AI kept 15-year-old zombie vuln alive, but its time is drawing near A security bug that surfaced fifteen years ago in a public post on GitHub has survived developers' attempts on its life. Despite multiple developer warnings about the 2010 GitHub Gist containing the path traversal vulnerability in 2012, 2014, and 2018, the flaw appeared in MDN Web Docs documentation and a Stack Overflow …
LLMs need banned.
The entire so-called training concept is broken.
You need to be an expert to know if the result is rubbish. Hallucination is a misleading anthropomorphic marketing excuse.
They use too much power.
The related chatbots for customer service are an insult and damaging to mental health.
"This experiment shows that the popular LLM chatbots have learned the vulnerable code pattern and can confidently generate insecure code snippets, even if the user specifically prompts them for a secure version [...] Thus, simply accepting LLM output is not a reliable thing to do."
It still puzzles me that many people do not understand that LLMs cannot be reliable. LLMs cannot reason and completely fail at logic. They often do produce very convincing (and correct!) output but they do not understand any of it - a LLM doesn't understand the word "vulnerable" or any word for that matter, it doesn't understand what a vulnerability in a code is etc. All it knows is the probability of the next word in a given context (while it does neither understand "context"). Expecting any logical reasoning from an LLM is like believing in miracles - prone to fail.
> All it knows is the probability of the next word in a given context ...
I don't entirely disagree with your conclusions, but the "next word" here is not correct. Transformer architectures essentially process text in parallel rather than sequentially, based on the entire available context.
> Expecting any logical reasoning from an LLM is like believing in miracles - prone to fail.
Then again, the same might be said for humans who are, of course, also "prone to fail" (some more than others). Like, for example, the (human) author of the original bug in the article, and the lazy (human) sods who copy-pasted it all over the internets without bothering (or failing) to understand the code.
The real point at issue is, surely, not whether LLMs/some-other-AI are "good at logic" or "understand" anything (whatever that means - and no, it's not obvious) -- but rather whether they do the job at hand better or worse than humans in the field. So sure, don't rely blindly on the soundness of AI-generated code without review, validation and thorough testing - because, after all, you wouldn't do that with human-generated code... would you?
"Probability of the next word" is a harsh abstraction of what actually happens. Although, I'd argue that being more specific is irrelevant for my point: LLMs are unreliable.
I agree with you, the question is what are LLMs good at. Logic isn't one but there already are many good use cases. All I'm saying is what you did as well: don't rely blindly on the soundness of AI-generated code, legal advice, abstracts etc. And yes, the same is true for human-generated stuff.
> "Probability of the next word" is a harsh abstraction of what actually happens. Although, I'd argue that being more specific is irrelevant for my point:
Yes, it was not my principal point, really - although it does speak to a depressingly naive (mis)understanding of how LLMs actually function, by people who ought to know better (not saying you're one of them!) There is a tendency to deride LLMs (and indeed machine learning in general) as "just" statistical machines. Sure, they are that; it's the "just" which is to my mind (ahem) unjustified: indeed there is accumulating evidence that biological cognition (and, who knows, perhaps intelligence) may well rest on statistical foundations; see e.g., Predictive Coding theory and the Free Energy Principle.
> ... LLMs are unreliable.
Yup. And my point (which you clearly agree with) is that so are humans: unreliability is not a necessarily an appropriate stick with which to bash AI - unless, in the interests of fairness, you are prepared to bash us humans with the same stick.
Worth mentioning that some AI (ML, if you prefer) is getting much, much better at logic; some of the new "reasoning" models such as DeepSeek, which use reinforcement learning recursively within the transformer architecture, are actually rather impressive1 (try them). It's not as if logic is some magic sauce that only we blessed humans are anointed with - in fact frequently we aren't.
1I am a mathematician/statistician by profession - I am not easily impressed by bad reasoning, amongst other things.
"unreliability is not a necessarily an appropriate stick with which to bash AI"
Of course it is!
"- unless, in the interests of fairness, you are prepared to bash us humans with the same stick."
Yes, I am. And I do. Your point? Have you ever seen and paid attention to TheGreatUnwashed? Would you buy a used car from J. Random Stranger?
The fact is, so-called "AI" is not reliable, you cant trust it. We're spending billions (trillions before we're done?) of dollars on something we can't trust! Not only can't we trust it, there is no way it will ever be trustworthy, by it's very nature. But we're sinking this amount of time and money (and energy) into it. KNOWING that it can't ever work. What kind of fucked up thinking is going into that particular investment?
Yes, I know, separating fools from their money is an ages-old investiment strategy, but C'MON! This is taking the mick ...
<shrug>We (for some values of "we") spend billions on paying crap, unreliable human coders with huge energy footprints - who also need their work reviewed by non-crap, reliable human coders.</shrug>
For what it's worth, I wasn't actually advocating for spending that kind of money on any technology that doesn't merit the expense (and no, I don't think LLMs in their current state do).
Frankly, I think I'm just bored to tears by the endless repetitive shrillness, and was trying to say something slightly different than the usual huff-pufferry, to broaden the perspective a little, even. Yes, we know {username here} is very, very angry about quote-AI-unquote because they've posted the exact same thing as a thousand other posters who are very, very angry about quote-AI-unqoute. It's become a tedious pissing contest about who can get the most upvotes for being the most self-righteously angriest. In the interests of making Reg forums readable, can we please stop doing that?
Oh, and my point was (I thought it was obvious) that these very-very-angry-about-"AI' commentards rarely concede that, actually, human coders, or, dare I say it, human intelligence, can, and frequently is, every bit as crap as and unreliable as qoute-AI-unquote1. In other words, can we at least have a level playing field for crapness and unreliability?
1I know this, because in the day job I spend way too much time reviewing the code and essays of crap, unreliable students, and way too often I swear to god I'd rather they cheated and used quote-AI-unquote.
The difference is that we can remove obviously bogus lines of "research", but we can't remove idiotic humans, nor those who take advantage of them.
We can, however, encourage other humans to ignore the obviously idiotic trains of thought.[0]
Arguably, your role as a teacher REQUIRES you to review the work of crap, unreliable students. Goes with the territory.
[0] Not that I'm holdin' my breath, y'unnerstan'. Consider, for example, all forms of religion, and how long they have been used to separate fools from their money ...
We can agree there.
It does feel as if the strategy of the industry is to create artificial idiots to encourage biological idiots (like there aren't enough of those) to part with their money.
Having said which, the technology is advancing apace. From a historical perspective, a mere twenty (maybe ten) years ago the current crop of AI agents would have seemed magical. So of course we moved the goalposts, raised the bar (insert metaphor). Some of the current rhetoric surrounding LLMs is eerily reminiscent of how we sneered at automatic human language translation not that long ago - a problem that today is virtually solved. LLMs will not be the last word - but human exceptionalism is strong. There will always be problems which of course are just too hard, which we are not holding our breath to see in our lifetime, like, dunno, controlled nuclear fusion, or heavier-than-air flight.
unreliability is not a necessarily an appropriate stick with which to bash AI - unless, in the interests of fairness, you are prepared to bash us humans with the same stick.
If LLM are (mis)used for tasks that require reliability, such as coding, I find it indeed the appropriate stick to bash it. And I will do the same with its human (ab)user. There is no fairness - AI is a tool and I treat and judge it as such (at least, until AI is comparable to an average human brain but I'm afraid that is not going to happen in my life time).
Regarding logic, LMs are getting better at keeping up the illusion of logic - still a far cry from actual logic from what I've seen (add a couple of steps and it fails). I'll happily dive into what you suggested though and get impressed. It's not that I believe AI cannot/can never act logically; but current LLMs by their very nature most likely will never.
I'm slightly intrigued about that "illusion" of logic - an argument (in the mathematical sense of a series of propositions) is either logical or it isn't. (And again, humans are not necessarily particularly good at that.)
I would, though seriously recommend you try the likes of DeepSeek (it is not a "straight" LLM), and tell us what you think.
My own experience with DeepSeek R1 was that I threw a hard statistical problem at it in an area in which I have expertise. Its response was lengthy (several pages), highly detailed, technically correct and included relevant citations from the literature. The most impressive thing was that it explained its (logical) reasoning as it went along, including backtracking when it hit a dead end and trying a different approach - much like a human expert, in fact. Apart from some slightly stilted language (irrelevant to the problem at hand) it could easily pass a Turing test for a human expert in that particular field. Not sure how it might perform in a different domain.
What I call illusion of logic is that some LLMs by now learnt to solve logic problems that involve planning of some steps to solve. But increasing the complexity slightly renders them useless - as opposed to an average human. An example of this is something like the Tower of Hanoi. A human, once they understand its logic, can solve it easily no matter what number of blocks you add, whereas transformer architectures quickly failed. It is like some humans that fail at logic: they can learn how to solve a certain problem but are unable to deduct the underlying principle and hence fail when the problem slightly changes. At first it appears as though they think logically but with a slightly different problem the illusion is busted (and yes, I do know a few such people).
Or take Chollet's ARC-AGI. Till end of 2024 LLMs failed miserably. But then Open AI (model 03?) managed to solve about 75 per cent. And now with ARC-AGI-2 they fail again (whereas humans easily reach 60 per cent).
DeepSeek still has to wait until after this trip - employer blocks access.
Right. It feels to me that you're not really talking about "logic", but rather about problem-solving - creativity, perhaps. Humans, as you imply, vary wildly in their abilities at problem-solving tasks, and these abilities are highly inhomogeneous across humans and tasks.
And problem-solving is not a straightforward issue. Here's an example from my own experience. I am a Sudoku addict1. Sodukus are actually almost trivially easy to solve from a "logical" standpoint - you just brute-force them in a fairly obvious way2. Unlike some other puzzles/games, Sudoku is nicely bounded - there are no combinatorial explosions. (For fun, I once wrote a C program to do this - it solved the most fiendish Sudoku in fractions of a millisecond.) Of course that is not at all how humans solve Sudokus - the brute-force method would be a giant pain. Rather, (expert) humans accumulate - by experience - a set of useful heuristic short-cuts which, as it happens, are closely tied to our visual pattern-scanning skills.
I think that last point is a significant hint as to why, e.g., LLMs tend to be poorer than humans at problem-solving. Those useful Sudoku heuristics are closely associated with our (evolved) physical embodiment in, and interaction with, the real world. This is not an original point, but I believe that machine problem-solving (on real-world tasks) will only catch up with human abilities if and when it too is "embodied" in some sense - when it is able to interact via sensorimotor loops wimicrosecondsth the world, as humans do, and thus accumulate experience. The only interaction currently available to LLMs and ML in general, at this stage, is a one-way street (input) via the proxy of human text - and that doesn't quite seem to cut it3.
If I find the time, I'll try asking DeepSeek to come up with strategies for solving Sudokus - that might be revealing... or perhaps not; in this case, I suspect the human text proxy is probably sufficient.
1My son once gave me a book of 300 fiendish Sudokus for my birthday. It destroyed my life for weeks, turning me into a hollow-eyed, antisocial, sleep-deprived husk (honestly, that's not me).
2Start filling in numbers from top-left, from the first open cell of the grid; try numbers 1-9 sequentially until you find a valid number, then move on to the next open cell (row-major or column-major, doesn't matter); if there is no valid number possible for the current cell, backtrack to the previous cell you filled, and increment the number there until you find a valid number. Repeat till done.
3Robotics offers a route here, although the current state of robotics would seem at this stage to be way too primitive to be effective as regards problem solving. No doubt that will change, though on what time scales I wouldn't even hazard a guess.
It's not new.
I've worked with people who repeatedly wrote insecure code, sometimes not even in their project. They were terminated on the suspicion of working for the Chinese government. The vulnerabilities, if known, would have been the first step to gaining complete system control.
I've also worked at places where the director of infrastructure said insecure code was fine. I didn't stay long enough to figure out who was behind it
We have trained the next generation of programmers to rely on LLM code generation for "productivity" reasons (greed, profits, etc..) Instead of being taught how to program, they are being taught how to prompt. Some of the biggest players in the IDE and OS world are pushing every type of AI in just about every programming tool they can shoe-horn "AI" into. "Don't think, just use your assistant." Let it do the thinking and work for you -- like magic!
The three-spinners have certainly woven a wicked fate for us all. A digital future with built in vulnerabilities -- as a "feature". It's cliche, but what could possibly go wrong?
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
