How to weaponize LLMs to auto-hijack websites AI models, the subject of ongoing safety concerns about harmful and biased output, pose a risk beyond content emission. When wedded with tools that enable automated interaction with other systems, they can act on their own as malicious agents. Computer scientists affiliated with the University of Illinois Urbana-Champaign ( …
Right on! IMHO, straight thinkers have a hard time comprehending the devious mind, which leaves the bulk of us at a disadvantage security-wise (and is why the FBI recommends courses in criminal behavior and psychology of evil to its trainees). Having AI models that can act as digital twins for criminals should be a great help for training activities, and to evaluate the eventual robustness of preventative counter-measures. Hopefully, other models get to compete with GPT-4 in this, in the not too distant future.
Interesting point. Maybe it would be possible to harness this tool as an addendum to security checks. You set up your network, make sure you've done your best, the let loose the AI to find out what you missed and/or what you need to check.
Of course, this would mean that said AI was not on the Internet, available to all. It would have to be brought on-prem, thus deployed by contract, and only used in-house. All of which will never happen.
So it's the miscreants that are going to have fun with it.
And don't come with the "only incremental abilities" argument. It's an LLM, it's supposed to learn, isn't it ?
Could this be the true beginning of Skynet ?
> It's an LLM, it's supposed to learn, isn't it ?
Not really.
None of the LLMs mentioned are learning at the point of use - they all have long (and costly) learning phases, building up all the internal weightings, and then a separate deployment phase when those weightings are run against the (user) input texts.
The releases - i.e. GPT-1 to GPT-2 ... GPT-4 - are *because* the models are not learning during deployment, but instead at most feed back into the ongoing training phase, which then spits out the next incremental/step release (whilst the learning phase chunders on, with various tweaks, to generate the next release after that).
Of course, as They are watching every time you use their LLM, no doubt those interactions are used (somehow?[1]) to feed back into the training then, yes, the *family* of models may be learning from the experience, but aside from the slow release cadence (which does mean They can say the improvements are "incremental" and not continuous, just 'cos it is so choppy) the feedback specifically related to breaking into websites is diluted by all the other interactions being recorded - all the telephone fraud and homework cheating - so specific improvements in one area are hard to pin down.
Unless, of course, They are watching out for specific usage, such as cracking websites, and preferentially use those to improve the training of the next release, but They would never do that. Why would they want to?[2]
[1] that feedback itself is interesting to consider, as to use a session as training input you need to be able to give the session - better, parts of a session - weightings: was this a good or bad session? On what criteria?[2]
[2] You really want a nice clean success/failure criterion to make best use of a session as training input. For example, "did we manage to break into a website (yes/no)?" Oh dear.
Broadly correct, but slight correction: the primary difference in the GPT generations is the size of the network, not just the generational dataset. As things stand, GPT-2 was 1.5B weights, GPT-3 was 175B weights, and GPT-4 is suspected (leaked) to be 1.8T weights split in 16 units, of which only two (dynamically chosen) are active at once.
Shame the open source models aren't (currently?) up to the task.
It would be handy to be able to have an on-hand cracker (on a separated LAN, in a VM) that could be run against one's website test build, for a small website builder. You international conglomerates can afford to hire the pentest team, the rest of us just need something that can run the pentests automagically, even if it takes an overnight run instead of twenty minutes.
Bonus points if it tells us what went wrong, rather than just crowing how easy that website was to bring down.
Dunno why this was downvoted; it's certainly true. Kali comes with a lot of free scanning and penetration tools, and learning to use many of them is pretty easy. There are tons of courses available, free and paid.
That said, if someone's interested in checking their own sites or getting started in website / web-application security analysis or penetration testing, I'd refer them to OWASP in general and this list of DAST tools. A number of them are free.
> It would be handy to be able to have an on-hand cracker (on a separated LAN, in a VM) that could be run against one's website test build
Initially I thought so too, but then I realized that there still is no way (in time and money) you might afford to prepare against relevant (i.e. recent or somewhat sophisticated) exploits, if only because those exploits haven't been found/made public yet. Nothing changes.
To put it simply, you'll only be able to check against yesteryear's exploits, and that's about all.
Last but not least, AFAIK 99% of all breaches are due to people being too lazy to patch known issues. AI won't change that...
Shame the open source models aren't (currently?) up to the task.
I suspect you could achieve similar performance with a dedicated sparse transformer model. Web-technology languages (HTML, Javascript, etc) are all much more regular than natural language, so the parameter count is less important. Put more resources into context-window length and specialized training: train the model with e.g. OWASP resources, particularly on WebGoat/WebWolf transcripts and that sort of thing.
This research was using already-available models because that was the hypothesis: that at least some generally-available models could do this kind of thing.
Frankly, you can almost certainly achieve good results without even using a DL stack. Combine a fuzzer with a large HMM, for example, trained and tuned with human-labeled data (and the usual techniques such as backoff), and you'd probably do pretty well at hijacking a lot of sites. The interesting bit here is seeing how far you can get with off-the-shelf tools.
> aggressive parasites hanging around biting everything that appears.
Maybe, if any large body (looking at you, governments) actually gave a damn about *really* helping web security, they could permanently run just such a beast and send you a report about how and why it was able to break in, with mitigation suggestions.
As they would need to make more money out it (instead of just redirecting some of the money set aside for pointless shouting and waving of arms about online security) they could instigate fines, based upon income derived from the website, the sites didn't improve b some deadline.
Yes, people love it when the government mounts DoS attacks on their sites.
Not that this hasn't been suggested before. The thing is, it takes really very little effort, compared to development cost, to download and run, say, Zed Attack Proxy (ZAP) against an internal version of the site. Or even production, for that matter. If people can't be bothered to do that, what makes you think they'd read a report from CISA or whatever?
And for that matter, there are plenty of bounty-hunting skiddies doing this already, and not a few actual security researchers. Again, they often get ignored.
But the question is how comprehensive is it?
Disappointing that with so much information on what makes an insecure website so many are still built that way.
Why? Because it's still (in 2024) easier to write a poorly secure site than a secure site.
Not exactly most peoples idea of "progress."
As always in the security field, defenders need to understand offensive tactics.
So the defenders("white hat hackers") should indeed use these AI tools to try to break into the systems to be protected. Also see "red team".
Having said that, AI is still "worm intelligence"(based on complexity and my real-world testing results) and the advanced tactics will still be developed by humans.
AI is essentially a neat form of automation of existing stuff. All the problems such as "hallucination" and "posing as perfect" will apply.
For legitimate security researchers, using LLMs to "try to break into the systems" is almost certainly a poor use of resources. We have much, much better vulnerability-scanning and penetration-testing tools.
The point of this research isn't to show that LLMs are good at finding website vulnerabilities. It's to show they can do it at all, thereby serving as Yet Another tool for the lowest tier of attackers — the script kiddies.
This post has been deleted by its author
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
