Closed models refuse to help researcher swat Linux bug

The guardrails that prevent closed-source, frontier models from aiding threat actors have turned into handcuffs that prevent those bots from helping to find and fix serious vulns. Daniel Fox Franke, a security researcher, was recently trying to track down the source of a segmentation fault in ripgrep, and found OpenAI's GPT-5.6 …

[Paul Crawford]

As for other aspects of IT, open-source (or open-weight) gives you a degree of isolation from what others want to impose upon you for dumb or expensive reasons.

[Anonymous Coward]

The way guardrails get in the way of security is why the US Government terminated their $200 million contract and deemed Anthropic a supply chain risk. See also why Hugging Face used a Chinese open weights models to fight off a recent chatbot attack.

[Anonymous Coward]

Anyone can read that sentence and see how it doesn't make sense, but thanks for parroting the administrations talking points.

If the guardrails get in the way by making the model useless, then it's hardly a supply chain risk. The model can't accomplish anything, so what's it putting at risk? This was petulance on display, no need to invent something more complex than that.

[Expect Great Things]

The nuance is that the models are, of course, normally quite useful. The supply chain risk comes from the fact that the guardrails prevent them from performing specific tasks.

It’s natural to assume that the administration was especially alarmed at the possibility that the models that were being widely adopted for their perceived usefulness would subsequently balk at performing morally bankrupt tasks (the very foundation of the administration’s agenda). In practice, the guard rails have proven to be much cruder.

[Tron]

The US are banning a lot of 'not made in the US' tech.

Kimi K3 is, whisper it, Chinese.

Governments love to ban stuff, especially foreign stuff. Plus open models might be considered a financial threat by Trump's chums in the AI industry.

Plus the pro-'guardrails' lobby will remind you to think of the children.

[Slant Four]

Re: The US are banning a lot of 'not made in the US' tech.

In top of that (money for their back pocket), we need to consider that the regime in Trumpistan want to replace/ban all the foreign tech that their citizens use for good 'ol "made in the USA" cause then they can spy on their citizens with backdoors in the US "stuff".

Then every citizen will end up with a dossier on what they watch, what they email, what they browse etc etc so the regime in Trumpistan can determine who might, for example, be a member of that mythical terrorist organization Antifa, might not be aligned with good 'ol American Christian "values", might not get their news and information from regime approved new outlets, might not have the correct binary sex orientation etc etc. Basically the Stasi on steroids and given the attitude of most of the Tech Bro's, they will willingly support the regime in Trumpistan to accumulate said dossiers.

if there is any spying to be done, it needs to be good 'ol American spying.

Bluck

[that one in the corner]

more investigation to do before I can think about shipping anything to LKML

Are we SURE this guy was using AI?

Aren't AI users supposed to report everything they've "found" immediately to the mailing list, without any of this annoying "investigation" malarky?

Is he trying to break the class's curve? ' Cos the Cool Kids won't thank him for doing that.

[meski-oz]

Just wow

However did we even find and solve bugs before AI?

(That's a rhetorical question, BTW)

[AtomicDog]

Re: Just wow

Much more slowly, albeit much more methodically.

Using LLMs is more like using an Uzi to shoot a target rather than learning how to use a sniper rifle.

[Anrtryg]

Re: Just wow

"Very American. Fire enough bullets and hope to hit the target"

[Anonymous Coward]

Re: Just wow

We did that too with fuzz testing.

Now these bullets have a little more focus (though artificially less than they should).

[vtcodger]

Re: Just wow

""Very American. Fire enough bullets and hope to hit the target"

Or just fire off a burst or two, pick the area with greatest density of bullet holes, and paint a target there. That's known as 'Texas Sharpshooting'

[VoiceOfTruth]

Re: Just wow

Sometimes those bugs are not found because nobody is looking for them. There is a lot of old code which works, which was committed years ago. It passed a few eyeballs at the time. We see reports in The Reg quite often about hardcoded passwords in Cisco (too convenient to be called bugs, so they are more likely to be backdoors). If those are genuine bugs, then nobody was looking for them.

I have occasionally mentioned my view that 'many eyes makes bugs shallow' is a fallacy. Those eyes have to be competent enough to understand sometimes very complex spaghetti. It's the same with OpenSSL. Let's face it: average programmers are not going to spot complex bugs in OpenSSL, even less understand the algorithms or if the algorithms have been coded correctly. They might spot a bug by luck, but that is not the same thing.

[DoctorNine]

I'm old. I remember the 'I Love Lucy Show'. Whenever Ricky would get overwhelmed by crazy American stuff, he would put his head in his hands and go, "AI AI AI..."

I never realized how prescient that Cuban singer actually was.

[Anonymous Coward]

Last sentence needs correction

"The US government has yet to articulate a coherent policy."

FTFY

[stiine]

" OpenAI's accidental attack, "

Accidental? There's not a chance in hell it was 'accidental'.

[thosrtanner]

This bit

"I never encountered any refusals from Sol itself: it knew that most of the classifier trips were inappropriate and always continued working with me in good faith to work around the problem."

AI doesn't have agency. How can it work with someone in good faith?

[osxtra]

Talk to the Hand

"From my perspective, an uncooperative tool is simply a broken one,"

Ah, I do miss the 'good old days' when we didn't anthropomorphize our tools.