Anthropic's magic code-sniffer: More Swiss cheese than cheddar, for now

OPINION In retrospect, calling it Mythos made it a hostage to fortune. Anthropic may have hoped that the name implied its AI code security model had mythical god-like powers, but there's an alternate reading. Another definition for Mythos is a set of beliefs of obscure origin which are incompatible with reality. That reality is …

[Pascal Monett]

"a set of beliefs of obscure origin which are incompatible with reality"

So, kind of like US politics at this point in time ?

[Anonymous Coward]

The fuzz

It's hard to understand how commentary about Mythos - as a way to productize security vulnerability research - can fail to mention alternate approaches. Anthropic spent $100M. Would it perhaps be cost-effective to spend $100M on fuzzing? (Yes, it would). Would it perhaps be much more cost-effective to spend $10M on fuzzing, and $90M on humans to fix discovered issues? (Yes, it certainly would).

How can an opinion piece about security ignore the current state of security?

[Flocke Kroes]

Re: The fuzz

Conventional wisdom: The software vendor can save $100M by not checking the code at all. Let the customers pay for the consequences.

[Clausewitz4.1]

Re: The fuzz

” Let the customers pay for the consequences.”

I believe you are in charge of Microsoft Windows security, sir

[P_Jamez]

Re: The fuzz

I have only heard it being mentioned once in the various media pieces about this. A estimate is Anthropic spent $100m on getting the results they just did. If one were to have paid humans $100m to find vulnerabilities, would they have done better?

[David Hicklin]

Re: The fuzz

I guess the difference here is that Anthropic spent $100m once but it can be used against every piece of newly generated codes over and over again.

Unlike the humans however, unless Anthropic spends another $100m it won't learn anything new as it goes along, so it will only find the old bugs - new ones will go undetected and we will be back where we started.

[Kurgan]

The Swiss cheese model of failure works less and less well the more the cheese tends to cheddar.

A quote worth remembering.

[Philo T Farnsworth]

Or, perhaps, milk to curdle.

[LionelB]

Unknown knowns

> It is very good at finding classes of vulnerability that humans know about, while not finding ones that they don’t.

How do we know there are classes of vulnerabilities we don't know about? (Asking for a friend.)

[Clausewitz4.1]

Re: Unknown knowns

” How do we know there are classes of vulnerabilities we don't know about? (Asking for a friend.)”

I wont be your friend, but I can answer that: human intelligence and creativity. You nailed why AI will never beat humans.

[LionelB]

Re: Unknown knowns

Your human intelligence and creativity tells you that you know about the existence of things that you don't know about… okay… I think the correct term for that is "religion". Here's sincerely hoping the AIs don't follow us down that route (but never say never).

[HereIAmJH]

Re: Unknown knowns

Out of curiosity, how often are we finding new classes of vulnerabilities?

The answers I get from google are that the most common and dangerous vulnerabilities have been around for decades. Cross-site scripting, SQL Injections, various bounds checking errors, etc.

I suspect the way we will find new exploits is to reliably fix the ones we know about and criminals will then focus on new vectors of attack.

[Jedit]

"a set of beliefs of obscure origin which are incompatible with reality"

Whenever I see Mythos I automatically think of the Cthulhu Mythos, which is basically the HP Lovecraft Expanded Universe - although the size of the Universe was but one of the many things that unnerved him.

So: alien, unknowable and the merest contact with it can have deleterious effects on your mind. Pretty accurate name for an AI, really.

[tonkei]

Re: "a set of beliefs of obscure origin which are incompatible with reality"

Whereas mythos just brings me back to a very drinkable greek lager!

[vtcodger]

Of foxes and henhouses

One modest quibble. Indeed, aircraft safety is a terrific model for responsible (e.g. not free market) handling of an important issue. But there is one relatively new class of airline safety issues that "classical" airline safety didn't handle well -- the Boeing-737Max fiasco. Why? Because the ultimate cause of the two deadly crashes looks to be an attempt to avoid the (substantial) costs of doing things "right" by pretending that no additional training was required for 737 pilots transitioning to the 737max.

The take away. Don't let the foxes guard your henhouses.

[Anonymous Coward]

Re: Of foxes and henhouses

Or appoint said Foxes to Self-Certify to the Hen House auditor.

Still plea bargain, pay a small fine and all is good in the end.

[JefcoPDrinkbeer]

(Aviation safety) "There’s no new undiscovered class of failure waiting in the wings."

Oh dear ...

[tiggity]

Beer

"In retrospect, calling it Mythos made it a hostage to fortune. Another definition for Mythos is a set of beliefs of obscure origin which are incompatible with reality."

It's also a Greek lager (pretty unexceptional, but served cold in a Greek beachside bar on a hot day it can be refreshing). I may work in IT but it will be a long while before hearing "Mythos" makes me think of "AI" and not beer.

[Dr Dan Holdsworth]

Aircraft are a poor analogy here

Modern jet aircraft are a purposefully very simple design; a streamlined cylinder with wings at the balance point and a tail at the back. This is easy to model but is not the most fuel-efficient design that can be made.

The most fuel-efficient flying designs are flying wings, but these lack cargo capacity. Adding cargo capacity turns these into blended wing-body designs, but the problem here is that this is much more difficult to model and may be prone to failure modes such as flat spins or similar unexpected things. An example of this sort of failure was an early hang glider model called the Gyre; the wing design here would lock into a dive at a certain angle of attack and once in this dive it was impossible to get out of the dive without a parachute.

This is the sort of problem that AI systems might be quite good at finding and fixing, if only we could trust the things.

[David Hicklin]

Re: Aircraft are a poor analogy here

> This is the sort of problem that AI systems might be quite good at finding and fixing, if only we could trust the things.

Sounds like the perfect breeding ground for a 737-Max scenario.

One other issue with the flying wing is that few of the passengers (or indeed any) will have a window to look out of. Sure you can put vid screens up but they it is a big psychological barrier to overcome. That and sitting alongside the fuel tanks*

* yes I know some planes have some under the fuselage, but the majority of it is in the wings.

[Rgordon]

There is a very obvious reason why these exploit/bug detecting AI models are being tested on mature, open source projects... They all have hundreds, if not more, serious/severe/critical bugs unresolved on places like Github. Those "critical" bugs are often unresolved because it is such a unique edgecase, or is only exploitable in unlikely scenarios/configurations, that everyone who actually uses the software, is never effected by it.

These bugs are what theyre "finding". Wouldnt a few admins for those codes anthropic "reported the critical bugs to" have come forward to confirm the efficacy of their claims?

It was wild reading your article. It started off strong, then pivoted to AI-utopianism. Like you know those bug submissions on Github? They have been public since the software was first published on a repository... If AI is only good at finding already understood bugs/exploits... And those bugs/exploits have been public information for over a decade in some cases... Then what is anthropic actually providing? Beyond calorie-free hype fodder for AI-grifters, and the rubes they have duped, to regurgitate.

Wake me up when a software used for a critical system is allowing an AI code agent to debug, maintain, compile, and publish prod code with no oversight.

[SP2000]

Queue the M&A

Regardless of the current capabilities of Mythos, or an adjacent model, the clock is already running on the stitch ups between Vuln. Mngt vendors and the model makers. The usual round of strategic partnership, acquisition, and eventual consortium model with defacto standards as to the data structure involved in model query and output.