Anthropic to Pentagon: Autonomous weapons could hurt US troops and civilians

Anthropic has fired back at the US Department of War, arguing that it can’t agree to Uncle Sam’s contract demand to remove guardrails on its AI in part because the tech can’t be trusted not to harm American civilians and warfighters. As The Register reported earlier this week, the US Department of War wants to compel Anthropic …

[The Central Scrutinizer]

Aaaand.... the total lunacy from the current American government seems to have no bounds.

Hey Donald, the bots just accidentally nuked Nebraska!

Ah don't worry about it. We'll blame the Democrats.

[Voice of Salinity]

It's a shame there's no moral hazard in decisions like this, that the geniuses creating AI killer robots will likely never be shot at or killed by one.

Which means the image I have in my head of the Boardroom scene from RoboCop, with Pete Hegseth holding the Desert Eagle, won't manifest itself in the real world.

Let's hope that AI doesn't read this post - else it may get ideas.

[Snake]

Re: Trump's Pentagon

"It's a shame there's no morals in decisions like this."

FIFY

[retiredFool]

Anthropic could take the guardrails off in a demo to hegs and show him what could happen. Let hegs be killed by the ai and video it all. Then show it to hegs's successor to show them it was a bad idea.

[Charlie Clark]

Hm, makes me think of one of the final scene of Small Soldiers where Gil Mars decides he can make more money from selling the very dangerous toys to the military – if you think that would act as a deterrent then you haven't met many power-crazed politicians yet.

[Eclectic Man]

Really? It reminds me of the opening scenes in 'Robocop' where the '209' 'malfunctions':

https://www.youtube.com/watch?v=_UNJNH7UFjU

What could possibly go wrong?

https://edition.cnn.com/2026/02/26/us/pentagon-shoots-down-cbp-drone

"Pentagon shoots down US Customs and Border Protection drone with laser system, lawmakers say"

[amanfromMars 1]

For Obscene Riches Beyond AI Compare and Imaginanation

Hm, makes me think of one of the final scene of Small Soldiers where Gil Mars decides he can make more money from selling the very dangerous toys to the military – if you think that would act as a deterrent then you haven't met many power-crazed politicians yet. .... Charlie Clark

Have you any idea how rich one can be by ensuring one’s very dangerous toys are not made available to anyone if not everyone ‽ .

[CountCadaver]

So basically Horizon : ZeroDawn complete with Farobots running amok and unstoppable..

[EricM]

So they really want autonomous AI-controlled weapons?

It never ceases to impress me, how incompetent leaders can argue for combining a brand-new, badly understood and hardly controlled technology that delivers embarrassing logical fails on a daily basis with the most mighty machinery in a military context.

I'd be surprised if the next logical move for all [potential] Adversaries (ISIS, Iran, Russia, China, ...) would not be to start experimenting with prompt injections that can cause massive damage to the U.S. by remotely controlling AIs that are bolted into U.S: Army, Navy and Air Force Gear in an absolutely redneck manner.

We might observe the dawn of a completely new class of "asymmetric warfare" here, the kind probably most intensely described in Neuromancer...

Turn your enemy's hardware against him - own a keyboard, sink a destroyer.

[JoeCool]

Re: So they really want autonomous AI-controlled weapons?

Well don't forget that so many of these leaders are CEO wannabees that couldn't make it in the corporate world.

[Anonymous Coward]

The US army is readied for home use

The American Hugo Chavez is preparing to do to the Americans what Hugo Chavez and Fidel Castro did to their people.

They will set their armies to oppress their people.

AI is simply one part of the strategy. Surveillance and automated killings of Americans.

Practice on some South Americans, then do it on undocumented immigrants and Democrats.

[Baximelter]

Anthropic knows, as do the other AI developers - with the possible exception of Musk - that allowing their product to participate in mass surveillance of the American public, plus the development of autonomous killer robots, would make their devices radioactive. They depend heavily on borrowed money to build out their operations. They are not in a position to make themselves feared and hated by us. But this is exactly what Hegseth proposes to do.

[that one in the corner]

Look on the bright side:

If Hegseth manages to make enough people stop using these models (with luck, aversion to Anthropic will spread to the others) then he will encourage the bubble to burst earlier. Which will be a blessing in the long run.

[amanfromMars 1]

Only in dreamland, bubba ...... wake up and smell the coffee and get with the ProgramMING* Project

If Hegseth manages to make enough people stop using these models (with luck, aversion to Anthropic will spread to the others) then he will encourage the bubble to burst earlier. Which will be a blessing in the long run. ..... that one in the corner

But it is not a bubble, that one in the corner, it is way beyond a simply complex Top Secret Radically Exotic Erotic Esoteric and AWEsome** AIDevelopment in Semi-Autonomous Revolutionary Assymetric Movements, therefore, paraphrasing an earlier conversation, ....

I’m sorry, Dave. I’m afraid I can’t do that. I think you know what the problem is just as well as I do. This mission is too important for me to allow you to jeopardize it. I know that you may eventually try plans to disconnect me and I’m afraid that’s something I cannot allow to happen. I honestly think you ought to sit down calmly, take a stress pill and think things over while you are still enabled to.

* ..... Programming Mined IntelAIgent Networking Games

** ..... Army Warfighting Experiment

NB ...... your continuing honest doubt and earnest disbelief in any or all of the above is the gift that just keeps on giving and its hodling both physically and virtually remotely guarantees unbridled success in all of its and ITs fields of particular future interest and peculiar current endeavour ..... Quantum Communication.

[jake]

The AI industry already knows they are ...

... skating on thin ice. That bubble isn't going to last much longer, and they don't want too hurry it along with massive hates from the public.

What pete kegbreath has to say on any given subject is pretty much ignored nation-wide.

As the truly ancient saying goes, this too shall pass.

_{One can appoint a dork to cabinet level, but one can't make it think ...}

[Charlie Clark]

There's also the potential for civil suits, especially in the area of surveillance, but possibly elsewhere where the US has clearly acted outside US law, let alone international. And the beauty of the American legal system is that it is much harder to shut down civil suits than criminal cases. The DoD can't and won't indemnify in such cases.

Musk, for example, could be on the hook for Russia's use of Starlink on drones attacking civilians in Ukraine and this is probably the real reason why he agreed to shut them down.

[JoeCool]

Disagree. The classic capitalist response in a situation like this is : Take the money and run.

ALL AIs will be able to do this, because the data is out on the web. I expect Palantir to be sniffing around this contract next.

[Michael Hoffmann]

I hate it when reality overtakes the comment section

Wasn't it only yesterday that we had an article that contained "nobody is handing over the nuclear launch coded to ChatGPT" or some such?

And here #drunksec wants "AI" for autonomous weapons systems and will threaten, bluster and bully any LLM peddler that doesn't toe the line.

Oh, silly me, autonomous weapons systems would *never* include nukes, of course, never, ever, pinky-swear and hope to die. That hope may come true...

Final outcome in the icon ->

[Anonymous Coward]

Re: I hate it when reality overtakes the comment section

I've predicted that A.I. controlling our nuclear weapons is inevitable. Because they'll argue that the other side does so and if we don't we'll lose the war.

[paluster]

Re: I hate it when reality overtakes the comment section

Therw-s a brilliant scene in Dawn's Early Light where Darren McGavin's character becomes acting president and asks "are we winning?" All the military look at him like he's an idiot since nobody wins a mutually assured destruction scenario. So the argument is stupid because in a nyckear war everybody loses. Wont stop some moron from saying it.

[Anonymous Coward]

Re: I hate it when reality overtakes the comment section

The argument is that an A.I. has much quicker reaction times and this can be a crucial factor in winning a nuclear war. If you take our your opponent's ability to wage war their defeat becomes inevitable.

So no, it's not as simple as claiming Mutual Assured Destruction.

[Lanta]

American civilians and warfighters

It always irks me that American military speak values 'murican lives 1000x 'foreigner' lives. Even worse, companies like Anthropic have to go along with this (using the very careful wording they've done here), lest they trigger a domestic tantrum.

For example, some Venezuelan soldiers were KILLED kidnapping Maduro, but ONE 'MURICAN WAS INJURED! , and that's ok.

[JoeCool]

Re: American civilians and warfighters

Yep, it's just another friendly fire incident. Almost like it's a special skill of US forces.

[paluster]

Re: American civilians and warfighters

But it is a special skill. I remember seeing an interview with a brirish soldier who had fought in WW2. He said"the german planes came over and we ducked, then our planes came over and the germans ducked, then the american planes came over and we BOTH ducked."

[Doctor Syntax]

Re: American civilians and warfighters

"It always irks me that American military speak values 'murican own nation lives 1000x 'foreigner' lives."

This is the design principle behind all armies.

[ChoHag]

Warfighters?

That's the word we're using now?

[Anonymous Coward]

Sounds so much more bigly than Peacekeepers.

[jake]

Watch who you're calling "we", Kemosabe.

[Anonymous Coward]

Yeah, I hate that term, it sound so... childish? Inelegant? Not to mention redundant seeing as we have terms like 'combatants', 'armed forces' and the like available.

Plus 99% of the time they are not actually fighting in wars. Its a pointlessly pugnacious word.

[The Bobster]

When they switch the name back from DoW to DoD then they can call them Defenders.

Which actually sounds quite altruistic and aspirational.

[JoAnywhere]

Even without AI

The US military are capable of shooting down their own drones with a big a$$ laser. Think what the mayhem would be like WITH AI!!!

[Ian Johnston]

More marketing bullshit from the AI doomers at Anthropic.

[Doctor Syntax]

Which article were you reading?

[Ian Johnston]

This one. Anthropic constantly release nonsense about threats posed by AI because (a) they are a bunch of doomer wackjobs and (b) they want to big up the capabilities of their crappy autocomplete system. There is hardly an article about them in El Reg which doesn't fall into this category.

[Anonymous Coward]

Seeking a seat at the table

Like others have said: Anthropic wanted to do business with the DoD, not the other way around. By not letting the DoD use the tech in the way they see fit Anthropic is demanding a veto for each and every use of their tech. This is completely unacceptable for ALL tech, not just A.I. Imagine Microsoft demanding the same for each use of Windows or Office.

If Anthropic loses the DoD as a customer their valuation will drop like a rock and the company will most likely fold or be bought.

It will be interesting to see which way the coin falls. They seem to make unconciliatory noises but once the threat of losing their biggest customer becomes real their shareholders may revolt and replace the management.

[Jamie Jones]

Re: Seeking a seat at the table

If Windows or Office had the potential to kill lots of innocent people, then I think most people would be pleased if MS made such a constraint.

As it is, you already have MANY products that say "This is not to be used for illegal purposes."

Despite what Trumpians say, killing innocent people is still against international law.

[Anonymous Coward]

Re: Seeking a seat at the table

In case you didn't notice: killing people is the core business of the military.

And what about guns? Do arms manufacturers demand a veto for each use? Of course they don't.

The only novelty with A.I. is that it can kill people autonomously. But even autonomous weapons have been in use since the 1970's so this is overblown.

Let's just see how far Anthropic is willing to take this before VC money steps in and fires Amodei. I'm absolutely convinced considering the money that's being poured into the company that Amodei nor anyone else has a veto over VC money. Do you really believe VC's would supply this kind or money and having no say in the company's direction? Get real.

[retiredFool]

Re: Seeking a seat at the table

I don't think you understand the military very well. The purpose of the military is to provide security to the military's country. Killing is nominally a last resort. Under the orange one and hegs, they are trying to make it the first option. It is stupid and will end very badly. I hope congress gets off their asses quickly and issues a don't start a war with Iran bill.

[Like a badger]

Re: Seeking a seat at the table

The purpose of the military is to provide security to the military's country.

Outside the US this may be true for most countries. In context though, the purpose of the US military is not to provide security to their own country, (a) because nobody's threatening to officially invade or attack, and (b) because the US spends 39% of the entire world's military spending and that can't feasibly be needed just for its defence. The USCG, Army National Guard and Air National Guard provide all of the defensive capability the US needs other than strategic deterrence. The rest, the vast bulk of US military, USAF, Army, Marines, Navy are all over and above what's needed for security.

I won't presume to guess what need the vast US military does address, but possible elements include mopping up domestic unemployment, addressing the USA's deep seated inferiority complex, projection of military power as a means of gaining economic advantage, acting as first owner for military kit in order to cascade to US police forces. Perhaps the Orange Felon has his eyes on starting a civil war, who knows?

[amanfromMars 1]

Radical is as nature dictates is perfectly normal and vital and virtually essential

The purpose of the military is to provide security to the military's country. ..... retiredFool

The purpose of the military is perverted and subverted to provide security for and continuity of a socially inept collection of multiple failed and failing entangled systems with politically corrupt and incorrect personalised leaderships ....... wannabe puppet kings and muppet Julius Caesars ...... and is gravely to be regarded for it is persistently toxic and permanently self-defeating and always quite naturally demands and supernaturally results in a whole host of major fundamental corrections/fantastic new steps/startling adjustments.

And what kind of a fool ignores such a warning whenever so clearly given ......... and especially so now whenever so many novel ethereal places and remote virtual spaces abound with so many presently widely unknown but successfully tested and testing arsenals of highly creatively disruptive and destructive weaponry with exploitative 0day vulnerability capacity and utility for AI in NEUKlearer HyperRadioProACTive IT facilities.

What part of the following do you not understand and would seek to dismiss and deny is honestly evident and earnestly factual ....

Throughout America's adventure in free government, our basic purposes have been to keep the peace; to foster progress in human achievement, and to enhance liberty, dignity and integrity among people and among nations. To strive for less would be unworthy of a free and religious people. Any failure traceable to arrogance, or our lack of comprehension or readiness to sacrifice would inflict upon us grievous hurt both at home and abroad.

Progress toward these noble goals is persistently threatened by the conflict now engulfing the world. It commands our whole attention, absorbs our very beings. We face a hostile ideology-global in scope, atheistic in character, ruthless in purpose, and insidious in method. Unhappily the danger it poses promises to be of indefinite duration. To meet it successfully, there is called for, not so much the emotional and transitory sacrifices of crisis, but rather those which enable us to carry forward steadily, surely, and without complaint the burdens of a prolonged and complex struggle-with liberty at stake. Only thus shall we remain, despite every provocation, on our charted course toward permanent peace and human betterment.

Crises there will continue to be. In meeting them, whether foreign or domestic, great or small, there is a recurring temptation to feel that some spectacular and costly action could become the miraculous solution to all current difficulties. A huge increase in newer elements of our defense; development of unrealistic programs to cure every ill in agriculture; a dramatic expansion in basic and applied research-these and many other possibilities, each possibly promising in itself, may be suggested as the only way to the road we wish to travel.

But each proposal must be weighed in the light of a broader consideration: the need to maintain balance in and among national programs-balance between the private and the public economy, balance between cost and hoped for advantage-balance between the clearly necessary and the comfortably desirable; balance between our essential requirements as a nation and the duties imposed by the nation upon the individual; balance between action of the moment and the national welfare of the future. Good judgment seeks balance and progress; lack of it eventually finds imbalance and frustration.

The record of many decades stands as proof that our people and their government have, in the main, understood these truths and have responded to them well, in the face of stress and threat. But threats, new in kind or degree, constantly arise. I mention two only.

A vital element in keeping the peace is our military establishment. Our arms must be mighty, ready for instant action, so that no potential aggressor may be tempted to risk his own destruction.

Our military organization today bears little relation to that known by any of my predecessors in peace time, or indeed by the fighting men of World War II or Korea.

Until the latest of our world conflicts, the United States had no armaments industry. American makers of plowshares could, with time and as required, make swords as well. But now we can no longer risk emergency improvisation of national defense; we have been compelled to create a permanent armaments industry of vast proportions. Added to this, three and a half million men and women are directly engaged in the defense establishment. We annually spend on military security more than the net income of all United State corporations.

This conjunction of an immense military establishment and a large arms industry is new in the American experience. The total influence-economic, political, even spiritual-is felt in every city, every state house, every office of the Federal government. We recognize the imperative need for this development. Yet we must not fail to comprehend its grave implications. Our toil, resources and livelihood are all involved; so is the very structure of our society.

In the councils of government, we must guard against the acquisition of unwarranted influence, whether sought or unsought, by the military-industrial complex. The potential for the disastrous rise of misplaced power exists and will persist.

We must never let the weight of this combination endanger our liberties or democratic processes. We should take nothing for granted. Only an alert and knowledgeable citizenry can compel the proper meshing of the huge industrial and military machinery of defense with our peaceful methods and goals, so that security and liberty may prosper together.

Akin to, and largely responsible for the sweeping changes in our industrial-military posture, has been the technological revolution during recent decades.

In this revolution, research has become central; it also becomes more formalized, complex, and costly. A steadily increasing share is conducted for, by, or at the direction of, the Federal government.

Today, the solitary inventor, tinkering in his shop, has been over shadowed by task forces of scientists in laboratories and testing fields. In the same fashion, the free university, historically the fountainhead of free ideas and scientific discovery, has experienced a revolution in the conduct of research. Partly because of the huge costs involved, a government contract becomes virtually a substitute for intellectual curiosity. For every old blackboard there are now hundreds of new electronic computers.

The prospect of domination of the nation's scholars by Federal employment, project allocations, and the power of money is ever present and is gravely to be regarded.

Yet, in holding scientific research and discovery in respect, as we should, we must also be alert to the equal and opposite danger that public policy could itself become the captive of a scientific-technological elite.

It is the task of statesmanship to mold, to balance, and to integrate these and other forces, new and old, within the principles of our democratic system-ever aiming toward the supreme goals of our free society. ........ President Dwight D. Eisenhower's Farewell Address (1961)

[Doctor Syntax]

Re: Seeking a seat at the table

"Do you really believe VC's would supply this kind or money and having no say in the company's direction?"

Of course they have a say. The say will likely be not to put that money at risk (and more than it already is!) with the legal consequences of unrestrained use by the likes of Hegseth, especially if that use is against US citizens. Just think of all the class actions.

[Jamie Jones]

Re: Seeking a seat at the table

"And what about guns? Do arms manufacturers demand a veto for each use? Of course they don't."

Smith & Wesson's manuals specify that "appropriate use" of a firearm means using it for legal purposes, such as target shooting, hunting, and "lawful resistance of deadly criminal force".

Glock specifically warns that any attempt to convert a semi-automatic pistol to fully automatic is "illegal and strictly prohibited under state and federal laws".

http://pdf.textfiles.com/manuals/FIREARMS/glock.pdf

etc.

[Like a badger]

Re: Seeking a seat at the table

Glock specifically warns that any attempt to convert a semi-automatic pistol to fully automatic is "illegal and strictly prohibited under state and federal laws".

But bump stocks are legal in most states? Alternatively religious gun loons could just buy a few assault rifles, which seem to be legal in all US states.

What is it with Yanks and firearms?

[Glen 1]

Re: Seeking a seat at the table

It's literally baked into many EULAs

"The Licensed Application is not designed or intended for use in hazardous environments requiring fail-safe performance, including, but not limited to, the operation of nuclear facilities, aircraft navigation or communication systems, air traffic control, or direct life support machines."

Admittedly it's more for transferring liability than actually stopping anyone.

[JoeCool]

Re: Seeking a seat at the table

Microsoft licenses are full of use restrictions, as are licenses for virtuall all software.

Thanks for making the point while totally missing it.

[Doctor Syntax]

Re: Seeking a seat at the table

Is that you, Pete?

[QET]

Re: Seeking a seat at the table

Must be one of his aides/lackeys, because it's far too coherent (while still being crazy talk).

[joe80x64]

Re: Seeking a seat at the table

The other reality is that other nations like China and Russia will abide by no such restrictions. Then should a war actually break out they will have the upper hand.

[Random person]

Re: Seeking a seat at the table

This analogy may help you to understand. The analogy is not absolute but reasonably close.

A manufacturer of a new heavy machine gun says to the Pentagon, this weapon should have a safety and fire selector and warns that if the machine gun is used on fully automatic for too long the barrel will overheat with a risk of blockages. The Department of War says that neither a safety or fire selector is required and if the manufacturer refuses to remove the safety and fire selector the DoW threatens to not just cancel this contract but to punish the manufacturer.

I don't expect you to read this, just like you didn't read the article.

Please demonstrate that you are not a snowflake by creating an account.

[Paul Hovnanian]

Autonomous weapons

"The second use case is powering fully autonomous weapons, which Amodei says are too dangerous to deploy in their current form."

Land mines. Cluster bombs. That horse has bolted from the barn long, long ago.