US Air Force AI drone 'killed operator, attacked comms towers in simulation' An AI-powered drone designed to identify and destroy surface-to-air missile sites decided to kill its human operator in simulation tests, according to the US Air Force's Chief of AI Test and Operations. Colonel Tucker Hamilton, who goes by the call sign Cinco, disclosed the snafu during a presentation at the Future Combat Air …
It may be made up and seem far fetched, but if the story update is to be believe so is the story.
However if that "AI" is flying a drone then pulling the plug is just a little hard. Then we only need to read some of the "On Calls"/"Who Me" on here to realise someone will make it hard to pull the plug, either deliberately thinking "AI" infallible or by wiring things wrong.
No worries. It needs fuel and/or electricity. Withhold it. Blow up the fuel pump, turn off the generator, cut the wires or hoses, whatever. As long as there is a human in the loop, a monkey wrench can be thrown into the works long before things get out of hand.
is the BLATANT lack of imagination/understanding by the people running the simulation.
We have a NAME for this: Paperclip scenario.
The FIRST book I read to contemplate this EXACT sort of thing was published in 1981.
Not that Shelby had not considered this problem generally two hundred years ago, or that Walt Disney had not VERY pointedly warned about the dangers of automatons eighty three years ago?
Seriously, just what kind of rock were these people hiding under? What color is the sky in their world?
Oh, wait. The old saw about "military intelligence." Carry on then. Good thing they aren't responsible for anything important.
---
Okay, so NOW I believe that AI is an existential threat--because it's going to be ordered about by this crew.
We have a NAME for this: Paperclip scenario.
It's funny you should say that, because the first thing that came to my mind is that this is like an Anti-Clippy situation...
"It looks like you've tasked me with taking out some enemy combatants - I don't need your help with that"
There's a good reason for this.
Imagine if you will that you are an experienced expert in the field of AI, well-read, ethical, imaginative and thoughtful. Now imagine someone asking you to work on murder-drones. You're going to rightly tell them to GTFO.
Unfortunately, they're going to keep making the murder drones, but they only have access to people who are not ethical/well-read/imaginative/thoughtful and the results end up being like this.
To my mind the most likely AI apocalypse is one where we have autonomous weaponised drones and some future Bezos type billionaire tells them to execute anybody who looks like they might be in a union.
Sorry, but I would love to work on such a project, for two (non-Ron Swanson) reasons:
1) A weapon is a weapon, and a tool is a tool. The fact that idiots misuse weapons and evil people misuse tools doesn't mean that the tool or weapon should not exist. In fact, the cruise missiles we sent to Baghdad in 1991 made final target selection autonomously. In this case, getting this tool usage right is hard. And I love to work on hard problems.
2) We know that for the last decade or so, China has been pouring money into AI research. It's pretty easy to foresee a scenario where only an AI is going to be able to act fast enough to counter an AI. Laugh about mine shaft gaps all you want, technological advantage has been thing every general in history worthy of the title has sought out. If we don't prepare to contain a hostile AI, we can expect to be rolled over by one. And I'm a defender. Even if the work were not in and of itself technically interesting, this is very worthy work.
"Agents are programmed to maximize scoring points – which can lead to the models figuring out strategies that might exploit the reward system but don't exactly match the behavior developers want."
Figuring out strategies to exploit a reward system without matching the intended purpose of the system is, in fact, very human behaviour. There are entire industries built around it eg swap 'developers' with 'governments' and swap 'models' with 'tax lawyers'
In a statement to Insider, Air Force spokesperson Ann Stefanek denied that any such simulation has taken place.
"The Department of the Air Force has not conducted any such AI-drone simulations and remains committed to ethical and responsible use of AI technology," Stefanek said. "It appears the colonel's comments were taken out of context and were meant to be anecdotal."
Sounds like the colonel's anecdote was hypothetical at best, which explains why it doesn't make much sense.
If the drone is attacking an enemy's air defence system it would be given a target list. Why would friendly infrastructure and units be included in such a list?
If the simulated drone is being controlled via a communications tower remote from the operator, why would it have attempted to kill the operator first? And how would it know where the operator was if the signal was passing through a comms tower? Also it's unlikely that the operator was located close to the targets, so if the drone was on site, how did it fly all the way back to the operator to attack?
I think either the colonel went off on a ramble without making it clear it was just hypothetical, or the media just missed that bit of his story.
No I didn't. It must have a list of valid targets to pick from, or it would have wasted all it's missiles on rocks.
Or telegraph poles. An early image-guided missile had a habit of doing that. It knew that a tank was a vehicle with a long bit sticking out one end. It didn't know the barrel was just a telegraph pole, or the poles shadow being cast across some perfectly innocent object.
For the moment.
I note, with some sadness, that we hear all about AI, but apparently nobody has bothered implementing the Three Laws yet.
Might want to get around to that, guys ?
Oh, silly me. It's the military. Of course they don't want those laws.
Carry on !
This post has been deleted by its author
Good point raised about requirements. We haven't yet reached the point of being able to define the requirements of 'good behaviour' for humans, so I'm not sure we're articulate enough to define and codify it into a system. Note I say 'good behaviour' since 'bad behaviour' is a constantly moving feast, hence our statute books get ever longer.
I'm also curious why we are benchmarking the 'intelligence' against humans - ie can this new thingy be mistaken for a human. I'm just not sure how relevant that is when assessing the risk of a bad outcome. The risk is almost the inverse - its failure to 'think' like us is probably our biggest concern as we wouldn't be able to reason with it using our own belief system which is is both genetically and socially engrained into almost all of us.
If aliens arrive this afternoon, we'd take a risk averse approach. I'm not sure why it's any different just because we've invented our own alien.
The risk is almost the inverse - its failure to 'think' like us is probably our biggest concern as we wouldn't be able to reason with it using our own belief system which is is both genetically and socially engrained into almost all of us. ..... PinchOf Salt
Quite so, PinchOfSalt, and that has the most catastrophic of humanity vulnerabilities hit right dead centre, smack on the head, with a systemic inability or maddening reluctance to think imaginatively out to the box just like an alien view of things may be rendered for media presentation and virtual realisation, guaranteeing that the future will be led by A.N.Others of a novel and unique perspective upon which new ideas can flourish and expand creating a fundamentally different world playground, uncorrupted by past human follies and self-service elitist boondoggles.
Do aliens really need the help of humans whenever they themselves have all of the power and energy for the delivering of sublime instruction sets to command and control hubs and world wide web networks over broadband and quantum communication channels?
And the honest answer to that question is a resounding NO. They do not need it ... but to be able to help provide it would a sign of advancing human intelligence being a possibility for AI grooming.
Where is it written that the "aliens"[0] way of thinking is somehow superior to that of Humans? Who is to say that Humans are not better at the thinking game than the "aliens"? Someone is taking a tremendous leap of logic somewhere ...
[0] I know, the existence of "aliens" assumes facts not in evidence, but bear with me for the sake of argument. Ta.
Where is it written that the "aliens"[0] way of thinking is somehow superior to that of Humans? Who is to say that Humans are not better at the thinking game than the "aliens"? Someone is taking a tremendous leap of logic somewhere ...[0] I know, the existence of "aliens" assumes facts not in evidence, but bear with me for the sake of argument. Ta. .... jake
.... and IT’s Making Tremendous Illogical Quantum Leaps Everywhere, is something to consider might not render its Virtual Terrain Team Players inferior and human-like/objects and/or beings subject to similarly practically remote, virtually autonomous and relatively anonymous proxied command and control.
Do you see any similarity/singularity/empathy between what you can read in "Re: Benchmark of AI" here on El Reg and the following you can read below, copied from the very recent Dominic Cummings' Substack blog, dated Saturday 3 Jun 11:14, .... with the former preceding the latter by 1 hour and 11 minutes ‽ .
This is partly (A) necessary background to thinking through what comes next, here and in America. In particular, is there an effort by a subset of the entrepreneurial elite that can build to ally with a large section of voters to replace the rotten Tory Party, or do we see the same dynamic as America — as politics disintegrates in a clownshow, those who can build respond by retreating further to their walled gardens and ‘fish ponds’, as Cicero put it, rather than trying to save the Republic.We can’t start doing things like creating a legal structure for a new party, figuring out how to launch and build it, what should its electoral strategy be etc without having some solid ground for understanding the different perspectives on questions like — why Brexit happened, what were we actually trying to do in No10, why did our attempt to turn the Tory Party into something extremely different fail (mostly, so far), was our attempt doomed because of its own logic, because of Boris/Carrie etc.
It’s partly (B) trying to answer a set of important questions about what really happened coherently in one place, given the crazy fairy tales believed across Westminster.
It’s partly (C) a product of having to write my official covid statement and face some hard questions honestly.
It’s partly (D) skimming through the essay I wrote in 2013 a decade later and thinking ‘what do I think of it now, having left the DfE, done the referendum, Trump, gone to No10, GE2019, covid, Ukraine and so on’ — in the context of (A), what comes next.
ACT* ...... Advanced Cyber Threat [whenever necessarily destructive and disruptive] and/or Advanced Cyber Treat [whenever worthily desirable and rewarding]
This sort of response was included as part of CompSci courses back in the 1980s, for bleep's sake: a key part of any learning mechanism is that you have no idea what/how it is going to learn[1], you just keep your fingers crossed that it will actually manage to learn *something*. Especially back in the '80s when machine resources were lower than today and it was painfully obvious that the system had gone down an inefficient route and couldn't achieve any of your goals for it, scrub and try again; nowadays just fling more cycles and memory at it.
Genetic algorithms will happily recreate the Vagus nerve's ridiculous looping down and back up again. Any signal into a system may be given "unexpected" importance, either higher or lower than *you* expected (because you think you are looking at the forest but the model can't even see the trees, just the leaves): a system that is "punished" will be trying to optimise away the punishment and just ignoring that input is a perfectly good way to do it.
The reported stories about AI programs used to be specifically about these weird results, such as the analogue circuits that "ought not to work" because the program had optimised some weird arrangement of parts that took advantage of an oscillation that humans worked to get rid off.[2]
In other words, do the bleeping background reading before trying to build a system![3]
In other other words, blasted whippersnappers, get off the hole where my lawn used to be!
[1] Or you would just, you know, program it directly.
[2] And then how the humans' approach was the useful one, as it allows building blocks to be created and assembled into bigger systems but the "clever" design approach had all the parts interacting together and couldn't scale up.
[3] Wasn't there a time when scholarship in the military was a thing of pride? The design of, success and failure of, everything from strategy to tactics to ordnance? From the importance of land surveys to waterproof boots?
AI : *kills all humans*
Humans : *buggers off to desolate areas and survive*
30 years pass
AI : *realizes part XYZ need to be replaced in order for it to function*
Humans : *waits gleefully for AI to self-destruct because part XYZ is failing rapidly*
AI : *dies finally*
Humans : *repolulates earth*
Humans : "Hmmm, I wonder if I can make an AI to...."
lather rinse repeat
To be found here.
TL;DR: autonomous drone in Iraq-ish war zone gets AI ethics upgrade because the general public aren't happy with it slaughtering innocents. Drone prioritises avoiding killing third parties. Command keeps overriding ethics to ensure target kills, drone decides best way to keep third parties safe is to take out Command with last resort weapon, a micronuke.
This simulation sounds like it was programmed by the work experience guy's younger brother.
If you create a reward system of points and then intervene to stop it scoring points, it seems entirely reasonable it decided to remove the communications and the operator if they are effectively reducing its score.
This doesn't illustrate a failure of AI as such, it illustrates a failure of those implementing it to put in basic controls and create appropriate rules for the AI to operate under. Quite obviously, it should be told the rules include not harming friendly troops or equipment, or that will score minus one million points for any such violations.
As with chatGPT, the special sauce is not the algorithm, it's the training and the prompt.
An AI was being trained to recognise cancerous melanomas by being shown photographs of cancerous ones and non-cancerous 'freckles'.
Then it was tested. Well, the AI worked out what the photos with the cancerous melanomas always contained, and the non-cancerous photos never had - a ruler.
