Reply to post:

The Register Lecture: AI turning on us? Let's talk existential risk

Peter2 Silver badge

Imagine for a second an intelligent virus (yes, I know, such a thing is well beyond our current capabilities, but this is a thought experiment) that manages to infect air traffic control workstations with the intent of causing as many deaths as possible. Or traffic light control systems. Or the emergency alert system.

Ok. Firstly, I don't think you understand how ATC works. I do because I have been taught to fly and have spoken to them via radio, which is how commands are passed. If aircraft seperation is compromised then ATC will know quite quickly via irate pilots shouting about it and they'll revert to their ermergency plans. As pilots are responsible for their aircraft there is unlikely to be any serious trouble if ATC packs up.

Traffic light systems are designed with physical safeguards such that they'd blow a fuse if you illuminated both sets of lights. Harm, zero as it just falls back to manual operation.

Emergency alert systems could cause people to panic and stand around doing nothing, but that's not going to cause the end of the world.

And that doesn't even get into the nightmare scenarios of hospital systems and infrastructure control systems. How many people do you think would die if medical equipment started putting out inaccurate data and all the lights went out? Heck, just shutting off gas pumps would result in millions of deaths in the US inside of a month.

I've worked for the NHS. My guess would be zero casualties, because everything critical is airgapped. Yeah, returning the wrong patient records wouldn't be good but that's about the most harm possible and the damage would have to be done by humans. Lights aren't going to go out because light switches aren't connected to computers. Power is backed up with generators that are tested weekly, the switchover and switchback to and from which causes more damage to computers in tests yearly than an AI could aspire to. FFS, UK hospitals are built with EM buffers on incoming lines designed to protect against a nearby nuclear detonation.

In the UK, petrol pumps are a very manual and offline process. Harm, zero.

The biggest harm would be that Just In Time supply systems would probably become SomewhatTooLate, which is suboptimal when it comes to things like food.

True, we don't have to worry about AI triggering a nuclear apocalypse directly, but what about sending falsified communications to all the world's nuclear powers making it seem like they were under attack?

Again, knowing something about these systems I know that they are designed by people who are considerably more paranoid than I am and have far less trust in technology and people programming it than I have, which is why everybody has their nukes set up to survive the first strike and then launch in response later.

They deal with alerts tolerably well. You know about the horror stories of training tapes of a full scale attack being ran on a live system by the USA during the cold war, right? It happened, yet failed to set off a nuclear war.

This is sort of like the X-Files. Things seem plausible when you don't know how they work, but the more you know the more it seems a bit silly.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021