Poltergeist attack could leave autonomous vehicles blind to obstacles – or haunt them with new ones Researchers at the Ubiquitous System Security Lab of Zhejiang University and the University of Michigan's Security and Privacy Research Group say they've found a way to blind autonomous vehicles to obstacles using simple audio signals. "Autonomous vehicles increasingly exploit computer-vision based object detection systems to …
This is worrying for people who want to rely on self-driving cars. I wonder whether they can also affect the lane-tracking features. As an older driver, I do find the automation in new cars intrusive such as lane tracking, 'foot off the accelerator' when approaching a roundabout, telling me I'm travelling over the speed limit when the limit is 30 not 20 as the car would have it, or that the speed limit on one minor road in the UK was 110mph.
My main problem with all this automation is that when the computer cannot cope there is the expectation from the manufacturers that the 'driver' will be alert and able to decide instantaneously what to do, when clearly anyone who is using lane tacking or automatic driving is unlikely to have been paying attention.
Only as worrying as the incompetence of most drivers.
As a disabled driver I very much appreciate lane tracking and adaptive cruise control, even if my car doesn't seem to grok that dual carriageway NSL is 70 rather than 60, or that speed limits persist around corners. At least use the manual speed limiter setting is nice and easy to use.
The fact that you are disabled does not matter. Your car has been modified to adapt to your disability so that you can drive it.
You're still the diver, whatever semi-autonomous thingamajigs you have, and if you cause an accident, you will still be held responsible.
As a society, we're getting ready to unleash a storm of metal controlled by a program that, in the best of circumstances, can only prove it works when the day is sunny, the road clean and the road markings new.
I want tests for when it rains, when it pours, when it snows, and when the road markings haven't been refreshed in 20 years.
Once an autonomous vehicle can cope with those conditions, then we can start worrying about audio attacks on detectors.
No, my car hasn't been adapted at all.
Do try not to spout rubbish.
Of course I am still responsible for safely controlling the vehicle, as I would have been if I ran into the back of someone whilst ABS was installed. The fact that the assistive technologies more than double the amount of time I can spend the wheel is irrelevant to the responsibility for the vehicle.
This isn't a car that is advertised as self driving in any way, it just has some assistance features. Having been driven in a Tesla using auto pilot it is a remarkable system. It wasn't, last time I checked, level 5.
No self driving vehicle needs to reach perfection, it needs to beat the average human. That is a task which is probably well within our reach the majority of the time.
Heck, I'd even be happy with a motorway level of autonomy. Drive to the motorway, relax, then drive off the motorway in a fresh state. Just reducing the amount of tired drivers on the roads off the motorways would be a significant benefit.
The problem with the "but sometimes" is that you ignore the benefits of the "most of the time".
So what if the car can't drive you all the way to your destination, if it can put you within ten miles without several hours of fatigue then it has massively improved the safety of all parts of the journey.
> No self driving vehicle needs to reach perfection, it needs to beat the average human.
The average human driver is shit, using that as the baseline is unconscionable. It'd make the roads more dangerous as while it might 'bring up' the level of those drivers who are below average, it is also going to 'pull down' those drivers who are above average down to that average level (since the self-driving will be doing the driving). Thus overall making the roads (IMO) no safer, and perhaps more dangerous, than now.
Such a system is responsible for human (and others, musn't forget the aliens who live amongst us, oh yeah, and the animals) lives, and as such, before giving a machine that level of responsibility, where there is no human in the loop to make the actual decisions, to take the responsibility, it needs to be far, far better than an "average human".
For a self-driving (level 5) system to be certified - IMHO - it needs to be at least as good as the 80th-percentile of professional drivers (that is, it is better than 80% of professional drivers), one might even argue as good as mid-level racing drivers (i.e. not F1/Indycar level, but above go-cart). None of this averaging across a whole population shit.
Paradoxically, the average human driver genuinely believes that (s)he is an 'above average driver'.
I reckon this is due to each driver having their own beliefs on what constitutes 'above average' and they grade other drivers and themselves by those criteria they value the most. So I am 'above average' at driving because I indicate my turns and lane changes, rarely break the posted speed limit, leave a good gap between my vehicle and the one in front and generally try to be a courteous driver. Oh and I stop for traffic lights that are red or turning red unless I definitely cannot.
I have never done a handbrake turn, do not nip in and out of traffic, and when someone is driving too close behind I slow down (as per the UK's Royal Society for the Prevention of Accidents advice) rather than speed up. This annoys drivers who reckon that car control, the ability to drive as fast as possible, and who only use their indicators when they are going to push into a lane and think the flashing amber light gives them right of way, makes for an 'above average driver'.
The philosopher, Stephan Korner once commented that people sometimes act contrary to their knowledge. You know, driving after midnight, that not every other driver is intoxicated, but if you are sensible you drive as if they all are drunk, just in case.
Oh. Hi Pascal.
Systems vulnerability testing either comes first, or it does not happen at all. Human beings are not nice creatures. The more money that goes into an effort the harder humans will work to keep the thing going.
Certainly, you need not-happy-path testing, and I was about to upvote you for saying that much. But to ignore the fundamentally hostile nature of the world in regards to these systems until the end is to insist on happy-path testing at a deeper level.
>I want tests for when it rains, when it pours, when it snows, and when the road markings haven't been refreshed in 20 years.
I suggest, given the substantive contents of the article to include tests for noise and whether the car behaves differently when the passengers are listening to Guns'n'Roses compared to Bob Dylan for example...
"Computer assisted" is the Big Lie™ of self-driving cars. You're either aware of what's going on enough to drive at the drop of a hat or you're not, and if you're paying that amount of attention then you might as well drive the damn thing yourself.
Every manufacturer knows this and knows that their cars' passengers will not be paying attention - they'll be talking to other passengers, reading the paper, admiring the scenery, or making rude gestures to cyclists. Designating one of these passengers to be "the driver" is simply a fig-leaf for the manufacturers' insurance requirements.
I am aware enough of what's going on, but the assistance virtually eliminates the cognitive load required to maintain gaze stabilisation. So I can take over at any point AND the assistance features are enabling me to do that for more than twice as long as I could without them.
I always welcome an additional set of eyes on the road, whether they belong to a passenger or the car.
Just because you can't maintain awareness of what's around you whilst assistance features make that easier doesn't mean noone can.
There is a point at which you start to trust the automated system too much, but that's a long way off for most automated systems at the moment.
I grew up on a farm. That means I was occasionally steering pickups when I was 5. I was driving tractors when I was 7. I was driving pickups when I was 8.
When I was 17, I tried cruise control for the first time. I was appalled by how much more difficult it was to keep my focus on the drive. I have occasionally resorted to it again in the last 37 years. And by "occasionally", I mean "most decades". I'm never comfortable with the reduction in focus.
Yes, the cognitive load goes down--that is the problem. The human brain is amazingly complex, and capable of doing a lot. Given it insufficient simulation, and it gets bored. And bored behind the wheel gets you dead.
>Given that most people can’t or won’t concentrate enough to drive safely anyway…
I use cruise control constantly, did even before I became disabled. Did it hinder concentration - no.
Reads as if you are speaking from personal experience, also like most people (and know I sometimes do it myself) you are over-estimating your abilities.
>Driving is already hard enough without having to monitor dials as well.
Dials (and knobs, switches and buttons) are easy compared to the multi-function touch displays much liked these days. I suggest if you are finding them difficult you may wish to assess whether you are actually fit to drive.
So you think it’s easier to keep aware of all the vehicles around you when you also have to look away from said other vehicles.
That makes complete sense. /s
I do hope your disdain for driver aids extends to servo assisted brakes, ABS, power steering, and everything else that is basically de tiger at this point. And as for the significant number of cars that will brake *for you* if they detect that you came off the throttle and went onto the brake very suddenly - because many incidents weren’t the result of lack of braking capacity in the vehicle, but lack of brake pressure applied.
"many incidents weren’t the result of lack of braking capacity in the vehicle, but lack of brake pressure applied."
The overwhelming majority of car crashes are the results of decisions or the lack of well before the brake pedal is applied, however firmly.
The only way the human driver will be given an incentive to improve their own skills is if it saves them money - which in essence is insurance companies giving discounts for decent ongoing driver training a la RoSPA ROADAR or IAM schemes (these are int he UK, I'm not aware of anything else similar elsewhere in the world).
I can see the logic - sort of - from the car manufacturers' points of view. "Let's eliminate the things which cause cars to come together, like not braking in time, crossing lanes, not looking properly into blind spots". The problem is that those speccing the systems to do this are probably in the same boat as other drivers in that the passed their tests many years ago and haven't had any standards checks since then.
I say this as someone who's a tutor for the local RoADAR group and sees a ex-police advanced driving instructor at least yearly to make sure I'm still up to scratch. And before anyone passes me a pork pie hat and some string back gloves you'd need to drive with me first to see if that particular stereotype fits.
“ The overwhelming majority of car crashes are the results of decisions or the lack of well before the brake pedal is applied, however firmly. ”
Well, there is that…. One could argue that that’s *all* incidents, decisions around maintenance and inspection being important decisions as well. But the point was that many people were hitting things when their (late) reactions were actually still “in time”.
Firmly of the belief that licenses should require retests every few years.
Firmly of the belief that various advanced driving courses should be mandated for higher performance vehicles.
Also firmly of the belief that keeping my eyes on the road is a good thing - and that’s what adaptive cruise control and lane keeping allow me to do.
> Firmly of the belief that licenses should require retests every few years.
I agree with this, certainly at least with respect to the 'written' portion, as road regulations change over time, and people usually don't go off reading acts of parliament to see what rules have changed.
For example, recently (well, about 7 years ago now) road rules changed here to allow motorcycle 'filtering', which had for decades been illegal. Filtering is the ability to, during slow-moving 'congestion' traffic (less than 25kph), to ride between lanes of traffic, and/or to 'filter' at traffic lights past the stopped vehicles to move up to the lights themselves, to the front of the queue. There are many additional requirements for this that I won't go into (not past heavy vehicles for example). The point is that I availed myself of these 'new' regulations and moved to the front of the queue stopped at traffic-lights, and some middle-aged wanker abused me and was going on about how it was illegal lane-splitting, and tried to run me off the road, being totally oblivious to the new rules. And I still find that now, that many are oblivious to these changes.
Therefore after that event, I think that every 10 years, on the 20/30/40th year etc, one should have to re-sit a written test that covers the basics plus specifically points out any changed road-rules that have come into force during the previous decade.
I wonder whether the person who tried to run you off the road because he thought you had broken road traffic regulations realises that running someone off the road is also a breach?
As for taking test every 10 years or so, that would probably be a good idea, but I reckon I might have to take a lesson or two now. Having just scrapped the car I've had for 22 years (it had a cassette player, not even a CD) I found that when I hired a new VW Golf, I had no idea how to manage the radio or satnav. Fortunately it was an automatic, so I never had the issue fo a hill start with the auto engine stop-start. The real issue was that in the event of an 'emergency stop', my learned reactions of 40+ years were to press both feet onto the pedals available, which is not ideal in an automatic.
I shall get a new copy of the Highway Code as soon as I am able, might even read it too.
>So you think it’s easier to keep aware of all the vehicles around you when you also have to look away from said other vehicles.
If your driving properly, you are looking away from individual vehicles around you all the time.
Actually, you'll find the essential instruments and warning indicators are statically positioned to be easily read with a quick glance...
>I do hope your disdain for driver aids extends..
No my disdain is for those aids that don't actually aid me, but take awareness and decision-making control away.
Years back I used cruise control on UK motorways, found it either meant I was doing less and so was more easily bored or it got in the way of responding to traffic conditions.
More recently, I've driven cares with lane awareness, similarly, I've found it got in the way of good driving more than being of assistance.
One of the helpful features of some systems is that they do (mostly) recognise the correct speed limit, so when on unfamiliar roads, this reminder is helpful.
So having driven a Volvo S60 2012 (with all the safety features) no I don't confuse cruise control, lane assist and other technologies that take over the driving with real driving aids such as servo assisted brakes, ABS, power steering, steering column mounted levers for windscreen wipers, lights etc. that only come into effect as and when I choose.
>And as for the significant number of cars that will brake *for you* if they detect that you came off the throttle and went onto the brake very suddenly
Actually, a more important consideration is detecting that the car in front has started to slow and that you need to slow and potentially slam your brakes on.
"No my disdain is for those aids that don't actually aid me,"
And since they don't help you they can't possibly help anyone else.
Remember when people didn't trust seatbelts, airbags, ABS... because they were potentially dangerous in some rare condition, or they interfered with you "driving" the vehicle.
">And as for the significant number of cars that will brake *for you* if they detect that you came off the throttle and went onto the brake very suddenly
Actually, a more important consideration is detecting that the car in front has started to slow and that you need to slow and potentially slam your brakes on."
Erm - that would be the adaptive cruise control technology you were just slamming - but in a state where it can't be turned off and is called collision avoidance.
dude, you are taking this personally, and not seeing the big picture.
You are not the issue. The issue is "some" tech is encouraging people to be less responsible by removing them from the equation, creating a big gap is security for everyone on the road. Are people stupid - yeah, is tech stupider - yes, it is just dumb. It can only do what it is told. Is it cool, sure. Is it human - no, it makes toast, to dark or to light and it's not good enough and gets people killed.
When your car can carry on a full on conversation at speed and out wit you every time, then and only then, I might consider letting it taxi me around. But that, is a loooooong ways away.
So you want to issue driving licenses based on the ability to hold a conversation?
I'll wait until the autonomous systems have passed a driving test - until then I'll still be driving, and taking advantage of all the assistive features I can - whether those are things like ABS to assist me in "Oh shit" moments, or things like lane keeping and ACC that help me on a normal journey.
"Dials (and knobs, switches and buttons) are easy compared to the multi-function touch displays much liked these days. I suggest if you are finding them difficult you may wish to assess whether you are actually fit to drive.'
I agree with your disdain for touchscreens as an in car interface.
However the closest I have ever come to another vehicle on the road was twenty year ago when I glanced down to grab the heater control. Massive great dial, so took no more than .1-.2 seconds to glance, but in that moment the vehicle in front started an emergency stop. I stopped with about 6" to spare, and was then overtaken by a small blue cloud that used to be part of my rear tyres.
My comment that driving was difficult wasn't that I found it particularly so - but that in general operating lethal machinery in public is difficult. The levels of concentration required to do it properly are much, much higher than most motorists manage. Reducing that level slightly, and reducing the time your eyes are away from the road at all, is a good thing.
Since I became disabled I have an additional cognitive load to deal with, and the aids are therefore even more important to me, previously they were useful gimmicks (i.e. I could do without them, but they were still an aid), now they are significantly important.
As a completely non driver aid example:
Keyless entry for a car. I had it, thought is was a fun gimmick, used it anyway. Replaced car, didn't have it any more. No big deal.
Then became disabled.
So now I come out of the supermarket, have to fish around in the bag for the key, open the boot, put the key back in the bag, put shopping and wheelchair in the boot, stagger round the car, fish the key out again, unlock the driver's door, put key back, get in, get key out for a third time, put key in ignition.
Because every action requires both hands, I can't just keep the key out - and suddenly keyless entry is no longer a gimmick but a major feature.
It's well proven that semi-autonomous is the most dangerous phase possible.
Have an example:
Your vehicle claims to be able to do adaptive cruise control, auto braking and lane following on motorways and dual carriageways.
This covers pretty much everything needed to get the car from one end of the M1 to the other without any driver input under most conditions.
One day there's sudden stationary traffic. Normally your car stops for this automatically, but today your car piles straight into the back of the stationary vehicle in front.
Why did that happen?
You saw the stationary traffic and trusted the car to stop.
For some reason the conditions happened to be such that the automation failed. You could not realise that the automation had failed until it was too late to avert the crash.
Instead of your thought process being "slowing traffic, uh oh, stopped, brake harder" it will have been "stopped traffic, why isn't the car slowing? Shouldn't it have reacted by now? *crunch*"
Even basic adaptive cruise control carries this risk because you won't be aware of slowing traffic until much later, especially in a "quiet" or "silent" vehicle.
Stopped traffic is a rather peculiar situation - most current systems don’t deal with it.
I don’t trust the car to slow/stop unless it’s already following something … You have to ask whether failure of the autonomous systems happens more of less often than failure of the nut behind the wheel.
Evidence at the moment is that it doesn’t. Both failure modes are catastrophic, but one system has the opportunity to learn from the mistake, whilst the other doesn’t.
My personal opinion is that the term 'automation' or ' auto' in its use as 'automation' or 'automatic' should be banned (excepting gearboxes as an established norm) untill such time as the manufacturer is legally responsible for driving not the owner.driver.
I agree wholheartedly that 'semi-autonomous' driving is the worst of all worlds with drivers bored and unfocussed yet still expected to take control in a split second emergency.
I can handle cruise control but I found that the more advanced cc such as radar cruise is quite distracting. You either mentally hand over judgement to the car or you are on edge as you catch up to a slower car.
If your distance is set long to avoid the "will it slow" worry, invariably some arse will cut in on your and the car will brake suddenly to put the distance in again. I've given up on it. Nice idea buit doesn't work for me.
Yes - but it's not the condition for which most current ACC systems are useful. It's bit like complaining that your toaster doesn't make coffee in the morning.
ACC is there so that in dense traffic you can set the cruise control and if the vehicle in front slows by a bit you don't have to keep changing the CC settings, it's adaptive.
I have mine set to max distance and don't find people pulling in front of me more than they would if I was either controlling that distance myself or had it set to a closer distance. Nor does it slam the brakes on when they inevitably do so at a motorway junction.
>ACC is there so that in dense traffic you can set the cruise control and if the vehicle in front slows by a bit you don't have to keep changing the CC settings, it's adaptive.
It's adaptive, but also dumb...
The current generation of ACC is of no help on the M42 (and stretches of the M1) with its dynamic variable speed limit.
I'm also concerned that you are allowing yourself to be distracted by having to manually set the ACC, something you may be doing many times in a journey.
If these systems cannot handle stopped traffic on the highway, then they absolutely have no place there. Stopped traffic wasn't even rare pre-COVID. The daily commute will get you that in my area.
But the places where it is rare is where you really need to handle it. It indicates that some major accident has already occurred.
Just no.
With regard to your second paragraph, this has been obvious for a very long time. Driver distraction as it's called is a variety of things, but includes autonomous functionality where a driver has less to do. That means they'll be looking at their phone, the pretty flowers, the inside of their eyelids - anywhere but the the road or conditions when it could matter most.
The likes of the US, EU, UK etc. should be coming up with stringent safety requirements that semi-autonomous vehicles must adhere to in order to be operate on public roads. That includes enforcing driver attention and doing so in a manner that is not easy to subvert.
It shouldn't be left to the likes of Tesla or other car manufacturers to come up with some lame ass solution that ticks a box but is easy to defeat or not fit for purpose in the real world.
It's to be expected that such attacks will be discovered, but they are no worse than the attacks that can be deployed against human drivers. For example, shining a 3W laser pointer in the eyes of a human driver will reduce their ability to detect obstacles. Audio attacks are similarly effective - playing Billy Ray Cyrus at sufficient volume will cause the driver to lose the will to live and steer of the nearest cliff.
They're obviously way worse than the attack you suggest, human driver will close/cover their eyes and try to come to a controlled stop. Making an autonomous vehicle think the stopped truck/bridge support isn't there at 70 mph I suspect will have a worse outcome for the passengers. I'm willing to play guinea pig as the passenger in the car driven by a human, if your game to be a passenger in the autonomous car?
Large posters of scantily clad women (or men) have been know to reduce the ability of humans to recognize road hazards.
Would a poster of say a Lamborghini Gallardo have a similar effect on autonomous systems? If not, we need to train them better if they are to eventually drive like humans.
The easiest and most effective attacks against autonomous vehicles are going to be low tech ones - putting some chewing gum on the camera, laying some cardboard boxes in the road and so forth. Even just standing in front of a car would work. I'm sure that if (ever) vehicles are fully autonomous that carjackers and griefers will have a field day with them.
I fear that there will be simpler attacks on AVs. Simply walk, or ride a bike, or drive a car, as if you were about to collide with the AV (in the knowledge that you will stop, obvs). The AV will be forced to take avoiding action.
A new form of "chicken" for the teenagers of tomorrow, perhaps?
> Equally simple, is to use a laser pointer to confuse the lidar.
That is unlikley to work as lidar, unlike laser pointers, tend to work in the non-visible part of the spectrum.
Laser pointers are meant to project a visible dot of light so they can be, you know, used to point out things to other people. That's what they are for.
Since lidar are designed to send out thousands (if not millions) of beams of light per second, and to operate around dozens/hundreds of other such units doing the same thing, using visible light would be problematic, don't you think? On top of that, there is often a lot of subtle frequency modulation and other techniques in the light sent out, such that a lidar can filter out light coming from other lidar units and only process light it emitted.
Early prototypes may have had this problem, but actual in-use production units don't.
That isn't to say someone couldn't build a device designed to emit light on the same wavelengh and moduation as in-service lidar. But this would be a targeted attack using special-purpose harware. As first (assuming you don't have the blueprints, source code, random keys used) you'd have to intercept the light form a target lidar unit, work out the modulation that one is using, and then be able to send that same signal back. Like intercepting a car-key fob, or RFID security tags, and so on. Doable, but not a random mass-attack.
At the moment, I cant think of an effective countermeasure.
These camera modules are generally third party and (fairly) standardised COTS. The onboard processing is 'correct' for its context. It could well prove a major refactor to avoid this attack vector. I wonder if any self driving incidents to date could be associated with (even inadvertent) for of image destabilisation?
Why should those security concerns be of interest in a world where high load heavy duty cranes are controlled via unsecured an unauthenticated remote controls ?
Not heavyweight enough for you ? Try the remote controls train workers use for shifting around whole trains in maintenance compounds.
At least cranes and trains tend not to run loose in the streets. Any vulnerability on autonomous whatevers adds to the risk list these things already have from their inherent flaws and their inability to cope with "untrained", unforeseen situations AKA input patterns. Put a low tech metallic sticker or a blob of zinc spray over the usual microwave/RADAR sensors and look at the output distortion. Use hairspray or similar substances on any optical lens. Your car cannot cope with either and not ensure safe braking and shutdown ?
Then it must never enter the real world.
The described attacks open a dangerous can of worms for "remotely" distorting sensors used to "safely" drive some speedy heavy weight around populated areas. I still wait for research in the field of "safe shutdown" when anything autonomous is confronted with corrupt or otherwise uninterpretable / "illegal" input, be it by intentional sabotage or occuring hardware or software faults.
Maybe an array of emergency sensors popping out of the hood reacting to questionable input and ensuring a safe shutdown / braking would help.
Right now all we know is that the device they tested was able to confuse a Samsung phone camera.
I don't buy their excuse as to why the paper didn't include a test on a car. I wonder if they did test it but it didn't work.
Even if the technology could scale up I wouldn't be worried because I can't see a practical use for it. If you want to target a particular person, crashing their vehicle doesn't guarantee the outcome. Plus the equipment is going to be as traceable as an IED. If you just have a thing about autonomous vehicles, there are cheaper and easier options..
Human drivers already don't see things and crash into them. So do semi-autonomous Teslas. So do all other forms of transportation including two people walking. I assume the same will be true for autonomous vehicles.
I assume they didn't use a car because of the cost.
It's also true that all the current semi-autonomous and autonomous testbed vehicles are using off-the-shelf parts, so this attack will work on them too.
What isn't known (and is out of scope for the paper) is whether any current systems can detect this attack and properly handle it using input from other sensors.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
