Risk Compensation
The problem isn't the AI, it's the trade-off between (a) taking risks to make sensible progress and (b) the rate of deaths and serious injuries that those risks result in, at a population level.
At the moment, with human drivers, UK society seems comfortable with killing around 7 people every day of the year in return for the benefits of being able to drive as we do.
With driverless cars, would we set the trade-off at, say 5 deaths per day? If we set the death rate too low then driverless cars will have to proceed extremely carefully to minimise the risk of death in the rare-but-possible event that a human pedestrian does something the AI does not or cannot expect. But could the families of those five dead people sue someone for deliberately designing the AI to take risks at a level that is known to result in deaths?
Then what about deliberate "jay-walking" to play "chicken" with driverless cars? Imagine the fun to be had creating traffic jams by getting a bunch of hooligans to pretend to be about to step into the road!
The problem is that we still allow fast-moving heavy machinery (cars, lorries) to exist in the same space as human beings (pedestrians, cyclists). Not something that is ever allowed on industrial sites, the HSE would require physical separation (like railways) or very low speed limits, banksmen for reversing vehicles, etc. See http://www.hse.gov.uk/workplacetransport/index.htm
Biting the hand that feeds IT
