"As soon as the given AI can drive at least as safely as an average human, it should be ok to use it in a self-driving car."
But there's a catch. AIs don't learn the same way we do, and in fact we don't always understand HOW we as humans ourselves learn. For example, there's the concept of intuition: the stuff we learn SUBconsciously, like the very subtle difference between a normal person and a suspicious one, between a car likely to stop and one likely to run the red light, the tells that a huge tree branch is going to fall in my path and I need to get out of the way BEFORE it actually falls (or it'll be too late), or perhaps the hints that the jerk in the corner is just trolling with the self-driving car that can't afford to risk hitting a pedestrian given half the chance. Since we don't know how ourselves we pick up on these subconscious hints, we have no way to teach them to an AI, so it doesn't learn those subtle things that can help prevent accidents without our even thinking about them. If you look up "self-driving cars intuition" you should probably find a few articles that wonder the same thing.
