". . . simply requiring a physical key in the circuit . . . would allow humans to remain in control there, eliminating that as a threat."
But that's exactly the type of forethought that is required!
My point, given above, at some length, is that we really should consider these ideas before they are needed so that regulations can be constructed accordingly. Private corporations will do what is best for them and if regulations aren't there to govern their actions, there is no reason to believe that any given 'common sense' move will be implemented.
If we think there is a risk in allowing self-driving cars to operate without requiring a human in the loop then we have to lay that down so companies are forced to comply. It might seem like a simple, common-sense step but that one step actually makes the idea of self-driving car-share much less useful. Requiring physical human interaction (in the form of a 'key' in actual proximity to the car) precludes any service where you book a car and it comes to your door, ready for you. You would have to go pick it up from a dedicated location.
Those locations must therefore be scattered around so there are enough vehicles available near where people need them. This limits the number of vehicles available in any area and takes up already limited parking space in densely-populated areas. Allowing no-interaction self-driving opens up a much greater user base as large numbers of cars can be stored in depots and these can be packed tighter, saving space.
It also also allows the idea of a drop-off where the car drops you off at your destination - like a taxi - and then goes back to base or off to another customer. That, again, helps with parking. If the car requires a human with a 'key' to move then the car needs to be parked somewhere at the destination. Which leads to people booking cars for longer than they need so they can park it and then drive it back later. Cars that don't require interaction allow there-and-back to be two separate trips, with the car not needing to be parked and idle in the interim. That's more efficient.
The idea, then, that just requiring a key solves everything is missing the benefits that can be had from other setups. My though is that the benefits means that zero-interaction cars will happen unless there are regulations like your proposed one to prevent it. People will, after all, prefer the option that is easier for them and, given the choice, a service that comes to them will be chosen over one they have to walk to; it's cheaper and more convenient.
A different question is whether self-driving cars should be able to be controlled remotely. This impacts even more as, not only would it prevent the above, it would also prevent law enforcement locking down cars - something I am sure they would love to be able to do.
It's not black-and-white, of course, which is why the discussions need to be had. Perhaps cars are able to be controlled remotely unless there is a human in them, in which case, they are sacrosanct as the risk to the occupant(s) and other road users and pedestrians would be too great to even allow the police to shut down a moving car remotely.
One thing you can be sure of - if it's not sorted out ahead of time, the result will be that law enforcement agencies will have unfettered access to monitor and manipulate your car in whatever way they deem 'necessary'.
I have ended up going on at some length again and the subject of self-driving cars is somewhat tangential but my point remains the same - we really need to work these things out before they are ubiquitous and history tells us that the distance from a useful technology being available and it becoming ubiquitous is rather short and has caught regulators off-guard time and again.