Re: Slartibartfast Hovercraft
One thing that comes to mind is that when most people use their phones, they hold them in front of their chests, below their faces, to see the screen. So the phone's front-facing camera could match the face and eyes of the person to determine that they are looking at the phone, as well as using the phone's positioning sensor to determine that it's being held in a particular position, before responding to whatever the person is saying.
What's wrong with a dedicated (hardware) button and push-to-talk? Would seem very natural (think walkie talkies) and without the background power drain of constantly listening out to everything? I can see accessibility issues along the lines that chap raised here the other day, but alternative arrangements can be made for those cases.