Re: Removable stem seems like a reasonable suggestion...
For those people whose ears they fit - they are actually pretty stable in place. They hook behind the 'tragus' (the flappy bit just in front of the ear canal entrance.
If your ears aren't 'apple approved' dimensions... then yes, they're going to fail.