Correct me if I'm wrong... but it seems that this is another example of security vs usability. There will always be this battle between the two, it's just picking a suitable demarkation that is important.
So... externally pluggable devices have near-direct hardware access for efficiency purposes otherwise the performance of them will be so poor and they will, in effect, often be useless or very non-desirable components. This bypasses the implied trust platform that exists within a system itself; Or, in other words, one trusts that the components that make up a system are trustworthy and as a result they can operate at full speed as close to the hardware layer as possible - which for computer components is pretty much essential. Any fix to this really has to be both at the hardware and software (OS) level.