Stop leaving spare USB ports active on machines that handle sensitive data.
FTFY. This was SOP not so long ago. And rivet the case shut, too. Repeat offenders to be issued with machines that have a solid blob of epoxy where the USB connectors were.
You don't need spare USB ports on a corporate/government machine. Peripherals should be shared per office, installs done from approved images over PXE and input devices should be permanently attached and only removable by a technician. Sort your physical security first, always. Then layer on the software and wetware policies.