don't freak out too much
I just checked my group's VMs on ESXi 6.5, just over 700 VMs, 98% linux, only one of them has a USB controller(a special purpose windows system), should be easy enough to remove.
To check, I used the tool govc, parsed the VM listing, and then used the device.info option to list all devices attached to each VM. I know powershell is popular with vmware folks, as a linux person I have never really used Powershell myself(yes I know it's available for linux).
https://github.com/vmware/govmomi/blob/master/govc/USAGE.md