What could possibly go wrong?
As a firmware developer/hacker kinda guy, this is a "spit your coffee over the keyboard" quality idea.
I did go and RTFA (well, spec) and they pay a lot of lip-service to CORS/security/etc it's still a pretty out-there idea.
There are actually few things in this world _less_ robust and securely programmed than your average USB peripheral - not only that but by definition a lot of them are in control of real physical things. USB peripheral firmware is totally the wild west... Ask any Linux developer trying to support something with Windows-only drivers - the first step is to basically run it under windows and reverse-engineer the device protocol to figure out how the hell the vendor set it up - but there is literally no telling what the device actually supports without reading and disassembling its firmware. USB devices are the blackest of black boxes, and there's millions of them, all with different firmware..
BRICKING ISSUE
Primarily, most USB devices are _exceedingly_ brickable; because most peripherals are flash-based MCUs, and most of them have some sort of firmware update procedure, and only very rarely does anyone even use the USB "DFU" standard; rather often, firmware updates are via what would (in any other system) be called hidden backdoors; e.g. HID command endpoints, hidden command messages, etc. Bulk endpoints with a "manufacturer specific" USB descriptor..
..There's very rarely a oh-shit-you-loaded-bad-firmware recovery mechanism either.
Even current USB devices that don't nominally require drivers (e.g. CDC, HID, Mass storage, etc) VERY often have these sorts of hidden backdoor things going on (for reflashing, debug, diagnostics, etc) because that's the way things have evolved for the last 20 years or so...
I think it's an interesting idea and (as a long time firmware developer including many USB peripherals) I mean that in an "oh wow the possibilities for shit going very badly are.. really interesting". I can think of a huge number of ways to turn previously useful peripherals into doorstops, and that's barely scratching the service of the issue. Yes.. but... WOAH THERE NELLY!
Letting the browser talk raw USB packets would be kinda like having an unpatched Windows 95 machine with every service turned on and all ports exposed to the internet.