1) The volumes (LUNs) being accessed as block storage ARE NOT physical devices. They are files stored on file systems.
No. They *can* be stored as files on a filesystem, but they don't have to be. They could instead be direct access to logical volumes such as LVM, and the overhead of that is very small.
3) The computational overhead to simulate a SCSI controller in software
... is irrelevant. If the clients are virtual machines then they'll expose block devices as virtio devices. There's no need to emulate a SCSI controller anywhere in the chain.
The problem is, PCIe is too slow for this. When facing NVMe and new storage technologies, the bus would max out at about 32 NVMe devices.
What "bus" are you talking about? The throughput of PCIe is determined by how many lanes you use. You can in principle build a switch fabric with as many lanes "in" and "out" as you like.
since NVMe and FC lack multicast, performance is simply doomed
Riiiiight. We all use multicast to our storage servers. Not.
Biting the hand that feeds IT
