Reply to EMCer and PCI
Disclaimer - Brocadian here:
Native FC is somewhat serialized with SCSI but that's SCSI not FC so let's make sure that we discuss the right things and not make mis-leading statements. Furthermore many FC target controllers including EMC's are not serial as such i.e. they have many queues etc. Now NVMe is better for flash than SCSI no doubt there, but running NVMe on FC for transport is something that the industry is working towards. FC can easily push 2M 4K IOPS with SCSI and the latency is much lower than 1ms. In addition you can get to ~70 us latency with flash on FC, I'm of course assuming an all flash device (no hybrid arrays). Making the argument that PCI-e is better than FC is comparing apples and oranges, it will largely depend on the usecase and the how the driver stack works on either technology.
But we are missing the biggest question here, extending PCI-e out of the server is not an easy task, PCI-e is very timing sensitive so a few meters can be done with a special adapter etc but a 1000 physical node network is much harder. Don't think you can build a PCI-E SAN the way you build FC SANs.