Reply to post:

Ah, well memory bandwidth / latency still matters, even if networking is a bottleneck.

There's some multi-machine architectures (e.g. OpenVPX) where the links between nodes are very fast, and do not involve CPU time at all. Such interconnects essentially bypass the CPU cores entirely to operate - they just DMA data straight into / out of RAM buffers. For CPUs like modern Xeons, this inevitably means passing through the CPU's memory subsystem (even if the cores aren't aware this is happening), so there is an art needed in ensuring that the amount of memory pressure coming from the processing leaves room for the network bandwidth too. That requires careful decomposition of the processing problem.

Older implementations actually involved a "bridge" chip between CPU, RAM and interconnect (bit bespoke, not done these days), the the RAM / Bridge being able to sustain full rate network and CPU memory demands.

The "biggest" example of this kind of approach is Tofu, used on Fugaku and the K machine (Japanese supercomputers); high speed low latency interconnect wired directly into the CPU's brain, not some peripheral out on a PCIe bus that requires CPU time to manage, run a stack for, etc. Both of these machines are notable for how their peak benchmark performance is fairly representative of application performance.