Rack-scale networks are the new hotness for massive AI training and inference workloads

If you thought AI networks weren't complicated enough, the rise of rack-scale architectures from the likes of Nvidia, AMD, and soon Intel has introduced a new layer of complexity. Compared to the scale-out networks, which typically use Ethernet or InfiniBand, the scale-up fabrics at the heart of these systems often employ …

[HuBo]

Ah-ha!

Interesting to see how rack-scale integration is expected to improve performance beyond the uplift in oomph from individual computational components. I guess it comes from the better sharing of distributed memory that comes from faster and less hoppy scale-up interconnects (a lesser need to move data closer to each individual GPU all the time?).

Makes me wonder if classical FP64 HPC could see similar improvements as AI in this, say a 1.6x speedup in dense matrix HPL (similar to training, as 4x over 2.5x), and a 12x speedup in sparse matrix HPCG (like inference, as 30x over 2.5x). Currently, the CPU-only Fugaku is neck-to-neck with the bigger CPU+GPU El Capitan on HPCG, so a 12x uplift for GPUs could be a huge deal (if it pans out) imho!

[Caver_Dave]

Backplane

Discussion of a backplane rather than cabling reminds me of a turn of the century "weather radar" system I was peripherally connected to (as in the development unit was about 20 feet from my desk for a few weeks.) IIRC there were 32 DSP computing node boards (3U or similar) connected via a nearly 4 foot tall and 3 foot wide backplane with active switching - the backplane replaced all the Ethernet (?) cabling and switches. The backplane had to be built in 2 parts as it was too large for any PCB processing house at the time.

The use of a backplane increased reliability, reduced installation time and errors, and the team reckoned weight (although it was surprisingly heavy.)