An NVIDIA Tesla supercomputing-specific GPU is the most LIKELY hardware for the GPU portion and is about 50 TeraFLOPS for 32 bit Single Precision Floating Point number crunching (or 100 TeraFLOPS for 16-bit Half-Precision Floating Point performance) and I suspect Microsoft went with AMD Rome 7742 CPUs for the 64-bit number crunching which maxes out at 2.3 TeraFLOPS for 64-bit Double Precision Floating Point operations or 4.6 TeraFLOPS for 32 bit Single Precision number crunching.
Since MUCH supercomputing work is done at Single Precision 32-bit number crunching, the total performance with 10,000 NVIDIA Teslas GPUs and 4,454 (i.e. 64-core CPUs) of the AMD EPYC Rome 7742 CPUs calculates out to 500,000 TeraFLOPS total for the GPU portion and 20,488 TeraFLOPS total for CPU portion or a Grand Total of 520,488 32-bits wide TeraFLOPS or about 520 PetaFLOPS at 32-bits wide. If you compare it against SUMMIT which 64-bits throughout, that means the Microsoft Supercomputer could do about 210 PetaFLOPS at 64-bit Double Precision Floating Point Number Crunching.
I should note however in MULTIPLE locations within the sprawling US military industrial complex, I know of at least FOUR of ExaFLOP+ scale 64-bit Double Precision supercomputers now operating!
Sooooooo, that’s actually not too bad for a civilian supercomputer!
P.S. Ours is the MOST POWERFUL SUPERCOMPUTER OF ALL !!! 119 ExaFLOPS at 128-bits wide Floating Point Operations SUSTAINED on 60 GHz GaAs substrates using combined-CPU/GPU/DSP super-chips !!!
WE STILL WIN BY A FAAAAAAAAAAAAAR MARGIN !!!!!!