HPE goes Cray for Nvidia's Blackwell GPUs, crams 224 into a single cabinet If you thought Nvidia's 120 kW NVL72 racks were compute dense with 72 Blackwell accelerators, they have nothing on HPE Cray's latest EX systems, which will pack more than three times as many GPUs into a single cabinet. Announced ahead of next week's Super Computing conference in Atlanta, Cray's EX154n platform will support up …
Why do these people go so crazy for density? 0.3 MW per rack is just stupid. The datacentre itself is going to be dwarfed by the electricity supply infrastructure, never mind the water chillers.
Why such a focus on how much compute they can squeeze into one rack? You have plenty of space and a limited overall power input, so why not just have more racks at lower density? Wouldn't it be cheaper, easier to maintain, less disastrous if a forklift knocks one over?
Is it maybe something to do with interconnect latency?
Yes, i think we are going to need a new voltage standard for all this shit (datacentres, EV chargers, industrial heat pumps), somewhere between 400V and 11kV.
Most of the other standards are separated by factors of two or three, but there's a factor of 27.5 in that gap.
Can someone tell the IEC, please?
For that size you would install solid copper busbars, a check online and you would need 100 x 6 mm bars @240v or 50x8 three phase to handle the load, I used to work for a company designing that stuff and it was fairly normal. You have to be very precise when calculating the runs as a bend in the wrong place is rather hard to fix.
We did one job where the system had to tested, the best method was to drop another copper bar across all the busbars at once, the machine had to withstand (I think as this was 30 years ago) 30 seconds without exploding.
Very dramatic when it fails…
I am reminded of the anecdotal "spray painting" incident, where a painter set his pot of ceiling paint on a handy busbar trio and it went poof.
We had the big long 480V plug-in style busses along the rafters at one site I worked at, you would slide back an access cover and stab on a breaker-box that would finger-connect to the 4 bars.
Lately with the cost of copper I've seen plants are going to 11kv/13.2kv/13.8kv to a distribution transformer in each zone.
Somehow the math works out that it's cheaper, likely because they can use smaller/cheaper aluminum feeders to the xformers.
In the 1980's we had one where two sections of busbar were bolted together. Well, they were supposed to be bolted. The installers had not put sufficient torque into the bolts and there was a small gap between the two connecting busbars. It worked for years until one morning it decided to fail - and it was dramatic. About a metre of copper just vanished. Fortunately no-one was near it at the time. Took is offline for a couple of weeks.
The one that I still wonder about is the night that a group of thieves broke into a substation for the area of the people I worked for were located - and stole the busbars from the substation.
Without leaving any crispies behind, this substation fed about a square mile of industrial and office spaces so they must have been huge.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
