LiquidStack CEO on why you shouldn't ignore immersion cooling

As chipmakers demand more power than ever, with some chips pushing 700 or even 1,000 watts in the case of some of Nvidia's upcoming parts, datacenter operators are having to get creative about the way they cool these chips. One of the technologies gaining popularity in the wake of these trends is immersion cooling, a process …

[Roland6]

We've had data tanks running up to 250 kilowatts in a 48U form factor for almost seven years now.

Given the typical size of a server rack cabinet, that’s circa 2 cubic metres of coolant ie. 2 tonnes of additional weight…

[Brewster's Angle Grinder]

Were you planning on putting it in orbit?

[John Riddoch]

Probably not, but 2 tons in a relatively small footprint will likely break many raised floors. You can't just wheel one of those cabinets into a standard data centre without extra planning and likely building works. Hence why these solutions work best on a "new build" DC where you can plan for the extra weight on your flooring.

[7teven 4ect]

Reinforcement

People won't be walking around in there, so you don't need regular office floor. Dig out a five-story basement, support 4x the density of server with a big metal frame thingy, designed for the purpose. Plus, depending on the density of the coolant, the buoyancy effect will reduce the weight the frame would have to bare, and the bedrock would hold the weight of the oil, frame, and server farm, no problem.

Then build many floors of mixed use space above it and heat it all using the waste heat.

[NoneSuch]

I suspect the BOFH Boss will end up in one of those tanks soon. Coroner found drowning with him found wearing bathing trunks, a mask and snorkel.

Tragic.

[Cybersaber]

Delicious possibilities

Or else the servers going offline because the PFY was experimenting with coolant oils that could double as deep fryers for onion bhajis.

[Cybersaber]

Here be monsters

No, because there are krakens up there and monsters are not part of most threat assessment scenarios. Many small green lives were lost in the pursuit of this knowledge.

[Richard 12]

... in a 48U form factor

Depends what the coolant is, and what it's replacing.

Large heatsinks are pretty darn heavy, so if you don't need those big blocks of metal then perhaps it evens out?

Sadly the whole interview was rather vague, I'd hoped for a sales pitch with some real meat, :(

[ArrZarr]

Re: ... in a 48U form factor

Bear in mind that you still need a heatsink on the CPU to transfer heat into the coolant. The Mineral Oil submerged PCs I've seen all use huge air-coolers anyway to get maximum heat transfer from CPU to coolant. The same heatsink will shift more heat in liquid immersion rather than pure air but you still need something.

The most common coolant for submerged PCs is mineral oil (unless that's changed in the last 7 years or so). 1L of mineral oil weight 860g, which is 86% of the mass of water. The weight of a Noctua NH-D15 is 1320g, so assuming you can knock 1kg off the cooler by immersion, that still only accounts for one and a bit liters of the oil.

I don't have the volume of a 1U server but I know it's way more than 1L

[7teven 4ect]

Deviant Olam on YouTube

Another advantage of this, is that provided all outgoing cabling is suitably hard to reach and interfere with, there is little chance that Deviant Olam, famous physical pen tester (also electronics, rfid, automatic door, elevator hacking etc), will let himself in, plug in a serial cable, and you know the rest.

[Richard 12]

Typo or mis-spoke?

you're only consuming 0.5 additional watts for cooling, versus half a watt for traditional air cooling systems.

0.5 and ½ are the same...

I assume he meant 0.05W vs 0.5W, so 1/10th the 'wastage', but unclear where the mistake occurred.

[dubious]

nope

We trialed immersion cooling years ago in our DC. It just made everthing more difficult. Swapping out a dead server was a horrible task, but it was the non-obvious things that were worse - any cables in the oil turned rigid as the plasticiser was leached out, and the oily coolant wicked up inside any cable. Fun when the leak alarms go off and you find oil dripping down the pdus and switches. Still finding that crap in places. Also with ours being a horizontal 40u-ish tank there wasn't really any overall space saving anyway.

[Lomskij]

Re: nope

We're running immersion cooling in our DCs with no problems - there are adaptors for power and network cables specifically to prevent the capilarity you described in your post. Regarding the lack of benefits - we're running 100kW worth of GPU servers in each 42U tank - good luck achieving that in air cooled racks.

[Lomskij]

The guy has no idea what he's talking about. PUE is the ratio between total energy consumed by the data centre and energy available to the IT equipment. PUE of 1.5 means that for 100kW consumed by the servers, 50kW is consumed by infrastructure such as power delivery and heat rejection. As a ballpark, these 50kW would break into: 10kW transmission loss (transformers, UPS etc), 10kW rack cooling, and 30kW heat removal - CRAH, chillers, water pumps etc.

While it is correct that immersion cooling tank has incredible power efficiency, especially compared to air cooled racks, the consumption of transformers and chillers will stay the same. So if you take a typical HPC data centre with PUE of 1.5 and replace all OCP racks with immersion cooling, it'll drop the PUE to 1.4 at best, not 1.05.

[Anonymous Coward]

Don't waste the heat

Data centres need to be positioned near to swimming pools so the excess heat can be used to heat them up. It would stop leisure centres going under due to the rising cost of energy.