How'd your servers get that baby-smooth look? Dutch and Brit cool kids dunk Supermicro systems in synthetic oil

Dutch liquid cooling specialist Asperitas and British systems integrator Boston have linked arms on a range of server systems based on "immersion cooling" – with the hardware submerged in giant tubs of synthetic oil, a dielectric compound very similar to Vaseline or baby oil. Four fully contained standalone systems were …

[Steve K]

Practical question

I can see the benefits of this, but from curiosity how do you remove the oil coolant from servers/parts when you need to replace hardware?

Or (since it is dielectric) does it not actually matter (as long as the parts don't slip out of your hands!!)

[chris 143]

Re: Practical question

I don't think it's significantly more magical than lift it out and try not to drip oil all over the floor.

I've seen demos where they'd more or less taken the top off a 1U/2U server and removed the fans then dunked it in a vat of oil.

Apparently ssds/helium hdds are happy enough being submerged too. Although I imagine it'd invalidate any warranty

[Mark 85]

Re: Practical question

I shudder to think of the messes and slippery floors doing that. I guess hot swappable is out of the question then?

[Gordon 10]

Re: Practical question

I shudder to think of what the lubed up BOFH's and engineers would look like afterwards.

[Steve K]

Re: Practical question

They'd have lovely, smooth skin?

[Roo]

Re: Practical question

At most datacenters I've worked with entire boxes are swapped out - the broken one taken away for further examination/repair/exchange/disposal. As long as they have nice self-sealing connectors for the plumbing they'll be fine.

The boxes will be substantially more dense than normal ones though - the oil will be heavier and they appear to be packing a lot more components into the same volume than a vanilla box... Perhaps professional wrestlers could do the job - they should be well practiced in grappling with heavy oily units. :)

[Anonymous Coward]

Cinzano?

> thanks to their self-contained nature, such systems can be deployed in any kind of building, in any climate – no data centre or dedicated server room required.

The heat still has to go somewhere. If the oil bath doesn't go through a heat exchanger then the heat is still going to warm the room, just a bit more quietly than if it had fans.

[chivo243]

Re: Cinzano?

...and a secondary warm water circuit ejects the heat from the system" Isn't this a heat exchange?

[Anonymous Coward]

Re: Cinzano?

> ...and a secondary warm water circuit ejects the heat from the system" Isn't this a heat exchange?

You're right of course, but that doesn't really fit with their 'self-contained' and 'any climate' statements. If they'd just said "needs less intrusive cooling", or something of that ilk I'd be fine with it.

TBH, I can't see my clients with traditional DCs buying this - if you've already got all the air con why would you then even consider adding more complexity to your cooling arrangements? And if you haven't got a traditional DC why wouldn't you use the cloud?

[Anonymous Coward]

Re: Cinzano?

If you had a lot of these, I would think the best solution would be a direct exchange system, where your HVAC has a "freon" (whatever the current substitute is called) loop piped to each cabinet.

The bigger problem would appear to be weight, though if you designed your datacenter around this you could dispense with the raised floor and it wouldn't matter if you had 3-4x the typical per sq ft weight.

[Arthur the cat]

A shame (Si) chips can't run hotter

Or you could use vegetable oil instead of mineral and combine your machine room with a chippie.

[Steve K]

Re: A shame (Si) chips can't run hotter

Or a water-bath/confit if temperatures aren't high enough for a chippie (sorry - been watching MasterChef)

[TRT]

Re: A shame (Si) chips can't run hotter

Sous vide-o output?

[GrapeBunch]

Re: A shame (Si) chips can't run hotter

Or perhaps the temps are about right for decarboxylation.

[vir]

Respect Your Elders

They need a Cray-2-style waterfall heat exchanger somewhere in the system, just to show who got there 30 years ago.

[Anonymous Coward]

Re: Respect Your Elders

But not the hideously expensive Fluorinert used as the medium.

I do sometimes wonder if Cray thought about using purified mineral oil at around 1/1000 of the cost and then thought, no, expensive computer needs expensive cooling system, and if everyone else thinks fluorinert is essential, they won't copy us.

[Chz]

Re: Respect Your Elders

I was going to say that the Fluorinert couldn't be huge next to the total system cost, but then I looked it up and a Cray-2 was "only" $15M. Seems cheap next to today's massive clusters!

Still, a 750lb barrel of Fluorinet is "just" $35k. Old Seymour didn't believe in doing anything if it wasn't all-out. Bless him. His biography is fascinating.

https://www.chemworld.com/Novec-FC-770-Fluorinert-Electronic-Liquid-p/novec-fc-770.htm

[Ken Moorhouse]

Not suitable for Bad Programmers...

All those Memory Leaks will lead to flooding.

[Tezfair]

new spin on old tech?

I recall this being a thing some 20 years ago

[Anonymous Coward]

Re: new spin on old tech?

tl;dr

Google aquarium pc

[martinusher]

Re: new spin on old tech?

Back in the early 1980s I worked on a R/F amplifier that was cooled by purified water. The water had to be ion free because the amplifier tube that was being cooled was sitting at around 7.5Kv. It worked well enough except in frosty weather -- the water froze when the prototype wasn't running.

(I've used a similar setup on load resistors I used for testing high power audio amplifiers. Nothing sophisticated here, I just dumped them in a bucket of water.)

[Colintd]

Re: new spin on old tech?

I've got a dummy generator load which is an oil barrel with water and caustic soda inside. Neutral (earthed at Genny) to barrel, lengthy of live wire stipped and dangled in middle of barrel. Works a treat as long as you have good ventilation (to avoid explosion).

[John Brown (no body)]

Re: new spin on old tech?

Those pole mounted transformers have been oil filled for many years too. As you and others have said, it's not new technology.

[Anonymous Coward]

Re: new spin on old tech?

"I recall this being a thing some 20 years ago"

me too - seem to recall TomsHardware building a system immersed in oil which they were able to overclock significantly.

[Adrian 4]

Re: new spin on old tech?

These things go in cycles (not just spin cycles..) - in 10 years time when liquid cooling is the norm, someone will be pushing the latest thing in air cooling, no plumbing, no leaks.

[Jan 0]

Re: new spin on old tech?

Nearer to 40 years ago, by Seymour Cray at CDC. I can't find any pictures but I recall the same design appearing in Wim Wender's film "Until the End of the World".

[elDog]

I don't believe it. Lend me a few top-end units so I can test myself.

I'll supply the mineral oil (baby oil), canola, corn, sesame, peanut should work also. Besides, think of the aromas!

Now to come up with a workload to put my demo units through their paces. Perhaps "Answer to the Ultimate Question of Life, The Universe, and Everything". Shirley with these specs the "Anna AIC (CLOUDzone243) is intended for GPU-based workloads like 3D rendering and machine learning – featuring the same 1,056 Xeon cores, but adding a whopping 331,776 CUDA cores across 72 integrated Nvidia cards" should be able to compute 42 in less than the 7.5 million years (probably performed on 8-bit CPUs.

Anybody have a spare hydro plant to loan me for a few million years?

[Anonymous Coward]

Re: I don't believe it. Lend me a few top-end units so I can test myself.

Now to come up with a workload to put my demo units through their paces

Windows 10 with Tetris ought to do it..

[Anonymous Coward]

Is this really comparable to 3M Novec

Not sure if this is comparable to 3M Novec as this sounds like a traditional liquid cooled system compared to Novecs novel boil off heat transfer method. I imagine the price differential would be similar to the performance differential between the two solutions. Either way its cool to see this tech making its way to our datacenters.

[Gnoitall]

May be more than just magic smoke

Jokes about the computer working hot enough to catch fire won't be that funny when the coolant ignites.

[Anonymous Coward]

Snake oil ?

[Starace]

So they've fixed the problems then?

Other times this has been tried it usually didn't work too well - poor circulation/localised cooling, reactions between components and the oil and issues with dielectric effects all tended to cause problems.

Also someone mentioned hard drives - they don't work for long either as either the pressure equalisation valve or a seal will give out and let oil in.

It's not that immersion cooling doesn't work at all, but you can't do it cheaply and expect a lot of success.

[ElReg!comments!Pierre]

Re: So they've fixed the problems then?

poor circulation/localised cooling

That's why you should use diesel as the coolant.

https://www.theregister.co.uk/2002/05/11/bofh_and_the_vax_cluster/

[circusmole]

How about using...

...castor based oil that used to go into my old Manx Norton. You then get the benefit of the glorious smell.

[Wellyboot]

Re: How about using...

Castrol-R - the racing smell.. good memories

[Anonymous Coward]

Re: How about using...

IIRC I read that in WW1 castor oil lubricant (used in aircraft engines so crude I just cannot image someone taking off in a plane powered by them) was inevitably taken in by the pilots in the exposed cockpits of the day, causing the runs.

The cure for the runs was whisky.

Part of the death rate was due to drunk pilots.

The thought of a data centre full of IT staff either headed for the loo or on the whisky bottle is a little concerning.

[M.V. Lipvig]

Pretty cool

Put your gear in clear glass cabinets, then when the workload gets heavy watch the oil start to boil. Be like working in Frankenstein's IT room.

[Anonymous Coward]

Re: Pretty cool

I've some datasets to train the nVidia version here that should quite do the job!

[My first thought as well.]

[Roland6]

However, due to their unusual footprint (600x1200mm), these systems are unlikely to serve as drop-in replacements for a standard 19-inch frame.

Note to self - they also won't fit in a 21-inch Open Rack standard cabinet footprint...

[Blockchain commentard]

19" rack is about 600 x 1000mm so remove the rack, slip this in (pun intended) and you only lose 200mm/8 inches of inter-rack corridor space. Can't see a problem really.

[Bronek Kozicki]

... and there is also a question of weight.

[Asperitas]

Hi all, we noticed a lot of questions around this article.

It’s impossible to answer all individually, that’s why we’ve planned a webinar on Friday, April 12th, 2019.

During this webinar Asperitas and Boston will cover all your questions. Register for free at https://eveeno.com/the-future-is-now

[HieronymusBloggs]

Dielectric

"The company uses mineral oil, which is much cheaper than proprietary dielectrics"

My computers use the cheapest dielectric of all (air) for cooling.

[G R Goslin]

Prior Art?

A long time ago, Shortly after the war, in fact, I read an article where the principle was tried in America. It was better than the system proposed here, in that it used a low boiling point fluid, One of the Freons, I believe. The advantage here is that taken away from the housing, the coollant would merely evaporate away, leaving the kit clean and dry. There was too the advantage of a boiling coolant, that the temperature of any part of the assembly could never achieve a temperature in excess of the boiling point, and the coolant didn't have to be pumped around the cooling circuit.