Crazy Texans dunk servers in DEEP FRYERS I first met the Green Revolution guys back at SC09 in Portland, Oregon. As I roamed the exhibit hall, people kept telling me to check out “those guys with the deep fryers full of servers”. At last I found them out in the lobby, which is the kids’ table section of the show. Above is a quick video of their demo that I shot …
Beware the caps. Most of the components on a circuit board will be fine in liquid, but not electrolytic capacitors - it seeps in slowly, over the course of months. I wouldn't be happy putting a server under oil unless I knew the manufacturer had properly tested it for reliability long-term immersed, and it remained covered under warranty.
I would have thought that would only apply to the older aluminum foil dielectric soaked paper high voltage capacitor, the sort you find in PSUs. But then you could remove the power supplies from the servers and mount them externally.
IMHO I don't think it would apply to the sealed tantalum bead or surface mounting types.
Just curious, does the oil in the "coolers" get hot enough to deep fry mars bars?
Paris, because like electrolytic capacitors, her capacity is usually prefixed with 'micro'
Ever been to a state fair in the US? You will immediately feel good about yourself, for one thing.
I will say that some of the things sound disgusting and while not my choice of food, are tasty. I had a nibble of a fried Snickers and it was good. (Of couse, that one bite gave me my daily calorie allowance.) Heck, Simona de Silvestro, a Swiss driver in Indycar and hardly a sterotypical state fair type, has developed a taste for corn dogs after a trip to a state fair.
I was wondering about that, but also about the impact of a immersive cooling oil with relative permittivity values of 2-3 on the characteristic impedance and propagation delay-matching of high speed PCB tracks?
As a designed-for-oil system no problem, but as a normal board in oil it might cause issues with very high speed systems.
Mineral oil has a flash point of 170°C and a boiling point of, I think, 310°C. It will burn, but by the time it starts burning the rest of your data centre is already so far gone that a regular air-cooled system would have been reduced to twisted metal and ash. In fact it may well preserve your hardware against fire and subsequent water damage.
Possibly.
Indeed it's been around for many years with enthusiasts and overclockers.
Usually they pop up mounted in a fish tank, with all the rocks, decorations and even a few plastic fish thrown in for good measure.
Cool to see someone's taken the idea to the server level, in theory it's a very efficient way of keeping things cool.
Aaah, synchronicity... first time I've first heard of mineral oil cooling was on Friday, when I've stumbled upon Puget Systems. Don't berate me.. I'm an accountant, not a techie. 8-)
And now I see that on the same day El Reg made an article about mineral oil cooling.
Cray were running their gear (Cray 2) in Fluorinert 20 years back. Tomorrow's World did something similar with a TV. Mind you, that may be why we have a hole in the ozone layer. Let's hope there's nothing nasty involving oil then...
I think Cray 2 kicked off the whole pimp my PC case pre Alienware. They looked lovely in the dark.
Cray found the same problems that these folks must have hit - laminar flow.beomes an issues at high heat transfer rates - the fluid does not move near the surface and at limit can end up "boiling". WIth cray the fix was "obvious" - instead of immersion use fuel injectors and use droplets of a specified size and use evaporation.
Of course there was more to it than simply pointing injectors at the chips - and the cooling fliud would have to have certain characteristics that the quoted oils do not have.
Looks like they have a design that would allow you to remove a single rack server.
You would need a rack like a dish rack to catch all the dripping oil, and would certainly take some time to clean up your hands afterwards too.
Not a very efficient way to replace parts like HD that need frequent maintenance in a high capacity data center.
Also worried about leaks in the "capsule" that protects the HD.
These ideas are old just with twists, just look at toms hw back in 2006!
http://www.tomshardware.com/reviews/strip-fans,1203.html
Plus, mineral oil comes from petroleum distillates while making gasoline and while the overall cost to acquire it is low, it still requires crude oil to make and hence not green at all. Then in one video they claim it saves 20% power by replacing fans, this may be true for calculating the entire datacenter, but i just don't see it for small environments, a server with a high end cpu fan only consumes 7-8W max, which is typically less than 10% of the overall wattage a server consumes. Add in mineral oil cost of manufacturing, cost of cooling infrastructure and your breaking even at best. I have seen datacenter shipping container manufacturers with this type of cooling, so again, there is nothing new about what they are doing.
Agreed this idea has been around for many decades, military and industrial use of oils, flouronert etc are common and old.
You miss the power calculations though, they didn't just remove the fans on the PC, they can also remove or severely downsize the HVAC/CRAC cooling units for the Data Center room, and the result of that is the ability to reduce the sizing on the UPS units. Both of these factors result in a large amount of savings, there are much better ways than the proposed to solve this....but I can't give away all my good ideas for free... :-)
They have several problems with their demo:
Mineral oil doesn't remove heat, it absorbs it (to a point) and then releases it into the room its housed in. This works great for home PC's that you heat up all evening playing games, and then cool down overnight, but would be terrible for a non-stop international supporting data center. They are glossing over that their demo is in a giant super AC cooled exhibit room.
Mineral oils also tend to work the stickers off of the MB, cards etc. Easy enough to remove these by hand for 20 servers for a demo, but not a realistic practice for 20,000 servers in a data center.
Also their Data Center would have to be a like a clean room, the open top would attract dust, hair etc and would become murky and dirty, and that could clog any pumps fans etc that are used to circulate the mineral oil over the hotter parts of the PC (PSU and CPU)
Liquids are like 3000 times more effective at cooling than air though.
I've seen working models, and proposals for liquid metals for cooling (not direct contact of course) alcohol (yikes) and many other industrial chemicals including and similar to flouronert.
Still good on them for trying and making people think
I totally agree, and I didn't fully run all the calculations and understand that fluid is more efficient at removing heat.
But the thing I was trying to imply, is that the grcooling isn't really all that efficient. It uses a 3.5KW pumping system that requires a water inlet of upto 120gpm to exchange heat. So the cost and efficiency of this makes no longer makes sense as the infrastructure to get water in a datacenter and power to run the pumps negates alot of the benefit.
Presumably they would be pumping oil around the tanks and via radiators. Plenty of scope to have various sedement bowls and filters in-line. This would keep the oil clean and clear of contaminants and be easy to maintain.
I built a homebrew oilcooled PC and used off the shelf automomotive oil filters and a cooler from a long deceased motorbike. As they are quite capable of keeping the oil in a diesel engine clean of contaminants I'm sure a rack of servers would not present any significant challenge-an internal combustion engine is a FAR mor demanding environment.
Thing is, an internal combustion engine will run for a considerable amount of time (until rather significant bits of it are dropping off) with the dirtiest old oil or even no oil at all and the roughest fuel you can find, I can quite imagine entire racks of servers going titsup because a filter clogs up or a pump flows less than it's meant to (all of which I'm sure will be monitored but), the slightest contamination in the oil and you've lunched your servers.
It must be at least 8-10 years since I put an optical disk into a drive in a server.
Most of these operations are lights out. You will mount ISO images of application images you need to install. You will build over the LAN via a build network. You can also mount images via the remote lights out type capabilities that most modern servers come with these days.
One of the questions I have is cable entries and exits into the oil bath. These will degrade and go very rigid in time which can cause complications.
Also the oil can "seep" via (I guess) capillary action down the cables (internally) and into equipment outside of the oil enclosure.
(Real world experience, I have had this happen with mineral oil which travelled UP around 8-10 inches above the level of the oil, and down the power supply cable and into a 4 way which got swamped with mineral oil.)
You raise some good points.
I think there are IP68 protected network cables available which wouldn't allow capillary seeping internally.
I think HDD's would be out of question also because if/when the oil gets to the platters the head and spindles would slow down and produce at least SMART errors. Network booting would of course help in this case..
Also, running ordinary HP/Dell/IBM/whatever servers without cpu/mobo/power fans would probably require tinkering with the mobo or a custom bios because a missing fan would probably cause several errors during boot, or the server log would be filled with error messages.
If the servers are connected via fiber, how much would the oil interfere between cable connector and the server's fibre interface?
Having myself designed and built servers and storage designed for ambient AIR cooling to 45C, the idea of oil-immersion cooling seems to me to be overkill for the data center.
Several major players have started shipping CRAC-free systems clusters in container modules, with densely packed kit, so the advantages of oil have flowed away. The article failed to mention the need for drip pans everywhere, and woe to anyone swapping out a disk drive!
How much would the volume of oil required for say a "typical" Virgin Media or similar data centre weigh?
You'll still need about the same weight of materials when you take out the cooling bits and add the oil proof boxes and pumps, give or take, but the nice light & fluffy air has been replaced with liquidy, dense & weighty oil. How many data centres are currently half way up a building and would need to be either relocated or reinforced to take the weight? Load bearing structures can only bear so much load.
I smell a cost implication here.
WATCH IT! A COST/BENEFIT ANALYSIS NEEDS TO BE CARRIED OUT!
Since we can't use Freon for this purpose any more, and, of course, water dissolves all sorts of ionic substances and quickly becomes a good conductor, the use of mineral oil to get more heat away from electronics more quickly has long been an obvious thing to try.
But there could be issues - like the one about capacitors pointed out here. And, ideally, to realize the full benefits of a medium that has a higher density and heat capacity, there should still be fans on the processors, modified ones that will circulate mineral oil instead of air.
Your electric supply transformers are all oil cooled, and they wouldn't have it any other way. They (almost) never catch fire/burn. So that is not a problem. Oil is cool. It is a good idea and the only problem they are going to have is HD failure. Think of the mess. Aside: transformer oil makes bucky balls, given enough zappity.
I agree it's efficient and don't see any problems with it, but I wouldn't want to be the one that has to pull out the drippy, slippery, gooey mess to change a failed RAM module or similar.
Plus, once you do pull a dead module and the oil gets on the socket contacts, will it make good contact with the new module, or would you have to thoroughly clean the contacts with solvent, let them dry, then install the module?
Old IBM iron used liquid cooling because it was so much more efficient to remove the heat. Oh wait, these blokes weren't born back then.... now CMOS power densities are approaching the old bipolar numbers and guess what?
Liquid cooling is baaaack!
Hope they do a patent search before getting too enthusiastic with a start up.
It looks as though the target market is a bit specialised, perhaps more super-computer applications than ordinary data centres. More efficient cooling at the chip level gives you the chance to "overclock". So the server boards may well be specially made to cope with the oil, and use heat sinks that are optimised for oil immersion.
Yes, nothing new under the sun, but getting it working for current styles of hardware is a pretty smart piece of work. Now, how about an oil-cooled Raspberry Pi?
[Note: deep fryers run hotter than electronic components can survive. It's about 50C above the mil-spec temperature upper limit]
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
