Intel plans immersion lab to chill its power-hungry chips Intel this week unveiled a $700 million sustainability initiative to try innovative liquid and immersion cooling technologies to the datacenter. The project will see Intel construct a 200,000-square-foot "mega lab" approximately 20 miles west of Portland at its Hillsboro campus, where the chipmaker will qualify, test, and demo …
From an article on Seymour Cray:
For his next machine, the Cray-2, he intended to shift from the usual silicon chips to faster, but unproven, gallium arsenide technology. Manufacturing difficulties forced him back to silicon, and the Cray-2 arrived, delayed, in 1985. Nevertheless, it broke the giga-flop (one thousand Mflops) barrier. This was the machine that was cooled by being completely immersed in an inert fluorocarbon liquid, the same liquid used as artificial human blood. Ever stylish, Cray included a decorative fountain in the coolant circulation system.
Will we get a fountain in these new systems?
An engineer threw me off a CDC 6400 mainframe one afternoon in the 1960s because it had sprung a leak and he needed to check the plumbing. I hadn't realized until then that the 6x00 (and 7600 as well) were liquid cooled.
BTW, in the 1990s, overclockers experimented with running x86 chips of the time immersed in mineral oil. It worked pretty well, but was, I'm told, kind of messy as the hot oil tended to work its way under the insulation of wiring to the power supply, keyboard, monitor, etc. At the terminus of the wires the oil would seep out.
From an article on Seymour Cray: <boards immersed in flourocarbons>
And in a successor company where a friend of min worked in the mid-80s (ETA systems in St. Paul, Minnesota) in the mid-80s, they were working on a supercomputer with CMOS CPU boards immersed in LN2. There were working prototypes, but there was also big fun due all the damage that occurred from thermal stress when you pulled a board out of the LN2 bath or (even worse) put a new one in.
I remember seeing a video of someone making a liquid-proof case and immersing a PC into oil. It worked find. Here it is: https://www.youtube.com/watch?v=YT_fvSQsJ1Q. I don't see where is Intel innovation here?
Reusing thermal energy for another application could also be a way avoid to waste it.
<quote>I don't see where is Intel innovation here?</quote>
<quote>This is important because liquid and immersion cooling requires completely different form factors than what's used in air cooled datacenters today. This unfamiliarity remains one of the technology's biggest inhibitors.</quote>
Is it that they have developed a totally different form factor specially for liquid cooled systems?
Air-cooled stuff like rackmount blades have all connectors on the back and front for ease of slotting into the frame, plugging in keyboards and replacing drives.
That's not very convenient for immersion cooling, as you'd prefer not to immerse the electrical and data connections if you can avoid it - the fluid will creep along the wires under the insulation unless special measures are taken.
So a different form factor makes sense. I presume one of the goals of this project is to come up with form factors that are easier to handle.
We're generating the power to run these systems and then need to generate power to cool them down, so that's a lot of energy running around and warming the planet. We need to start thinking about the consequences of all our "upgrades" - are we going to move to a carbon-free world where everyone needs an AC unit?
Maybe 5 years ago or so, portable A/C became thing "everyone" bought in Australia. The uptake was to such an extent large proportions of the population suddenly had a multi-kW demand per head that wasn't there before in the space of 2-3 years.
This happened to happen at the same time as a bunch of shutdowns of older coal plant, and contributed to nationwide system problems.
In a world where temperatures are going up - for whatever reason - demand for A/C is going to increase, as is demand for power to run them. Which, given the marginal fuel of choice in most countries is gas; any demand you add to the system that wasn't there before is by definition, gas powered. This includes electric cars!
Honestly, I can come to only one conclusion and that is that the planet is well and truly f*#£4% short of being able to get alternative generation sources out in far more extreme numbers than we have done already. Pro Petrochemical ranters / climate change deniers can be found everywhere including on El Reg hence A/C; as I am sure they will leap on this comment as "ahhhh, no, windmills don't work all the time" yada. Which is correct, but if you can cut 50% of gas demand by using them, why wouldn't you use them?
WIth enough of them, backed up by storage; you cut gas demand entirely outside of chemical feedstock.
Climate change or not, this has GOT to be a good thing to do. We have to get on with it.
"Which, given the marginal fuel of choice in most countries is gas; any demand you add to the system that wasn't there before is by definition, gas powered. This includes electric cars!"
And given the efficiency differences between electric cars and ICE vehicles... you can burn oil in a power plant and get more miles than you can by refining that oil and putting in an ICE vehicle.
That we can use gas instead of oil is an even more significant improvement, and the ability (though not yet realised) to use that giant battery as a plug in home battery will make a huge difference to the very concept of generation/supply.
(i.e. I agree, but that storage doesn't have to be grid scale, it can be simply very large numbers of domestic scale systems).
Pricing incentives, and cars/inverters with a network connection are the way to go.
It should be obvious but I don't see this as the whole solution, but we *must* move in this direction, and do so rapidly - so that EVSE equipment is built with the capabilities built in.
> given the marginal fuel of choice in most countries is gas; any demand you add to the system that wasn't there before is by definition, gas powered. This includes electric cars!
Yep, and along with heat pumps, the distribution network is going to need a serious injection of Copper in order to support them.
Your electric meter doesn't show you how much energy is wasted in the "low"-voltage wiring that runs under (or over) your street, but it is significant, and it goes up with the square of current. If you've ever noticed your lights dimming slightly when you turn the electric cooker/shower/kettle on, then please DO NOT get an electric car. Your local distribution network is overloaded and an electric car would waste a huge chunk of the juice it consumes (and any DC load like an electric car will typically draw MORE current as the voltage drops). In the worst case, electric cars in Australia will cause overhead lines to catch fire, with all the consequences that entails.. The distribution network was never designed for them.
> "ahhhh, no, windmills don't work all the time" yada.
Well, sorry but they don't. And batteries will never fix that, not even if we poured all the world's resources into making more batteries.
What could actually save us would be to de-regulate nuclear power. Nuclear power is actually pretty simple and easy, and it would be extremely cheap if people weren't so bloody scared of it. Well what is more scary, slowly starving/freezing/burning to death, or ionising radiation that has been part of the background of this planet since it first became a planet?
As for nuclear weapons proliferation, there are modern reactor designs (Thorium, etc.) that can actually burn nuclear waste and ageing weapons stockpiles as fuel - no need to mine any more uranium, we have enough nuclear fuel to last us centuries. But there are big vested interests (oil companies) who are dead set against that, and they love to frighten and manipulate the hippie types.
Wind power is indeed not good for grid stability, and it benefits the oil and gas industry enormously because of that fact. As soon as the wind stops blowing, the gas power stations start printing money, because every country needs to keep the lights on, and since we shunned nuclear, gas is all we have to do that.
> WIth enough of them, backed up by storage; you cut gas demand entirely outside of chemical feedstock.
No, really you won't. A country the size of the UK needs about 30-40GW to power its grid. When the wind stops blowing, we need 20GW of gas, because we only have 5GW of nuclear (and don't even get me started on the folly of Biomass)
https://gridwatch.co.uk/
For the last week of April and the first week of May, the UK had almost no wind, and used a horrendous 6TWh of gas-fired electricity over those two weeks.
For scale, a factory that produces 1GWh/year of battery capacity is apparently called a Gigafactory. The entire world has around 1 TWh of batteries IN TOTAL, and that includes all cars and grid-storage banks. We would need the world's entire stock of batteries three times over just to replace gas in the UK for one week of calm weather.
And when you factor in the energy cost of producing those batteries in the first place, you can see it's a mug's game.
(I'm not saying that batteries don't help at all - they DO help in frequency regulation, providing a replacement for mechanical inertia, to stabilise the grid against sudden load changes so that you have the extra few minutes required to spin up a big GAS turbine - but they do NOT provide bulk energy storage on anywhere near the order of magnitude that we need, and they never will)
Batteries don’t need to be lead-acid or lithium-ion. Grid scale pumped storage plus wind gets you the cover you need.
And yes, pumped storage can be done on that scale; see Rheenergise. Much cheaper and longer lived equipment than batteries; and doesn’t involve flooding valley after valley.
Wow, that's amazing. Just one teeny-tiny snag: What is this magical R-19 fluid that they say is 2.5 times heavier than water and just as low viscosity? What is it made of (they don't say..) and how do you propose to manufacture billions of tonnes of the stuff? (the nice thing about water you see, is we literally have oceans of it and it rains from the heavens) Even if that were plausible, what is the pollution hazard?
That is the purest greenwash I have ever seen. It is amazing that people like yourself are daft enough to give it credit.
(I'm not surprised that it gets investment though - investors will put their money into anything that they think other people will invest in, so as long as they are in the cool kids club they can pull their money out before the stupid losers. see: Ponzi Scheme)
If they had one single chemist or chemical engineer on their team, or if they would give the formula for their (presumably patent-protected) R-19, then my eyebrows might not rise so high.
It's a mineral oil. I look after other systems which use mineral oils with properties not so far removed that described in the sales pitch - not as high a density as they claim, but certainly low viscosity compared to water. The consequences of leaks of the mineral oils that I look after are fairly negligible. And we successfully contain tens-of-millions of litres of the stuff with only marginal losses per year (of the order of 0.01%). The ideas of the system could be made to work, albeit less effectively, with lower density mineral oils.
Hiding behind patents is something of a necessity as you are well aware. Let your genie out, and the idea will be pilfered by all and sundry.
No question, it's an investment scheme with the usual risks on it. It's also one behind a proven set of technologies that can be scaled up and used just about everywhere without NIMBYism preventing development. The component parts for pumped storage have been around a long, long time. Only the oil element is particularly novel. If you want ponzi, go look at Crypto-farce.
And, not to put too fine a point on it, if this isn't a solution, what is? Burn oil and gas till it runs out? Pray that fusion can be made to work in anything other than H-Bomb format for more than a minute?
Between gambling on something that is known to work, and something that's been 50-years away for the last 50-years I am very much inclined to favour the former. Including building more conventional nuke (however, NIMBY means that isn't gonna happen).
Er, mineral oil is usually less dense than water. Even what sigma-aldrich describes as "heavy mineral oil" has a density of 0.88g/ml i.e. 12% lighter than water.
Bituminous tar might be slightly heavier (certainly not 2.5x), but its viscosity is somewhat problematic...
I have never heard of an oil which is 2x denser than water AND comparable in viscosity.
Mercury could be an option if it wasn't so rare and toxic..
Patents are not something to hide-behind. The idea is that when you patent something, you publish it to the world and use the patent system to protect the idea. But if there is no patent, then it is either "unprotected" against being pilfered, or more likely, it doesn't exist.
> And, not to put too fine a point on it, if this isn't a solution, what is? Burn oil and gas till it runs out? Pray that fusion can be made to work in anything other than H-Bomb format for more than a minute?
As I said: We have some extremely good fission designs that are capable of burning nuclear waste and even old weapons stockpiles with no need to mine or enrich any more uranium. That is literally free energy. CANDU is one example, but there are even better (intrinsically safer = cheaper) reactor designs that we were developing before the world (or at least the West) turned against nuclear.
NIMBYs be damned, nukes are what we need. But you are right, the public attitude, poorly-defined regulation (ALARA principle) and lack of expertise in the UK means that while the Chinese can build a nuke plant in China for less than £5 billion in 5 years, we (or even the Chinese) can't build one in the UK for £25 billion in 10 years.
That was the original use. A cursory look at BM Reports will show that Dinorwig and Cruachan's operations have changed massively over last 30 years. Instead of being "emergency supply when a nuke trips", they operate much more intra-day as balancing services versus variable wind and gas output.
Given that grid losses are very well quantified but you fail to mention them... the rest of the whinge is a bit pointless.
Don't get me wrong, I agree that we should be moving to EVs and Heat Pumps (both offer significant advantages even if we still power them with dead dinosaurs), but your fears for the grid are somewhat misplaced.
However I absolutely agree that we need to get behind more distributed nuclear options. To me motorway service stations are a perfect location for micro plants - they already have decent grid connections, and can have them upgraded, they are generally a little way away from nimby territory and can use the "waste" heat to keep the buildings warm.
EVs have the potential advantage, and I know this isn't yet available, of being an extremely large domestic battery. A domestic load is trivial for the battery to handle, and cars have these nice big DC pins. If we could run the vehicle as a load balancer, since we only need the full range a handful of times a year then the grid would benefit from EVs. In the UK a typical EV would need 4-5kWh/day (20 miles, 4-5m/kWh), and battery sizes are overwhelmingly >50kWh (compared with 13.5 in a powerwall).
"For the last week of April and the first week of May, the UK had almost no wind, and used a horrendous 6TWh of gas-fired electricity over those two weeks."
Nuclear would be an excellent replacement, but the 6TWh used here... are 6TWh that we *don't* use for the other weeks... That's not a bad trade off, although a different marginal source would be good - and that's one reason I lean towards a large number of distributed micro nuclear plants rather than a couple of big ones.
News Flash: Man Made Climate Change is a *MYTH*. CO2 is at EQUILIBRIUM, and is VERY bad at absorbing black body radiation (the thing that defines a 'greenhouse gas') for temperatures ACTUALLY found on Earth... *ESPECIALLY* when you compare it to something like WATER. (CO2 is actually effective on MARS where it's avg -80F or similar even with only that thin CO2 atmosphere)
Saving money on electric bills, however, *IS* important. THAT bottom line determines whether people get a raise next year (or the boss gets his new expensive chair). For THIS reason, liquid cooling make sense. NOT for 'that other bogus UN-SCIENTIFIC reason'.
There will be no peer reviewed references because with regards to this it is bollocks, as normal from Bob.
It's has truths but attributing the different facts to incorrect consequences.
Water vapour does contribute greatly to increases in global warming, increases in water vapour does heat the planet. But to get more water vapour in the atmosphere, the temperature would need to increase (higher temp atmosphere can hold more water vapour). That's where co2 comes in, it has increased dramatically in the atmosphere over the last 200 years. The amount in the last 60 years is the same as what would occur over 5000 years due to natural fluctuations in temperatures that wound increase co2 content in the atmosphere.
This increase in co2, causes an increase in atmospheric temperatures (not caught up to the actual content yet), which increases the amount of water vapour that can be held, which increases the temperature, which increases the co2 and water vapour, which increases....
Co2 concentrations are small but greatly affect the temp of the planet, without it, the world would be a frozen ball.
"Co2 concentrations are small but greatly affect the temp of the planet, without it, the world would be a frozen ball."
If I remember my planetary history right, it was the emergence of plants that triggered the first ice age - they took all the CO2 out of the atmosphere and the temperatures crashed.
Eventually volcanoes added enough greenhouse gases to start the ice melting
News Flash: Man Made Climate Change is a *MYTH*
Oh, thank fuck for that! Here was me worried that the overwealming scientific consensus for the last 30 years was that it wasn't. But you've capitalised it and used little stars too, so that's made me feel much better.
(Sure, actual evidence, or at least an explanation of your years of research and experience in the field so that I can be sure your opinion is worth more than that of my cat would be helpful, but at least I can sleep easy now).
...and is VERY bad at absorbing black body radiation (the thing that defines a 'greenhouse gas')
What do you mean by this? My physics knowlage petered out at A level many years ago, and as I understood it back then black body radiation was used in the explanation of the ultra violet catastrophy. What do you mean by it in this context and how does that equate to the environment?
To be fair, you are correct, that despite being the poster child of global warming CO2 isn't actually that effective as a greenhouse gas, having a 'global warming potential' of 1. There are other gasses that are much much wors, for example R-134A (a common refrigerant) is over 1000 times worse, or R-404A (another refrigerant) which is nearly 4000 worse.
It's a good job none of either of these ever get released into the atmosphere.
(Ironically, the switch to high pressure CO2 as a refrigerant could help reduce these kind of emissions significantly).
Sulphur Hexafluoride's global warming potential is vastly worse again that those refrigerants.
It's used in large quantities on transmission and distribution networks for insulation purposes; a job it's very good at. Hence it's high GWP potential.
Leaks are a thing, and very different design philosophies can be seen between manufacturers. GEC Alstom basically regard leaks as inevitable so they all pish the stuff regularly. Mitsubishi on the other hand their philosophy has been one of stop the leaks happening at all costs.
There are various synthetic, branded alternatives to SF6 being touted, all of which have GWP potential and/or carcinogenic properties.
The best idea would be a return to air-insulated equipment; however nobody makes them anymore because it's big, expensive and very, very noisy. (Like, artillery piece going boom noise).
Genuinely the best compromise on the market right now appears to be a nitrogen/CO2 mix under pressure. Not perfect but would be a vast improvement on SF6. There is the not so minor problem of how do you rewire the entire country to change out the old tech... Start today with budget in abundance, you'll be lucky to be done in 30 years time. The phrase HAH comes to mind.
Call me weird, but I think I'd rather be pursuing energy efficiency and ditching un-necessary processes to save power, than finding ways to jam more power through...
TCO on a datacentre does of course depend very heavily on the Air-con; and something more efficient than running giant fridges definitely has a place. But how much of the workload grind really needed to exist in the first place?
for speed inside of a CPU, higher speed means higher current. Lower voltage reduces power (current times voltage) and smaller logic gate sizes reduce current. The materials determine the practical limits on logic gate sizes and voltages, as well as operating temperature ranges. A *LOT* of variables go into this mix, and current designs are asymptotically hitting those limits.
Just to put things into perspective, anyway... (the physics reasons behind the need for better cooling).
Oh, and component density and location on the wafer plays a big part in sinking the heat away from the places that generate it the most also. Gotta be "clever":.
the main reason why higher speed needs higher current has to do with internal capacitance. It requires a bit of extra current to flip the charge on a gate from a 1 to a 0 or vice versa when you ALSO have to overcome the parasitic capacitance. Think of it as being like a tiny unwanted battery that needs to be charged and discharged in the right direction every time you change the logic state. And there are ZILLIONS of those, constantly being charged and discharged at gigahertz frequencies. No WONDER it heats up! I could get into reactive vs resistive current too but in the interest of being brief I think that paints the right picture.
I went to see a company, Iceotope, in a garage in Sheffield who were doing this with Intel processors ten years ago. They're still in existence.
So Intel appears to be happy to let small companies take the risk, develop the technology, prove the concept, then leap in with huge feet once the issues have been solved and claim it for themselves?
That's what it looks like to me, anyway.
It feels like we’ve come full circle from when Intel’s Pentium 4 ended up running too hot. IIRC one solution they were looking at back then was for micro channels throughout the CPU for liquid to pass through so it could be more efficiently cooled.
I never heard about it again, so either that was when Intel dumped the Pentium 4 and went back to the Pentium 3 architecture or they never got it to work right.
I explained the reasons for power consumption vs speed in an earlier post, above. Reducing power consumption is a goal, not always achievable. And there are patents that would need licensing, no doubt, if Intel starts copying AMD. They each have their own kind of "cleverness" no doubt, and this will rearrange in the future as these things continue to get smaller, run on lower voltages, and crank up the clock frequencies.
(so I'd think Intel is probably trying to become less power hungry than AMD while AMD tries to keep their edge on Intel, for the moment at least)
There are better ways to reduce power than clock cycles. Write your machine code level logic such that Instructions that complete in fewer clocks for instance.
Since resistance is a function of temperature, jamming 600w onto a CPU die is particularly dumb. What workload can you possibly have that needs that spot bandwidth at CPU? The answer is none, as it will be limited by networking and I/O.
Bad solution to a problem of intels own making, and the software worlds intransigence towards ever more abstraction & bloat.
Possibly a better approach would be to review what we are actually getting out of all rush for "AI".
Increasingly it is needing monumental amounts of resources in compute, data centres, power & cooling etc. Yes it can do some funky stuff but 600w for the chip alone. It does not matter how "green" or "renewable" the energy and cooling for the DC is, AI is still using huge amounts of power. It is similar to the situation around bitcoin mining but because it is "AI" it is acceptable.
I am sure that there are benefits, it is just that with IT, the impact or doing it and the resources used are quietly buried.
Look at the proposal Meta (Facebook) had for a site in the Netherlands:
https://www.datacenterdynamics.com/en/news/facebook-owner-meta-suspends-zeewolde-netherlands-data-center-due-to-political-pushback/#:~:text=Facebook%20owner%20Meta%20has%20paused,favor%20of%20reconsidering%20the%20project.
So 200MW and a footprint in hectares. that is an awful lot of power and space.
Last time I saw someone testing immersion cooling the answer was that it worked very well, but it was an absolute nightmare for maintenance.
Apparently, even if you can rig a way of draining the thing down without flooding the surrounding area with vile, slippery gunk, all the bits are liberally coated with it and it doesn't wash off easily.
Oh and you still need heatsinks to get the heat out of the hot components and into the gunk, same as you do with air.
Just what is wrong with the tried, tested and available off the shelf solution? Water cool the hot bits. All you need to build is the appropriate heat exchangers for your componentry to plug into the cooling loop.
In the data center I worked in for almost a decade (until mid 2021), immersion cooling showed up in perhaps 2018. Conventional 2U/4node servers were used (minus the fans & using a metal foil versus thermal paste for the CPU heatsinks).
"Apparently, even if you can rig a way of draining the thing down without flooding the surrounding area with vile, slippery gunk, all the bits are liberally coated with it and it doesn't wash off easily."
As the person handling server re-cycling, I have no idea what vendor would have been glad to see greasy pallets of servers, to load onto their truck...
"At scale you don't bother servicing/replacing an individual unit."
We repaired them. I personally added a second M.2 drive to 400 of them. Quite messy work and somewhat dangerous as pulling servers sleds vertically out of a mineral oil vat, being slippy gravity wants them to fall back in...Scale? Well perhaps modest in the scheme of things (perhaps 1,000 total "swimming" servers).
They use HP Moonshots immersed in some sort of oil (they won't say what sort, though). The excess heat goes to the local swimming pool, for which they get some token tax cuts and a good bargaining position with local municipality. They say they ordered so many Moonshots they were able to get a deal from HP under which they didn't have to buy all those high availability fans, accounting for savings of 20 € apiece, which in turn paid for all the necessary immersion cooling infrastructure. They also said they are utilizing the Moonshots to such a level that the air cooling would struggle and they would have to spec the AC to such a gradient that the energy consumption would render the traditional air cooling financially infeasible.
At an exhibition years ago, a company had a CRT telly, working, submerged in a large fish tank complete with plastic fish and foliage.
When I asked, it turned out to be dry cleaning fluid or something. Unfortunately I failed to ask what would happen if the tv set fire to the whole thing.
Intel haven't really come up with anything new here.
With Intel/nVidia power consumption, the KFC Console concept is becoming a reality.
Except in the data centres they'd be able to actually cook the chicken rather than just keep it hot.
Maybe it would be an idea to build fast food outlets into data centres, or vice versa.
Or build data centres next to housing developments so that the heat can be used to heat the homes cheaply.
A book on nanotechnology/molecular assembly positing a supercomputer about the size of your fist that would boil so many gallons of water per second pumped through it.
Apologies for lack of details my head very much elsewhere right now. Multiple small children wanting things, fighting, and yabbering on like there was LSD on the pizza.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
