How thermal management is changing in the age of the kilowatt chip As Moore's Law slowed to a crawl, chips, particularly those used in AI and high-performance computing (HPC), have steadily gotten hotter. In 2023 we saw accelerators enter the kilowatt range with the arrival of Nvidia's GH200 Superchips. We've known these chips would be hot for a while now – Nvidia has been teasing the CPU-GPU …
They're putting RAM inside the processor package. HBM. It's more expensive and less flexible, but faster. You don't need pads for the RAM bus, so you can make that bus much wider and faster.
High end graphics cards, AI accelerators, and other expensive chips are getting this first, but I expect that in time it will roll out much more widely. The whole "multiple chips in a package" thing is getting much more common, and the RAM chips are just another chip to put in that package.
《Given that they run at 1.8V or lower, then that means the pins/balls/bondwires are carrying in excess of 1kA. That's the bit I can't get my head around!》
I would consider higher voltage and lower current and distribute and down convert on the chip(s.)
1.0 kA DC is insane - I vaguely recall that most of the current is adjacent (under) the conductor's surface so what must the current densities (V/m2) be?
Probably time to revisit some cryogenic technologies: Josephson junction?
A sudden coolant failure not a pretty picture - 2 kJ/sec with nowhere to go works out about 0.5°K /sec for a litre of water but a probably lot less than a litre and probably not water.
A circulating coolant has to transfer the heat somewhere which probably involves vast quantities of fresh water or massive fans and radiators - thermodynamic duck shoving - not great for the environment or planet generally.
"I would consider higher voltage and lower current and distribute and down convert on the chip(s.)"
That is exactly what they do, the voltage that is supplied to the CPU is the SoC voltage, this is then down converted to the core voltage on package. Intel use Digital Linear Fully Integrated Voltage Regulators and AMD uses a setup with Digital Low Dropout Regulators.
Err, except that linear regulators and LDOs do not "convert voltage to current" like a switching regulator can.
If an LDO wants to convert 1.8V to 1.5V, it draws the same current at 1.8V as it supplies at 1.5V, and it throws the excess energy away as heat.
These devices are used for smoothing (so that the resistance of the bondwires does not cause voltage ripple), not for power supply efficiency.
No, the trouble with a series circuit is that each element always takes the same current. That's Kirchoff's Current Law.
If one of your CPU cores wired up in series wanted LESS current than the others, then it would suddenly see a higher voltage than it wanted and get fried.
There's also a problem of not having a common ground voltage between CPU cores. How do they communicate when core 1 is at 1.2V and core 10 is at 12V?
Yes, completely correct. That is why I was saying groups of CPU so that you would be able to better average each step in the series.
Another thing you would need is load management so as to even out the current draw.
I had read somewhere that there was a push to optical buss between components. If this is the case, it would solve the common voltage for data interconnection.
No good ideas make it to production without facing the firing squad. Guys, see how many holes you can make and solutions to plug them.
It may spark an idea that will save massive Datacenter power consumption.
> 1.0 kA DC is insane - I vaguely recall that most of the current is adjacent (under) the conductor's surface so what must the current densities (V/m2) be?
We used to worry about Electromigration. (High current density over time damaging the atomic structure of the conductor and increasing resistance). Is that not a problem with today's chips?
I remember worrying about that 15 years ago, and I can only imagine it's gotten worse as features have gotten smaller.
My pet theory is that we're on the verge of a physics and economics breakdown where parts wear out faster and faster and yields drop lower and lower while costs soar ever higher. In fact, I would hazard a guess that this is part of why modern processors (and not just the "central" variety!) are more prone to failure versus units from, say, the 80's. Which begs the question, when will this start negatively affecting appliance prices, like Tesla vehicles..? Actually, the automotive market would probably be a good place to watch for early signs of problems with longevity, as that should be reflected in vehicle prices and repair rates (if you can find that data on the open market).
If you look inside a modern computer you'll see that the CPU and GPU are surrounded by synchronous buck converters to make ~1 volt at hundreds of amps. That feeds a whole lot of chip pins. If all goes well nothing burns up. There's no practical tech to do this inside the chiplet package. 1 cubic cm of buck converter gets you about 40 watts.
The next practical step would be teaching AI how to improve chips and code. Both contain many layers of simplified structures that exist to keep complexity withing human grasp. A powerful analyzer could shortcut all the formalities and intermediate steps.
1.0 kA DC is insane - I vaguely recall that most of the current is adjacent (under) the conductor's surface so what must the current densities (V/m2) be?
That's the skin effect and it applies to signals, getting progresively worse as frequency increases. It's a non issue for DC power supplies that are not supposed to vary in voltage.
Air cooling isn’t necessarily as incapacious as liquid cooling. It just depends on what your air is.
There’s some avionics out there that use external airflow admitted directly into processing racks. If you have infinite high speed external airflow available, air cooling does just nicely, and if it’s pre chilled to -40C so much the better.
Admittedly one has to deal with rain, bugs, ice, chaff, dust, shrapnel, careless ground crew and smoke being blown in with the cooling air too, but that’s manageable.
My father used to be in charge of the cooling at a meat packing plant, so a lot of his days were spent thinking about ammonia. The plant he worked at happened to be in a climate where things could get quite cold in the winter, well below zero, so one day he just asked why they don't simply open a door on the roof in the winter instead of spending all kinds of money to pump in ammonia. This apparently saved the company so much money when used across all their plants in sufficiently cold climates they gave him an award.
IBM mainframes had liquid cooling in the 80s. The modules themselves were encased in liquids and the entire apparatus was cooled through a cooling tower where pipes were ran under a raised floor that was also used as a way to bring in cold air and hide cables. Ah, a trip through time, that's when a 2.5 GB disk drive cost $60K and was as big as a closet, never mid the mutli i/o channel controller in another closet size cabinet.
Sadly the Crays of the era used a constant amount of electricity regardless of workload and were described as looking like a "big resistor" as far as the building's electrical schematics were concerned. Probably why they required somewhat exotic coolant; IBM and ICL mainframes just used distilled water. Talking of which, I forget how much heat e.g. an IBM TCM (thermal coupling module) was expected to deal with, I think about 300W each for a 3090, each CPU having around 15 TCMs (that number may be completely wrong but it was quite a few).
Sadly the Crays of the era used a constant amount of electricity regardless of workload and were described as looking like a "big resistor" as far as the building's electrical schematics were concerned.
Depends on which era we're talking about. That was certainly the case with the early ECL Crays, later they moved on to CMOS (much later than the rest of the industry). Probably early '90s or thereabouts.
AFAIK this was just standard, at least at the time; I would assume that hybrid heating & cooling systems are more common nowadays, but I've assumed lots of things that turn out not to be the case. ISTR some of the newer ICL stuff I saw 30 years back used what IBM calls a radiator in that they were water-cooled but instead of being plumbed in to under-floor piping, they dissipated the heat straight into the air, much the same as a modern water-cooled PC (except much bigger obvs.) and then the air was chilled to uncomfortably low levels by a battery of (very squeaky) air-con units situated around the computer room's periphery. I presume this was viable as they likely generated much less unwanted heat than their ancestors; part of me thinks it's a slightly inelegant solution, but another part of me also unwittingly found the outlet pipe of a CPU on one occasion and it was very ouchie. Nobody had thought to warn me beforehand "oh btw there are some very hot pipes down there lol" and I hadn't realised 1st gen AS/400s were water-cooled.
According to IBM research, nanosheet transistors (the type being employed by TSMC for N2 and Intel for 20A/18A, though each have their own marketing names) which will appear on the server market in 2026 or so double their performance level if you cool them to LN2 level temperatures.
Since cooling to that level is relatively cheap it is an easy decision to make for doubling performance. I don't know how much additional input power the chip requires, but safe to say it will require higher power levels. So datacenters are going to become even more power dense by the end of the decade!
> According to IBM research, nanosheet transistors (the type being employed by TSMC for N2 and Intel for 20A/18A, though each have their own marketing names) which will appear on the server market in 2026 or so double their performance level if you cool them to LN2 level temperatures.
> Since cooling to that level is relatively cheap it is an easy decision to make for doubling performance. I don't know how much additional input power the chip requires, but safe to say it will require higher power levels. So datacenters are going to become even more power dense by the end of the decade!
I've not followed this for several years so I don't know what the current state-of-the-art is, but at CERN we've been experimenting with/using carbon-dioxide evaporative cooling for years Here's an early article which was my first DDG search result. More papers are out there if you use the right keywords.
https://iopscience.iop.org/article/10.1088/1748-0221/6/01/C01091
Castrol used the slogan "Liquid Engineering" to promote Castrol GTX.
They should revive that and use it for DS cooling as its doubly applicable to the liquid and the application.
https://www.castrol.com/en/global/corporate/products/data-centre-and-it-cooling.html
I'd think that it would be an idea to built a 1-2 RU module on a baseplate of preframulated Amerlite that included an embedded cooling labyrinth with quick connects on the back side and the rack would have a manifold with nipples to connect up to a central cooling system. Air cooling is noisy although cheaper for a small system. Not getting as much into each rack doesn't seem like a big deal if each new RU of space used is a multiple of processing over what it's replacing. Bonus if this gains back floor space for other needs since that comes at a cost as well.
Liquid cooling also can be routed for use in a building's HVAC system for heating in winter much more easily and also routed to the most economical method of rejecting the waste heat based on the exterior ambient temp at other times. If cooling is just fans pushing around cold air generated from classic aircon, there's no way to economize in the same way as noted in Aerogems story.
Yes co-generation should be an obvious answer to economic use of waste heat. If one uses the waste heat as part of a hot water heating system through a heat exchanger, it might be possible to locate into urban areas and sell the process heat. Single pass systems where the maximum extraction of useful energy is not practiced is on the way out. There will be problems as things evolve but the big ones will not be in the engineering.
The amount of power needed now surely means we must be on the edge of a change of approach. The thermal conductivity of silicon will eventually represent a hard limit, and we are already more or less at the limit of band gap.
Materials wise there are few alternatives, Gallium-Arsenide the perennial never-makes-it-because economics. Photonics? Voodoo?
"Materials wise there are few alternatives, Gallium-Arsenide the perennial never-makes-it-because economics. Photonics? Voodoo?"
Protein?
Granted, proteins are damned slow, easy to overcook at even mild temperatures, easy to denature in millins of other ways and complex to build
but one advantage of protein is that it can be grown in a vat, like chemical meats or beer.
The one great advantage to protein is that it is scalable. From insect complexity to dolphin and beyond. That networking it isn't easy is a programming issue,
or perhaps one of engineering and I'm sure that's solvable.
Protein is also proven technology. Bugs and birds have been using it for many, many weeks.
Yeah, Water Cooling for home use is nothing new.
But uh... do you just add a 1/4" pipe and quick disconnects to each RU and pump cold water from a chiller outside with a tiny pump straight on the chip like an AIO? Using water straight in the pipes would make so much sense. Instead of HVAC, you just get a large" pipe with 4C water feeding the whole rack or room pumped from outside.
I mean, Nuclear Power plants do it with 20 inch pipes, why not?
Did anybody come up with standards for water piping on this already?
"The next big breakthrough will be figure out how to make effecient thermo-electric materials and layer those into the process, so that the waste heat can be drawn off and used to offset the power bill, or potentially have components that power up as it gets hotter."
I spent a nice hour with an engineer at Teledyne that worked on RTG's and the current technologies for turning heat into electricity via thermo-electric materials has very low efficiency. Their customers are looking for sealed-box systems that will work for decades ala Voyager or something that will operate for years at a time in a meteorological buoy in the arctic that can't be visited very often. The use of Stirling engines is something they were working on to increase efficiency yet still be able to deliver long enough lifetimes. Obviously, a mechanical system will wear out sooner than something that's solid state, but how long does that rover really need to perform?
At one point, I was trying to figure out how to take the heat from my ICE car's exhaust to generate electricity that would offset the load on the alternator. Some quick 1st order analysis was showing me the cost and complexity was far too high to be worthwhile. I'd to better installing a roof rack and a solar PV panel.
"Given all the unwanted heat produced, maybe we should be building datacenters next to buildings like hospitals, so for at least part of the year it could be used."
Big plus one. Outside of a city, putting greenhouses between facilities would also be a good idea so fruits and veg can be grown all year for more local production. Even with all of the jam I made from the last season, it's not going to last until I get strawberries from the garden next. As it's a miserable rainy day where I am, there will be a proper time out for tea. There goes another half jar of preserves.
What is all this power being used for? Is it running "AI assistants" which are even less helpful than the script-driven call centres? Is it parsing the text (JSON) which we send between all our microservices? Is it analysing our internet history to sell us more crap?
Happy New Year.
