GPUs could explode to multiple TB with new storage-inspired memory tech

Virtually every high-end GPU and AI accelerator relies on high bandwidth memory (HBM), which can shuffle data around at multiple terabytes a second but can only reach into the gigabytes, with models often needing to be shared across multiple processors. However, an emerging storage technology could change that, boosting …

[very angry man]

why not plug in

why not put a socket of five on the GPU so you can plug in more , like ram or type c, then when it wares out you can replace it or upgrade it

[BoHu]

Don't believe the HBF

This sounds great for capacity, and not too bad on bandwidth, but seems to end-up pretty much a single-use tech, valuable only for the decode phase of LLM processing as performed by GPUs. I'm not sure that dataflow engines (also targeted at decode) would benefit from it. And the limitations on "write endurance before it wears out" really constrain the broader utility of this concept imho. So I think this HBF will go nowhere at all, neither fast, nor slow.

Spin-Transfer Torque Magnetoresistive Random Access Memory (STT-MRAM) though (plus Ferroelectric RAM (FeRAM), shiitake mushroom memristors, and whathaveyous), that could have a more broadly successful future iiuc [ https://www.theregister.com/offbeat/2025/10/30/theres-mushroom-for-improvement-in-fungal-computing/1278270 ].

And really, with AI (so-called) capabilities growing only logarithmically with every exponential increase in the number of parameters its 'models' use, something's gotta give eventually, and hopefully that won't be our energy grid, water supply, or RAM and SSDs -- from the perspective of reason, it should be their unsustainable girth that gets to be subjected to a strict mediterranean diet-like regimen of no insane-nonsense slimming down, not the stuff that right upholds our pursuit of happiness and even our very lives ...

[steelpillow]

Capacity-lifetime anybody?

The write cycle limitations of NAND Flash eat into its effective capacity, as data gets stashed hither and thither to spin out each cell's - and hence the chip's - lifetime. Capacity utilisation - net capacity if you like - can be unnervingly low, if you want a decent life out of it. HBF is going to have to reflect that compromise, and that will eat away at its advantages over HBM. What we really need is a standardised data capacity-lifetime metric, with the balance on your HBF chips configurable in userland.

[Bebu sa Ware]

In a nutshell…

good old fashioned interleaving which multiplies your bandwidth (bytes/sec) by the degree of interleaving but your latency (sec) remains roughly the same.

The usual hack for latency is to interpose one or more levels of faster cache between the storage and system which introduces its own problems if the storage is shared and modifiable.

Old problems in new guises.

I seem to recall a lot the same ground was covered in Curt Schimmel's UNIX Systems for Modern Architectures (1994) which I ought to fish out to re·read.

[Bartholomew]

Do to DRAM what they did to FLASH

DRAM in some ways is analogous to SLC (but without the write cycle limit, nor data saved on power off and stores enough electrons in each individual cell to not need data to be recreated using Low-Density Parity-Checks on every single read).

So why not store two bits per DRAM cell like MLC. Or three bits per cell like TLC. Or 4 bits per cell like QLC. Or 5 bits a cell like PLC.

It would mean that every DRAM refresh would be far more complex. Every single read would be regenerating valid data from corrupted data in the cells. And every write would require reads to regenerate the new parity.

The access speed would probably sit between current DRAM and current FLASH (because slow flash erase cycles would not be needed). There is no write cycle limit to DRAM cells, If each cell stored enough electrons for 5 bits that would be 32x the amount of memory (well a good bit less than that allowing for parity, but the idea is reasonable).