How OpenAI used a new data type to cut inference costs by 75% Whether or not OpenAI's new open weights models are any good is still up for debate, but their use of a relatively new data type called MXFP4 is arguably more important, especially if it catches on among OpenAI's rivals. The format promises massive compute savings compared to data types traditionally used by LLMs, allowing …
Next up, (de-)compressing weights on the fly to use less than a bit per weight. When that is no longer enough, then the weights must be encoded in the quantum states of the fabric of the computer chips. That will also be insufficient to satisfy the curious language models and the weights will have to be encoded in the quantum state of the universe itself.
But since we are already living in this universe, we can be assured that the universe already has the quantum state necessary to encapsulate the quantum state for the models and because we do not see the result of this quantum state we must conclude that either the whole language model stuff is folly or the universe simply prevents us from reading the proper weights. So, a dead and alive situation waiting for the wave to collapse and the model to cease to exist. Speaking statistical probabilistic, of course.
QI (Quantum Intelligence) will ensure there is no resource consumption any longer. As a matter of fact, QI will reverse any and all negative resource consumption. QI will give you all answers you always wanted but never dared to ask(*) and, at the same time, extract energy from the zero point field and convert it to usable heat that will heat our buildings and supply electricity for centuries to come. It will be a Grand New QI World were no one has to work and everybody has everything they always wanted.
(*) right until someone asks the QI how the universe actually works which causes the universe to vanish instantly and reinvent itself so no QI or other being or system ever again may come into existence to discover or answer how the universe works.
As a science graduate and computing MSc I didn’t understand any of that…. but it seems to imply that you don’t need a iPhone 16 (or greater) to run Apple Intelligence.
An iPhone 8, Raspberry Pi or Mr Fusion would do ??
Time to offload the NVidia stock.
Chinese Deep Mind will be pissing themselves.
As I understand it you take a block of 32 sixteen bit values and factor out the 8 bit exponent that keeps the maximum information when the modified values are reduced to 4 bits (here FP4 I think)
Basically (re)encoding 32×16 bits as 8 + 32×4 bits but not necessarily losslessly. 64:17 compression ?
I guess this implies that adjacent parts of a model's representation are fairly close numerically but it also strikes me that the 8 bit exponents are also proxies for the block of 32 values they each apply to.
An expensive way of reinventing jpeg etc ? ;)
Time to design my next selfOnly.Custom( T-shirt );
...Lower resolution seems to be the way of modern life (people watching movies filmed in 4K on their phones.)...IMax->SmartWatch is next?
...That day in the mid/late 1980's when I got stand in the center of a Cray X-MP/24 at Bell Labs in New Jersey, I think the machine would have sighed if I said the future is FP4.
First, is there a typo in the 3rd set, the re-scaled version? Starting from `BF16 values, 0.0625, 0.375, 0.078125, and 0.25` seems like our poor 0.25 got missed :-(
> 4-bit floating point value by the scaling factor, resulting in: 0.0625, 0.375, 0.09375, and 0.0625.
Shouldn't that last one be 0.25 as it ought to fit in a 16-bit space, just like 0.375.
Also, two questions about the scaling factor: First, how are these 32 (or 16) common-scaling blocks determined? Memory-adjacent? Some higher order data structure?
Second, in the training phase, does that factor evolve in time? i.e. Do the scaling factors appear in the very outset of a training run, as a factor of the initial data? Or does the model have to train for a while before block commonalities are apparent?
p.s.
> This is going to get really nerdy, really quickly here, so we won't judge if you want to jump straight to the why it matters section.
Nah, we love this stuff. Fascinating and from the looks of it, quite significant. Esp. from the PoV of running LLMs locally on some mid/upper range GPUs with constrained memory.
I hope you find the following article as interesting and positive as I did.
Best wishes.
I (am no expert on AI calculations but...) suspect it's more about convergence. How wide you can cut the peel before it isn't marmalade but jellied oranges you are making. I wouldn't be surprised if the next thing is variable depth FP, use the more expensive depth later in the calculations.
Not at all if you keep both things. Keep the full-res image somewhere, but only send the downscaled one. When a user zooms in, crop the relevant portion, downscale it less, and send that. Result: a picture that looks like you can zoom with a lot more detail than you'd expect but you don't have the bandwidth costs of sending the whole thing down the pipe.
Whether this works for inference is another question. If they train a model on FP16 and quantize it for use, then they still have the unquantized version around somewhere. I don't know whether there's any real ability to or benefit in cutting over from one to the other mid-inference.
No, I speak as if FP16 is the original precision, because you're right that you can't go any higher than that. Whatever the original precision they used before quantization, that's as high as you can go, just as my zoom with resolution method won't let you get any more resolution than the camera originally gave you. With the camera, you can degrade it and then recover information later by keeping those copies in parallel. If you can do that usefully with an LLM, then the high precision version will be whatever they had the first time.
Im no AI guy but.... does this mean its the same quality answer but quicker, or does this mean shittier answer quicker?
Playing with local models suggests that the lower the quantization the smaller the model the quicker the response the crappier the answer.
For some things like categorizing, smaller models have been fine, but for some activities which require better more statistically accurate responses bigger models always win.
Is this not just a reaction to ChatGPT being way to expensive to run, and they want to down size the models so that companies can run ChatGPT on their own resources? (all for a monthly per user fee of coarse). This would make sense since they made $3.7 Billion in revenue and a made a healthy loss of $5 Billion. If they could try and move the compute to the big users and just charge a monthly fee for the privilege. Making a $5 Billion loss does not scale for very long. Its not like more users = less compute. I do not understand the business case?
EDIT ... Reading the expectations...The New York Times reports that OpenAI projects it'll make $11.6 billion in 2025, and assuming that OpenAI burns at the same rate it did in 2024 — spending $2.25 to make $1 — OpenAI is on course to burn over $26 billion in 2025 for a loss of $14.4 billion.
The problem with answering that is that they have very little information about the quality of any answer and don't care because, if they did care, they would still be trying to improve their models rather than selling them. Quantization on its own means the quality will degrade, but a newer model, trained on better data, could still produce better output quantized than an older model did unquantized. This version of quantization will produce better results than normal FP4 does at the cost of the model being bigger; it's almost as if bits aren't infinitely compressible.
Also, you can't predict losses the way you did because burn rate is not directly proportional to revenue. Although it's expensive to execute their models, most of their expense is in the training process somewhere, so if more people pay them to use the models, their burn ratio will decline.
