Boffins detail new algorithms to losslessly boost AI perf by up to 2.8x We all know that AI is expensive, but a new set of algorithms developed by researchers at the Weizmann Institute of Science, Intel Labs, and d-Matrix could significantly reduce the cost of serving up your favorite large language model (LLM) with just a few lines of code. Presented at the International Conference on Machine …
I'm having trouble with this analogy "The whole concept is a bit like predictive text on a modern smartphone. As you type, it tries to guess what you're going to say next. When it's right, you can complete the sentence with a single tap; when it's wrong, you just type it out yourself."
I know what the answer should be with predictive text because of prior knowledge. How does the LLM know that the predicted tokens are right or wrong? Does it have prior knowledge?
The trick is that LLMs are a lot faster if they can evaluate multiple queries in parallel. So we use the small model to predict the next ten tokens, then on the *assumption* that those are all going to be correct we feed all ten prefixes into the big model to get ten next tokens.
Optimally, what happens is the big LLM happens to output at each step the token that the small model has guessed. In that case we just got ten tokens for the price of one. But usually, the big model calculates at least one token differently from the small one; in that case we just throw away all the tokens after it and restart from that point.
So you say "Hello " and the small model says "workdays", you complete:
- Hello
- Hello w
- Hello wo
- Hello wor
- Hello work
- Hello workd
and so on, which you can do efficiently in parallel.
The true (big) completions are: 'Hello [w]', 'Hello w[o]', 'Hello wo[r]', 'Hello wor[l]', 'Hello 'work[d]'.
Now you just go through: "w was right, cool. o was right, cool. r was right, cool. k was wrong, it was l instead, so I got 4 tokens for the price of one" and start over from "Hello worl".
> ...they could just switch the whole thing off and stop wasting energy and time.
To me (and possibly you) computers and phones are something we didn't have when we were kids. We played outside with others and had a grand old time in the sandpit.
Kids born today don't know about that and get their screen fix from lazy parents trying to keep then quiet. In twenty years, your quote will be heresy and subject to on the spot execution.
You could absolutely do that. But optimally you want to run the draft model on separate hardware (cpu/gpu) anyways so you can keep the big model maximally busy. At that point, so long as the draft model runs faster than the big model, there's no point to speeding it up further, because the big model is what limits your throughput anyway.
