Intel, Ampere show running LLMs on CPUs isn't as crazy as it sounds

Popular generative AI chatbots and services like ChatGPT or Gemini mostly run on GPUs or other dedicated accelerators, but as smaller models are more widely deployed in the enterprise, CPU-makers Intel and Ampere are suggesting their wares can do the job too – and their arguments aren't entirely without merit. To be clear, …

[Tom Chiverton 1]

"optimizations to Llama.cpp "

They'll be contributing those back, right?

[Vikingforties]

Yep, https://github.com/AmpereComputingAI/llama.cpp

[Rich 2]

Missing the point - AGAIN

“running LLMs on CPUs isn't as crazy as it sounds”

Yes it is!!! It’s might be possible but it’s a complete waste of electricity.

OK - a quick thought experiment… I have a PC and I run an LLM on it. And it works. Now what? What fucking use is it?

[Lon24]

Re: Missing the point - AGAIN

Llama, on my desktop, can produce better bash scripts then I can. OK, that's a low bar. They even (gasp) include useful documented comments so hacking them into something useful. gets me a better result faster. Maybe I should have memorised the Linux toolbox but I don't want to be mistaken for a bot ;-)

[LybsterRoy]

Re: Missing the point - AGAIN

Please tell me the percentage of the population using a PC who a) are running Linux and b) have the faintest idea what a bash script is and c) give a shit anyway.

[TheWeetabix]

Re: Missing the point - AGAIN

On this site? Probably quite a few. You’ll find you’re in the minority here with no tech skills.

[Anonymous Coward]

Re: Missing the point - AGAIN

> What fucking use is it?

Well, LLMs are known for making things up, so, yes, it may be able to suggest something novel to add into your love life.

[Snowy]

When your best is a CPU.

Everything can run on a CPU.

[G.Y.]

Re: When your best is a CPU.

Everything can CRAWL on a CPU ...

(cf. running FP code without an FPU)

[Rik Myslewski]

Damn good article

Thanks for the well thought-through, effectively data-supported article.

You write, “Intel has only shared performance at batch size of one.” Let’s hope that when Xeon 6 is available, some enterprising soul will provide us with more-applicable benchmarks.