back to article Intel, Ampere show running LLMs on CPUs isn't as crazy as it sounds

Popular generative AI chatbots and services like ChatGPT or Gemini mostly run on GPUs or other dedicated accelerators, but as smaller models are more widely deployed in the enterprise, CPU-makers Intel and Ampere are suggesting their wares can do the job too – and their arguments aren't entirely without merit. To be clear, …

  1. Tom Chiverton 1 Silver badge

    "optimizations to Llama.cpp "

    They'll be contributing those back, right?

    1. Vikingforties

      Yep, https://github.com/AmpereComputingAI/llama.cpp

  2. Rich 2 Silver badge

    Missing the point - AGAIN

    “running LLMs on CPUs isn't as crazy as it sounds”

    Yes it is!!! It’s might be possible but it’s a complete waste of electricity.

    OK - a quick thought experiment… I have a PC and I run an LLM on it. And it works. Now what? What fucking use is it?

    1. Lon24

      Re: Missing the point - AGAIN

      Llama, on my desktop, can produce better bash scripts then I can. OK, that's a low bar. They even (gasp) include useful documented comments so hacking them into something useful. gets me a better result faster. Maybe I should have memorised the Linux toolbox but I don't want to be mistaken for a bot ;-)

      1. LybsterRoy Silver badge

        Re: Missing the point - AGAIN

        Please tell me the percentage of the population using a PC who a) are running Linux and b) have the faintest idea what a bash script is and c) give a shit anyway.

        1. TheWeetabix Bronze badge

          Re: Missing the point - AGAIN

          On this site? Probably quite a few. You’ll find you’re in the minority here with no tech skills.

    2. Anonymous Coward
      Anonymous Coward

      Re: Missing the point - AGAIN

      > What fucking use is it?

      Well, LLMs are known for making things up, so, yes, it may be able to suggest something novel to add into your love life.

  3. Snowy Silver badge
    Facepalm

    When your best is a CPU.

    Everything can run on a CPU.

    1. G.Y.

      Re: When your best is a CPU.

      Everything can CRAWL on a CPU ...

      (cf. running FP code without an FPU)

  4. Rik Myslewski

    Damn good article

    Thanks for the well thought-through, effectively data-supported article.

    You write, “Intel has only shared performance at batch size of one.” Let’s hope that when Xeon 6 is available, some enterprising soul will provide us with more-applicable benchmarks.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like