I know of only one person who wants to run an LLM
And he is an architecture astronaut that the rest of the group uses as a barometer of what not to do.
He doesn't trust ISPs or mail providers, so he hosts his own mail server, which has been down for months.
He won't use an FTP server, preferring to set up a microservice architecture, with SwarmKit to orchestrate it in a VM, of course.
And he needs an $8,000 four-processor Ryzen desktop with 128GB of ram, 16TB of SSD, and 6 NICs so that everything can be run in a dozen separate VMs.
Naturally, he's trying to run his own LLM on his desktop.
That's the target audience for desktop AI. The people whose lights dim in their house when they fire up a compiler because their computers use more power than their air conditioners.
Everyone else is using web-based ProtonMail or the like for mail, they use a shared DropBox account to share files, and they run $300-$1,000 computers (for the gamers) that get just as much done, if not more, because they're not rebuilding the internet service infrastructure on their desks.
The average user doesn't care about running Proxmox, or Kubernetes, or setting up an industrial data warehouse. And they don't care about having an LLM on their desktop, either.
Computer makers are salivating over AI processors because it means being able to sell high margin machines again, and since the market doesn't currently have AI chips, there's a potential market of billions of machines to be sold. The only thing missing is any indication that end users asking for it.
Does anyone else remember 15 years ago, when TV makers were pushing 3D television sets? You couldn't find a TV without 3D for a while. How many do you see today?
Sure, AI and LLMs are important, and people will use them. Remotely. End users love email, too, but they're not running out to install and configure SMTP servers on their desktop, they just use a remote server that provides it for them. The same will be true for LLMs.
I'm sure that AI features will creep into regular desktop chipsets; that's probably inevitable. But needing a dedicated AI chip is a solution in search of a problem.