At last, a good reason to buy an AI PC: Reining in runaway token bills

Corporate PC buyers haven’t rushed to buy AI PCs but analyst firm Gartner thinks the machines can now do an important new job: Running AI workloads on the desktop to provide a hedge against tearaway token bills. The firm on Monday published a Strategic Roadmap for Agentic AI PCs in which Research Vice President Steve Kleynhans …

[that one in the corner]

Road to tech success

You hand your task over to the big LLM; CFO sees token cost, finally realises it isn't cost effective.

Trim task down, hand it to self-hosted cloud LLM; CFO sees electric bill & rack capex, finally realises it isn't cost effective.

Trim task down, run it on PC-hosted NPUs; CFO sees forced laptop refresh cycle in the RAMpocalypse, realises it isn't cost effective.

Sysop & sane half of dev team wave 20 line shell script in the air, wondering what they have to do to actually get CFO to listen, for once, please.

[Dan 55]

Re: Road to tech success

Have you got a charge code for that 20-line script?

[Anonymous Coward]

What next?

Force AI into your toaster? Just ask Dave Lister how that turns out.

[Scotthva5]

How about a muffin?

[Ken Shabby]

Aah, so you're a waffle man!

[werdsmith]

Have Gartner only just worked this out?

Meanwhile, businesses have been running local AI models for a long time.

[Guy de Loimbard]

Gartner always keen to remain relevant!

[Ian Johnston]

Gartner: Pope shits in woods, bears catholic.

[doublelayer]

That depends what kind of model you're talking about. In my experience, many businesses aren't trying to run local LLMs. Most of the better machines used by developers easily can run a small one, or they could run big open weights models on local servers, but I still see people preferring to use the biggest proprietary models. Maybe because those models are better, but they wouldn't know that unless they had tried the alternative and they haven't bothered yet.

The problem with comparing the biggest models and smaller ones at many scales is that they have to define what success is. The tiniest quantized LLM that can run somewhat quickly on a CPU in 2 GB of RAM will still pretend to fulfill all the requests it's given. It will just fail on most of them. Bigger models fail less often, but if they start creating benchmarks and comparing them, they might discover that the fantastic models they're using now are failing more tasks than they assumed.

[Pascal Monett]

Gartner thinks

Well that's a first.

[phuzz]

Re: Gartner thinks

In a rare case of Gartner not talking nonsense:

By 2030, 70 percent of the corporate PC installed base will be capable of running some local GenAI workloads.

might even come true. As long as PC manufacturers haven't got bored of jumping on the AI bandwagon by then, and we define "capable" carefully.

Whether anyone will actually be trying to run AI models locally in 2030 is a completely different question.

[martinusher]

Probably don't need anything that special

A higher performance PC, the sort you'd use for gaming, can run usable LLMs. They won't be state of the art, of course, and we're not talking about training but for the vast majority of everyday tasks they are likely to prove adequate.

With all this software and network bloat we tend to forget just how powerful a typical PC is, even a 'last year's model'. Given the cost of buying (thank you, memory) new systems there's likely a rather healthy market in using applications that make optimal use of existing processing power. Its not just Linux, Open Source etc. either -- there's just so much crud in modern software that the actual useful work done is for many people a small fraction of the available computing power. Its one thing to trade convenience for speed but you can overdo it (and vendors aren't exactly falling over themselves to suggest alternatives to "just buy more of the same").

[DS999]

You don't need an "AI PC" for that

If you have a halfway decent GPU you already have more token throughput than the NPU in those AI PCs. Something that was midrange five years ago would be more than enough.

[doublelayer]

Re: You don't need an "AI PC" for that

RAM is a big problem with that, because even the top-end cards for personal machines didn't tend to put very much in, which limits how large a model you can fit in them. Admittedly, that used to be a much bigger difference than it is now. A CPU/NPU attached to replaceable RAM could have it upgraded for much more capacity than replacing a GPU, but now, that RAM will be very expensive anyway. Still, it's more likely for someone to have a machine with 16 GB of internal RAM than a GPU with 16 GB of VRAM, meaning they can run a larger model slowly on the CPU or NPU if they've got one rather than having to go with something small enough to run quickly in their graphics memory.

[Groo The Wanderer - A Canuck]

Re: You don't need an "AI PC" for that

llama.cpp is your friend if you lack VRAM.

[DrewPH]

Gartner thinks...

As soon as I saw those words I stopped reading.

[Anonymous Coward]

NPU was all hypium

How many NPUs are totally unused because of lack of software? The most effective local LLM models run on way higher specs than just a minimum NPU, more like dual 5090 cards or a Nvidia Blackwell attached to its own system on a chip. This is one of the biggest frauds in PC history, everyone just moved on to Strix Halo

[Groo The Wanderer - A Canuck]

Rather, I look forward to the day when the industry stops trying to bolt "AI" on everything whether it makes sense to have it or not, and gets back to work solving business and industry issues instead of lining the already-far-too-deep pockets of Altman, Musk, et. al.

In the meantime, I'm expecting the announcement that Microsquishy has released "Copilot Calculator" any day now...