The NPU: Neural processing unit or needless pricey upsell?

If you haven't heard of neural processing units (NPUs) by now, you must have missed a year's worth of AI marketing from Intel, AMD, and Qualcomm. In the past 12 months, these AI-focused processors have been touted as the next essential upgrade – one that everyone apparently needs to make the most of artificial intelligence. …

[Steve Davies 3]

You forgot the evil empire...

called Apple.

Haven't they had NPU's in some shape or form in their CPU for some years now?

All silently working in the background to steal your data and send it to the mothership (or so the detractors try to tell us)

They must be even worse than Intel or Amd because there is no choice now to have a device without an NPU...

El Reg is slipping or has the change in ownership made it a more apple friendly place. We deserve to know.

[doublelayer]

Re: You forgot the evil empire...

To the extent that Apple is stealing your data, they're not using the NPU to do it. They use the NPU for a few functions that run on the device, including speech recognition and a bunch of things the camera can do, both while taking photos and sorting them. Any data stealing is done on the classic CPU. You don't need special hardware to manage it. The degree to which your data is being stolen and by whom is left as an exercise for the reader or the reader's unjustified assumption if they have one.

[DS999]

Re: You forgot the evil empire...

Apple's NPU isn't taking up as much die space as AMD/Intel's, less than 5% of the die space of M4. It is not quite as powerful overall but nearly three times as good as AMD's in terms of performance per area (probably a result of years of refinement since they've been including them in their designs since 2017)

Apple has an advantage since they aren't having the requirements dictated to them by a third party like AMD/Intel are having with Microsoft needing an NPU to support Recall in Windows 12, nor do they have a competitor like AMD & Intel have in each other having marketing incentive for exceeding the requirements so they can claim they are "better" at AI than the other guy. I wouldn't be surprised to see the percentage of die area devoted to AI increasing in the x86 world not because it is needed but because they want to have more than the other does.

[I should coco]

Re: You forgot the evil empire...

So?

[Dave 126]

Re: You forgot the evil empire...

> You forgot the evil empire... called Apple.

He didn't forget. From the article:

"NPUs do add efficiency, particularly in mobile devices where every watt saved is valuable"

Apple's NPUs started on the iPhone to assist Apple-written software, particularly on the camera side of things. Apple's engineers evidentially determined at some point that an NPU would aid them in delivering a product that would sell (in this case, a phone that does fancy trickery to improve photographs without depleting the battery). Google then followed suit and added NPUs to its Pixel phones, which were sold on their photography chops.

The article is about CPU vendors adding an NPU of dubious utility to a laptop chip and then asking more money for it. Whilst Apple's M series chips do feature an NPU, they don't use it as the main selling point.

[nautica]

Almost correct...

From the opening paragraph--

"If you haven't heard of neural processing units (NPUs) by now, you must have missed a year's worth of AI marketing from Intel, AMD, and Qualcomm."

Should have read...

"If you haven't heard of neural processing units (NPUs) by now, you must have missed a year's worth of AI marketing hype from Intel, AMD, and Qualcomm."

[MiguelC]

Re: Almost correct...

TBH, whenever marketing is involved, hype is implied

[Anonymous Coward]

Re: Almost correct...

Opening paragraph?

Not really ...

It's actually in the title, like writing on the wall.

The NPU: Neural processing unit or needless pricey upsell?

.

[Anonymous Coward]

May I have some Optane with that?

I heard that this 3D memory will fill the "cache-RAM gap", whatever that means, it's gotta be awesome!

/s wouldn't it be delightful if we learned Intel was somehow using all that written down 3D X-Point silicon to build their their NPUs? I mean, memory is massively parallel, too!

[Richard 12]

What does an NPU actually do?

I mean the actual functionality, not the hype.

GPUs run a "map" function on a huge array of data at selectable floating point resolution. The "reduce" performance tends to be pretty awful and better done by CPU. So I can use them for tasks requiring a 'map' as long as I can readily get the data in and the results back out.

What does an NPU do?

Can I write an NPU 'shader'? If so, what is the hardware optimised for?

[steelpillow]

Re: What does an NPU actually do?

Was wondering the same thing. Some kind of shonky hardware almost-a-neural net or something?

[JoeCool]

Re: What does an NPU actually do?

My understanding is that NPUs run low-precision SIMD ops ( is that a Map ? )

But the issue is that there is no standard for the hardware, instruction set, or app type and interface.

IE it seems that to be practical, the current NPU implementation has to support AI apps individually ( per chip vendor, per language, per OS )

I am willing to bet that there is one AI app that is supported "off the shelf" and that app comes from M$

[Spazturtle]

Re: What does an NPU actually do?

There is DirectML which you can use as a vendor agnostic API, it added support for NPUs about 2 months ago.

[JoeCool]

Re: What does an NPU actually do?

Thanks for mentioning DirectML.

Their initial announcement is one fo the things that showed how specific the support was:

"developer preview support for NPU acceleration in DirectML ... enables support for a subset of models on new Windows 11 devices with Intel Core Ultra processors with Intel AI boost."

I'm not clear how much broader support for that API is. The only other announcement I saw claimed support for Copilot on SnapdragonX on Win11.

[Spazturtle]

Re: What does an NPU actually do?

It's the same as Direct3D where each GPU needs a Direct3D driver, with DirectML each NPU will need a DirectML driver.

[Sorry that handle is already taken.]

Re: What does an NPU actually do?

I looked up Neural Processing Unit on Wikipedia and the article it redirects to is currently so vague as to be essentially useless. Someone's actually left an editing note in the text asking someone else to flesh it out...

Hey Register: would you mind adding some technical detail about what an NPU is and what it does (and how and why it's superior to CPUs and GPUs for the same tasks) to this article?

[sabroni]

Re: Hey Register: would you mind adding some technical detail

It's on Wikipedia. Do it yourself.

[Anonymous Coward]

Re: Hey Register: would you mind adding some technical detail

That only works if we know what it does.

[Androgynous Cow Herd]

Re: Hey Register: would you mind adding some technical detail

C'mon -It's 2024, almost 2025. Do the modern thing and use AI to make some shit up!

[Spazturtle]

Re: What does an NPU actually do?

An NPU is basically a simple compute only GPU that is optimised for INT8 and INT16 performance compared to normal GPU which is optimised for FP32.

[munnoch]

Re: What does an NPU actually do?

Yes, its a castrated GPU and GPU's are castrated FPU's with a bit of vectorisation thrown in.

Airthmetic is quite heavy on clock cycles for general purpose CPU's hence the desire to offload it to dedicated silicon. All we are seeing is a range of designs optimised for different levels of precision as there is a direct (inverse) correlation between precision and throughput.

[Benegesserict Cumbersomberbatch]

Re: What does an NPU actually do?

Can't use it to render or ray trace spheroids, then.

[Anonymous Coward]

Re: What does an NPU actually do?

NPUs are mostly lower-precision matrix-multiplication accelerators since that's most of what a neural network is.

You can use a GPU to do matrix multiplication but you have to do all the steps explicitly, which is inefficient. And they're not always optimized for lower-precision operations.

NPUs also usually have some acceleration for the other operations that are necessary to evaluate neural networks, e.g., various activation functions, output normalization layers, pooling layers, etc.

Anybody saying that an NPU does roughly the same thing as a CPU or a GPU (or vice versa) has catastrophically misunderstood the situation.

[Anonymous Coward]

Re: has catastrophically misunderstood the situation.

Every accusation....

This is a shit GPU with some helper functions. Wow, I've catastropically misunderstood that and my brain has ruptured.

[Mentat74]

All I see...

Is another possible attack-vector for malware, spyware and ransomware...

What memory does this NPU have access to ?

Are there any protections built-in to prevent access to certain memory regions ?

I think it's just a matter of time before someone finds a way to get these things to do something they shouldn't.

[Mot524]

Re: All I see...

NPUs have specific functionality, i.e., doing neural network stuff. Not malware.

What you're suggesting is akin to saying that somebody is going to figure out how to take advantage of a chip's video encoder to run malware.

[chuckufarley]

I think a lot of AI implementations...

...Will wind up being built into Cloud Computing. If you let the users run models on their own hardware you give them too much control and can't charge them as much for it.

[ShameElevator]

Re: I think a lot of AI implementations...

Apple (and maybe the others) try to let things get handled on device and then send any thing my complex to the cloud. Apple have their AI cloud that is supposed to be private for these things. At what point things get send to the cloud I don’t know.

[Anonymous Coward]

Multi tasking?

How do NPUs handle multi tasking? In fact can / do they at all? Does an executable "claim" it and prevent others from using it? Does the NPU need to run a complete iteration of the neural net, then hand over to other processes? I really want to know how this stuff works.

Maybe I should ask ChatGPT or Gemini, but they'll tell me something that they're 100% sure of, but could be a complete hallucination.

[chuckufarley]

Re: Multi tasking?

My GPU can multitask very well indeed. If you believe the hype around AI and GenAI then you have to believe that any hardware that claims to be an AI accelerator will be good at multitasking because AI will be used for everything if the hype is true. On the other hand, how much L2/L3 cache do the NPUs need vs how much do they get?

[gnasher729]

Re: Multi tasking?

From what I've seen, cache bandwidth or access time is quite irrelevant. What counts is pure memory bandwidth, Which can be 100 to 280 GB/sec on a not-top-of-the-range Mac.

[Tron]

It's a scam to sell more hardware.

Because W11's fake 'minimum requirements' didn't con enough of us.

Same reason why they block older versions, remove legacy compatibility and demand subs rather than standalone.

The TOPS rating is like those vehicle mpg, emissions and EV range ratings that aren't real world. It's not vendor and customer, it's predator and prey or farmer and sheep. They are no better than politicians, which is the worst thing you can say about anyone.

NPU = FAU. **** All Use.

[chuckufarley]

Re: It's a scam to sell more hardware.

It can't be a scam to sell more hardware. Otherwise we would all have Itanium desktops, No?

[Uncle Slacky]

Poorly written set of poll options

Shirley the first and third choices in your poll are just saying the same thing in different ways?

[Andy The Hat]

Re: Poorly written set of poll options

They are but don't ... already got my coat ...

[doublelayer]

Re: Poorly written set of poll options

I agree. My view would go along the lines of "You probably won't miss an NPU if you don't have one, but there are some possible benefits, and if enough people have one then some software will use it and it will become standard like all the other things that every desktop has now but was an optional extra before". It's not the most concise option, but none of the provided options comes close.

[DJO]

Re: Poorly written set of poll options

If games like Civilization could use this to improve the quality of the "AI" players in their games it be worth having, but as ever, I suspect the first few generations of NPUs are for most use cases going to be useless crap, we won't see any meaningful use in the gaming sector for a few years.

[Anonymous Coward]

Re: Poorly written set of poll options

Last time I played Civ, it was possible to beat a tank using guys armed with clubs. I think lack of AI isn't the main issue.

[JacobZ]

Under a rock

Apparently I've been living under a rock for a year because I have never heard of these things. And I thought I was reasonably current with tech.

Or perhaps I did hear about them, promptly filed them under Next Overhyped Thing, and completely ignored them since?

[chuckufarley]

Re: Under a rock

If it turns out to be Option B then it's also what you will have to pay for and never use for decades to come. When did programs stop using Intel's MMX or AMD's 3DNow?

[phuzz]

Re: Under a rock

When did programs stop using Intel's MMX or AMD's 3DNow?

In a way, they didn't. The new instructions that Intel called MMX, were added to by AMD and called 3DNow. Intel then added more instructions and called them SSE. SSE went through various versions with more instructions added, and then became AVX, and so on.

So, if you're using an x86 CPU, it almost certainly has the MMX instructions, plus a whole lot of newer ones. Whether or not they're used by software depends on the compiler used to compile each executable.

[Helcat]

Do we actually need AI on our computers? What is the real world benefit? At what cost (not just the extra cost for the CPU - what about power draw, heat, memory usage, performance, reliability etc?) Will the AI be like the LLM's (aka Kit-Bash solutions? If you're wondering about that - take a look at some of the AI generated images and texts... they're... unreal) Will they deliver what's promised? Can we opt out?

The latter, to me, is the most important point: Can I take the non-AI version, please? After all, if it's a choice of take it or leave it, then we can't do a real world comparison as to if these NPU's are delivering anything to us other than bigger bills. Got to do side-by-side comparisons so we can make informed decisions as to if NPU is useful or just that Needlessly Pricy Upsell.

[doublelayer]

If the AI is running:

Power draw: Better than if it's on a CPU and/or GPU.

Heat: See above.

Memory usage: Basically unchanged. Large models use lots of RAM and it doesn't much matter where they're processed.

Performance: By running as much of the AI on the NPU as possible, load on the CPU is reduced.

Reliability: I don't have any reason to expect the NPU to have any reliability differences compared to the other parts of the chip. If your CPU cores don't start failing randomly, the NPU probably won't either.

If the AI is not running:

Power draw: Mostly negligible. There will be some current going to the NPU, but like most other components, it can be put into a deep sleep mode when nothing is using it.

Heat: See power.

Others: Since nothing is using the NPU, your other resources won't be affected much.

Cost: You've paid for silicon you're not using.

[LVPC]

I disagree. I have zero use for LLM fake "artificial intelligence" and went out of my way to buy a not-latest generation CPU for my latest build this year.

I'd rather have more cores. And not have to worry about NPU hardware flaws open to get more exploits.

[katrinab]

Real-world use-case:

I am scanning a load of old till receipts, some of which are very faded and barely legible.

My options are to scan them into Photoshop or similar and play about with levels and exposure until I get something that is legible, or I can use the Document Scanner on my iPhone which mostly manages to get good-enough results automatically and pretty much instantly.

It is the Apple Neural Engine that is doing all the heavy lifting here. I believe a recent Pixel will achieve similar results with its Tensor Core.

Obviously the Large Language Model stuff is fundamentally flawed and will never work.

[Groo The Wanderer]

They're "new pocket unloaders" - their primary purpose is to unload cash from your pocket.

[Anonymous Coward]

NPU

Not Particularly Useful

Now Pay Us

[Anonymous Coward]

Re: NPU

Not Particularly Useful

Now Pay Us

8^D !!! ----> Have one on me. Yours is probably the best post* in this thread.

* with the exception of this one, obviously. 8^°

.

[Rich 2]

Can I switch it off?

If one of the marketing points is to save power then it seems to me that switching off anything related to “AI” (no such thing) would be a jolly good place to start

[Doctor Syntax]

"Manufacturers promote NPUs as essential for future-proofing laptops in an AI-driven world."

There's a bit of an assumption in there.

[LVPC]

It's more than "a bit. of an assumption."

who the hell wants to live in an "AI-driven world?" Seriously , who's the boss here?

[Alumoi]

Apple, Microsoft, Google....

[Anonymous Coward]

... a portly herd of rotund llamas, heftily spitting at us ...

[jcday]

Intriguing

I wonder why they're doing in hardware stuff that actually is better done in software. "Efficiency" isn't a reason, since there's better efficiency gains elsewhere.

Neural Nets are, ultimately, a very extreme example of SIMD-type work. Ideally, you want something akin to a vector processor, because all neurons are performing exactly the same instructions at the same time, just on different data sets.

This is partly why GPUs are good, because GPUs can do limited amounts of SIMD stuff.

[Bebu sa Ware]

Strikes me...

If you have a spare 15% of your silicon for this nonsense why not make that 15% 'soft' so that different and potentially more useful functionality could be dynamically assigned by replacing the microcode on demand.

A completely soft CPU might have some merit too. ;)

(A *really old* idea, though.)

[blcollier]

Re: Strikes me...

> more useful functionality could be dynamically assigned by replacing the microcode on demand

You mean like... some kind of array of gates, that can be re-programmed... in the field... Huh, what a cool idea, a Gate Array that's Programmable in the Field - maybe we could call it a GAPF.

Wait... Hang on... That already exists, doesn't it...

I am indeed gently poking fun, the fact you used the term 'soft CPU' already tells me you know what an FPGA is ;). But I do remember when FPGAs were supposed to be the Next Big Thing in server silicon, being reprogrammed on-the-fly to execute specialist tasks.

But FPGAs did give us cycle-accurate replication of classic computers and console systems, without the need for software hacks/workarounds or interpolation layers, so there's that to be grateful for :).

[doublelayer]

Re: Strikes me...

Because then we'd have an article asking whether the FPGA is a field-programmable gate array or a futile profit-gaining addition. An NPU can be used to efficiently compute some classes of software, and if you don't run those or you do but you can't or don't optimize to use it, it's useless. An FPGA on a general-purpose computer is quite similar, just with a different set of things it's useful for. Since most things where you would use an FPGA are also done in software with an efficiency cost, you have to ask how many things an average computer does where access to an FPGA would provide an efficiency boost to justify the cost of the silicon and how likely each is to be written in a way that can use the FPGA.

[An_Old_Dog]

Re: Strikes me...

You'd want a more-modern way of loading microcode into the writable control store than the old computers used. (Somewhere I have a floppy diskette with Data General-provided microcode for an Eclipse on it.)

[jglathe]

The thing I can't wrap my head around

NPUs aren't that new, but the size and TOPS they deliver now may be. But, old or new NPU, where are the documentation and libraries for it? This should be in the device tree of the SoC. Odd as it may be, I wouldn't care much for Windows, that's big corp's playground if anything. With out this, nobody will touch it unless he's paid for it.

[m4r35n357]

NPU === HH

Hallucination hardware

[Anonymous Coward]

Upsell?

Is it really an up sell though? You can get cheap SBCs with an NPU onboard...quite a few have had them for a while.

For the number of TOPS you usually get with a built in NPU, you might as well have it as not...they're pretty cheap...they're more geared to working with sensor data though than anything else...so I can see an NPU being useful for managing fan curves, adjusting your webcam, noise cancellation and other stuff like that.

I'm all for it, trouble is though we need critical mass of adoption to make developing the software worthwhile.

What we're more likely to see them used for is aimbots and ESP in games. Which is cool...but also really, really lame.

[Alan Mackenzie]

What does "upsell" (noun) mean?

Upsell (noun)? I've never encountered the word before. Is it part of some USAmerican dialect?

It made it hard for me to tick one of the three boxes on the quiz.

[Toe Knee]

Re: What does "upsell" (noun) mean?

As an American, let me enlighten you:

Imagine you are at your “favorite” fast food establishment getting a half-kilo-with-cheese burger.

The friendly employee behind the counter asks “would you like to make that a meal for 75 cents more?”

That pleasant query is the dreaded “upsell”.

[Alan Mackenzie]

Re: What does "upsell" (noun) mean?

Many thanks!

[doublelayer]

Re: What does "upsell" (noun) mean?

If you know the verb upsell, then the noun is the same. To upsell something, you try to add on extra things that cost more money or to replace the cheap thing with an expensive alternative after the customer has decided what they actually want. An upsell is the thing you add on to do that or, alternatively, the act of upselling itself. In this case, it's not really an upsell because it's just put there and you don't really get a choice about it; if you want a certain type of CPU and it only comes with an NPU, then you'll have to buy an NPU to get the rest of it. To be a more traditional upsell, you would have to persuade the customer to add it on or to choose a more expensive board by marketing the included NPU.

I've heard people in the UK use both words. I don't know if it was an American term when it started, and it could easily have come from any country to begin with, but it's pretty global now.

[katrinab]

Re: What does "upsell" (noun) mean?

It is Marketing Speak for selling someone a more expensive product than they originally asked for.

[nautica]

Re: What does "upsell" (noun) mean?

...think it's a verb; not a noun.

It only comes close to being a noun when it's a gerund, as in "upselling".

But I cud be serosly bad rong; I've been rong befoar.

[Mot524]

Look at Apple

If you're not sure what value an NPU provides, look at what Apple has been doing with theirs for years (not including all their new "Apple Intelligence" stuff):

- Image processing stack for photos and video (means they don't have to put dedicated image processing hardware on their chips BTW)

- Text/image/facial recognition for indexing photos/images

- Facial recognition for Face ID

- Speech to text, improves latency for e.g. Siri, dictation, etc.

- Presumably translation in their language translation app

- Probably improving microphone recording quality for e.g. FaceTime?

A lot of people on this thread seem to think that an NPU is only good for running LLMs but they seem to have missed most of the point.

[Mot524]

NPU efficiency

https://john-zhihong-wang.medium.com/benchmarking-apple-macbook-pro-m1-for-deep-learning-inference-101428887da7

Mileage varies depending on which network you run, but for MobileNet it looks like the M1 GPU is 50% faster than the M1 CPU, and the M1 NPU is 250% faster than the GPU (350% as fast).

Also, fun fact, the M1 NPU maxes out at about 3.6 watts whereas the M1 GPU uses 10-15 watts.

So there's a huge performance gain, and a power efficiency gain of about ~12x.

So for everybody who's wondering "why don't they just use the GPU to do this NPU stuff" you're basically wondering why anybody would do something 10+ times more efficiently.

[DJO]

Re: NPU efficiency

Yes it does stuff efficiently. But is it doing stuff the user actually want doing?

[david1024]

well

What they really need, given the lack of NPU standards... is an "AI-READY" PC... that would have a slot for a module. That way they can upsell the 5%-10% and just have a slot that will likely never get used... then they can upgrade the slot next year and orphan the 1st gen AI-Ready slots. Power users will absolutely need that most expensive NPU upgrade for their AI-READY PC for a few hundred more bucks!

Can't lose!

[An_Old_Dog]

Die Space Usage

I'd rather the ~15% die space usage consumed by an NPU used instead for cache memory. That would be directly-useful by the non-AI-enhanced, mostly-single-threaded programs I run.

[Jonjonz]

Our Tech Bro overlords have been slurping PC surveillance for years, the NPU make life easier for that. The NPU eats that data, analyzes it, and sends the digest report back to the tech bros. This way they can gather more data, but only have to transmit the filtered, categorized and summarized cream back to them.

No doubt the NPU, to function fully for its supposed consumer benefits, will also serve as a conduit front end to assist with making cloud based solutions preferable to native solutions, further bringing all PC use into the cloud where you pay rent and own no software that operates independently on the PC.

[phillr]

A RTX 4000 - class GPU has a TOPS rating somewhere in the 1,000 range. Why should I get a CPU with an added NPU that is in the range of 40-50 TOPS, when with a software shim, I should be able to use the processing horsepower I already have installed?