An NPU is a fixed-function matrix engine for small, quantized models, and its TOPS rating is a peak INT8 figure that does not predict how fast a device writes text.⁠1 The claim that an NPU speeds up on-device text generation sits at rung 2, a lab demonstration, because three research groups that measured mobile NPUs disagree on which phase of generation it accelerates.⁠2⁠3⁠4 An independent test of a 40 TOPS NPU measured 6.9 tokens per second on a 1.5-billion-parameter model, and its authors attribute the ceiling mainly to memory bandwidth.⁠5

The status board places each claim on the maturity ladder; for devices, rung 5 means a product on sale whose claims an independent lab has tested. No claim below reaches it in the sources I opened.

NameRung (1-5)Best evidenceAs of
Copilot+ NPU features (40+ TOPS floor)4Microsoft says the features are designed for the NPU and improve battery life;⁠6 no independent power test foundread 2026-11-08
Hailo-10H NPU, 40 TOPS rated3Independent preprint measured 6.9 tok/s at 1.87 W, against Hailo’s own 6.82 tok/s;⁠5 one unit, one model2026-06-07
Hexagon NPU speeding up LLM phases2Three groups report three different phase splits⁠2⁠3⁠42026-07-06
Intel Core Ultra NPU, Llama 2 7B at 18.55 tok/s4Intel’s own MLPerf Client v0.6 run, measured April 2025⁠72025-05-05
Snapdragon 8 Elite Gen 6 NPU, “50% higher prefill”2Qualcomm’s figure for INT4 models, relayed by Android Authority;⁠8 no test of a retail phone found2026-09-10
Apple A20 Pro, “twice the AI processing”4Neural Engine doubled to 32 cores, memory bus width undisclosed;⁠9 Apple’s claim against the A19 Pro, no independent test found, U.S. sales from 18 September⁠102026-09-09

Seven sources are older than six months and flagged where they appear: Qualcomm’s TOPS guide (reprinted June 2024), Intel’s MLPerf post (May 2025), the SOSP paper (October 2025), Microsoft’s developer guide (page dated November 2025), the Edge AI and Vision summary (January 2026), UL’s Procyon notice (January 2026) and the Leiden thesis (April 2026).

An NPU is a fixed-shape matrix engine rated at its peak INT8 rate

An NPU is built around arrays of multiply-accumulate units. The SOSP 2025 paper describes the Snapdragon 8 Gen 3’s NPU as having multiple 32-by-32 systolic arrays that hold weights still while activations stream through, which saves memory traffic on regular matrix math.⁠2 The price is rigidity. A tensor dimension under 32 costs the same as a full tile, reordering the same multiplication changed speed sixfold in one test, and the NPU runs static graphs, so engines pad a 300-token prompt to 512.⁠2 Microsoft adds that many NPUs handle only low-bit integer math, so models must be quantized first.⁠6

Reported Qualcomm’s own guide defines TOPS as two times the number of MAC units times the clock frequency, divided by one trillion, quoted at peak frequency and, as the industry standard, at INT8 precision.⁠1 The same guide says a high TOPS number alone does not guarantee good AI performance and lists memory bandwidth and software optimization among the other factors.⁠1 Microsoft uses the figure as a gate: many Copilot+ features need an NPU rated at 40 or more TOPS.⁠6

Opinion A TOPS figure that names no precision and no workload describes a ceiling.

Text generation waits on memory bandwidth more than on compute

Generating text has two phases. Prefill reads the whole prompt in one batch and is compute-bound, while decode emits one token at a time and is memory-bound.⁠2 Reported The Edge AI and Vision Alliance’s summary of Chandra and Krishnamoorthi’s January 2026 survey puts mobile memory bandwidth at 50 to 90 GB/s, against 2 to 3 TB/s for data-center GPUs, and says moving weights from 16-bit to 4-bit cuts memory traffic per token fourfold.⁠11 It also says available RAM is often under 4 GB after operating-system overhead.⁠11

Measured The SOSP team measured a theoretical 68 GB/s of memory bandwidth on the Snapdragon 8 Gen 3, with any single processor reaching only 40 to 45 GB/s during decode and GPU and NPU together about 60 GB/s.⁠2 Their engine decoded Llama-8B at up to 14 tokens per second.⁠2 The independent preprint on the 40 TOPS Hailo-10H attributes its 6.9 tokens per second mainly to on-module memory bandwidth, while noting that the deployment setup also contributes.⁠5

Peak compute helps prefill, but memory bandwidth sets the pace of every token after the first.⁠2

Three studies give three answers on where the NPU wins

SOSP 2025: the NPU wins prefill, the GPU wins decode

Measured The Shanghai Jiao Tong team’s SOSP 2025 paper, tested on a Snapdragon 8 Gen 3 and 8 Elite, found the NPU dominant in prefill and the GPU faster in decode.⁠2 Running both together made its engine 1.34 to 6.02 times faster end to end than GPU-only and NPU-only engines.⁠2 In a prefill ablation, a naive NPU port of Llama-8B ran 62.5% slower than the GPU baseline because the NPU’s graphs had to be rebuilt for each prompt length.⁠2 The paper is peer-reviewed and older than six months.

July 2026 preprint: the NPU wins prefill, the CPU wins decode

Measured A July 2026 preprint by Cai and colleagues, covering five frameworks and three backends, reports that NPUs excel at compute-bound prefill while CPUs beat every other backend in memory-bound decoding.⁠3 It finds framework choice produced NPU performance gaps of up to 10 times, and scheduling flaws that waste up to 40% of energy.⁠3 The authors estimate a tuned configuration could cut NPU-backend energy by up to 54.8%, which is a projection, not a measurement.⁠3

April 2026 thesis: the CPU wins prefill, the NPU wins decode

Measured A Leiden University master’s thesis dated April 2026 used llama.cpp’s Hexagon backend on a Snapdragon 8 Gen 3 and found the reverse. On the main matrix operator, the NPU ran 1.27 to 1.62 times slower than the CPU in prefill and 1.5 to 1.7 times faster in decode.⁠4 End to end, the decode gain shrank to 1.05 to 1.20 times, because lightweight operators took 8 to 22 times longer to dispatch than to run.⁠4 In a battery-statistics test, offloading layers to the NPU raised drain by 17% to 52%.⁠4

Two of the three put the NPU ahead in prefill, so the thesis is the outlier. Its author says the Hexagon backend’s operators were written from scratch and attention still falls back to the CPU, and its energy figure comes from Android battery statistics, not chip-level power measurement.⁠4 I weight the peer-reviewed SOSP paper and the five-framework preprint more on what the silicon can do, and the thesis more on how immature the open-source path is. The three studies use different software stacks, so none replicates another, which is why the claim holds at rung 2.

NPUs earn their place on small, steady jobs at low power

Reported Microsoft’s training material says Live Captions, Windows Studio Effects and Click to Do run on a Copilot+ PC’s NPU, and credits the NPU’s efficiency for letting them run continuously without draining the battery.⁠12 The developer guide says Windows assigns work across the NPU, CPU and GPU, and that Windows ML falls back to the GPU or CPU when the NPU path is unavailable.⁠6 Both are Microsoft’s statements, and I found no independent power measurement of these features.

Measured The one independent test of a dedicated NPU running a language model that I found put the Hailo-10H at 6.9 tokens per second and 1.87 W of whole-system power, against 131.7 tokens per second and 34.1 W for a laptop RTX 4050 on battery.⁠5 The authors report near-zero variance across 20 iterations and no throttling, and say a 500-token reply would take about 72 seconds, which suits background work and not live chat.⁠5 They caution that their energy-per-token figures use different measurement scopes and are not directly comparable.⁠5

Reported Intel’s own MLPerf Client v0.6 run, measured in April 2025 on a Core Ultra 9 288V laptop with Llama 2 7B, reported 18.55 tokens per second and a 1.09-second first token on the NPU.⁠7 That is Intel’s measurement on Intel’s configuration, and it is older than six months.

Opinion On current evidence, an NPU’s job is small models at low power.

Launch claims omit baselines and sustained-load behavior

Reported Qualcomm says the Snapdragon 8 Elite Gen 6 NPU delivers 50% higher prefill performance, faster decoding, and support for mixture-of-experts models up to 30 billion parameters.⁠8 Android Authority notes the 50% applies to INT4 models and is not a 50% gain in AI performance overall, and that Qualcomm did not disclose the bandwidth of the new shared memory.⁠8

Reported Apple’s newsroom comparisons, as relayed by iGeeksBlog, give the A20 Pro twice the AI processing of the A19 Pro, tied to a dual 16-core Neural Engine, plus 50% more memory bandwidth.⁠10 Notebookcheck notes Apple has not disclosed the memory bus width.⁠9 Neither report gives a TOPS figure, and iGeeksBlog stresses these are Apple’s comparisons, not independent tests.⁠10

Measured Phones add a thermal limit that peak ratings ignore. In the independent preprint, an iPhone 16 Pro lost 41.5% of its peak throughput over 20 back-to-back runs and a Galaxy S24 Ultra about 15%.⁠5 Both ran on the GPU, because MLX does not target the Neural Engine and the Android build used the Adreno GPU, so the result shows the heat limit on sustained inference, not NPU behavior.⁠5

ClaimWhat it measuresHow to check it
“45 TOPS”Peak INT8 throughput at top clockAsk for tokens per second and watts on your model⁠1⁠5
“Copilot+ ready”The 40+ TOPS NPU requirement for Copilot+ featuresCheck which features you use run on the NPU⁠6⁠12
“50% faster”A gain over the maker’s last chip, at one precisionFind the baseline, precision and workload⁠8
“Runs 30B models”A capability claim; RAM and bandwidth set the speedLook for independent tokens per second and RAM figures⁠11
“Twice the AI processing”A ratio against the previous chipWait for a lab to rerun a stated workload⁠10

What would move it up a rung

The claim that an NPU speeds up on-device text generation reaches rung 3 when two independent groups run the same model, quantization and framework on retail phones and publish prefill speed, decode speed and joules per token for the CPU, GPU and NPU. The three studies above use different software stacks, so none replicates another.⁠2⁠3⁠4

Qualcomm’s Gen 6 claim and Apple’s A20 Pro claim move up when a lab reruns a stated workload on retail units against the previous generation and publishes the result beside the maker’s figure.⁠8⁠10 The first candidates are reviews of phones carrying the Snapdragon 8 Elite Gen 6 chips and of the iPhone 18 Pro.

Standard benchmarks are the place to watch. UL’s Procyon AI Text Generation benchmark added NPU support across major Windows vendors in January 2026,⁠13 and MLPerf Client v0.6 included NPU tests in 2025.⁠7 Neither the Intel post nor UL’s notice publishes a power figure, so joules per token is the number still missing.