AI Wiki

Neural Processing Unit

AI HardwareAI InferenceEdge computing
14 min read
Updated
Suggest editHistoryTalk31 citationsFact-checked
RawGraph

A neural processing unit (NPU) is a processor, or a block inside a larger chip, designed specifically to run neural network inference at high throughput and low power. Where a CPU executes a few complex instructions at a time, an NPU packs large arrays of simple multiply-accumulate units that work through the matrix arithmetic of a trained model, usually at reduced numeric precision such as INT8.[28] Flagship smartphone systems on a chip began adding them in 2017, and laptop processors from Intel, AMD, and Qualcomm followed from 2023.

The term covers a family of vendor-branded designs rather than a single architecture. Apple's Neural Engine, Qualcomm's Hexagon NPU, Google's mobile TPU inside its Tensor phone chips, Huawei's Kirin NPUs, Intel's AI Boost, and AMD's XDNA all do broadly the same job. Vendors advertise them in TOPS (trillions of operations per second), a figure that became a purchasing threshold in May 2024 when Microsoft set 40+ TOPS as the entry bar for its Copilot+ PC program.[17]

The case for doing inference on the device rather than in the cloud rests on privacy, latency, offline availability, and battery life. Google, describing Gemini Nano running on the Pixel 8 Pro, put it in exactly those terms: on-device processing helps "prevent sensitive data from leaving the phone" and lets features work "without a network connection."[25] Because an NPU handles this work at a fraction of the energy a CPU or GPU would burn, always-on features such as live captioning, camera processing, and wake-word detection become practical on battery power.

Origins

Qualcomm appears to have coined the term in its modern sense. In October 2013 the company introduced its Zeroth program for "brain-inspired computing" and wrote that it wanted "to create, define and standardize this new processing architecture," which it named the "Neural Processing Unit (NPU)."[1] Zeroth itself never shipped as a discrete chip, but the name stuck.

The first neural accelerator deployed at mass scale in the modern era ran in servers, not phones. Google revealed in May 2016 that its Tensor Processing Unit, "a custom ASIC we built specifically for machine learning," had already been running inside its data centers for more than a year, powering search ranking and AlphaGo.[2] The TPU established the template of a dedicated ASIC trading numeric precision for operations per watt, but it was a datacenter part; the NPU label came to mean the same idea scaled down into a consumer SoC.

That scaling down happened, three times over, in the autumn of 2017:

  • On September 2, 2017, at IFA in Berlin, Huawei unveiled the Kirin 970, which it called "Huawei's first mobile AI computing platform featuring a dedicated Neural Processing Unit (NPU)." Built on a 10nm process with 5.5 billion transistors, the chip's heterogeneous architecture delivered what Huawei claimed was "up to 25x the performance with 50x greater efficiency" for AI tasks compared with running them on the CPU; in an image-recognition benchmark it processed 2,000 images per minute.[4]
  • On September 12, 2017, Apple announced the A11 Bionic in the iPhone X. Its "neural engine" used a dual-core design performing up to 600 billion operations per second and powered Face ID and Animoji.[3]
  • On October 17, 2017, Google described Pixel Visual Core, "Google's first custom-designed co-processor for consumer products," which shipped inside the Pixel 2. Aimed at HDR+ photography and machine learning, it ran HDR+ five times faster than the application processor at less than a tenth of the energy.[5]

Within two years an NPU of some kind was standard equipment in flagship phones, and the marketing race over TOPS figures had begun.

How an NPU differs from a CPU, GPU, and TPU

All four processor types can execute neural network math; they differ in what they give up to do it well. A CPU is fully general and correspondingly slow at the dense matrix multiplication that dominates deep learning. A GPU, built for graphics but repurposed for parallel arithmetic, remains the workhorse for training large models on NVIDIA and other hardware, and a laptop's integrated GPU often outruns its NPU in raw throughput. The NPU's advantage is efficiency at a narrow job: sustained, low-power inference on quantized models, leaving the CPU and GPU free for other work. Microsoft's developer documentation describes Windows 11 assigning work across all three, with the NPU "using energy on AI tasks more efficiently than a CPU or GPU resulting in longer device battery life."[28]

The distinction from a TPU is mostly one of deployment. Google's TPUs are datacenter accelerators, run in its own facilities and sold through Google Cloud for training and serving models.[2][31] An NPU is an on-device block, sized for one user's workload and thermally constrained by a phone or laptop chassis. Google blurs its own terminology: the on-device machine learning engine inside its Tensor phone chips is also called a TPU, but functionally it is a smartphone NPU.[8]

Most NPU designs rely on quantization. Models are trained in 16-bit or 32-bit floating point, then converted to lower-precision formats before deployment; as Microsoft's Copilot+ documentation notes, "many NPU devices, however, only support integer math in lower bit format, such as INT8, for increased performance and power efficiency."[28] Lower precision means fewer transistors and less memory traffic per operation, which is where the efficiency comes from. It also means the software stack matters: a model must be converted, quantized, and compiled for each vendor's NPU, through frameworks such as ONNX Runtime with vendor execution providers on Windows.[28]

Smartphone NPUs

Apple Neural Engine

Apple has shipped a Neural Engine in every A-series chip since the A11 and every Mac chip since the M1, and it publishes an operations-per-second figure for most generations. Verified vendor figures:

ChipAnnouncedNeural EngineApple's stated performance
A11 BionicSeptember 2017dual-coreup to 600 billion operations/second [3]
A12 BionicSeptember 20188-coreup to 5 trillion operations/second [6]
M1November 202016-core11 trillion operations/second [7]
A17 ProSeptember 202316-core35 trillion operations/second; up to 2x faster than A16 [9][10]
M4May 202416-core38 trillion operations/second [22]
A18September 202416-coreruns machine learning models up to 2x faster than A16 [23]

When it introduced the M4 in May 2024, Apple claimed its Neural Engine was "more powerful than any neural processing unit in any AI PC today."[22] Note that Apple's press releases quote raw operations per second without stating the numeric precision behind the figure, which makes direct comparison with competitors' INT8 TOPS ratings unreliable.

The Neural Engine became the anchor for Apple Intelligence, announced in June 2024. Apple's on-device foundation model is "a ~3 billion parameter on-device language model" compressed with a mixed 2-bit and 4-bit quantization scheme averaging 3.7 bits per weight; on an iPhone 15 Pro, Apple reported time-to-first-token latency of about 0.6 milliseconds per prompt token and generation at 30 tokens per second.[24]

Qualcomm Hexagon

Qualcomm's NPU grew out of the Hexagon DSP line in Snapdragon chips. With the Snapdragon 8 Gen 3, announced at the Snapdragon Summit on October 24, 2023, the company built the launch around on-device generative AI: the Hexagon NPU was 98 percent faster than the prior generation with 40 percent better performance per watt, and Qualcomm called its AI Engine the first to support generative AI models of up to 10 billion parameters entirely on the phone.[11]

Google Tensor

Google introduced Tensor, "the first ever system on a chip made by Google," with the Pixel 6 in 2021. The chip was co-designed with Google Research around where machine learning models were heading and includes an on-device TPU for camera, speech, and translation workloads.[8] In December 2023 Google made the Pixel 8 Pro, running Tensor G3, the first phone to run Gemini Nano, its smallest large language model variant, powering Summarize in Recorder and Smart Reply in Gboard fully offline.[25]

The AI PC: NPUs reach laptops

NPUs stayed a phone feature for six years. That changed in 14 months across 2023 and 2024:

PlatformVendorAnnouncedNPUVendor NPU rating
Ryzen 7040 "Phoenix"AMD2023 (shipped mid-2023)XDNA (Ryzen AI)10 TOPS [16]
Snapdragon X EliteQualcommOctober 24, 2023Hexagon45 TOPS [12][13]
Core Ultra "Meteor Lake"IntelDecember 14, 2023NPU (AI Boost)about 11 TOPS; roughly 34 platform TOPS [14][15]
Ryzen 8040 "Hawk Point"AMDDecember 2023XDNA16 TOPS [16]
Ryzen AI 300 "Strix Point"AMDJune 2, 2024XDNA 2up to 50 TOPS [18]
Core Ultra 200V "Lunar Lake"IntelSeptember 2024NPU 448 INT8 TOPS [20][21]

AMD moved first, quietly: the Ryzen 7040 mobile series shipped in mid-2023 with an NPU built on XDNA, an architecture derived from the AI engines AMD acquired with Xilinx.[16] Intel's launch was louder. The Core Ultra series, released December 14, 2023, was Intel's first client processor with an integrated NPU, part of a disaggregated chiplet design built on the Intel 4 process, and Intel branded the moment the arrival of the "AI PC."[14][15]

Qualcomm supplied the chip that defined the category's performance bar. The Snapdragon X Elite, unveiled at the October 2023 Snapdragon Summit with a 12-core Arm-compatible Oryon CPU, carried a Hexagon NPU rated at 45 TOPS.[12][13]

Microsoft then turned that number into policy. On May 20, 2024, it announced Copilot+ PCs, "a new category of Windows PCs designed for AI" built around silicon "capable of an incredible 40+ TOPS," with features such as real-time translation and image generation running locally; the first wave, launched June 18, 2024 starting at $999, ran exclusively on Snapdragon X Elite and X Plus.[17] Copilot+ support expanded to AMD's Ryzen AI 300 series and Intel's Core Ultra 200V ("Lunar Lake") in an update announced September 3, 2024.[19] Lunar Lake's fourth-generation NPU delivers 48 INT8 TOPS, clearing the Copilot+ threshold that Meteor Lake's NPU missed by a wide margin.[20][21] AMD's Ryzen AI 300, announced at Computex on June 2, 2024, pairs Zen 5 CPU cores with an XDNA 2 NPU rated at up to 50 TOPS.[18]

The requirement had a side effect: it created the first mainstream market in which laptops are marketed and disqualified on an AI-specific spec. Meteor Lake and Hawk Point machines, sold as AI PCs in early 2024, do not qualify as Copilot+ PCs.

TOPS as a marketing metric

TOPS is a theoretical peak: the maximum number of operations the multiply-accumulate array could complete per second if fully utilized. As a comparison tool it has well-documented problems.

The first is precision. A TOPS figure is only meaningful alongside the numeric format it assumes, since a chip performs roughly twice as many INT8 operations as FP16 operations in the same silicon. Intel specifies INT8 for Lunar Lake's 48 TOPS;[21] Apple's operations-per-second claims specify no precision at all.[3][22] One industry explainer summarizes the resulting incomparability bluntly: TOPS persists in marketing because it is simple and large, and "every vendor can claim leadership by choosing the precision and configuration that maximises their number."[29]

The second is sparsity. Some vendors quote figures that assume structured sparsity, in which a portion of weights are zero and skipped, effectively doubling the headline number; Qualcomm published an explainer in July 2025 distinguishing "dense TOPS" from "sparse TOPS," arguing that dense figures represent true hardware performance and that sparse figures mislead without context.[30]

The third is that peak arithmetic is often not the bottleneck. Real workloads, especially language models, are frequently memory-bound: the NPU stalls waiting for weights to arrive rather than running out of compute, so memory bandwidth, cache design, and the quality of drivers and compilers determine how much of the theoretical figure an application ever sees.[29] Two chips with similar TOPS ratings can perform very differently on the same model, which is why benchmark suites such as MLPerf and per-model measurements are more informative than the headline number. Microsoft's own developer guidance for Copilot+ PCs tells developers to measure NPU performance with system traces and per-inference timings.[28]

What runs on an NPU

The workloads NPUs handle have expanded in roughly three waves. The first was perception: Face ID and Animoji on the A11,[3] scene-aware photography on the Kirin 970 and Pixel Visual Core,[4][5] and later always-on speech recognition and live captioning. Google's original Tensor pitch centered on running "state of the art ML" for camera and speech features, such as Live Caption and Recorder transcription, without draining the battery.[8]

The second wave was generative. Qualcomm's 10-billion-parameter on-device ceiling for the Snapdragon 8 Gen 3,[11] Gemini Nano's offline summarization on the Pixel 8 Pro,[25] Apple's ~3 billion parameter Apple Intelligence model,[24] and Copilot+ features such as local image generation[17] all moved small language models and diffusion-style workloads onto consumer NPUs, with edge AI quantization techniques making the models fit.

The third wave is operating-system plumbing. Windows 11 exposes the NPU as a first-class resource: Task Manager shows NPU utilization alongside CPU and GPU, Windows ML selects the appropriate vendor execution provider automatically (Qualcomm's QNN, Intel's OpenVINO), and models ship pre-quantized through catalogs such as Qualcomm AI Hub.[28]

Recent developments

Chipmakers have continued to iterate. Intel's Panther Lake generation, launched as the Core Ultra Series 3 at CES in January 2026, is the first client chip built on the Intel 18A process and carries a fifth-generation NPU rated at up to 50 INT8 TOPS, a design that prioritizes efficiency and die area over a bigger peak figure.[27]

Apple took a different route with the M5, announced October 15, 2025: alongside the 16-core Neural Engine, each of the chip's ten GPU cores gained a dedicated Neural Accelerator, giving over four times the peak GPU compute for AI compared with M4.[26] The move signals that the NPU is no longer the only, or even primary, AI engine in a consumer SoC; inference work is spreading back across GPU and CPU with the OS or framework choosing placement per task.[28]

The gap between spec-sheet TOPS and delivered value remains the category's open question. Laptop NPU ratings went from roughly 10 TOPS in 2023 to roughly 50 in 2026,[16][27] yet how much of that capacity applications actually use depends on quantized model availability and per-vendor software support,[28][29] and the dense-versus-sparse accounting dispute shows vendors still control the denominator of their own metric.[30]

See also

References

  1. ^Qualcomm OnQ Blog. "Introducing Qualcomm Zeroth Processors: Brain-Inspired Computing." October 2013. qualcomm.com/...rocessors-brain-inspired-computing
  2. ^Google Cloud Blog. "Google supercharges machine learning tasks with TPU custom chip." May 2016. cloud.google.com/...earning-tasks-with-custom-chip
  3. ^Apple Newsroom. "The future is here: iPhone X." September 12, 2017. apple.com/...the-future-is-here-iphone-x
  4. ^Huawei. "Huawei Reveals the Future of Mobile AI at IFA 2017." September 2, 2017. huawei.com/...mobile-ai-ifa-2017
  5. ^Google Blog. "Pixel Visual Core: image processing and machine learning on Pixel 2." October 17, 2017. blog.google/...essing-and-machine-learning-pixel-2
  6. ^Apple Newsroom. "iPhone XS and iPhone XS Max bring the best and biggest displays to iPhone." September 12, 2018. apple.com/...e-best-and-biggest-displays-to-iphone
  7. ^Apple Newsroom. "Apple unleashes M1." November 10, 2020. apple.com/...apple-unleashes-m1
  8. ^Google Blog. "Google Tensor debuts on the new Pixel 6 this fall." 2021. blog.google/...introducing-google-tensor
  9. ^Apple Newsroom. "Apple unveils iPhone 15 Pro and iPhone 15 Pro Max." September 12, 2023. apple.com/...s-iphone-15-pro-and-iphone-15-pro-max
  10. ^Wikipedia. "Apple A17." en.wikipedia.org/...Apple_A17
  11. ^Qualcomm. "Qualcomm Launches Premium Snapdragon 8 Gen 3 to Bring Generative AI to the Next Wave of Flagship Smartphones." October 24, 2023. qualcomm.com/...dragon-8-gen-3-to-bring-generative
  12. ^Qualcomm. "Qualcomm Unleashes Snapdragon X Elite: The AI Super-Charged Platform to Revolutionize the PC." October 2023. qualcomm.com/...x-elite--the-ai-super-charged-plat
  13. ^Qualcomm. "Snapdragon X Elite" (product specifications). qualcomm.com/...snapdragon-x-elite
  14. ^Intel Newsroom. "Intel Core Ultra Ushers in the Age of the AI PC" (Core Ultra launch). December 14, 2023. newsroom.intel.com/...ltra-client-computing-news-1
  15. ^Wikipedia. "Meteor Lake." en.wikipedia.org/...Meteor_Lake
  16. ^TechPowerUp. "AMD Ryzen 8040 Series 'Hawk Point' Mobile Processors Announced with a Faster NPU." December 2023. techpowerup.com/...ors-announced-with-a-faster-npu
  17. ^Microsoft Official Blog. "Introducing Copilot+ PCs." May 20, 2024. blogs.microsoft.com/...introducing-copilot-pcs
  18. ^HotHardware. "AMD Unveils Ryzen AI 300 Strix Point CPUs At Computex." June 2, 2024. hothardware.com/...ai-300-series-announce-computex
  19. ^Windows Experience Blog. "Copilot+ PCs expand availability with new AMD and Intel silicon." September 3, 2024. blogs.windows.com/...ith-new-amd-and-intel-silicon
  20. ^Intel Newsroom. "Intel Launches Core Ultra 200V Series Mobile Processors." September 2024. intel.com/...core-ultra-200v-series-mobile
  21. ^Wikipedia. "Lunar Lake." en.wikipedia.org/...Lunar_Lake
  22. ^Apple Newsroom. "Apple introduces M4 chip." May 7, 2024. apple.com/...apple-introduces-m4-chip
  23. ^Apple Newsroom. "Apple introduces iPhone 16 and iPhone 16 Plus." September 9, 2024. apple.com/...troduces-iphone-16-and-iphone-16-plus
  24. ^Apple Machine Learning Research. "Introducing Apple's On-Device and Server Foundation Models." June 10, 2024. machinelearning.apple.com/...ple-foundation-models
  25. ^Google Blog. "Pixel Feature Drop: December 2023." December 6, 2023. blog.google/...pixel-feature-drop-december-2023
  26. ^Apple Newsroom. "Apple unleashes M5, the next big leap in AI performance for Apple silicon." October 15, 2025. apple.com/...p-in-ai-performance-for-apple-silicon
  27. ^Wikipedia. "Panther Lake (microprocessor)." en.wikipedia.org/...Panther_Lake_(microprocessor)
  28. ^Microsoft Learn. "Copilot+ PCs developer guide." learn.microsoft.com/...npu-devices
  29. ^TechnoLynx. "AI TOPS on the Spec Sheet: Why the Headline Number Does Not Predict Real Performance." technolynx.com/...ai-tops-explained-misleading-spec
  30. ^Qualcomm OnQ Blog. "Dense TOPS vs. Sparse TOPS: What's the Difference?" July 22, 2025. qualcomm.com/...s-sparse-tops-whats-the-difference
  31. ^Google Cloud. "Introduction to Cloud TPU." docs.cloud.google.com/...intro-to-tpu

Improve this article

Add missing citations, update stale details, or suggest a clearer explanation. Every suggestion is reviewed for sourcing before it goes live.

v1 · 2,894 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent adversarial fact-check at creation (wanted38 campaign, 2026-07-24): every claim verified against primary sources by a dedicated verification agent; corrections applied before publication.

Cite this page: AI Wiki. "Neural Processing Unit." aiwiki.ai, updated 24 Jul 2026, fact-checked 24 Jul 2026. CC BY 4.0. https://aiwiki.ai/wiki/npu

Suggest edit