AI Chip

An AI chip is an integrated circuit, or a tightly integrated multi-die semiconductor package, designed or selected to execute artificial intelligence workloads efficiently. The term covers physical devices ranging from a phone processor containing an embedded neural-network engine to a discrete data-center processor packaged with High Bandwidth Memory. Common designs emphasize matrix or tensor arithmetic, data reuse, supported low-precision formats, and movement of weights and activations through a memory hierarchy.^[1]

This article uses "AI chip" for the physical semiconductor device and product implementation. For the broader functional category, including on-die engines, add-in modules, servers, and multi-device systems, see AI accelerator. An AI chip can contain several accelerator engines, while an accelerator can comprise several chips plus host processors, memory, networking, firmware, and software. Industry usage overlaps, but the terms are not exact synonyms.

"AI chip" is an informal umbrella rather than a single standardized processor class. A graphics processing unit, a custom ASIC, a reconfigurable device, and an embedded neural processor can all qualify when they are built or deployed for AI. Their usefulness cannot be determined from transistor count or peak operations alone. Model structure, numerical format, memory capacity and bandwidth, compiler coverage, latency target, power boundary, and system configuration all affect delivered results.^[1][31]

Scope and terminology

The word "chip" is used loosely in product literature. Keeping the physical levels separate prevents incorrect comparisons.

Scroll sideways for more →

This page focuses on the first three levels. It discusses modules and systems only where they define a chip specification or measurement boundary.

Several labels describe different, overlapping properties. GPU, ASIC, NPU, TPU, and chiplet labels sit on different axes, so treating them as one flat list of mutually exclusive chip types is misleading. A useful taxonomy separates implementation substrate, programming model, execution organization, physical integration, deployment class, and target workload.^[1][31]

• A neural processing unit is a vendor and industry label for a processor or engine specialized for neural-network operations. It does not specify one instruction set, die size, or power class.^[1][18]
• An ASIC is a circuit designed for a particular application or domain. Google's Tensor Processing Unit is one named family of machine-learning ASICs, but "TPU" is not a generic standards-body category.^[6]
• An FPGA is a reconfigurable integrated circuit on which an AI data path can be implemented after manufacturing. It trades some fixed-function density and efficiency for post-fabrication flexibility.^[1]
• "GPU" describes a programmable parallel-processor lineage. Modern data-center GPUs combine general vector or scalar execution with dedicated matrix units, so GPU and AI chip are not mutually exclusive labels.^[1][12]
• "Chiplet," "monolithic," and "wafer scale" describe physical integration. "Systolic," "SIMD," "SIMT," and "dataflow" describe execution or data movement. None is, by itself, a workload or market category.

An embedded AI block may share a system-on-chip with CPU cores, graphics, image processing, media, security, and I/O. A discrete data-center chip may instead devote much more area and package power to compute, memory interfaces, and chip-to-chip communication. Both fit this article's definition even though their deployment constraints differ greatly.^[12][18][19]

Historical development

Regular arrays of processing elements predate modern neural networks. H. T. Kung's 1982 paper described systolic architectures in which data rhythmically pass through arrays of cells, providing an architectural root for later matrix engines without claiming a modern AI product category.^[2]

As programmable graphics hardware became more capable, researchers mapped machine-learning operations to GPUs. A 2009 study by Raina, Madhavan, and Ng reported large workload-specific gains for deep unsupervised learning on a GPU, while also identifying host-to-device transfer as a practical cost.^[3] AlexNet's 2012 training used an efficient GPU convolution implementation and helped establish GPUs as a central platform for deep-learning research.^[4] These results are historical examples, not performance comparisons with current hardware.

Dedicated neural-network silicon emerged alongside this GPU path. DianNao, published in 2014, demonstrated a small-footprint accelerator architecture aimed at ubiquitous machine learning.^[5] Google deployed its first TPU for data-center inference in 2015 and published the design in 2017. That custom ASIC used a 65,536-element 8-bit multiply-accumulate matrix and 28 MiB of software-managed on-chip memory, illustrating the value of specializing arithmetic and memory organization for a known workload class.^[6]

Later products expanded in several physical directions at once: tensor units were added to programmable GPUs; cloud operators designed internal ASIC families; client processors integrated NPUs; and high-end packages combined several logic dies with stacked memory. Wafer-scale and compute-in-memory research explored alternatives to conventional die and memory boundaries. The result is a heterogeneous field rather than one converged chip architecture.^[1][31]

Architecture of an AI chip

Compute organization

Neural networks repeatedly apply multiply-accumulate operations, reductions, nonlinear functions, normalization, data rearrangement, and control. Dense matrix multiplication is therefore prominent, but a useful chip also needs vector or scalar paths for operations that do not map efficiently to a matrix array. Google's first TPU paired its matrix unit with other execution and memory resources, while Qualcomm describes its embedded Hexagon NPU as a combination of scalar, vector, tensor, and shared-memory components.^[6][18]

Designs may organize multiply-accumulate units as systolic arrays, spatial processing elements, vector lanes, SIMT cores, or mixtures of these. Eyeriss demonstrated a row-stationary spatial dataflow intended to reuse weights and activations locally for convolutional networks.^[7] It is one design point, not a universally best dataflow. Transformer attention, recommendation models, sparse mixtures of experts, convolutional networks, and small sensor models expose different shapes and bottlenecks.

Memory and data movement

Arithmetic units are useful only when data arrive quickly enough. An AI chip commonly combines registers, local scratchpads or SRAM, shared cache, off-package or in-package DRAM, and direct-memory-access engines. Designers choose which tensors remain stationary, which move between processing elements, and where intermediate results are materialized. Larger local storage can improve reuse but consumes die area and power.^[1][7]

The Roofline model relates attainable performance to operational intensity and the ceilings imposed by compute throughput and memory bandwidth.^[8] Real AI chips have more than one bandwidth ceiling because registers, SRAM, cache, high-bandwidth memory, host memory, and inter-chip links differ. A model that exceeds one level's capacity may move to a slower level, so a nominally compute-rich chip can deliver low utilization. Capacity and bandwidth must therefore be considered together.

High-end packages often place HBM stacks beside logic on an interposer or other advanced substrate. Embedded products more often share a unified memory pool or use mobile DRAM to meet area, cost, and standby-power limits. On-chip SRAM bandwidth and off-chip memory bandwidth are different measurements and should not be placed in the same comparison column without a clear boundary.

Numerical formats and sparsity

Smaller numerical formats can reduce storage, data movement, and circuit area, and can increase arithmetic throughput when the chip implements an appropriate path. Training commonly requires careful accumulation and scaling, while inference often tolerates more aggressive quantization. The FP8 E4M3 and E5M2 formats, for example, were evaluated with 16-bit-comparable results in specific training settings, not as a guarantee for every model or task.^[9]

Peak throughput does not automatically double whenever bit width is halved. Packing, accumulation format, tensor shape, alignment, memory pressure, compiler support, and the exact hardware path all matter. A chip may advertise separate dense and sparse peaks. Sparse acceleration may require a fixed pattern, metadata, or pruning procedure, and unstructured zeros do not necessarily activate the advertised path. Model quality must be checked at the chosen precision and sparsity.

Control, I/O, and reliability

AI chips also include schedulers, address generation, synchronization, firmware processors, memory controllers, host interfaces, telemetry, error detection, and often media or data-movement engines. These functions occupy area that is absent from a simple matrix-throughput calculation but are necessary for a usable device.^{[1][6][18][20]}

Large packages add further requirements: coherent or explicitly managed inter-die links, clock and power distribution, known-good-die testing, fault isolation, thermal sensors, and repair or redundancy. A product can expose several physical dies as one logical device, or expose partitions that software schedules independently. The packaging choice and programming model are related but not identical.^[13][16]

Training, inference, and deployment targets

Training computes forward activations, gradients, and parameter updates. It usually needs numerical behavior suitable for accumulation, storage for parameters and optimizer state, and communication paths for parallel execution. Inference uses trained parameters to produce outputs, but it is not one uniform workload. Batch size, latency target, sequence length, model architecture, and serving policy can move the bottleneck between compute, memory, and communication.^[1][10]

For autoregressive Transformers, prompt processing and token-by-token generation have different behavior. Research on Transformer inference shows that hardware partitioning and batch choices change with latency and throughput goals, so the slogan "inference is memory-bound" is not a universal chip-design rule.^[10] Large-batch prompt processing can use matrix units heavily; small-batch decoding may be limited by repeated weight or key-value-cache access.

These requirements produce distinct product emphases:

Scroll sideways for more →

Hardware and software are co-designed. A chip requires drivers, firmware, a compiler or kernel toolchain, optimized libraries, framework integration, profiling tools, and a compatibility policy. Runtime systems can partition a graph across providers and fall back to a CPU when an operator is unsupported.^[25] A theoretically efficient chip with incomplete operator, datatype, or shape coverage may spend substantial time moving data across that boundary.

Design, fabrication, and packaging

An AI-chip program begins with workloads and product constraints rather than a peak-operations target. Designers profile operators, tensor shapes, reuse, latency, accuracy, memory footprint, power, cost, and expected software evolution. They then select compute arrays, local memory, interconnect, I/O, number formats, control, and fault-management features. Hardware description, verification, synthesis, physical design, timing closure, and sign-off are performed against a foundry process design kit before tapeout.

A process node describes a manufacturing process and its design rules. Modern node names do not correspond to one directly comparable physical dimension; Intel's explanation notes that current names are closer to representations of density or performance changes than literal feature measurements.^[11] Node name alone does not establish transistor density, frequency, energy efficiency, yield, or product quality.

After fabrication, wafers are tested and cut into dies unless the design deliberately retains a larger wafer-scale fabric. Yield depends partly on die area and defect density. Smaller dies can improve the probability of obtaining usable pieces from a wafer, but a multi-die design then pays for inter-die interfaces, package area, assembly, testing, and more complex power and thermal behavior.^[11] AMD's MI300 family illustrates a production package that combines multiple vertically stacked compute dies, I/O dies, and eight HBM3 stacks through an internal fabric.^[13]

Chiplets let designers mix functions, reuse dies across products, or manufacture different functions on different processes. UCIe defines an open package-level die-to-die interface and protocol stack; its stated use cases include systems-in-package that exceed a single reticle and mix chiplet components.^[14] Proprietary links are also common. An interface standard does not by itself guarantee that arbitrary dies can be combined, because power, clocks, packaging, firmware, security, validation, and thermal design must also agree.

Advanced packaging is a functional part of many AI chips, not merely a protective enclosure. TSMC's CoWoS family, for example, integrates logic and HBM through silicon or redistribution-layer interposers.^[15] NIST identifies power delivery, heat removal, test, repair, and reliability as central challenges when many dies are assembled tightly enough to act like one device.^[16] Package-level memory and interconnect gains therefore come with manufacturing-capacity and validation constraints.

Wafer-scale integration takes a different approach. Cerebras reports that its WSE-3 uses a 46,225 mm2 wafer-scale device with 4 trillion transistors and 900,000 AI-oriented cores.^[17] Those vendor specifications demonstrate a physical form, not a general speed ranking. The chip is used inside a larger system, and its headline peak cannot be compared directly with a conventional package without matching workload, precision, power, and system boundaries.

Representative chip families

The following examples show distinct implementation choices. They are not exhaustive and do not rank current products. Product generations and availability change more quickly than the architectural distinctions.

Scroll sideways for more →

These examples also show why GPU, ASIC, NPU, cloud chip, and wafer scale should not be forced into one mutually exclusive taxonomy. NVIDIA's device is both a GPU and a multi-die AI chip. Google's TPU, Meta's MTIA, and Microsoft's Maia are custom accelerator products with different workload and system goals. Qualcomm's NPU is an on-die subsystem. Cerebras changes the manufacturing and integration boundary.^[1][31]

Experimental physical approaches

The production examples above use digital CMOS arithmetic with SRAM and DRAM hierarchies. Research devices explore different physical tradeoffs. An IBM-led 2023 demonstration used 35 million phase-change-memory devices across 34 analog tiles, plus digital-to-analog and analog-to-digital peripheral circuits and an on-chip mesh. The authors reported up to 12.4 TOPS/W of chip-sustained performance and demonstrated a speech-recognition model across five chips, while distinguishing software-equivalent from near-software-equivalent accuracy.^[23]

Neuromorphic chips instead model event-driven neurons and synapses. Intel's Loihi 2 is a research processor with programmable neuron models, local memory, asynchronous communication, and on-chip learning support.^[24] Its events, neurons, and synapses are not directly comparable with dense floating-point tensor operations. Analog, in-memory, and neuromorphic results should be evaluated on end-to-end tasks, accuracy, peripheral circuits, data conversion, programmability, and fabrication maturity rather than a single operations-per-second figure.

Reading chip specifications

A defensible comparison begins by fixing the physical and workload boundary.

Scroll sideways for more →

Peak TOPS or FLOPS is a capability of a specified arithmetic path, not an application-performance result. A matrix engine may be underused because a layer is too small, has an unsupported shape, performs non-matrix work, or waits for data. The Roofline model provides one way to reason about compute and bandwidth ceilings, but synchronization, launch overhead, inter-chip communication, and software add further limits.^[8]

Benchmark results should match the intended workload and system boundary. MLPerf Training measures time to reach a defined quality target, while its divisions and availability categories constrain what is being compared.^[26] MLPerf Inference defines scenarios and quality targets for complete submissions rather than abstract chips.^[27] A result from an eight-chip server is evidence about that configured system and software stack, not a chip-only score. Vendor-reported product figures remain useful when their precision, sparsity, workload, configuration, and measurement method are explicit.

Power deserves the same care. Thermal design power is an engineering envelope, not measured energy per inference or training step. Lower precision can reduce energy, but higher utilization can also raise instantaneous power. Package cooling capacity, idle behavior, memory, host processors, and networking determine whether a chip-level efficiency improvement reduces total system energy.^[27]

Industry and supply chain

AI-chip production depends on a specialized chain: architecture and circuit design, reusable IP and electronic-design automation, semiconductor equipment and materials, wafer fabrication, memory, substrates and interposers, assembly, test, firmware, and system integration. NIST describes the semiconductor supply chain as global, specialized, and interconnected, with geographic concentration among critical suppliers.^[28] A company that designs an AI chip may therefore depend on other firms for leading-edge logic, HBM, advanced packaging, and manufacturing equipment.

Foundry and packaging capacity are distinct constraints. TSMC's 2025 annual report describes leading-edge logic, CoWoS, SoIC, InFO, and other packaging or stacking technologies as separate investments supporting high-performance and AI products.^[29] HBM also combines memory fabrication with stack assembly and integration beside a logic die. Consequently, a shortage or qualification delay in packaging, memory, substrates, or test can limit product supply even when logic wafers are available.

Advanced computing integrated circuits are also subject to trade and security policy. As of the July 28, 2026 research cutoff, the U.S. Export Administration Regulations included controls and end-use provisions for specified advanced-computing ICs, supercomputers, advanced-node integrated circuits, semiconductor manufacturing equipment, and related parties or destinations.^[30] Classifications, thresholds, exceptions, license policies, and entity listings change. A product name or marketing label alone does not determine legal status, and current official rules and professional compliance advice are required for a transaction.

Limitations and tradeoffs

Specialization trades flexibility for potential gains in throughput, latency, energy, area, or cost on selected workloads. A fixed data path can become less useful when model operators, tensor shapes, or number formats change. More programmable chips can adapt through kernels and compilers but devote area and power to control, scheduling, and generality.^[1][31]

Memory capacity can be as important as arithmetic. A model that does not fit at the desired precision must be partitioned, streamed, or reduced, each of which changes latency and communication. More HBM and wider interfaces increase package complexity, power delivery, heat flux, and cost. More chiplets can improve reuse and yield but add inter-die traffic and assembly risk. Wafer-scale designs avoid some package crossings but require defect-tolerance and unusual cooling and system integration.^{[1][13][16][17]}

No chip is optimal for every stage of AI. Training, interactive generation, high-throughput serving, recommendation, vision, and always-on sensing reward different balances. The meaningful unit of selection is therefore a verified workload running through a supported software stack on a clearly bounded device or system, with model quality and operational constraints measured alongside speed.

References

1. ^Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel Emer, "Efficient Processing of Deep Neural Networks: A Tutorial and Survey," Proceedings of the IEEE 105(12), 2017. arxiv.org/...1703.09039
2. ^H. T. Kung, "Why Systolic Architectures?," Computer 15(1), 1982. doi.org/...MC.1982.1653825
3. ^Rajat Raina, Anand Madhavan, and Andrew Ng, "Large-scale Deep Unsupervised Learning using Graphics Processors," ICML, 2009. icml.cc/...218.pdf
4. ^Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," NeurIPS, 2012. papers.nips.cc/...b9d6b76c8436e924a68c45b-Abstract
5. ^Tianshi Chen et al., "DianNao: A Small-Footprint High-Throughput Accelerator for Ubiquitous Machine-Learning," ASPLOS, 2014. doi.org/...2541940.2541967
6. ^Norman P. Jouppi et al., "In-Datacenter Performance Analysis of a Tensor Processing Unit," ISCA, 2017. research.google/...sis-of-a-tensor-processing-unit
7. ^Yu-Hsin Chen, Joel Emer, and Vivienne Sze, "Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks," ISCA, 2016. research.nvidia.com/...taflow-convolutional-neural
8. ^Samuel Williams, Andrew Waterman, and David Patterson, "Roofline: An Insightful Visual Performance Model for Multicore Architectures," Communications of the ACM 52(4), 2009. www2.eecs.berkeley.edu/...EECS-2008-134.pdf
9. ^Paulius Micikevicius et al., "FP8 Formats for Deep Learning," 2022. arxiv.org/...2209.05433
10. ^Reiner Pope et al., "Efficiently Scaling Transformer Inference," MLSys, 2023. proceedings.mlsys.org/...ca2780-Abstract-mlsys2023
11. ^Intel, "Explaining Common Chip Terms," accessed July 28, 2026. newsroom.intel.com/...explaining-common-chip-terms
12. ^NVIDIA, "Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era," August 22, 2025. developer.nvidia.com/...owering-the-ai-factory-era
13. ^AMD, "AMD Instinct MI300 series microarchitecture," accessed July 28, 2026. instinct.docs.amd.com/...mi300
14. ^UCIe Consortium, "Universal Chiplet Interconnect Express Specifications," accessed July 28, 2026. uciexpress.org/specifications
15. ^TSMC, "CoWoS," accessed July 28, 2026. 3dfabric.tsmc.com/...cowos
16. ^National Institute of Standards and Technology, "National Advanced Packaging Manufacturing Program," accessed July 28, 2026. nist.gov/...vanced-packaging-manufacturing-program
17. ^Cerebras Systems, "WSE-3," accessed July 28, 2026. cerebras.ai/chip
18. ^Qualcomm, "Qualcomm Hexagon NPU," accessed July 28, 2026. qualcomm.com/...hexagon
19. ^Apple, "Apple unleashes M1," November 10, 2020. apple.com/...apple-unleashes-m1
20. ^Meta, "MTIA v1: Meta's first-generation AI inference accelerator," May 18, 2023. ai.meta.com/...ining-inference-accelerator-AI-MTIA
21. ^Amazon Web Services, "AWS Trainium Research," accessed July 28, 2026. aws.amazon.com/...research
22. ^Microsoft, "Microsoft Ignite 2023: AI transformation and the technology driving change," November 15, 2023. blogs.microsoft.com/...e-technology-driving-change
23. ^S. Ambrogio et al., "An analog-AI chip for energy-efficient speech recognition and transcription," Nature 620, 2023. nature.com/...s41586-023-06337-5
24. ^Intel Labs, "Taking Neuromorphic Computing to the Next Level with Loihi 2," 2021. intel.com/...romorphic-computing-loihi-2-brief.pdf
25. ^ONNX Runtime, "Execution Providers," accessed July 28, 2026. onnxruntime.ai/...execution-providers
26. ^MLCommons, "MLPerf Training," accessed July 28, 2026. mlcommons.org/...training
27. ^MLCommons, "MLPerf Inference: Datacenter," accessed July 28, 2026. mlcommons.org/...inference-datacenter
28. ^National Institute of Standards and Technology, "Vision for Success: Facilities for Semiconductor Materials and Manufacturing Equipment," accessed July 28, 2026. nist.gov/...-materials-and-manufacturing-equipment
29. ^TSMC, "2025 Annual Report," 2026. investor.tsmc.com/...english
30. ^U.S. Bureau of Industry and Security, "Export Administration Regulations, Part 744," current through July 27, 2026. bis.gov/...744
31. ^Albert Reuther et al., "Survey and Benchmarking of Machine Learning Accelerators," IEEE HPEC, 2019. doi.org/...HPEC.2019.8916327