# AI Accelerator

> Source: https://aiwiki.ai/wiki/ai_accelerator
> Updated: 2026-07-28
> Categories: AI Hardware, AI Inference, AI Infrastructure
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution to "AI Wiki (aiwiki.ai)".

An AI accelerator is hardware designed or configured to execute artificial intelligence and machine learning workloads more efficiently than a general-purpose processor executing the same workload without specialized acceleration. The category includes programmable processors, fixed or configurable engines, and complete systems built around them. Common examples include [graphics processing units](/wiki/gpu), tensor or matrix processors, [neural processing units](/wiki/npu), application-specific integrated circuits, and field-programmable gate arrays. Their advantage usually comes from parallel arithmetic, data reuse, specialized number formats, and reduced control overhead, but usable performance depends on memory, interconnects, software, and model quality as well as arithmetic units.[1]

This article uses "AI accelerator" as a functional and system-level category. The related [AI chip](/wiki/ai_chip) article covers the narrower physical semiconductor device and product category. An AI chip may contain one or more accelerator engines, while an accelerator may be an on-die block, a whole chip, an add-in module, or a multi-chip server or cluster. The terms overlap in industry usage, but they are not exact synonyms.

Accelerators are used for model training and inference in data centers, personal computers, mobile devices, vehicles, robots, and embedded systems. No design is fastest or most efficient for every workload. Matrix shape, batch size, sequence length, numerical format, supported operators, memory capacity, latency target, power limit, and the maturity of the compiler and runtime can all change the result.

## Scope and terminology

AI accelerator terminology is not governed by one universal taxonomy. Several labels describe different dimensions and can apply to the same device:

| Dimension | Examples | What the label describes |
| --- | --- | --- |
| Implementation substrate | GPU, ASIC, FPGA | How the hardware is implemented and how much it can be reconfigured |
| Integration level | IP block, system-on-chip engine, discrete chip, module, server, cluster | The physical and system boundary |
| Execution organization | SIMD or SIMT, vector, systolic array, spatial dataflow, coarse-grained reconfigurable array | How operations and data movement are scheduled |
| Workload role | Training, inference, recommendation, vision, language, signal processing | The intended computation |
| Deployment class | Data center, client, mobile, automotive, industrial, edge | The power, latency, cost, and environmental constraints |

A GPU is a programmable many-core processor whose parallel execution and memory bandwidth make it useful for dense tensor work. Modern AI-oriented GPUs also contain matrix engines, so "GPU" and "tensor accelerator" can describe different parts of the same product. An [application-specific integrated circuit](/wiki/asic) describes a design and implementation class, not a unique execution model. It may implement a systolic array, vector engine, spatial dataflow, or a mixture of these.

An NPU is an industry label for a processor or processor block specialized for neural-network operations. Arm describes its Ethos NPU as an inference accelerator for matrix and tensor operations, while Qualcomm describes Hexagon as a heterogeneous NPU combining scalar, vector, and tensor engines with shared memory.[2][3] These examples show why NPU should not be defined solely by device size or by one microarchitecture. The name often appears in low-power client and edge systems, but it does not specify a universal instruction set or performance class.

A [Tensor Processing Unit](/wiki/tpu) is Google's product-family name for its custom machine-learning processors, rather than a generic standards-defined class. Google's first published TPU was an inference ASIC with a 65,536-element 8-bit multiply-accumulate matrix and software-managed on-chip memory.[4] Other companies use terms such as tensor core, matrix core, neural engine, or AI engine for related capabilities. Those labels should be interpreted from the relevant architecture documentation rather than treated as interchangeable specifications.

A [field-programmable gate array](/wiki/fpga) uses reconfigurable logic and routing. It can implement a data path tailored to a model without fabricating a new ASIC, but compilation time, available logic and memory, clock rate, and development complexity constrain what is practical. "Dataflow" describes an execution and mapping approach, and "wafer scale" describes an integration approach; neither is a separate alternative to GPU, ASIC, or FPGA in a mutually exclusive taxonomy.

## Workloads

### Training

Training repeatedly performs forward and backward passes, computes gradients, updates parameters, and often retains activations or recomputes them. Large training jobs can require more memory for parameters, gradients, optimizer state, and activations than one accelerator provides. Data, tensor, pipeline, and expert parallelism distribute those objects and operations across devices, but add collective communication and synchronization. Megatron-LM, for example, inserted communication operations into a model-parallel Transformer implementation and reported an 8.3-billion-parameter model trained across 512 GPUs.[5] The result is a historical system demonstration, not a general scaling guarantee.

Training hardware therefore needs more than high matrix throughput. It benefits from numerical range suitable for gradient calculations, memory capacity, efficient collective communication, error detection, checkpointing, and a software stack that can schedule overlapping compute and communication. Small models or irregular workloads may leave a large accelerator underused, while very large models may be limited by interconnect bandwidth or memory capacity rather than arithmetic.

### Inference

Inference applies a trained model. It may optimize for low single-request latency, high batch throughput, low energy per result, deterministic response time, or operation within a fixed thermal envelope. A data-center service can batch requests to improve utilization, while an interactive or safety-critical system may prioritize tail latency. Embedded inference often has tighter memory and power limits and may integrate sensor preprocessing, CPU control, GPU work, and an NPU in one system-on-chip.

Generative Transformer inference has distinct prefill and autoregressive decode phases. Prefill processes the input sequence in parallel and can have high arithmetic intensity. Decode produces tokens sequentially and, especially at small batch sizes, can spend much of its time moving weights and key-value cache data. A study on TPU v4 found materially different optimal partitioning and utilization for low-latency and high-throughput prefill and decode configurations.[6] The exact boundary depends on model architecture, sequence length, batch size, quantization, and implementation; "training is compute-bound and inference is memory-bound" is too broad to be a rule.

Other workloads create different bottlenecks. Convolutional networks reuse filters and feature maps, recommendation models combine dense computation with large embedding tables and all-to-all traffic, mixture-of-experts models route tokens among experts, and graph neural networks often use irregular memory access. Accelerator selection must start with a measured workload rather than a model parameter count alone.

## Architecture

### Compute organization

Most neural-network accelerators devote substantial hardware to multiply-accumulate operations used in matrix multiplication, convolution, and related tensor contractions. GPUs expose large numbers of programmable threads and increasingly include dedicated matrix instructions. Systolic arrays move operands rhythmically through a grid of processing elements. Spatial and dataflow architectures map operations and data paths onto processing elements so that intermediate values can remain local.

Specialization can improve throughput or energy efficiency by replacing some instruction decoding, speculation, caching, and general control with regular compute and explicitly managed storage. The tradeoff is reduced flexibility. Control-heavy code, dynamic shapes, unsupported operators, data-dependent sparsity, and small tensors can fail to fill a wide matrix engine. Accelerators commonly retain scalar or vector units and rely on host CPUs for orchestration and fallback.

### Memory hierarchy and dataflow

Moving data can cost more time and energy than performing an arithmetic operation. Eyeriss demonstrated this point for convolutional networks by using a row-stationary dataflow to reuse weights and activations in local storage and reduce partial-sum movement.[7] The broader lesson is not that one dataflow is always best, but that register files, local SRAM, caches, high-bandwidth memory, and off-package memory must be managed with the model's reuse patterns in mind.

The Roofline model relates attainable performance to peak compute, memory bandwidth, and operational intensity, meaning operations performed per byte transferred from the measured memory level.[8] A kernel below the model's ridge point is bandwidth-limited under the stated assumptions; adding arithmetic units alone will not raise its ceiling. Real accelerators have several memory levels and additional latency, synchronization, and communication limits, so a hierarchical or measured Roofline analysis is more useful than comparing a single peak number.

Algorithm and software changes can alter data movement without changing the chip. FlashAttention, for example, tiled exact attention to reduce reads and writes between GPU high-bandwidth memory and on-chip SRAM.[9] This illustrates hardware-software co-design: an accelerator's effective capability depends on algorithms that fit its memory hierarchy, not only its physical peak throughput.

### Numerical formats and sparsity

Accelerators support combinations of floating-point, integer, and sometimes block-scaled formats. Lower-bit formats can reduce storage and memory traffic and allow more arithmetic units in a fixed area, but they do not automatically provide an equal speedup. The hardware must implement the format, the kernel must use the relevant execution path, shapes must fill the units, and scaling and accumulation must preserve acceptable model quality.

The FP8 proposal introduced E4M3 and E5M2 encodings and reported results comparable with 16-bit baselines on the authors' evaluated image and language training tasks.[10] That evidence supports FP8 as a practical option for those tested settings, not a guarantee for every model. [Quantization](/wiki/quantization) calibration, rounding, overflow, underflow, accumulator precision, and sensitive layers require validation against an explicit quality target.

Sparsity presents a similar boundary. Skipping zeros can save work only when the hardware and software recognize the sparsity pattern without excessive indexing, load imbalance, or conversion overhead. NVIDIA's A100 architecture, for example, accelerated a documented 2:4 structured pattern rather than arbitrary sparsity.[11] Peak "sparse TOPS" should therefore not be compared with dense throughput unless the model, pattern, accuracy, and counting convention match.

### Packaging, memory, and interconnect

An accelerator system has several distinct communication levels:

- On-die and on-package fabrics connect compute engines, caches, memory controllers, and chiplets.
- Host links connect an accelerator to CPUs and system memory.
- Scale-up links connect accelerators within a server or tightly coupled pod.
- Scale-out networks connect servers and clusters.

The levels are not interchangeable. PCI Express 6.0, for example, specifies a 64 GT/s host and device interconnect with up to 256 GB/s in an x16 configuration, while Universal Chiplet Interconnect Express defines a package-level die-to-die interface.[12][13] A product may use those standards alongside proprietary on-package or scale-up links and Ethernet or InfiniBand at cluster scale.

Modern packages may combine multiple compute dies, I/O dies, and high-bandwidth memory. AMD's MI300 documentation, for example, describes stacked compute dies, I/O dies, HBM3, and an on-package fabric as one accelerator assembly.[14] This is also why a system discussion cannot be reduced to the specifications of one silicon die.

At cluster scale, topology and collective operations become architectural concerns. Google's TPU v4 system used optical circuit switches to reconfigure links among accelerator nodes and included separate sparse-processing cores for embedding-heavy models.[15] Such features are workload and system specific. Link bandwidth alone does not determine scaling: message size, topology, routing, collective algorithm, contention, software overlap, stragglers, and failures also matter.

### Host and software stack

An accelerator normally depends on a host processor, device firmware and drivers, compiler passes, operator libraries, runtime scheduling, memory allocation, communication libraries, and framework integration. A model that can be represented in a framework is not necessarily supported efficiently on every backend.

TVM showed an end-to-end compiler approach that performs graph and operator optimizations and maps workloads across CPUs, GPUs, FPGAs, and accelerator backends.[16] MLIR provides reusable intermediate representations at multiple abstraction levels for heterogeneous and domain-specific compilation.[17] These systems address portability and optimization, but portable source code does not imply equal performance across targets.

Runtime partitioning also exposes hardware limits. ONNX Runtime asks each execution provider which nodes or subgraphs it supports, assigns supported regions to that provider, and can use a lower-priority provider such as the CPU for the remainder.[18] A fallback can preserve correctness while adding transfers and synchronization that erase an expected speedup. Operator coverage, dynamic-shape support, kernel quality, graph-fusion behavior, debugging tools, and long-term software maintenance are therefore part of accelerator evaluation.

## Representative deployment classes

The following classes overlap. The examples illustrate system forms and are not an exhaustive product list or a performance ranking.

| Class | Typical strengths | Typical constraints | Representative examples |
| --- | --- | --- | --- |
| Programmable data-center GPU | Broad operator coverage, mature parallel programming, training and inference | High power and cooling demand; efficiency depends on utilization and software | NVIDIA data-center GPUs and AMD Instinct accelerators[11][14] |
| Cloud-operated custom accelerator | Hardware and software co-designed for a provider's services and network | Availability and tooling may be tied to one cloud | Google TPU and AWS Trainium or Inferentia[4][19] |
| Client or embedded NPU | Low-power inference near sensors and users; integration with other SoC engines | Smaller memory, thermal limits, and narrower operator support | Arm Ethos and Qualcomm Hexagon[2][3] |
| FPGA accelerator | Reconfigurable data paths and interfaces after manufacturing | Tool complexity, compile time, and finite on-chip resources | Vendor and research FPGA inference designs[1][16] |
| Research or specialized architecture | Can explore spatial, dataflow, wafer-scale, in-memory, analog, or photonic approaches | Evidence may rely on prototypes or simulation; software and manufacturing maturity vary | DianNao and Eyeriss are influential published digital examples[7][20] |

Product families evolve quickly. Exact supported formats, memory capacity, software versions, availability, and benchmark status should be verified from a dated architecture document and reproducible result rather than inferred from a family name.

## History

AI acceleration draws on earlier work in parallel array processors, digital signal processors, vector computing, and application-specific hardware. H. T. Kung's 1982 paper "Why Systolic Architectures?" described regular arrays that rhythmically compute and pass data, an organization later reused in tensor processors.[21]

GPUs became important to machine learning as researchers mapped learning algorithms onto highly parallel graphics hardware. A 2009 ICML paper reported that its GPU implementation of large deep belief networks was up to 70 times faster than its dual-core CPU implementation, while also documenting memory-transfer constraints.[22] AlexNet's 2012 ImageNet system used an efficient GPU implementation to train a large convolutional network, helping establish GPUs as a practical platform for deep learning research.[23] Both are workload- and era-specific results, not current CPU-GPU comparisons.

Dedicated neural-network architectures followed. DianNao, published at ASPLOS 2014, was a small-footprint accelerator for neural and machine-learning operations and received a conference best-paper award.[20] Google deployed its first TPU for data-center inference in 2015 and published its architecture and contemporary CPU and GPU comparison in 2017.[4] Eyeriss, published in 2016, made data reuse and movement energy central design concerns.[7]

Later systems integrated matrix engines into programmable GPUs and client processors, adopted lower-precision arithmetic, and scaled custom accelerators through high-bandwidth memory and specialized interconnects. The development did not converge on one architecture. Instead, it produced a heterogeneous landscape in which general programmability, workload specialization, memory organization, deployment scale, and software ecosystems are traded against one another.

## Evaluation

Peak TOPS or FLOPS is a property of a specified arithmetic path under a counting convention, not a complete application benchmark. A defensible comparison should disclose at least:

| Measure | Required context |
| --- | --- |
| Latency | Input shape, batch size, percentile, warmup, host overhead, and whether preprocessing and transfers are included |
| Throughput | Workload, batch or concurrency, latency constraint, and complete system configuration |
| Time to train | Model, dataset, target quality, optimizer, device count, software version, and convergence rule |
| Numerical quality | Baseline, metric, calibration or retraining method, and accepted tolerance |
| Memory | Usable capacity, measured bandwidth, model state, temporary buffers, and cache assumptions |
| Scaling | Device count, topology, collective algorithm, parallelism method, and single-device baseline |
| Energy | Measurement boundary, duration, workload, achieved quality, and idle treatment |
| Cost | Date, region, purchase or rental model, utilization, support, networking, and facility assumptions |

[MLPerf](/wiki/mlperf) addresses part of this problem by defining models, datasets, quality targets, scenarios, divisions, availability categories, compliance rules, and submission metadata.[24] Its inference power results measure average AC power for the entire system at the wall during the corresponding benchmark. MLCommons explicitly states that thermal design power and power-supply ratings are not validated substitutes.[24]

Results still require like-for-like interpretation. Closed and Open divisions permit different levels of model modification, availability categories distinguish purchasable systems from preview and research systems, and results may later be corrected or invalidated. Comparisons should use the same benchmark version, task, scenario, quality target, division, availability class, and measurement boundary.

Vendor microbenchmarks can be useful when their kernels, shapes, formats, sparsity, clocks, and power boundaries are disclosed. They should not be presented as application-wide speedups. Likewise, a percentage of peak can reveal utilization but does not by itself establish useful latency, energy, or model quality.

## Limitations and tradeoffs

- **Specialization versus flexibility:** Fixed data paths can be efficient for supported operations but age poorly when model structures or number formats change. More programmable designs accept control and area overhead to support a wider workload range.
- **Memory and communication walls:** A device can have idle arithmetic units because operands, weights, key-value state, gradients, or collective messages cannot arrive fast enough.
- **Numerical risk:** Lower precision and sparsity can change model outputs. Validation must use the deployed model, data distribution, and quality threshold.
- **Software dependence:** Missing kernels, graph breaks, CPU fallback, compiler regressions, and version-specific behavior can dominate end-to-end performance.
- **Scale and reliability:** Large jobs add topology, synchronization, checkpoint, repair, and straggler costs. A single-device benchmark does not predict cluster efficiency.
- **Power and cooling:** Package power is not facility energy. Hosts, memory, networks, storage, fans, cooling, and utilization affect the system boundary.
- **Evidence quality:** Vendor-reported peak figures, roadmaps, prices, and selected benchmarks can be accurate within their definitions while remaining unsuitable for a broad comparison. Independent reproduction and dated configuration records reduce this risk.

An accelerator is therefore best evaluated as one component of a measured hardware-software system. The useful question is not which architecture has the largest headline number, but which complete system meets the workload's quality, latency, throughput, energy, reliability, and cost constraints.

## References

1. Sze, V., Chen, Y.-H., Yang, T.-J., and Emer, J. S. "Efficient Processing of Deep Neural Networks: A Tutorial and Survey." Proceedings of the IEEE 105, no. 12 (2017): 2295-2329. https://arxiv.org/abs/1703.09039
2. Arm. "Build Edge AI Applications on Arm." Arm Developer, accessed July 28, 2026. https://developer.arm.com/edge-ai
3. Qualcomm. "Qualcomm Hexagon NPU." Accessed July 28, 2026. https://www.qualcomm.com/processors/hexagon
4. Jouppi, N. P., et al. "In-Datacenter Performance Analysis of a Tensor Processing Unit." Proceedings of the 44th Annual International Symposium on Computer Architecture (2017): 1-12. https://research.google/pubs/in-datacenter-performance-analysis-of-a-tensor-processing-unit/
5. Shoeybi, M., et al. "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism." arXiv:1909.08053, version 4, 2020. https://arxiv.org/abs/1909.08053
6. Pope, R., et al. "Efficiently Scaling Transformer Inference." Proceedings of Machine Learning and Systems 5 (2023). https://proceedings.mlsys.org/paper_files/paper/2023/hash/c4be71ab8d24cdfb45e3d06dbfca2780-Abstract-mlsys2023.html
7. Chen, Y.-H., Emer, J., and Sze, V. "Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks." Proceedings of the 43rd Annual International Symposium on Computer Architecture (2016): 367-379. https://research.nvidia.com/publication/2016-06_eyeriss-spatial-architecture-energy-efficient-dataflow-convolutional-neural
8. Williams, S., Waterman, A., and Patterson, D. "Roofline: An Insightful Visual Performance Model for Multicore Architectures." Communications of the ACM 52, no. 4 (2009): 65-76. https://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-134.pdf
9. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." Advances in Neural Information Processing Systems 35 (2022). https://arxiv.org/abs/2205.14135
10. Micikevicius, P., et al. "FP8 Formats for Deep Learning." arXiv:2209.05433, version 2, 2022. https://arxiv.org/abs/2209.05433
11. NVIDIA. "NVIDIA A100 Tensor Core GPU Architecture." White paper, version 1.0, 2020. https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-ampere-architecture-whitepaper.pdf
12. PCI-SIG. "PCI Express 6.0 Specification." Accessed July 28, 2026. https://pcisig.com/pci-express-6.0-specification
13. UCIe Consortium. "Universal Chiplet Interconnect Express Specifications." Accessed July 28, 2026. https://www.uciexpress.org/specifications
14. AMD. "AMD Instinct MI300 series microarchitecture." Accessed July 28, 2026. https://instinct.docs.amd.com/develop/gpu-arch/mi300.html
15. Jouppi, N. P., et al. "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings." Proceedings of the 50th Annual International Symposium on Computer Architecture (2023): 1-14. https://arxiv.org/abs/2304.01433
16. Chen, T., et al. "TVM: An Automated End-to-End Optimizing Compiler for Deep Learning." 13th USENIX Symposium on Operating Systems Design and Implementation (2018): 578-594. https://www.usenix.org/conference/osdi18/presentation/chen
17. Lattner, C., et al. "MLIR: Scaling Compiler Infrastructure for Domain Specific Computation." 2021 IEEE/ACM International Symposium on Code Generation and Optimization (2021): 2-14. https://research.google/pubs/mlir-scaling-compiler-infrastructure-for-domain-specific-computation/
18. ONNX Runtime. "Execution Providers." Accessed July 28, 2026. https://onnxruntime.ai/docs/execution-providers/
19. Amazon Web Services. "AWS Trainium: Getting started." Accessed July 28, 2026. https://aws.amazon.com/ai/machine-learning/trainium/getting-started/
20. Chen, T., et al. "DianNao: A Small-Footprint High-Throughput Accelerator for Ubiquitous Machine-Learning." Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (2014): 269-284. https://doi.org/10.1145/2541940.2541967
21. Kung, H. T. "Why Systolic Architectures?" Computer 15, no. 1 (1982): 37-46. https://doi.org/10.1109/MC.1982.1653825
22. Raina, R., Madhavan, A., and Ng, A. Y. "Large-scale Deep Unsupervised Learning using Graphics Processors." Proceedings of the 26th International Conference on Machine Learning (2009): 873-880. https://icml.cc/Conferences/2009/papers/218.pdf
23. Krizhevsky, A., Sutskever, I., and Hinton, G. E. "ImageNet Classification with Deep Convolutional Neural Networks." Advances in Neural Information Processing Systems 25 (2012). https://papers.nips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
24. MLCommons. "MLPerf Inference: Datacenter." Accessed July 28, 2026. https://mlcommons.org/benchmarks/inference-datacenter/