Graphics processing unit

A graphics processing unit (GPU) is a processor designed to execute many similar operations in parallel. GPUs were developed for graphics workloads, in which the same transformations and shading calculations are applied to large numbers of vertices, fragments, or pixels. Their programmable arithmetic resources are also used for general-purpose computing and for many artificial intelligence workloads.^[1][2]

A GPU is not simply a faster CPU. A CPU usually devotes more of its silicon and power budget to low-latency execution, large caches, branch prediction, and a relatively small number of complex instruction streams. A GPU devotes more resources to arithmetic throughput and keeps many lightweight threads in flight. This design can work well when a problem exposes substantial data parallelism, but it can perform poorly when work is serial, branch-heavy, too small to occupy the device, or dominated by memory transfers.^[6][10]

Modern AI systems commonly use GPUs for training and inference because operations such as matrix multiplication, convolution, normalization, and attention can be expressed as large parallel kernels. GPUs are only one class of AI accelerator, however. CPUs, tensor processors, application-specific integrated circuits, field-programmable gate arrays, and other devices can be preferable for particular latency, power, cost, software, or deployment constraints. No fixed GPU-versus-CPU speedup applies across workloads.^[10]

The GPUs sold for AI in the mid-2020s differ from earlier general-purpose GPUs in four specific ways, each covered below: dedicated matrix-multiply engines rather than only vector arithmetic units; support for numerical formats far narrower than the 32-bit floating point that once served as the default; on-package high-bandwidth memory instead of conventional graphics DRAM; and high-speed coherent links that make dozens of devices behave, for some purposes, like one. Together these changes moved the usual performance limit away from arithmetic and toward memory traffic and communication, which is why a modern data-center GPU can be simultaneously rich in floating-point capability and starved of the bandwidth needed to use it.

Reading GPU specifications requires care. Vendor tables often lead with the throughput available when a matrix operand has been pruned into a hardware-supported sparsity pattern, and that figure is exactly twice the ordinary dense figure for the same format. A throughput number that does not state both its numerical format and whether it is dense or sparsity-assisted cannot be compared with anything. This article states both wherever it quotes a figure. Product specifications cited here are current as of August 2026 and are used to illustrate architectural points rather than to catalogue products; the wiki maintains separate pages for individual devices such as the NVIDIA H100, NVIDIA B200, and NVIDIA Vera Rubin platforms.

Architecture

Throughput-oriented execution

A GPU contains multiple parallel processing units, called streaming multiprocessors in NVIDIA documentation and compute units in AMD terminology. A program launches many threads, which the hardware groups for execution. In NVIDIA's CUDA model, a warp contains 32 threads. Threads in a warp start at the same program address but have their own registers and instruction state. When they take different branches, the hardware may need to execute the paths separately, reducing the useful work performed per instruction.^[6]

This execution style is often called single instruction, multiple thread, or SIMT. It resembles vector or single instruction, multiple data execution, but exposes a thread-oriented programming model. Other vendors use different names and group widths, and widths can vary by architecture. Software should not assume that every GPU has NVIDIA's 32-thread warp organization.

GPUs hide latency by maintaining many runnable thread groups. When one group waits for data, a scheduler can issue work from another group. Effective latency hiding therefore depends on having enough independent work and enough registers and local memory to keep multiple groups resident. A kernel that uses too many per-thread resources can reduce occupancy, but maximum occupancy does not by itself guarantee maximum performance.^[6]

Memory hierarchy

GPU performance depends on data movement as much as arithmetic. A discrete accelerator normally has its own device memory and communicates with a host processor over an interconnect. Integrated GPUs and some system-on-chip designs share physical memory with a CPU, although they still have caches and access rules that affect performance.

Common levels of a GPU memory hierarchy include:

• registers private to a thread;
• low-latency on-chip memory shared by threads in a workgroup or block;
• one or more hardware-managed caches;
• off-chip device memory, which may use GDDR or high-bandwidth memory; and
• host memory reached through an interconnect or a unified-memory mechanism.

Registers and on-chip shared memory are fast but limited. Device memory has much greater capacity but higher latency. Efficient kernels arrange accesses so adjacent threads use nearby addresses, reuse values from on-chip storage, and avoid unnecessary transfers between host and device. CUDA documentation calls the first pattern coalescing. Unified virtual addressing or managed memory can simplify programming, but it does not remove the physical cost of moving pages or cache lines.^[6][27]

Data-center parts have also added large last-level caches to reduce traffic to device memory. AMD's Instinct MI350X, for example, carries 256 MB of what AMD calls Infinity Cache shared across its compute dies, in addition to 4 MB of L2 per die and 32 KB of L1 per compute unit.^[33] A cache of that size can hold intermediate tensors that would otherwise be written to and re-read from device memory, but it is far too small to hold the weights of a large model, so it does not remove the bandwidth limit described later in this article.

High-bandwidth memory

Data-center GPUs use high-bandwidth memory (HBM) rather than the GDDR memory found on consumer graphics cards. HBM stacks several DRAM dies vertically, connects them with through-silicon vias, and places the stacks on the same package as the processor, linked through a silicon interposer. The design trades clock speed for interface width: an HBM stack presents a very wide bus at a modest per-pin data rate, where GDDR uses a narrower bus at a higher rate. Placing the memory on-package also shortens the wires, which reduces the energy spent per bit moved.

JEDEC published the HBM4 standard, JESD270-4, in April 2025. It specifies transfer rates up to 8 Gb/s across a 2048-bit interface, giving up to 2 TB/s per stack; it doubles the independent channels per stack from 16 in HBM3 to 32, each split into two pseudo-channels; and it supports 4-high, 8-high, 12-high, and 16-high stacks built from 24 Gb or 32 Gb dies.^[43] Capacity per device is therefore bounded by how many stacks a package can host and how tall each stack can be, and bandwidth by interface width times data rate. Both limits are set by packaging as much as by the memory dies.

The practical effect is visible across recent generations.

Scroll sideways for more →

Memory capacity and memory bandwidth answer different questions and are often confused. Capacity decides whether a model, its activations, and its caches fit on the device at all. Bandwidth decides how quickly that state can be read, which for much of AI inference is the quantity that sets the achievable speed.

Specialized execution units

General GPU arithmetic units handle integer and floating-point instructions. Graphics processors also contain fixed-function units for operations such as texture sampling and rasterization. Data-center and AI-oriented designs may add matrix-multiply engines.

NVIDIA introduced Tensor Cores in its Volta architecture. In the matrix-multiply-accumulate operation documented for Volta, the multiplicand matrices were FP16, while the accumulator and output matrices could be FP16 or FP32; NVIDIA's white paper describes the principal mixed-precision path as FP16 input with FP32 accumulation.^[12] Later products support additional formats and shapes, so Volta's operation should not be treated as a specification for every Tensor Core generation. AMD's CDNA architecture includes matrix cores, and Intel's Xe GPU architecture describes Xe Matrix Extensions alongside vector engines.^[7][13]

Specialized matrix hardware increases peak throughput only for supported operations, data types, dimensions, and layouts. Software libraries may pad or transform tensors to use those units. An advertised peak rate therefore does not state the speed of an arbitrary model or kernel.

What a matrix engine actually does

A tensor core is a hardware unit that computes a small matrix multiply-accumulate, D = A x B + C, as a single instruction across a group of threads, rather than issuing a stream of scalar or vector fused multiply-add operations. The advantage is not that multiplication becomes faster in isolation. It is that a single instruction retires many multiply-accumulate operations, so the per-operation cost of instruction fetch, decode, scheduling, and register file access is amortized across the whole tile, and operands can be held in a dedicated datapath instead of being read from the general register file for every product.

Three consequences follow, and all of them matter more than the headline throughput number.

First, the tiles have fixed shapes. A matrix engine accelerates a general matrix multiply only when the operation can be decomposed into supported tile shapes with acceptable padding. Small matrices, awkward dimensions, or unusual layouts leave much of the unit idle. NVIDIA's own performance guidance is built around tile quantization and wave quantization effects for exactly this reason.^[18]

Second, the engines are mixed precision by construction. The inputs are held in a narrow format while the accumulator is wider, typically FP32 for 16-bit and 8-bit inputs. This is what makes low-precision matrix multiplication numerically usable: the rounding error of the individual products is bounded by the input format, but the summation over the reduction dimension, where error would otherwise accumulate, happens at higher precision.^[12][37]

Third, only matrix work benefits. Elementwise operations, normalizations, softmax, activation functions, data movement, and reductions still run on the general arithmetic units and are usually limited by memory bandwidth. A model whose runtime is dominated by such operations sees little of the matrix engine's peak, which is one reason kernel fusion and attention-specific kernels have such large effects.

Every major vendor now ships an equivalent unit: NVIDIA Tensor Cores, AMD Matrix Cores (1,024 of them on an MI350X module),^[33] and Intel's Xe Matrix Extensions.^[7] Google's tensor processing unit reaches the same objective with a systolic array, a different structure that also amortizes control overhead across a large block of multiply-accumulate hardware.^[51]

Numerical formats and the low-precision progression

The most consequential architectural change of the past decade is that the arithmetic got narrower. Each halving of the element width halves the bytes that must be read to perform a given multiplication and, on hardware that supports the format natively, roughly doubles the peak matrix throughput. The cost is representational: fewer bits means less dynamic range, less precision, or both.

Scroll sideways for more →

The progression is not simply "fewer bits are better." Each format solves a specific problem.

TF32 exists because FP32 matrix multiplication was the bottleneck in training codes that did not want to change their numerics. It keeps FP32's 8-bit exponent, so no rescaling is needed, but truncates the mantissa to FP16's 10 bits and accumulates in FP32. NVIDIA introduced it with the Ampere architecture, where it is the default path for FP32 matrix operations in the deep-learning libraries.^[37] On an A100, TF32 matrix throughput is 156 TFLOPS dense against 19.5 TFLOPS for classical FP32 vector arithmetic.^[29]

FP16 and BF16 divide the 16-bit budget differently. FP16 keeps 10 mantissa bits but only 5 exponent bits, so gradient values can underflow to zero; the standard remedy is loss scaling plus an FP32 master copy of the weights, as described by Micikevicius and colleagues.^[15] BF16 keeps FP32's 8-bit exponent and accepts a 7-bit mantissa, so the same range is available without rescaling, at the cost of coarser resolution. Both are supported at identical throughput on current NVIDIA and AMD data-center parts.^[28][33]

FP8 arrived with two encodings rather than one, standardized in a joint proposal from NVIDIA, Arm, and Intel: E4M3 for the forward pass, where precision matters more, and E5M2 for gradients, where dynamic range matters more. The authors reported that FP8 training matched the result quality of 16-bit training across convolutional networks, recurrent networks, and transformers, including runs at 175 billion parameters, and also examined post-training quantization of models originally trained in 16 bits.^[34] Because a single per-tensor scale cannot cover the range of every tensor, FP8 training in practice depends on per-tensor scaling factors that are updated during the run, which is what NVIDIA's Transformer Engine library automates.

At four bits, per-tensor scaling is no longer sufficient, and the industry moved to block scaling. The Open Compute Project's Microscaling (MX) specification, version 1.0, defines a shared scale factor over a block of 32 elements, with the scale itself stored as an 8-bit power of two (E8M0) and the elements as FP8, FP6, FP4, or INT8.^[35] NVIDIA's NVFP4 uses a smaller block of 16 elements and an E4M3 scale with mantissa bits rather than a pure power of two, plus a second per-tensor FP32 scale; NVIDIA states that the finer blocks and higher-resolution scale reduce quantization error relative to the power-of-two approach.^[36] The important general point is that a 4-bit format is not a bare 4-bit number: it is a 4-bit element plus a shared scale, and the effective storage cost and accuracy depend on the block size as much as on the element width.

What each step costs in accuracy is workload dependent and has to be measured. The pattern reported across the literature is that the risk rises as the format narrows and as the format is applied to more of the computation. TF32 and BF16 are generally drop-in for training. FP8 requires a scaling strategy and is now routine for both training and inference. FP4 is used mainly for inference weights and activations with block scaling, and its accuracy depends on the quantization recipe as much as the hardware. In every case, the correct evidence is an end-to-end quality measurement on the target task, not a peak operations-per-second figure. See quantization, post-training quantization, and quantization-aware training for the methods.

There is a second benefit to narrow formats that is often overlooked and, for inference, is usually the larger one. Halving the width of the weights halves the bytes that must be streamed from memory for each forward pass. On a workload whose limit is memory bandwidth rather than arithmetic, that alone can nearly halve the time per step even if the matrix engine's extra throughput goes unused.

Structured sparsity and how to read a throughput figure

From the Ampere generation onward, NVIDIA data-center GPUs contain matrix units that can skip a specific pattern of zeros. The pattern, called 2:4 or fine-grained structured sparsity, requires that in every group of four consecutive values along the reduction dimension, at least two are zero. That fixes the sparsity level at 50 percent and makes the compressed representation cheap: the nonzero values are stored contiguously at half the original size, alongside small metadata indices recording which of the four positions each surviving value occupied. The hardware reads the metadata, selects the matching activations, and performs half as many multiply-accumulate operations, which yields twice the math throughput when the first operand of the matrix multiply is in this compressed form.^[30]

NVIDIA researchers described a train-prune-retrain workflow for producing such weights: train a dense model normally, prune each group of four weights to a 2:4 pattern, then repeat the original training schedule on the surviving weights with the same hyperparameters. They reported that this recipe preserved accuracy across a wide range of tasks and architectures without per-model hyperparameter search.^[30]

The distinction that matters to a reader is between that peak-math property and the end-to-end result. The 2x applies to a supported general matrix multiply whose first operand has actually been pruned and compressed. It does not apply to elementwise work, to attention over a key-value cache, to layers whose shapes are unsupported, or to any model that was never pruned into the pattern. Published measurements bear this out. PyTorch engineers reported a 1.3x speedup across the forward and backward passes of the linear layers in a ViT-L MLP block on A100 GPUs, and about 6 percent end-to-end wall-clock reduction for DINOv2 ViT-L training on ImageNet, with top-1 accuracy of 82.7 against 82.8 for the dense baseline. They also noted that the speedup depended on a fast custom sparsification kernel: using the stock cuSPARSELt compression path, the compression call alone took 380 microseconds against a dense-to-sparse matrix multiply saving of 151 microseconds, so the optimization lost money.^[61]

Most models distributed as open weights are dense. For the typical deployment, the dense figure is the one that applies.

This matters because vendors put the sparsity-assisted number in the headline position. The pattern is consistent and, once known, easy to spot.

Scroll sideways for more →

Applied to a single device, the arithmetic looks like this. The H100 SXM datasheet lists 3,958 TFLOPS of FP8 tensor throughput and 1,979 TFLOPS for BF16 and FP16, all marked as shown with sparsity, with the footnote stating that specifications are one-half lower without it. The dense figures are therefore 1,979 TFLOPS at FP8 and 989 TFLOPS at BF16.^[28] AMD prints both columns for the MI350X: 4.614 PFLOPS of FP8 dense against 9.2274 PFLOPS with sparsity, and 9.2275 PFLOPS of FP4 dense against 18.455 PFLOPS with sparsity.^[33] For the HGX B200 board of eight GPUs, NVIDIA lists 144 PFLOPS of FP4 with sparsity and 72 PFLOPS dense, that is 18 and 9 PFLOPS per GPU.^[31]

Two rules follow. If two figures are quoted for the same format and one is exactly twice the other, the larger is the sparsity-assisted number. If only one figure is quoted, check the footnote, because a single unqualified number in a marketing table is more often the sparse one. Comparisons that mix a sparse figure for one device with a dense figure for another are meaningless, and this error is common in secondary coverage.

Chiplets and multi-die packaging

Lithography imposes a maximum area for a single die, the reticle limit, of roughly 800 square millimetres on current equipment. Designs that need more transistors than that must be split across multiple dies and reassembled in the package. Both major GPU vendors have crossed that line.

NVIDIA's Blackwell architecture places two reticle-limited dies, 208 billion transistors in total, on one package, joined by an interface NVIDIA calls NV-HBI at 10 TB/s, and presents the result to software as a single GPU with a unified memory space.^[62] NVIDIA states that the Rubin GPU carries 336 billion transistors across two compute dies.^[59] AMD went further and earlier in die count: an MI350X module contains eight accelerator compute dies of 32 compute units each, built on a 3 nm process, sitting on two mirrored I/O dies built on 6 nm, with 256 MB of last-level cache shared across the compute dies.^[33]

The engineering trade is straightforward. Smaller dies yield better, and mixing process nodes lets logic that does not benefit from the leading node stay on a cheaper one. Against that, every byte crossing a die boundary costs latency and energy, and if the cross-die link is slow relative to on-die traffic the device develops non-uniform memory behaviour that software must manage. The vendors' stated design goal in both cases is to make the multi-die device behave as one, which is why the die-to-die bandwidth figures are quoted in terabytes per second.

Multi-die construction also depends on advanced packaging, principally 2.5D silicon interposer processes such as the CoWoS family from TSMC, which is also what places the HBM stacks beside the compute dies. That step is a separate production line from wafer fabrication, with its own equipment and lead times, so a finished accelerator requires capacity in logic wafers, HBM, and packaging at the same time.

Development

Early graphics accelerators implemented a mostly fixed rendering pipeline. As vertex and fragment stages became programmable, researchers mapped non-graphics calculations onto graphics APIs and described the field as general-purpose computation on graphics hardware. A 2005 survey documented this transition and the programming difficulties created by expressing computations as graphics operations.^[1]

NVIDIA marketed the GeForce 256 in 1999 as "the world's first GPU." This is a company marketing claim, not proof that one firm invented all graphics processors or that no earlier device meets a broader definition.^[3] The term subsequently became the common name for processors that combine graphics functions with highly parallel programmable execution.

The release of CUDA in November 2006 gave developers a C-like environment for general-purpose computation on NVIDIA GPUs without requiring them to formulate every task as a graphics pipeline.^[4] The Khronos Group released OpenCL 1.0 in December 2008 as a royalty-free specification for parallel programming across heterogeneous processors.^[5] These systems helped establish GPU computing as a general high-performance computing method.

GPUs also became central to modern deep learning. The 2012 AlexNet paper trained a large convolutional network on two NVIDIA GTX 580 GPUs with 3 GB of memory each. The authors reported that training took five to six days and described how the network was divided between the two devices.^[14] This result was historically influential, but its particular hardware, model, dataset, and timing should not be generalized to present systems.

The subsequent decade added the features that define an AI GPU. The table below lists the first appearance of each in NVIDIA's data-center line, which is the line most often used as a reference point; competing vendors added equivalents on their own schedules.

Scroll sideways for more →

The direction of travel is visible in that table. Arithmetic width fell from 32 bits to 4. Memory capacity and bandwidth per device rose by roughly an order of magnitude. Package-level integration replaced single-die scaling. AMD followed a parallel path, reaching FP6 and FP4 matrix support and 288 GB of HBM3E with the CDNA 4 generation in 2025,^[33] and announcing the Instinct MI400 series with HBM4 and the Helios rack-scale system at its Advancing AI event on 23 July 2026.^[60]

Programming models and software

Kernels, threads, and synchronization

A GPU program commonly consists of host code and one or more device kernels. The host allocates or identifies memory, moves or maps data, launches kernels, and retrieves results. A kernel is executed by many threads. CUDA organizes threads into blocks and blocks into a grid; threads within a block can cooperate through shared memory and block-level synchronization.^[6]

Synchronization scope matters. A barrier within one block does not normally synchronize every block in a grid. Operations that span devices require explicit communication or a library that implements it. Incorrect assumptions about ordering can produce race conditions even when a program appears to work on one device.

Kernel launches and host-device transfers have overhead. Combining small operations, keeping data on the device, and exposing enough work per launch can improve utilization. These techniques are workload-dependent: fusing operations can also increase register use, duplicate computation, or complicate numerical behavior.

Vendor and cross-platform ecosystems

CUDA is NVIDIA's platform and programming model for its GPUs. It includes a compiler toolchain, runtime, profilers, and optimized libraries. CUDA source and binaries are not a vendor-neutral GPU interface.^[6]

AMD's HIP provides a C++ runtime and kernel language intended to make GPU code portable between AMD and NVIDIA platforms. AMD documents mechanical CUDA-to-HIP translation tools, but also notes that porting can require manual work. Source portability does not guarantee identical numerical behavior or performance.^[8]

OpenCL defines a cross-platform host and kernel API for heterogeneous computation.^[5] SYCL is a Khronos standard that provides a single-source C++ programming model built on heterogeneous backends.^[9] An implementation must still support the target hardware, and performance tuning may depend on the device's execution width, memory hierarchy, compiler, and libraries. This is the distinction between functional portability and performance portability.

Machine-learning users often interact with frameworks instead of writing kernels directly. TensorFlow, PyTorch, and JAX construct tensor operations and dispatch them through runtime systems and vendor libraries. NVIDIA's cuDNN, for example, supplies tuned primitives for convolutions, attention, matrix multiplications, normalization, and related neural-network operations.^[16] Framework placement is not always automatic or optimal: TensorFlow documents explicit device placement, memory-allocation settings, and multi-GPU strategies.^[17]

Why CUDA persists, and what competes with it

CUDA's durability is usually attributed to a single feature, but it rests on several that reinforce each other. The first is age: the platform has been shipping since 2006, so nearly two decades of published kernels, tutorials, university courses, and internal codebases assume it. The second is the library layer rather than the language. A team adopting a different platform does not merely need a compiler; it needs equivalents of cuBLAS, cuDNN, CUTLASS, NCCL, cuSPARSELt, the profiling tools, and the inference runtimes, each tuned per architecture. The third is that the frameworks most researchers use were developed against NVIDIA hardware, so new operations tend to appear there first and to be tuned there most. The fourth is availability: NVIDIA parts are offered by every major cloud, which makes them the default target for anything intended to run in more than one place. NVIDIA reported data-center revenue of $197.3 billion within total revenue of $215.9 billion for fiscal 2026, a scale that funds continuous software investment.^[57]

None of that makes the platform technically unassailable, and the alternatives have narrowed the functional gap considerably.

AMD's ROCm is the most direct competitor. Descriptions of it as an immature stack date from the late 2010s and are no longer accurate as a blanket statement. AMD's own product documentation for the MI350 series lists support for PyTorch, TensorFlow, JAX, ONNX Runtime, Kokkos, Raja, SGLang, Triton, and vLLM, and states that ROCm delivers day-zero support for new models through collaborations with model developers.^[33] ROCm 7.0 shipped in 2025 with support for the MI350X and MI355X and OCP FP8 formats, and point releases have continued through the 7.x series.^[49] AMD submitted MI355X results in both the closed and open divisions of MLPerf Inference v6.0 in April 2026.^[58] The remaining differences reported by practitioners are less about whether frameworks run and more about breadth: the number of hardware generations supported at once, the maturity of profiling and debugging tools, coverage of less common operations, and the depth of the third-party kernel ecosystem. Those are real costs, and they are narrower than they were.

Three other approaches attack the problem from different angles. Triton, originally developed at OpenAI, is a Python-embedded language for writing tiled kernels in which the compiler handles the mapping to threads, memory hierarchy, and matrix units; the design goal stated in the original paper was to let a programmer express blocked algorithms while the compiler handles the intra-block scheduling.^[48] It matters less as a portability layer than as the substrate that machine-generated kernels target. Intel's oneAPI packages SYCL implementations, libraries, and tools for a multi-vendor target set; its governance moved to the UXL Foundation, whose steering members include Arm, Codeplay, Fujitsu, Google Cloud, Imagination, Intel, Qualcomm, and Samsung, and which has a formal liaison with Khronos over the SYCL standard.^[50] AMD's HIP takes the pragmatic route of resembling CUDA closely enough that mechanical translation covers much of a port.^[8]

Compilation stacks

Between a framework and a kernel sits a compiler, and this layer has become the main mechanism by which portability is delivered in practice. Rather than hand-writing a kernel for every operation and every architecture, the compiler fuses operations, chooses tile sizes, and emits code.

PyTorch's torch.compile is the widely deployed example. TorchDynamo captures Python bytecode into a graph without giving up eager-mode flexibility; TorchInductor, the default backend, lowers that graph into Triton for GPUs and C++ with OpenMP for CPUs. Ansel and colleagues reported a geometric mean speedup of 2.27x for inference and 1.41x for training across more than 180 real-world models on an NVIDIA A100, outperforming six other PyTorch compiler backends.^[47] The architectural point is that the performance-critical GPU code is generated, not written, which means a new backend needs a code generator rather than a rewritten kernel library. JAX takes a comparable route through XLA, and Pallas provides a lower-level kernel-authoring path within it.

The limits are worth stating plainly. Compilers do well at fusing elementwise chains and at reducing memory traffic; they do less well at matching a hand-tuned matrix multiply or attention kernel, which is why libraries such as CUTLASS and FlashAttention remain in use underneath. Compilation also costs time at startup and can produce different numerics than the eager path, which interacts with the reproducibility issues discussed below.

GPUs in artificial intelligence

Why neural networks map to GPUs

Training and running a neural network repeatedly applies tensor operations to batches of data. Dense layers and attention contain matrix multiplications. A convolutional neural network applies filters over many spatial positions and channels. These operations have regular structure and can perform many independent multiply-accumulate operations, which makes them suitable for parallel hardware.

The mapping is not automatic. Matrix multiplication is most efficient when dimensions are large enough to create many thread blocks and when each value loaded from memory is reused for many arithmetic operations. Small matrices, unfavorable dimensions, transposes, sparse access patterns, and frequent synchronization can lower utilization. NVIDIA's deep-learning performance guide uses arithmetic intensity and tile reuse to explain why matrix shape and batch size affect realized throughput.^[18]

AI frameworks lower high-level operations into kernels or library calls. A single model layer can invoke several kernels for arithmetic, reductions, data layout, and activation functions. Compiler systems may fuse some of them to reduce memory traffic. End-to-end runtime includes all of these operations, not only the fastest matrix multiplication.

Precision in practice

The formats themselves are described under architecture above. What matters at the model level is which parts of a computation each format is applied to, and how the resulting accuracy is verified.

Mixed-precision training combines lower-precision storage or arithmetic with selected higher-precision operations. Micikevicius and colleagues described training with FP16 weights, activations, and gradients while retaining an FP32 master copy of weights, using loss scaling to preserve small gradients, and accumulating selected operations in FP32.^[15] These techniques addressed the numerical behavior of the models and hardware studied; they do not mean that every network trains correctly after a global conversion to FP16. The same caution applies one step down: FP8 training works, but it works because the scaling factors are managed, not because the format is inherently sufficient.

Quantized inference similarly requires measurement against an accuracy target. Lower precision can reduce memory traffic and increase throughput, but calibration error, unsupported operators, conversions between formats, or small batches can reduce the benefit. Peak low-precision operations per second are therefore not a substitute for an end-to-end benchmark. A common and defensible pattern in deployed systems is to quantize the weights aggressively, because they dominate the bytes read, while keeping activations, accumulations, and numerically sensitive layers such as normalization and the output projection at higher precision.

Training

Deep learning training includes forward evaluation, gradient calculation, optimizer updates, data input, and often checkpointing. GPU memory must accommodate model parameters, activations retained for backpropagation, gradients, optimizer state, temporary workspaces, and framework overhead. The largest model that fits in memory is not necessarily the one with the highest throughput.

Larger batches can increase parallelism and arithmetic intensity, but batch size also affects optimization and statistical efficiency. Reporting only examples per second can be misleading if different systems reach different model quality. MLPerf Training was designed around end-to-end time to reach a defined quality target on specified workloads, with rules intended to make system comparisons more meaningful.^[19]

For models that do not fit on one device, training can combine several parallelization methods:

• data parallelism replicates a model and assigns different examples to each worker, then combines gradients;
• tensor parallelism divides operations or tensors within a layer among devices;
• pipeline parallelism places different layer groups on different devices and passes microbatches through the resulting stages; and
• state sharding divides parameters, gradients, or optimizer state among workers.

Megatron-LM research combined data, tensor, and pipeline parallelism and documented that their interactions affect communication, idle time, memory use, and kernel efficiency.^[20] GPipe showed how microbatching can keep pipeline stages active, but also illustrated the idle "bubble" at the beginning and end of a pipeline schedule.^[21] ZeRO partitions model states to reduce per-device memory use in data-parallel training.^[22] Later work added expert parallelism for mixture-of-experts models and context parallelism for long sequences; FSDP is the sharding implementation built into PyTorch, and gradient checkpointing trades recomputation for activation memory.

Which method to use is largely determined by which resource is scarce, and the choices interact with the interconnect. Tensor parallelism produces the heaviest and most latency-sensitive traffic, because devices exchange activations several times per layer; it is normally confined to devices sharing a fast scale-up link. Pipeline parallelism produces less traffic but introduces idle bubbles. Data parallelism produces one large gradient exchange per step, which tolerates a slower network but scales in message size with the model.

Scaling is not linear. Workers exchange gradients, activations, parameters, or optimizer state, and synchronization can leave devices idle. NCCL implements collective operations such as all-reduce, all-gather, reduce-scatter, and broadcast for NVIDIA systems.^[23] Performance depends on message sizes, topology, interconnect bandwidth, software, and the balance of computation among workers. A speedup reported on one cluster size and model is not a universal property of the GPU.

Inference

Inference has different objectives from training. An interactive service may prioritize latency for one request, while an offline service may maximize throughput by batching many requests. A model can also be limited by memory capacity or bandwidth rather than arithmetic.

Autoregressive large language model inference illustrates the distinction. Processing an input prompt can expose parallel work across input tokens. Generating later tokens is sequential across decoding steps for each sequence, although batching supplies parallel work across sequences. Weights and the attention key-value cache must be read or retained, so memory traffic and capacity can dominate. Increasing batch size can improve throughput while increasing waiting time and memory use. The next section works through why.

Reported inference rates are meaningful only with context: model and precision, batch size, input and output lengths, latency statistic, sampling method, hardware count, serving software, and accuracy or quality checks. A tokens-per-second value from one setup cannot be transferred to another without those conditions.

Memory bandwidth and the economics of serving

This section is the practical core of the article. For most current AI inference, the quantity that determines speed and cost is not how many floating-point operations a GPU can perform but how many bytes it can read per second, and how many useful operations it performs per byte read.

Arithmetic intensity and machine balance

Arithmetic intensity, also called operational intensity, is the number of arithmetic operations a kernel performs per byte transferred from device memory. Machine balance is the corresponding property of the hardware: peak arithmetic throughput divided by peak memory bandwidth, expressed in operations per byte. A kernel whose intensity is below the machine balance cannot reach peak arithmetic throughput no matter how it is written, because it will finish its arithmetic before the memory system can deliver the next operands. This is the Roofline framework applied to a specific device.^[11]

Machine balance has been rising for decades, and this is the underlying reason memory has become the limit. Gholami and colleagues quantified the divergence: over twenty years, peak server hardware FLOPS scaled at 3.0x every two years, while DRAM bandwidth scaled at 1.6x and interconnect bandwidth at 1.4x over the same interval.^[38] The compounded gap is the "memory wall," and it means each new generation demands more data reuse from software just to stay in the same regime.

The following table computes machine balance at BF16 or FP16 dense matrix throughput, which is the fairest common basis across vendors.

Scroll sideways for more →

All matrix figures above are dense. The NVIDIA values for Ampere and later are derived from the published tables by halving the sparsity-assisted numbers, as those tables instruct.^[28][32] Two observations follow. First, machine balance roughly doubled between the V100 and the H100, meaning a kernel that was marginally compute bound on the older part is memory bound on the newer one without any change to the software. Second, the H200 row is the exception that proves the rule: it is the only entry whose balance moved down, because it is an H100 with more and faster memory and identical arithmetic.

At narrower formats the picture worsens. Using FP8, where the weights occupy one byte instead of two, the H100's dense balance is roughly 591 operations per byte and the GB200's roughly 625. Every step down in precision doubles the arithmetic and halves the bytes for the same operand count, so it moves the balance point further away.

Why decoding is bandwidth bound

Consider a single linear layer in a transformer with N weight values stored at b bytes each, evaluated for a batch of B sequences during autoregressive generation. Each sequence contributes one token per step. The layer performs about 2NB floating-point operations and must read about Nb bytes of weights. Its arithmetic intensity is therefore about 2B/b operations per byte, independent of the layer's size.

At BF16 and batch size 1, that is roughly 1 operation per byte, against a machine balance of about 295 on an H100. The device is idle for well over 99 percent of its arithmetic capacity, not because the software is poor but because there is nothing else to do with the weights once they arrive. Reaching the balance point requires a batch of a few hundred concurrent sequences. NVIDIA's own guidance states the conclusion directly: during decoding, "the speed at which the data (weights, keys, values, activations) is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens."^[41]

The consequence can be stated as a hard floor. A dense 70-billion-parameter model with 8-bit weights requires roughly 70 GB of reads per decoding step. On a device with 8 TB/s of memory bandwidth, that is about 9 milliseconds per step, or about 110 tokens per second per sequence, before any arithmetic, attention traffic, kernel overhead, or bandwidth inefficiency is counted. No amount of additional floating-point capability lowers that number. More bandwidth, fewer bytes per weight, fewer weights read per token, or more sequences sharing each read are the only levers.

Prefill behaves differently. Processing a prompt multiplies the weight matrix by many token vectors at once, so intensity scales with the number of tokens in flight and the phase is normally compute bound. This asymmetry is why serving systems increasingly separate prefill and decode onto different resources, and why reported "tokens per second" figures are ambiguous unless the phase and the batch composition are stated. Pope and colleagues measured both ends of this range on the same system, reporting 29 milliseconds per token at low batch size and 76 percent model FLOPS utilization during large-batch processing.^[42]

Mixture-of-experts architectures change the arithmetic by reading only the routed experts for each token. The GPT-OSS 120B model used in MLPerf Inference v6.0, for example, has about 117 billion total parameters with about 5.1 billion active per token.^[58] That reduces bytes read per token substantially, at the cost of higher memory capacity requirements and routing-dependent traffic patterns that complicate batching.

The key-value cache

Attention adds a second and growing source of memory traffic. Each generated token attends to the keys and values of all preceding tokens, and recomputing them every step would be wasteful, so they are cached. NVIDIA gives the per-token cache size as 2 x layers x (heads x head dimension) x bytes per element, and the total as batch size x sequence length x 2 x layers x hidden size x element size. For Llama 2 7B at batch size 1 and 4,096 tokens in FP16, that is about 2 GB.^[41]

Three properties of that cache drive serving cost.

It grows linearly in both batch size and context length. Doubling the batch to improve weight amortization also doubles the cache, and long contexts multiply it further, so capacity becomes the binding constraint on how large a batch a device can hold.

It does not amortize across the batch. Weights are shared by every sequence in a batch, so reading them once serves all of them. Each sequence has its own cache, so the attention phase reads roughly the same bytes per sequence regardless of batch size and stays bandwidth bound at any batch size. This is the reason for architectural responses such as multi-query and grouped-query attention, which shrink the cache by sharing key and value heads across query heads. Pope and colleagues showed that the smaller cache of multi-query attention permits far longer contexts at the same memory budget.^[42]

It fragments. Naive allocation reserves contiguous memory for the maximum possible sequence length, wasting most of it. The vLLM system applied operating-system paging ideas to the cache, allocating it in fixed-size blocks that need not be contiguous and can be shared across requests, and reported 2x to 4x throughput improvements over FasterTransformer and Orca at comparable latency, with the largest gains on longer sequences and larger models.^[40] See PagedAttention and continuous batching.

Attention itself was also memory bound for a separate reason: standard implementations materialized the full attention score matrix in device memory. FlashAttention restructured the computation to tile it through on-chip SRAM, avoiding those reads and writes entirely, and reported a 3x speedup on GPT-2 at sequence length 1K, 2.4x on long-range arena tasks, and a 15 percent end-to-end improvement on BERT-large training relative to the MLPerf 1.1 record.^[39] It is an exact algorithm; the gain came entirely from moving fewer bytes.

Compute-rich and bandwidth-starved

The two regimes can coexist within the same request. A serving system may run prefill at high arithmetic utilization and then spend most of its wall-clock time in decode at a small fraction of the same device's floating-point capability. Averaged over a workload, a GPU can therefore report high occupancy and busy schedulers while achieving single-digit percentages of peak FLOPS, and both measurements are correct.

The clearest available demonstration is the H100 to H200 comparison. The two parts have identical Hopper compute, listing the same tensor throughput to the digit in every format. The H200 has 141 GB of HBM3E at 4.8 TB/s against the H100's 80 GB of HBM3 at 3.35 TB/s. NVIDIA states that the H200 delivers 1.9 times the inference performance of the H100 on Llama 2 70B and 1.6 times on GPT-3 175B.^[44] A memory upgrade with no additional arithmetic bought most of a generational improvement in inference.

The practical implications for anyone sizing or buying hardware are direct. For interactive serving of a dense model, compare devices on memory bandwidth and capacity before comparing them on peak FLOPS. Expect throughput per device to improve steeply with batch size until the cache exhausts memory, and expect per-request latency to degrade at the same time. Treat the weight footprint, not the parameter count, as the cost driver, because quantizing weights reduces bytes read proportionally. Measure at the target latency budget, since the achievable batch size is set by the latency target and the achievable throughput is set by the batch size.

Interconnect and scale-out

Once a workload exceeds one device, the network becomes part of the processor. Modern AI systems use two distinct networks with different technologies and purposes: a scale-up fabric that connects a modest number of GPUs tightly enough to be treated almost as one device, and a scale-out network that connects those groups into a cluster.

Scale-up: NVLink, NVSwitch, and Infinity Fabric

NVLink is NVIDIA's point-to-point GPU interconnect. Its bandwidth per GPU has roughly doubled each generation.

Scroll sideways for more →

Source: NVIDIA.^[45] For comparison, a PCIe Gen5 x16 slot provides 128 GB/s bidirectionally,^[28] so fifth-generation NVLink carries roughly fourteen times as much traffic per GPU as the host bus it sits beside. That ratio is the reason tensor-parallel work is placed inside an NVLink domain and almost never across a PCIe boundary.

NVSwitch extends the point-to-point links into an all-to-all domain. NVIDIA states that the Hopper-generation switch supported 8-GPU domains at 7.2 TB/s aggregate, the Blackwell generation supports 72-GPU domains at 130 TB/s, and the Rubin generation supports 72-GPU domains at 260 TB/s.^[45] Within such a domain every GPU can reach every other at full link bandwidth, which is what allows a group of GPUs to be programmed as if it held one large pool of memory.

AMD's equivalent scale-up fabric is Infinity Fabric. An MI350X module exposes seven scale-up links at 144 GB/s each and participates in coherent memory sharing across the eight accelerators on a universal baseboard.^[33] UALink is an industry effort to standardize an open scale-up interconnect for accelerators, positioned against the proprietary alternatives.

Scale-out: InfiniBand and Ethernet

Between nodes, AI clusters have historically used InfiniBand, which provides remote direct memory access, credit-based flow control, and adaptive routing well suited to the bursty all-reduce traffic of distributed training. Ethernet with RDMA over Converged Ethernet has been the alternative, traditionally weaker on lossless behaviour and congestion control but cheaper, more familiar to network operators, and supplied by more vendors.

That gap narrowed with the Ultra Ethernet Consortium, which published Specification 1.0 on 11 June 2025: a full communication stack over Ethernet, running to more than 560 pages, aimed specifically at AI and high-performance computing traffic patterns.^[46] AMD describes itself as a founding member of the consortium and states that it builds its accelerators for Ethernet-based AI networking.^[33] NVIDIA, which acquired the leading InfiniBand vendor in 2020, now sells both an InfiniBand line and an Ethernet line for AI back-end networks. Published market-share estimates for the two technologies come from analyst firms rather than from vendor disclosures and should be treated accordingly.

The reason the choice matters is that distributed training performance is governed by collective operations, not by point-to-point transfers. An all-reduce over a large gradient tensor completes only when its slowest participant does, so tail latency and congestion behaviour affect step time more than headline link speed. See InfiniBand, Ethernet, Ultra Ethernet, and NVIDIA Spectrum-X.

The rack as the unit of purchase

The combination of large models, tensor and expert parallelism, and a fast scale-up fabric has changed what is actually bought. A single GPU is rarely the meaningful unit for frontier work; the rack is.

NVIDIA's GB200 NVL72 places 36 Grace CPUs and 72 Blackwell GPUs in one liquid-cooled rack, presenting 13.4 TB of HBM3E at 576 TB/s aggregate within a 130 TB/s NVLink domain. NVIDIA lists the rack's NVFP4 tensor throughput as 1,440 PFLOPS with sparsity and 720 PFLOPS dense, and its FP16/BF16 throughput as 360 PFLOPS with sparsity.^[32] The Rubin-generation successor keeps the 72-GPU domain and doubles the NVLink aggregate to 260 TB/s.^[45] AMD announced a comparable rack-scale system, Helios, alongside the Instinct MI400 series at its Advancing AI event on 23 July 2026.^[60]

Buying at rack granularity has consequences beyond procurement. Power density rises to the point that liquid cooling becomes a requirement rather than an option, which constrains which buildings can host the equipment. The International Energy Agency identifies accelerated servers as an important driver of rising power density and energy use in AI-oriented data centres.^[26] The unit of failure also grows: a fault that removes a rack removes an entire NVLink domain, so checkpointing intervals and job restart behaviour have to be designed around it. See AI data center and GPU cluster.

Performance analysis

Parallelism is necessary but not sufficient

A GPU benefits from a workload only when useful parallel work outweighs launch, transfer, synchronization, and scheduling costs. In a historical comparison of 14 throughput-oriented applications, Lee and colleagues optimized both an Intel CPU and an NVIDIA GPU and reported an average GPU speedup of 2.5 times, while analyzing cases that favored each architecture.^[10] The exact result is tied to 2010 hardware and software. Its lasting lesson is that claims of universal 10-fold, 100-fold, or greater GPU acceleration are not valid without a controlled workload-specific comparison.

A fair CPU-GPU comparison should use optimized implementations on both platforms, include data-transfer costs when the application requires them, hold output quality constant, and report the complete hardware and software configuration. Comparing a tuned GPU library with an untuned single-threaded CPU program measures implementation choices as well as processors.

Roofline model

The Roofline model relates a kernel's performance to peak arithmetic throughput, memory bandwidth, and operational intensity, defined as operations performed per byte transferred from main memory. Its upper bound is the lesser of peak compute performance and memory bandwidth multiplied by operational intensity.^[11]

This model distinguishes two broad regimes. A low-intensity kernel is memory-bound under the model, so additional arithmetic units alone do not increase its ceiling. A high-intensity kernel may be compute-bound. Real performance remains below the ceiling because of instruction dependencies, cache behavior, communication, launch overhead, divergence, and other limits. The model is a diagnostic framework, not a prediction that every kernel reaches the roof.

The section on memory bandwidth above applies this model to specific devices and to the phases of language-model inference; the machine balance figures there are the ridge points of each device's roofline.

Interpreting specifications

GPU specifications commonly list:

• memory capacity and bandwidth;
• supported numerical formats;
• peak arithmetic rates for selected formats;
• thermal design power;
• host and device interconnects; and
• software and virtualization support.

These quantities answer different questions. Memory capacity determines whether a working set fits, while bandwidth limits how quickly data can be moved. Peak arithmetic rate assumes a particular instruction mix and enough independent work. Some rates count a fused multiply-add as two operations. Some quote structured-sparsity throughput that requires a supported sparsity pattern. Thermal design power is a design specification, not a direct measurement of energy for a job.

A short checklist covers most of the ways a comparison goes wrong:

1. Which numerical format does the figure describe? A number without a format is not a number.
2. Is it dense or sparsity-assisted? If the same table shows one value that is exactly double another for the same format, the larger is the sparse one.
3. Is it per device, per board, or per rack? Eight-GPU boards and 72-GPU racks are quoted in the same units as single chips.
4. Does the memory figure describe capacity or bandwidth, and is it per device or aggregated?
5. Is the interconnect figure unidirectional or bidirectional?
6. Is the comparison baseline the previous generation at the same precision, or at a different one? A generational multiple that combines a precision change with a hardware change describes both.

For procurement or scientific reporting, benchmark the intended model, dataset, precision, batch or request distribution, quality target, and complete software stack. Include warmup, measurement duration, error bars or repeated runs where relevant, and the energy boundary if energy is reported.

Benchmarks

MLPerf, maintained by MLCommons, is the closest available approximation to a controlled cross-vendor comparison, because submissions must meet a defined quality target on a defined workload and are reviewed by other submitters. MLPerf Inference v6.0, published on 1 April 2026, drew submissions from 24 organizations including AMD, Google, Intel, NVIDIA, and several cloud providers, and added benchmarks based on the open-weight GPT-OSS 120B model, text-to-video generation, DLRMv3, and vision-language models.^[58]

The benchmark's limitations should be understood alongside its value. Closed-division results constrain the model and the optimizations, which aids comparability but understates what a tuned deployment can achieve; open-division results allow more latitude and are correspondingly less comparable. Submissions reflect vendor engineering effort as well as hardware, and the set of submitters is self-selected. A benchmark result is evidence about a configuration, not a property of a chip.

Limitations and risks

Irregular and serial workloads

GPUs are less suitable when a task contains little parallel work, unpredictable pointer chasing, frequent branching within thread groups, or fine-grained synchronization. The CPU may also be preferable when a calculation is small enough that launch and transfer overhead dominate. Hybrid applications often keep control-flow and preprocessing work on CPUs while using GPUs for dense kernels.

Memory and communication

Accelerator memory is finite. Models that exceed it require recomputation, offloading, sharding, or lower-precision storage, each of which changes runtime and complexity. Multi-GPU execution adds collective communication and topology effects. Adding devices can make a poorly partitioned workload slower.

Numerical behavior and reproducibility

Floating-point addition and multiplication are not generally associative. Parallel reductions can combine values in a different order from a CPU implementation and produce different rounding results.^[25] Frameworks may also choose different algorithms on different devices or software versions. PyTorch states that completely reproducible results are not guaranteed across releases, platforms, or CPU and GPU execution, and notes that deterministic operations can be slower.^[24]

Differences at the last bits are not automatically errors, but they can affect numerically unstable algorithms. Validation should use tolerances and task-level quality measures chosen for the application rather than require bitwise equality without justification. Narrow formats widen the gap: two systems running the same model at FP4 with different block-scaling implementations can produce visibly different outputs while both being correct implementations of their respective specifications.

Energy, cooling, and utilization

High-performance GPUs can create dense electrical and cooling loads. The International Energy Agency identifies accelerated servers as an important driver of rising power density and energy use in AI-oriented data centers.^[26] A GPU's rated power is not the energy consumed by an entire training run or data center. Facility cooling, CPUs, memory, networking, storage, utilization, and job duration also contribute.

Board power has risen sharply across generations: the A100 SXM is rated at 400 W, the H100 SXM at up to 700 W, and the AMD MI350X at a 1,000 W maximum total board power.^[29][28][33] At those densities air cooling becomes impractical for dense configurations, which is why rack-scale systems are liquid cooled.

Energy comparisons should specify the measurement boundary and output achieved. A device with higher instantaneous power can use less total energy if it completes equivalent work much sooner, while an underutilized accelerator can waste capacity. Carbon impact additionally depends on location, time, and electricity supply.

Software dependence

Real performance depends on compilers, kernels, libraries, drivers, and framework support. Vendor-specific APIs can provide mature optimization but increase switching costs. Cross-platform source code can still need architecture-specific tuning. Long-term deployments should evaluate software maintenance, supported numerical formats, debugging and profiling tools, and availability of tested libraries alongside hardware specifications.

Supply, cost, and policy

Manufacturing and packaging

An AI GPU depends on three separate supply chains that must arrive together: leading-edge logic wafers, HBM stacks, and advanced packaging that joins them. Any one of the three can gate output, and the packaging step in particular has been repeatedly identified by vendors and foundries as a constraint, because 2.5D interposer capacity cannot be added as quickly as wafer starts. HBM has three qualified suppliers, SK hynix, Samsung, and Micron, and its availability has tracked closely with accelerator availability.

The commercial scale involved is unusual for a component category. NVIDIA reported fiscal 2026 revenue of $215.9 billion, of which $197.3 billion came from its data-center segment.^[57]

Export controls and market segmentation

Since 2022 the United States has restricted exports of advanced computing chips through the Bureau of Industry and Security. The control is keyed to a calculated metric rather than a product name. Total Processing Performance is defined as 2 x MacTOPS x the bit length of the operation, aggregated across the chip, where MacTOPS is the theoretical peak multiply-accumulate rate in tera-operations per second; a companion metric, performance density, divides TPP by die area. Data-center chips at TPP of 4,800 or above, or at TPP of 1,600 or above with performance density of 5.92 or above, fall under export control classification number 3A090.a.^[55]

Because the threshold is a formula, vendors can and do design variants that sit beneath it, which produced a segmented market with distinct product lines for restricted destinations, such as NVIDIA's H20.

The policy has continued to move. On 15 January 2026 BIS revised its licensing posture for China and Macau from a presumption of denial to case-by-case review for chips below a TPP of 21,000 and total DRAM bandwidth below 6,500 GB/s, naming the NVIDIA H200 and AMD MI325X. Approval is conditioned on the chip being commercially available in the United States, sufficient domestic supply, a cap limiting shipments to China and Macau to no more than half of a customer's United States shipments, know-your-customer procedures by the consignee, independent United States laboratory testing before each shipment, and restrictions on remote access through infrastructure-as-a-service. Reexports from third countries and exports to other embargoed destinations remain prohibited.^[54]

The practical result is a market with at least three tiers: unrestricted destinations, license-eligible destinations subject to conditions, and prohibited destinations. That segmentation affects which hardware is available where, at what price, and consequently which models can be trained or served in which jurisdictions. See export controls and Entity List.

Renting compared with owning

Both models are in wide use, and the choice turns on utilization, holding period, and who bears obsolescence risk.

Ownership provides control over scheduling, data locality, and the software stack, and converts a variable cost into a fixed one, which is attractive at high sustained utilization. It requires securing power, cooling, space, and network capacity, and it exposes the owner to the residual value of the hardware. Renting from a cloud provider or a specialized GPU operator transfers those risks at a markup, allows capacity to track demand, and gives access to a generation before a buyer could take delivery. Published rates vary widely by provider, region, generation, and commitment length, and a spot price observed on one day is not a cost basis.

The obsolescence question has a clear public data point. Amazon disclosed that effective 1 January 2025 it shortened the estimated useful life of a subset of its servers and networking equipment from six years to five, citing the increased pace of technology development particularly in artificial intelligence and machine learning, and anticipated a resulting decrease in 2025 operating income of approximately $0.7 billion.^[56] That reversed an earlier extension and is the most direct public statement by a large operator that AI hardware is aging faster than previously assumed.

Older hardware does not become worthless when a new generation ships; it moves down the workload stack toward smaller models, fine-tuning, batch inference, and development work. But the value of a given device is tied to its memory bandwidth and capacity relative to the models people want to run, which is why parts with generous memory retain utility longer than their FLOPS ratings would suggest. See neocloud, CoreWeave, Lambda Labs, and NVIDIA DGX Cloud.

Alternatives to GPUs

A GPU is a general-purpose parallel processor that happens to suit AI workloads well. Several classes of hardware trade generality for efficiency on a narrower target.

Google's tensor processing unit is the longest-running alternative in production. It has no graphics hardware and organizes its matrix arithmetic as a systolic array, an arrangement in which operands flow through a grid of multiply-accumulate cells so that each value is reused across many cells without returning to a register file. Jouppi and colleagues described the first-generation design, a 256 by 256 array of 8-bit multiply-accumulate units built for inference.^[51] The seventh-generation TPU7x, called Ironwood, was announced in April 2025 and became generally available on Google Cloud in November 2025; Google lists 2,307 TFLOPS of BF16 and 4,614 TFLOPS of FP8 per chip, 192 GB of HBM3E at 7.37 TB/s, 1.2 TB/s of inter-chip interconnect per chip, and pods of up to 9,216 chips.^[52][53] Google publishes single values with no sparsity qualifier, so those are dense figures and are directly comparable with the dense GPU numbers above. The design differences from a GPU are real but narrower than they once were: both now pair a large matrix engine with HBM and a dedicated scale-up network, and both face the same machine balance problem.

Scroll sideways for more →

Two cautions apply to every comparison in that table. First, an accelerator built around holding weights in SRAM buys latency by giving up capacity per device, so the comparison changes entirely with model size; a design that is dramatically faster on a 20-billion-parameter model may require many more devices for a 400-billion-parameter one. Second, hardware specialized for one model architecture carries architecture risk. The shift toward mixture-of-experts routing and long-context attention changed the memory access pattern of frontier models within a few years, and hardware fixed to the previous pattern loses its advantage.

The general rule is that the more precisely a workload is known and the longer it stays fixed, the more a specialized device can win; the more the workload is expected to change, the more the GPU's generality is worth paying for.

Selecting a processor

The relevant question is not whether GPUs are generally better, but whether a particular system satisfies a particular workload. Important questions include:

1. Does the computation expose enough parallel work at the required request size?
2. Does the model and its temporary state fit in memory?
3. Is performance limited by arithmetic, memory bandwidth, communication, or latency?
4. Are the required operations and numerical formats well supported?
5. Does the software stack meet portability, maintenance, and reproducibility needs?
6. What end-to-end quality, latency, throughput, energy, and cost does the system achieve?

For inference workloads specifically, question 3 usually resolves to memory bandwidth, and a shorter path to an answer is available: estimate the bytes that must be read per token, divide by the device's memory bandwidth to get a floor on step time, compare that floor with the latency target, and derive the batch size that the remaining headroom permits. If the required batch size exceeds what memory capacity allows once the key-value cache is counted, the device is too small regardless of its arithmetic rating.

For training, the corresponding questions are whether the model state fits under the chosen sharding scheme, whether the parallelization strategy's communication fits inside the available scale-up domain, and what the resulting step time and time-to-quality are.

A benchmark should match the deployment objective. Training requires time to a defined quality target. Interactive inference requires a latency distribution at a specified load. Offline inference emphasizes completed work per unit time. Scientific applications may prioritize numerical error and reproducibility. The same GPU can rank differently under each objective.

See also

• AI chip
• AI accelerator
• Tensor Processing Unit
• Tensor core
• High-bandwidth memory
• NVIDIA Volta
• NVIDIA Hopper
• NVIDIA Blackwell
• NVLink
• InfiniBand
• HGX
• ROCm
• Quantization

References

1. ^John D. Owens, David Luebke, Naga Govindaraju, Mark Harris, Jens Kruger, Aaron E. Lefohn, and Timothy J. Purcell. "A Survey of General-Purpose Computation on Graphics Hardware." *Eurographics 2005, State of the Art Reports*, 2005, pp. 21-51. research.nvidia.com/...mputation-graphics-hardware
2. ^John Nickolls and William J. Dally. "The GPU Computing Era." IEEE Micro, 2010. pages.cs.wisc.edu/...ieeemicro10_gpu.pdf
3. ^NVIDIA. "NVIDIA Unveils the GeForce 256, the World's First GPU." 1999. nvidia.com/...time_99
4. ^NVIDIA Technical Blog. "CUDA Refresher: Getting Started with CUDA." 2020. developer.nvidia.com/...-getting-started-with-cuda
5. ^Khronos Group. "OpenCL 1.0 Released." 2008. khronos.org/...opencl_1.0_released
6. ^NVIDIA. "CUDA Programming Guide." docs.nvidia.com/...cuda-programming-guide
7. ^Intel. "Intel Xe GPU Architecture." oneAPI GPU Optimization Guide. intel.com/...intel-xe-gpu-architecture
8. ^AMD. "HIP Programming Model." ROCm Documentation. rocm.docs.amd.com/...programming_model
9. ^Khronos Group. "SYCL." khronos.org/sycl
10. ^Victor W. Lee et al. "Debunking the 100X GPU vs. CPU Myth: An Evaluation of Throughput Computing on CPU and GPU." ISCA, 2010. cs.umd.edu/...Lee-GPU.pdf
11. ^Samuel Williams, Andrew Waterman, and David Patterson. "Roofline: An Insightful Visual Performance Model for Multicore Architectures." 2008. www2.eecs.berkeley.edu/...EECS-2008-134.pdf
12. ^NVIDIA. "NVIDIA Tesla V100 GPU Architecture." 2017. nvidia.com/...Volta-Architecture-Whitepaper-v1.0.pdf
13. ^AMD. "Hardware implementation." HIP Documentation. rocm.docs.amd.com/...hardware_implementation
14. ^Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. "ImageNet Classification with Deep Convolutional Neural Networks." NeurIPS, 2012. proceedings.neurips.cc/...436e924a68c45b-Paper.pdf
15. ^Paulius Micikevicius et al. "Mixed Precision Training." ICLR, 2018. arxiv.org/...1710.03740
16. ^NVIDIA. "cuDNN Documentation." docs.nvidia.com/...latest
17. ^TensorFlow. "Use a GPU." tensorflow.org/...gpu
18. ^NVIDIA. "Deep Learning Performance Guide: Matrix Multiplication." docs.nvidia.com/...rformance-matrix-multiplication
19. ^Peter Mattson et al. "MLPerf Training Benchmark." Proceedings of Machine Learning and Systems, 2020. proceedings.mlsys.org/...f25efb8912945f7-Paper.pdf
20. ^Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM." *SC '21: International Conference for High Performance Computing, Networking, Storage and Analysis*, 2021, article 58, pp. 1-15. doi.org/...3458817.3476209
21. ^Yanping Huang et al. "GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism." NeurIPS, 2019. proceedings.neurips.cc/...6b1c5722a46aa2-Paper.pdf
22. ^Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models." *SC20: International Conference for High Performance Computing, Networking, Storage and Analysis*, 2020, pp. 1-16. doi.org/...SC41405.2020.00024
23. ^NVIDIA. "Collective Operations." NCCL User Guide. docs.nvidia.com/...collectives
24. ^PyTorch. "Reproducibility." docs.pytorch.org/...randomness
25. ^NVIDIA. "Floating Point and IEEE 754 Compliance for NVIDIA GPUs." docs.nvidia.com/...floating-point
26. ^International Energy Agency. "Energy demand from AI." iea.org/...energy-demand-from-ai
27. ^NVIDIA. "CUDA C++ Best Practices Guide." docs.nvidia.com/...cuda-c-best-practices-guide
28. ^NVIDIA. "NVIDIA H100 Tensor Core GPU Datasheet." Specification table footnote: "Shown with sparsity. Specifications 1/2 lower without sparsity." resources.nvidia.com/...nvidia-h100-tensor-c
29. ^NVIDIA. "NVIDIA A100 Tensor Core GPU Datasheet." November 2020. System specifications footnote: "With sparsity." nvidia.com/...update-a4-nvidia-1485612-r12-web.pdf
30. ^Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. "Accelerating Sparse Deep Neural Networks." arXiv:2104.08378, 16 April 2021. arxiv.org/...2104.08378
31. ^NVIDIA. "NVIDIA HGX Platform." Specification table footnotes: "Specification in Sparse | Dense" and "Dense is 1/2 sparse spec shown." nvidia.com/...hgx
32. ^NVIDIA. "NVIDIA GB200 NVL72." Specification table footnotes: "Specification in sparse | dense" and "Specification in sparse. Dense is one-half sparse spec shown." nvidia.com/...gb200-nvl72
33. ^AMD. "AMD Instinct MI350X GPU" datasheet, June 2025 (document LE-92701-00 6/25). AI peak theoretical performance table with separate dense and "W/SPARSITY" columns. amd.com/...mi350x
34. ^Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, and Hao Wu. "FP8 Formats for Deep Learning." arXiv:2209.05433, September 2022. arxiv.org/...2209.05433
35. ^Open Compute Project. "OCP Microscaling Formats (MX) Specification Version 1.0." 2023. opencompute.org/...-formats-mx-v1-0-spec-final-pdf
36. ^NVIDIA Technical Blog. "Introducing NVFP4 for Efficient and Accurate Low-Precision Inference." developer.nvidia.com/...te-low-precision-inference
37. ^NVIDIA Technical Blog. "Accelerating AI Training with NVIDIA TF32 Tensor Cores." developer.nvidia.com/...ing-with-tf32-tensor-cores
38. ^Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, and Kurt Keutzer. "AI and Memory Wall." *IEEE Micro*, vol. 44, 2024, pp. 33-39. arxiv.org/...2403.14123
39. ^Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Re. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." NeurIPS, 2022. arxiv.org/...2205.14135
40. ^Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP, 2023. arxiv.org/...2309.06180
41. ^NVIDIA Technical Blog. "Mastering LLM Techniques: Inference Optimization." developer.nvidia.com/...ues-inference-optimization
42. ^Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. "Efficiently Scaling Transformer Inference." arXiv:2211.05102, November 2022. arxiv.org/...2211.05102
43. ^JEDEC. "JEDEC and Industry Leaders Collaborate to Release JESD270-4 HBM4 Standard." 16 April 2025. jedec.org/...ase-jesd270-4-hbm4-standard-advancing
44. ^NVIDIA. "NVIDIA H200 Tensor Core GPU." Specification table; tensor-core figures footnoted "With sparsity." nvidia.com/...h200
45. ^NVIDIA. "NVLink and NVLink Switch." nvidia.com/...nvlink
46. ^Ultra Ethernet Consortium. "Ultra Ethernet Consortium Launches Specification 1.0." 11 June 2025. ultraethernet.org/...ernet-for-ai-and-hpc-at-scale
47. ^Jason Ansel et al. "PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation." *ASPLOS '24*, April 2024. doi.org/...3620665.3640366
48. ^Philippe Tillet, Hsiang-Tsung Kung, and David Cox. "Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations." *MAPL 2019*. doi.org/...3315508.3329973
49. ^AMD. "ROCm 7.0.1 Release Notes." ROCm Documentation. rocm.docs.amd.com/...release-notes
50. ^UXL Foundation. uxlfoundation.org
51. ^Norman P. Jouppi et al. "In-Datacenter Performance Analysis of a Tensor Processing Unit." ISCA, 2017. arxiv.org/...1704.04760
52. ^Google Cloud. "TPU7x (Ironwood)." Cloud TPU documentation. docs.cloud.google.com/...tpu7x
53. ^Google. "Ironwood: The first Google TPU for the age of inference." 9 April 2025. blog.google/...ironwood-tpu-age-of-inference
54. ^Morgan Lewis. "BIS Revises Export Review Policy for Advanced AI Chips Destined for China and Macau." January 2026. morganlewis.com/...ps-destined-for-china-and-macau
55. ^Center for Security and Emerging Technology, Georgetown University. "Explainer: The Commerce Department's October 2023 Export Control Update." cset.georgetown.edu/...bis-2023-update-explainer
56. ^Amazon.com, Inc. Annual Report on Form 10-K for fiscal year 2024, disclosing a change in the estimated useful life of a subset of servers and networking equipment from six years to five, effective 1 January 2025. sec.gov/...browse-edgar
57. ^NVIDIA. "NVIDIA Announces Financial Results for Fourth Quarter and Fiscal 2026." February 2026. nvidianews.nvidia.com/...h-quarter-and-fiscal-2026
58. ^MLCommons. "MLCommons Releases New MLPerf Inference v6.0 Benchmark Results." 1 April 2026. mlcommons.org/...mlperf-inference-v6-0-results
59. ^NVIDIA Technical Blog. "Inside the NVIDIA Vera Rubin Platform: Six New Chips, One AI Supercomputer." 5 January 2026. developer.nvidia.com/...chips-one-ai-supercomputer
60. ^AMD Newsroom. "AAI 2026: AMD Launches AMD Instinct MI400 Series GPUs for Frontier AI, HPC." 23 July 2026. newsroom.amd.com/...aai-2026-mi400-instinct-update
61. ^PyTorch. "Accelerating Neural Network Training with Semi-Structured (2:4) Sparsity." 20 June 2024. pytorch.org/...accelerating-neural-network-training
62. ^NVIDIA. "NVIDIA Blackwell Architecture." nvidia.com/...blackwell-architecture