AI Infrastructure

AI infrastructure is the hardware, facilities, networking, storage, and software used to develop, train, evaluate, deploy, and operate artificial intelligence systems. It includes more than accelerator chips. A production stack may extend from dataset storage and model-development tools through servers, cluster interconnects, schedulers, inference runtimes, monitoring, electrical distribution, and cooling. The relevant boundary depends on the workload: a research cluster, a cloud inference service, and an embedded deployment need different infrastructure.

The layers are coupled. A processor with high peak arithmetic throughput can remain underused if its memory system cannot supply operands, a distributed job can stall behind one slow worker, and an inference service can run out of memory for attention state before it exhausts compute. Facility power and cooling also constrain how much equipment can operate at a site. For these reasons, AI infrastructure is evaluated as a system rather than as a list of chips or data centers.^[1][2]

Scope and layers

An AI chip is one component of AI infrastructure, while an AI accelerator is a processor or device optimized for relevant workloads. A data center supplies the physical environment for many deployments, but not all AI infrastructure is centralized. Workloads also run in enterprise server rooms, laboratories, vehicles, phones, industrial equipment, and other edge systems.

Scroll sideways for more →

This layered view is not a mandatory architecture. Small systems can combine most functions on one machine, while large installations split them into independently operated services. Production machine-learning platforms also contain extensive data, configuration, validation, and monitoring code around the model itself. Research on technical debt in machine learning warns that hidden data dependencies, feedback loops, configuration, and changes in the external world can make this surrounding system more complex than the learning code.^[27][29]

Compute, memory, and node design

AI computation commonly uses GPUs, TPUs, other application-specific processors, CPUs with vector or matrix instructions, or combinations of them. A compute node usually pairs processors with host memory, accelerator memory, storage interfaces, and network adapters. The choice depends on the operations a model performs, the numerical formats it supports, the amount of state that must remain resident, and the cost of moving data.^[1][2]

Deep-learning accelerators devote substantial hardware to dense or sparse tensor operations and use a memory hierarchy to reuse weights, activations, and partial results. Data movement can consume more time and energy than arithmetic, so local memories and dataflows are central architectural choices.^[1] The Roofline model expresses a related limit: attainable performance is bounded either by peak compute or by memory bandwidth multiplied by a workload's operational intensity. A peak FLOPS or operations-per-second figure therefore does not predict application performance by itself.^[34]

Software can change that balance without changing the processor. FlashAttention, for example, tiles exact attention computation so that it performs fewer reads and writes between GPU high-bandwidth memory and on-chip SRAM than a conventional materialization of the full attention matrix.^[6] This is one reason an infrastructure comparison must identify the kernels and software version as well as the device.

Memory requirements differ between lifecycle stages:

Scroll sideways for more →

This distinction matters when sizing a system. Standard data parallelism replicates model state on every worker, while ZeRO partitions optimizer states, gradients, and parameters across data-parallel processes to reduce redundant device-memory use.^[4] In language-model serving, the key-value cache changes size as requests arrive and generate tokens. The vLLM work showed that fragmentation and duplicated cache blocks can restrict batching, and introduced PagedAttention to allocate that state in non-contiguous blocks.^[7]

Numerical precision is another system-level choice. Lower-precision formats can reduce storage, memory traffic, and arithmetic cost, but only if the hardware, kernels, accumulation behavior, and model-quality checks support them. Quantization can also change accuracy and may introduce conversion or dequantization overhead. Valid comparisons state the format and accuracy condition rather than presenting unlike precision modes as interchangeable.^[1][6]

Google's TPU v4 illustrates why a processor generation is also a system design. The published machine combined accelerator chips, CPU hosts, a three-dimensional interconnect, and reconfigurable optical circuit switches. The switches could route around unavailable components and select topologies for different communication patterns.^[2] The example does not establish that one interconnect or accelerator is universally superior; its measured comparisons are tied to the paper's systems and workloads.

Networks and collective communication

Distributed training and inference move tensors at several physical scales. "Scale-up" generally refers to a tightly coupled accelerator domain within a server, rack, or pod. "Scale-out" connects nodes or accelerator domains across a cluster. The boundary is implementation-specific, but the distinction helps separate memory-semantic, low-latency fabrics from routed cluster networks.

Communication libraries expose collective operations that match common parallel algorithms. NCCL, for example, documents operations including AllReduce, Broadcast, Reduce, AllGather, ReduceScatter, and point-to-point send and receive.^[13]

Scroll sideways for more →

The required topology follows the traffic pattern. Backpropagation often relies heavily on reductions, while sparse expert models add all-to-all traffic. The TPU v4 paper notes that these patterns stress a network differently and uses reconfigurable topology to address availability and bisection-bandwidth needs.^[2] UALink 200G 1.0 defines a scale-up accelerator and switch fabric, while Ultra Ethernet Specification 1.0 defines an Ethernet-based stack for scale-out AI and high-performance-computing traffic.^[32][33] These specifications describe interfaces and protocol behavior, not guaranteed application performance.

Network planning accounts for per-link bandwidth, aggregate bisection bandwidth, latency, oversubscription, congestion control, cabling, and the path between each device and its network interface. Placement matters because two allocations with the same number of accelerators can have different topologies. A synchronous job also progresses at the speed of its slowest participating rank.^[2][13] In a 2025 study of a shared production cluster, GREYHOUND's authors observed transient slowdowns caused by contention, device degradation, thermal throttling, and network congestion; their reported frequency and impact apply to that measured environment, not to every cluster.^[12]

Data, storage, and checkpoints

Data infrastructure supplies training and evaluation examples, but capacity alone is not enough. A reproducible pipeline records dataset versions, transformations, filtering, schemas, access rules, and the relationship between source data and generated artifacts. "Datasheets for Datasets" proposed documenting a dataset's motivation, composition, collection process, preprocessing, uses, distribution, and maintenance to make such decisions inspectable.^[25] Production platforms add automated checks for schema, statistics, anomalies, and training-serving skew before data reaches a model.^[29]

Storage is usually tiered. Object or distributed file storage provides durable capacity; local solid-state storage and memory caches reduce repeated remote reads; metadata services track versions and ownership. The correct layout depends on record size, access order, shuffling, compression, and the number of readers. A pipeline that delivers high sequential bandwidth can still perform poorly under small random reads or when thousands of workers request the same shards at once.^[10][29]

Checkpoints serve several purposes:

• restart training after a failure;
• evaluate intermediate states without stopping the main job;
• transfer state from pretraining to later training stages;
• preserve a reproducible model artifact;
• change hardware allocation or parallelism when the checkpoint format permits it.

A training checkpoint may include parameters, optimizer state, progress counters, random-number-generator state, and metadata needed to resume consistently. At large scale, writing this state can pause training or saturate storage. ByteCheckpoint addresses checkpoint representation, resharding, storage backends, and I/O stalls across multiple training configurations.^[10] Universal Checkpointing likewise separates checkpoint structure from a specific parallel strategy so that a job can reload under a different device count or sharding scheme.^[11] These systems demonstrate the problem and evaluated approaches; their reported speedups should not be transferred to unrelated configurations.

Durability policy is a tradeoff. More frequent checkpoints reduce repeated work after a failure but consume more bandwidth and storage. Retention also has to distinguish recoverable training state from release artifacts that need longer preservation, review, and access controls.^[10][11]

Training infrastructure

Large training jobs distribute work because a model or its state does not fit on one device, because one device would take too long, or both. Several forms of parallelism can be combined:

Scroll sideways for more →

Megatron-LM demonstrated intra-layer tensor parallelism and combined it with data parallelism for transformer training.^[3] ZeRO analyzes the memory redundancy of ordinary data parallelism and partitions three classes of model state in stages.^[4] GShard provides a model in which a compiler partitions annotated computations and describes sparse expert routing across devices.^[5] The techniques address different constraints, so "more parallelism" is not a complete optimization goal. A layout can lower memory use while increasing communication, add idle pipeline time, or produce small kernels that use the hardware poorly.

PyTorch separates distributed data-parallel training, fully sharded data parallelism, tensor parallelism, and pipeline parallelism in its distributed overview.^[14] Other frameworks expose similar concepts, but configuration and checkpoint formats are not automatically portable. Infrastructure teams must version the framework, compiler, collective library, drivers, container image, kernels, and model code together.

The scheduler must allocate a compatible set of devices and place them with suitable network connectivity. It may also enforce priorities, quotas, reservations, and preemption. Long synchronous jobs need health checks and a recovery procedure for both fail-stop errors and subtler "fail-slow" behavior. Checkpoint completion, validation, and restore time are therefore operational metrics, not background implementation details.^[10][12][14]

Training performance is meaningful only with a quality condition. MLPerf Training measures wall-clock time to reach a specified quality target for a defined model and dataset; its Closed division constrains implementations more tightly than its Open division.^[17][31] A raw examples-per-second number can be increased by changing batch size, numerical precision, data, or convergence behavior, so those conditions must accompany the result.

Inference infrastructure

Inference systems turn a model artifact into responses under workload-specific service objectives. They load weights, preprocess inputs, schedule work, manage accelerator memory, execute kernels, postprocess outputs, and record telemetry. Online services add routing, authentication, admission control, rate limits, autoscaling, and failure handling. Offline inference emphasizes completed work per unit time, while interactive systems usually impose latency limits.^[7][9][18]

Autoregressive language-model serving has two computationally different phases. The prefill phase processes the input context and creates attention state. The decode phase produces subsequent tokens iteratively while reading the retained key-value cache. DistServe found that co-locating these phases can create interference and evaluated disaggregating them onto separate resources to optimize the number of requests meeting time-to-first-token and time-per-output-token objectives.^[8] The result supports phase-aware planning, but it does not mean disaggregation is best at every load or model size.

Batching improves device use by combining requests, yet fixed request-level batching can leave capacity idle when sequences finish at different times. Orca introduced iteration-level scheduling so a serving system could admit and remove work between generation iterations.^[9] vLLM combined a scheduler with PagedAttention to reduce key-value-cache waste and support larger batches.^[7] These works measure particular models, hardware, and baselines, so their throughput ratios are not general constants.

Useful serving metrics distinguish different reader experiences:

Scroll sideways for more →

MLPerf Inference defines different scenarios, including offline and server-style workloads, and requires accuracy targets alongside performance measurement.^[18][30] Comparisons should match model, dataset, accuracy tier, scenario, software, numerical format, accelerator count, and power boundary. Results from a throughput-oriented offline run cannot be substituted for interactive latency.

Scheduling and platform software

Cluster software connects resource allocation to model execution. Kubernetes schedules Pods by filtering nodes that cannot satisfy requirements and scoring the remaining feasible nodes; plugins can extend the scheduling framework.^[15] Slurm manages resources, partitions, jobs, and job steps for batch-oriented clusters.^[16] Ray provides distributed tasks and stateful actors, and its original systems paper describes a scheduler and object store designed for heterogeneous, dynamic AI workloads.^[28] These tools operate at different abstraction levels and can be combined.

AI workloads add requirements that a generic CPU request does not express:

• accelerator type, count, memory, and supported numerical formats;
• topology and network-interface placement;
• compatible drivers, firmware, runtime, and collective libraries;
• local storage and dataset locality;
• gang allocation for workers that must start together;
• isolation between tenants and control over device access;
• priority, quota, reservation, and preemption policy.

Platform software also manages the model lifecycle. A production pipeline records code and data versions, validates inputs, trains and evaluates candidates, approves artifacts, deploys them, monitors behavior, and supports rollback. TFX is one documented production-scale example with components for data analysis, validation, training, evaluation, serving, and pipeline orchestration.^[29] Model cards propose recording intended use, evaluation procedures, relevant factors, and limitations with a released model.^[26] These records do not prove that a system is safe or suitable, but they make review and comparison more precise.

Observability spans both infrastructure and model behavior. Operators track device errors, temperatures, throttling, memory pressure, network retries, queue depth, storage latency, checkpoint progress, and scheduler state. They also monitor model-specific quality, drift, policy violations, and unexpected use where appropriate. Correlation identifiers and versioned deployment metadata help connect a response to the model, runtime, configuration, and data pipeline that produced it.^[23][24][29]

Facilities, electricity, and cooling

Data-center AI systems depend on an electrical and thermal chain: utility or on-site supply, switchgear and transformers, uninterruptible power systems, rack distribution, server power conversion, cooling equipment, and heat rejection. Redundant paths and backup generation improve continuity but add equipment and conversion losses. The U.S. Department of Energy's 2024 design guide covers IT equipment, environmental conditions, air management, cooling, electrical systems, heat recovery, and operational metrics. It recommends treating IT efficiency first because reductions in computing load also reduce supporting electrical and cooling demand.^[19]

Higher rack power density can require rear-door heat exchangers or direct liquid cooling when air systems cannot remove heat within equipment limits. Liquid cooling moves heat efficiently near the processors, but it does not eliminate heat rejection, pumps, water-quality controls, leak management, or facility design. Site climate, water constraints, supply temperature, and the intended equipment determine the appropriate system.^[19][22]

Power usage effectiveness (PUE) is the ratio of total data-center energy to energy used by IT equipment. ISO/IEC 30134-2:2026 defines its determination, measurement categories, reporting, and interpretation.^[21] PUE measures facility overhead within a stated boundary. It does not measure useful model work, model quality, carbon intensity, water use, or hardware utilization, so a lower PUE does not by itself establish a better AI system. Comparisons also need consistent measurement boundaries and time periods.

Global and national estimates require careful scope labels. The International Energy Agency estimated that all data centers, not AI alone, used about 415 TWh in 2024, roughly 1.5 percent of global electricity consumption. Its base case projects about 945 TWh in 2030 and identifies accelerated servers as almost half of the net increase. These are modeled projections, not scheduled consumption.^[20] Lawrence Berkeley National Laboratory estimated 176 TWh of U.S. data-center electricity use in 2023, 4.4 percent of national consumption, then presented a 2028 scenario range of 325-580 TWh because future accelerator shipments, operating patterns, and cooling choices are uncertain.^[22]

Energy procurement claims also need physical and contractual boundaries. A global average does not describe a locally constrained grid, and the IEA notes that data centers' spatial concentration can make grid integration challenging. Capacity planning therefore includes the site's firm electrical limit, interconnection schedule, backup strategy, expected utilization, heat-rejection capacity, water use where relevant, and the emissions-accounting method.^[19][20]

Reliability, security, and governance

Reliability is designed across failure domains. A cluster may tolerate a failed disk, link, accelerator, server, rack, or power path differently. Health checks must distinguish a crashed component from one that returns incorrect results or performs slowly. Recovery procedures include retrying work, replacing a worker, rerouting communication, restoring a checkpoint, reducing an allocation, or failing over a service. The expected cost of recovery depends on failure frequency, checkpoint interval, restore bandwidth, model-loading time, and whether the workload can change its parallel layout.^[10][11]

Security controls cover identities, network paths, data, model artifacts, software images, secrets, and hardware administration. Shared infrastructure needs tenant isolation and authorization at both the control plane and device level. Provenance is needed for datasets, dependencies, checkpoints, models, and deployment configurations. Logs and stored prompts or outputs may themselves contain sensitive information, so observability requires retention and access rules.

The NIST AI Risk Management Framework describes secure and resilient operation as one characteristic of trustworthy AI and places testing, evaluation, verification, validation, documentation, and monitoring across the lifecycle.^[23] Its Generative AI Profile adds actions concerning third-party components, training and evaluation data provenance, incident disclosure, continuous monitoring, and content provenance.^[24] The frameworks are voluntary risk-management guidance, not certifications and not substitutes for sector-specific legal or safety requirements.

Operational governance defines who can provision expensive capacity, introduce data, approve an artifact, change a runtime, access logs, and roll back a release. Separation between development, evaluation, and approval can reduce conflicts of interest. Infrastructure should retain enough evidence to reproduce material decisions without collecting unnecessary personal or confidential data.^[23][24]

Deployment models and capacity planning

Organizations can rent public-cloud instances, reserve dedicated cloud capacity, use a colocation provider, operate their own facility, deploy at the edge, or combine these approaches. Price is one of several factors in the decision.

Scroll sideways for more →

Capacity planning starts with a workload distribution rather than a model name. Inputs include model state, context or sample sizes, output lengths, batch behavior, arrival-rate distribution, quality target, latency objectives, training duration, checkpoint policy, and expected growth. The design then tests memory fit, communication volume, storage traffic, failure behavior, facility limits, and utilization. Peak demand, average demand, and recoverable backlog are different quantities.^[17][18][19]

Cost boundaries should be stated. Hardware purchase price or hourly instance price omits facility, network, storage, support, software, reserved-capacity, idle-capacity, and staff costs. Cost per token or training run is also sensitive to what is counted, whether failed and evaluation work is included, and whether the result meets the required quality.

Measurement and comparison

No single metric captures an AI infrastructure system. A useful evaluation keeps four levels separate:^[17][18][19]

Scroll sideways for more →

Benchmark reports should identify the exact system under test and the rules used. MLPerf's training and inference suites provide reference implementations, accuracy or quality constraints, scenarios, divisions, availability categories, and submission metadata.^[17][18] The accompanying benchmark papers explain why a fixed quality target and representative scenarios are necessary for comparison.^[30][31] Submitted results can still differ in hardware scale, software, availability status, and allowed optimizations, so rankings must be interpreted within the matching slice.

For an internal system, end-to-end measurement is more useful than peak specifications alone. Profiling should locate time in data input, compute kernels, memory transfers, collectives, synchronization, checkpointing, queueing, and postprocessing. Load tests should use representative request lengths and arrival patterns. Failure tests should demonstrate restore and failover behavior. Energy measurements should state whether they cover accelerators, servers, or the full facility. A result without these boundaries can be precise but misleading.^{[1][10][12][18][19]}

References

1. ^Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel Emer, "Efficient Processing of Deep Neural Networks: A Tutorial and Survey," Proceedings of the IEEE 105(12), 2017. arxiv.org/...1703.09039
2. ^Norman P. Jouppi et al., "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings," ISCA, 2023. arxiv.org/...2304.01433
3. ^Mohammad Shoeybi et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism," 2019. arxiv.org/...1909.08053
4. ^Samyam Rajbhandari et al., "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models," 2019. arxiv.org/...1910.02054
5. ^Dmitry Lepikhin et al., "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding," 2020. arxiv.org/...2006.16668
6. ^Tri Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," NeurIPS, 2022. arxiv.org/...2205.14135
7. ^Woosuk Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention," SOSP, 2023. arxiv.org/...2309.06180
8. ^Yinmin Zhong et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving," OSDI, 2024. usenix.org/...zhong-yinmin
9. ^Gyeong-In Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models," OSDI, 2022. usenix.org/...yu
10. ^Borui Wan et al., "ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development," NSDI, 2025. usenix.org/...wan-borui
11. ^Xinyu Lian et al., "Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Parallelism," USENIX ATC, 2025. usenix.org/...lian
12. ^Tianyuan Wu et al., "GREYHOUND: Hunting Fail-Slows in Hybrid-Parallel Training at Scale," USENIX ATC, 2025. usenix.org/...wu-tianyuan
13. ^NVIDIA, "NCCL Collective Operations," NCCL User Guide. docs.nvidia.com/...collectives
14. ^PyTorch, "Overview of Distributed PyTorch." docs.pytorch.org/...dist_overview
15. ^Kubernetes, "Kubernetes Scheduler." kubernetes.io/...kube-scheduler
16. ^SchedMD, "Slurm Overview." slurm.schedmd.com/overview
17. ^MLCommons, "MLPerf Training." mlcommons.org/...training
18. ^MLCommons, "MLPerf Inference Documentation." docs.mlcommons.org/inference
19. ^U.S. Department of Energy, "Best Practices Guide for Energy-Efficient Data Center Design," July 2024. energy.gov/...ractice-guide-data-center-design.pdf
20. ^International Energy Agency, "Energy and AI: Energy Demand from AI," 2025. iea.org/...energy-demand-from-ai
21. ^ISO and IEC, "ISO/IEC 30134-2:2026, Information technology - Data centres key performance indicators - Part 2: Power usage effectiveness (PUE)," 2026. iso.org/...30134-2
22. ^Arman Shehabi et al., "2024 United States Data Center Energy Usage Report," Lawrence Berkeley National Laboratory, 2024. eta-publications.lbl.gov/...nergy-usage-report.pdf
23. ^Elham Tabassi, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," NIST AI 100-1, 2023. doi.org/...NIST.AI.100-1
24. ^Chloe Autio et al., "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile," NIST AI 600-1, 2024. doi.org/...NIST.AI.600-1
25. ^Timnit Gebru et al., "Datasheets for Datasets," Communications of the ACM 64(12), 2021. doi.org/...3458723
26. ^Margaret Mitchell et al., "Model Cards for Model Reporting," FAT*, 2019. doi.org/...3287560.3287596
27. ^D. Sculley et al., "Hidden Technical Debt in Machine Learning Systems," NeurIPS, 2015. papers.nips.cc/...896fcaf2674f757a2463eba-Abstract
28. ^Philipp Moritz et al., "Ray: A Distributed Framework for Emerging AI Applications," OSDI, 2018. usenix.org/...moritz
29. ^Denis Baylor et al., "TFX: A TensorFlow-Based Production-Scale Machine Learning Platform," KDD, 2017. research.google/...scale-machine-learning-platform
30. ^Vijay Janapa Reddi et al., "MLPerf Inference Benchmark," 2019. arxiv.org/...1911.02549
31. ^Peter Mattson et al., "MLPerf Training Benchmark," 2019. arxiv.org/...1910.01500
32. ^Ultra Accelerator Link Consortium, "UALink_200 Revision 1.0 Specification," April 2025. ualinkconsortium.org/...n_v1.0_Evaluation_Copy.pdf
33. ^Ultra Ethernet Consortium, "Ultra Ethernet Specification v1.0," June 11, 2025. ultraethernet.org/...UE-Specification-6.11.25.pdf
34. ^Samuel Williams, Andrew Waterman, and David Patterson, "Roofline: An Insightful Visual Performance Model for Multicore Architectures," Communications of the ACM 52(4), 2009. www2.eecs.berkeley.edu/...EECS-2008-134.pdf