Central processing unit

A central processing unit (CPU) is the general-purpose processor that executes a computer's instruction stream. It fetches instructions from memory, decodes them, performs arithmetic and logic on values held in registers, and writes results back. Everything else in a modern machine, including GPU accelerators, storage, and networking, is set in motion by software running on a CPU. The first microprocessor sold commercially as a component, Intel's 4-bit 4004, was released on November 15, 1971 with 2,300 transistors on a 10 micrometre process and a clock rate of 740 to 750 kHz ^[1]. A single 2026 server socket carries hundreds of cores: AMD's 6th Gen EPYC 9996 has 256 of them, and 512 threads with simultaneous multithreading enabled ^[2].

In AI the CPU's role has been redefined twice. Neural networks were first trained on CPUs, then lost nearly all of that work to GPUs between 2009 and 2012, once researchers demonstrated order-of-magnitude speedups from massively parallel hardware ^[3][4]. What remained was not small. CPUs run the orchestration, data loading, and serving logic around every large model; they act as host processors inside GPU systems; and they still execute inference directly for models that are small enough, quantized enough, or served at low enough concurrency that an accelerator is not worth its cost.

The last decade has also pushed matrix hardware back into the CPU itself. Intel added Advanced Matrix Extensions (AMX) to Xeon in 2023, Arm specified a Scalable Matrix Extension in Armv9.2-A, and open-source runtimes such as llama.cpp made quantized CPU inference routine on ordinary laptops and servers ^[5][6][7].

How a CPU executes a program

A CPU is organized around a few classic parts: an arithmetic logic unit (ALU) that performs integer and bitwise operations, a floating-point unit, a control unit that sequences instruction execution, and a register file holding the operands currently in use. A program counter tracks the address of the next instruction. The instruction set architecture (ISA) is the contract between hardware and software: it defines the visible registers, the instruction encodings, the memory model, and the privilege levels, and it is what lets a binary compiled once run on chips from different vendors and generations.

Between the register file and main memory sits a cache hierarchy built from SRAM. A first-level cache of tens of kilobytes per core answers in a few cycles, a private second level in the megabyte range answers in a few dozen, and a shared last-level cache serves the whole socket. DRAM access costs hundreds of cycles. Ampere's AmpereOne parts, for example, give each core 2 MB of private L2 alongside 64 MB of system-level cache ^[8]. Most of the engineering effort in a modern core goes into hiding that memory latency rather than into arithmetic.

The techniques that do the hiding accumulated over sixty years. Pipelining overlaps the stages of successive instructions. Superscalar issue starts several instructions per cycle. Out-of-order execution lets instructions whose operands are ready run ahead of stalled ones: the CDC 6600 of 1964 used scoreboarding for this, the IBM System/360 Model 91 of 1966 added register renaming through Tomasulo's algorithm, and the technique reached commodity x86 with NexGen's Nx586 in 1994 and Intel's Pentium Pro in 1995 ^[9]. Branch prediction and speculative execution keep the pipeline fed across control flow. Simultaneous multithreading fills the leftover issue slots with a second thread, which is what doubles the thread count a socket reports to the operating system ^[2].

Single instruction, multiple data (SIMD) units add the other kind of parallelism. Intel announced AVX-512 in July 2013 and first shipped it in the Xeon Phi x200 (Knights Landing) in 2016, widening the vector register file to 32 registers of 512 bits plus eight opmask registers ^[10]. Vector Neural Network Instructions (VNNI) arrived with Cascade Lake in 2019 to multiply and accumulate byte and word pairs for deep learning, and a bfloat16 extension followed with Cooper Lake in 2020 ^[10]. AMD added AVX-512 with Zen 4 in 2022, and Intel announced AVX10 in July 2023 as a versioned successor ^[10]. On the Arm side, the Scalable Vector Extension lets an implementation pick a hard-coded vector width between 128 and 2,048 bits in 128-bit steps, so the same binary runs across designs of different widths ^[6].

Instruction set architectures

Three ISAs matter for AI infrastructure, and they differ as much in governance as in engineering.

Scroll sideways for more →

The practical difference shows up in who can build a chip. x86-64 is proprietary and cross-licensed between Intel and AMD, so a cloud provider cannot simply design its own x86 core ^[11]. Arm sells both finished core designs, the Neoverse line for servers, and architectural licences that let a licensee build a custom core, so Amazon, Google, Microsoft, NVIDIA, Ampere Computing, and Qualcomm can all ship CPUs of their own ^[12]. That is why hyperscaler-designed server silicon is almost entirely Arm. RISC-V removes the licence entirely. In October 2025 RISC-V International was granted PAS submitter status by ISO/IEC JTC1, allowing it to submit the RISC-V Instruction Set Manual for consideration as an international standard; the organization notes that no ISA has previously become an international standard ^[14]. A 2026 forecast published by RISC-V International projects 35.9 billion RISC-V SoC units and $318 billion of associated revenue by 2031, with AI accelerators the largest revenue segment ^[15].

Moore's law, Dennard scaling, and the turn to multicore

Gordon Moore's 1965 article in Electronics observed that the component count at minimum cost was doubling roughly every year, and projected the trend forward a decade; he revised the cadence to roughly every two years at the 1975 IEEE International Electron Devices Meeting ^[16]. That is a statement about density, not speed. The reason density translated into speed for thirty years was a separate result: Robert Dennard and five co-authors showed in the October 1974 IEEE Journal of Solid-State Circuits that as MOSFET dimensions shrink, voltage and current scale down with length, so power density stays constant ^[17].

Dennard scaling broke down around 2005-2007. Leakage current and threshold voltage stopped scaling with size, so power density rose as transistors shrank, and the heat had nowhere to go ^[17]. Intel's cancellation of the Tejas and Jayhawk processors in 2004 was an early public sign of the resulting power wall ^[17]. Clock frequency flattened; transistor counts kept climbing. The industry spent the extra transistors on more cores rather than faster ones, which meant the gains only reached software that could be split across cores ^[18].

That transition is the structural reason AI hardware went the way it did. Once a CPU core could not simply be made faster, the returns moved to specialization and to parallelism, and the workloads with the most parallelism to exploit found the hardware built for it.

Why AI training moved to GPUs

Training a neural network is dominated by dense matrix multiplication over large batches. The work is regular, has few data-dependent branches, and reuses each loaded value many times. That profile does not reward the machinery a CPU core spends most of its area on. Out-of-order windows, branch predictors, and deep cache hierarchies exist to make irregular, latency-sensitive code fast; a matrix kernel already knows which data it needs and when.

The empirical case was made early. Rajat Raina, Anand Madhavan, and Andrew Ng reported at ICML 2009 that a GPU implementation of deep belief network learning ran "up to 70 times faster than a dual-core CPU implementation for large models," cutting training of a four-layer network with 100 million parameters from several weeks to about a day ^[3]. AlexNet won ImageNet in 2012 using what its authors called "a very efficient GPU implementation of convolutional nets" ^[4]. By the time Google published its first Tensor Processing Unit paper at ISCA 2017, the reference points were already accelerators: the TPU was reported as "on average about 15X - 30X faster than its contemporary GPU or CPU," with performance per watt 30 to 80 times higher ^[19].

Memory bandwidth compounds the gap. An H100 SXM module is rated at 3.35 TB/s of memory bandwidth ^[20]. Server CPUs feed their cores through DDR channels: Intel says the Xeon 6700P is planned to support 8,000 MT/s RDIMMs at one DIMM per channel on select SKUs, a 25% increase over the 6,400 MT/s parts, and AMD's 6th Gen EPYC platforms support up to 16 channels of 12.8 GT/s MRDIMM ^[21][2]. Even those configurations sit well below a single datacenter GPU's high bandwidth memory. Software closed the loop: CUDA gave GPUs a mature kernel and library ecosystem that CPU vendors have spent the following decade trying to match for AI.

CPU inference and matrix extensions

CPUs never lost inference the way they lost training, and vendors have pushed matrix hardware into the core to keep it. Intel introduced AMX in June 2020 and first shipped it in the Sapphire Rapids Xeon generation released in January 2023 ^[5][22]. AMX adds two-dimensional tile registers, eight of them, each 16 rows of 64 bytes, operated on by a Tile Matrix Multiply unit (TMUL) that performs matrix multiply-and-accumulate in BF16 and INT8; a 4th Gen Xeon core can perform 2,048 INT8 or 1,024 BF16 operations per cycle, and an FP16 variant was documented in September 2022 ^[5]. The state is large enough that Linux gives AMX its own dynamically enabled XSTATE components, XTILECFG and XTILEDATA, so that programs must ask for permission and prove they have a large enough signal stack before using them ^[23]. Arm's answer is the Scalable Matrix Extension, introduced in Armv9.2-A with matrix tile storage, on-the-fly transposition, and outer-product accumulation over SVE vectors; SME2 in Armv9.4-A added multi-vector instructions along with 2-bit and 4-bit weight compression and binary-network support ^[6].

The software stack that reaches this hardware is mostly open source. oneDNN, an Apache-2.0 library governed by the UXL Foundation, provides the optimized primitives and covers x86-64 and AArch64, with experimental support for IBM Power, s390x, and 64-bit RISC-V; PyTorch, TensorFlow, ONNX Runtime, OpenVINO, and PaddlePaddle all build on it ^[24]. Intel's separate PyTorch extension, which targeted AVX-512 VNNI and AMX with FP32, BF16, and INT8 or INT4 weight-only quantization, was archived in March 2026 after Intel upstreamed most of its features; Intel now recommends using PyTorch directly ^[25].

The runtime that did the most for CPU inference in practice is llama.cpp, the main development playground for Georgi Gerganov's ggml tensor library ^[7][26]. It aims at "LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware," with AVX, AVX2, AVX512 and AMX paths on x86, NEON and Accelerate on Apple silicon, and RVV on RISC-V, plus 1.5-bit through 8-bit integer quantization ^[7]. Tools built on it, including Ollama, put local model execution on machines with no discrete GPU at all, which is the practical foundation of much on-device AI and edge computing work.

CPU inference makes sense where capacity and flexibility matter more than peak throughput: small models, low or bursty concurrency, latency-tolerant batch jobs, and workloads that must run on hardware already deployed. Memory capacity is the CPU's structural advantage. An NVIDIA DGX B200 host carries 2 TB of system memory against 1,440 GB of GPU memory across eight B200 GPUs, and Intel cites up to 8 TB of system memory on Xeon 6 platforms ^[27][28]. For mixture of experts models, where only a fraction of parameters is active per token, that ratio changes the arithmetic of where weights can live.

CPUs as host processors in AI systems

Every GPU system needs a CPU. In NVIDIA's x86 systems the host is a Xeon: a DGX B200 pairs eight B200 GPUs with two Intel Xeon 8570 processors of 56 cores each, 2 TB of memory, and NVMe storage ^[27]. Intel says the Xeon 6776P fills the same role in Blackwell-based DGX B300 systems, and announced in March 2026 that Xeon 6 would be the host CPU in NVIDIA DGX Rubin NVL8 systems ^[28]. Intel's framing of the job is direct: the host CPU "governs orchestration, memory access, model security, and throughput across GPU-accelerated systems" ^[28]. AMD makes the same argument for its own parts, with CEO Lisa Su saying that for AI host nodes, 6th Gen EPYC CPUs "deliver the speed and memory bandwidth to keep accelerators fully fed" ^[2].

NVIDIA's rack-scale systems go further and use an Arm CPU the company designs itself. NVIDIA Grace has 72 Arm Neoverse V2 cores; the Grace CPU Superchip joins two of them over a 900 GB/s NVLink-C2C link for 144 cores and up to 1 TB/s of LPDDR5X bandwidth ^[29]. The GH200 Grace Hopper Superchip pairs one Grace with a Hopper GPU and 480 GB of LPDDR5X, coherent across the link ^[30]. A GB200 NVL72 rack contains 36 Grace CPUs, 2,592 Neoverse V2 cores in total, alongside 72 NVIDIA Blackwell GPUs ^[31]. The successor Vera CPU uses a custom NVIDIA core called Olympus, which NVIDIA claims delivers twice the single-threaded performance, three times the core-to-core bandwidth, and 40% lower memory latency than competing chiplet designs; as of July 2026 Vera Rubin NVL72 racks were ramping into production at CoreWeave, Google Cloud, Microsoft Azure, and Oracle Cloud Infrastructure ^[32].

The server CPU market

The x86-64 server market is an Intel and AMD duopoly, while the Arm side of the data center is split between the merchant vendor Ampere and CPUs the cloud providers commission or design for their own fleets.

Scroll sideways for more →

The Arm push is measurable in deployment rather than in datasheets. AWS Graviton powers over 350 EC2 instance types used by more than 120,000 customers, and AWS says Meta is deploying Graviton "at scale starting with tens of millions of cores" ^[35][36]. Google's Axion C4A instances, generally available since 31 October 2024, are claimed to offer up to 65% better price-performance and up to 60% better energy efficiency than comparable current-generation x86 instances, with Bigtable, Spanner, BigQuery, and the YouTube ads platform among the internal users ^[37]. Intel, meanwhile, has reframed its pitch around the control plane: its Xeon 6+ announcement argues that "as AI becomes more agentic, the constraints shift to orchestration, concurrency, and data movement," and an Intel data center fellow argued in July 2026 that memory bandwidth is the most overlooked metric in AI systems because processors otherwise "spend valuable cycles waiting for data" ^[34][21].

Limitations

Two structural limits shape what a CPU can do for AI. The first is parallelism: adding cores helps only the fraction of a workload that can be split across them, and single-thread performance has grown slowly since the mid-2000s, so the multicore era rewarded workloads like rendering, serving, and training far more than it rewarded serial code ^[18]. The second is bandwidth. Peak arithmetic throughput on a CPU has grown faster than the memory system feeding it, and Intel's own 2026 case for 8,000 MT/s RDIMMs, followed by second-generation MRDIMMs at 8,800 MT/s on Xeon 6900P in the first quarter of 2027, is an admission that the socket is bandwidth-limited for AI work ^[21].

Speculative execution, one of the main sources of single-thread performance, also turned out to be a security liability. Meltdown and Spectre, disclosed in January 2018, exploit speculation to leak memory across isolation boundaries: Meltdown "breaks the most fundamental isolation between user applications and the operating system," and Spectre "breaks the isolation between different applications" ^[38]. Meltdown affected essentially every Intel processor implementing out-of-order execution since 1995, and Spectre was verified on Intel, AMD, and Arm parts ^[38]. The mitigations, in microcode, kernels, and compilers, cost performance on exactly the code paths that CPUs are supposed to be good at, and the class of attack has continued to generate new variants since.

See also

• GPU
• AI accelerator
• NVIDIA Grace
• RISC-V
• Tensor core
• llama.cpp

References

1. ^"Intel 4004", Wikipedia. en.wikipedia.org/...Intel_4004
2. ^"AAI 2026: AMD Delivers Full-Stack Compute for the Agentic AI Era", AMD press release, 23 July 2026. ir.amd.com/...stack-compute-for-the-agentic-ai-era
3. ^Rajat Raina, Anand Madhavan and Andrew Y. Ng, "Large-scale Deep Unsupervised Learning using Graphics Processors", Proceedings of the 26th International Conference on Machine Learning (ICML), Montreal, 2009. icml.cc/...218.pdf
4. ^Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks", NIPS 2012. papers.nips.cc/...b9d6b76c8436e924a68c45b-Abstract
5. ^"Advanced Matrix Extensions", Wikipedia. en.wikipedia.org/...Advanced_Matrix_Extensions
6. ^"AArch64", Wikipedia (SVE, SVE2, SME and SME2 sections). en.wikipedia.org/...AArch64
7. ^llama.cpp repository README, ggml-org. github.com/...llama.cpp
8. ^"Cloud Native Processors", Ampere Computing product pages. amperecomputing.com/...processors
9. ^"Out-of-order execution", Wikipedia. en.wikipedia.org/...Out-of-order_execution
10. ^"AVX-512", Wikipedia. en.wikipedia.org/...AVX-512
11. ^"x86-64", Wikipedia. en.wikipedia.org/...X86-64
12. ^"ARM architecture family", Wikipedia. en.wikipedia.org/...ARM_architecture_family
13. ^"RISC-V", Wikipedia. en.wikipedia.org/...RISC-V
14. ^Andrea Gallo, "RISC-V Takes First Step Toward International Standardization as ISO/IEC JTC1 Grants PAS Submitter Status", RISC-V International, 21 October 2025. riscv.org/...risc-v-jtc1-pas-submitter
15. ^James De Vile, "Behind The Scenes of SHD Group's 2026 RISC-V Market Forecast", RISC-V International, 19 June 2026. riscv.org/...shd-forecast-2026
16. ^"Moore's law", Wikipedia. en.wikipedia.org/...Moore%27s_law
17. ^"Dennard scaling", Wikipedia (citing R. H. Dennard et al., IEEE Journal of Solid-State Circuits SC-9(5):256-268, October 1974). en.wikipedia.org/...Dennard_scaling
18. ^Karl Rupp, "42 Years of Microprocessor Trend Data", February 2018. karlrupp.net/...years-of-microprocessor-trend-data
19. ^Norman P. Jouppi et al., "In-Datacenter Performance Analysis of a Tensor Processing Unit", ISCA 2017, arXiv:1704.04760. arxiv.org/...1704.04760
20. ^"NVIDIA H100 Tensor Core GPU", NVIDIA product specifications. nvidia.com/...h100
21. ^Srini Krishna, "Memory Bandwidth May Be the Most Overlooked AI Performance Metric", Intel Newsroom, 20 July 2026. newsroom.intel.com/...looked-ai-performance-metric
22. ^"Xeon", Wikipedia. en.wikipedia.org/...Xeon
23. ^"x86 XSTATE", Linux kernel documentation. docs.kernel.org/...xstate
24. ^oneDNN repository README, UXL Foundation. github.com/...oneDNN
25. ^Intel Extension for PyTorch repository README. github.com/...intel-extension-for-pytorch
26. ^ggml.ai. ggml.ai
27. ^"Introduction to NVIDIA DGX B200", NVIDIA DGX B200 User Guide. docs.nvidia.com/...introduction-to-dgxb200
28. ^"Intel Xeon 6 Used as Host CPUs in NVIDIA DGX Rubin NVL8 Systems", Intel Newsroom, 16 March 2026. newsroom.intel.com/...vidia-dgx-rubin-nvl8-systems
29. ^"NVIDIA Grace CPU", NVIDIA. nvidia.com/...grace-cpu
30. ^"Hopper (microarchitecture)", Wikipedia. en.wikipedia.org/...Hopper_(microarchitecture)
31. ^"NVIDIA GB200 NVL72", NVIDIA. nvidia.com/...gb200-nvl72
32. ^"NVIDIA Vera Rubin Driving Performance Per Watt, Lowest Token Cost for Partners Worldwide", NVIDIA blog, 21 July 2026. blogs.nvidia.com/...vera-rubin
33. ^"AMD Announces Production Ramp of Next-Generation AMD EPYC Processor 'Venice' on TSMC 2nm Process Technology", AMD press release, 21 May 2026. ir.amd.com/...enice-on-tsmc-2nm-process-technology
34. ^"Intel Puts Agentic AI to Work with Xeon 6+, Networking, and AI Systems", Intel Newsroom, 31 May 2026. newsroom.intel.com/...xeon-6-networking-ai-systems
35. ^Esra Kayabali, "Now available: Amazon EC2 M9g and M9gd instances powered by new AWS Graviton5 processors", AWS News Blog, 10 June 2026. aws.amazon.com/...-by-new-aws-graviton5-processors
36. ^"AWS Graviton Processor", Amazon Web Services. aws.amazon.com/...graviton
37. ^"Try C4A, the first Google Axion processor", Google Cloud blog. cloud.google.com/...e-first-google-axion-processor
38. ^"Meltdown and Spectre", Graz University of Technology. meltdownattack.com