PCI Express

PCI Express (PCIe) is a high-speed serial interconnect standard used to attach processors to graphics cards, network adapters, storage devices, and accelerators inside nearly every modern computer and server. It is maintained by PCI-SIG, the PCI Special Interest Group, which has published seven generations of the base specification since 2003, each roughly doubling the data rate of the previous one ^[1]. PCIe replaced three older interfaces at once: conventional PCI, PCI-X, and the Accelerated Graphics Port (AGP) ^[1].

The design change that made this possible was abandoning the shared parallel bus. Conventional PCI put every device on a common set of wires running at a common clock. PCIe instead uses point-to-point topology, with separate serial links connecting each device to a root complex on the host, and switches fanning those links out to more devices ^[1]. Each link is built from lanes, and each lane is two unidirectional differential pairs (four wires total) that carry data in both directions at once ^[1]. The standard defines link widths of x1, x2, x4, x8, and x16, with x12 and x32 also defined up to and including PCIe 5.0; x16 is the largest width in common use ^[1]. Lane count is negotiated automatically at initialization, and a link can down-configure itself to fewer lanes if some are unreliable ^[1].

For AI infrastructure, PCIe matters in a specific and slightly awkward way. It is the standard path between host memory and an AI accelerator, the path from NVMe storage into GPU memory, and the path a network adapter uses to reach the GPU. It is also, by a wide margin, the slowest of the links inside a modern training node. An NVIDIA H100 reaches its own high-bandwidth memory at 3.35 TB/s and its neighbors over NVLink at 900 GB/s, but reaches the host over PCIe Gen5 at 128 GB/s ^[2]. Much of the engineering described below exists to keep that gap from stalling the accelerators.

Origins and governance

PCIe was developed under the codename 3GIO, after an earlier working name of HSI (High Speed Interconnect), by a body called the Arapaho Work Group that began as Intel engineers before widening to include other companies ^[1]. The standard was created in 2003 by Intel, Dell, HP, and IBM, and PCI-SIG introduced PCIe 1.0a that year with a per-lane transfer rate of 2.5 GT/s ^[1].

PCI-SIG has kept a steady cadence since. It publishes the base specification, plus companion documents covering card form factors, cabling, and connectors, and it releases interim point revisions between major generations. The optical engineering change notice published in 2025, for example, amends both the PCIe 6.4 specification and the PCIe 7.0 specification ^[3].

Generations and signaling

Every generation has doubled the per-lane rate, but the mechanism for doing so has changed twice. The first two generations used 8b/10b line coding, which spends 20 percent of the raw bit rate on encoding overhead. PCIe 3.0 switched to 128b/130b, cutting that overhead to roughly 1.5 percent and allowing effective throughput to nearly double even though the signaling rate rose only from 5 GT/s to 8 GT/s ^[1].

PCIe 6.0, released on 11 January 2022, changed the physical signaling instead ^[5]. It adopted PAM4, which uses four voltage levels to encode two bits per symbol rather than the two levels of NRZ, so the data rate doubles without a proportional increase in symbol rate ^[4]. PAM4 has smaller voltage margins and a higher raw error rate, so PCIe 6.0 also added forward error correction and reorganized traffic into fixed 256-byte flow control units, or FLITs, replacing the variable-length packet handling of earlier generations ^[1][4]. A three-way interlaced FEC scheme plus a cyclic redundancy check protects each FLIT, of which 242 bytes carry data ^[1]. PCIe 7.0 keeps both PAM4 and the 256-byte FLIT structure, so reaching 128 GT/s required doubling the symbol rate as well ^[4].

Scroll sideways for more →

Source: PCI-SIG generation data as compiled in ^[1].

Reading the bandwidth numbers

Vendors and PCI-SIG usually quote a larger figure than the table above, because they count both directions and use the raw signaling rate. PCI-SIG announced PCIe 6.0 as delivering up to 256 GB/s over x16 ^[5], and PCIe 7.0 is quoted at up to 512 GB/s bidirectionally over x16 ^[6]. The arithmetic for PCIe 5.0 is 32 GT/s times 16 lanes times two directions, divided by 8 bits per byte, which gives the familiar 128 GB/s that NVIDIA prints in its data sheets ^[2]. Subtracting encoding overhead and halving for a single direction gives the 63 GB/s in the table. Both conventions are legitimate; they answer different questions, and they are easy to mix up when comparing hardware.

Form factors and connectors

The base specification defines the protocol; separate documents define the physical shapes it comes in. The Card Electromechanical (CEM) specification covers the familiar edge-connector add-in card. Slot power is capped by card class: x1 cards are limited to 10 W combined, x4 and wider cards to 25 W, and a full-size x16 graphics card may draw up to 75 W combined once software has configured it as a high-power device ^[1]. Anything beyond that requires auxiliary power, which is why high-power accelerator cards carry supplementary connectors; the 16-pin 12VHPWR connector supports up to 600 W ^[1].

Storage uses different shapes on the same electrical foundation. M.2 is the compact drive format that succeeded mSATA, U.2 carries up to four PCIe lanes in a 2.5-inch package, and the Enterprise and Data Center Standard Form Factor (EDSFF) defines the module shapes used in servers ^[7]. PCI-SIG also standardized cabling: the CopprLink internal and external cable specifications carry PCIe 5.0 and 6.0 signaling, at 32.0 and 64.0 GT/s respectively, beyond the confines of a single board ^[1].

The Open Compute Project defines a separate accelerator packaging family under its Open Accelerator Infrastructure effort, comprising the OCP Accelerator Module (OAM), a Universal Baseboard (UBB), and a Host Interface Board ^[8]. AMD builds its Instinct platforms this way: the MI300X platform is a UBB 2.0 baseboard hosting eight MI300X OAM accelerators with 192 GB of HBM3 each, fully meshed over Infinity Fabric, and connected upstream at x16 and 32 GT/s ^[9].

What SXM is, and what it is not

NVIDIA's SXM modules are often described as an alternative to PCIe. That is half right. SXM, which expands to Server PCI Express Module, is a socketed mezzanine form factor that mounts GPUs flat on a baseboard rather than vertically in a slot, and it delivers power through the socket itself, letting an H100 draw its full 700 W without external cables ^[10]. What it is not is a replacement for the host bus. NVIDIA's own specifications list PCIe Gen5 at 128 GB/s for the SXM versions of both the H100 and the H200, the same host link the card versions use ^[2][11]. The difference is what happens between GPUs: SXM parts get the full-width NVLink connection to their neighbors, while a card version relies on a bridge or on PCIe itself.

The same split shows up in the published specifications. Card variants run at lower power and lower peak tensor throughput in exchange for fitting a standard chassis, even where the memory configuration matches, as it does between the H200 SXM and H200 NVL.

Scroll sideways for more →

Sources: NVIDIA H100 and H200 product specifications ^[2][11].

CXL and the shared physical layer

Compute Express Link (CXL) reuses the PCIe electrical and physical layers to carry a cache-coherent protocol, so a CXL device plugs into a standard PCIe slot ^[12]. CXL multiplexes three sub-protocols: CXL.io, which is built on PCIe and handles configuration, link initialization, device discovery, interrupts, DMA, and register access; CXL.cache, which lets a device coherently cache host memory; and CXL.mem, which lets the host address device-attached memory with load and store instructions ^[12].

The version history tracks PCIe closely. CXL 1.0 arrived in March 2019 on the PCIe 5.0 physical layer, CXL 2.0 in November 2020 added switching, and CXL 3.0 in August 2022 moved to the PCIe 6.0 physical interface and PAM4 coding ^[12]. The CXL Consortium released CXL 4.0 on 18 November 2025, doubling the rate from 64 GT/s to 128 GT/s, keeping the 256-byte FLIT format, adding native x2 width and support for up to four retimers, and retaining backward compatibility with CXL 3.x, 2.0, 1.1, and 1.0 ^[13]. Derek Rohde, the consortium's president and a principal engineer at NVIDIA, described the release as reflecting "our member companies' commitment to driving open standards that empower data center innovation" ^[13].

In practice CXL has found more traction as a memory expansion and pooling interface than as an accelerator attach point, since accelerator vendors ship proprietary coherent links of their own.

PCIe in AI systems

Feeding the accelerators

Three data flows in a training or inference node cross PCIe: host-to-device transfers of weights and activations, storage-to-device transfers of training data and checkpoints, and network-to-device transfers for distributed training. NVIDIA's GPUDirect family exists to shorten the last two.

GPUDirect RDMA "enables a direct path for data exchange between the GPU and a third-party peer device using standard features of PCI Express," using PCI base address registers so a network adapter can read and write GPU memory directly ^[14]. The constraint is topological: the two devices must sit under the same upstream PCIe root complex, and paths that cross between CPU sockets are documented as limited or unreliable ^[14]. This is why AI server designs place network adapters and GPUs under the same PCIe switch, and why switch and retimer silicon became a product category of its own.

GPUDirect Storage applies the same idea to drives. It "enables a direct data path for direct memory access (DMA) transfers between GPU memory and storage, which avoids a bounce buffer through the CPU," and NVIDIA's documentation notes that without it "an extra copy through a bounce buffer in the CPU is necessary, which introduces latency and lowers effective bandwidth" ^[15]. NVIDIA introduced the technology with the observation that "fast GPUs are increasingly starved by slow I/O" ^[16].

Storage and data loading

Nearly all AI storage is NVMe, an interface specification for non-volatile media attached over PCIe, first released as version 1.0 in March 2011 ^[7]. NVMe's advantage over the older AHCI interface is parallelism suited to many-core hosts: it supports up to 65,535 queues with up to 65,536 commands each, against AHCI's single 32-command queue ^[7]. A DGX B200 illustrates the typical arrangement, with two 1.92 TB NVMe M.2 drives mirrored for the operating system and eight 3.84 TB NVMe U.2 drives striped as a data cache, alongside two Intel Xeon 8570 PCIe Gen5 processors ^[17].

Whether storage can keep accelerators busy is now benchmarked directly. MLCommons published MLPerf Storage v2.0 results on 4 August 2025, adding a checkpointing workload and reporting that submitted systems served roughly twice the number of accelerators as in the previous round ^[18]. The organization framed the practical question as whether a storage system can absorb a cluster's checkpoints "without slowing the system down to an unacceptable level" ^[18].

Limits

PCIe is a general-purpose bus doing a job that specialized fabrics do better, and the industry has responded by routing around it wherever the cost is justified.

Bandwidth is the visible limit. NVIDIA states that its sixth-generation NVLink delivers 3,600 GB/s per GPU, which the company describes as "over 14x the bandwidth of PCIe Gen6" ^[19]. Within a GPU cluster, NVLink Switch fabrics reach 130 TB/s across a 72-GPU GB200 NVL72 domain and 260 TB/s in the announced sixth-generation systems ^[19]. Collective operations that would saturate PCIe run entirely inside those domains, leaving PCIe to handle the host relationship. Competing efforts such as UALink for scale-up and Ultra Ethernet for scale-out pursue the same separation with open standards.

Reach is the second limit. PAM4 signaling at 64 and 128 GT/s degrades quickly over copper, which is why retimers have become standard components in AI servers rather than optional ones, and why PCI-SIG opened a workgroup on making the architecture more optical-friendly. In June 2025 the group announced what it called the industry's first standards-based PCIe optical solution, an optical aware retimer engineering change notice amending the PCIe 6.4 and 7.0 specifications to allow "extended reach across racks and pods." PCI-SIG president Al Yanes said the group expects "initial adoption to occur in data center applications like AI/ML and cloud" ^[3].

The third limit is adoption lag. A finished specification is not shipping silicon, and the gap has widened as signaling has become harder. PCIe 5.0 was specified in 2019 and is still the host interface on shipping Hopper systems and on DGX B200 systems built around Blackwell GPUs ^[2][11][17]. PCIe 6.0 hardware was originally expected in enterprise markets in early 2024 with PCIe 7.0 following in 2027; a 2024 revision to the schedule pushed the PCIe 7.0 compliance program out to 2028 ^[20].

Recent developments

PCI-SIG released the final PCIe 7.0 specification on 11 June 2025 at its developers conference, delivering 128.0 GT/s per lane and up to 512 GB/s bidirectionally over x16, and named cloud computing, 800-gigabit Ethernet, and artificial intelligence as its initial targets ^[1][6]. Less than two months later, on 5 August 2025, the group announced work on PCIe 8.0 at 256.0 GT/s and up to 1 TB/s bidirectionally over x16, due in 2028 ^[1][21].

Deployment is now catching up on the previous generation. AMD launched its sixth-generation EPYC processors, codenamed Venice, at its Advancing AI 2026 event with up to 256 cores per socket, 1.6 TB/s of memory bandwidth, and 128 lanes of PCIe Gen6 at 64 GT/s; trade coverage in July 2026 called it the first server CPU with PCIe Gen6 and the only one offering it to accelerators, with SP7 parts scheduled for the fourth quarter of 2026 ^[22]. On the connectivity side, Astera Labs reported record revenue of $308.4 million for the quarter ended 31 March 2026, up 93 percent year over year, and attributed the growth to its PCIe 6 portfolio, including Scorpio P-Series switches spanning 32 to 320 lanes and Aries 6 retimers ^[23].

Both announcements reflect the position PCIe has held for two decades. It rarely wins on raw bandwidth against a purpose-built accelerator fabric, but every processor, drive, and network adapter in the data center already speaks it, so the host side of an AI system is built on it regardless.

See also

• NVLink
• NVSwitch
• InfiniBand
• NVIDIA HGX
• GPU computing
• AI accelerator

References

1. ^Wikipedia, "PCI Express." en.wikipedia.org/...PCI_Express
2. ^NVIDIA, "NVIDIA H100 Tensor Core GPU" product specifications. nvidia.com/...h100
3. ^Signal Integrity Journal, "PCI-SIG Announces Industry's First Standard-Based PCIe Optical Solution," 11 June 2025. signalintegrityjournal.com/...cie-optical-solution
4. ^Rambus, "The Ultimate Guide to PCI Express." rambus.com/...the-ultimate-guide-to-pci-express
5. ^HPCwire, "PCI-SIG Releases PCIe 6.0 Specification," 11 January 2022. hpcwire.com/...sig-releases-pcie-6-0-specification
6. ^Synopsys, "PCI Express 7.0 IP Solutions." synopsys.com/...pci-express-7
7. ^Wikipedia, "NVM Express." en.wikipedia.org/...NVM_Express
8. ^Open Compute Project, "Server/OAI" project wiki. opencompute.org/...OAI
9. ^AMD, "AMD Instinct MI300X," Instinct Customer Acceptance Guide. instinct.docs.amd.com/...mi300x
10. ^Wikipedia, "SXM (socket)." en.wikipedia.org/...SXM_(socket)
11. ^NVIDIA, "NVIDIA H200 Tensor Core GPU" product specifications. nvidia.com/...h200
12. ^Wikipedia, "Compute Express Link." en.wikipedia.org/...Compute_Express_Link
13. ^CXL Consortium, "CXL Consortium Releases the Compute Express Link 4.0 Specification Increasing Speed and Bandwidth," 18 November 2025. computeexpresslink.org/...e_FINAL_Website-Copy.pdf
14. ^NVIDIA, "GPUDirect RDMA," CUDA Toolkit documentation. docs.nvidia.com/...gpudirect-rdma
15. ^NVIDIA, "GPUDirect Storage Overview Guide." docs.nvidia.com/...overview-guide
16. ^NVIDIA Developer Blog, "GPUDirect Storage: A Direct Path Between Storage and GPU Memory." developer.nvidia.com/...gpudirect-storage
17. ^NVIDIA, "Introduction to the NVIDIA DGX B200 System," DGX B200 User Guide. docs.nvidia.com/...introduction-to-dgxb200
18. ^MLCommons, "New MLPerf Storage v2.0 Benchmark Results," 4 August 2025. mlcommons.org/...mlperf-storage-v2-0-results
19. ^NVIDIA, "NVLink and NVLink Switch." nvidia.com/...nvlink
20. ^TechSpot, "PCIe 6.0 and 7.0 deployment encounters delays," 14 June 2024. techspot.com/...60-70-deployment-encounters-delays
21. ^Phoronix, "PCI Express 8.0 Aims For 256 GT/s In 2028," 5 August 2025. phoronix.com/...PCI-Express-8.0-Announcement
22. ^StorageReview, "AMD 6th Gen EPYC Venice: 256 Cores, 1.6TB/s, and the First PCIe Gen 6 Server CPU," 23 July 2026. storagereview.com/...e-first-pcie-gen-6-server-cpu
23. ^Astera Labs, "Astera Labs Reports First Quarter 2026 Financial Results." asteralabs.com/...t-quarter-2026-financial-results