# Remote direct memory access

> Source: https://aiwiki.ai/wiki/rdma
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: AI Compute, AI Hardware, AI Infrastructure
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Remote direct memory access." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/rdma
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Remote direct memory access** (**RDMA**) is a networking technique in which a network adapter moves data directly between the memory of one computer and the memory of another, without the operating system copying the data through kernel buffers and, for one-sided operations, without involving the remote processor at all. Applications set up a channel through the operating system once and then post transfers straight to the adapter, which handles segmentation, reliability and placement in hardware [1][2]. RDMA is the defining feature of [InfiniBand](https://aiwiki.ai/wiki/infiniband), and it is also carried over Ethernet by RoCE (RDMA over Converged Ethernet) and by the TCP-based iWARP protocols; all three expose essentially the same programming interface, known as verbs [2].

In AI infrastructure, RDMA connects the servers of large [GPU cluster](https://aiwiki.ai/wiki/gpu_cluster)s; the authors of a 2026 transport paper from AMD, Broadcom, Intel, Microsoft, NVIDIA and OpenAI call the RoCEv2 reliable connection the dominant open standard in production AI/ML clusters [26]. Collective communication libraries such as [NCCL](https://aiwiki.ai/wiki/nccl) use RDMA for communication between nodes, and the GPUDirect RDMA extension lets a network adapter read and write GPU memory directly instead of staging data in host memory [14][16]. As of the June 2026 [TOP500](https://aiwiki.ai/wiki/top500) list, InfiniBand and Ethernet with RoCE together connected 376 of the 500 systems, according to the InfiniBand Trade Association [27]. Meta trained [Llama 3](https://aiwiki.ai/wiki/llama_3) 405B on a RoCE cluster [21], Microsoft reported in 2023 that about 70% of traffic in Azure was RDMA [17], and in 2026 engineers from six companies and, separately, Meta published new transports that keep the RDMA programming model while changing how it behaves on the wire [26][28].

## How does RDMA work?

### The problem it solves

A conventional TCP/IP stack is byte-stream oriented and needs the operating system for every operation. On the receive side the kernel places arriving bytes in its own buffers and then copies them into the application's buffer [2]. The IETF problem statement for RDMA over IP (RFC 4297, December 2005) argued that data movement, specifically copying, was a major source of host overhead in high-speed networking, and that the architectural fix was for the network adapter to place data directly into application buffers [1]. It collected earlier measurements in support, including a 1989 study of TCP on a Sun-3/60 workstation that attributed 48% of the measured overhead to copying [1].

Microsoft and Mellanox researchers gave a modern comparison in 2015. On two Windows servers with 40 Gb/s adapters, driving the link at full speed with 4 MB messages cost TCP more than 20% of the CPU cycles across all cores, while the RDMA client stayed under 3% and the RDMA server used almost none. The average latency of a 2 KB transfer was 25.4 microseconds over TCP against 1.7 microseconds for an RDMA read or write and 2.8 microseconds for an RDMA send [12].

### Kernel bypass and zero copy

RDMA removes the copies and the per-operation system calls. The operating system is used to register memory and to create the communication channel; after that, the application posts work requests directly to the adapter from user space. The adapter segments an outgoing message into packets sized for the path, the packets cross the network, and the receiving adapter reassembles them directly in the destination application's virtual memory. The receiving application is notified only once the whole message has arrived [2]. These properties are usually summarized as kernel bypass (no operating system on the data path) and zero copy (no intermediate buffers) [2][3].

### Verbs, queue pairs and memory registration

The programming interface is called verbs. On Linux, the user-space verbs library is libibverbs, part of the rdma-core package [2][23]. Its main objects are:

- **Queue pair (QP).** The communication endpoint, made of a send queue and a receive queue. NVIDIA's programming manual describes a QP as roughly equivalent to a socket. Both sides must create and initialize a QP before transferring data, and a connection manager can be used to exchange the information needed to connect them [2].
- **Completion queue (CQ).** A first-in, first-out queue of completion entries that tells the application which work requests have finished and with what status. Work queues from many QPs can share one CQ [2].
- **Memory region (MR).** Before an adapter can touch a buffer, the application registers it. Registration pins the pages so they cannot be swapped out, checks permissions, and writes the virtual-to-physical translation into the adapter. Each registration returns a local key (lkey) used by the local adapter and a remote key (rkey) that the application hands to a peer so that the peer's adapter can access that memory [2].
- **Protection domain and memory windows.** A protection domain groups QPs with the memory regions they may use, and memory windows let an application grant and revoke remote access to parts of a registered region more cheaply than re-registering it [2].

### Operations

RDMA offers two styles of transfer. Two-sided operations (send and receive) resemble socket messaging: the receiver must have posted a buffer in advance, and it decides where incoming data lands. One-sided operations (RDMA read and RDMA write) name a remote virtual address and rkey, and they complete without any notification to the remote application, whose only involvement was registering the memory and handing out the key [2]. The IETF version of the protocol uses the same split, calling advertised buffers "tagged" and identifying them with a steering tag (STag) [3].

| Operation | Style | What happens |
|---|---|---|
| Send / receive | Two-sided | Data goes to the next buffer the receiver posted; the receiver is notified [2] |
| Send with immediate | Two-sided | As send, plus a 4-byte value delivered in the completion [2] |
| RDMA write | One-sided | Data is written into a remote registered buffer; the remote side is not notified [2] |
| RDMA write with immediate | One-sided | As RDMA write, but the remote side receives the immediate value as a notification [2] |
| RDMA read | One-sided | Data is fetched from a remote registered buffer into a local one [2] |
| Atomic fetch-and-add, compare-and-swap | One-sided | A remote value is updated atomically and its previous value returned [2] |

### Transport modes

A QP is created with a transport type. A **reliable connection** (RC) links one QP to exactly one other QP and delivers messages reliably and in order, much like a TCP connection. An **unreliable connection** (UC) is also one-to-one but does not retry lost packets. An **unreliable datagram** (UD) QP can exchange single-packet messages with any other UD QP, with no ordering or delivery guarantee, much like UDP, and supports multicast. In the verbs API, RDMA reads and atomic operations are available only on reliable connections [2].

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/6t041Lr5FCY" title="Everything You Wanted to Know About RDMA But Were Too Proud to Ask"></iframe>
<figcaption>"Everything You Wanted to Know About RDMA But Were Too Proud to Ask", video from SNIAVideo, published March 26, 2025. <a href="https://www.youtube.com/watch?v=6t041Lr5FCY">Watch on YouTube</a>.</figcaption>
</figure>

## History

Industry standards for low-latency cluster messaging predate InfiniBand. In December 1997 Compaq, Intel and Microsoft announced the completed Virtual Interface (VI) Architecture specification, an industry-standard interface for high-speed communication between clustered servers over system area networks, to which more than 100 companies had contributed [4]. The InfiniBand Trade Association (IBTA) was formed in August 1999 to build InfiniBand as a high-performance, programmable RDMA fabric [5].

In parallel the IETF standardized RDMA over IP networks. RFC 4296 (architecture) and RFC 4297 (problem statement) appeared in December 2005 [6][1], followed in October 2007 by the RDMA Protocol (RDMAP, RFC 5040), Direct Data Placement (DDP, RFC 5041) and Marker PDU Aligned framing for TCP (MPA, RFC 5044) [3][7][8]. Early versions of these protocols had been written by an industry group, the RDMA Consortium, before the IETF finished its own [8]. The suite, which RFC 7306 (June 2014) defines as iWARP (RFCs 5040, 5041, 5044 and 6581), runs RDMAP over DDP over MPA over TCP, and DDP can also run directly over SCTP [3][31]. RFC 7306 itself extended the protocol with atomic operations and immediate data [31].

The IBTA released the first RoCE specification in 2010, bringing the benefits of RDMA to Ethernet networks [9]. Its successor, RoCEv2, was announced on 16 September 2014; it added the ability to route RoCE traffic across Layer 3 IP networks, which the IBTA presented as a step toward hyperscale deployments [9][10]. IANA registered UDP port 4791 for "IP Routable RocE" in October 2014 [11]. Microsoft's papers on DCQCN congestion control (2015) and on running RoCEv2 across its data centers (2016) documented what it took to make RDMA work at cloud scale [12][13].

| Date | Milestone |
|---|---|
| December 1997 | VI Architecture 1.0 completed by Compaq, Intel and Microsoft [4] |
| August 1999 | InfiniBand Trade Association formed [5] |
| December 2005 | IETF publishes RFC 4296 and RFC 4297 on RDMA over IP [6][1] |
| October 2007 | IETF publishes RDMAP (RFC 5040), DDP (RFC 5041) and MPA (RFC 5044) [3][7][8] |
| 2010 | IBTA releases the first RoCE specification [9] |
| September 2014 | IBTA announces routable RoCEv2 [9] |
| October 2014 | IANA registers UDP port 4791 for routable RoCE [11] |
| August 2015 | Microsoft and Mellanox publish DCQCN at SIGCOMM [12] |
| August 2016 | Microsoft publishes "RDMA over Commodity Ethernet at Scale" at SIGCOMM [13] |
| April 2023 | Microsoft reports about 70% of Azure traffic is RDMA [17] |
| March 2024 | Meta describes two 24,576-GPU clusters, one on RoCE and one on InfiniBand [18] |
| August 2024 | Meta presents its RoCE training network at SIGCOMM [19] |
| June 2025 | Ultra Ethernet Consortium publishes Specification 1.0 [29] |
| June 2026 | MRC transport paper published by AMD, Broadcom, Intel, Microsoft, NVIDIA and OpenAI authors [26] |
| July 2026 | IBTA says its forthcoming Release 2.1 will include MRC [27] |
| August 2026 | Meta announces the MetaRoCE transport [28] |

## RDMA transports

All three families share the verbs interface but differ in their physical, link and transport layers [2].

| | InfiniBand | RoCE (v1 / v2) | iWARP |
|---|---|---|---|
| Standards body | IBTA [5] | IBTA [9] | IETF [3] |
| Carried over | Native InfiniBand links and switches [2] | Ethernet; v2 adds UDP/IP so traffic can be routed [9][10] | TCP (or SCTP) over IP [3] |
| Loss handling | Credit-based link-level flow control [2] | Traditionally a lossless Ethernet configured with Priority Flow Control [2][12] | Reliable, in-order delivery of TCP [8] |
| June 2026 TOP500 systems | 293 [27] | Counted with InfiniBand in a combined 376 [27] | Not reported [27] |

### InfiniBand

InfiniBand was designed around RDMA. Its host channel adapters translate the application's virtual addresses into physical ones so that user programs can drive the adapter directly, its link layer uses credit-based flow control so that switches do not drop packets, and a subnet manager assigns addresses and computes routes [2]. The [InfiniBand](https://aiwiki.ai/wiki/infiniband) article covers the fabric, its speed grades and NVIDIA's Quantum switches in detail.

### RoCE

RoCE is the IBTA's standard for RDMA over Ethernet [2]. The original version stayed within one Ethernet Layer 2 network; RoCEv2 puts the transport inside UDP and IP headers so that standard IP routing, monitoring and firewalling apply, while remaining transparent to applications [9][10][11]. NVIDIA's programming manual describes RoCE as relying on Priority Flow Control (PFC) from Data Center Bridging Ethernet to provide lossless connectivity, and as avoiding the TCP transport that iWARP requires; NVIDIA, as a RoCE vendor, also claims RoCE has lower latency and CPU overhead than iWARP [2].

### iWARP

iWARP runs RDMA over a TCP byte stream. The MPA layer adapts TCP for DDP: it keeps TCP's reliable, in-order delivery while preserving the record boundaries that direct placement needs, and it allows integrated adapters that combine TCP, MPA and DDP in one implementation [8]. Linux support for iWARP adapters is part of rdma-core, which ships an iWARP port-mapper daemon (iwpmd) and user-space support for the software iWARP driver (siw) [23].

## Lossless Ethernet and congestion control

RoCEv2 deployments have relied on a drop-free Ethernet fabric, provided by Priority Flow Control. When a switch or adapter queue passes a threshold, it sends a PAUSE frame upstream and the upstream device stops transmitting on that priority until it receives a RESUME. PFC supports up to eight priority classes, but it pauses a whole port and priority rather than a single flow, which causes head-of-line blocking and unfairness between flows [12].

Microsoft's 2016 account of running RoCEv2 in production for about a year and a half described the problems this created at scale. To route RDMA beyond a single VLAN the company designed a PFC scheme keyed on the IP DSCP field, and it had to deal with PFC-induced deadlock, an RDMA transport livelock, and storms of PFC pause frames sent by network adapters. The authors concluded that these issues could be solved and that RDMA could replace TCP for communication inside a data center [13].

DCQCN (Data Center Quantized Congestion Notification), published by Microsoft and Mellanox in 2015, added end-to-end congestion control to RoCEv2. It builds on QCN (Quantized Congestion Notification) and DCTCP, needs only standard RED and ECN marking from switches, and implements the rest of the algorithm in the network adapters; it was implemented in Mellanox adapters and, at the time of publication, was being deployed in Microsoft's data centers [12].

Meta's AI training network later moved away from DCQCN. Meta first used DCQCN on its RoCE training network but pivoted to letting the collective library manage congestion [19]. For its 400G deployments Meta ran without DCQCN, relying on PFC plus receiver-driven admission in which the receiver grants permission to send with clear-to-send packets, and reported stable performance for more than a year [20]. The Llama 3 paper likewise reports running a 24K-GPU cluster without DCQCN, using deep-buffer spine switches and better load balancing instead [21].

## GPUDirect RDMA

GPUDirect RDMA is an NVIDIA technology, introduced with Kepler-class GPUs and CUDA 5.0, that gives a third-party PCI Express device such as a network adapter a direct data path to GPU memory using standard features of PCI Express [14]. The adapter reads and writes GPU memory through the GPU's PCI BAR window, and NVIDIA's documentation says the nvidia-peermem kernel module lets InfiniBand and RoCE adapters (ConnectX-3 VPI or newer) access GPU memory "without needing to copy data to host memory" [14]. NVIDIA describes it as one member of a family that also includes GPUDirect Peer to Peer (direct GPU-to-GPU copies over [PCIe](https://aiwiki.ai/wiki/pcie) or [NVLink](https://aiwiki.ai/wiki/nvlink)), GPUDirect Storage and GPUDirect for Video [15].

The data path depends on the server's PCIe topology. NVIDIA's documentation requires the GPU and the adapter to share an upstream PCIe root complex, says the best performance comes when only PCIe switches sit between them, and warns that a path crossing the processor interconnect between CPU sockets may be very slow or unreliable. The IOMMU must be disabled or set to pass-through mode [14].

In the cloud, Amazon Web Services exposes the same idea under its own names. Its Elastic Fabric Adapter supports RDMA read on all Nitro v4 and later instances and RDMA write on most of them; "Accelerator Data Transfer" corresponds to GPUDirect RDMA on NVIDIA GPU instances, and "Accelerator Command Initiation", which lets the accelerator start transfers itself, corresponds to GPUDirect Async [22].

## Role in AI clusters

### Collective communication

[Distributed training](https://aiwiki.ai/wiki/distributed_training) requires GPUs to exchange data constantly. In Meta's description, the four to eight GPUs inside a training node talk over NVLink, while communication between nodes crosses the network, and Meta's paper says that using TCP/IP sockets for that traffic is prone to degraded performance from CPU overhead and added latency [19]. NVIDIA's collective library NCCL has an "IB/RoCE" transport built on InfiniBand verbs and falls back to IP sockets when that transport is disabled. Its environment variables select which host channel adapters to use and which global ID index to use in RoCE mode [16]. DeepSeek's expert-parallel library DeepEP lists "NVLink for intranode communication" and an "RDMA network for internode communication" among its requirements, and as of November 2026 its README says it is fully tested on InfiniBand and theoretically compatible with RoCE [30].

### Meta's RoCE clusters

Meta has published detailed accounts of RDMA in its AI training clusters. In March 2024 it described two clusters of 24,576 [NVIDIA H100](https://aiwiki.ai/wiki/nvidia_h100) GPUs, one built on RoCE with Arista 7800 switches and Wedge400 and Minipack2 rack switches, and one built on NVIDIA Quantum2 InfiniBand, both connecting 400 Gb/s endpoints [18]. Its SIGCOMM 2024 paper gives three reasons for choosing RoCE: it keeps the RDMA verbs semantics that training software already used, it let Meta reuse much of its existing Ethernet data center design and tooling, and it rests on open standards with support from multiple vendors [19]. Meta also separated GPU training traffic into a dedicated "backend" network so that it could evolve independently of the rest of the data center [19].

The Llama 3 paper describes the RoCE cluster used for Llama 3 405B. Each rack held 16 GPUs in two servers under one Minipack2 top-of-rack switch; 192 racks formed a pod of 3,072 GPUs with full bisection bandwidth; and eight pods were joined by aggregation switches into a 24K-GPU cluster, with a 1:7 oversubscription at the aggregation layer. Smaller Llama 3 models were trained on the InfiniBand cluster, and Meta says it tuned both to give equivalent performance for these workloads [21].

Load balancing turned out to be a central problem. Training produces a small number of large flows, which standard equal-cost multipath (ECMP) hashing spread poorly across the available paths [20][21]. For Llama 3, Meta's collective library opened 16 flows between each pair of GPUs, and an enhanced ECMP scheme hashed on additional fields in the RoCE header [21]. Meta's engineering blog says hashing on the destination queue pair field, combined with more queue pairs per connection, improved AllReduce performance by up to 40% [20].

### Storage and other upper-layer protocols

RDMA also carries storage and remote procedure call traffic. Standards include iSCSI Extensions for RDMA (iSER, RFC 7145) and an RDMA transport for Remote Procedure Call (RFC 8166), and NVIDIA lists SCSI RDMA Protocol, SMB, Lustre and NFS among the protocols that run over its RDMA adapters [24][25][2]. Microsoft uses RDMA for Azure Storage both between compute virtual machines and storage clusters and inside the storage clusters, and supports intra-region RDMA in all Azure public regions [17].

### Software stack

On Linux the kernel's InfiniBand subsystem and the rdma-core user-space package provide the RDMA stack for all three transports. rdma-core includes libibverbs, the librdmacm connection manager and libibumad, plus user-space providers for many adapter drivers, among them mlx5_ib, bnxt_re, irdma, efa, mana_ib and ionic_rdma. It also supports two kernel software implementations that run over an ordinary network interface: rdma_rxe (soft-RoCE) and siw (soft-iWARP) [23]. Applications reach AWS's Elastic Fabric Adapter through the Libfabric API, and the adapter uses Amazon's own Scalable Reliable Datagram (SRD) protocol for congestion control and reliability [22].

## Limits of classic RoCE and newer transports

The authors of the MRC paper summarize three problems for large training clusters: RC is single-path and recovers from loss with go-back-N retransmission, its usual congestion model (lossless Ethernet with PFC, or DCQCN) does not scale to large clusters, and it is very vulnerable to link and fabric failures [26]. Meta's engineers likewise describe the probabilistic nature of hash-based load balancing as a lasting drawback [20].

Three efforts in 2025 and 2026 aim to fix this while keeping RDMA semantics:

| Transport | Sponsor | Approach |
|---|---|---|
| Ultra Ethernet Transport | [Ultra Ethernet](https://aiwiki.ai/wiki/ultra_ethernet) Consortium | Specification 1.0 published June 2025, described by the consortium as modern RDMA for Ethernet and IP [29] |
| Multipath Reliable Connection (MRC) | Authors from [AMD](https://aiwiki.ai/wiki/amd), [Broadcom](https://aiwiki.ai/wiki/broadcom), [Intel](https://aiwiki.ai/wiki/intel), [Microsoft](https://aiwiki.ai/wiki/microsoft), [NVIDIA](https://aiwiki.ai/wiki/nvidia) and [OpenAI](https://aiwiki.ai/wiki/openai); developed under the [Open Compute Project](https://aiwiki.ai/wiki/open_compute_project) | Extends RoCEv2 RC with per-packet multipath, sender-based congestion control, selective acknowledgments and fast failover [26] |
| MetaRoCE | Meta | Treats Ethernet as lossy (no PFC), sprays packets over many paths, places data out of order, retransmits only lost packets [28] |

MRC keeps RC's reliable, ordered completion semantics for software while spraying the packets of one connection across many paths, and its authors say it combines ideas from Ultra Ethernet and the IBTA and is already in production [26]. In July 2026 the IBTA said MRC would be the most significant addition in its forthcoming InfiniBand Architecture Specification Release 2.1 (Volumes 1 and 2), describing it as an extension of the RoCEv2 reliable connection model with multipath support, contributed jointly by AMD, Broadcom, Intel, Microsoft, NVIDIA and OpenAI [27].

Meta announced MetaRoCE in August 2026. It gives each connection several paths, each with its own UDP source port so that switches hash them differently, tracks round-trip time and ECN marks per path, uses a 256-bit selective-acknowledgment bitmap per path, and combines ECN-based rate control with fair-share rate hints from the receiver. Meta says existing verbs software works without modification, reported about 86% throughput at 1% packet loss on a 64-node AMD GPU cluster with [AMD Pensando](https://aiwiki.ai/wiki/amd_pensando) adapters, and said it would release the specification, a reference implementation and a compliance suite through the Open Compute Project [28]. At the OCP Global Summit in October 2026, Meta announced open-source MetaRoCE implementations [32].

## Adoption

RDMA fabrics connect most TOP500 supercomputers. According to the IBTA's analysis of the June 2026 list, InfiniBand connected 293 systems (59% of the list, up from 276 in November 2025), and InfiniBand and Ethernet with RoCE together connected 376 systems, or 75% of the list; InfiniBand connected 62 of the top 100 [27]. In commercial AI, Meta trained Llama 3 models on RoCE and InfiniBand clusters side by side [21], about 70% of Azure's traffic was RDMA as of 2023 [17], and AWS offers RDMA operations through its own SRD-based adapter [22].

## See also

- [InfiniBand](https://aiwiki.ai/wiki/infiniband)
- [Ultra Ethernet](https://aiwiki.ai/wiki/ultra_ethernet)
- [NVIDIA Spectrum-X](https://aiwiki.ai/wiki/nvidia_spectrum_x)
- [NVLink](https://aiwiki.ai/wiki/nvlink)
- [NCCL](https://aiwiki.ai/wiki/nccl)
- [ConnectX](https://aiwiki.ai/wiki/connectx)
- [BlueField](https://aiwiki.ai/wiki/bluefield)
- [Distributed training](https://aiwiki.ai/wiki/distributed_training)
- [Distributed computing](https://aiwiki.ai/wiki/distributed_computing)
- [Data center](https://aiwiki.ai/wiki/data_center)

## References

1. Romanow, A., Mogul, J., Talpey, T., Bailey, S. "Remote Direct Memory Access (RDMA) over IP Problem Statement". RFC 4297, IETF, December 2005. https://www.rfc-editor.org/rfc/rfc4297.html
2. NVIDIA. "RDMA Aware Networks Programming User Manual", Rev. 1.7 (August 2020). NVIDIA documentation. https://docs.nvidia.com/rdma-aware-networks-programming-user-manual-1-7.pdf
3. Recio, R., Metzler, B., Culley, P., Hilland, J., Garcia, D. "A Remote Direct Memory Access Protocol Specification". RFC 5040, IETF, October 2007. https://www.rfc-editor.org/rfc/rfc5040.html
4. Microsoft. "Compaq, Intel and Microsoft Announce Completion of the Virtual Interface Architecture Specification". Microsoft News, December 16, 1997. https://news.microsoft.com/1997/12/16/compaq-intel-and-microsoft-announce-completion-of-the-virtual-interface-architecture-specification/
5. InfiniBand Trade Association. "The IBTA Celebrates 20 Years of Growth and Industry Success". IBTA, August 27, 2019. https://infinibandta.org/the-ibta-celebrates-20-years-of-growth-and-industry-success/
6. Bailey, S., Talpey, T. "The Architecture of Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) on Internet Protocols". RFC 4296, IETF, December 2005. https://www.rfc-editor.org/rfc/rfc4296.html
7. Shah, H., Pinkerton, J., Recio, R., Culley, P. "Direct Data Placement over Reliable Transports". RFC 5041, IETF, October 2007. https://www.rfc-editor.org/rfc/rfc5041.html
8. Culley, P., Elzur, U., Recio, R., Bailey, S., Carrier, J. "Marker PDU Aligned Framing for TCP Specification". RFC 5044, IETF, October 2007. https://www.rfc-editor.org/rfc/rfc5044.html
9. InfiniBand Trade Association. "InfiniBand Trade Association Releases Updated Specification for Remote Direct Memory Access over Converged Ethernet (RoCE)". IBTA, September 16, 2014. https://www.infinibandta.org/?p=1250
10. Chirgwin, R. "Oi! You noisy servers! Talk among yourselves and stop bothering that poor router!". The Register, September 17, 2014. https://www.theregister.com/2014/09/17/rocev2_spec_lands/
11. Internet Assigned Numbers Authority. "Service Name and Transport Protocol Port Number Registry" (entry: roce, UDP 4791, "IP Routable RocE", registered 2014-10-17). IANA. https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?search=4791
12. Zhu, Y., Eran, H., Firestone, D., Guo, C., Lipshteyn, M., Liron, Y., Padhye, J., Raindel, S., Haj Yahia, M., Zhang, M. "Congestion Control for Large-Scale RDMA Deployments". Proceedings of ACM SIGCOMM 2015, August 2015. https://www.microsoft.com/en-us/research/publication/congestion-control-for-large-scale-rdma-deployments/
13. Guo, C., Wu, H., Deng, Z., Soni, G., Ye, J., Padhye, J., Lipshteyn, M. "RDMA over Commodity Ethernet at Scale". Proceedings of ACM SIGCOMM 2016, August 2016. https://www.microsoft.com/en-us/research/publication/rdma-commodity-ethernet-scale/
14. NVIDIA. "GPUDirect RDMA" (GPUDirect RDMA 13.5 documentation). NVIDIA CUDA documentation. https://docs.nvidia.com/cuda/gpudirect-rdma/index.html
15. NVIDIA. "NVIDIA GPUDirect". NVIDIA Developer. https://developer.nvidia.com/gpudirect
16. NVIDIA. "Environment Variables". NCCL 2.32.3 documentation. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html
17. Bai, W., et al. "Empowering Azure Storage with RDMA". 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI '23), April 2023. https://www.usenix.org/conference/nsdi23/presentation/bai
18. Lee, K., Gangidi, A., Oldham, M. "Building Meta's GenAI Infrastructure". Engineering at Meta, March 12, 2024. https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/
19. Gangidi, A., Miao, R., Zheng, S., Bondu, S. J., Goes, G., Morsy, H., Puri, R., Riftadi, M., Shetty, A. J., Yang, J., Zhang, S., Jimenez Fernandez, M., Gandham, S., Zeng, H. "RDMA over Ethernet for Distributed AI Training at Meta Scale". Proceedings of ACM SIGCOMM 2024, August 2024. https://doi.org/10.1145/3651890.3672233
20. Gangidi, A., Zeng, J. H. "RoCE networks for distributed AI training at scale". Engineering at Meta, August 5, 2024. https://engineering.fb.com/2024/08/05/data-center-engineering/roce-network-distributed-ai-training-at-scale/
21. Llama Team, AI @ Meta. "The Llama 3 Herd of Models". arXiv:2407.21783, July 2024. https://arxiv.org/abs/2407.21783
22. Amazon Web Services. "Elastic Fabric Adapter for AI/ML and HPC workloads on Amazon EC2". Amazon EC2 User Guide. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html
23. linux-rdma. "rdma-core: RDMA core userspace libraries and daemons". GitHub. https://github.com/linux-rdma/rdma-core
24. Ko, M., Nezhinsky, A. "Internet Small Computer System Interface (iSCSI) Extensions for the Remote Direct Memory Access (RDMA) Specification". RFC 7145, IETF, April 2014. https://www.rfc-editor.org/rfc/rfc7145.html
25. Lever, C. (ed.), Simpson, W., Talpey, T. "Remote Direct Memory Access Transport for Remote Procedure Call Version 1". RFC 8166, IETF, June 2017. https://www.rfc-editor.org/rfc/rfc8166.html
26. Sohan, R., Spada, E., Davis, E., Handley, M., et al. "The Multipath Reliable Connection (MRC) Transport". arXiv:2606.18170, June 2026 (revised November 2026). https://arxiv.org/abs/2606.18170
27. InfiniBand Trade Association. "RDMA Networking Powers the June 2026 TOP500 while IBTA Advances New Specifications". IBTA, July 31, 2026. https://infinibandta.org/rdma-networking-powers-the-june-2026-top500-while-ibta-advances-new-specifications/
28. Srinivasan, A., Spring, N., Baldonado, O., Krishnamurthy, R. "MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet". Engineering at Meta, August 24, 2026. https://engineering.fb.com/2026/08/24/networking-traffic/metaroce-rdma-transport-ai-ethernet/
29. Ultra Ethernet Consortium. "Ultra Ethernet Consortium (UEC) Launches Specification 1.0 Transforming Ethernet for AI and HPC at Scale". UEC, June 11, 2025. https://ultraethernet.org/ultra-ethernet-consortium-uec-launches-specification-1-0-transforming-ethernet-for-ai-and-hpc-at-scale/
30. deepseek-ai. "DeepEP". GitHub. https://github.com/deepseek-ai/DeepEP
31. Shah, H., Marti, F., Noureddine, W., Eiriksson, A., Sharp, R. "Remote Direct Memory Access (RDMA) Protocol Extensions". RFC 7306, IETF, June 2014. https://www.rfc-editor.org/rfc/rfc7306.html
32. Sunkad, R., Baldonado, O. "OCP Summit 2026: Powering AI with Open Innovations in Scale-Up, Scale-Out, and Scale-Across Networks". Engineering at Meta, October 12, 2026. https://engineering.fb.com/2026/10/12/open-source/ocp-summit-2026-powering-ai-with-open-innovations-in-scale-up-scale-out-and-scale-across-networks/

