NCCL, the NVIDIA Collective Communications Library, is a library of topology-aware communication primitives for NVIDIA GPU systems. It supplies collective operations such as all-reduce, broadcast, reduce, all-gather, reduce-scatter, all-to-all, gather, and scatter, together with point-to-point send and receive. NCCL is designed to move data directly between GPU buffers over available interconnects, including PCI Express, NVLink, InfiniBand, and IP networks. Its API resembles the collective portion of MPI, but operations are associated with CUDA streams rather than being ordinary host-blocking calls.[1][2]

NCCL is a communications component, not a complete distributed training framework. Higher-level systems create process groups, divide models or data among workers, schedule computation, and invoke NCCL through a framework backend. This division of responsibility makes NCCL important to multi-GPU deep learning without making it responsible for a training algorithm, optimizer, checkpoint format, or failure-recovery policy.[1][3]

Role in distributed GPU computing

Distributed GPU workloads repeatedly transform data that is partitioned across ranks. In data parallelism, each rank commonly computes local gradients and then participates in an all-reduce or a reduce-scatter followed by an all-gather. Sharded training uses all-gather to materialize parameters and reduce-scatter to repartition gradients. Tensor and sequence parallel implementations exchange activations or partial results with collective and point-to-point operations. NCCL implements the communication layer used by many such workloads while the framework determines when each operation is issued.[24][25][26]

An NCCL communicator defines a set of ranks and the communication context shared by them. Applications can create a communicator from a unique identifier distributed out of band, or initialize all devices within one process with ncclCommInitAll. A collective call names device buffers, an element count, a datatype, an operation where applicable, a communicator, and a CUDA stream. The call enqueues GPU work; completion relative to later GPU work is governed by stream ordering and CUDA synchronization.[1][15]

This design has two practical consequences. First, NCCL can discover the machine topology and choose paths and algorithms without requiring the training code to encode every link. Second, correct use still depends on application-wide agreement: ranks must issue compatible operations in a compatible order, and buffer lifetimes must extend until the queued work is complete. NCCL cannot repair an application that disagrees about counts, datatypes, roots, or operation order.[2][14][15]

History and release development

NVIDIA announced NCCL in April 2016 after developing it as a research project for efficient single-node collectives. The original public description emphasized five collectives, ring algorithms, CUDA kernels, and GPUDirect peer-to-peer transfers within a multi-GPU server.[6] The scope expanded to multi-node communication in the NCCL 2 series. NVIDIA described NCCL 2.3 in September 2018 as fully open source and documented socket, InfiniBand Verbs, and GPUDirect RDMA paths for communication between servers.[7]

The library has since added algorithms, transports, APIs, and recovery mechanisms rather than following a single algorithmic line:

Period	Development
2016	The initial public release focused on single-node GPU collectives implemented with CUDA kernels and ring-based movement.[6]
2018	NCCL 2.3 expanded the documented multi-node and open-source scope, including sockets, InfiniBand, and GPUDirect RDMA.[7]
2019	NCCL 2.4 introduced double binary trees for lower-latency all-reduce at large scale while retaining rings where they performed better.[8]
2020	NCCL 2.7 added point-to-point `ncclSend` and `ncclRecv`, enabling grouped peer exchanges and communication patterns beyond the original collectives.[9]
2023 to early 2025	Later 2.x releases added NVLink SHARP support, scalable initialization, and the Parallel Aggregated Trees algorithm for all-gather and reduce-scatter.[10][16]
July 2025	NCCL 2.27 added symmetric-memory features, communicator shrinking, and Direct NIC support on qualifying Grace Blackwell systems. NVIDIA also reported improvements from NVLink and InfiniBand SHARP paths.[17]
Late 2025	NCCL 2.28 introduced an experimental device API, dedicated host APIs for all-to-all, gather, and scatter, and copy-engine collective support.[18][21]
Late 2025 to February 2026	NCCL 2.29 added one-sided host operations, communicator growth, monitoring support, and the initial NCCL4Py package. The 2.29.7 release changed most project code to the Apache License 2.0 while retaining some BSD-licensed portions.[19][33][40]
April to June 2026	NCCL 2.30 added features including GIN-based networking work, TMA-selected symmetric kernels, and further one-sided and copy-engine development. NVIDIA listed NCCL 2.30.7 for download on June 9, 2026.[4][5][20]

Version availability depends on the CUDA and platform packages supplied by NVIDIA or a downstream framework. A feature shown in current documentation therefore cannot be assumed to exist in every NCCL version shipped with every training environment. Applications that rely on a recently added operation, algorithm, or environment-variable value should check the installed NCCL version rather than treating the current manual as a historical description.[4][10]

Communication operations

NCCL operations use ranks numbered from zero to n - 1. Collective semantics follow familiar MPI-style patterns, but the input and output are GPU-accessible buffers and execution is stream ordered.[1][2]

Operation	Result
All-reduce	Reduces corresponding elements from every rank and returns the result to every rank.
Broadcast	Copies a root rank's buffer to every rank.
Reduce	Reduces corresponding elements to a designated root rank.
All-gather	Concatenates one contribution from each rank and returns the complete sequence to every rank.
Reduce-scatter	Reduces corresponding inputs and distributes disjoint result segments among ranks.
All-to-all	Sends a distinct equal-sized segment from every rank to every rank.
Gather	Collects one contribution from each rank at a root rank.
Scatter	Distributes distinct root-buffer segments to the ranks.
Send and receive	Transfers data between a named sender and receiver. Both sides must participate with compatible counts and datatypes.

The dedicated all-to-all, gather, and scatter host APIs entered the documented library in NCCL 2.28.3. Older NCCL documentation showed how applications could compose these patterns from grouped send and receive operations, so source code written for an older release can express the same pattern differently.[9][18]

Point-to-point communication is two-sided: a send must be matched by a receive. NCCL documentation recommends placing mutually dependent sends and receives in a group so that they can be scheduled together instead of allowing one call to wait for a peer operation that has not yet been submitted. NCCL 2.29 separately introduced one-sided host APIs, including put, get, and signal operations, which have different setup and synchronization requirements.[9][19]

Reduction operations support documented numerical operators over supported datatypes. The exact datatype and operator set is version dependent. Applications should consult the API documentation for their installed release, especially for pre-multiplied sum operators or low-precision formats, rather than infer support from CUDA's datatype catalog.[2][4]

Execution model

For the conventional host-side collective interface, NCCL normally launches communication work as CUDA kernels. NVIDIA's overview describes each collective as a single kernel that performs both communication and local computation, reducing the need for host-side staging between phases.[1] NCCL 2.28 added an experimental device API that instead lets user kernels perform supported communication operations through a device communicator. The device API includes lower-level synchronization and memory-access concepts and should not be conflated with the long-established host API.[21]

NCCL selects an execution plan using message size, rank count, discovered topology, available transports, and enabled algorithms and protocols. A 2025 IEEE HOT Interconnects paper that profiled NCCL 2.22 found materially different algorithm and protocol choices across platforms and message ranges. That study is evidence about the measured release and systems, not a promise that every later release makes the same choice.[11]

The library also distinguishes algorithms from wire or kernel protocols. Current configuration documentation lists algorithm names including Ring, Tree, CollnetChain, CollnetDirect, NVLS, NVLSTree, and PAT. It lists the LL, LL128, and Simple protocols. By default, NCCL chooses automatically; setting NCCL_ALGO or NCCL_PROTO constrains that choice and can reduce performance or, for an unsupported LL128 combination, risk data corruption.[10]

NCCL group calls have three documented uses:

managing multiple GPUs from one host thread;
aggregating communication operations to reduce launch overhead; and
submitting concurrent point-to-point operations as one scheduling unit.[14]

ncclGroupStart and ncclGroupEnd delimit the group. Calls inside the group are not guaranteed to be complete or independently observable before the group is closed. All participating ranks must still issue operations in a globally compatible order.[14][15]

Algorithms and protocols

Ring algorithms

In a ring all-reduce, ranks are arranged in a logical cycle. A reduce-scatter circulates chunks while accumulating partial reductions, followed by an all-gather that circulates the reduced chunks. For large messages, the amount transferred per rank approaches twice the data size as the rank count grows. Patarasuk and Yuan analyzed bandwidth-optimal ring-style all-reduce algorithms under a standard communication model; their result helps explain the ring's strong large-message bandwidth behavior, although an NCCL implementation also depends on real topology and protocol costs.[12]

The ring has a linear number of communication steps in the rank count. Its throughput can be excellent when links are well utilized, but its latency becomes less attractive for small messages or very large groups. NCCL can create multiple channels and map logical rings onto the physical topology so that more than one path carries chunks concurrently.[6][7][11]

Double binary trees

NCCL 2.4 added double binary trees for all-reduce. Two complementary trees split the data so that most ranks are an internal node in one tree and a leaf in the other. The tree depth grows logarithmically with rank count, reducing the number of latency-bearing steps relative to a ring while using both trees to maintain bandwidth. NVIDIA reported large gains on the Summit system for small all-reduce messages and stated that NCCL continued to choose rings where ring performance was better.[8] The underlying two-tree approach has also been studied in peer-reviewed collective-communication research.[13]

PAT, NVLS, CollNet, and network offload

Parallel Aggregated Trees, or PAT, was introduced for all-gather and reduce-scatter. NVIDIA describes PAT as using logarithmic communication steps while retaining bandwidth efficiency and avoiding the power-of-two restrictions of classic recursive doubling. It targets cases where ring step count dominates latency.[16]

NVLS uses collective support exposed by recent NVLink and NVSwitch systems. CollNet algorithms use a network-aware structure rather than treating every inter-node link as an ordinary ring edge. SHARP-capable fabrics can offload or accelerate parts of a collective in supported configurations. These names do not identify one universally superior path: hardware generation, switch firmware, message size, operation, rank layout, plugin availability, and NCCL version all affect whether NCCL can select them.[10][17]

The device API added further choices such as multimemory-based collectives and GIN networking. These interfaces expose capabilities to application kernels and have more demanding memory-registration and synchronization rules than a normal host ncclAllReduce call. Their presence does not eliminate the conventional host-side algorithms.[21]

Topology and transports

NCCL discovers relationships among GPUs, CPU sockets, PCI Express switches, network interfaces, and supported high-speed links. It uses this information to select paths and to construct channels, rings, trees, and other plans. Topology awareness matters because a logical rank order that crosses a slow CPU or PCI Express boundary unnecessarily can waste available bandwidth.[1][11]

Within a host, NCCL can use direct GPU peer-to-peer paths over NVLink or PCI Express where the platform and driver permit them. When direct peer access is unavailable, it can use shared-memory paths. Across hosts, documented transports include InfiniBand Verbs and IP sockets. GPUDirect RDMA allows a supported network adapter to access GPU memory without staging the payload through ordinary host buffers, subject to the GPU, NIC, PCI Express, driver, and memory-registration requirements of the system.[1][7][10]

Topology discovery is constrained by what the operating system and container expose. NCCL troubleshooting guidance specifically calls out GPU peer-to-peer capability and PCI Access Control Services settings as possible causes of degraded or failed direct transfers. Container and virtual-machine environments can also hide or misrepresent devices, shared memory, or network interfaces. The selected path should therefore be verified with NCCL logs and a controlled test rather than inferred solely from a hardware inventory.[36][37][38]

The principal data paths can be summarized as follows:

Scope	Examples	Important constraints
GPU to GPU within a host	NVLink, PCI Express peer-to-peer	Peer access, topology, PCIe routing, and platform ACS behavior
GPU exchange through host-accessible memory	Shared memory or cuMem host allocation	Container shared-memory limits, NUMA support, and NCCL version
Host to host	InfiniBand Verbs or IP sockets	Interface selection, routing, firewall and port range, NIC plugin, and RDMA configuration
Accelerated collective fabric	NVLS, CollNet, InfiniBand SHARP, GIN on supported releases	Specific hardware, software, topology, and release requirements

Programming model and correctness

Most multi-process applications obtain an ncclUniqueId on one rank, distribute it using another control plane, and call ncclCommInitRank on every rank with the same rank count and unique identifier. Single-process applications can use ncclCommInitAll. Framework deployments commonly assign one process to each GPU, while NCCL also supports one process managing multiple devices through grouped initialization and calls.[14][15][24]

NCCL operations are asynchronous with respect to the host once successfully enqueued. In the default blocking communicator mode, some API calls can block during initialization or other setup. A communicator created with nonblocking configuration can return ncclInProgress; the application then polls ncclCommGetAsyncError until the operation completes or reports an error. This NCCL communicator mode is distinct from framework-specific environment variables that contain the string NCCL.[15][39]

After an asynchronous communication error, the communicator may need to be aborted and recreated. Newer releases also support ncclCommShrink, which creates a communicator that excludes failed or unwanted ranks under documented constraints, and NCCL 2.29 added ncclCommGrow. These APIs provide building blocks; they do not by themselves reconstruct model state, replay a failed optimizer step, or choose a new data partition.[15][17][19]

Applications using multiple communicators must maintain a consistent host launch order across ranks. Current NCCL documentation describes implicit launch ordering, available since NCCL 2.26, as an aid when enabled, but still requires matching host-side issue order. Mismatched order can deadlock. The same general rule applies to collective signatures: all ranks in a collective must agree on the operation and compatible buffer layout.[14][15]

CUDA Graph capture is supported for documented NCCL operations and releases, but capture does not remove ordering requirements. Graph registration and buffer-registration features are version and topology dependent. Applications should treat graph capture, memory registration, and one-sided communication as separate opt-in mechanisms whose constraints must be checked against the installed release.[4][15][20]

Framework integration

NCCL is commonly reached through a framework process-group abstraction rather than called directly:

PyTorch documents NCCL as the recommended backend for CUDA GPU distributed training. DistributedDataParallel divides parameters into buckets so that gradient reduction can overlap with backward computation when gradients become ready. The overlap is orchestrated by PyTorch; NCCL performs the submitted communication.[24][25]
FSDP uses process groups for parameter all-gathers and gradient reduce-scatters in its sharded execution. The FSDP design paper describes the relationship between sharding, computation, and collectives, while the implementation controls their schedule.[26][27]
DeepSpeed initializes distributed communication with NCCL as its documented default backend for GPU training, including when an MPI launcher supplies rank metadata.[28]
Megatron-LM constructs NCCL-backed process groups for forms of model parallelism and pipeline parallelism. Its current source also exposes NCCL configuration options for selected groups.[29]
TensorFlow provides an NcclAllReduce cross-device implementation for MirroredStrategy. That class is one integration point; it does not imply that every TensorFlow collective or deployment uses NCCL.[30]
Horovod described a ring-all-reduce approach for distributed training and supports GPU communication through NCCL in appropriate builds. Its orchestration layer remains separate from NCCL.[31]
JAX CUDA distributions depend on compatible NCCL releases for supported multi-GPU communication features. The exact lowering and feature set vary by JAX, XLA, CUDA, and NCCL version, so it is more accurate to verify a specific JAX release than to claim that every JAX collective maps directly to NCCL.[32]

Framework variables must not be confused with library variables. For example, current PyTorch documentation lists variables such as TORCH_NCCL_USE_COMM_NONBLOCKING. Names like these configure PyTorch's ProcessGroupNCCL behavior. NCCL's own variables generally use the NCCL_ prefix and are documented in the NCCL environment-variable reference.[10][39]

Performance measurement

The nccl-tests project supplies correctness and performance executables for NCCL operations. Tests report algorithm bandwidth, computed from payload size divided by measured time, and bus bandwidth, a normalization intended to make results more comparable across collective types. For all-reduce with n ranks, the documented conversion is:

bus bandwidth = algorithm bandwidth * 2 * (n - 1) / n

For all-gather and reduce-scatter, the factor is (n - 1) / n; broadcast and reduce use a factor of one. These values are analytical normalizations, not direct counters for every byte traversing every physical link.[22][23]

Meaningful benchmarks record operation, datatype, payload range, rank count, GPUs per process, process placement, NCCL and CUDA versions, network plugin, topology, and whether results are in-place. Warm-up, clock state, concurrent workload, and CPU affinity can also affect measurements. A single bandwidth number without this context cannot establish expected H100, Blackwell, or cluster-wide performance.

NCCL debug output can reveal the selected interfaces, channels, algorithms, protocols, and topology decisions, but logging itself can perturb a test. NVIDIA classifies several debug variables as troubleshooting aids and advises against leaving them set in production. Performance reports should state any forced algorithm or protocol because such settings disable part of NCCL's automatic selection.[10]

Configuration and troubleshooting

NCCL reads configuration from environment variables and, in current releases, from configuration files. Variables fall into system-configuration and debugging categories. The official environment reference is versioned, so a deployment should use the manual that matches its installed library.[10]

Variable	Purpose
`NCCL_SOCKET_IFNAME`	Selects or excludes IP interface name prefixes for socket communication.
`NCCL_IB_HCA`	Selects or excludes InfiniBand/RoCE interfaces and ports.
`NCCL_NET_GDR_LEVEL`	Controls the topology distance at which GPU Direct RDMA is used.
`NCCL_P2P_DISABLE`	Disables direct GPU peer-to-peer transport for diagnosis or configuration.
`NCCL_ALGO`	Restricts enabled algorithms. Leaving it unset permits automatic selection.
`NCCL_PROTO`	Restricts enabled protocols. NVIDIA warns that forcing LL128 where it is unsupported can corrupt data.
`NCCL_DEBUG`	Controls NCCL logging level. `NCCL_DEBUG_SUBSYS` narrows logged subsystems.
`NCCL_TOPO_DUMP_FILE`	Writes detected topology information to a file for diagnosis.

Common failure classes have different remedies:

Peer-to-peer failure: Verify CUDA peer access and the platform's PCI Express ACS and IOMMU behavior. Disabling P2P can isolate the path, but it is a diagnostic or deliberate configuration choice, not a universal fix.[36]
Shared-memory failure: Containers may provide insufficient /dev/shm. Current NCCL releases can use cuMem host allocations under supported CUDA driver/runtime and NUMA conditions, with documented fallback behavior in later releases. Container NUMA exposure and shared-memory limits remain relevant.[37]
Network hang or unexpected socket path: NCCL may select an interface that is administratively up but cannot reach peer ranks. Pinning NCCL_SOCKET_IFNAME, checking routing, and allowing the configured TCP port range can distinguish interface selection from firewall failure.[38]
Collective deadlock: Confirm that every rank reaches the same operation with matching count, datatype, root, and order. For point-to-point exchanges, submit mutually dependent operations in a group.[2][9][14]
Asynchronous error: Poll the communicator in nonblocking mode, propagate failure through the application control plane, and abort or shrink the communicator as the chosen recovery design requires.[15][17]
Unexpectedly low bandwidth: Run nccl-tests, inspect topology and selected transports, compare against the correct bus-bandwidth formula, and remove old forced algorithm or debug settings before drawing a hardware conclusion.[10][22][23]

Because environment-variable behavior changes over time, copying an old tuning recipe can disable a newer algorithm or force a protocol that was valid only on another platform. Automatic selection is the appropriate baseline for comparison; overrides should answer a measured, reproducible problem.

Related libraries and licensing

MPI implementations can provide GPU-aware collectives and broader message-passing functionality. Open MPI, for example, documents CUDA-aware collective components and cases in which GPU buffers may be staged through host memory. NCCL has a narrower GPU-communication scope and integrates through CUDA streams, while MPI supplies a larger programming model that includes host communication, communicators, datatypes, and process coordination.[1][35]

AMD's RCCL provides collective communication for ROCm GPU systems and intentionally exposes NCCL-like API names. Hardware and software requirements differ, so source-level similarity should not be described as universal binary compatibility or identical performance.[34]

The NCCL repository is publicly available. Its current license file states that most of the project is licensed under the Apache License 2.0 and that some portions retain their original three-clause BSD license. This replaced the older, simpler characterization of NCCL as wholly BSD-licensed and should be checked again if the project changes its licensing layout.[3][33]

References

[1] NVIDIA, "Overview," NCCL User Guide, version 2.30.7. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/overview.html

[2] NVIDIA, "Collective Operations," NCCL User Guide. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html

[3] NVIDIA, "NVIDIA Collective Communication Library (NCCL)," GitHub repository. https://github.com/NVIDIA/nccl

[4] NVIDIA, "NCCL 2.30.7 Release Notes," 2026. https://docs.nvidia.com/deeplearning/nccl/archives/nccl_2307/release-notes/rel_2-30-7.html

[5] NVIDIA, "NCCL Downloads," listing NCCL 2.30.7 on June 9, 2026. https://developer.nvidia.com/nccl/nccl-download

[6] Sylvain Jeaugey, "Fast Multi-GPU Collectives with NCCL," NVIDIA Technical Blog, April 7, 2016. https://developer.nvidia.com/blog/fast-multi-gpu-collectives-nccl/

[7] NVIDIA, "NCCL 2.3: Scaling Multi-GPU Applications Across Multiple Nodes," NVIDIA Technical Blog, September 26, 2018. https://developer.nvidia.com/blog/?p=12093

[8] Sylvain Jeaugey, "Massively Scale Your Deep Learning Training with NCCL 2.4," NVIDIA Technical Blog, February 4, 2019. https://developer.nvidia.com/blog/massively-scale-deep-learning-training-nccl-2-4/

[9] NVIDIA, "Point-to-point communication," NCCL User Guide. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/p2p.html

[10] NVIDIA, "Environment Variables," NCCL User Guide. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html

[11] Zhiyi Hu, Linta Tang, Ting Yang, Suren Byna, Zhaoguo Wang, Zhen Zheng, and Wyatt Lloyd, "Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms," 2025 IEEE Symposium on High-Performance Interconnects, pp. 48-59, 2025. https://doi.org/10.1109/HOTI66940.2025.00024

[12] Pitch Patarasuk and Xin Yuan, "Bandwidth optimal all-reduce algorithms for clusters of workstations," Journal of Parallel and Distributed Computing, 69(2), pp. 117-124, 2009. https://doi.org/10.1016/j.jpdc.2008.09.002

[13] Peter Sanders, Jochen Speck, and Jesper Larsson Traff, "Two-tree algorithms for full bandwidth broadcast, reduction and scan," Parallel Computing, 35(12), pp. 581-594, 2009. https://doi.org/10.1016/j.parco.2009.09.001

[14] NVIDIA, "Group Calls," NCCL User Guide. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/groups.html

[15] NVIDIA, "Communicator Creation and Management," NCCL User Guide. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/communicators.html

[16] NVIDIA, "New Scaling Algorithm and Initialization with NVIDIA Collective Communications Library 2.23," NVIDIA Technical Blog. https://developer.nvidia.com/blog/new-scaling-algorithm-and-initialization-with-nvidia-collective-communications-library-2-23/

[17] NVIDIA, "Enabling Fast Inference and Resilient Training with NCCL 2.27," NVIDIA Technical Blog, July 14, 2025. https://developer.nvidia.com/blog/enabling-fast-inference-and-resilient-training-with-nccl-2-27

[18] NVIDIA, "NCCL 2.28.3 Release Notes," 2025. https://docs.nvidia.com/deeplearning/nccl/archives/nccl_2297/release-notes/rel_2-28-3.html

[19] NVIDIA, "NCCL 2.29.2 Release Notes," 2025. https://docs.nvidia.com/deeplearning/nccl/archives/nccl_2293/release-notes/rel_2-29-2.html

[20] NVIDIA, "NCCL 2.30.3 Release Notes," 2026. https://docs.nvidia.com/deeplearning/nccl/archives/nccl_2307/release-notes/rel_2-30-3.html

[21] NVIDIA, "Device API," NCCL User Guide. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/deviceapi.html

[22] NVIDIA, "NCCL Tests," GitHub repository. https://github.com/NVIDIA/nccl-tests

[23] NVIDIA, "NCCL Tests Performance," GitHub repository documentation. https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md

[24] PyTorch, "Distributed communication package - torch.distributed," PyTorch documentation. https://docs.pytorch.org/docs/stable/distributed.html

[25] PyTorch, "DistributedDataParallel," PyTorch documentation. https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html

[26] PyTorch, "FullyShardedDataParallel," PyTorch documentation. https://docs.pytorch.org/docs/stable/fsdp.html

[27] Yanli Zhao et al., "PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel," arXiv:2304.11277, 2023. https://arxiv.org/abs/2304.11277

[28] DeepSpeed, "Getting Started," DeepSpeed documentation. https://www.deepspeed.ai/getting-started/

[29] NVIDIA, "Megatron Core parallel state," Megatron-LM source code. https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/parallel_state.py

[30] TensorFlow, "tf.distribute.NcclAllReduce," TensorFlow API documentation. https://www.tensorflow.org/api_docs/python/tf/distribute/NcclAllReduce

[31] Alexander Sergeev and Mike Del Balso, "Horovod: fast and easy distributed deep learning in TensorFlow," arXiv:1802.05799, 2018. https://arxiv.org/abs/1802.05799

[32] JAX, "Change log," GitHub repository. https://github.com/jax-ml/jax/blob/main/CHANGELOG.md

[33] NVIDIA, "NCCL License," GitHub repository. https://raw.githubusercontent.com/NVIDIA/nccl/master/LICENSE.txt

[34] AMD, "RCCL documentation," ROCm documentation. https://rocm.docs.amd.com/projects/rccl/en/develop/

[35] Open MPI, "MPI collectives," Open MPI v5.0.4 release notes. https://docs.open-mpi.org/en/v5.0.4/release-notes/mpi-collectives.html

[36] NVIDIA, "GPU Direct," NCCL User Guide: Troubleshooting. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/gpu_troubleshooting.html

[37] NVIDIA, "Runtime and MPI Issues," NCCL User Guide: Troubleshooting. https://docs.nvidia.com/deeplearning/nccl/archives/nccl_2307/user-guide/docs/troubleshooting/runtime_and_mpi_issues.html

[38] NVIDIA, "Networking Issues," NCCL User Guide: Troubleshooting. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html

[39] PyTorch, "CUDA Environment Variables," PyTorch documentation. https://docs.pytorch.org/docs/stable/cuda_environment_variables.html

[40] NVIDIA, "NCCL v2.29.7 Release," GitHub Releases, February 27, 2026. https://github.com/NVIDIA/nccl/releases/tag/v2.29.7-1

Editing: NCCL (NVIDIA Collective Communications Library)