DeepEP

DeepEP is an open-source communication library built by DeepSeek for distributed training and inference, written first for Mixture-of-Experts (MoE) models. It provides fast all-to-all kernels for the two communication steps an MoE layer depends on, usually called dispatch and combine, and it covers both the high-throughput case used in training and prefill and the latency-sensitive case used in decoding. DeepSeek released it on 25 February 2025, the second day of its "Open Source Week," and described it there as "the first open-source EP communication library for MoE model training and inference." ^[1][2][10]

The library matters because MoE only saves compute if the routing it depends on is cheap to move across GPUs. DeepEP is the piece that makes that movement fast, and the first version was tuned to match the exact MoE design used in DeepSeek-V3. ^[4][11]

Two later rewrites widened the scope. The V2 release of April 2026 refactored expert parallelism, replaced the NVSHMEM backend with NVIDIA's lighter NCCL Gin backend, and added experimental primitives for pipeline parallelism, context parallelism and remote memory access. V2.5, merged on 29 September 2026, split the buffer API into four objects, added bucket collectives and redundant-expert load-balancing primitives, and deleted V1 and its NVSHMEM dependency outright. The README now expands the name as "DeepEveryParallel" and describes the project as "a high-performance communication library for machine learning training and inference" rather than an MoE-only library. ^[1][12][13] A matching port for Huawei Ascend NPUs, DeepEP-Ascend, appeared on 30 September 2026. ^[14]

Why MoE needs fast all-to-all communication

A dense transformer sends every token through the same feed-forward network. A Mixture-of-Experts layer instead holds many expert networks and routes each token to only a few of them, picked by a small gating function. That lets the total parameter count grow while the compute spent per token stays roughly flat, which is the main reason MoE has become common in large language models. ^[5]

The catch shows up once the experts are spread across many GPUs, a setup called expert parallelism. A given GPU holds some experts but receives tokens that may belong to experts living anywhere in the cluster. So before the experts can run, every GPU has to ship each token to whichever GPU owns its chosen expert. That is an all-to-all exchange, and it happens twice per MoE layer.

The first exchange is dispatch. Each GPU sends its tokens out to the GPUs that own the selected experts, so that every expert receives the tokens routed to it. The experts then do their feed-forward computation locally. The second exchange is combine. The expert outputs travel back to the GPUs that originally held those tokens, where they are reassembled in the right order so the rest of the network can continue. ^[1]

Because dispatch and combine sit on the critical path of every MoE layer, their cost is felt on every forward pass in training and on every step of inference. If the all-to-all is slow, the expensive GPUs spend their time waiting on the network instead of doing math, and the efficiency that MoE promised on paper quietly disappears. DeepEP exists to keep that exchange close to the limits of the hardware. ^[1][3]

The two kernel sets of V1

The original library split the problem into two kernel families because training and decoding stress the network in different ways. ^[11]

The first set was the normal, or high-throughput, kernels. These move large batches of tokens and care most about raw bandwidth, so they suited training and the prefill phase of inference, where a long prompt is processed in one shot. They were written for asymmetric-domain bandwidth forwarding, meaning they were designed to push data efficiently from the fast NVLink domain inside a node out to the slower RDMA domain between nodes. DeepSeek aligned these kernels with the group-limited gating algorithm described in the DeepSeek-V3 paper, which restricts how many separate machines a token's experts can land on and so keeps the cross-node traffic bounded. ^[4][11]

The second set was the low-latency kernels. Decoding generates one token at a time, so the batches are tiny and the thing that hurts is the fixed delay of each round trip rather than total bandwidth. These kernels used pure RDMA and were stripped down to make a single dispatch or combine as quick as possible. They are the kernels you reach for when serving an interactive model, where every millisecond of added latency is felt directly by the user. ^[11]

V2 folded both into one EPBuffer interface with a single expanded layout for grouped expert GEMMs, so from April 2026 onward the split is a historical description of V1 rather than a description of the API. ^[12]

NVLink, RDMA, FP8, and overlap

A modern GPU cluster has two very different kinds of links, and DeepEP is written to respect both. Inside a single server, GPUs talk over NVLink, NVIDIA's high-bandwidth on-node interconnect. Between servers, they talk over an RDMA network, typically InfiniBand, which lets one machine read and write another machine's memory without going through the operating system on each hop. The V1 normal kernels forwarded across both domains, using NVLink for the intranode leg and RDMA for the internode leg, while the low-latency kernels leaned on RDMA for the cross-node transfers; from June 2025 the low-latency path also used NVLink where it could. ^[11][15]

DeepEP supports FP8 for the dispatch step. Sending tokens in an 8-bit floating-point format rather than 16-bit roughly halves the bytes on the wire, which is a direct win for an operation whose cost is dominated by data movement. DeepSeek's reference setting dispatches in FP8 and combines in BF16, the same low-precision direction it pushed throughout the DeepSeek-V3 work; combine inputs must be BF16 in the current API. ^[1][4]

The third idea is overlap. Communication and computation can run at the same time if the software is arranged carefully, so that a GPU keeps computing while more tokens are still arriving. In V1 the low-latency path did this with a hook-based mechanism that, by DeepSeek's account, did not consume any streaming multiprocessor resources. V2 gave that up: its release note says that "0 SM RDMA low-latency EP is no longer supported," and the current README states plainly that "EP dispatch and combine require GPU SMs; zero-SM RDMA EP is not supported." Overlap is now arranged differently, through deferred dispatch and combine epilogues: the call returns an event, the framework queues independent work, and calling .wait() runs the epilogue on the current stream when the result is actually needed. ^{[1][11][12][13]}

The experimental non-EP primitives follow the same pattern. V2 advertised zero-SM Engram and pipeline parallelism over RDMA and zero-SM context-parallel all-gather driven by the copy engine; in the current API, all-gather is still copy-engine driven with num_sms=0, while reduce-scatter and all-reduce use SMs. ^[1][12]

Performance and hardware

The figures DeepSeek has published have changed with every rewrite, and the README no longer carries a performance table at all, so the numbers below are tied to the version that published them.

V1, as published in February 2025

The original benchmarks were run on H800 GPUs, quoted at roughly 160 GB/s of NVLink bandwidth, each connected to a ConnectX-7 InfiniBand card rated at 400 Gb/s, which DeepSeek put at roughly 50 GB/s of usable bandwidth per card. DeepEP V1 targeted NVIDIA Hopper GPUs, and the README said it might support more architectures or devices later. The normal-kernel numbers follow the DeepSeek-V3/R1 pretraining setting of 4,096 tokens per batch, a hidden size of 7,168, top-4 groups, top-8 experts, FP8 dispatch and BF16 combine. The column labelled "#EP" is the size of the expert-parallel group, in ranks, not the number of experts: the README states the expert count separately in the benchmark setting, and the intranode row's value of 8 is one NVLink node. ^[11][14]

The low-latency figures follow a DeepSeek-V3/R1 production setting of 128 tokens per batch, a hidden size of 7,168, top-8 experts, FP8 dispatch and BF16 combine, again sweeping the EP group size rather than the expert count. ^[11]

Two contributions moved these numbers during 2025. The README recorded that optimizations from Tencent's Network Platform Department improved performance by up to 30 percent on 22 April 2025, and that the low-latency kernels began using NVLink wherever possible on 5 June 2025. By the v1.2.1 tag of 16 September 2025, the last and only tagged release, the same tables reported 58 GB/s dispatch at EP32 internode and 51 GB/s at EP64, and low-latency dispatch of 77 us at 98 GB/s and combine of 114 us at 127 GB/s at EP8, against the 163 us and 318 us of February. ^[15][16]

V2, as published in April 2026

The V2 release note reports its own measurements at 8,192 tokens per batch, a hidden size of 7,168, top-8 experts, FP8 dispatch and BF16 combine. SM90 is Hopper and SM100 is Blackwell; DeepSeek notes these are logical bandwidths, so a figure such as 90 GB/s for an EP 8 x 2 topology includes local-rank traffic. ^[12]

DeepSeek's summary of the comparison is that V2 reaches up to 1.3 times V1's peak performance while using up to four times fewer SMs, and that for V3-style legacy training the SM count fell from 24 to between 4 and 6 at equivalent or better performance. It declined to publish results for larger EP configurations, saying only that it expects the kernels to keep saturating hardware bandwidth at scale. Those are self-reported numbers from the release note. ^[12]

Version history

V2.5's load-balancing primitives are worth singling out. EPBuffer.lb_prefetch_weights pushes expert weights and quantization scales to redundant expert slots over NVLink before the expert computation runs, and EPBuffer.lb_reduce_grads reads the redundant experts' FP32 gradients back over NVLink and adds them into the original experts during the backward pass. DeepSeek credits the dynamic expert-replication approach to MoonEP from Moonshot AI and to UltraEP, and says the replication plan and token rerouting stay with the integrating framework rather than inside DeepEP. ^[1][13]

Requirements and dependencies

The dependency list changed completely between V1 and V2.5. V1 needed Hopper GPUs, Python 3.8 or newer, CUDA 12.3 or newer, PyTorch 2.1 or newer and a modified build of NVSHMEM, NVIDIA's library for GPU-initiated communication, which DeepEP used to drive its RDMA transfers. ^[8] As of 1 October 2026 the README asks for Linux, Python 3.10 or newer, Hopper or newer NVIDIA GPUs, CUDA Toolkit 13.1 or newer, PyTorch 2.10 or newer with CUDA support, NCCL 2.32.3 or newer, a C++20 compiler with std::format, NVLink inside a node and an RDMA network between nodes. Installation builds only the host C++ extension; the GPU kernels themselves are compiled for the current device at runtime by DeepJIT, DeepSeek's small JIT compilation library, so installing does not need a visible GPU or a TORCH_CUDA_ARCH_LIST. NVSHMEM is no longer a dependency at all. ^{[1][11][13][17]}

On the fabric side, DeepSeek says DeepEP is fully tested with InfiniBand and is theoretically compatible with RDMA over Converged Ethernet. The README recommends segregating expert-parallel traffic from other traffic across InfiniBand virtual lanes, selecting the service level through the buffer's sl_idx argument, leaving adaptive routing enabled under all load conditions, and disabling congestion control for maximum-bandwidth workloads once the fabric configuration has been validated. The code repository is released under the MIT License. ^[1]

DeepEP-Ascend

On 30 September 2026 DeepSeek published DeepEP-Ascend, a port of the library to Huawei's Ascend NPUs, as part of a set of six open-source infrastructure components for the Huawei Ascend platform alongside TileLang, DeepGEMM-Ascend, TileKernels, FlashMLA and DeepSelect. The main DeepEP README's news section summarizes the port as offering the "same API and full performance on HUAWEI Ascend 950 NPUs." ^[1][14]

The Ascend repository implements expert-parallel all-to-all dispatch and combine for MoE, including FP8 dispatch and deferred epilogues, and carries work-in-progress primitives for pipeline parallelism, Bucket collectives for context and data parallelism, and Engram remote memory access. Its public buffer APIs are deliberately aligned with the CUDA version's EPBuffer-based V2.5 interface, and the Python package keeps the name deep_ep. The kernels themselves are Ascend C, not CUDA: they drive communication through Huawei's HCCL and HCOMM libraries plus UBMEM and URMA, and are compiled at runtime by the same DeepJIT. The README's acknowledgement thanks Huawei's asc-comm team. ^[14]

The requirements are specific. The repository asks for an Ascend 950 (A5) part with UBMEM connectivity and UBC_CTP/URMA channels between ranks, an even number of ranks for multi-rank communication, and a CANN installation providing Ascend C, the Bisheng compiler and HCCL/HCOMM headers and libraries, with Python 3.10 or newer and a matching torch_npu. The validated stack it names is Ascend 950DT with CANN 9.2.0, Python 3.12, PyTorch 2.13.0+cpu and torch_npu 2.13.0rc1. The README states that kernel support on other Ascend generations or CANN versions is not established by its measurements. ^[14]

Its published bandwidths were measured on Ascend 950DT NPUs with CANN 9.2.0 at 16,384 tokens of per-rank capacity, hidden size 7,168, top-6 routing over 256 experts, 32 AI cores, expert alignment 128 and no bias, with FP8 dispatch using row-major scales and BF16 combine, over the supernode's external Clos network. The ranges cover all ranks. ^[14]

DeepSeek's own framing of those numbers is that sustained dispatch reaches roughly 90 to 95 percent of the physical payload bandwidth limit for EP sizes up to 32, and that larger EP sizes and combine remain under optimization because combine carries extra local-reduction overhead and contends with URMA for HBM. The measurements come with an unusual caveat: they were collected on a proof-of-concept hardware development kit supplied to DeepSeek with additional manual configuration, not a publicly distributed release, and the README points users instead to Huawei's Q3 commercial HDK release for Atlas 850E, whose public availability it describes as planned for around 15 October 2026, subject to Huawei's own schedule. Users on earlier proof-of-concept versions are told they may see lower bandwidth. Several features are explicitly unfinished: Ascend reduce-scatter and all-reduce kernels, the communication kernels behind the expert load-balancing APIs, and hybrid communication, CPU-backed Engram storage and graph capture. ^[14]

Unlike the CUDA repository and the sibling DeepGEMM-Ascend, which are MIT-licensed, the DeepEP-Ascend repository carried no license file as of 1 October 2026. ^[1][14]

Reporting on the wider release, the German outlet The Decoder said the components were announced through DeepSeek's official WeChat channel and that DeepSeek said Huawei "fully supported" the work, citing Reuters; it added that the two companies also optimized a supernode of 128 Ascend 950 chips. Pandaily reported the same 128-card design as Huawei's jointly defined SuperPoD Flex supernode with a UBL128 networking scheme, and quoted DeepEP-Ascend's EP8 figures as DeepSeek saying they approach hardware limits. Both outlets presented the release as aimed at the software gap between Chinese accelerators and NVIDIA's CUDA ecosystem. ^[18][19]

Experimental branches and community forks

The README keeps a list of branches and pull requests maintained outside the main line, with the warning that their features and dependencies apply only to those branches. As of 1 October 2026 it lists a Hybrid-EP branch using TMA instructions for minimal SM usage, larger NVLink domains, PCIe kernels for non-NVLink environments and NVFP4 support; a Mori-EP branch adding ROCm and AMD GPU support in low-latency mode through AMD's MORI backend; an nvDev branch carrying newer CUDA features such as Compute Fabric Transport; a zero-copy pull request contributed by Tencent's Network Platform Department; an Eager pull request that removes a round trip by avoiding RDMA atomics; and an AntGroup-Opt series from Ant Group's Network Platform Department. Named community forks include uccl-ep for heterogeneous GPUs and NICs, a multi-queue-pair dual-port variant from Infrawaves, Ant Group's DeepXTrace slow-rank analyzer and AMD's MORI library. DeepSeek acknowledges the NCCL team for its support of the Gin backend. ^[1]

Place in DeepSeek's open-infra suite

DeepEP is one of a set of low-level tools DeepSeek published during Open Source Week, and the pieces are meant to fit together. FlashMLA, released the day before, is a decoding kernel for the multi-head latent attention used in DeepSeek-V3. DeepGEMM, released the day after, is an FP8 matrix-multiply library that includes the grouped GEMMs an MoE layer runs once its tokens have been dispatched. Day four added DualPipe, a pipeline-parallel scheduling method, and EPLB, an expert-parallel load balancer that tries to keep the experts evenly busy so no single GPU becomes the bottleneck in the all-to-all. Day five added 3FS, a parallel file system, and the smallpond data-processing framework; the sixth and final post, on 1 March 2025, was a write-up of the DeepSeek-V3/R1 inference system rather than a repository. ^[2][6][7]

Seen together, the suite is roughly the data path of a DeepSeek-V3 MoE layer broken into reusable parts. DeepEP moves the tokens, DeepGEMM does the expert math, EPLB decides where the experts live, and FlashMLA handles attention. DeepSeek framed the whole release as opening up infrastructure it had already tested in production rather than as a research prototype, calling the repositories "humble building blocks of our online service: documented, deployed, and battle-tested in production." The lower-level groundwork all of this builds on is described under CUDA. ^[2][3]

Adoption and significance

DeepEP drew attention because DeepSeek presented it as the first open release of a communication library aimed specifically at MoE expert parallelism, an area that had mostly lived inside the private stacks of large labs. Efficient all-to-all is one of the harder, less visible parts of training and serving an MoE model, and DeepSeek shipped a version it said it used itself, with the benchmarks to back it. ^[2][3]

That had a practical effect on the wider community. Anyone building or serving large MoE models on Hopper hardware suddenly had a concrete, optimized reference instead of writing the kernels from scratch, and the library slotted naturally into stacks already running DeepSeek-R1 style architectures. Cloud and serving teams picked it up quickly: the LMSYS group used DeepEP in a prefill-decode-disaggregated, large-scale expert-parallel deployment of DeepSeek across 96 H100 GPUs, and Microsoft's Azure High Performance Computing blog published a tuning guide for DeepEP on Azure in May 2025, focused on CPU, GPU and memory affinity during initialization. It also fed the broader read on DeepSeek through early 2025, which is that the lab competed less by spending more on hardware and more by squeezing the hardware it had, with the all-to-all layer as one clear example. ^[9][20]

By October 2026 the repository had about 10,200 stars and 1,456 forks on GitHub, and newer MoE communication libraries were benchmarking themselves against it: Moonshot AI's MoonEP publishes its comparisons against DeepEP V2, and DeepSeek in turn built V2.5's redundant-expert primitives on the approach MoonEP and UltraEP explored. ^[1][13][21]

Limitations

DeepEP is deliberately narrow, and the constraints follow from that. As of 1 October 2026 it requires Hopper or newer NVIDIA GPUs, so it does not cover older NVIDIA generations or other vendors' accelerators in the main line; AMD support lives on the separate Mori-EP branch. It assumes NVLink within a node and an RDMA network between nodes, which is the layout of a serious training cluster but not of a single workstation or a commodity setup, and RDMA operations specifically require the NCCL Gin backend. Bucket collectives, Engram and pipeline parallelism are all marked experimental, and NVLink reductions need NCCL multimem support. V2 also notes that its buffers consume more memory than V1's. ^[1][12]

Several V1 constraints are gone. The patched NVSHMEM build, the zero-SM RDMA low-latency path and the undefined-behavior PTX load instruction that V1 used for extra bandwidth, with DISABLE_AGGRESSIVE_PTX_INSTRS as the escape hatch, were all removed when V1 was deleted in V2.5. What persists is the shape of the problem: the kernels are built around the routing patterns DeepSeek's own models use, so a very different gating scheme may not map onto them as cleanly. ^[1][11][13]

See also

• DeepSeek
• DeepSeek-V3
• DeepSeek V4
• Mixture-of-Experts
• Expert parallelism
• FlashMLA
• DeepGEMM
• TileLang
• NCCL
• Huawei Ascend

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17DeepSeek. "DeepEP: an efficient expert-parallel communication library." GitHub. github.com/...DeepEP
2. ^1 ^2 ^3 ^4 ^5DeepSeek. "open-infra-index: Open Source Week." GitHub. github.com/...open-infra-index
3. ^1 ^2 ^3Nazneen, N. "DeepSeek AI Releases DeepEP: An Open-Source EP Communication Library for MoE Model Training and Inference." MarkTechPost, February 2025. marktechpost.com/...e-model-training-and-inference
4. ^1 ^2 ^3DeepSeek-AI. "DeepSeek-V3 Technical Report." arXiv:2412.19437. arxiv.org/...2412.19437
5. ^Fedus, William, Barret Zoph, and Noam Shazeer. "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity." arXiv:2101.03961. arxiv.org/...2101.03961
6. ^DeepSeek. "FlashMLA: Efficient Multi-head Latent Attention Kernels." GitHub. github.com/...FlashMLA
7. ^DeepSeek. "DeepGEMM: clean and efficient BLAS kernel library on GPU." GitHub. github.com/...DeepGEMM
8. ^NVIDIA. "NVSHMEM: GPU-initiated communication for NVIDIA GPUs." NVIDIA Developer. developer.nvidia.com/nvshmem
9. ^LMSYS Org. "Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs." LMSYS Blog, May 2025. lmsys.org/...2025-05-05-large-scale-ep
10. ^DeepSeek. "update day 2 - DeepEP." open-infra-index commit history, 25 February 2025. github.com/...main
11. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11DeepSeek. "DeepEP" README as published on 25 February 2025 (commit 84d3d6fd). GitHub. github.com/...README.md
12. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8DeepSeek. "[Public release 26/04] Introducing EPv2: faster EP, and Engram/PP/CP supports." DeepEP pull request #605, merged 29 April 2026. github.com/...605
13. ^1 ^2 ^3 ^4 ^5 ^6 ^7DeepSeek. "DeepEP V2.5." DeepEP pull request #763, merged 29 September 2026. github.com/...763
14. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9DeepSeek. "DeepEP-Ascend: A high-performance communication library for machine learning training and inference on Huawei Ascend NPUs." GitHub. github.com/...DeepEP-Ascend
15. ^1 ^2DeepSeek. "DeepEP" README at tag v1.2.1. GitHub. github.com/...README.md
16. ^1 ^2DeepSeek. "Stable release v1.2.1." DeepEP releases, 16 September 2025. github.com/...v1.2.1
17. ^DeepSeek. "DeepJIT: A lightweight library for xPU kernel JIT compilation." GitHub. github.com/...DeepJIT
18. ^Schreiner, Maximilian. "China's AI industry closes ranks as Deepseek ships open-source software for Huawei's Ascend chips." The Decoder, 30 September 2026. the-decoder.com/...ftware-for-huaweis-ascend-chips
19. ^Pandaily. "DeepSeek Open-Sources Ascend Versions of TileLang, DeepGEMM and DeepEP as Huawei Details SuperPoD Flex." 30 September 2026. pandaily.com/...lang-deepgemm-deepep-superpod-flex
20. ^Microsoft Azure High Performance Computing Blog. "Achieving Optimal Performance for DeepSeek Expert Parallelism (DeepEP) on Azure." 16 May 2025. techcommunity.microsoft.com/...4414699
21. ^Moonshot AI. "MoonEP: A Perfectly Balanced Expert Parallelism Library via Dynamic Redundant Experts." GitHub README, section "Communication vs DeepEP v2". Accessed 1 October 2026. github.com/...MoonEP