NVIDIA Dynamo

NVIDIA Dynamo is an open-source, low-latency distributed inference serving framework designed to deploy and scale generative AI and reasoning models across large GPU clusters. NVIDIA describes the project, on its GitHub homepage, as a "Datacenter Scale Distributed Inference Serving Framework," and at its launch positioned it as the successor to the NVIDIA Triton Inference Server for large-scale LLM workloads.^[4][3] Announced by NVIDIA on March 18, 2025, at GTC 2025 in San Jose, California, Dynamo addresses the operational challenges of running frontier large language models (LLMs) in production at datacenter scale.^[1] The framework is available under the Apache 2.0 license and is hosted at github.com/ai-dynamo/dynamo.^[4]

Dynamo introduces four named innovations for inference optimization at fleet scale: disaggregated serving (splitting the prefill and decode phases across different GPUs), a GPU Planner for dynamic GPU scheduling, an LLM-aware Smart Router that routes requests to minimize KV cache recomputation, and a low-latency communication library (NIXL) for accelerated GPU-to-GPU data transfer.^[1] In NVIDIA's launch benchmarks, these techniques boosted the number of tokens generated by over 30x per GPU when serving DeepSeek-R1 671B on the GB200 NVL72 platform, and doubled the throughput and revenue of serving Llama models on the existing NVIDIA Hopper platform using the same number of GPUs.^[1]

CEO Jensen Huang described Dynamo during the GTC keynote as the "operating system of an AI factory," drawing a parallel to the industrial-era dynamo (an electrical generator) that powered the first factory revolution.^[12] In the launch announcement, Huang framed the project's purpose plainly: "To enable a future of custom reasoning AI, NVIDIA Dynamo helps serve these models at scale, driving cost savings and efficiencies across AI factories."^[1] The project reached production maturity with the release of Dynamo 1.0 in March 2026, by which point it had been adopted by cloud providers including AWS, Microsoft Azure, Google Cloud, and Oracle Cloud Infrastructure, as well as AI-native companies like Perplexity and Cursor.^[5]

What problem does Dynamo solve? Distributed inference at scale

As LLMs grew from tens of billions to hundreds of billions of parameters, single-GPU inference became impractical for production serving. A model such as DeepSeek-R1, with 671 billion parameters and a 128,000-token context window, requires tens of terabytes of memory and substantial all-to-all communication bandwidth to serve at low latency. Spreading inference across multiple GPUs and nodes introduces new coordination problems that existing frameworks were not designed to solve at scale.

Traditional inference frameworks like NVIDIA Triton Inference Server were designed around single-node, multi-framework serving. They work well for vision, NLP, and smaller language model workloads but do not natively handle the two-phase computation pattern of transformer-based LLMs, where a prompt-processing stage (prefill) and a token-generation stage (decode) have fundamentally different hardware demands.

Beyond hardware constraints, reasoning models introduced a further challenge. Models that "think" by generating extended chain-of-thought sequences before producing a final answer can produce tens of thousands of internal tokens per request. This dramatically amplifies the memory and compute demands compared to standard chat-style inference, and it increases the variance in output length, making it harder to allocate GPU resources ahead of time.

Existing single-engine solutions such as vLLM, SGLang, and TensorRT-LLM each addressed parts of this problem but required operators to manage their own request routing, GPU scaling, and KV cache placement. NVIDIA Dynamo was built to provide an orchestration layer that sits above these inference engines and handles the coordination work across thousands of GPUs. NVIDIA states that Dynamo "orchestrates and accelerates inference communication across thousands of GPUs" to keep AI factories running at the lowest possible cost.^[1]

How does NVIDIA Dynamo work? Architecture

Dynamo is an orchestration framework rather than a self-contained inference engine. It integrates with existing inference backends (vLLM, SGLang, TensorRT-LLM) and adds a set of coordinating services that handle scheduling, routing, memory management, and inter-GPU communication.^[2] The primary language for the framework is Rust (approximately 55% of the codebase), chosen for its performance and memory-safety properties, with Python (about 30%) used for extensibility and user-facing APIs, and Go used for certain infrastructure components.^[4]

The codebase was approximately at version 1.1.0 as of May 2026 and had accumulated over 6,700 GitHub stars and contributions from more than 70 community members.^[4] By June 2026 the repository had grown to roughly 7,200 stars and 1,200 forks, with a language split of about 53% Rust, 33% Python, and 12% Go.^[4] Version 1.2.0, the fifteenth feature release, followed on June 2, 2026.^[20] By September 6, 2026, the repository showed roughly 7,980 stars and 1,550 forks, and the latest stable tag was v1.4.2, published August 29, 2026.^[4][20]

Disaggregated serving

The central architectural innovation in Dynamo is disaggregated serving: the physical separation of the prefill phase and the decode phase onto different GPUs or groups of GPUs.

In standard (aggregated) serving, each GPU or GPU group performs both prefill and decode for every request. Prefill is compute-bound: it processes the input prompt tokens in a single parallel forward pass. Decode is memory-bandwidth-bound: it generates tokens one at a time, repeatedly loading the KV cache from GPU memory. Running both on the same hardware forces a compromise. Decode underutilizes matrix multiplication units, while prefill competes for memory bandwidth with active decodes.^[2]

Dynamo separates these phases so each can be optimized independently:

• Prefill workers run with lower tensor parallelism, since prefill benefits from dense matrix math rather than all-to-all communication.
• Decode workers run with higher tensor parallelism, which distributes the KV cache across more GPUs and reduces memory pressure per device.

Once a prefill worker finishes processing a prompt, it transfers the resulting KV cache blocks to a designated decode worker over a high-speed interconnect. The decode worker then takes over and generates the response token by token.

Dynamo also implements conditional disaggregation: not every request goes to a remote prefill worker. If the prompt is short or the decode worker already has a high prefix cache hit rate for that request, Dynamo routes the prefill locally on the decode worker to avoid unnecessary transfer overhead. The disaggregated router makes this decision at runtime based on two configurable thresholds: the minimum prefill length required to justify remote processing, and the maximum queue depth of the remote prefill pool.^[8]

Experimental results published by NVIDIA showed this design achieving up to a 6x throughput improvement on DeepSeek-R1 running on GB200 NVL72 hardware in medium-latency scenarios, compared to aggregated serving on the same hardware.^[7] For Llama 70B on Hopper-generation GPUs, disaggregated serving roughly doubled throughput.^[2]

KV-aware routing

After disaggregated serving, KV-aware routing is Dynamo's second major mechanism for reducing redundant computation.

In a fleet with many decode workers, the same prompt prefix often appears across multiple requests (for example, a long system prompt shared by all users of a particular application). If each request lands on a different decode worker, each worker computes and stores its own copy of the KV cache for that prefix. This wastes both compute and memory.

The NVIDIA Dynamo Smart Router maintains a global, cluster-wide map of which KV cache blocks are resident on which workers.^[2] It uses a Radix Tree (the same data structure used in PagedAttention for local cache management) to index prefixes by their token hash. Two backend implementations are available: a single-threaded RadixTree and a ConcurrentRadixTree using a thread pool for higher throughput under heavy load.^[9]

When a new request arrives, the router computes an overlap score between the incoming token sequence and the cached blocks on each worker. It then routes the request to the worker with the highest cache overlap while also accounting for load balance.^[9] Workers with heavy decode queues receive lower routing weight regardless of cache overlap, preventing a single overloaded worker from degrading the user experience.

The router exposes a configurable overlap weight parameter that operators can tune to trade off TTFT (time to first token) against ITL (inter-token latency). Higher overlap weight steers requests more aggressively toward cache-rich workers, which reduces redundant prefill and cuts TTFT. Lower overlap weight distributes load more evenly, which reduces ITL for already-running decodes.^[9]

NVIDIA reported that on a dataset of 100,000 real user queries to a DeepSeek-R1 deployment (with average input lengths of 4,000 tokens and output lengths of 800 tokens), KV-aware routing achieved a 3x reduction in TTFT and a 2x reduction in average request latency compared to naive round-robin routing.^[2]

Baseten, an inference endpoint company, deployed Dynamo for Qwen3 Coder 480B and measured a 50% reduction in average TTFT, a 34% reduction in time-per-output-token, a 61% increase in requests per second, and an 89% KV cache hit rate across four replicas, compared to serving the same model without Dynamo's routing.^[10]

NIXL: NVIDIA Inference Xfer Library

KV cache transfer between disaggregated prefill and decode workers requires extremely low-latency point-to-point data movement that general-purpose networking libraries are not optimized for. Dynamo includes the NVIDIA Inference Xfer Library (NIXL), a hardware-agnostic communication library built specifically for moving KV cache blocks between GPU memory regions.^[2]

NIXL supports five transport backends:

• RDMA over InfiniBand
• RDMA over Converged Ethernet (RoCE) via UCX
• TCP as a fallback for non-RDMA environments
• NVMe-oF for transfers involving SSD-backed KV storage
• S3-compatible object storage^[16]

Transfers are non-blocking: a prefill worker can issue a NIXL write to a decode worker's VRAM and immediately begin processing the next request without waiting for the transfer to complete. This allows GPU compute and data movement to overlap, reducing idle time.

To minimize per-transfer metadata overhead, NIXL caches memory descriptors in etcd (a distributed key-value store). Only block IDs need to be included in each request message; the receiving worker looks up the full descriptor from etcd. Contiguous blocks are also consolidated into a single transfer operation where possible.

For configurations where prefill and decode workers run with different tensor parallelism degrees (which changes the KV layout), NIXL includes high-performance kernels that transpose KV blocks during transfer, eliminating the need to reshape data at either end.

On GB200 NVL72 systems, NIXL can exploit NVLink's 1.8 TB/s per-GPU bidirectional bandwidth for transfers within the same NVLink domain, which is approximately 36 times faster than 400 Gbps Ethernet.^[7]

KV Block Manager

GPU HBM memory is the scarcest resource in large-scale inference. A single DeepSeek-R1 request with a 128,000-token context window can occupy several gigabytes of KV cache per GPU. Under heavy load, KV cache pressure forces the eviction of cached prefixes just as they become useful to subsequent requests.

The KV Block Manager (KVBM) extends the effective KV cache capacity by tiering storage across multiple memory types in order of latency and cost:

1. GPU HBM (fastest, most expensive)
2. CPU DRAM
3. Local NVMe SSD
4. Remote storage (S3 or Azure Blob)^[2]

When a KV block is evicted from GPU memory due to capacity pressure, KVBM writes it to the next available tier rather than discarding it. If the same prefix is requested again and its blocks are in CPU memory, they can be prefetched back to GPU memory much faster than recomputing them. KVBM maintains a cluster-wide event log of block locations so the Smart Router can account for which workers have which blocks in any tier, not just in GPU HBM.

KVBM is available as a pip-installable module that can be added independently to vLLM or TensorRT-LLM deployments without requiring the full Dynamo stack.^[17]

SLO Planner

The SLO Planner is Dynamo's autoscaling component. It continuously monitors GPU utilization, KV block occupancy, and the depth of the prefill request queue across the cluster.^[6] Based on operator-defined service level objectives (SLOs) for TTFT and ITL, the Planner decides when to rebalance resources between prefill workers and decode workers, or when to scale the total GPU count up or down.

Conventional autoscalers based on GPU utilization percentage behave poorly for LLM inference because prefill and decode phases use the same GPUs in very different ways. A decode-heavy workload may show moderate GPU utilization while prefill requests queue up unserved. The Planner addresses this by tracking inference-specific metrics rather than generic hardware counters.^[6]

A benchmark by NVIDIA using simulated workload bursts showed the Planner achieving 80% fewer SLA breaches compared to a fixed topology deployment, at approximately 5% lower total cost of ownership.

The Planner's forecasting layer, introduced with version 0.4 in August 2025, predicts incoming traffic using time-series models such as ARIMA and Prophet and pre-computes the minimum worker counts needed to hold SLO targets as load shifts.^[21] In January 2026, Microsoft and NVIDIA published a joint engineering series on running the system on Azure Kubernetes Service, pairing a pre-deployment profiler that automates configuration search with the runtime SLO-based Planner; in the published example, a Qwen3-32B-FP8 deployment targeting a 500 ms TTFT and 30 ms ITL scaled from one to two prefill workers within minutes under rising load while holding its latency targets.^[22]

ModelExpress

Starting a new inference worker replica typically requires loading the full model checkpoint from network storage, which can take minutes for a 671B parameter model. During traffic spikes, this startup latency limits how quickly additional capacity can be brought online.

ModelExpress accelerates replica startup by loading the model once on an initial worker and then streaming the weights to additional workers over NVLink using NIXL. Because NVLink bandwidth far exceeds storage I/O bandwidth for in-domain transfers, this process is substantially faster than reading from a shared filesystem. NVIDIA reported a 7x reduction in model startup time for large mixture-of-experts (MoE) models using this approach.^[17]

Dynamo Snapshot

In June 2026, NVIDIA released Dynamo Snapshot in limited preview, a checkpoint-and-restore system that attacks worker cold starts from a different angle than ModelExpress. Snapshot serializes the full state of a warmed-up inference worker, both GPU-side and CPU-side, and restores it on the same or a different node, skipping model loading and engine initialization entirely.^[18] The system combines two tools: the CUDA driver's checkpoint capability, exposed through the cuda-checkpoint utility, dumps GPU device state into CPU memory, while CRIU (Checkpoint/Restore in Userspace) serializes the host process tree to disk.^[19] A GPU Memory Service decouples large model weights from process state so weights can be restored concurrently over high-bandwidth paths such as GPUDirect Storage, while a KV cache unmap step and parallel memory restoration with Linux asynchronous I/O keep checkpoint sizes and restore times down.^[18]

In a proof-of-concept configuration with striped local NVMe SSDs, Snapshot restored a gpt-oss-120b vLLM worker in under 5 seconds, which NVIDIA reported as an up to 21x reduction in startup time.^[18][19] The initial preview supports single-GPU vLLM workers in runc-managed containers, deployed through a privileged Kubernetes DaemonSet installed via Helm, and requires NVIDIA driver version 580 or newer.^[19]

Shadow Engine Recovery

In August 2026, NVIDIA introduced Shadow Engine Recovery, called Shadow Engine Failover in the current documentation, as an experimental, opt-in preview for restoring inference capacity after an engine process fails while its node and GPUs remain healthy. The design keeps an initialized passive engine on the same GPUs as the active engine so that process replacement does not repeat the full weight-loading and engine-warmup path.^[25][26]

A per-GPU GPU Memory Service (GMS) sidecar owns the physical HBM allocations for model weights independently of either engine process. Using the CUDA Virtual Memory Management API, the active and shadow engines map the same physical weight bytes at their own virtual addresses. GMS is not on the inference read path after that mapping. The shadow creates its communicators, captures CUDA graphs, completes warmup, and then parks while retaining its CUDA context, graphs, communicators, and weight mappings. It neither stores a second physical copy of the weights nor materializes a KV cache while parked.^[25]

Each worker pod contains active and shadow engine containers, the GMS sidecar, and a shared POSIX file lock. The active engine holds the lock, maintains the live KV cache, and registers with the frontend router. If that process exits or a liveness probe terminates it, the operating system releases the lock. The shadow then acquires the lock, maps the resident weights, creates a fresh, empty KV cache, and registers with the router. The failed engine can restart in the background as the next shadow.^[25][27]

NVIDIA evaluated this path in its own synthetic benchmark. The test used a two-worker fleet on NVIDIA B200 nodes, with one worker per node. Each worker used tensor parallelism of 8 to serve GLM-5.2 quantized to NVFP4, with a 200K maximum context and an FP8 KV cache. A single frontend sent requests round-robin; each synthetic request contained 32,000 input tokens and requested 1,000 output tokens, at 0.7 requests per second. NVIDIA stated that the engine build and configuration were the same in both runs and that the presence of a preinitialized shadow was the only difference. It deliberately terminated one worker and measured how long the fleet took to serve again with two workers.^[25]

The 7.3-second recovery result, including its 1.7-second detection and 5.6-second promotion split, and the 283-second cold-restart comparison are internal, vendor-reported results from that single synthetic setup. They have not been independently reproduced and should not be generalized to other models, hardware, request mixes, or failure conditions.^[25]

The preview has a narrow failure boundary. It is intended for an engine-process failure on a healthy node with healthy GPUs, driver, and GMS. It does not preserve in-flight requests, network sockets, or the prior KV cache, and it does not diagnose or repair the process defect. It is not hardware fault tolerance for GPU, node, or rack loss and provides no cross-node recovery after such a loss.^[26] NVIDIA's August launch article also said hardware, node, and multi-node failures were outside the preview's coverage.^[25] Current documentation lists vLLM support for both single-node and multinode model topologies, but that describes where the model can run, not recovery from losing one of those nodes.^[26]

Current prerequisites include Kubernetes 1.34 or newer with DRA v1 enabled, the NVIDIA GPU DRA driver, a matching DRA DeviceClass such as the default gpu.nvidia.com, a backend image and command that support GMS such as vLLM with --load-format gms, and enough GPU memory for GMS plus both engine processes. NVIDIA's published example enables the v1beta1 experimental.failover and experimental.gpuMemoryService fields.^[26][27] The current documentation recommends non-production evaluation unless the exact backend, topology, and failure mode have been validated. It also says not to combine Dynamo Snapshot with GMS and reports that admission currently blocks the combination because of known GPU-driver restore issues, superseding the launch article's broader statement about composing the two features.^[25][26]

Grove

Grove is Dynamo's Kubernetes operator. It provides a single declarative API for deploying inference workloads ranging from simple single-pod setups to complex multi-node disaggregated configurations. Grove handles topology-aware gang scheduling, automatically placing related prefill and decode pods on GPUs that share NVLink connectivity to maximize transfer speeds.^[15]

Grove replaces the manual YAML configuration required to set up multi-node inference deployments. Operators specify service-level objectives and hardware constraints; Grove generates the appropriate Kubernetes resource definitions and manages placement.

Grove was published as a standalone open-source project (ai-dynamo/grove) in November 2025 and ships as a modular component of Dynamo. It models a deployment through three hierarchical Kubernetes custom resources: PodCliques (role-specific pod groups such as prefill, decode, or routing), PodCliqueScalingGroups (bundles of components that scale together), and PodCliqueSets (the complete workload definition with startup ordering and spread constraints). Its hierarchical gang scheduling guarantees minimum viable combinations, for example at least one prefill and one decode worker, while letting each component type scale independently.^[15]

AIConfigurator

AIConfigurator is a simulation tool that helps operators choose prefill-to-decode GPU ratios and other serving topology parameters before deploying a workload. It simulates more than 10,000 deployment configurations and recommends the one that best satisfies the specified SLOs given the available GPU budget. Community contributors from Mooncake and Alibaba added SGLang support to the AIConfigurator during the Dynamo 1.0 cycle. In 2026 the tool also gained an attention-FFN disaggregation (AFD) mode, described in the section on heterogeneous decode below.^[32][33][34]

Which inference engines does Dynamo support? Multi-engine support

Dynamo does not replace existing inference engines. It sits above them as an orchestration layer, managing scheduling, routing, and memory across whichever backends the operator chooses. At launch NVIDIA listed PyTorch, SGLang, TensorRT-LLM, and vLLM as supported open-source tools.^[1]

TensorRT-LLM

TensorRT-LLM is NVIDIA's own inference engine, optimized for maximum throughput on NVIDIA GPUs through custom CUDA kernels, quantization (FP8, INT8, INT4, NVFP4), speculative decoding, and other low-level hardware optimizations. It delivers the highest single-node throughput of any publicly available engine for NVIDIA hardware but requires significant engineering effort to set up and is tightly coupled to specific NVIDIA GPU generations.

When paired with Dynamo, TensorRT-LLM handles the per-GPU computation while Dynamo handles cross-node coordination, KV cache routing, and autoscaling. This combination is the primary deployment path for operators who want maximum throughput on NVIDIA Blackwell hardware.

vLLM

vLLM is a community-developed inference engine that introduced PagedAttention, a technique for managing KV cache in non-contiguous memory pages to reduce fragmentation and improve memory efficiency. vLLM has broad model support and a large developer community. It is typically easier to set up than TensorRT-LLM and supports a wide range of hardware beyond NVIDIA GPUs.

Dynamo augments vLLM deployments with cross-node KV-aware routing and the KVBM tiered caching system, which vLLM cannot provide on its own. The KVBM is pip-installable alongside vLLM without requiring the full Dynamo stack.

SGLang

SGLang is an inference framework developed by the LMSYS group at UC Berkeley. Its core innovation, RadixAttention, extends the prefix caching concept from single-node settings to workloads with complex shared-prefix patterns such as multi-turn chat and retrieval-augmented generation (RAG). SGLang generally outperforms vLLM on workloads with high prefix overlap.

Dynamo extends SGLang's prefix caching across multiple nodes. Where SGLang's RadixAttention caches prefixes per worker, Dynamo's Smart Router maintains a global view of which workers hold which prefixes across the entire cluster and routes new requests accordingly.

The LMSYS group and NVIDIA published benchmarks in February 2026 showing SGLang on GB300 NVL72, coordinated by Dynamo, achieving 25x higher throughput compared to H200-based single-node setups.^[11] The deployment paired Dynamo's prefill-decode disaggregation and KV-aware router with SGLang's HiCache radix tree, ran MoE experts and dense GEMMs in NVFP4 precision, and measured the gain at a 50 tokens-per-second-per-user interactivity target.^[11]

How fast is NVIDIA Dynamo? Performance

Benchmark results for NVIDIA Dynamo vary significantly by model size, hardware generation, and workload characteristics. The figures below represent reported results from NVIDIA, LMSYS, and third-party sources.

Hardware comparison: Hopper vs. Blackwell

Note: The 30x figure from NVIDIA's March 2025 announcement compares disaggregated Blackwell performance against non-disaggregated Hopper performance, combining hardware gains from GB200 NVL72 and software gains from Dynamo's disaggregated serving.^[12] The SemiAnalysis InferenceX benchmark published in March 2026 reported 7x throughput improvement attributable specifically to Dynamo's software stack on Blackwell hardware.^[23]

InferenceX (formerly InferenceMAX), SemiAnalysis's continuously re-run open-source inference benchmark, attributed the 7x figure to disaggregated serving combined with wide expert parallelism on GB200 NVL72 in its March 3, 2026 update.^[23] As of mid-2026, NVIDIA's product page advertises up to 50x higher MoE model throughput for GB300 NVL72 systems running Dynamo relative to Hopper-based systems, a figure that, like the 30x claim, combines hardware and software generations.^[3]

KV-aware routing impact

SLO Planner

How does Dynamo compare with standalone inference frameworks?

Dynamo occupies a different layer than the inference engines it integrates with. It is an orchestration and coordination framework, not a GPU computation engine. This distinction matters when choosing a deployment approach.

vLLM, SGLang, and TensorRT-LLM all work well as standalone engines for deployments that fit on a small number of GPUs. As deployments scale to tens or hundreds of nodes, the coordination overhead of managing KV cache placement, request routing, and autoscaling by hand grows substantially. Dynamo's value increases with cluster size.

For teams not yet at multi-node scale, running vLLM or SGLang standalone remains simpler and avoids the operational overhead of managing Dynamo's supporting services (etcd, NATS, Kubernetes CRDs).

How does Dynamo differ from NVIDIA Triton Inference Server?

Dynamo is described by NVIDIA as the successor to NVIDIA Triton Inference Server for LLM workloads.^[3] Triton Inference Server, first released in 2018, was designed as a general-purpose model serving platform supporting multiple frameworks (PyTorch, TensorFlow, ONNX, TensorRT) and model types (image classification, NLP, audio, recommendation). It remains in active development under the name Dynamo-Triton and continues to receive production branch support for existing enterprise deployments.^[24]

Dynamo and Dynamo-Triton serve different purposes. Dynamo-Triton handles diverse model types on single nodes or small multi-GPU configurations. Dynamo handles multi-node LLM serving with disaggregated inference, KV-aware routing, and the other LLM-specific features described above. NVIDIA positions the two products as complementary: enterprises can continue using Dynamo-Triton for their existing general-purpose inference workloads while adopting Dynamo for new large-scale LLM deployments.

What is NVIDIA Dynamo used for? Use cases

Large-scale LLM serving

Dynamo's primary use case is serving frontier LLMs at datacenter scale. Deployments involving tens of nodes benefit most from its KV-aware routing (which reduces redundant prefill computation across the fleet) and disaggregated serving (which prevents decode phases from being starved by prefill activity).

Reasoning model serving

Reasoning models such as DeepSeek-R1 and similar chain-of-thought models produce highly variable output lengths. A single request may generate thousands of tokens of internal reasoning before producing a short final answer. This variance makes static GPU allocation inefficient: allocating for peak output length wastes resources on shorter responses, while allocating for average output length causes queue buildup during reasoning-heavy traffic.

Dynamo's Planner addresses this by monitoring prefill queue depth and decode KV block utilization in real time and rebalancing GPUs between prefill and decode pools as the workload character shifts.

Multimodal inference

Dynamo 1.0 extended disaggregation to multimodal workloads through a three-stage encode-prefill-decode (EPD) pipeline.^[17] Vision encoders run on designated encode workers, text prefill runs on prefill workers, and autoregressive decode runs on decode workers. Each stage can be scaled independently. NVIDIA reported a 30% TTFT reduction and 25% throughput gain for multimodal workloads using this architecture, relative to a single-stage serving setup.^[17]

A CPU-backed LRU cache stores image embeddings so that repeated requests referencing the same image do not trigger redundant GPU encoding.^[17]

Agentic AI

Agentic workloads (multi-step pipelines where models call tools, reflect on results, and generate follow-up queries) produce heterogeneous traffic patterns: short reflexive completions interleaved with long planning sequences. Dynamo 1.0 added priority-based routing that accepts hints from the application layer about each request's latency sensitivity and expected output length, routing time-critical requests to lower-queue workers even if their cache overlap is not optimal.^[17]

With the 1.0 release, NVIDIA reported up to 4x lower time to first token for agentic pipelines built with the NVIDIA NeMo Agent Toolkit on Hopper GPUs, along with a 1.5x throughput increase from agentic-focused optimizations.^[17]

Attention-FFN disaggregation and heterogeneous decode

NVIDIA's introduction of the Groq 3 LPX rack at GTC 2026 gave Dynamo a second kind of disaggregation to orchestrate. Where prefill-decode disaggregation moves a request's KV cache once, attention-FFN disaggregation (AFD) splits the decode phase itself: the attention layers, which read the growing per-request KV cache, run on one pool of hardware, and the feed-forward or mixture-of-experts layers, whose weights are fixed in size, run on another, with intermediate activations exchanged for every generated token. In NVIDIA's deployment the attention pool is a Vera Rubin NVL72 and the FFN pool is an LPX rack of 256 Groq 3 LPUs holding 128 GB of on-chip SRAM, a memory that suits fixed-size weights but not a KV cache that grows with context.^[28] NVIDIA's product page summarizes the mode as Rubin GPUs and LPUs boosting decode "by jointly computing every layer of the AI model for every output token."^[39]

NVIDIA's technical blog post of March 16, 2026, by Kyle Aubrey and Farshad Ghodsian, is the source for Dynamo's role. It says that making heterogeneous decode practical "requires software that can classify requests, route work by latency targets, move intermediate activations with low overhead, and keep tail latency stable under bursty, variable traffic," and that "NVIDIA Dynamo provides that orchestration layer by coordinating disaggregated serving and disaggregated decode across heterogeneous backends." The mechanics, as the post describes them: "Dynamo routes prefill to GPU workers to process the large context and build the KV cache. During decode, Dynamo orchestrates the AFD loop where GPUs run attention over the accumulated KV cache, intermediate activations are handed off to LPUs for FFN/MoE execution, and outputs return to the GPUs to continue token generation." The post credits three existing Dynamo capabilities, "KV-aware routing, low-overhead transfers, and latency-target-driven scheduling," with keeping interactive sessions out of long queues and holding tail latency steady as concurrency and request shapes vary.^[28] Its Figure 7, titled "NVIDIA Dynamo Orchestrates Heterogeneous Compute" with the subtitle "KV-aware routing for prefill and disaggregated decode (ATTN <-> FFN)", shows a Dynamo bar spanning an NVL72 (72 Rubin GPUs, with prefill GPUs and decode GPUs labelled ATTN, fed by a KV cache over "NVLink or Ethernet") and an LPX rack (256 LPUs, labelled decode LPUs FFN), joined by a double-headed arrow labelled "Interim Decode Activations", "Repeat per token", over Ethernet.^[28]

The KV-aware routing and latency-target scheduling that the LPX post leans on are the same features described elsewhere in this article. The Dynamo 1.0 post of the same day describes the router as evaluating "worker queue depth and relevant KV cache information on each worker" and making "a probabilistic decision using a weighted combination of these factors," and describes agent hints for latency sensitivity and expected output length that let the router order queues "so user-facing turns run before background work." That post does not mention AFD, LPX, or LPUs; the connection between those features and heterogeneous decode is drawn only in the LPX posts.^[17][28]

A second NVIDIA post on August 24, 2026, by Seth Weidman, Kirthi Devleker, and Andrew Ling, widened the picture. It lists three ways to pair the two racks: standard prefill-decode disaggregation, in which NVL72 handles prefill and hands off the KV cache once per turn and LPX performs the entire decode step from weights held in SRAM; attention-FFN disaggregation, in which "Vera Rubin NVL72 computes attention and holds the KV cache in DRAM, while Groq 3 LPX executes the FFN layers. Only intermediate tokens are sent between racks, once per full-attention layer"; and external-drafter speculative decoding, in which LPX runs a small draft model ahead of the target model on NVL72 and "only draft tokens cross the link." The same post reports that Artificial Analysis measured a median 3,431 output tokens per second for Gemma 4 31B at 100K input context on an NVIDIA-hosted LPX system, and that NVIDIA's own run of the SPEED-Bench coding set produced a median of 4,767 tokens per second. Those figures describe a 31B model on the LPX rack; the post does not attribute them to an AFD configuration or say which serving mode was used.^[29]

Status in the open-source repositories

NVIDIA's posts describe its own deployment, and readers should not assume that the same capability is visible in the Apache 2.0 code. GitHub searches of the ai-dynamo/dynamo issue tracker on September 6, 2026 for AFD, attention-FFN, LPU, LPX, and Groq found no merged AFD serving mode. What they did find is summarized below.

As of September 6, 2026, therefore, Dynamo's orchestration of the AFD loop between Vera Rubin NVL72 and Groq 3 LPX is documented in NVIDIA's blog posts and modelled in the AIConfigurator planning tool, while the serving runtime's public repository shows only a closed placeholder pull request and a related speculative-decoding proposal. Whether NVIDIA's LPX deployments run on unreleased Dynamo code, on a separate integration, or on a build that will be published later is not stated in any source consulted for this article.

Who uses NVIDIA Dynamo? Adoption

NVIDIA's March 2026 Dynamo 1.0 production announcement listed the following adopters:^[5]

Cloud providers:

• AWS (integrates Dynamo with Amazon EKS, P5/P6 EC2 instances via EFA)^[14]
• Microsoft Azure
• Google Cloud
• Oracle Cloud Infrastructure

Cloud GPU providers:

• CoreWeave
• Together AI (integrates with the Together Inference Engine for cross-node scaling)
• Nebius
• Alibaba Cloud (also contributed SGLang support to AIConfigurator)

AI-native companies:

• Perplexity (serves hundreds of millions of monthly requests; CTO Denis Yarats cited Dynamo for driving "inference-serving efficiencies")^[1]
• Cursor

Inference endpoint providers:

• Baseten (deployed for Qwen3 Coder 480B; measured 50% TTFT reduction)^[10]
• Deep Infra
• Fireworks

Enterprises:

• ByteDance
• Meituan
• PayPal
• Pinterest
• AstraZeneca
• BlackRock
• Tencent Cloud

Cohere's SVP Saurabh Baji said the company expects Dynamo to help "deliver a premier user experience to enterprise customers."^[1]

The 1.0 announcement further named NVIDIA cloud partners Crusoe, DigitalOcean, Gcore, GMI Cloud, Lightning AI, Nscale, and Vultr, the AI-native company Hebbia, and enterprises including Coupang, Instacart, Shopee, and SoftBank Corp. among adopters, and stated that Dynamo and TensorRT-LLM optimizations integrate natively into open-source frameworks including LangChain, llm-d, LMCache, SGLang, and vLLM.^[5] NVIDIA said more than 30 organizations were running Dynamo in production at the time of the 1.0 release.^[17] Announcing it, Jensen Huang said: "Inference is the engine of intelligence, powering every query, every agent and every application."^[5]

When was NVIDIA Dynamo released? Release history and 2026 developments

Dynamo moved from first public release to production status in roughly one year:

Development previews of version 1.3.0, published in early June 2026, added tool-calling parser parity across model families, OpenAI-compatible embeddings serving, a /v1/realtime protocol surface, and topology-aware routing, alongside model-specific preview builds for DeepSeek-V4 Pro on TensorRT-LLM, NVIDIA Nemotron-3 Super and Ultra on vLLM, Moonshot AI's Kimi K2.6, and NVIDIA Cosmos 3 text-to-image and text-to-video generation through the vLLM-Omni backend.^[20] The model-specific preview builds continued through the summer: Kimi K3 on vLLM (July 27), Nemotron 3.5 Lightning on vLLM and TensorRT-LLM with MTP, DFlash, and DSpark speculative-decoding recipes (August 11), Qwen3.8-2.4T-A95B-FP8 on vLLM and SGLang (August 27), Thinking Machines' Inkling on vLLM 0.28.0 with GB300 agentic recipes (September 3), Gemma 4 31B on TensorRT-LLM 1.3.0rc25 (September 3), and DeepSeek-V4-Pro-0813 on vLLM 0.28.0 with 1M-token context recipes (September 4). Each is labelled an experimental dev build that is not recommended for production, and the container images are published multi-arch for amd64 and arm64.^[20]

Is NVIDIA Dynamo open source?

Yes. Dynamo is released under the Apache 2.0 license. NVIDIA described it at launch as "fully open source," and the project is hosted at github.com/ai-dynamo/dynamo under the ai-dynamo GitHub organization.^[1][4]

As of May 2026, the project had accumulated over 6,700 GitHub stars and contributions from more than 70 individuals.^[4] NVIDIA runs biweekly office hours and weekly development meetings for community contributors. A Discord server is available for developer discussion.

The enterprise version of Dynamo is available through NVIDIA AI Enterprise and via NVIDIA NIM microservices, which provide pre-configured container images with validated hardware support for production deployments.^[1]

Limitations

Dynamo carries several constraints that operators should consider before adopting it:

NVIDIA hardware dependency. Dynamo requires NVIDIA GPUs running CUDA. AMD and Intel GPU support is not available. The framework is optimized for Ampere and later NVIDIA architectures, with the largest performance gains on Blackwell (H100 successor) and GB200 NVL72 hardware.^[12]

Operational complexity. A full Dynamo deployment requires Kubernetes, etcd, and NATS JetStream as supporting services, in addition to the Dynamo components themselves.^[4] This is significantly more complex to operate than a standalone vLLM or SGLang instance. Teams without existing Kubernetes expertise face a steep setup curve.

ARM64 support is experimental. The x86_64 architecture is the primary supported target. ARM64 support exists but is marked experimental as of version 1.1.0.

Python version constraints. The KV Block Manager requires Python 3.12, which is currently supported only on Ubuntu 24.04. Operators on other distributions need to build from source or use the provided container images.

Single-node deployments gain little. The core benefits of Dynamo (cross-node KV routing, disaggregated prefill/decode, cluster-wide autoscaling) apply to multi-node deployments. A single GPU or a single node with multiple GPUs served by a standalone vLLM or SGLang instance has similar performance without the added infrastructure overhead.

SGLang KVBM still in development. As of version 1.1.0, the KV Block Manager integration with SGLang is incomplete and still under active development.

See also

• Mooncake (LLM serving)
• vLLM
• SGLang
• TensorRT
• NVIDIA Triton Inference Server
• Disaggregated serving
• PagedAttention
• RadixAttention
• Inference optimization
• NVIDIA GB300 NVL72
• DeepSeek-R1
• NVIDIA NIM

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11NVIDIA Newsroom. "NVIDIA Dynamo Open-Source Library Accelerates and Scales AI Reasoning Models." March 18, 2025. nvidianews.nvidia.com/...cales-ai-reasoning-models
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10NVIDIA Technical Blog. "Introducing NVIDIA Dynamo, A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Models." March 2025. developer.nvidia.com/...caling-reasoning-ai-models
3. ^1 ^2 ^3NVIDIA Developer. "Dynamo Inference Framework." developer.nvidia.com/dynamo
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9GitHub. ai-dynamo/dynamo repository. github.com/...dynamo
5. ^1 ^2 ^3 ^4 ^5NVIDIA Newsroom. "NVIDIA Enters Production With Dynamo, the Broadly Adopted Inference Operating System for AI Factories." March 16, 2026. nvidianews.nvidia.com/...dynamo-1-0
6. ^1 ^2 ^3NVIDIA Technical Blog. "NVIDIA Dynamo Adds GPU Autoscaling, Kubernetes Automation, and Networking Optimizations." May 20, 2025. developer.nvidia.com/...d-networking-optimizations
7. ^1 ^2 ^3 ^4NVIDIA Technical Blog. "How NVIDIA GB200 NVL72 and NVIDIA Dynamo Boost Inference Performance for MoE Models." developer.nvidia.com/...performance-for-moe-models
8. ^NVIDIA Dynamo Documentation. "Disaggregation: Separating Prefill and Decode for Enhanced Performance." docs.nvidia.com/...disaggregated-serving
9. ^1 ^2 ^3NVIDIA Dynamo Documentation. "KV Cache Routing." docs.nvidia.com/...kv_cache_routing
10. ^1 ^2 ^3 ^4 ^5 ^6 ^7Baseten. "How Baseten achieved 2x faster inference with NVIDIA Dynamo." baseten.co/...-faster-inference-with-nvidia-dynamo
11. ^1 ^2 ^3LMSYS Blog. "Unlocking 25x Inference Performance with SGLang on NVIDIA GB300 NVL72." February 2026. lmsys.org/...2026-02-20-gb300-inferencex
12. ^1 ^2 ^3The Register. "A closer look at Dynamo, Nvidia's 'operating system' for AI inference." March 23, 2025. theregister.com/...nvidia_dynamo
13. SemiAnalysis. "NVIDIA GTC 2025 -- Built For Reasoning, Vera Rubin, Kyber, CPO, Dynamo Inference, Jensen Math, Feynman." March 19, 2025. semianalysis.com/...-inference-jensen-math-feynman
14. ^AWS. "Accelerate generative AI inference with NVIDIA Dynamo and Amazon EKS." aws.amazon.com/...ith-nvidia-dynamo-and-amazon-eks
15. ^1 ^2NVIDIA Technical Blog. "Streamline Complex AI Inference on Kubernetes with NVIDIA Grove." developer.nvidia.com/...bernetes-with-nvidia-grove
16. ^Spheron Blog. "NVIDIA NIXL and Disaggregated Inference: Move KV Caches Across GPUs at Wire Speed." spheron.network/...l-disaggregated-inference-guide
17. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10NVIDIA Technical Blog. "How NVIDIA Dynamo 1.0 Powers Multi-Node Inference at Production Scale." March 2026. developer.nvidia.com/...-dynamo-1-production-ready
18. ^1 ^2 ^3NVIDIA Technical Blog. "NVIDIA Dynamo Snapshot: Fast Startup for Inference Workloads on Kubernetes." June 2026. developer.nvidia.com/...ce-workloads-on-kubernetes
19. ^1 ^2 ^3MarkTechPost. "NVIDIA AI Releases Dynamo Snapshot: A CRIU-Based Fast Startup System for AI Inference on Kubernetes." June 5, 2026. marktechpost.com/...for-ai-inference-on-kubernetes
20. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13GitHub. "Releases - ai-dynamo/dynamo." github.com/...releases
21. ^1 ^2NVIDIA Technical Blog. "Dynamo 0.4 Delivers 4x Faster Performance, SLO-Based Autoscaling, and Real-Time Observability." August 13, 2025. developer.nvidia.com/...nd-real-time-observability
22. ^InfoQ. "NVIDIA Dynamo Planner Brings SLO-Driven Automation to Multi-Node LLM Inference." January 2026. infoq.com/...nvidia-dynamo-ai-kubernetes
23. ^1 ^2SemiAnalysis. "InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX." March 2026. newsletter.semianalysis.com/...nvidia-blackwell-vs
24. ^NVIDIA Developer. "Dynamo-Triton Open-Source Software." developer.nvidia.com/dynamo-triton
25. ^1 ^2 ^3 ^4 ^5 ^6 ^7NVIDIA Technical Blog. "Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo." August 25, 2026. developer.nvidia.com/...-recovery-in-nvidia-dynamo
26. ^1 ^2 ^3 ^4 ^5NVIDIA Dynamo Documentation. "Shadow Engine Failover." Accessed September 3, 2026. docs.nvidia.com/...shadow-engine-failover
27. ^1 ^2GitHub. "Active-passive GPU failover example - ai-dynamo/dynamo." Accessed September 3, 2026. github.com/...agg_failover.yaml
28. ^1 ^2 ^3 ^4NVIDIA Technical Blog. "Inside NVIDIA Groq 3 LPX: The Low-Latency Inference Accelerator for the NVIDIA Vera Rubin Platform." By Kyle Aubrey and Farshad Ghodsian. March 16, 2026 (updated April 2, 2026). developer.nvidia.com/...nvidia-vera-rubin-platform
29. ^NVIDIA Technical Blog. "How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin." By Seth Weidman, Kirthi Devleker, and Andrew Ling. August 24, 2026. developer.nvidia.com/...ntext-on-nvidia-vera-rubin
30. ^GitHub. "feat(sglang): Add AFD (Attention-FFN Disaggregation) infrastructure," pull request #6570, ai-dynamo/dynamo. Opened and closed February 25, 2026. github.com/...6570
31. ^GitHub. "[FEATURE]: Detailed Usage Guide for AFD Support in Dynamo," issue #4653, ai-dynamo/dynamo. Opened November 27, 2025; closed May 26, 2026. github.com/...4653
32. ^1 ^2GitHub. "[feat] Support AFD," issue #888, ai-dynamo/aiconfigurator. Opened April 22, 2026. github.com/...888
33. ^1 ^2GitHub. "feat(afd): add AFD estimate mode," pull request #1129, ai-dynamo/aiconfigurator. Merged June 8, 2026. github.com/...1129
34. ^1 ^2GitHub. "feat(afd): add AFD default CLI mode built on the v2 Task architecture," pull request #1323, ai-dynamo/aiconfigurator. Merged August 5, 2026. github.com/...1323
35. ^GitHub. "DEP: Native external speculative decoding with independent target and draft pools," issue #13889, ai-dynamo/dynamo. Opened August 27, 2026. github.com/...13889
36. ^GitHub. "[Vision] Toward Dynamo 2.0," issue #9208, ai-dynamo/dynamo. Opened May 6, 2026. github.com/...9208
37. ^GitHub. "feat(lpu): add opaque engine_data field to nvext response with per-request opt-in via extra_fields," pull request #8099, ai-dynamo/dynamo. Merged April 21, 2026. github.com/...8099
38. ^GitHub. "feat(afd) : Rebase/afd onto upstream," pull request #1597, ai-dynamo/aiconfigurator. Opened August 27, 2026. github.com/...1597
39. ^NVIDIA. "NVIDIA Groq 3 LPX: Inference Accelerator for Agentic AI" (product page). Accessed September 6, 2026. nvidia.com/...lpx