# NVIDIA Triton Inference Server

> Source: https://aiwiki.ai/wiki/nvidia_triton_inference_server
> Updated: 2026-06-21
> Categories: AI Inference, Deep Learning, Developer Tools, NVIDIA
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

*See also: [Model Deployment](/wiki/model_deployment), [artificial intelligence applications](/wiki/artificial_intelligence_applications), and [GPU Computing](/wiki/gpu_computing)*

## Introduction

NVIDIA Triton [Inference](/wiki/inference) Server is open-source [model deployment](/wiki/model_deployment) software that lets teams run trained models from any [machine learning](/wiki/machine_learning) or [deep learning](/wiki/deep_learning) framework on any processor (GPU, CPU, or other accelerator) behind a single standardized serving interface.[^1] Developed by [NVIDIA](/wiki/nvidia), licensed under the BSD 3-Clause license, and distributed as part of NVIDIA AI Enterprise, Triton serves models built with [TensorRT](/wiki/tensorrt), [PyTorch](/wiki/pytorch), [TensorFlow](/wiki/tensorflow), [ONNX](/wiki/onnx), [OpenVINO](/wiki/openvino), Python, and RAPIDS FIL, and adds production features such as dynamic batching, concurrent model execution, and model ensembles to maximize throughput on [GPU](/wiki/gpu_computing)- or CPU-based infrastructure.[^12] NVIDIA describes Triton as open-source software that, "available with NVIDIA AI Enterprise," lets teams "run inference on trained machine learning or deep learning models from any framework on any processor: GPU, CPU, or other."[^12]

Triton delivers fast and scalable AI across cloud, on-premises, edge, and embedded devices, and the source is hosted on GitHub at the `triton-inference-server/server` repository.[^1][^17]

Originally called TensorRT Inference Server, the project was renamed to Triton Inference Server in 2020 to better reflect its multi-framework support beyond [TensorRT](/wiki/tensorrt) alone. In March 2025, NVIDIA folded Triton into the broader [NVIDIA Dynamo](/wiki/nvidia_dynamo) inference platform, and the product is now officially referred to as NVIDIA Dynamo-Triton (described on NVIDIA's developer site as "NVIDIA Dynamo-Triton, formerly NVIDIA Triton Inference Server").[^11] Both the older "Triton Inference Server" name and the newer "Dynamo-Triton" name remain in active use across NVIDIA's documentation and marketing as of 2026. Despite the rebranding, the core software remains the same open-source project with monthly container releases on NVIDIA NGC. As of June 2026, the latest release is version 2.69.0, corresponding to the 26.05 NGC container, with new container builds published roughly every month.[^19]

Triton supports a wide range of [deep learning](/wiki/deep_learning) and [machine learning](/wiki/machine_learning) frameworks, handles dynamic batching and concurrent model execution, exposes both HTTP/REST and [gRPC](/wiki/grpc) endpoints compliant with the KServe inference protocol, publishes [Prometheus](/wiki/prometheus) metrics for monitoring, and integrates with Kubernetes for orchestration.[^1] It is one of the inference engines that powers NVIDIA NIM microservices and is used in production at companies ranging from startups to large enterprises.[^13]

## How does Triton Inference Server work?

Triton follows a modular architecture built around several core components that work together to receive inference requests, schedule them efficiently, and dispatch them to the appropriate backend for execution.[^2]

### Request Flow

Inference requests enter Triton through one of three interfaces: HTTP/REST, gRPC, or the in-process C API. Each incoming request is routed to a per-model scheduler based on the model name specified in the request. The scheduler may hold the request temporarily to form a batch, then passes the batched request to the backend responsible for executing that model. After the backend completes the computation, the result travels back through the scheduler and out through the same interface that received the original request.[^2]

### Model Repository

The model repository is a file-system-based store of all models that Triton makes available for inference.[^3] Triton is launched with the `--model-repository` flag pointing to one or more repository paths. Each model occupies its own subdirectory, and within that directory there are numbered version subdirectories containing the actual model files. The general layout is:

```
<model-repository-path>/
  <model-name>/
    config.pbtxt
    1/
      model.plan      (TensorRT)
    2/
      model.plan
  <another-model>/
    config.pbtxt
    1/
      model.onnx      (ONNX Runtime)
```

Triton supports model repositories on local file systems as well as cloud object storage services including Amazon S3 (`s3://`), Google Cloud Storage (`gs://`), and Azure Blob Storage (`as://`).[^3] This makes it straightforward to deploy Triton in cloud environments without copying model files to local disk. Recent releases have extended this further: the 26.05 container added Azure Managed Identity authentication for Azure Storage model repositories, removing the need to embed storage credentials directly.[^19]

A version policy in each model's configuration controls which versions are active at any time. The three policies are `all` (serve every version), `latest` (serve the most recent *n* versions), and `specific` (serve only the listed versions). The default is to serve the single latest version.[^3]

### Model Configuration

Each model's behavior is governed by a Protocol Buffers text file named `config.pbtxt`.[^4] Key configuration fields include:

| Field | Description |
|---|---|
| `backend` or `platform` | Specifies which execution backend to use (e.g., `tensorrt_plan`, `pytorch_libtorch`, `onnxruntime_onnx`) |
| `max_batch_size` | Largest batch the model accepts. Set to 0 for models that do not support batching. |
| `input` / `output` | Defines tensor names, data types, and dimensions for each model input and output |
| `instance_group` | Controls how many parallel copies of the model to run and on which devices (GPU or CPU) |
| `dynamic_batching` | Enables and configures the dynamic batcher for the model |
| `sequence_batching` | Enables sequence-aware batching for stateful models |
| `ensemble_scheduling` | Defines the pipeline of models and tensor mappings for an ensemble model |
| `optimization` | Specifies framework-level acceleration (e.g., TensorRT for ONNX, OpenVINO for CPU) |
| `model_warmup` | Pre-runs inference requests at load time to eliminate cold-start latency |
| `version_policy` | Controls which model versions are active |

Triton can also auto-generate a minimal configuration for many backends if no `config.pbtxt` is provided, using the `--strict-model-config=false` flag.[^4]

## What frameworks does Triton support?

One of Triton's defining strengths is its support for a wide variety of model formats and execution backends. Each backend is responsible for loading and executing models of a particular type, and backends are implemented against Triton's stable Backend C API so they can be developed and distributed independently of the core server.[^10]

| Backend | Framework / Format | Default Model Filename | Notes |
|---|---|---|---|
| TensorRT | [TensorRT](/wiki/tensorrt) Plans | `model.plan` | GPU-optimized inference with INT8/FP16 precision. Plans are specific to GPU compute capability. |
| PyTorch | [PyTorch](/wiki/pytorch) TorchScript and PyTorch 2.0 | `model.pt` | Supports both TorchScript-serialized models and newer PyTorch export formats |
| TensorFlow | [TensorFlow](/wiki/tensorflow) SavedModel and GraphDef | `model.savedmodel` or `model.graphdef` | Supports TensorFlow 1.x and 2.x models |
| ONNX Runtime | [ONNX](/wiki/onnx) models | `model.onnx` | Broad compatibility with any framework that exports ONNX. Supports TensorRT and OpenVINO acceleration. |
| OpenVINO | [OpenVINO](/wiki/openvino) IR format | `model.xml` + `model.bin` | Optimized CPU inference from Intel |
| Python | Custom Python code | `model.py` | Enables arbitrary preprocessing, postprocessing, or model logic in Python. Also serves as the host for vLLM backend. |
| TensorRT-LLM | [Large language models](/wiki/large_language_model) optimized with TensorRT-LLM | Compiled engine files | Optimized specifically for LLM inference with in-flight batching and paged KV cache |
| vLLM | LLMs via [vLLM](/wiki/vllm) engine | Python-based | Runs vLLM as a Triton backend, combining vLLM's PagedAttention with Triton's serving infrastructure |
| FIL (Forest Inference Library) | [XGBoost](/wiki/xgboost), LightGBM, [scikit-learn](/wiki/scikit_learn) RandomForest, RAPIDS cuML | Treelite format | High-performance tree-based model inference with SHAP explainability on GPUs and CPUs |
| DALI | NVIDIA Data Loading Library | `model.dali` | Hardware-accelerated data preprocessing pipelines |
| Custom C++ | User-defined backends | Varies | Triton's Backend C API allows developers to write entirely custom backends |

This multi-framework support means that an organization can serve a [TensorRT](/wiki/tensorrt)-optimized vision model, a [PyTorch](/wiki/pytorch) text classifier, and a Python-based preprocessing pipeline all from the same Triton instance, without needing separate serving infrastructure for each.[^10] NVIDIA's product materials summarize this as the ability to deploy models on any major framework, including TensorFlow, PyTorch, Python, ONNX, TensorRT, RAPIDS cuML, XGBoost, scikit-learn RandomForest, OpenVINO, and custom C++ backends.[^12]

## What is dynamic batching in Triton?

Dynamic batching is the single Triton feature that provides the largest performance improvement for most workloads. When enabled, the dynamic batcher combines individual inference requests that arrive within a short time window into a single larger batch before sending that batch to the model for execution.[^9] This allows the GPU to process more data per kernel launch, increasing throughput substantially.

### How Dynamic Batching Works

When an inference request arrives, the dynamic batcher places it in a queue. The batcher continuously checks whether the queued requests can form a batch of a preferred size. If a preferred batch size is reached, the batch is dispatched immediately. If not, the batcher waits up to a configurable delay (`max_queue_delay_microseconds`) for additional requests to arrive. Once the delay expires or the preferred size is met, whichever comes first, the batch is sent to the backend.[^9]

Key configuration parameters for dynamic batching include:

| Parameter | Description |
|---|---|
| `preferred_batch_size` | A list of batch sizes the batcher prefers to form (e.g., `[4, 8]`) |
| `max_queue_delay_microseconds` | Maximum time a request may wait in the queue before the batcher sends a partial batch |
| `priority_levels` | Number of priority levels for the queue. Higher-priority requests are batched first. |
| `default_priority_level` | The priority assigned to requests that do not specify one |
| `preserve_ordering` | When true, responses are returned in the same order as requests arrived |

In the worked example in NVIDIA's optimization documentation, enabling dynamic batching on an Inception ONNX model raised throughput from about 73 inferences per second (without batching) to 272 inferences per second with eight concurrent requests, and NVIDIA notes this came "without increasing latency compared to not using the dynamic batcher."[^5]

### Sequence Batching

For stateful models that must process ordered sequences of requests (such as recurrent networks or models with temporal context), Triton provides a sequence batcher. The sequence batcher ensures that all requests belonging to the same sequence are routed to the same model instance, maintaining state across requests. Configuration options include sequence timeout duration and control signals for sequence start, end, ready, and correlation ID.[^9]

## Concurrent Model Execution

Triton can run multiple instances of one or more models simultaneously, overlapping compute and memory-transfer operations to maximize GPU and CPU utilization.

### Instance Groups

The `instance_group` configuration field controls how many parallel copies of a model are loaded and on which devices.[^4] By default, Triton creates one instance per available GPU for each model. This can be customized to run multiple instances on a single GPU, distribute instances across specific GPUs, run instances on CPU, or mix GPU and CPU instances.

For example, the following configuration runs two instances on GPU 0 and one on CPU:

```
instance_group [
  { count: 2, kind: KIND_GPU, gpus: [0] },
  { count: 1, kind: KIND_CPU }
]
```

Having multiple instances allows Triton to overlap memory transfer operations with inference computation. While one instance is executing a forward pass, another can be loading input data. In NVIDIA's optimization example, allowing two instances of the Inception ONNX model raised throughput from roughly 73 to about 110 inferences per second at a concurrency of 2, although the benefit varies by model architecture.[^5]

### Rate Limiter

Triton includes a rate limiter that controls the rate at which requests are scheduled across model instances. This is useful when multiple models share the same GPU and you want to prevent one model from consuming all the compute resources. The rate limiter assigns resource costs to each model instance and ensures that the total resource consumption stays within configured limits.[^2]

## Model Ensembles and Pipelines

Triton accommodates modern inference requirements where a single client query may involve multiple models with pre- and post-processing steps.

### Ensemble Models

An ensemble model represents a pipeline of one or more models connected through their input and output tensors.[^7] The ensemble scheduler manages the dataflow between component models, routing the output tensors of one step to the input tensors of the next step as defined in the configuration. This avoids intermediate network round trips because all models in the ensemble execute within the same Triton instance.

The ensemble scheduler works as follows:

1. Maps ensemble input tensors to the inputs of the first component model
2. Sends an internal request to the first model when all its inputs are ready
3. Collects output tensors from the completed model
4. Routes those outputs to dependent downstream models according to the configured tensor mappings
5. Repeats until all pipeline steps are complete
6. Returns the final output tensors as the ensemble response

Ensemble models can include components running on different devices (some on GPU, some on CPU) and using different frameworks. For example, a pipeline might use a Python model for preprocessing, an ONNX model for feature extraction, and a TensorRT model for classification, all chained together in a single ensemble.

The `max_inflight_requests` setting, added in the 25.12 release, prevents memory accumulation when upstream models produce outputs faster than downstream models consume them. When this limit is reached, the scheduler pauses upstream models until downstream processing catches up.[^20]

### Business Logic Scripting (BLS)

For pipelines that require loops, conditionals, data-dependent branching, or other custom logic that cannot be expressed as a static dataflow graph, Triton offers Business Logic Scripting. BLS is most commonly written in the Python backend, where, starting with the 21.08 release, a set of utility functions lets a Python model issue inference requests to other models served by the same Triton instance. Custom C++ backends can do the equivalent through Triton's in-process C API. Either way, BLS provides full programmatic control over the inference pipeline while still using Triton's optimized model execution for each individual model call.[^21]

## Inference Protocols and APIs

Triton exposes inference capabilities through standardized network protocols, making it easy to integrate with existing application infrastructure.

### HTTP/REST and gRPC Endpoints

Triton serves requests on three default ports:

| Port | Protocol | Purpose |
|---|---|---|
| 8000 | HTTP/REST | Inference requests, model management, health checks |
| 8001 | gRPC | Inference requests, model management, health checks |
| 8002 | HTTP | Prometheus metrics endpoint |

Both the HTTP and gRPC interfaces implement the standard inference protocol proposed by the KServe project.[^8] The available API endpoints include:

| Endpoint Category | Description |
|---|---|
| Health | Server liveness and readiness probes; model readiness checks |
| Metadata | Server version and extension info; per-model metadata including input/output specifications |
| Inference | Synchronous inference requests; gRPC also supports bi-directional streaming for sequence models |
| Model Management | Load and unload models at runtime without restarting the server |
| Statistics | Per-model inference statistics including request counts and latencies |
| Model Repository | Query available models in the repository |

### KServe Integration

Triton implements the KServe V2 inference protocol, making it a drop-in serving runtime for [KServe](/wiki/kserve) (formerly KFServing) deployments on Kubernetes. KServe is the standard model inference platform on Kubernetes, and Triton's protocol compliance means it can be used as the inference backend in KServe InferenceService resources without any custom adapters. This enables features like canary deployments, autoscaling, and traffic routing managed by KServe's control plane.

Triton also extends the KServe protocol with additional capabilities including shared memory support, model configuration queries, tracing, logging, and statistics endpoints.[^8]

### gRPC Streaming

The gRPC interface supports bi-directional streaming inference RPCs in addition to standard unary calls. Streaming is useful in scenarios where a sequence of inference requests must be routed to the same Triton server instance (for example, behind a load balancer), or when order-critical sequences need to maintain a persistent connection. NVIDIA recommends using unary gRPC calls for standard inference and reserving streaming for situations that specifically require it.

gRPC connections can be secured with SSL/TLS, and response compression can be configured for bandwidth-sensitive deployments.[^8] The `--grpc-infer-thread-count` flag was exposed as a server option in the 25.04 release to let operators tune the number of handler threads for gRPC inference requests.[^22]

## Metrics and Monitoring

Triton provides comprehensive Prometheus-compatible metrics for monitoring inference performance and resource utilization in production.

### Prometheus Metrics Endpoint

By default, Triton exposes metrics at `http://localhost:8002/metrics`.[^6] The endpoint address can be customized with the `--metrics-port` and `--metrics-address` flags. Metrics are pulled by Prometheus scrapers and are not pushed to any remote server.

### Available Metrics

| Metric Category | Key Metrics | Description |
|---|---|---|
| Inference Counts | `nv_inference_request_success`, `nv_inference_request_failure`, `nv_inference_count`, `nv_inference_exec_count` | Track successful and failed requests, total inferences performed, and batch execution counts per model |
| Latency (Counters) | `nv_inference_request_duration_us`, `nv_inference_queue_duration_us`, `nv_inference_compute_input_duration_us`, `nv_inference_compute_infer_duration_us`, `nv_inference_compute_output_duration_us` | Break down end-to-end request time into queue time, input processing, model execution, and output processing |
| Latency (Histograms) | `nv_inference_first_response_histogram_ms` | Experimental histogram of time to first response (enable with `--metrics-config histogram_latencies=true`) |
| Latency (Summaries) | `nv_inference_request_summary_us`, `nv_inference_queue_summary_us` | Experimental quantile summaries (enable with `--metrics-config summary_latencies=true`) |
| GPU | `nv_gpu_utilization`, `nv_gpu_memory_total_bytes`, `nv_gpu_memory_used_bytes`, `nv_gpu_power_usage`, `nv_energy_consumption` | GPU utilization rate, memory usage, power consumption, and energy since startup. Collected via DCGM. |
| CPU | `nv_cpu_utilization`, `nv_cpu_memory_total_bytes`, `nv_cpu_memory_used_bytes` | System-level CPU and memory usage (Linux only) |
| Pinned Memory | `nv_pinned_memory_pool_total_bytes`, `nv_pinned_memory_pool_used_bytes` | Pinned memory pool utilization (available since release 24.01) |
| Response Cache | `nv_cache_num_hits_per_model`, `nv_cache_num_misses_per_model`, `nv_cache_hit_duration_per_model`, `nv_cache_miss_duration_per_model` | Cache hit/miss rates and lookup durations per model |
| Pending Requests | `nv_inference_pending_request_count` | Number of requests awaiting backend execution |

These metrics integrate naturally with [Grafana](/wiki/grafana) dashboards and Kubernetes-based monitoring stacks.[^6] When running Triton on Kubernetes, a PodMonitor or ServiceMonitor resource tells Prometheus to scrape the metrics endpoint from all Triton pods.

## Performance Optimization

Triton provides several tools and techniques for maximizing inference throughput and minimizing latency.

### Performance Analyzer (perf_analyzer)

The Performance Analyzer is a command-line tool that sends synthetic inference requests to a running Triton instance and measures throughput and latency at various concurrency levels. It is the primary tool for benchmarking model performance and testing the effects of configuration changes such as batch size, instance count, and precision settings.[^5]

### GenAI-Perf Analyzer

For [large language models](/wiki/large_language_model) and multimodal models, the GenAI-Perf Analyzer extends Performance Analyzer with LLM-specific metrics including time to first token, inter-token latency, and output token throughput.

### Model Analyzer

The Triton Model Analyzer automates the process of finding the optimal deployment configuration for one or more models. It sweeps through combinations of batch sizes, instance counts, and precision settings, runs performance tests for each combination, and reports the configurations that meet specified quality-of-service constraints (for example, maximum p99 latency) while maximizing throughput. Model Analyzer also profiles GPU memory usage, which is essential for determining how many models can share a single GPU.

### Framework-Specific Acceleration

Triton supports backend-specific optimizations that can dramatically improve performance:

- **TensorRT acceleration for ONNX models**: By configuring TensorRT as an execution accelerator in the ONNX backend's optimization policy, ONNX models can be compiled into TensorRT engines at load time. In NVIDIA's DenseNet ONNX example, this improved throughput from 138.2 to 273.8 inferences per second while cutting latency roughly in half (from about 14,500 to 7,300 microseconds).[^5]
- **OpenVINO acceleration for CPU inference**: ONNX models running on CPU can be accelerated by configuring [OpenVINO](/wiki/openvino) as the CPU execution accelerator.
- **NUMA-aware placement**: On multi-socket CPU servers, Triton's host policy configuration can bind model instances to specific NUMA nodes and CPU cores, optimizing memory access patterns.

### Response Cache

Triton includes an optional response cache that stores inference results for repeated inputs. When an identical request arrives, the cached result is returned without re-executing the model. This is particularly useful for workloads with high input repetition, such as lookup-heavy recommendation pipelines.

### Model Warmup

The `model_warmup` configuration option triggers a set of inference requests when a model is first loaded, ensuring that GPU kernels are compiled and caches are populated before production traffic arrives. This eliminates the latency spike that would otherwise occur on the first real request.

## GPU, CPU, and Mixed Inference

Triton supports inference on a variety of hardware platforms:

- **NVIDIA GPUs**: All [CUDA](/wiki/cuda)-capable NVIDIA GPUs, including datacenter GPUs (A100, H100, B200), workstation GPUs, and Jetson edge devices (though 26.02 / version 2.66.0 was the final release to ship Jetson artifacts on GitHub)[^29]
- **x86 CPUs**: Using backends like ONNX Runtime, OpenVINO, and the Python backend
- **ARM CPUs**: Supported for edge and embedded deployments
- **AWS Inferentia**: Custom accelerator chips from Amazon Web Services

Mixed inference configurations are common in production. For example, a computer vision pipeline might run a lightweight Python-based preprocessing model on CPU while the heavy neural network runs on GPU. Triton's instance group configuration makes this straightforward by allowing each model to specify its own target device.

## How does Triton serve large language models?

Triton supports inference for [large language models](/wiki/large_language_model) through multiple backends:[^10]

- **TensorRT-LLM backend**: Provides maximum performance for LLM inference on NVIDIA GPUs with features including in-flight batching, paged KV cache, INT4/INT8/FP8 quantization, tensor parallelism across multiple GPUs, and pipeline parallelism across multiple nodes.[^10] These optimizations can compound: in one NVIDIA study, reusing the [KV cache](/wiki/kv_cache) by offloading it to CPU memory accelerated time to first token by up to 14x on x86-based H100 systems for multi-turn workloads.[^23]
- **vLLM backend**: Runs the [vLLM](/wiki/vllm) engine within Triton, bringing vLLM's PagedAttention memory management and continuous batching to Triton's serving infrastructure.
- **Python backend**: Can host any Python-based LLM framework as a custom model.

For very large models that do not fit on a single GPU, Triton supports model partitioning across multiple GPUs within a single server or across multiple servers using tensor parallelism and pipeline parallelism.

## Production Use and NVIDIA NIM

Triton is widely deployed in production environments across industries including healthcare, finance, retail, manufacturing, and logistics.

### NVIDIA NIM

NVIDIA NIM (NVIDIA Inference Microservices) are pre-built, production-ready containers that package optimized AI models with the NVIDIA inference stack.[^13] Historically, NIM containers used Triton Inference Server alongside [TensorRT](/wiki/tensorrt) and TensorRT-LLM to deliver optimized inference, with models that had already been optimized, configured for Triton, and tested extensively, reducing deployment times from weeks to minutes.[^13] As NIM has matured, its backend selection has broadened: a modern NIM container inspects the model's format, architecture, and quantization and automatically chooses an optimal runtime among [vLLM](/wiki/vllm), [SGLang](/wiki/sglang), and TensorRT-LLM, and the unified NIM workflow integrates with both the Triton Inference Server and [NVIDIA Dynamo](/wiki/nvidia_dynamo).[^24]

NIM microservices are available through NVIDIA's own hosted endpoints as well as through major cloud providers including AWS, Google Cloud, and Microsoft Azure.[^13]

### Cloud Platform Support

Triton is supported as a serving runtime on a broad range of cloud platforms and managed ML services:

- Amazon SageMaker, Amazon EKS, and Amazon ECS
- Google Vertex AI and Google Kubernetes Engine (GKE)
- Microsoft Azure Machine Learning and Azure Kubernetes Service (AKS)
- [Alibaba Cloud](/wiki/alibaba_cloud)
- Oracle Cloud Infrastructure Data Science Platform
- HPE Ezmeral

### Kubernetes Deployment

Triton is distributed as a Docker container, making it straightforward to deploy on any Kubernetes cluster.[^18] In a Kubernetes environment, Triton benefits from:

- Horizontal pod autoscaling based on GPU utilization or custom inference metrics
- Rolling updates for zero-downtime model version changes
- Service mesh integration for traffic management and observability
- Integration with KServe for standardized model serving workflows

NVIDIA has documented patterns for running Triton at scale on Kubernetes together with Multi-Instance GPU (MIG), which partitions a single A100 or H100 into isolated GPU slices so that several Triton pods can share one physical GPU with guaranteed quality of service.[^18]

### Enterprise Support

NVIDIA provides enterprise-grade support for Triton through the NVIDIA AI Enterprise subscription, which includes guaranteed response times, priority security notifications, regular production branch updates with a 9-month support lifecycle, and access to NVIDIA AI experts.[^11]

## How does Triton compare to vLLM, TGI, and other serving frameworks?

The table below compares Triton with other popular inference serving solutions as of early 2026.[^14][^15]

| Feature | NVIDIA Triton | [vLLM](/wiki/vllm) | Text Generation Inference (TGI) | [Ray Serve](/wiki/ray_serve) | [BentoML](/wiki/bentoml) |
|---|---|---|---|---|---|
| **Primary Focus** | General-purpose, multi-framework serving | LLM serving | LLM serving | General-purpose serving with autoscaling | ML model packaging and serving |
| **Supported Frameworks** | TensorRT, PyTorch, TensorFlow, ONNX, OpenVINO, Python, FIL, vLLM, TensorRT-LLM, DALI | Hugging Face Transformers (LLMs) | Hugging Face Transformers (LLMs) | Any Python model; integrates vLLM, TensorRT-LLM | Any Python model; can use Triton as a runner |
| **Dynamic Batching** | Yes (configurable per model) | Continuous batching | Continuous batching | Yes (custom batching logic) | Yes (adaptive batching) |
| **Multi-Model Serving** | Yes (concurrent execution on same GPU) | No (single model per instance) | No (single model per instance) | Yes (multi-deployment composition) | Yes (multi-model services) |
| **Model Ensemble/Pipeline** | Native ensemble scheduler and BLS | No | No | Deployment graph composition | Service composition |
| **LLM Optimization** | TensorRT-LLM backend with paged KV cache, in-flight batching | PagedAttention, continuous batching | Flash attention, continuous batching | Delegates to vLLM or TensorRT-LLM | Delegates to underlying runtime |
| **Protocol** | KServe-compliant HTTP + gRPC | OpenAI-compatible HTTP | OpenAI-compatible HTTP | HTTP (custom endpoints) | HTTP (custom endpoints) |
| **GPU Support** | NVIDIA GPUs, multi-GPU, multi-node | NVIDIA, AMD, Intel, TPU | NVIDIA GPUs | Any (via backend) | Any (via backend) |
| **CPU Inference** | Yes (OpenVINO, ONNX Runtime) | Limited | No | Yes | Yes |
| **Autoscaling** | Via Kubernetes/KServe | Manual or via orchestrator | Manual or via orchestrator | Built-in autoscaling with custom policies | Via BentoCloud or Kubernetes |
| **Metrics** | Prometheus (GPU, latency, throughput, cache) | Prometheus | Prometheus | Prometheus, custom metrics | Prometheus |
| **Ease of Setup** | Moderate to complex (config.pbtxt, model repository) | Simple (few CLI flags) | Simple (Docker container) | Moderate (Python decorators) | Simple (Python decorators) |
| **Language** | Core in C++; Python wrapper (PyTriton) | Python | Rust core, Python interface | Python | Python |
| **License** | BSD 3-Clause | Apache 2.0 | Apache 2.0 (was HFOSL) | Apache 2.0 | Apache 2.0 |
| **Status (2026)** | Active development (Dynamo-Triton) | Active development | Maintenance mode (since Dec 2025) | Active development | Active development |

### When to Choose Each

- **NVIDIA Triton** is the best fit for enterprises running complex, multi-model inference pipelines on NVIDIA hardware, especially when models span multiple frameworks and require fine-grained performance tuning. It is the only option that provides native concurrent model execution on optimized C++ backends with ensemble orchestration.
- **[vLLM](/wiki/vllm)** is the default choice for teams focused on serving large language models with high throughput. Its PagedAttention memory management achieves the best memory utilization among LLM serving frameworks, and its [OpenAI](/wiki/openai)-compatible API simplifies integration.
- **TGI** (Text Generation Inference) was a strong option for [Hugging Face](/wiki/hugging_face)-centric teams, but its maintainers placed it in maintenance mode on December 11, 2025, accepting only minor bug fixes and documentation changes going forward. Hugging Face now recommends vLLM or [SGLang](/wiki/sglang) for new deployments.[^25]
- **[Ray Serve](/wiki/ray_serve)** excels at multi-model composition with built-in autoscaling and is well suited for teams already using the Ray ecosystem. It can delegate LLM inference to vLLM or TensorRT-LLM while handling orchestration, routing, and scaling.
- **[BentoML](/wiki/bentoml)** prioritizes developer experience with a Pythonic API for packaging and versioning models. Starting with BentoML v1.0.16, Triton can be used as a runner within BentoML, combining BentoML's ease of use with Triton's high-performance inference.[^16]

## PyTriton: Python-Native Interface

PyTriton is a Flask/FastAPI-like interface for Python developers who want to use Triton's serving capabilities without writing `config.pbtxt` files or structuring model repositories manually.[^12] With PyTriton, developers define inference functions as decorated Python callables and bind them to a Triton instance programmatically. This enables rapid prototyping and testing while maintaining access to Triton's dynamic batching, concurrent execution, and HTTP/gRPC serving.

PyTriton is particularly useful for serving custom preprocessing logic, prototype models during development, and inference pipelines that are easiest to express in pure Python.

## Model Orchestration

Triton includes model orchestration functionality designed for efficient multi-model inference at scale. The orchestration service loads models on demand, unloads inactive models to free GPU memory, and allocates resources effectively by placing as many models as possible on a single GPU server. This is especially valuable in multi-tenant environments where hundreds of models may be registered but only a subset is actively serving traffic at any given time.

## Ecosystem Integrations

Triton is supported by a variety of cloud platforms, [MLOps](/wiki/mlops) tools, and services:

- **Cloud platforms**: Alibaba Cloud, Amazon EKS, Amazon ECS, Amazon SageMaker, Google GKE, Google Vertex AI, HPE Ezmeral, Microsoft AKS, Azure Machine Learning, Oracle Cloud Infrastructure Data Science
- **Orchestration**: Kubernetes, KServe, Docker, NVIDIA Fleet Command
- **Monitoring**: [Prometheus](/wiki/prometheus), Grafana, Datadog (via integration)
- **MLOps tools**: [MLflow](/wiki/mlflow), [Kubeflow](/wiki/kubeflow), Seldon Core, BentoML
- **Data preprocessing**: NVIDIA DALI, RAPIDS

## Success Stories

Companies such as Amazon, American Express, Siemens Energy, and [Perplexity AI](/wiki/perplexity_ai) have successfully adopted NVIDIA Triton in production. [Perplexity](/wiki/perplexity) AI, for example, serves over 400 million search queries per month using the NVIDIA inference stack, combining H100 GPUs, Triton, and TensorRT-LLM to run more than 20 models simultaneously, and it has worked with NVIDIA's Triton engineering team to deploy disaggregated prefill and decode serving.[^13] American Express uses Triton for real-time fraud detection, while Siemens Energy applies it to AI-based remote monitoring for physical inspections.[^13]

## Developer Resources

NVIDIA provides comprehensive documentation and learning resources for Triton:

- **Official documentation**: Full user guide, API reference, and backend-specific guides at `docs.nvidia.com`[^1]
- **GitHub repositories**: Source code, examples, and issue tracking under the `triton-inference-server` GitHub organization
- **NGC containers**: Pre-built Docker containers released monthly on NVIDIA NGC
- **NVIDIA LaunchPad**: Free hosted labs for hands-on experience with Triton
- **Tutorials**: Step-by-step guides covering installation, model deployment, performance optimization, and integration with popular frameworks
- **Community forums**: Platform for connecting with other Triton users, sharing best practices, and getting help with deployment challenges

## Is Triton the same as NVIDIA Dynamo?

In March 2025, NVIDIA introduced [NVIDIA Dynamo](/wiki/nvidia_dynamo) at GTC, a separate open-source, low-latency inference framework designed for distributed serving of generative AI and reasoning models.[^26] Dynamo focuses on disaggregated serving, where the prefill (prompt processing) and decode (token generation) phases of LLM inference are split across different GPU pools for optimal resource utilization. It pairs this with a KV-cache-aware request router that hashes incoming prompts, tracks where matching key-value blocks already live, and routes each request to the GPU that maximizes cache reuse, plus dynamic GPU scheduling that shifts capacity between prefill and decode as demand shifts.[^26] Triton Inference Server has been folded under the Dynamo platform umbrella, and the combined offering is marketed as NVIDIA Dynamo-Triton.[^11]

It is worth being precise about how the two products relate, because the names are easy to confuse. Dynamo-Triton is the renamed, general-purpose Triton Inference Server for serving models of any type across any framework. NVIDIA Dynamo is a distinct, newer datacenter-scale framework aimed specifically at distributed LLM inference, and NVIDIA positions it as complementing Dynamo-Triton with LLM-specific optimizations such as disaggregated serving, prefix caching, and offloading the KV cache to lower-cost storage.[^11] Dynamo is built around modular components including NIXL for high-speed GPU-to-GPU KV-cache transfer (over NVLink, InfiniBand, or UCX), KVBM for memory management, and Grove for scaling.[^27]

### NVIDIA Dynamo 1.0

On March 16, 2026, NVIDIA announced that Dynamo had entered production with the release of Dynamo 1.0, which the company describes as an open-source "inference operating system" for AI factories.[^27] NVIDIA reports that Dynamo can boost the number of requests served by up to 7x on [Blackwell](/wiki/blackwell)-generation GPUs, "as demonstrated in the recent SemiAnalysis InferenceX benchmark" running DeepSeek-R1.[^28] Dynamo 1.0 integrates with open-source frameworks including [vLLM](/wiki/vllm), [SGLang](/wiki/sglang), LMCache, and [llm-d](/wiki/llm_d), and NVIDIA lists production adopters spanning cloud providers (AWS, Microsoft Azure, Google Cloud, Oracle Cloud Infrastructure) and AI-native companies such as [Perplexity](/wiki/perplexity), Cursor, ByteDance, PayPal, Pinterest, Baseten, and Fireworks.[^27] For teams that have standardized on Triton, this means the broader NVIDIA inference roadmap is increasingly centered on Dynamo for large-scale, multi-node generative AI, while Dynamo-Triton remains the workhorse for single-server and multi-framework serving.

### Continued Triton Releases and Support

Independent of the Dynamo branding, the Triton/Dynamo-Triton codebase continues to ship monthly NGC containers, reaching version 2.69.0 (container 26.05) in June 2026.[^19] Recent releases have added a Rust gRPC client library, Azure Managed Identity authentication for model repositories, `GPU_DEVICE_IDS` support for pinning vLLM models to specific GPUs, and a series of hardening fixes such as capping the number of HTTP request chunks to prevent memory exhaustion.[^19] One notable platform change is that Triton 26.02 (version 2.66.0) was the last release to publish Jetson artifacts on GitHub, signaling a wind-down of first-party Jetson packages for newer versions.[^29]

NVIDIA AI Enterprise customers using Triton continue to receive production branch support for their existing deployments, with monthly patches for security vulnerabilities and a 9-month lifecycle for API stability.[^11]

NVIDIA continues to invest in Triton's development, incorporating new features and improvements based on user feedback and industry needs. Areas of ongoing work include expanded backend support, improved orchestration capabilities for multi-tenant serving, enhanced LLM inference performance, and deeper integration with the Dynamo distributed inference framework.[^11]

## References

1. NVIDIA. "NVIDIA Triton Inference Server Documentation." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
2. NVIDIA. "Triton Architecture." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/architecture.html
3. NVIDIA. "Model Repository." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_repository.html
4. NVIDIA. "Model Configuration." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_configuration.html
5. NVIDIA. "Optimization." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/optimization.html
6. NVIDIA. "Metrics." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/metrics.html
7. NVIDIA. "Ensemble Models." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/ensemble_models.html
8. NVIDIA. "Inference Protocols and APIs." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inference_protocols.html
9. NVIDIA. "Batchers." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/batcher.html
10. NVIDIA. "Triton Inference Server Backend." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/README.html
11. NVIDIA. "Dynamo-Triton Open-Source Software." *developer.nvidia.com*. https://developer.nvidia.com/dynamo-triton
12. NVIDIA. "Triton Inference Server for Every AI Workload." *nvidia.com*. https://www.nvidia.com/en-us/ai/dynamo-triton/
13. NVIDIA. "NVIDIA NIM Microservices." *nvidia.com*. https://www.nvidia.com/en-us/ai-data-science/products/nim-microservices/
14. PremAI. "LLM Inference Servers Compared: vLLM vs TGI vs SGLang vs Triton (2026)." *blog.premai.io*. https://blog.premai.io/llm-inference-servers-compared-vllm-vs-tgi-vs-sglang-vs-triton-2026/
15. Clarifai. "vLLM vs Triton vs TGI: Choosing the Right LLM Serving Framework." *clarifai.com*. https://www.clarifai.com/blog/model-serving-framework/
16. BentoML. "BentoML Or Triton Inference Server? Choose Both!" *bentoml.com*. https://www.bentoml.com/blog/bentoml-or-triton-inference-server-choose-both
17. GitHub. "triton-inference-server/server." *github.com*. https://github.com/triton-inference-server/server
18. NVIDIA. "Deploying NVIDIA Triton at Scale with MIG and Kubernetes." *developer.nvidia.com*. https://developer.nvidia.com/blog/deploying-nvidia-triton-at-scale-with-mig-and-kubernetes/
19. GitHub. "Release Release 2.69.0 corresponding to NGC container 26.05." *github.com*. https://github.com/triton-inference-server/server/releases/tag/v2.69.0
20. NVIDIA. "Triton Inference Server Release 25.12." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-25-12.html
21. NVIDIA. "Business Logic Scripting." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/bls.html
22. NVIDIA. "Triton Inference Server Release 25.04." *docs.nvidia.com*. https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/Chunk786889861.html
23. NVIDIA. "5x Faster Time to First Token with NVIDIA TensorRT-LLM KV Cache Early Reuse." *developer.nvidia.com*. https://developer.nvidia.com/blog/5x-faster-time-to-first-token-with-nvidia-tensorrt-llm-kv-cache-early-reuse/
24. NVIDIA. "Simplify LLM Deployment and AI Inference with a Unified NVIDIA NIM Workflow." *developer.nvidia.com*. https://developer.nvidia.com/blog/simplify-llm-deployment-and-ai-inference-with-unified-nvidia-nim-workflow/
25. GitHub. "huggingface/text-generation-inference." *github.com*. https://github.com/huggingface/text-generation-inference
26. NVIDIA. "NVIDIA Dynamo, A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Models." *developer.nvidia.com*. https://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/
27. NVIDIA. "How NVIDIA Dynamo 1.0 Powers Multi-Node Inference at Production Scale." *developer.nvidia.com*. https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/
28. NVIDIA. "NVIDIA Enters Production With Dynamo, the Broadly Adopted Inference Operating System for AI Factories." *nvidianews.nvidia.com*. https://nvidianews.nvidia.com/news/dynamo-1-0
29. GitHub. "Releases - triton-inference-server/server." *github.com*. https://github.com/triton-inference-server/server/releases