# DeepInfer

> Source: https://aiwiki.ai/wiki/deepinfer
> Updated: 2026-10-20
> Categories: AI Inference, Software Development
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "DeepInfer." aiwiki.ai, 20 Oct 2026. https://aiwiki.ai/wiki/deepinfer
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

DeepInfer is an open-source inference framework developed by [DeepSeek](https://aiwiki.ai/wiki/deepseek) for running its models across accelerator clusters. DeepSeek announced its public release on October 20, 2026. The initial release supports [DeepSeek V4.1-Flash](https://aiwiki.ai/wiki/deepseek_v4_1_flash) and provides an offline inference engine, rather than a complete, ready-to-deploy online service.[1][2][16]

## Release and compatibility

| Item | Documented support |
| --- | --- |
| Developer | DeepSeek |
| Public announcement | October 20, 2026 |
| Framework license | Apache License 2.0 |
| Supported model at release | DeepSeek-V4.1-Flash |
| NVIDIA hardware | SM100 GPUs |
| Huawei hardware | Ascend 950 NPUs |
| Operating system | Linux |
| Compiler and language prerequisites | GCC 13.0 or newer; Python 3.12 |
| NVIDIA software prerequisite | CUDA 13 |
| Ascend software prerequisite | CANN 9.2 |

These are the release's documented targets and prerequisites, not a claim that every NVIDIA or Ascend device is supported.[1][2][3]

## Relationship to other inference projects

The repository credits [vLLM](https://aiwiki.ai/wiki/vllm) as its main original foundation, with further inspiration from [SGLang](https://aiwiki.ai/wiki/sglang) and [NVIDIA TensorRT-LLM](https://aiwiki.ai/wiki/tensorrt_llm). The vLLM project acknowledged the release and said it intended to use ideas from DeepInfer in further V4.1-Flash optimization.[1][4]

The academic work behind vLLM addresses [KV cache](https://aiwiki.ai/wiki/kv_cache) fragmentation and sharing through PagedAttention. That work supplies background on the serving problem; its benchmark results are not measurements of DeepInfer.[5]

## Distributed execution

DeepInfer combines attention [data parallelism](https://aiwiki.ai/wiki/data_parallelism), balanced [context parallelism](https://aiwiki.ai/wiki/context_parallelism), and [expert parallelism](https://aiwiki.ai/wiki/expert_parallelism) for [mixture-of-experts](https://aiwiki.ai/wiki/mixture_of_experts) computation.[1][16][20] Its kernel dependencies include [FlashMLA](https://aiwiki.ai/wiki/flashmla), [DeepGEMM](https://aiwiki.ai/wiki/deepgemm), [DeepEP](https://aiwiki.ai/wiki/deepep), TileKernels, DeepSelect, and [TileLang](https://aiwiki.ai/wiki/tilelang).[17]

Expert-parallel load balancing uses recent token-routing statistics to decide which experts need extra physical replicas and how to place those replicas across devices. The implementation distinguishes the model's logical experts from physical copies, so replicating a busy expert changes its placement rather than the model's logical expert count.[6]

The deployment documentation describes a production layout with separate vision-encoder, prefill, and decode stages. Prefill processes the input prompt, while decode generates subsequent tokens. DeepSeek's described production deployment uses attention data parallelism alongside expert parallelism; it says tensor and context parallelism were not in use there. This production choice is distinct from the public engine's supported configuration options.[7]

## Decoding and scheduling

### DSpark verification

DeepInfer supports [speculative decoding](https://aiwiki.ai/wiki/speculative_decoding) through DSpark. Draft tokens carry confidence estimates, and the scheduler uses their expected acceptance probabilities to allocate verification work. Variable-length verification avoids checking the same fixed number of draft tokens for every request. DeepSeek's model report describes DSpark as a semi-autoregressive drafting architecture with confidence-scheduled verification.[8]

The public configuration enables variable-length verification with `DEEPINFER_ENABLE_MTP_VARLEN=1`. `DEEPINFER_USE_NEXTN_DECODE` sets the maximum draft length, documented as five by default.[18]

### Asynchronous scheduling

The scheduler overlaps CPU scheduling for a subsequent batch with accelerator execution of the current batch. This reduces the idle interval otherwise occupied by CPU scheduling and post-processing. DeepSeek describes the design as following existing SGLang and NanoFlow implementations, rather than presenting asynchronous overlap as an entirely new serving technique.[7]

### Sliding-window bounded replay

SWA bounded replay reconstructs sliding-window attention state from a short suffix instead of replaying a much longer history. The model report describes this reconstruction as approximate. It should not be confused with exact reconstruction of every attention state.[8]

The repository contains separate `mask` and `compact` implementations and verification utilities that compare their metadata and model outputs. That equivalence check compares two implementations of the truncation operation; it does not establish equivalence between bounded replay and unrestricted full-history computation.[9]

## Sampling and structured outputs

The `SamplingParams` interface includes temperature, top-p, top-k, maximum output tokens, per-sequence random seeds, and structured-output arguments. A missing `seq_seed` produces a randomly selected 64-bit seed. Supplying the same fixed seed is therefore a separate step from configuring batch-invariant computation.[10]

| Setting | Documented purpose and restriction |
| --- | --- |
| `DEEPINFER_BATCH_INVARIANT=1` | Batch-invariant model computation; documented for NVIDIA only |
| Batch invariance plus `DEEPINFER_ENABLE_MTP_VARLEN=0` and a fixed `seq_seed` | Documented NVIDIA configuration for identical outputs from identical input tokens and sampling parameters |
| `so_args={"schema_type": ..., "schema": ...}` | Output constraints passed to `SamplingParams` |

The repository also includes a regression test that compares a prompt's batched and single-request outputs with batch invariance enabled. The test compares token IDs and probability information; its presence is a test specification, not an independent benchmark of all hardware configurations.[1][11]

Structured output compilation uses XGrammar. The compiler dispatches JSON schemas, regular expressions, and EBNF grammars to the corresponding compilation routines; JSON-schema compilation uses strict mode. Compiled constraints then restrict the tokens available during generation. This controls output format, not the factual accuracy of the generated content.[12][7]

## Benchmarks and interpretation

The repository publishes separate throughput-latency plots for prefill and decode. DeepSeek reports measuring the prefill plot on four SM100 GPUs and the decode plot on 64 SM100 GPUs, each inside one scale-up domain.[1]

The benchmark driver measures steady-state intervals after warmup. Its decode throughput uses full-acceptance accounting: the token count includes all configured draft tokens whether or not a workload would actually accept them. The plotting utility applies an assumed draft-acceptance rate to fixed-length decode sweeps, adjusting both per-user speed and throughput; it rejects this calculation for variable-length verification runs. Consequently, a plotted number must be read together with its acceptance assumption, context length, concurrency, draft length, and device configuration.[13][14]

The examples also warn that profiler timing can distort performance and should not be used to derive steps per second. The sweep scripts disable profiling for measurement points. These are developer-provided engine measurements, not end-to-end measurements of an independently operated online service.[13]

### Model-quality checks

The evaluation suite separates generation from scoring and supplies reference Pass@1 values for checking a V4.1 deployment.[15]

| Evaluation | Tasks | Samples per task | Reported reference Pass@1 |
| --- | ---: | ---: | ---: |
| AIME 2026 | 30 | 32 | 99.9% |
| GPQA Diamond | 198 | 8 | 93.4% |
| HLE-Full | 2,500 | 1 | 42.6% |

For AIME and GPQA, Pass@1 is mean single-sample accuracy across the repeated samples, not the proportion of questions solved by any one of those attempts. HLE uses one sample per task. The documentation calls these reference figures indicative sanity checks, with variation expected between runs.[15]

Scoring defaults to an LLM judge. Rule-based scoring is available for AIME and GPQA, but not HLE. A run's Pass@1 is considered valid only when its `coverage_complete` field is true. These checks measure the configured model's answers; they do not measure serving throughput.[15]

## Running the public release

The official build selects its backend with `DEEPINFER_ACCELERATOR`, then builds dependencies and the extension from source. For NVIDIA, the build and offline example run as follows:[19][20]

```bash
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py
```

The examples use `deepseek-ai/DeepSeek-V4.1-Flash-1017` as their default checkpoint. A model identifier triggers a download through the Hugging Face cache; an explicit local model directory can be used instead. `DEEPINFER_HF_MODEL_DIR` changes the default model location without overriding an explicitly supplied `--model_name`.[16]

The examples provide expert-parallel, data-parallel, and context-parallel configurations, plus text and image regression inputs. Their offline tests include perplexity baselines and optional checks for regular-expression, JSON, and EBNF output constraints.[20]

## Public release boundaries

The public engine does not by itself reproduce DeepSeek's complete online service. The README lists a production KV cache system, vision-encoder/prefill/decode disaggregation, and load balancing across prefill and decode data-parallel replicas as additional requirements not included in the release. DeepSeek's deployment Q&A explains that cache management and request load balancing depend on internal infrastructure.[1][7]

This exclusion concerns the surrounding production system. It should not be read as saying the engine performs inference without attention-state caching or lacks the expert-placement balancing code included in the repository.[6][7]

## References

1. DeepSeek-AI. [DeepInfer: official repository and README](https://github.com/deepseek-ai/DeepInfer). Release documentation reviewed October 20, 2026.
2. DeepSeek. [Public DeepInfer announcement](https://x.com/deepseek_ai/status/2112476496283775135). October 20, 2026.
3. DeepSeek-AI. [DeepInfer Apache License 2.0](https://github.com/deepseek-ai/DeepInfer/blob/main/LICENSE).
4. vLLM. [Acknowledgment of DeepInfer's release and vLLM foundation](https://x.com/vllm_project/status/2112480719952138301). October 20, 2026.
5. Kwon, Woosuk, et al. [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180). 2023.
6. DeepSeek-AI. [DeepInfer expert-parallel load-balancing policy](https://github.com/deepseek-ai/DeepInfer/blob/main/deepinfer/model_executor/eplb/policy.py).
7. DeepSeek-AI. [DeepSeek V4.1 Flash Inference Deployment Q&A](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md). Reviewed October 20, 2026.
8. DeepSeek-AI. [DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression](https://arxiv.org/abs/2609.19969). September 17, 2026. See sections 2.4.3 and 3.2.2.
9. DeepSeek-AI. [YoCo mask/compact equivalence verification](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/yoco/README.md).
10. DeepSeek-AI. [DeepInfer sampling parameters](https://github.com/deepseek-ai/DeepInfer/blob/main/deepinfer/sampling_params.py).
11. DeepSeek-AI. [Batch-invariant regression test](https://github.com/deepseek-ai/DeepInfer/blob/main/tests/models/test_batch_invariant.py).
12. DeepSeek-AI. [DeepInfer structured-output compiler](https://github.com/deepseek-ai/DeepInfer/blob/main/deepinfer/structure_output/compiler.py).
13. DeepSeek-AI. [Offline inference and benchmark examples](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/README.md).
14. DeepSeek-AI. [Throughput-latency plotting implementation](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/sweep/plot_pareto.py).
15. DeepSeek-AI. [Benchmark evaluation methodology and reference results](https://github.com/deepseek-ai/DeepInfer/blob/main/eval/README.md).
16. DeepSeek-AI. [Offline model and launch configuration](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/_config.py) and [example usage](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/README.md).
17. DeepSeek-AI. [Vendored kernel libraries](https://github.com/deepseek-ai/DeepInfer/blob/main/.gitmodules) and [Python kernel dependencies](https://github.com/deepseek-ai/DeepInfer/blob/main/requirements.txt).
18. DeepSeek-AI. [Environment-variable definitions](https://github.com/deepseek-ai/DeepInfer/blob/main/deepinfer/environ.py).
19. DeepSeek-AI. [Backend selection and build entry point](https://github.com/deepseek-ai/DeepInfer/blob/main/build.sh).
20. DeepSeek-AI. [Offline inference and regression example](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/offline_inference.py).