DeepInfer

DeepInfer is an open-source inference framework developed by DeepSeek for running its models across accelerator clusters. DeepSeek announced its public release on October 20, 2026. The initial release supports DeepSeek V4.1-Flash and provides an offline inference engine, rather than a complete, ready-to-deploy online service.^[1][2][16]

DeepSeek describes DeepInfer as the framework behind its own large-scale production services. In a deployment Q&A published with the code, and in a companion technical article on the Chinese platform Zhihu, the company said a single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour at its off-peak ("valley") API pricing. The US$38 figure is revenue per GPU-hour, not a cost figure, and both numbers are company-reported.^[7][24][35] The Zhihu article also published production statistics, including an average input cache hit rate of 96% and an average time to first token of 400 milliseconds.^[35][36]

Release and compatibility

These are the release's documented targets and prerequisites, not a claim that every NVIDIA or Ascend device is supported.^[1][2][3]

Announcement and companion release

DeepSeek's four-post announcement on X called DeepInfer its "in-house inference framework" and listed high-performance kernels, an optimized inference engine, distributed inference, speculative decoding, SWA bounded replay and structured outputs among its contents. The company said inference code for future DeepSeek models and ongoing optimizations would also be published in the repository.^[2][21]

The same thread announced deepseek-recipe, which DeepSeek describes as its production token encoding and decoding component, aligned with the processing logic of the official DeepSeek API. It is a set of Rust libraries with Python bindings. It converts Messages, Chat Completions and Responses API requests into a common conversation format, renders DeepSeek V4 and V4.1 prompts, encodes them into token IDs, and parses model output (reasoning content, tool calls and structured output) back into the matching response format. The caller supplies model inference, tool execution and HTTP transport. deepseek-recipe is released under the MIT License, separately from DeepInfer's Apache 2.0 license.^[7][21][22] Packages of the library predate the announcement: versions 0.1.0 and 0.1.1 were uploaded to PyPI on September 10, 2026. The announcement coincided with version 0.2.0, dated October 20 in the project's changelog, which changed the public API of every crate and of the Python package and added resolution of the model version (V4 or V4.1) from a request's model name. As of October 31, 2026, 0.2.0 was the current release, and the repository's last commit dated from October 19.^[38][39]

According to the deployment Q&A, the encoder records the positions of the special tokens it inserts itself and encodes special-token strings that appear in user input, tool results or tool-call arguments as ordinary text, so that such text cannot inject control tokens. It also encodes the ordinary-text segments between special tokens in parallel, which DeepSeek says reduces tokenization time for very long prompts.^[7] On the output side, the end-of-reasoning marker and the DSML tool-call token are recognized by token ID, so the same text spelled out by the model in ordinary tokens stays literal content. For streaming, the parser buffers incomplete UTF-8 characters, keeps state across the reasoning, body and tool-argument phases, and uses incremental KMP matching so that a delimiter split across chunks is still recognized. Tool names are returned as soon as they are parsed, and tool arguments are streamed as JSON fragments rather than assembled into a complete object first.^[7]

Earlier position on open-sourcing the engine

In April 2025, DeepSeek published a note explaining why it would not then open-source its internal inference engine. It described the engine as based on an early fork of vLLM, heavily customized for DeepSeek models and tightly coupled to internal infrastructure such as cluster management tools, and said that as "a small research team focused on developing better models" it lacked the bandwidth to maintain a large open-source project. It said it would instead contribute standalone components and optimizations to existing open-source projects.^[23]

DeepSeek describes DeepInfer as the in-house inference framework that powers its production services, released as an offline inference engine rather than the complete production stack.^[1][2][24] DeepSeek says its KV cache management and load-balancing code is not yet fully open-sourced because those components are tightly coupled to its internal infrastructure, and points readers to a Zhihu blog post that describes their design.^[7][24]

Relationship to other inference projects

The repository credits vLLM as its main original foundation, with further inspiration from SGLang and NVIDIA TensorRT-LLM. The vLLM project acknowledged the release and said it intended to use ideas from DeepInfer in further V4.1-Flash optimization.^[1][4]

The academic work behind vLLM addresses KV cache fragmentation and sharing through PagedAttention. That work supplies background on the serving problem; its benchmark results are not measurements of DeepInfer.^[5]

Distributed execution

DeepInfer combines attention data parallelism, balanced context parallelism, and expert parallelism for mixture-of-experts computation.^[1][16][20] As in the inference system DeepSeek built for DeepSeek V3 and DeepSeek-R1, the deployment Q&A sets the attention data-parallel degree equal to the expert-parallel degree.^[7] Its kernel dependencies include FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and TileLang.^[17]

Expert-parallel load balancing uses recent token-routing statistics to decide which experts need extra physical replicas and how to place those replicas across devices. The implementation distinguishes the model's logical experts from physical copies, so replicating a busy expert changes its placement rather than the model's logical expert count.^[6]

The deployment documentation describes a production layout with separate vision-encoder, prefill, and decode stages, a form of disaggregated serving. Prefill processes the input prompt, while decode generates subsequent tokens. DeepSeek's described production deployment uses attention data parallelism alongside expert parallelism; it says tensor and context parallelism were not in use there. This production choice is distinct from the public engine's supported configuration options.^[7]

Production deployment described by DeepSeek

The deployment Q&A describes the production system DeepSeek runs around DeepInfer for V4.1-Flash. The figures in this section are DeepSeek's own descriptions as published on October 20, 2026, not independent measurements.^[7]

DeepSeek gives three design goals for the system: support for accelerators from multiple vendors; prefill on ordinary 8-GPU machines and decode on ordinary 64- to 128-GPU supernodes; and retention of the KV cache for more than 72 hours, so that repeated context can skip prefill computation on a cache hit.^[7]

DeepSeek says Wide-EP substantially reduces per-step memory traffic in the MoE expert layers and, with expert-parallel load balancing, raises decode throughput. The Q&A includes an EP8 versus EP64 decode chart produced with the offline benchmark script. It reports a measured MoE load imbalance (slowest rank divided by the mean) of 1.074 for prefill at EP4/DP4, 1.045 at EP2/DP2 and 1.090 for decode.^[7] DeepSeek's Zhihu article gives lower prefill figures, measured on online data, of 1.0618 at EP4/DP4 and 1.0320 at EP2/DP2, and the same 1.090 for decode.^[35]

The expert-parallel load balancer gives each GPU a few redundant expert slots that host replicas of busy experts. For prefill, DeepSeek starts from four redundant experts per GPU, collects expert load over a 100-step window and reassigns experts every 1,000 steps. For decode it starts from one redundant expert per GPU (seven experts per GPU at EP64 after rounding, excluding the shared expert), with a 1,000-step window and reassignment every 10,000 steps. Expert weights are shuffled within the group over NVLink symmetric memory.^[7]

The main KV cache uses FP4 with per-16 E4M3 scaling, which DeepSeek likens to NVFP4 without the global scale. The indexer KV cache and the routed-expert weights use MXFP4 with per-32 UE8M0 scaling. The Q&A also says its "Mega MoE" implementation is significantly faster than unfused MoE within a scale-up domain, regardless of EP size or batch size.^[7]

Keeping the block KV cache on a shared storage cluster, rather than in local host memory, lets any prefill machine hit any cached prefix, so prefill routing does not need prefix affinity. The sliding-window part of the cache stays in host memory on the machine that produced it, and other machines fetch it over a 400G network on a hit. DeepSeek attributes the longer chatbot time-to-live to a heavier long tail of repeated cache hits and reports that a TTL of about three days costs chatbot traffic 0.5% to 1% of KV cache hit rate. Because V4.1-Flash compresses its KV cache heavily, the Q&A says prefill and decode can be split across data centers with about 400 Gbps of interconnect per 8 GPUs; DeepSeek's own inference nodes sit in different but adjacent data centers.^[7]

For the storage tier, the Q&A recommends KV cache servers with 20 SSDs of 30.73 TB each, at least 40 GB/s of read and 5 GB/s of write throughput per server, and enough total capacity to hold 24 hours of data, with the number of servers scaled to the size of the inference deployment. Data on 3FS uses 10+2 erasure coding, so each group tolerates two simultaneous node failures; the Q&A says a read goes to 11 storage nodes of the group at once and any 10 responses are enough to decode it. The kvcache-store service has no expiry logic of its own. Instead, a garbage-collection worker on each storage node evicts the least recently used cache under an LRU policy to keep disk usage below a watermark.^[7]

Cache matching is a metadata-only hash-chain prefix match that must finish before a request is routed, so its latency cannot be hidden. Fetching KV data from 3FS into host shared memory runs asynchronously in a separate process, and swap-in from shared memory to HBM overlaps per-layer host-to-device transfers with forward computation. Because the CUDA copy engine performs poorly on fine-grained KV-block copies, swap-in uses GDR, with an RDMA network card saturating PCIe bandwidth without consuming streaming multiprocessors. Decode does not offload KV cache, since it is not limited by cache capacity, but the KV cache it produces is still written to storage to raise the hit rate.^[7]

The sliding-window (SWA) part of the cache is saved only at 128-token boundaries. Each stored sequence occupies 20 layers x 134 tokens x 528 bytes, or 1,415,040 bytes, in an MXFP8 format, and an in-memory directory partitioned across the entrypoint machines records which machine holds it. Under DeepSeek's hit policy, the block KV cache sets the hit length, capped at L - 134 for an input of L tokens (a 128-token window plus six tokens of speculative backtracking). If no SWA state is available for the final block, the hit length falls back by one 128-token block, which is recomputed.^[7]

The prefill load balancer sorts requests into four context-length tiers (under 16K, 16K-64K, 64K-256K and 256K tokens or more) and uses a fitted cost model to size each prefill chunk within a 32,768-token budget. The decode load balancer spreads requests across all decode GPUs while balancing both request count and total KV cache length. As a heuristic, it sends short requests to GPUs with few running requests but a relatively large total KV cache, and long requests to GPUs with a small total KV cache; the Q&A and the Zhihu article refer readers to the intra-group scheduling algorithm of DualPath, a SIGCOMM 2026 paper on agentic inference over disaggregated KV cache storage, for details.^[7][35] A separate vision-encoder load balancer sends each image to a randomly chosen vision-encoder machine and, on that machine, to the GPU currently processing the fewest image patches. DeepSeek says its load balancing reaches roughly 80% of a perfectly balanced offline benchmark.^[7]

Autoscaling and fault tolerance

Prefill capacity is scaled on the number of non-cached tokens and decode capacity on the number of running requests, with rate limits and upper and lower bounds on each adjustment. A decode engine that receives a shutdown notice enters a one-minute grace period, after which any unfinished requests are forwarded, together with their KV cache, to other decode engines. Deployments keep spare machines in the pool and forbid scale-in while quota utilization is above 0.8.^[7]

DeepSeek says engine restarts take minutes, and that fault tolerance comes from redundant prefill and decode engine capacity plus request-level rerouting rather than from faster restarts. Because all GPUs of a large-EP instance form one communication domain, a single failed GPU makes the whole instance unusable; on restart, the platform sets the EP size to the number of healthy GPUs the instance actually receives.^[7]

DeepSeek's technical article on Zhihu

On October 20, 2026, DeepSeek published a technical article titled "DeepSeek V4.1 Flash 大规模推理部署实践" ("Large-scale inference deployment practice for DeepSeek V4.1 Flash") on its verified organization account on Zhihu, a Chinese question-and-answer and blogging platform. The deployment Q&A links to it as a description of the system's overall architecture and key implementation, including the design of the KV cache management and load-balancing components whose code has not yet been fully open-sourced, and says the Q&A builds on the article and will be updated to answer more detailed questions.^[7][35] Most of the article's text also appears, in English, in the Q&A: the design goals, parallelism choices, load balancers, KV cache storage, autoscaling, DSpark, asynchronous scheduling, structured outputs and deepseek-recipe internals described elsewhere on this page are in both. The account below is based on an archived copy of the article captured on October 21, 2026, and on Chinese press coverage of it.^[35][36][37]

Reported production statistics

The article includes a table of what it calls actual online operating data for the V4.1-Flash service, with a note that the machines keep some spare capacity to absorb traffic fluctuations. These statistics are not in the deployment Q&A.^[35]

The two prefill throughput figures count input tokens with and without the tokens served from the KV cache. A note under the table says the prefill-to-decode ratio in the actual deployment is close to 1:1 and repeats the average revenue of US$38 per GPU-hour at valley pricing. The table does not give a measurement period.^[35] Xinzhiyuan, noting both the Zhihu article and the Q&A, reported the 77K-token average input, the 400 ms average first-token latency, the 230 tokens-per-second output speed and the 96% cache hit rate as DeepSeek's officially published online statistics.^[36] They are company-reported figures for DeepSeek's own traffic.

Storage details not in the Q&A

The article adds performance figures for the 3FS storage tier. It says the cluster is optimized for random reads and writes at the granularity of one KV cache block, about 112 KiB, and reaches a peak aggregate read bandwidth of 10 TiB/s and a peak aggregate write bandwidth of about 850 GiB/s.^[35][37] Block KV cache is managed in 128-token pages and, following the model's architecture, holds the main KV and indexer keys of four layers.^[35]

For the sliding-window cache, which stays in the host memory of the machine that produced it, the article reports from online data a time-to-live of about 180 seconds and a hit rate above 98%. DeepSeek says SWA state is reused only within an active session, over minutes, so a small pool with fast turnover covers the sessions active at any time; misses caused by eviction are recomputed with encoder SWA bounded replay.^[35][37]

The article calls this approach, keeping SWA state out of persistent storage and recomputing it on a miss, the main change in KV cache management from V4 to V4.1. DeepSeek's inference system for DeepSeek V4 wrote SWA state to persistent storage at two positions and, on a miss, ran a full forward pass over a number of tokens equal to the layer count times the window, restoring the window state exactly. According to the article, that path needed enough SSD capacity to push the miss rate very low, let SWA state take up nearly half of the persistent KV cache capacity, and did not return a matching hit benefit, because a block KV prefix can be hit repeatedly over several days while SWA state is reused only within an active session, over minutes. The article adds that the "Zero SWA Caching" approach proposed in the V4 technical report, which persists no SWA state and relies only on exact recomputation, was not adopted in the V4.1 inference system because its cost in production was too high.^[35]

Decoding and scheduling

DSpark verification

DeepInfer supports speculative decoding through DSpark. Draft tokens carry confidence estimates, and the scheduler uses their expected acceptance probabilities to allocate verification work. Variable-length verification avoids checking the same fixed number of draft tokens for every request. DeepSeek's model report describes DSpark as a semi-autoregressive drafting architecture with confidence-scheduled verification.^[8] DeepSeek's Zhihu article explains the drafting change relative to DFlash: DSpark adds to the drafter's base logits a low-rank logit bias that depends only on the previous token ID, a lightweight sequential correction intended to restore local conditioning between adjacent tokens and raise draft accuracy. The article refers readers to the DSpark paper, posted to arXiv in July 2026, for the method.^[35][40]

The public configuration enables variable-length verification with DEEPINFER_ENABLE_MTP_VARLEN=1. DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length, documented as five by default.^[18]

In production, the draft model proposes five tokens at a time. DeepSeek reports that variable-length verification cut the average number of draft tokens verified per step by about 30% (from 5.00 to between 3.2 and 4.4) while leaving the average accepted length nearly unchanged (3.29 versus 3.20). From EP64/DP64 benchmark data, it estimates a 4% to 7% throughput gain at its production operating point of about 60 concurrent sequences per rank.^[7] The Zhihu article gives a single figure for the online deployment: compared with always verifying five tokens, variable-length verification raised throughput by 6%.^[35][37]

These are DeepSeek's estimated per-rank throughput figures from its own benchmark, not end-to-end service measurements.^[7]

Asynchronous scheduling

The scheduler overlaps CPU scheduling for a subsequent batch with accelerator execution of the current batch. This reduces the idle interval otherwise occupied by CPU scheduling and post-processing. DeepSeek describes the design as following existing SGLang and NanoFlow implementations, rather than presenting asynchronous overlap as an entirely new serving technique.^[7]

Asynchronous scheduling interacts with DSpark: when the next step is scheduled, the confidence values from the current step have not yet returned. The CPU therefore fixes only the total verification budget, and a TileLang kernel on the GPU computes each request's draft length.^[7]

Sliding-window bounded replay

SWA bounded replay reconstructs sliding-window attention state from a short suffix instead of replaying a much longer history. The model report describes this reconstruction as approximate. It should not be confused with exact reconstruction of every attention state.^[8]

According to the deployment Q&A, replay recomputes only the most recent 128 tokens and limits each query's attention to keys inside the replayed segment, which makes the recomputation cost constant regardless of context length. For the decoder layers, exact reconstruction would require running the last 2,560 tokens through 20 decoder layers; DeepSeek says recomputing only the last 128 prompt tokens instead nearly halves total prefill computation. The README's default V4.1 settings are DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140; the retained suffix must cover the SWA window plus the maximum backtracking length.^[1][7]

The repository contains separate mask and compact implementations and verification utilities that compare their metadata and model outputs. That equivalence check compares two implementations of the truncation operation; it does not establish equivalence between bounded replay and unrestricted full-history computation.^[9]

Sampling and structured outputs

The SamplingParams interface includes temperature, top-p, top-k, maximum output tokens, per-sequence random seeds, and structured-output arguments. A missing seq_seed produces a randomly selected 64-bit seed. Supplying the same fixed seed is therefore a separate step from configuring batch-invariant computation.^[10]

The repository also includes a regression test that compares a prompt's batched and single-request outputs with batch invariance enabled. The test compares token IDs and probability information; its presence is a test specification, not an independent benchmark of all hardware configurations.^[1][11]

Structured output compilation uses XGrammar. The compiler dispatches JSON schemas, regular expressions, and EBNF grammars to the corresponding compilation routines; JSON-schema compilation uses strict mode. Compiled constraints then restrict the tokens available during generation. This controls output format, not the factual accuracy of the generated content.^[12][7] In DeepSeek's production system, compilation runs asynchronously on a dedicated structured-output machine, and each request retrieves its compiled grammar automaton before it enters the engine.^[7]

Benchmarks and interpretation

The repository publishes separate throughput-latency plots for prefill and decode. DeepSeek reports measuring the prefill plot on four SM100 GPUs and the decode plot on 64 SM100 GPUs, each inside one scale-up domain.^[1]

The chart captions give the configurations. The prefill sweep uses EP4/DP4 with "384+32 experts", a 131,072-token context and 140-token decoder bounded replay. Its non-cached input throughput rises from about 60,000 tokens per second per GPU at a batch of 4,096 tokens per DP rank (roughly 70 ms per prefill step) to just under 90,000 at 65,536 tokens (roughly 730 ms per step). The decode sweep uses EP64/DP64 with "384+64 experts", a 131,072-token context, a fixed three-token draft (nextn=3) and an assumed per-token acceptance rate of 75%, which the caption converts to τ = 3.25. Its output throughput rises from about 2,600 tokens per second per GPU at 8 requests per DP rank, at about 325 tokens per second per user, to about 26,000 at 256 requests, at about 100 tokens per second per user. These values are read from the published charts and are approximate.^[1] The Zhihu article reproduces the same decode chart and a slightly different prefill chart, which levels off at about 87,000 tokens per second per GPU.^[35]

The benchmark driver measures steady-state intervals after warmup. Its decode throughput uses full-acceptance accounting: the token count includes all configured draft tokens whether or not a workload would actually accept them. The plotting utility applies an assumed draft-acceptance rate to fixed-length decode sweeps, adjusting both per-user speed and throughput; it rejects this calculation for variable-length verification runs. Consequently, a plotted number must be read together with its acceptance assumption, context length, concurrency, draft length, and device configuration.^[13][14]

The examples also warn that profiler timing can distort performance and should not be used to derive steps per second. The sweep scripts disable profiling for measurement points. These are developer-provided engine measurements, not end-to-end measurements of an independently operated online service.^[13]

Model-quality checks

The evaluation suite separates generation from scoring and supplies reference Pass@1 values for checking a V4.1 deployment.^[15]

For AIME and GPQA, Pass@1 is mean single-sample accuracy across the repeated samples, not the proportion of questions solved by any one of those attempts. HLE uses one sample per task. The documentation calls these reference figures indicative sanity checks, with variation expected between runs.^[15]

Scoring defaults to an LLM judge. Rule-based scoring is available for AIME and GPQA, but not HLE. A run's Pass@1 is considered valid only when its coverage_complete field is true. These checks measure the configured model's answers; they do not measure serving throughput.^[15]

Running the public release

The official build selects its backend with DEEPINFER_ACCELERATOR, then builds dependencies and the extension from source. For NVIDIA, the build and offline example run as follows:^[19][20]

export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py

The examples use deepseek-ai/DeepSeek-V4.1-Flash-1017 as their default checkpoint. A model identifier triggers a download through the Hugging Face cache; an explicit local model directory can be used instead. DEEPINFER_HF_MODEL_DIR changes the default model location without overriding an explicitly supplied --model_name.^[16]

The examples provide expert-parallel, data-parallel, and context-parallel configurations, plus text and image regression inputs. Their offline tests include perplexity baselines and optional checks for regular-expression, JSON, and EBNF output constraints.^[20]

Public release boundaries

The public engine does not by itself reproduce DeepSeek's complete online service. The README lists a production KV cache system, vision-encoder/prefill/decode disaggregation, and load balancing across prefill and decode data-parallel replicas as additional requirements not included in the release. DeepSeek's deployment Q&A explains that cache management and request load balancing depend on internal infrastructure.^[1][7]

This exclusion concerns the surrounding production system. It should not be read as saying the engine performs inference without attention-state caching or lacks the expert-placement balancing code included in the repository.^[6][7]

DeepSeek says its internal versions of kernel libraries such as DeepGEMM and FlashMLA are broadly the same as the public ones, with some differences in the indexer and "mega" kernel series. The public FlashMLA lacks one hand-tuned SASS optimization, which DeepSeek says makes its decoding kernel 5% to 10% slower, and end-to-end performance differs by at most 5%. The internal draft model is somewhat newer, with what DeepSeek calls a small performance difference.^[7]

Reception and economics analysis

Press coverage

Technology outlets including SiliconANGLE and TechNode covered the release. SiliconANGLE reported the 155-trillion-token and US$38-per-GPU-hour figures from the Q&A, noted that the two run-to-run determinism features work only on NVIDIA hardware and that all published performance benchmarks are DeepSeek's own NVIDIA results, and observed that Hopper and older NVIDIA architectures are not on the supported-hardware list.^[24] TechNode reported that the framework supports NVIDIA GPUs and Huawei Ascend processors and that DeepSeek plans to keep releasing inference code for future models through the project.^[25]

The South China Morning Post reported that DeepSeek gave the revenue figure in a technical article on Zhihu and that the company did not disclose its operating costs or profit margins. It linked the release to posts by DeepSeek senior researcher Chen Deli, who called the contest between open- and closed-source AI "a decisive battle between two futures, two destinies for humanity", and by Cui Tianyi, who leads DeepSeek's Harness team and said the company hoped more businesses and organizations would "deploy DeepSeek models and profit from doing so".^[27] Quartz and The Next Web, citing Bloomberg, reported that no outside party had yet benchmarked DeepInfer's efficiency gains, and that companies hosting DeepSeek's models could now apply the same serving methods themselves.^[28][29]

The Next Web reported that the repository had passed 8,000 GitHub stars by the afternoon of release day.^[28] As of October 31, 2026, it had about 24,500 stars and 2,950 forks, no tagged releases, and five commits on its main branch; the most recent, on October 25, added strict validation of boolean environment variables.^[26]

Revenue figure and margin estimates

The US$38 figure is revenue per GPU-hour at "valley" (off-peak) API pricing. DeepSeek's off-peak rates are half of its peak rates; peak hours are 01:00-04:00 and 06:00-10:00 UTC, Monday through Friday, excluding Chinese public holidays. As of October 30, 2026, off-peak prices for V4.1-Flash (deepseek-flash) were US$0.003 per million cache-hit input tokens, US$0.15 per million cache-miss input tokens and US$0.60 per million output tokens.^[34] Neither the deployment Q&A nor the Zhihu article states how many GPUs the cluster contains or names the accelerator model it uses, and DeepSeek did not publish its costs. A note under the article's statistics table says that the deployed prefill-to-decode ratio is close to 1:1 and that the US$38 average is calculated at valley prices without regard to peaks and troughs.^[7][27][35]

Several commentators converted the revenue figure into margin estimates. These are third-party calculations built on assumed GPU rental prices, not figures disclosed by DeepSeek, and they leave out costs other than the GPUs themselves.

Synced (Jiqizhixin), in an article republished by 36Kr, repeated the 82% calculation and noted that it subtracts only the GPU rental. Synced argued that a large gap between DeepSeek's API prices and its costs would remain even after other service expenses, while reporting that Hacker News commenters disagreed: some derived higher margins from lower GPU costs, and others said the full lifecycle investment in the model had to be counted.^[30] The same article contrasted DeepSeek's three- to seven-day KV cache retention with Anthropic, OpenAI and Google, whose prompt caching retention it put mostly between five minutes and 24 hours.^[30] Physicist Steve Hsu, quoting Jin's post, attributed the efficiency behind such margins to architectural innovation rather than distillation.^[33]

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10DeepSeek-AI. DeepInfer: official repository and README. Release documentation reviewed October 20, 2026.
2. ^1 ^2 ^3 ^4DeepSeek. Public DeepInfer announcement. October 20, 2026.
3. ^DeepSeek-AI. DeepInfer Apache License 2.0.
4. ^vLLM. Acknowledgment of DeepInfer's release and vLLM foundation. October 20, 2026.
5. ^Kwon, Woosuk, et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. 2023.
6. ^1 ^2DeepSeek-AI. DeepInfer expert-parallel load-balancing policy.
7. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32DeepSeek-AI. DeepSeek V4.1 Flash Inference Deployment Q&A. Reviewed October 20, 2026.
8. ^1 ^2DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. September 17, 2026. See sections 2.4.3 and 3.2.2.
9. ^DeepSeek-AI. YoCo mask/compact equivalence verification.
10. ^DeepSeek-AI. DeepInfer sampling parameters.
11. ^DeepSeek-AI. Batch-invariant regression test.
12. ^DeepSeek-AI. DeepInfer structured-output compiler.
13. ^1 ^2DeepSeek-AI. Offline inference and benchmark examples.
14. ^DeepSeek-AI. Throughput-latency plotting implementation.
15. ^1 ^2 ^3DeepSeek-AI. Benchmark evaluation methodology and reference results.
16. ^1 ^2 ^3DeepSeek-AI. Offline model and launch configuration and example usage.
17. ^1 ^2DeepSeek-AI. Vendored kernel libraries and Python kernel dependencies.
18. ^DeepSeek-AI. Environment-variable definitions.
19. ^DeepSeek-AI. Backend selection and build entry point.
20. ^1 ^2 ^3DeepSeek-AI. Offline inference and regression example.
21. ^1 ^2DeepSeek. Announcement thread, posts 2/4, 3/4 and 4/4. October 20, 2026.
22. ^1 ^2DeepSeek-AI. deepseek-recipe: official repository and README. Accessed October 28, 2026.
23. ^DeepSeek-AI. The Path to Open-Sourcing the DeepSeek Inference Engine. April 14, 2025.
24. ^1 ^2 ^3 ^4Riley, Duncan. DeepSeek open-sources DeepInfer, the inference framework behind its production services. SiliconANGLE, October 20, 2026.
25. ^TechNode. DeepSeek open-sources inference framework with support for Nvidia and Huawei chips. October 21, 2026.
26. ^GitHub. Repository metadata for deepseek-ai/DeepInfer and commit history. Accessed October 31, 2026.
27. ^1 ^2Lee, Chong Ming. DeepSeek seeks 'total victory' over closed-source AI with new open-source tool. South China Morning Post, October 21, 2026.
28. ^1 ^2Constantin, Ana Maria. DeepSeek open-sources DeepInfer, the engine that runs its AI models. The Next Web, October 20, 2026.
29. ^Tolomia, Cris. DeepSeek is open-sourcing the software it uses to run its AI models at scale. Quartz, October 20, 2026.
30. ^1 ^2Synced (Jiqizhixin). DeepSeek开源推理框架，有人算出AI的毛利：82% (DeepSeek open-sources its inference framework; someone calculated AI's gross margin: 82%). Republished by 36Kr, October 22, 2026.
31. ^Jin, Yuchen. Post on DeepInfer's revenue figure. X, October 20, 2026.
32. ^swyx. Post on DeepSeek's Zhihu deployment article. X, October 20, 2026.
33. ^Hsu, Steve. Post on DeepSeek inference efficiency. X, October 22, 2026.
34. ^DeepSeek. Models & Pricing. DeepSeek API Docs. Accessed October 30, 2026.
35. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16DeepSeek. DeepSeek V4.1 Flash 大规模推理部署实践 (Large-scale inference deployment practice for DeepSeek V4.1 Flash). Zhihu, October 20, 2026. Read via an archive.today copy captured October 21, 2026, including its statistics table and benchmark charts.
36. ^1 ^2 ^3Xinzhiyuan. 突发，DeepSeek重磅开源核心推理引擎！ (Breaking: DeepSeek open-sources its core inference engine). Republished by BAAI Hub, October 21, 2026.
37. ^1 ^2 ^3 ^4Bu, Shuqing. DeepSeek全面开源推理框架：兼容英伟达、华为昇腾芯片，单集群日吞吐155万亿Token，每卡每小时收入38美元 (DeepSeek fully open-sources its inference framework). Wallstreetcn, republished by 36Kr, October 20, 2026.
38. ^DeepSeek-AI. deepseek-recipe changelog and commit history. Accessed October 31, 2026.
39. ^Python Package Index. deepseek-recipe release history. Accessed October 31, 2026.
40. ^Cheng, Xin, et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv, July 6, 2026.