DeepSeek Sparse Attention (DSA)

DeepSeek Sparse Attention (DSA) is a trainable, fine-grained sparse attention mechanism introduced by the Chinese AI company DeepSeek in its experimental model DeepSeek-V3.2-Exp, released on September 29, 2025. DSA pairs a lightweight scoring module that DeepSeek calls the "lightning indexer" with a fine-grained token-selection step, so that each query attends to only a small, fixed number of the most relevant earlier tokens (2,048 by default) instead of to the entire preceding sequence. This cuts the dominant cost of the core attention computation from quadratic, O(L^2), to roughly O(L*k) in the sequence length L, where k is the number of selected tokens and far smaller than L in long contexts. DeepSeek reported that the technique preserves output quality, closely matching the benchmark scores of its predecessor DeepSeek-V3.1-Terminus, while reducing long-context training and inference cost enough that the company lowered its published API prices by more than half (50 percent or more) on the same day. ^[1][2][5]

DeepSeek summarized the mechanism in its release announcement: DSA "achieves fine-grained sparse attention with minimal impact on output quality" while "boosting long-context performance & reducing compute cost." ^[1]

What is DeepSeek Sparse Attention?

DSA is DeepSeek's production response to the long-context attention bottleneck, and it is the engineering successor to the company's earlier research method Native Sparse Attention (NSA). Where NSA was validated on a model trained from scratch, DSA was added to an already-trained flagship through continued training, which let DeepSeek measure the cost of sparsity against a near-identical dense baseline. The company described DeepSeek-V3.2-Exp as an experimental, intermediate step toward a next-generation architecture rather than a finished product, and it kept the model's training configuration deliberately aligned with V3.1-Terminus so that any change in benchmark numbers could be attributed to the attention mechanism itself. ^[1][2]

The model that carries DSA is built on the same mixture-of-experts (MoE) and multi-head latent attention (MLA) foundation as DeepSeek-V3, a 671-billion-parameter MoE with roughly 37 billion parameters active per token. DSA is layered on top of MLA rather than replacing it: the indexer scores tokens with small projections of each layer's input hidden states, and what it selects are MLA's compressed latent key-value entries, which are then read through the ordinary MLA attention path. ^[2][3][12]

Why is sparse attention needed? Background on DeepSeek-V3.2-Exp and MLA

In a standard transformer, every query token attends to every preceding key token, so producing the output for a sequence of length L costs on the order of L^2 query-key interactions. For the long context windows that modern large language models target, this quadratic term dominates both the compute of training and the memory traffic of decoding, where each newly generated token must read the entire stored key-value (KV) cache. ^[3]

DeepSeek had already attacked the memory side of this problem with MLA, which compresses the per-layer KV cache into a small shared low-rank latent so that far less data is stored and moved during decoding. MLA reduces the size of what attention reads, but it does not by itself reduce the number of tokens each query must consider, so the attention computation is still effectively quadratic in context length. DSA closes that gap by making the set of attended tokens sparse. Because it operates on MLA's latent representations and selects whole latent KV entries that are shared across all query heads, the two mechanisms compose cleanly: MLA shrinks each entry, and DSA reduces how many entries are touched. ^[2][3]

DeepSeek-V3.2-Exp itself is a continued-training derivative of DeepSeek-V3.1-Terminus. DeepSeek froze the training recipe of the earlier model and changed essentially one thing, the attention mechanism, in order to isolate the effect of DSA. ^[1][2]

How does DeepSeek Sparse Attention work?

DSA has two components: a lightning indexer that produces a cheap relevance score for every preceding token, and a fine-grained selector that keeps only the highest-scoring tokens for full attention. ^[1][3]

What is the lightning indexer?

For a query at position t, the indexer assigns each earlier position s an index score by combining a small number of lightweight per-head dot products. In the notation of DeepSeek's technical report the score is

I(t,s) = sum over j of w(t,j) * ReLU( q(t,j) . k(s) ),

where q(t,j) is the indexer query for head j, k(s) is the indexer key at position s, w(t,j) is a learned per-head weight, and the sum runs over the indexer's heads. Two design choices make this scan affordable. The indexer uses far fewer heads than the main attention (64 indexer heads of dimension 128 against the model's 128 attention heads, per the released configuration) and can be implemented in FP8, and it uses a ReLU nonlinearity instead of a softmax, which DeepSeek says it chose "for throughput consideration". In the technical report, q(t,j) and w(t,j) are derived from the query token's hidden state h(t) and k(s) from the preceding token's hidden state h(s), so the scoring runs on small projections rather than on the full attention heads. The indexer's scoring is itself formally quadratic in L, but because each per-token operation is so light, its wall-clock and floating-point cost are minor relative to dense attention, and the expensive part of the layer is what follows. ^{[3][7][10][12][13]}

How does fine-grained token selection work?

Given the index scores, the selector keeps only the top-k positions for each query, with k set to 2,048 by default, and builds a sparse attention mask that hides every other token. The main MLA attention is then computed over just those k selected entries. Selection is fine-grained, meaning it operates at the level of individual tokens rather than over fixed contiguous blocks. This is the central technical distinction between DSA and the block-based schemes that preceded it, and the README of DeepSeek's release repository says that DSA "achieves fine-grained sparse attention for the first time". Because MLA in its decoding form behaves like multi-query attention, with one latent KV entry shared across all query heads, the same 2,048 selected entries serve every head of a token, so the selection is computed once per query rather than once per head. With k held constant, the core attention cost grows only linearly with context length, giving the O(L*k) behavior that motivates the design. ^{[2][3][7][10]}

How is DSA trained into the model?

DSA was instilled through continued training of DeepSeek-V3.1-Terminus in two phases. In a dense warm-up stage of 2.1 billion tokens (1,000 steps of 16 sequences of 128K tokens, at a learning rate of 1e-3), the main model is frozen and only the lightning indexer is trained. The indexer learns to imitate the dense model's own attention by minimizing a Kullback-Leibler divergence between its predicted token importances and the attention weights the dense model actually produces, so the indexer learns to point at the tokens the model would have attended to anyway. In the second, sparse stage of 943.7 billion tokens (15,000 steps of 480 sequences of 128K tokens, at a learning rate of 7.3e-6), the top-k selector is switched on with 2,048 tokens selected per query and the whole model, indexer included, is trained with sparse attention active so that the network adapts to operating under sparsity; the indexer input is detached from the computational graph, so the indexer still learns only from its KL loss while the main model learns only from the language-modeling loss. DeepSeek then kept the same post-training pipeline it used for V3.1-Terminus: specialist models for mathematics, competitive programming, general logical reasoning, agentic coding and agentic search (plus writing and general question answering) were trained with large-scale RL and distilled into the final checkpoint, followed by a single mixed reinforcement-learning stage using Group Relative Policy Optimization (GRPO) that merges reasoning, agent and human-alignment training rather than running them as separate stages. ^[7][12]

How much more efficient is DSA, and does it hurt quality?

The headline efficiency claim is the reduction of the core attention term from O(L^2) to O(L*k). Because the indexer is engineered to be far cheaper per token than the main attention, the overall long-context cost is dominated by the sparse attention over k tokens rather than by the full sequence. DeepSeek packaged high-performance kernels for the mechanism, releasing sparse-attention kernels in FlashMLA, indexer logit kernels in DeepGEMM, and additional kernels in TileLang. ^[2][10]

On quality, DeepSeek aligned the configurations of V3.2-Exp and V3.1-Terminus and published a side-by-side comparison showing near parity across a broad benchmark suite. The results are mixed in the small, with V3.2-Exp slightly ahead on several reasoning and agentic-search tasks and slightly behind on a few knowledge and coding tasks. For three of the drops (GPQA-Diamond, Humanity's Last Exam and HMMT 2025) the technical report's explanation is that V3.2-Exp generates fewer reasoning tokens, and it states that the gap closes when intermediate checkpoints producing a comparable number of tokens are used; the report offers no explanation for the Aider-Polyglot drop, which at 1.6 points is larger than GPQA-Diamond's. ^{[1][2][10][12]}

The efficiency gain showed up most directly in price. On release day DeepSeek cut its API prices by more than 50 percent: the cache-miss input price fell to 0.28 US dollars per million tokens (from 0.56), the cache-hit input price to 0.028 dollars per million, and the output price to 0.42 dollars per million. The saving grows with context length, since the cost reduction comes from replacing a quadratic term with a near-linear one. An analysis summarized by DeepLearning.AI estimated that long-context inference became roughly 6 to 7 times cheaper than on V3.1-Terminus, citing illustrative per-million-token costs of about 0.10 dollars versus 0.60 dollars at 32,000 tokens of input and about 0.30 dollars versus 2.30 dollars at 128,000 tokens. ^[4][5][6]

DeepSeek released the model weights on Hugging Face under the MIT License, and the open release made DSA immediately reproducible. Serving frameworks including vLLM and SGLang added support on the day of release, and on November 17, 2025 DeepSeek noted a correction to a Rotary Position Embedding discrepancy in the indexer module of its earlier inference code. ^[2][6]

How does DSA differ from Native Sparse Attention and other sparse attention?

DSA is the direct descendant of NSA, the trainable sparse-attention method DeepSeek published in February 2025 and which shared a Best Paper Award at ACL 2025. NSA runs three parallel branches over the history, a coarse compression branch, a fine-grained selection branch, and a local sliding window, and blends them with a learned gate, all built around hardware-aligned block selection. DSA keeps NSA's goals of trainable, fine-grained sparsity and hardware-aware kernels but simplifies the design dramatically: it drops the multi-branch gate in favor of a single lightning indexer plus a top-k selector, it selects individual tokens rather than contiguous blocks, and it is bolted onto MLA through continued training rather than being baked in from scratch. ^[2][3][8]

DSA also belongs to the same wave as Mixture of Block Attention (MoBA), released within days of NSA in February 2025 by Moonshot AI and deployed in its Kimi assistant. MoBA carries the mixture-of-experts routing idea into attention, partitioning the context into blocks and routing each query to its top-k blocks by a dot product against each block's mean-pooled key. NSA, MoBA, and DSA together mark a shift away from inference-only sparsification, in which a dense model is pruned after the fact, and toward sparsity that is a native, trained-in property of the model. DSA's distinguishing feature within that group is its token-level granularity: rather than choosing blocks, its indexer scores and its selector keeps individual key-value entries. ^[3][8][9]

Related and derivative work

DSA's lightning indexer has become a reference point for a second wave of work that treats sparse selection, rather than the attention computation itself, as the cost to attack. Because the indexer still scans every preceding token for every query at every layer, its O(L^2) term can come to dominate per-layer attention cost as contexts lengthen, and several 2026 papers target that step directly. ^[14][15][16]

SparDA's authors position their Forecast projection explicitly between DSA and InfiniGen, the 2024 OSDI system that prefetches CPU-resident KV entries before each layer using the raw hidden state as a cross-layer proxy: SparDA "learns a cross-layer Forecast projection" that they describe as "bridging DSA's trained decoupled indexer with InfiniGen's lookahead", and in their related-work summary, "SparDA extends DSA's lightning indexer to block-sparse attention, operating one layer ahead and pruning to one Forecast head per GQA group." The Forecast is trained the way DSA's indexer is warmed up, by KL divergence against the original selector's attention distribution restricted to the top-k set, though the paper skips DSA's second, full-model sparse training stage because its base models are already sparse-pretrained. The same paper reports that DSA has been adopted by GLM-5 as well as DeepSeek-V3.2, that DeepSeek-V4's Compressed Sparse Attention path is built on top of DSA, and lists extending its lookahead design to token-level DSA as future work. ^[14]

A separate line of work, NOSA from THUNLP and OpenBMB (October 2025, revised January 2026), classifies DSA as an element-wise (token-level) selection scheme and argues that block-wise selection is the better fit for KV cache offloading, because gathering individual tokens from host memory makes poor use of PCIe bandwidth; NOSA therefore constrains how many blocks change between decoding steps rather than refining the indexer. ^[17] Together these papers move the target from the attention computation itself, which sparse attention has already reduced from O(L^2) to O(L*k), toward the two costs the SparDA paper says still grow with context length after that reduction: selection overhead and KV cache capacity. ^[14]

Why does DeepSeek Sparse Attention matter?

DSA is notable less as a new idea than as a demonstration that fine-grained, learned sparse attention can be deployed at frontier scale with, on DeepSeek's own evaluations, no substantial loss of quality and with an immediate, large reduction in serving cost. By aligning V3.2-Exp's recipe with V3.1-Terminus and open-sourcing the weights, the kernels, and the benchmark comparison, DeepSeek gave the community a clean controlled study of sparse attention on a 671-billion-parameter model, something earlier sparse-attention papers could only approximate at smaller scale. The mechanism was subsequently carried forward into DeepSeek's broader V3.2 line, with a later technical report, "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models," presenting the architecture beyond its initial experimental framing. For long-context applications, where the quadratic cost of attention had been the binding constraint, DSA reframed the problem as one of cheaply selecting which few thousand tokens actually matter for each query. ^[1][2][11]

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7DeepSeek-AI. "Introducing DeepSeek-V3.2-Exp." DeepSeek API Docs, September 29, 2025. api-docs.deepseek.com/...news250929
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11DeepSeek-AI. "DeepSeek-V3.2-Exp" (model release, technical report, and kernels). GitHub, September 29, 2025. github.com/...DeepSeek-V3.2-Exp
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Raschka, Sebastian. "A Technical Tour of the DeepSeek Models from V3 to V3.2." Ahead of AI, 2025. magazine.sebastianraschka.com/...technical-deepseek
4. ^DeepLearning.AI. "DeepSeek-V3.2-Exp Streamlines Processing Using a 'Lightning Indexer,' Boosting Efficiency." The Batch, October 2025. deeplearning.ai/...ing-indexer-boosting-efficiency
5. ^1 ^2TechCrunch. "DeepSeek releases 'sparse attention' model that cuts API costs in half." September 29, 2025. techcrunch.com/...odel-that-cuts-api-costs-in-half
6. ^1 ^2VentureBeat. "DeepSeek's new V3.2-Exp model cuts API pricing in half to less than 3 cents per 1M input tokens." September 29, 2025. venturebeat.com/...ng-in-half-to-less-than-3-cents
7. ^1 ^2 ^3Golge, Eren. "Model check: DeepSeek-V3.2-Exp, Fine-Grained Sparse Attention for Efficient Long-Context LLMs." Substack, 2025. erogol.substack.com/...check-deepseek-v32-exp-fine
8. ^1 ^2Yuan, Jingyang, et al. "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention." arXiv:2502.11089, February 16, 2025. arxiv.org/...2502.11089
9. ^Lu, Enzhe, et al. "MoBA: Mixture of Block Attention for Long-Context LLMs." arXiv:2502.13189, February 18, 2025. arxiv.org/...2502.13189
10. ^1 ^2 ^3 ^4MarkTechPost. "DeepSeek V3.2-Exp Cuts Long-Context Costs with DeepSeek Sparse Attention (DSA) While Maintaining Benchmark Parity." September 30, 2025. marktechpost.com/...e-maintaining-benchmark-parity
11. ^DeepSeek-AI. "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models." arXiv:2512.02556, December 2025. arxiv.org/...2512.02556
12. ^1 ^2 ^3 ^4DeepSeek-AI. "DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention" (technical report PDF in the release repository). GitHub, September 29, 2025. github.com/...DeepSeek_V3_2.pdf
13. ^DeepSeek-AI. "deepseek-ai/DeepSeek-V3.2-Exp" model repository, config.json (index_n_heads, index_head_dim, index_topk, num_attention_heads). Hugging Face. huggingface.co/...config.json
14. ^1 ^2 ^3 ^4Fu, Yaosheng, Guangxuan Xiao, Xin Dong, Song Han, and Oreste Villa. "SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference." arXiv:2606.04511, June 3, 2026. arxiv.org/...2606.04511
15. ^1 ^2Bai, Yushi, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, and Juanzi Li. "IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse." arXiv:2603.12201, March 12, 2026. arxiv.org/...2603.12201
16. ^1 ^2Xu, Yufei, et al. "HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention." arXiv:2603.28458, March 30, 2026 (v4, September 10, 2026; COLM 2026). arxiv.org/...2603.28458
17. ^Huang, Yuxiang, Pengjie Wang, Jicheng Han, Weilin Zhao, Zhou Su, Ao Sun, Hongya Lyu, Hengyu Zhao, Yudong Wang, Chaojun Xiao, Xu Han, and Zhiyuan Liu. "NOSA: Native and Offloadable Sparse Attention." arXiv:2510.13602, October 15, 2025 (v2, January 29, 2026). arxiv.org/...2510.13602