Native Sparse Attention (NSA)
Native Sparse Attention (NSA) is a hardware-aligned, natively trainable sparse attention mechanism introduced in February 2025 by researchers at DeepSeek, Peking University and the University of Washington. It is designed to make long-context language model training and inference cheaper while keeping the accuracy of dense (full) attention. Instead of letting every query attend to every earlier token, NSA runs three parallel branches over the key-value sequence: a coarse-grained compression branch, a fine-grained block-selection branch, and a local sliding window, blended by a learned gate. In kernel benchmarks on an 8-GPU NVIDIA A100 system at a 64k-token (65,536-token) context, the authors report up to 9.0 times faster forward propagation and 6.0 times faster backward propagation than a Triton implementation of FlashAttention-2, and an expected decoding speedup of up to 11.6 times, estimated from the reduction in key-value cache reads. A 27-billion-parameter model pretrained with NSA matched or exceeded an identically trained full-attention model on the authors' general, long-context and reasoning evaluations. The method is described in the paper "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention" by Jingyang Yuan and 14 co-authors, which was one of four Best Paper award winners at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025) in Vienna. [1][2][3][7]
| At a glance | |
|---|---|
| Paper | "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention" [1] |
| Authors | Jingyang Yuan (first author, internship at DeepSeek-AI), Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Liang Wenfeng, Wangding Zeng [1] |
| Affiliations | DeepSeek-AI; Peking University (School of Computer Science, PKU-Anker LLM Lab); University of Washington [1] |
| Preprint | arXiv:2502.11089, v1 February 16, 2025; v2 February 27, 2025 [1] |
| Published version | ACL 2025 (Volume 1: Long Papers), pages 23078-23097, DOI 10.18653/v1/2025.acl-long.1126 [2] |
| Award | ACL 2025 Best Paper (one of four) [7] |
| Branches | compression (block 32, stride 16), selection (block 64, top 16 blocks), sliding window (512 tokens) [1] |
| Test model | 27B-total, 3B-active GQA + MoE transformer, 30 layers [1] |
| Kernels | Triton; compression and window branches use FlashAttention-2-compatible kernels, selection uses a custom group-centric kernel [1] |
What is Native Sparse Attention?
The core problem NSA targets is that the self-attention operation in a transformer has computational cost that grows quadratically, O(L^2), with sequence length L. For modern context windows of tens or hundreds of thousands of tokens, attention comes to dominate both the compute of the forward and backward passes during training and the memory traffic of the key-value (KV) cache during decoding. Sparse attention, which computes only a subset of the query-key interactions, is the obvious remedy, but two practical obstacles had limited earlier methods. First, many sparse schemes are applied only at inference time on top of a model that was pretrained with full attention, so the model never learns to operate with the sparsity and a train-inference mismatch degrades quality. Second, theoretical reductions in floating point operations frequently fail to produce real wall-clock speedups because the sparse access pattern is irregular and does not map well to GPU memory hierarchies or to optimized kernels such as FlashAttention. [1]
NSA's contribution is to address both obstacles at once through two stated design principles: it is hardware-aligned, meaning its sparse pattern is organized into contiguous blocks and its kernels are written to keep the GPU's arithmetic units busy and to cooperate with grouped-query attention; and it is natively trainable, meaning the sparsity is present from the start of pretraining rather than bolted on afterward. The paper's abstract states that NSA "employs a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection to preserve both global context awareness and local precision." [1] The authors report that an NSA model pretrained end-to-end matches or exceeds an equivalent full-attention model on general, long-context, and reasoning benchmarks, while delivering large speedups on 64k-token sequences. [1]
Why is long-context attention so expensive?
In a standard attention layer, every query token attends to every preceding key token, so producing the output for a sequence of length L requires on the order of L^2 query-key dot products and the same number of weighted value sums. Doubling the context roughly quadruples the attention work. During autoregressive decoding the bottleneck shifts from arithmetic to memory: each newly generated token must read the entire stored KV cache, so the per-token cost grows linearly with the context already produced and the KV cache becomes a dominant consumer of memory bandwidth. The NSA paper cites a theoretical estimate that softmax attention accounts for 70 to 80 percent of total latency when decoding 64k-length contexts. [1]
The paper frames this with arithmetic intensity, the ratio of compute operations to memory accesses. Training and prefilling run batched matrix multiplications with high arithmetic intensity and are compute-bound, while decoding generates one token per forward pass but must load the whole KV cache, so it is memory-bandwidth-bound. A useful sparse method therefore has two different jobs: cut computation during training and prefilling, and cut memory reads during decoding. [1]
A long line of work has tried to make attention cheaper. The NSA paper groups the closest prior methods into fixed sparse patterns (sliding windows, attention sinks as in StreamingLLM, and the local-plus-global layout of Longformer), dynamic KV-cache pruning (H2O, SnapKV and related eviction policies), and query-aware selection (Quest, InfLLM, HashAttention, ClusterKV, MInference, SeerAttention). DeepSeek's own multi-head latent attention, introduced with DeepSeek-V2, attacks the memory side differently by compressing the KV cache into a latent vector. [1][23] The NSA authors argue that the earlier sparse methods share weaknesses they set out to avoid. Some sparsify only one phase: H2O applies sparsity during decoding but needs expensive preprocessing at prefill, while MInference accelerates only prefill. Some clash with grouped-query and multi-query attention: in Quest each head selects its own KV subset, so in a GQA model the memory traffic is the union of all heads' selections in a group. And some are hard to train: k-means clustering (ClusterKV) or SimHash selection (MagicPIG) block gradients, while token-granular selection (HashAttention) scatters memory reads and prevents FlashAttention-style blockwise kernels. [1]
How does NSA work?
NSA replaces the full key and value history of each query q_t with a compact, query-dependent set of remapped keys and values. In the paper's notation, several mapping strategies c produce key-value sets K~_t^c and V~_t^c, each is attended separately, and the outputs are combined as o*_t = sum over c of g_t^c * Attn(q_t, K~_t^c, V~_t^c), where c ranges over C = {cmp, slc, win} (compression, selection, sliding window) and g_t^c in [0, 1] is a gate score. The total number of remapped keys, N_t, is kept much smaller than the context length t, which is where the sparsity comes from. [1]
Token compression
The compression branch gives the query a cheap, global view of the whole history. Consecutive keys, and separately values, are grouped into blocks of length l taken every d tokens, and each block is mapped to a single compressed key or value by a small learnable multilayer perceptron phi that includes an intra-block position encoding. The paper uses a stride smaller than the block length (d < l) so that blocks overlap, which it says mitigates information fragmentation; the configuration in the experiments is l = 32 and d = 16. Because a long sequence collapses to a much shorter sequence of compressed tokens, attending to them is inexpensive, and the resulting attention scores double as a coarse map of which regions of the context matter most. [1]
Token selection
Coarse compression alone loses fine detail, so the selection branch restores it by attending at full resolution to a small number of important blocks. NSA reuses the softmax attention scores already computed between the query and the compressed keys as importance scores, so selection adds little extra computation. When the selection blocks (size l') are larger than the compression blocks, the scores of the compression blocks that overlap each selection block are summed to give that block's score. The top-n blocks by score are then kept and their original, uncompressed keys and values are concatenated for attention. In the experiments l' = 64 and n = 16, and the 16 blocks include one initial block and two local blocks that are always active. [1]
Two design choices make this branch hardware-friendly. Selecting whole contiguous blocks rather than scattered individual tokens produces coalesced memory reads and suits Tensor Core matrix units; the authors add that attention scores tend to be spatially continuous, so neighboring keys usually have similar importance, an observation they also report in visualizations of their full-attention model. And when the model uses grouped-query or multi-query attention, the importance scores of all query heads in a group are summed so the heads share one set of selected blocks, which means the KV cache is loaded once per group rather than once per head during decoding. [1]
Sliding window
The third branch is a plain local window over the most recent w tokens, set to w = 512 in the paper. The authors explain that local patterns adapt faster and can dominate learning, which could stop the model from learning to use the compression and selection tokens. Giving local context its own branch lets the other two branches specialize in long-range information without being shortcut by local patterns. To further prevent shortcut learning, NSA gives each of the three branches its own independent keys and values, which the paper says adds only marginal computation. [1]
Gated combination
The outputs of the three branches are merged into the layer output by a gate. For each branch, an MLP with a sigmoid activation computes a gate score between 0 and 1 from the input features, and the final output is the gated sum of the three branch attentions. [1]
| NSA component | Role | Paper setting |
|---|---|---|
| Compression block / stride | coarse global summary | length l = 32, stride d = 16 |
| Selection block / count | fine-grained important regions | block l' = 64, keep top n = 16 (including 1 initial and 2 local blocks) |
| Sliding window | local recent context | w = 512 tokens |
| Gate | combine the three branches | MLP plus sigmoid, one score per branch |
| Keys and values | per-branch | independent sets for compression, selection and window |
During decoding, NSA needs to load at most about (s - l)/d compressed tokens, n x l' selected tokens and w window tokens, where s is the cached sequence length. With the settings above that is 2,048 tokens at an 8k context and 5,632 tokens at a 64k context, against 8,192 and 65,536 tokens for full attention: [1]
| Context length | Full attention (tokens loaded per step) | NSA (tokens loaded per step) | Expected decoding speedup |
|---|---|---|---|
| 8,192 | 8,192 | 2,048 | 4x |
| 16,384 | 16,384 | 2,560 | 6.4x |
| 32,768 | 32,768 | 3,584 | 9.1x |
| 65,536 | 65,536 | 5,632 | 11.6x |
The paper presents these figures as memory-access volume per attention operation, measured in equivalent tokens, and notes that because decoding is memory-bound the expected speedup is approximately linear in that volume. [1]
What makes NSA hardware-aligned and natively trainable?
The hardware-aligned principle shows up in NSA's custom kernels, written in Triton and built on the same IO-aware, blockwise philosophy as FlashAttention. The paper targets architectures with shared KV caches (GQA and MQA) because plain multi-head attention is memory-intensive and inefficient for decoding. The compression and sliding-window branches are compatible with existing FlashAttention-2 kernels, so the new work is a kernel for the selection branch. FlashAttention loads contiguous blocks of queries into on-chip SRAM, but in NSA neighboring queries may need different KV blocks, so that strategy would waste memory traffic. NSA instead groups queries by GQA group: [1]
- Group-centric data loading. For each query position t, the kernel loads the queries of all heads in one GQA group together with their shared list of selected block indices.
- Shared KV fetching. In the inner loop, it loads the selected contiguous key and value blocks into SRAM one after another, with a kernel block size that divides the selection block size.
- Outer loop on grid. Because the inner-loop length (proportional to n) is nearly the same for every query block, the query and output loops are handed to Triton's grid scheduler.
The authors say this achieves near-optimal arithmetic intensity by removing redundant KV transfers through group-wise sharing and by balancing work across the GPU's streaming multiprocessors. [1] The design relies on having many query heads per GQA group: the paper's model uses 4 groups of 16 heads. A later kernel paper, Flash Sparse Attention, argues that this loop order is only efficient with a relatively large number of query heads per group, whereas many existing LLMs use far fewer (see the follow-up section below). [1][14]
The natively trainable principle is the conceptual heart of the paper. Rather than train a dense model and then discard attention at inference, NSA is part of the architecture during pretraining, so the model adapts its representations to the sparse pattern and there is no train-test gap to pay at deployment. The abstract states that this reduces "pretraining computation without sacrificing model performance." [1] The top-n block choice itself is a discrete step, and the paper does not train a separate scorer for it: the block scores are by-products of the compression branch's attention, so no auxiliary loss is needed. The authors contrast this with alternatives they tried and abandoned, described below. [1]
How was NSA trained and evaluated?
The authors pretrained an NSA model and an otherwise identical full-attention baseline. The backbone follows what the paper calls common practice in state-of-the-art LLMs, combining grouped-query attention with a mixture-of-experts design in the DeepSeekMoE style: [1]
| Setting | Value |
|---|---|
| Total / active parameters | 27B / 3B |
| Layers / hidden size | 30 / 2,560 |
| Attention | GQA, 4 groups, 64 heads in total; d_q = d_k = 192, d_v = 128 |
| MoE | 72 routed experts plus 2 shared experts, top-6 routing; the first layer uses a SwiGLU MLP instead of MoE |
| Pretraining | 270B tokens of 8k-length text |
| Long-context adaptation | continued training and supervised fine-tuning on 32k-length text with YaRN |
| Speed benchmarks | 8-GPU A100 system, Triton kernels for both NSA and the FlashAttention-2 baseline |
Both models were trained to full convergence, and the paper's pretraining loss curve shows the NSA model with consistently lower loss than the full-attention model. The token count is given as 270 billion in the experimental-setup section and as 260 billion in the introduction, in both the arXiv and ACL versions. [1][2]
How accurate and how fast is NSA?
General benchmarks
On nine knowledge, reasoning and coding benchmarks, the authors report that NSA outperformed the full-attention model on 7 of 9 metrics and on average (0.456 against 0.443). The largest gains were on DROP (+0.042) and GSM8K (+0.034); full attention scored higher on MMLU and MBPP. The authors note that most samples in these benchmarks fit inside the local context window of the inference-time sparse baselines, which makes those baselines effectively equivalent to full attention, so they compared NSA only against full attention here. [1]
| Benchmark (setting) | Full attention | NSA |
|---|---|---|
| MMLU (5-shot acc.) | 0.567 | 0.565 |
| MMLU-Pro (5-shot acc.) | 0.279 | 0.286 |
| CMMLU (5-shot acc.) | 0.576 | 0.587 |
| BBH (3-shot acc.) | 0.497 | 0.521 |
| GSM8K (8-shot acc.) | 0.486 | 0.520 |
| MATH (4-shot acc.) | 0.263 | 0.264 |
| DROP (1-shot F1) | 0.503 | 0.545 |
| MBPP (3-shot pass@1) | 0.482 | 0.466 |
| HumanEval (0-shot pass@1) | 0.335 | 0.348 |
| Average | 0.443 | 0.456 |
Long-context benchmarks
On a 64k-context needle-in-a-haystack test, the authors report perfect retrieval accuracy at every depth. On LongBench, they compared NSA with full attention and with four inference-time sparse methods (H2O, InfLLM, Quest, and Exact-Top, which computes full attention scores and keeps each query's top-n keys). For a fair comparison every sparse baseline was given a budget of 2,560 activated tokens per query, the average NSA activates at a 32k sequence length, including the leading 128 tokens and 512 local tokens; some LongBench subsets on which all models scored poorly were excluded. [1]
| Method | LongBench average (selected subsets) |
|---|---|
| H2O | 0.303 |
| InfLLM | 0.383 |
| Quest | 0.392 |
| Exact-Top | 0.423 |
| Full attention | 0.437 |
| NSA | 0.469 |
NSA's largest margins over full attention were on the multi-hop QA tasks HotpotQA (+0.087) and 2WikiMultihopQA (+0.051), on the LCC code task (+0.069) and on English passage retrieval (+0.075). It was below full attention on three subsets: English MultiFieldQA (0.503 against 0.512), GovReport summarization (0.307 against 0.324) and Chinese passage retrieval (0.550 against 0.560). [1]
Chain-of-thought reasoning
To test whether the sparse model could learn long chain-of-thought reasoning, the authors used knowledge distillation from DeepSeek-R1 rather than reinforcement learning, which they describe as less effective on smaller models: both models were fine-tuned on 10 billion tokens of 32k-length mathematical reasoning traces. The resulting models were scored on AIME 2024, averaging 16 samples per question at temperature 0.7 and top-p 0.95. Only full attention was used as a baseline because the inference-time sparse methods do not support training. [1]
| Generation token limit | Full Attention-R | NSA-R |
|---|---|---|
| 8,192 | 0.046 | 0.121 |
| 16,384 | 0.092 | 0.146 |
As the abstract summarizes, "the model pretrained with NSA maintains or exceeds Full Attention models across general benchmarks, long-context tasks, and instruction-based reasoning." [1]
Kernel speed
The speed comparison pits Triton-based NSA kernels against a Triton-based FlashAttention-2 kernel on the same backend. The speedup grows with context length: [1]
| Context length | Forward speedup | Backward speedup |
|---|---|---|
| 8k | 2.1x | 1.1x |
| 16k | 3.8x | 2.0x |
| 32k | 6.3x | 3.4x |
| 64k | 9.0x | 6.0x |
The authors attribute the gain to blockwise memory access that maximizes Tensor Core utilization through coalesced loads and to loop scheduling that eliminates redundant KV transfers. For decoding, the headline 11.6x at 64k is the expected speedup derived from KV-cache memory access in the table above, not a separate wall-clock benchmark. [1]
Design alternatives the authors rejected
Section 6 of the arXiv preprint (moved to an appendix in the ACL version) describes approaches the team tried before settling on NSA. Key-clustering strategies such as ClusterKV were rejected because of the overhead of dynamic clustering, load imbalances between clusters (worse in mixture-of-experts systems with expert parallelism), and the need for periodic reclustering. Two other blockwise selection strategies were tested on a 3B-parameter model with a similar architecture: an auxiliary-loss-based learnable selector (conceptually similar to SeerAttention, supervised with KL divergence against mean-pooled attention scores) and a heuristic, parameter-free selector following Quest, including a variant that used full attention for the first 1,000 steps before switching. The paper reports that both had worse training loss than NSA. A visualization of the attention map of the authors' 27B full-attention model, showing blockwise clustering of attention scores, is presented as the inspiration for block-based selection. [1]
Limitations and critiques
The ACL version of the paper adds a limitations section: the work considers sparsity within the attention map of a single layer, leaving cross-layer sparsity to future work, and the Triton implementation carries some abstraction overhead compared with native CUDA kernels. [2] As of the InfLLM-V2 paper (September 2025), the NSA authors had not published their code, so that team used a community Triton implementation for its NSA experiments. [15]
Later papers have raised several criticisms:
- Extra parameters and short-context cost. The MiniCPM4 technical report (June 2025) argues that NSA's three attention components add parameters, raising computation on short sequences and tripling key-value storage during pretraining. [16]
- Short-to-long adaptation. The InfLLM-V2 authors report that when they converted a model pretrained with dense attention on short sequences into an NSA model for long-context fine-tuning, NSA disrupted the training loss and gave unsatisfactory results, although they note that the original paper trained NSA from scratch. [15]
- Kernel generality. Flash Sparse Attention (August 2025) argues that NSA's kernel loop order is efficient only when each GQA group has many query heads, which limits its use on popular LLMs with smaller groups. [14]
How does NSA compare to other sparse-attention methods?
NSA builds on the lineage of FlashAttention: it does not change attention's mathematics so much as supply blockwise, SRAM-tiling kernels that make its specific sparse pattern fast, and its full-attention baseline is itself a FlashAttention-2 implementation. Where FlashAttention computes dense attention exactly but efficiently, NSA computes a learned sparse subset efficiently. [1][6]
NSA appeared two days before Mixture of Block Attention (MoBA), posted to arXiv on February 18, 2025 by researchers at Moonshot AI, which says MoBA was already deployed to support Kimi's long-context requests. MoBA also makes block-sparse attention trainable, but it does so by applying the mixture-of-experts routing idea directly to attention: the context is partitioned into blocks and each query is routed to its top-k blocks according to the inner product between the query and the mean-pooled keys of each block, with no learned gating parameters. Compared with MoBA's single routing mechanism, NSA adds the explicit compression and sliding-window branches and the three-way gate, and builds its selection kernel around GQA groups. Both works demonstrated that sparsity learned during training can match full attention, a convergence from two laboratories in the same week. [1][4]
DeepSeek Sparse Attention (DSA) is the sparse mechanism DeepSeek shipped in its experimental DeepSeek-V3.2-Exp model on September 29, 2025, and carried into DeepSeek-V3.2. DSA uses a lightweight "lightning indexer" to score the relevance of each preceding token to the current query, and a fine-grained selector then keeps only the top-k tokens for attention; it is instantiated on top of multi-head latent attention. DeepSeek's technical reports cite the NSA paper once, to support the point that at the kernel level each key-value entry must be shared across multiple queries for efficiency, which is why DSA uses the MQA mode of MLA. The reports do not otherwise describe how DSA relates to NSA. The two differ in granularity: NSA selects contiguous blocks and adds compression and window branches, while DSA selects individual tokens with a separate indexer, and DeepSeek's release README says DSA "achieves fine-grained sparse attention for the first time." With the V3.2-Exp release DeepSeek also announced API price cuts of more than 50 percent. [5][8][9][10]
In June 2026, MiniMax published MiniMax Sparse Attention (MSA), the mechanism behind MiniMax M3. The MSA paper describes natively trained sparse-attention designs, NSA among them, as its closest prior work and distinguishes MSA by combining per-GQA-group top-k sharing with block-level selection through a lightweight index branch trained with a KL alignment loss. [19] Together, NSA, MoBA, DSA and their successors mark a shift away from inference-only sparse attention toward sparsity that is trained into the model.
Open implementations and follow-up work
Community implementations appeared within days of the preprint; the InfLLM-V2 authors later noted that the NSA code had not been published and used one of them. GitHub figures below were retrieved from the GitHub API on October 31, 2026. [11][12][13][14][15]
| Repository | Created | Description | License | Stars (Oct 31, 2026) |
|---|---|---|---|---|
| lucidrains/native-sparse-attention-pytorch | February 19, 2025 | PyTorch implementation of the NSA sparse attention pattern | MIT | 816 |
| fla-org/native-sparse-attention | February 21, 2025 | Triton kernels; added an online top-k selection kernel (February 25, 2025) and full training support with the flame framework (March 15, 2025) | MIT | 1,026 |
| XunhaoLai/native-sparse-attention-triton | February 24, 2025 | Triton implementation; used for the NSA baseline in the InfLLM-V2 paper | Apache-2.0 | 285 |
| Relaxed-System-Lab/Flash-Sparse-Attention | August 17, 2025 | Alternative NSA kernels from the FSA paper | Apache-2.0 | 590 |
Selected follow-up research, by date of first arXiv posting:
| Date | Work | Relationship to NSA |
|---|---|---|
| March 12, 2025 | TabNSA (Eslamian and Cheng) | Combines NSA with a TabMixer backbone for tabular data learning. [20] |
| June 9, 2025 | MiniCPM4 technical report (OpenBMB) | Introduces InfLLM v2 trainable sparse attention and positions it against NSA and MoBA; claims its top-k block selection cuts computation by 60 percent compared with NSA. [16] |
| June 14, 2025 | BSA: Ball Sparse Attention (Brita et al.) | Adapts NSA to unordered point sets using the Ball Tree structure of the Erwin Transformer. [21] |
| August 14, 2025 | Erwin NSA (Lapautre et al.) | Combines NSA with the Erwin architecture for physical-science point clouds (cosmology, molecular dynamics, air pressure). [22] |
| August 25, 2025 | FSA: Flash Sparse Attention (Yan et al.) | Alternative NSA kernel for small GQA groups; reports up to 3.5x (average 1.6x) kernel-level latency reduction, up to 1.25x end-to-end training speedup and up to 1.36x prefill speedup over the vanilla NSA kernel. [14] |
| September 29, 2025 | InfLLM-V2 (Zhao et al.) | Dense-sparse switchable attention that reuses dense attention parameters instead of NSA's extra key-value sets; follows NSA's block settings (compression block 32, stride 16, block 64) and GQA group size of 16; used to train the open-sourced MiniCPM4.1-8B. [15] |
| October 2, 2025 | VideoNSA (Song et al., ICLR 2026 per the arXiv listing) | Adapts Qwen2.5-VL with NSA on video tokens and dense attention on text, trained end to end on a 216K video instruction dataset; reports scaling to 128K tokens. [17] |
| November 2, 2025 | Hu et al. | Alternates local and global (compression and selection) attention across layers and adds latent attention to NSA's branches; reports a 50 percent KV-cache memory reduction versus NSA, with experiments on 340M to 1.3B models. [18] |
| June 11, 2026 | MiniMax Sparse Attention (Lai et al.) | Cites NSA among the closest prior work; per-GQA-group block selection with a separate index branch. [19] |
Continue reading
- AttentionAttention is a family of operations in neural networks that computes an output by assigning data-dependent weights to a collection of representations and combining them.Mentioned 92 times above
- Sparse attentionSparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token attend to only a subset of other tokens in a sequenceMentioned 20 times above
- DeepSeekDeepSeek is a Chinese artificial intelligence company based in Hangzhou. It was founded in 2023 by Liang Wenfeng, who had previously co-founded the quantitative investment firm High-Flyer.Mentioned 13 times above
- Flash AttentionFlashAttention is a family of exact, input/output-aware algorithms and GPU kernels for scaled dot-product attention.Mentioned 12 times above
- Context windowA context window is the finite token sequence that a language model can process for one invocation.Mentioned 11 times above
- Peking UniversityPeking University (PKU; Chinese: 北京大学; pinyin: Běijīng Dàxué), often abbreviated as Beida (北大), is a public research university in BeijingMentioned twice above
More to read
- Large Language Model
- NVIDIA A100
- DeepSeek Sparse Attention (DSA)
- MiniMax M3
- Mixture of Block Attention (MoBA)
- KV Cache
Topics:Deep LearningLarge Language ModelsModel ArchitectureNeural Networks
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40Yuan, Jingyang; Gao, Huazuo; Dai, Damai; Luo, Junyu; Zhao, Liang; Zhang, Zhengyan; Xie, Zhenda; Wei, Y. X.; Wang, Lean; Xiao, Zhiping; Wang, Yuqing; Ruan, Chong; Zhang, Ming; Liang, Wenfeng; Zeng, Wangding. "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention." arXiv:2502.11089, February 16, 2025. arxiv.org/...2502.11089
- ^1 ^2 ^3 ^4Yuan, Jingyang, et al. "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), Volume 1: Long Papers, pages 23078 to 23097, Vienna, Austria, July 2025. aclanthology.org/2025.acl-long.1126
- ^Wang, Hannah. "DeepSeek founder shares best paper award at top global AI research conference." South China Morning Post, July 31, 2025. scmp.com/...ward-top-global-ai-research-conference
- ^Lu, Enzhe, et al. "MoBA: Mixture of Block Attention for Long-Context LLMs." arXiv:2502.13189, February 18, 2025. arxiv.org/...2502.13189
- ^DeepSeek-AI. "DeepSeek-V3.2-Exp" (model release introducing DeepSeek Sparse Attention). September 29, 2025. github.com/...DeepSeek-V3.2-Exp
- ^Dao, Tri; Fu, Daniel Y.; Ermon, Stefano; Rudra, Atri; Re, Christopher. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." arXiv:2205.14135, May 2022. arxiv.org/...2205.14135
- ^1 ^2ACL 2025. "Awards" (Best Paper list). 63rd Annual Meeting of the Association for Computational Linguistics. 2025.aclweb.org/...awards
- ^DeepSeek-AI. "DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention." Technical report, September 2025. github.com/...DeepSeek_V3_2.pdf
- ^DeepSeek-AI. "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models." arXiv:2512.02556, December 2, 2025. arxiv.org/...2512.02556
- ^DeepSeek. "Introducing DeepSeek-V3.2-Exp." DeepSeek API Docs news, September 29, 2025. api-docs.deepseek.com/...news250929
- ^fla-org. "native-sparse-attention" (Efficient Triton implementations for Native Sparse Attention). GitHub repository, created February 21, 2025; accessed October 31, 2026. github.com/...native-sparse-attention
- ^Wang, Phil (lucidrains). "native-sparse-attention-pytorch." GitHub repository, created February 19, 2025; accessed October 31, 2026. github.com/...native-sparse-attention-pytorch
- ^Lai, Xunhao. "native-sparse-attention-triton." GitHub repository, created February 24, 2025; accessed October 31, 2026. github.com/...native-sparse-attention-triton
- ^1 ^2 ^3 ^4Yan, Ran; Jiang, Youhe; Chen, Zhuoming; Mai, Haohui; Chen, Beidi; Yuan, Binhang. "FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel." arXiv:2508.18224, August 25, 2025. arxiv.org/...2508.18224 (code: github.com/...Flash-Sparse-Attention)
- ^1 ^2 ^3 ^4Zhao, Weilin, et al. "InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation." arXiv:2509.24663, September 29, 2025. arxiv.org/...2509.24663
- ^1 ^2MiniCPM Team. "MiniCPM4: Ultra-Efficient LLMs on End Devices." arXiv:2506.07900, June 9, 2025. arxiv.org/...2506.07900
- ^Song, Enxin, et al. "VideoNSA: Native Sparse Attention Scales Video Understanding." arXiv:2510.02295, October 2, 2025. arxiv.org/...2510.02295
- ^Hu, Yuxuan, et al. "Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies." arXiv:2511.00819, November 2, 2025. arxiv.org/...2511.00819
- ^1 ^2Lai, Xunhao, et al. "MiniMax Sparse Attention." arXiv:2606.13392, June 11, 2026. arxiv.org/...2606.13392
- ^Eslamian, Ali; Cheng, Qiang. "TabNSA: Native Sparse Attention for Efficient Tabular Data Learning." arXiv:2503.09850, March 12, 2025. arxiv.org/...2503.09850
- ^Brita, Catalin E., et al. "BSA: Ball Sparse Attention for Large-scale Geometries." arXiv:2506.12541, June 14, 2025. arxiv.org/...2506.12541
- ^Lapautre, Nicolas, et al. "Natively Trainable Sparse Attention for Hierarchical Point Cloud Datasets." arXiv:2508.10758, August 14, 2025. arxiv.org/...2508.10758
- ^DeepSeek-AI. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." arXiv:2405.04434, May 2024. arxiv.org/...2405.04434
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
5 revisions · v5 · 4,960 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independently verified 2026-10-31 against the NSA paper (arXiv 2502.11089 v1/v2 and the ACL 2025 Anthology version), ACL awards page, DeepSeek V3.2 reports, MoBA paper and GitHub API; 4 verifier defects fixed before stamping.
Cite this page: AI Wiki. "Native Sparse Attention (NSA)." aiwiki.ai, updated 31 Oct 2026, fact-checked 31 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/native_sparse_attention


