Citation and evidence

Native Sparse Attention (NSA)

25 min full readUpdated 23 references

This article's verification

Report a problem with this article

Edit article
More

Cite this article

Native Sparse Attention (NSA) is a hardware-aligned, natively trainable sparse attention mechanism introduced in February 2025 by researchers at DeepSeek, Peking University and the University of Washington. It is designed to make long-context language model training and inference cheaper while keeping the accuracy of dense (full) attention. Instead of letting every query attend to every earlier token, NSA runs three parallel branches over the key-value sequence: a coarse-grained compression branch, a fine-grained block-selection branch, and a local sliding window, blended by a learned gate. In kernel benchmarks on an 8-GPU NVIDIA A100 system at a 64k-token (65,536-token) context, the authors report up to 9.0 times faster forward propagation and 6.0 times faster backward propagation than a Triton implementation of FlashAttention-2, and an expected decoding speedup of up to 11.6 times, estimated from the reduction in key-value cache reads. A 27-billion-parameter model pretrained with NSA matched or exceeded an identically trained full-attention model on the authors' general, long-context and reasoning evaluations. The method is described in the paper "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention" by Jingyang Yuan and 14 co-authors, which was one of four Best Paper award winners at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025) in Vienna. [1][2][3][7]

At a glance
Paper"Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention" [1]
AuthorsJingyang Yuan (first author, internship at DeepSeek-AI), Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Liang Wenfeng, Wangding Zeng [1]
AffiliationsDeepSeek-AI; Peking University (School of Computer Science, PKU-Anker LLM Lab); University of Washington [1]
PreprintarXiv:2502.11089, v1 February 16, 2025; v2 February 27, 2025 [1]
Published versionACL 2025 (Volume 1: Long Papers), pages 23078-23097, DOI 10.18653/v1/2025.acl-long.1126 [2]
AwardACL 2025 Best Paper (one of four) [7]
Branchescompression (block 32, stride 16), selection (block 64, top 16 blocks), sliding window (512 tokens) [1]
Test model27B-total, 3B-active GQA + MoE transformer, 30 layers [1]
KernelsTriton; compression and window branches use FlashAttention-2-compatible kernels, selection uses a custom group-centric kernel [1]

Expanded article table

What is Native Sparse Attention?

The core problem NSA targets is that the self-attention operation in a transformer has computational cost that grows quadratically, O(L^2), with sequence length L. For modern context windows of tens or hundreds of thousands of tokens, attention comes to dominate both the compute of the forward and backward passes during training and the memory traffic of the key-value (KV) cache during decoding. Sparse attention, which computes only a subset of the query-key interactions, is the obvious remedy, but two practical obstacles had limited earlier methods. First, many sparse schemes are applied only at inference time on top of a model that was pretrained with full attention, so the model never learns to operate with the sparsity and a train-inference mismatch degrades quality. Second, theoretical reductions in floating point operations frequently fail to produce real wall-clock speedups because the sparse access pattern is irregular and does not map well to GPU memory hierarchies or to optimized kernels such as FlashAttention. [1]

NSA's contribution is to address both obstacles at once through two stated design principles: it is hardware-aligned, meaning its sparse pattern is organized into contiguous blocks and its kernels are written to keep the GPU's arithmetic units busy and to cooperate with grouped-query attention; and it is natively trainable, meaning the sparsity is present from the start of pretraining rather than bolted on afterward. The paper's abstract states that NSA "employs a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection to preserve both global context awareness and local precision." [1] The authors report that an NSA model pretrained end-to-end matches or exceeds an equivalent full-attention model on general, long-context, and reasoning benchmarks, while delivering large speedups on 64k-token sequences. [1]

Why is long-context attention so expensive?

In a standard attention layer, every query token attends to every preceding key token, so producing the output for a sequence of length L requires on the order of L^2 query-key dot products and the same number of weighted value sums. Doubling the context roughly quadruples the attention work. During autoregressive decoding the bottleneck shifts from arithmetic to memory: each newly generated token must read the entire stored KV cache, so the per-token cost grows linearly with the context already produced and the KV cache becomes a dominant consumer of memory bandwidth. The NSA paper cites a theoretical estimate that softmax attention accounts for 70 to 80 percent of total latency when decoding 64k-length contexts. [1]

The paper frames this with arithmetic intensity, the ratio of compute operations to memory accesses. Training and prefilling run batched matrix multiplications with high arithmetic intensity and are compute-bound, while decoding generates one token per forward pass but must load the whole KV cache, so it is memory-bandwidth-bound. A useful sparse method therefore has two different jobs: cut computation during training and prefilling, and cut memory reads during decoding. [1]

A long line of work has tried to make attention cheaper. The NSA paper groups the closest prior methods into fixed sparse patterns (sliding windows, attention sinks as in StreamingLLM, and the local-plus-global layout of Longformer), dynamic KV-cache pruning (H2O, SnapKV and related eviction policies), and query-aware selection (Quest, InfLLM, HashAttention, ClusterKV, MInference, SeerAttention). DeepSeek's own multi-head latent attention, introduced with DeepSeek-V2, attacks the memory side differently by compressing the KV cache into a latent vector. [1][23] The NSA authors argue that the earlier sparse methods share weaknesses they set out to avoid. Some sparsify only one phase: H2O applies sparsity during decoding but needs expensive preprocessing at prefill, while MInference accelerates only prefill. Some clash with grouped-query and multi-query attention: in Quest each head selects its own KV subset, so in a GQA model the memory traffic is the union of all heads' selections in a group. And some are hard to train: k-means clustering (ClusterKV) or SimHash selection (MagicPIG) block gradients, while token-granular selection (HashAttention) scatters memory reads and prevents FlashAttention-style blockwise kernels. [1]

How does NSA work?

NSA replaces the full key and value history of each query q_t with a compact, query-dependent set of remapped keys and values. In the paper's notation, several mapping strategies c produce key-value sets K~_t^c and V~_t^c, each is attended separately, and the outputs are combined as o*_t = sum over c of g_t^c * Attn(q_t, K~_t^c, V~_t^c), where c ranges over C = {cmp, slc, win} (compression, selection, sliding window) and g_t^c in [0, 1] is a gate score. The total number of remapped keys, N_t, is kept much smaller than the context length t, which is where the sparsity comes from. [1]

Token compression

The compression branch gives the query a cheap, global view of the whole history. Consecutive keys, and separately values, are grouped into blocks of length l taken every d tokens, and each block is mapped to a single compressed key or value by a small learnable multilayer perceptron phi that includes an intra-block position encoding. The paper uses a stride smaller than the block length (d < l) so that blocks overlap, which it says mitigates information fragmentation; the configuration in the experiments is l = 32 and d = 16. Because a long sequence collapses to a much shorter sequence of compressed tokens, attending to them is inexpensive, and the resulting attention scores double as a coarse map of which regions of the context matter most. [1]

Token selection

Coarse compression alone loses fine detail, so the selection branch restores it by attending at full resolution to a small number of important blocks. NSA reuses the softmax attention scores already computed between the query and the compressed keys as importance scores, so selection adds little extra computation. When the selection blocks (size l') are larger than the compression blocks, the scores of the compression blocks that overlap each selection block are summed to give that block's score. The top-n blocks by score are then kept and their original, uncompressed keys and values are concatenated for attention. In the experiments l' = 64 and n = 16, and the 16 blocks include one initial block and two local blocks that are always active. [1]

Two design choices make this branch hardware-friendly. Selecting whole contiguous blocks rather than scattered individual tokens produces coalesced memory reads and suits Tensor Core matrix units; the authors add that attention scores tend to be spatially continuous, so neighboring keys usually have similar importance, an observation they also report in visualizations of their full-attention model. And when the model uses grouped-query or multi-query attention, the importance scores of all query heads in a group are summed so the heads share one set of selected blocks, which means the KV cache is loaded once per group rather than once per head during decoding. [1]

Sliding window

The third branch is a plain local window over the most recent w tokens, set to w = 512 in the paper. The authors explain that local patterns adapt faster and can dominate learning, which could stop the model from learning to use the compression and selection tokens. Giving local context its own branch lets the other two branches specialize in long-range information without being shortcut by local patterns. To further prevent shortcut learning, NSA gives each of the three branches its own independent keys and values, which the paper says adds only marginal computation. [1]

Gated combination

The outputs of the three branches are merged into the layer output by a gate. For each branch, an MLP with a sigmoid activation computes a gate score between 0 and 1 from the input features, and the final output is the gated sum of the three branch attentions. [1]

NSA componentRolePaper setting
Compression block / stridecoarse global summarylength l = 32, stride d = 16
Selection block / countfine-grained important regionsblock l' = 64, keep top n = 16 (including 1 initial and 2 local blocks)
Sliding windowlocal recent contextw = 512 tokens
Gatecombine the three branchesMLP plus sigmoid, one score per branch
Keys and valuesper-branchindependent sets for compression, selection and window

Expanded article table

During decoding, NSA needs to load at most about (s - l)/d compressed tokens, n x l' selected tokens and w window tokens, where s is the cached sequence length. With the settings above that is 2,048 tokens at an 8k context and 5,632 tokens at a 64k context, against 8,192 and 65,536 tokens for full attention: [1]

Context lengthFull attention (tokens loaded per step)NSA (tokens loaded per step)Expected decoding speedup
8,1928,1922,0484x
16,38416,3842,5606.4x
32,76832,7683,5849.1x
65,53665,5365,63211.6x

Expanded article table

The paper presents these figures as memory-access volume per attention operation, measured in equivalent tokens, and notes that because decoding is memory-bound the expected speedup is approximately linear in that volume. [1]

What makes NSA hardware-aligned and natively trainable?

The hardware-aligned principle shows up in NSA's custom kernels, written in Triton and built on the same IO-aware, blockwise philosophy as FlashAttention. The paper targets architectures with shared KV caches (GQA and MQA) because plain multi-head attention is memory-intensive and inefficient for decoding. The compression and sliding-window branches are compatible with existing FlashAttention-2 kernels, so the new work is a kernel for the selection branch. FlashAttention loads contiguous blocks of queries into on-chip SRAM, but in NSA neighboring queries may need different KV blocks, so that strategy would waste memory traffic. NSA instead groups queries by GQA group: [1]

  1. Group-centric data loading. For each query position t, the kernel loads the queries of all heads in one GQA group together with their shared list of selected block indices.
  2. Shared KV fetching. In the inner loop, it loads the selected contiguous key and value blocks into SRAM one after another, with a kernel block size that divides the selection block size.
  3. Outer loop on grid. Because the inner-loop length (proportional to n) is nearly the same for every query block, the query and output loops are handed to Triton's grid scheduler.

The authors say this achieves near-optimal arithmetic intensity by removing redundant KV transfers through group-wise sharing and by balancing work across the GPU's streaming multiprocessors. [1] The design relies on having many query heads per GQA group: the paper's model uses 4 groups of 16 heads. A later kernel paper, Flash Sparse Attention, argues that this loop order is only efficient with a relatively large number of query heads per group, whereas many existing LLMs use far fewer (see the follow-up section below). [1][14]

The natively trainable principle is the conceptual heart of the paper. Rather than train a dense model and then discard attention at inference, NSA is part of the architecture during pretraining, so the model adapts its representations to the sparse pattern and there is no train-test gap to pay at deployment. The abstract states that this reduces "pretraining computation without sacrificing model performance." [1] The top-n block choice itself is a discrete step, and the paper does not train a separate scorer for it: the block scores are by-products of the compression branch's attention, so no auxiliary loss is needed. The authors contrast this with alternatives they tried and abandoned, described below. [1]

How was NSA trained and evaluated?

The authors pretrained an NSA model and an otherwise identical full-attention baseline. The backbone follows what the paper calls common practice in state-of-the-art LLMs, combining grouped-query attention with a mixture-of-experts design in the DeepSeekMoE style: [1]

SettingValue
Total / active parameters27B / 3B
Layers / hidden size30 / 2,560
AttentionGQA, 4 groups, 64 heads in total; d_q = d_k = 192, d_v = 128
MoE72 routed experts plus 2 shared experts, top-6 routing; the first layer uses a SwiGLU MLP instead of MoE
Pretraining270B tokens of 8k-length text
Long-context adaptationcontinued training and supervised fine-tuning on 32k-length text with YaRN
Speed benchmarks8-GPU A100 system, Triton kernels for both NSA and the FlashAttention-2 baseline

Expanded article table

Both models were trained to full convergence, and the paper's pretraining loss curve shows the NSA model with consistently lower loss than the full-attention model. The token count is given as 270 billion in the experimental-setup section and as 260 billion in the introduction, in both the arXiv and ACL versions. [1][2]

How accurate and how fast is NSA?

General benchmarks

On nine knowledge, reasoning and coding benchmarks, the authors report that NSA outperformed the full-attention model on 7 of 9 metrics and on average (0.456 against 0.443). The largest gains were on DROP (+0.042) and GSM8K (+0.034); full attention scored higher on MMLU and MBPP. The authors note that most samples in these benchmarks fit inside the local context window of the inference-time sparse baselines, which makes those baselines effectively equivalent to full attention, so they compared NSA only against full attention here. [1]

Benchmark (setting)Full attentionNSA
MMLU (5-shot acc.)0.5670.565
MMLU-Pro (5-shot acc.)0.2790.286
CMMLU (5-shot acc.)0.5760.587
BBH (3-shot acc.)0.4970.521
GSM8K (8-shot acc.)0.4860.520
MATH (4-shot acc.)0.2630.264
DROP (1-shot F1)0.5030.545
MBPP (3-shot pass@1)0.4820.466
HumanEval (0-shot pass@1)0.3350.348
Average0.4430.456

Expanded article table

Long-context benchmarks

On a 64k-context needle-in-a-haystack test, the authors report perfect retrieval accuracy at every depth. On LongBench, they compared NSA with full attention and with four inference-time sparse methods (H2O, InfLLM, Quest, and Exact-Top, which computes full attention scores and keeps each query's top-n keys). For a fair comparison every sparse baseline was given a budget of 2,560 activated tokens per query, the average NSA activates at a 32k sequence length, including the leading 128 tokens and 512 local tokens; some LongBench subsets on which all models scored poorly were excluded. [1]

MethodLongBench average (selected subsets)
H2O0.303
InfLLM0.383
Quest0.392
Exact-Top0.423
Full attention0.437
NSA0.469

Expanded article table

NSA's largest margins over full attention were on the multi-hop QA tasks HotpotQA (+0.087) and 2WikiMultihopQA (+0.051), on the LCC code task (+0.069) and on English passage retrieval (+0.075). It was below full attention on three subsets: English MultiFieldQA (0.503 against 0.512), GovReport summarization (0.307 against 0.324) and Chinese passage retrieval (0.550 against 0.560). [1]

Chain-of-thought reasoning

To test whether the sparse model could learn long chain-of-thought reasoning, the authors used knowledge distillation from DeepSeek-R1 rather than reinforcement learning, which they describe as less effective on smaller models: both models were fine-tuned on 10 billion tokens of 32k-length mathematical reasoning traces. The resulting models were scored on AIME 2024, averaging 16 samples per question at temperature 0.7 and top-p 0.95. Only full attention was used as a baseline because the inference-time sparse methods do not support training. [1]

Generation token limitFull Attention-RNSA-R
8,1920.0460.121
16,3840.0920.146

Expanded article table

As the abstract summarizes, "the model pretrained with NSA maintains or exceeds Full Attention models across general benchmarks, long-context tasks, and instruction-based reasoning." [1]

Kernel speed

The speed comparison pits Triton-based NSA kernels against a Triton-based FlashAttention-2 kernel on the same backend. The speedup grows with context length: [1]

Context lengthForward speedupBackward speedup
8k2.1x1.1x
16k3.8x2.0x
32k6.3x3.4x
64k9.0x6.0x

Expanded article table

The authors attribute the gain to blockwise memory access that maximizes Tensor Core utilization through coalesced loads and to loop scheduling that eliminates redundant KV transfers. For decoding, the headline 11.6x at 64k is the expected speedup derived from KV-cache memory access in the table above, not a separate wall-clock benchmark. [1]

Design alternatives the authors rejected

Section 6 of the arXiv preprint (moved to an appendix in the ACL version) describes approaches the team tried before settling on NSA. Key-clustering strategies such as ClusterKV were rejected because of the overhead of dynamic clustering, load imbalances between clusters (worse in mixture-of-experts systems with expert parallelism), and the need for periodic reclustering. Two other blockwise selection strategies were tested on a 3B-parameter model with a similar architecture: an auxiliary-loss-based learnable selector (conceptually similar to SeerAttention, supervised with KL divergence against mean-pooled attention scores) and a heuristic, parameter-free selector following Quest, including a variant that used full attention for the first 1,000 steps before switching. The paper reports that both had worse training loss than NSA. A visualization of the attention map of the authors' 27B full-attention model, showing blockwise clustering of attention scores, is presented as the inspiration for block-based selection. [1]

Limitations and critiques

The ACL version of the paper adds a limitations section: the work considers sparsity within the attention map of a single layer, leaving cross-layer sparsity to future work, and the Triton implementation carries some abstraction overhead compared with native CUDA kernels. [2] As of the InfLLM-V2 paper (September 2025), the NSA authors had not published their code, so that team used a community Triton implementation for its NSA experiments. [15]

Later papers have raised several criticisms:

  • Extra parameters and short-context cost. The MiniCPM4 technical report (June 2025) argues that NSA's three attention components add parameters, raising computation on short sequences and tripling key-value storage during pretraining. [16]
  • Short-to-long adaptation. The InfLLM-V2 authors report that when they converted a model pretrained with dense attention on short sequences into an NSA model for long-context fine-tuning, NSA disrupted the training loss and gave unsatisfactory results, although they note that the original paper trained NSA from scratch. [15]
  • Kernel generality. Flash Sparse Attention (August 2025) argues that NSA's kernel loop order is efficient only when each GQA group has many query heads, which limits its use on popular LLMs with smaller groups. [14]

How does NSA compare to other sparse-attention methods?

NSA builds on the lineage of FlashAttention: it does not change attention's mathematics so much as supply blockwise, SRAM-tiling kernels that make its specific sparse pattern fast, and its full-attention baseline is itself a FlashAttention-2 implementation. Where FlashAttention computes dense attention exactly but efficiently, NSA computes a learned sparse subset efficiently. [1][6]

NSA appeared two days before Mixture of Block Attention (MoBA), posted to arXiv on February 18, 2025 by researchers at Moonshot AI, which says MoBA was already deployed to support Kimi's long-context requests. MoBA also makes block-sparse attention trainable, but it does so by applying the mixture-of-experts routing idea directly to attention: the context is partitioned into blocks and each query is routed to its top-k blocks according to the inner product between the query and the mean-pooled keys of each block, with no learned gating parameters. Compared with MoBA's single routing mechanism, NSA adds the explicit compression and sliding-window branches and the three-way gate, and builds its selection kernel around GQA groups. Both works demonstrated that sparsity learned during training can match full attention, a convergence from two laboratories in the same week. [1][4]

DeepSeek Sparse Attention (DSA) is the sparse mechanism DeepSeek shipped in its experimental DeepSeek-V3.2-Exp model on September 29, 2025, and carried into DeepSeek-V3.2. DSA uses a lightweight "lightning indexer" to score the relevance of each preceding token to the current query, and a fine-grained selector then keeps only the top-k tokens for attention; it is instantiated on top of multi-head latent attention. DeepSeek's technical reports cite the NSA paper once, to support the point that at the kernel level each key-value entry must be shared across multiple queries for efficiency, which is why DSA uses the MQA mode of MLA. The reports do not otherwise describe how DSA relates to NSA. The two differ in granularity: NSA selects contiguous blocks and adds compression and window branches, while DSA selects individual tokens with a separate indexer, and DeepSeek's release README says DSA "achieves fine-grained sparse attention for the first time." With the V3.2-Exp release DeepSeek also announced API price cuts of more than 50 percent. [5][8][9][10]

In June 2026, MiniMax published MiniMax Sparse Attention (MSA), the mechanism behind MiniMax M3. The MSA paper describes natively trained sparse-attention designs, NSA among them, as its closest prior work and distinguishes MSA by combining per-GQA-group top-k sharing with block-level selection through a lightweight index branch trained with a KL alignment loss. [19] Together, NSA, MoBA, DSA and their successors mark a shift away from inference-only sparse attention toward sparsity that is trained into the model.

Open implementations and follow-up work

Community implementations appeared within days of the preprint; the InfLLM-V2 authors later noted that the NSA code had not been published and used one of them. GitHub figures below were retrieved from the GitHub API on October 31, 2026. [11][12][13][14][15]

RepositoryCreatedDescriptionLicenseStars (Oct 31, 2026)
lucidrains/native-sparse-attention-pytorchFebruary 19, 2025PyTorch implementation of the NSA sparse attention patternMIT816
fla-org/native-sparse-attentionFebruary 21, 2025Triton kernels; added an online top-k selection kernel (February 25, 2025) and full training support with the flame framework (March 15, 2025)MIT1,026
XunhaoLai/native-sparse-attention-tritonFebruary 24, 2025Triton implementation; used for the NSA baseline in the InfLLM-V2 paperApache-2.0285
Relaxed-System-Lab/Flash-Sparse-AttentionAugust 17, 2025Alternative NSA kernels from the FSA paperApache-2.0590

Expanded article table

Selected follow-up research, by date of first arXiv posting:

DateWorkRelationship to NSA
March 12, 2025TabNSA (Eslamian and Cheng)Combines NSA with a TabMixer backbone for tabular data learning. [20]
June 9, 2025MiniCPM4 technical report (OpenBMB)Introduces InfLLM v2 trainable sparse attention and positions it against NSA and MoBA; claims its top-k block selection cuts computation by 60 percent compared with NSA. [16]
June 14, 2025BSA: Ball Sparse Attention (Brita et al.)Adapts NSA to unordered point sets using the Ball Tree structure of the Erwin Transformer. [21]
August 14, 2025Erwin NSA (Lapautre et al.)Combines NSA with the Erwin architecture for physical-science point clouds (cosmology, molecular dynamics, air pressure). [22]
August 25, 2025FSA: Flash Sparse Attention (Yan et al.)Alternative NSA kernel for small GQA groups; reports up to 3.5x (average 1.6x) kernel-level latency reduction, up to 1.25x end-to-end training speedup and up to 1.36x prefill speedup over the vanilla NSA kernel. [14]
September 29, 2025InfLLM-V2 (Zhao et al.)Dense-sparse switchable attention that reuses dense attention parameters instead of NSA's extra key-value sets; follows NSA's block settings (compression block 32, stride 16, block 64) and GQA group size of 16; used to train the open-sourced MiniCPM4.1-8B. [15]
October 2, 2025VideoNSA (Song et al., ICLR 2026 per the arXiv listing)Adapts Qwen2.5-VL with NSA on video tokens and dense attention on text, trained end to end on a 216K video instruction dataset; reports scaling to 128K tokens. [17]
November 2, 2025Hu et al.Alternates local and global (compression and selection) attention across layers and adds latent attention to NSA's branches; reports a 50 percent KV-cache memory reduction versus NSA, with experiments on 340M to 1.3B models. [18]
June 11, 2026MiniMax Sparse Attention (Lai et al.)Cites NSA among the closest prior work; per-GQA-group block selection with a separate index branch. [19]

Expanded article table

Continue reading

More to read

Topics:Deep LearningLarge Language ModelsModel ArchitectureNeural Networks

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40Yuan, Jingyang; Gao, Huazuo; Dai, Damai; Luo, Junyu; Zhao, Liang; Zhang, Zhengyan; Xie, Zhenda; Wei, Y. X.; Wang, Lean; Xiao, Zhiping; Wang, Yuqing; Ruan, Chong; Zhang, Ming; Liang, Wenfeng; Zeng, Wangding. "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention." arXiv:2502.11089, February 16, 2025. arxiv.org/...2502.11089
  2. ^1 ^2 ^3 ^4Yuan, Jingyang, et al. "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), Volume 1: Long Papers, pages 23078 to 23097, Vienna, Austria, July 2025. aclanthology.org/2025.acl-long.1126
  3. ^Wang, Hannah. "DeepSeek founder shares best paper award at top global AI research conference." South China Morning Post, July 31, 2025. scmp.com/...ward-top-global-ai-research-conference
  4. ^Lu, Enzhe, et al. "MoBA: Mixture of Block Attention for Long-Context LLMs." arXiv:2502.13189, February 18, 2025. arxiv.org/...2502.13189
  5. ^DeepSeek-AI. "DeepSeek-V3.2-Exp" (model release introducing DeepSeek Sparse Attention). September 29, 2025. github.com/...DeepSeek-V3.2-Exp
  6. ^Dao, Tri; Fu, Daniel Y.; Ermon, Stefano; Rudra, Atri; Re, Christopher. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." arXiv:2205.14135, May 2022. arxiv.org/...2205.14135
  7. ^1 ^2ACL 2025. "Awards" (Best Paper list). 63rd Annual Meeting of the Association for Computational Linguistics. 2025.aclweb.org/...awards
  8. ^DeepSeek-AI. "DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention." Technical report, September 2025. github.com/...DeepSeek_V3_2.pdf
  9. ^DeepSeek-AI. "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models." arXiv:2512.02556, December 2, 2025. arxiv.org/...2512.02556
  10. ^DeepSeek. "Introducing DeepSeek-V3.2-Exp." DeepSeek API Docs news, September 29, 2025. api-docs.deepseek.com/...news250929
  11. ^fla-org. "native-sparse-attention" (Efficient Triton implementations for Native Sparse Attention). GitHub repository, created February 21, 2025; accessed October 31, 2026. github.com/...native-sparse-attention
  12. ^Wang, Phil (lucidrains). "native-sparse-attention-pytorch." GitHub repository, created February 19, 2025; accessed October 31, 2026. github.com/...native-sparse-attention-pytorch
  13. ^Lai, Xunhao. "native-sparse-attention-triton." GitHub repository, created February 24, 2025; accessed October 31, 2026. github.com/...native-sparse-attention-triton
  14. ^1 ^2 ^3 ^4Yan, Ran; Jiang, Youhe; Chen, Zhuoming; Mai, Haohui; Chen, Beidi; Yuan, Binhang. "FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel." arXiv:2508.18224, August 25, 2025. arxiv.org/...2508.18224 (code: github.com/...Flash-Sparse-Attention)
  15. ^1 ^2 ^3 ^4Zhao, Weilin, et al. "InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation." arXiv:2509.24663, September 29, 2025. arxiv.org/...2509.24663
  16. ^1 ^2MiniCPM Team. "MiniCPM4: Ultra-Efficient LLMs on End Devices." arXiv:2506.07900, June 9, 2025. arxiv.org/...2506.07900
  17. ^Song, Enxin, et al. "VideoNSA: Native Sparse Attention Scales Video Understanding." arXiv:2510.02295, October 2, 2025. arxiv.org/...2510.02295
  18. ^Hu, Yuxuan, et al. "Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies." arXiv:2511.00819, November 2, 2025. arxiv.org/...2511.00819
  19. ^1 ^2Lai, Xunhao, et al. "MiniMax Sparse Attention." arXiv:2606.13392, June 11, 2026. arxiv.org/...2606.13392
  20. ^Eslamian, Ali; Cheng, Qiang. "TabNSA: Native Sparse Attention for Efficient Tabular Data Learning." arXiv:2503.09850, March 12, 2025. arxiv.org/...2503.09850
  21. ^Brita, Catalin E., et al. "BSA: Ball Sparse Attention for Large-scale Geometries." arXiv:2506.12541, June 14, 2025. arxiv.org/...2506.12541
  22. ^Lapautre, Nicolas, et al. "Natively Trainable Sparse Attention for Hierarchical Point Cloud Datasets." arXiv:2508.10758, August 14, 2025. arxiv.org/...2508.10758
  23. ^DeepSeek-AI. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." arXiv:2405.04434, May 2024. arxiv.org/...2405.04434

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

5 revisions · v5 · 4,960 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independently verified 2026-10-31 against the NSA paper (arXiv 2502.11089 v1/v2 and the ACL 2025 Anthology version), ACL awards page, DeepSeek V3.2 reports, MoBA paper and GitHub API; 4 verifier defects fixed before stamping.

Cite this page: AI Wiki. "Native Sparse Attention (NSA)." aiwiki.ai, updated 31 Oct 2026, fact-checked 31 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/native_sparse_attention

Edit article