# Native Sparse Attention (NSA)

> Source: https://aiwiki.ai/wiki/native_sparse_attention
> Updated: 2026-10-31
> Fact-checked: 2026-10-31
> Categories: Deep Learning, Large Language Models, Model Architecture, Neural Networks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Native Sparse Attention (NSA)." aiwiki.ai, 31 Oct 2026. https://aiwiki.ai/wiki/native_sparse_attention
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

Native [Sparse Attention](https://aiwiki.ai/wiki/sparse_attention) (NSA) is a hardware-aligned, natively trainable sparse [attention](https://aiwiki.ai/wiki/attention) mechanism introduced in February 2025 by researchers at [DeepSeek](https://aiwiki.ai/wiki/deepseek), [Peking University](https://aiwiki.ai/wiki/peking_university) and the University of Washington. It is designed to make [long-context](https://aiwiki.ai/wiki/context_window) [language model](https://aiwiki.ai/wiki/large_language_model) training and inference cheaper while keeping the accuracy of dense (full) attention. Instead of letting every query attend to every earlier token, NSA runs three parallel branches over the key-value sequence: a coarse-grained compression branch, a fine-grained block-selection branch, and a local sliding window, blended by a learned gate. In kernel benchmarks on an 8-GPU [NVIDIA A100](https://aiwiki.ai/wiki/nvidia_a100) system at a 64k-token (65,536-token) context, the authors report up to 9.0 times faster forward propagation and 6.0 times faster backward propagation than a Triton implementation of [FlashAttention](https://aiwiki.ai/wiki/flash_attention)-2, and an expected decoding speedup of up to 11.6 times, estimated from the reduction in key-value cache reads. A 27-billion-parameter model pretrained with NSA matched or exceeded an identically trained full-attention model on the authors' general, long-context and reasoning evaluations. The method is described in the paper "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention" by Jingyang Yuan and 14 co-authors, which was one of four Best Paper award winners at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025) in Vienna. [1][2][3][7]

| At a glance | |
| --- | --- |
| Paper | "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention" [1] |
| Authors | Jingyang Yuan (first author, internship at DeepSeek-AI), Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, [Liang Wenfeng](https://aiwiki.ai/wiki/liang_wenfeng), Wangding Zeng [1] |
| Affiliations | DeepSeek-AI; Peking University (School of Computer Science, PKU-Anker LLM Lab); University of Washington [1] |
| Preprint | arXiv:2502.11089, v1 February 16, 2025; v2 February 27, 2025 [1] |
| Published version | ACL 2025 (Volume 1: Long Papers), pages 23078-23097, DOI 10.18653/v1/2025.acl-long.1126 [2] |
| Award | ACL 2025 Best Paper (one of four) [7] |
| Branches | compression (block 32, stride 16), selection (block 64, top 16 blocks), sliding window (512 tokens) [1] |
| Test model | 27B-total, 3B-active GQA + MoE transformer, 30 layers [1] |
| Kernels | Triton; compression and window branches use FlashAttention-2-compatible kernels, selection uses a custom group-centric kernel [1] |

## What is Native Sparse Attention?

The core problem NSA targets is that the [self-attention](https://aiwiki.ai/wiki/self_attention) operation in a [transformer](https://aiwiki.ai/wiki/transformers) has computational cost that grows quadratically, O(L^2), with sequence length L. For modern context windows of tens or hundreds of thousands of tokens, attention comes to dominate both the compute of the forward and backward passes during training and the memory traffic of the key-value (KV) cache during decoding. Sparse attention, which computes only a subset of the query-key interactions, is the obvious remedy, but two practical obstacles had limited earlier methods. First, many sparse schemes are applied only at inference time on top of a model that was pretrained with full attention, so the model never learns to operate with the [sparsity](https://aiwiki.ai/wiki/sparsity) and a train-inference mismatch degrades quality. Second, theoretical reductions in floating point operations frequently fail to produce real wall-clock speedups because the sparse access pattern is irregular and does not map well to GPU memory hierarchies or to optimized kernels such as FlashAttention. [1]

NSA's contribution is to address both obstacles at once through two stated design principles: it is hardware-aligned, meaning its sparse pattern is organized into contiguous blocks and its kernels are written to keep the GPU's arithmetic units busy and to cooperate with [grouped-query attention](https://aiwiki.ai/wiki/grouped_query_attention); and it is natively trainable, meaning the sparsity is present from the start of pretraining rather than bolted on afterward. The paper's abstract states that NSA "employs a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection to preserve both global context awareness and local precision." [1] The authors report that an NSA model pretrained end-to-end matches or exceeds an equivalent full-attention model on general, long-context, and reasoning benchmarks, while delivering large speedups on 64k-token sequences. [1]

## Why is long-context attention so expensive?

In a standard attention layer, every query token attends to every preceding key token, so producing the output for a sequence of length L requires on the order of L^2 query-key dot products and the same number of weighted value sums. Doubling the context roughly quadruples the attention work. During autoregressive decoding the bottleneck shifts from arithmetic to memory: each newly generated token must read the entire stored [KV cache](https://aiwiki.ai/wiki/kv_cache), so the per-token cost grows linearly with the context already produced and the KV cache becomes a dominant consumer of memory bandwidth. The NSA paper cites a theoretical estimate that softmax attention accounts for 70 to 80 percent of total latency when decoding 64k-length contexts. [1]

The paper frames this with arithmetic intensity, the ratio of compute operations to memory accesses. Training and prefilling run batched matrix multiplications with high arithmetic intensity and are compute-bound, while decoding generates one token per forward pass but must load the whole KV cache, so it is memory-bandwidth-bound. A useful sparse method therefore has two different jobs: cut computation during training and prefilling, and cut memory reads during decoding. [1]

A long line of work has tried to make attention cheaper. The NSA paper groups the closest prior methods into fixed sparse patterns (sliding windows, attention sinks as in [StreamingLLM](https://aiwiki.ai/wiki/streaming_llm), and the local-plus-global layout of [Longformer](https://aiwiki.ai/wiki/longformer)), dynamic KV-cache pruning (H2O, SnapKV and related eviction policies), and query-aware selection (Quest, InfLLM, HashAttention, ClusterKV, MInference, SeerAttention). DeepSeek's own [multi-head latent attention](https://aiwiki.ai/wiki/multi-head_latent_attention), introduced with DeepSeek-V2, attacks the memory side differently by compressing the KV cache into a latent vector. [1][23] The NSA authors argue that the earlier sparse methods share weaknesses they set out to avoid. Some sparsify only one phase: H2O applies sparsity during decoding but needs expensive preprocessing at prefill, while MInference accelerates only prefill. Some clash with grouped-query and multi-query attention: in Quest each head selects its own KV subset, so in a GQA model the memory traffic is the union of all heads' selections in a group. And some are hard to train: k-means clustering (ClusterKV) or SimHash selection (MagicPIG) block gradients, while token-granular selection (HashAttention) scatters memory reads and prevents FlashAttention-style blockwise kernels. [1]

## How does NSA work?

NSA replaces the full key and value history of each query q_t with a compact, query-dependent set of remapped keys and values. In the paper's notation, several mapping strategies c produce key-value sets K~_t^c and V~_t^c, each is attended separately, and the outputs are combined as o*_t = sum over c of g_t^c * Attn(q_t, K~_t^c, V~_t^c), where c ranges over C = {cmp, slc, win} (compression, selection, sliding window) and g_t^c in [0, 1] is a gate score. The total number of remapped keys, N_t, is kept much smaller than the context length t, which is where the sparsity comes from. [1]

### Token compression

The compression branch gives the query a cheap, global view of the whole history. Consecutive keys, and separately values, are grouped into blocks of length l taken every d tokens, and each block is mapped to a single compressed key or value by a small learnable [multilayer perceptron](https://aiwiki.ai/wiki/multilayer_perceptron) phi that includes an intra-block position encoding. The paper uses a stride smaller than the block length (d < l) so that blocks overlap, which it says mitigates information fragmentation; the configuration in the experiments is l = 32 and d = 16. Because a long sequence collapses to a much shorter sequence of compressed tokens, attending to them is inexpensive, and the resulting attention scores double as a coarse map of which regions of the context matter most. [1]

### Token selection

Coarse compression alone loses fine detail, so the selection branch restores it by attending at full resolution to a small number of important blocks. NSA reuses the softmax attention scores already computed between the query and the compressed keys as importance scores, so selection adds little extra computation. When the selection blocks (size l') are larger than the compression blocks, the scores of the compression blocks that overlap each selection block are summed to give that block's score. The top-n blocks by score are then kept and their original, uncompressed keys and values are concatenated for attention. In the experiments l' = 64 and n = 16, and the 16 blocks include one initial block and two local blocks that are always active. [1]

Two design choices make this branch hardware-friendly. Selecting whole contiguous blocks rather than scattered individual tokens produces coalesced memory reads and suits [Tensor Core](https://aiwiki.ai/wiki/tensor_core) matrix units; the authors add that attention scores tend to be spatially continuous, so neighboring keys usually have similar importance, an observation they also report in visualizations of their full-attention model. And when the model uses grouped-query or multi-query attention, the importance scores of all query heads in a group are summed so the heads share one set of selected blocks, which means the KV cache is loaded once per group rather than once per head during decoding. [1]

### Sliding window

The third branch is a plain local window over the most recent w tokens, set to w = 512 in the paper. The authors explain that local patterns adapt faster and can dominate learning, which could stop the model from learning to use the compression and selection tokens. Giving local context its own branch lets the other two branches specialize in long-range information without being shortcut by local patterns. To further prevent shortcut learning, NSA gives each of the three branches its own independent keys and values, which the paper says adds only marginal computation. [1]

### Gated combination

The outputs of the three branches are merged into the layer output by a gate. For each branch, an MLP with a sigmoid activation computes a gate score between 0 and 1 from the input features, and the final output is the gated sum of the three branch attentions. [1]

| NSA component | Role | Paper setting |
| --- | --- | --- |
| Compression block / stride | coarse global summary | length l = 32, stride d = 16 |
| Selection block / count | fine-grained important regions | block l' = 64, keep top n = 16 (including 1 initial and 2 local blocks) |
| Sliding window | local recent context | w = 512 tokens |
| Gate | combine the three branches | MLP plus sigmoid, one score per branch |
| Keys and values | per-branch | independent sets for compression, selection and window |

During decoding, NSA needs to load at most about (s - l)/d compressed tokens, n x l' selected tokens and w window tokens, where s is the cached sequence length. With the settings above that is 2,048 tokens at an 8k context and 5,632 tokens at a 64k context, against 8,192 and 65,536 tokens for full attention: [1]

| Context length | Full attention (tokens loaded per step) | NSA (tokens loaded per step) | Expected decoding speedup |
| --- | --- | --- | --- |
| 8,192 | 8,192 | 2,048 | 4x |
| 16,384 | 16,384 | 2,560 | 6.4x |
| 32,768 | 32,768 | 3,584 | 9.1x |
| 65,536 | 65,536 | 5,632 | 11.6x |

The paper presents these figures as memory-access volume per attention operation, measured in equivalent tokens, and notes that because decoding is memory-bound the expected speedup is approximately linear in that volume. [1]

## What makes NSA hardware-aligned and natively trainable?

The hardware-aligned principle shows up in NSA's custom kernels, written in Triton and built on the same IO-aware, blockwise philosophy as FlashAttention. The paper targets architectures with shared KV caches (GQA and MQA) because plain multi-head attention is memory-intensive and inefficient for decoding. The compression and sliding-window branches are compatible with existing FlashAttention-2 kernels, so the new work is a kernel for the selection branch. FlashAttention loads contiguous blocks of queries into on-chip SRAM, but in NSA neighboring queries may need different KV blocks, so that strategy would waste memory traffic. NSA instead groups queries by GQA group: [1]

1. **Group-centric data loading.** For each query position t, the kernel loads the queries of all heads in one GQA group together with their shared list of selected block indices.
2. **Shared KV fetching.** In the inner loop, it loads the selected contiguous key and value blocks into SRAM one after another, with a kernel block size that divides the selection block size.
3. **Outer loop on grid.** Because the inner-loop length (proportional to n) is nearly the same for every query block, the query and output loops are handed to Triton's grid scheduler.

The authors say this achieves near-optimal arithmetic intensity by removing redundant KV transfers through group-wise sharing and by balancing work across the GPU's streaming multiprocessors. [1] The design relies on having many query heads per GQA group: the paper's model uses 4 groups of 16 heads. A later kernel paper, Flash Sparse Attention, argues that this loop order is only efficient with a relatively large number of query heads per group, whereas many existing LLMs use far fewer (see the follow-up section below). [1][14]

The natively trainable principle is the conceptual heart of the paper. Rather than train a dense model and then discard attention at inference, NSA is part of the architecture during pretraining, so the model adapts its representations to the sparse pattern and there is no train-test gap to pay at deployment. The abstract states that this reduces "pretraining computation without sacrificing model performance." [1] The top-n block choice itself is a discrete step, and the paper does not train a separate scorer for it: the block scores are by-products of the compression branch's attention, so no auxiliary loss is needed. The authors contrast this with alternatives they tried and abandoned, described below. [1]

## How was NSA trained and evaluated?

The authors pretrained an NSA model and an otherwise identical full-attention baseline. The backbone follows what the paper calls common practice in state-of-the-art LLMs, combining grouped-query attention with a [mixture-of-experts](https://aiwiki.ai/wiki/mixture_of_experts) design in the DeepSeekMoE style: [1]

| Setting | Value |
| --- | --- |
| Total / active parameters | 27B / 3B |
| Layers / hidden size | 30 / 2,560 |
| Attention | GQA, 4 groups, 64 heads in total; d_q = d_k = 192, d_v = 128 |
| MoE | 72 routed experts plus 2 shared experts, top-6 routing; the first layer uses a SwiGLU MLP instead of MoE |
| Pretraining | 270B tokens of 8k-length text |
| Long-context adaptation | continued training and supervised fine-tuning on 32k-length text with [YaRN](https://aiwiki.ai/wiki/yarn) |
| Speed benchmarks | 8-GPU A100 system, Triton kernels for both NSA and the FlashAttention-2 baseline |

Both models were trained to full convergence, and the paper's pretraining loss curve shows the NSA model with consistently lower loss than the full-attention model. The token count is given as 270 billion in the experimental-setup section and as 260 billion in the introduction, in both the arXiv and ACL versions. [1][2]

## How accurate and how fast is NSA?

### General benchmarks

On nine knowledge, reasoning and coding benchmarks, the authors report that NSA outperformed the full-attention model on 7 of 9 metrics and on average (0.456 against 0.443). The largest gains were on DROP (+0.042) and GSM8K (+0.034); full attention scored higher on MMLU and MBPP. The authors note that most samples in these benchmarks fit inside the local context window of the inference-time sparse baselines, which makes those baselines effectively equivalent to full attention, so they compared NSA only against full attention here. [1]

| Benchmark (setting) | Full attention | NSA |
| --- | --- | --- |
| MMLU (5-shot acc.) | 0.567 | 0.565 |
| MMLU-Pro (5-shot acc.) | 0.279 | 0.286 |
| CMMLU (5-shot acc.) | 0.576 | 0.587 |
| BBH (3-shot acc.) | 0.497 | 0.521 |
| GSM8K (8-shot acc.) | 0.486 | 0.520 |
| MATH (4-shot acc.) | 0.263 | 0.264 |
| DROP (1-shot F1) | 0.503 | 0.545 |
| MBPP (3-shot pass@1) | 0.482 | 0.466 |
| HumanEval (0-shot pass@1) | 0.335 | 0.348 |
| Average | 0.443 | 0.456 |

### Long-context benchmarks

On a 64k-context [needle-in-a-haystack](https://aiwiki.ai/wiki/needle_in_a_haystack) test, the authors report perfect retrieval accuracy at every depth. On [LongBench](https://aiwiki.ai/wiki/longbench), they compared NSA with full attention and with four inference-time sparse methods (H2O, InfLLM, Quest, and Exact-Top, which computes full attention scores and keeps each query's top-n keys). For a fair comparison every sparse baseline was given a budget of 2,560 activated tokens per query, the average NSA activates at a 32k sequence length, including the leading 128 tokens and 512 local tokens; some LongBench subsets on which all models scored poorly were excluded. [1]

| Method | LongBench average (selected subsets) |
| --- | --- |
| H2O | 0.303 |
| InfLLM | 0.383 |
| Quest | 0.392 |
| Exact-Top | 0.423 |
| Full attention | 0.437 |
| NSA | 0.469 |

NSA's largest margins over full attention were on the multi-hop QA tasks HotpotQA (+0.087) and 2WikiMultihopQA (+0.051), on the LCC code task (+0.069) and on English passage retrieval (+0.075). It was below full attention on three subsets: English MultiFieldQA (0.503 against 0.512), GovReport summarization (0.307 against 0.324) and Chinese passage retrieval (0.550 against 0.560). [1]

### Chain-of-thought reasoning

To test whether the sparse model could learn long [chain-of-thought](https://aiwiki.ai/wiki/chain_of_thought) reasoning, the authors used [knowledge distillation](https://aiwiki.ai/wiki/knowledge_distillation) from [DeepSeek-R1](https://aiwiki.ai/wiki/deepseek_r1) rather than [reinforcement learning](https://aiwiki.ai/wiki/reinforcement_learning), which they describe as less effective on smaller models: both models were fine-tuned on 10 billion tokens of 32k-length mathematical reasoning traces. The resulting models were scored on [AIME](https://aiwiki.ai/wiki/aime) 2024, averaging 16 samples per question at temperature 0.7 and top-p 0.95. Only full attention was used as a baseline because the inference-time sparse methods do not support training. [1]

| Generation token limit | Full Attention-R | NSA-R |
| --- | --- | --- |
| 8,192 | 0.046 | 0.121 |
| 16,384 | 0.092 | 0.146 |

As the abstract summarizes, "the model pretrained with NSA maintains or exceeds Full Attention models across general benchmarks, long-context tasks, and instruction-based reasoning." [1]

### Kernel speed

The speed comparison pits Triton-based NSA kernels against a Triton-based FlashAttention-2 kernel on the same backend. The speedup grows with context length: [1]

| Context length | Forward speedup | Backward speedup |
| --- | --- | --- |
| 8k | 2.1x | 1.1x |
| 16k | 3.8x | 2.0x |
| 32k | 6.3x | 3.4x |
| 64k | 9.0x | 6.0x |

The authors attribute the gain to blockwise memory access that maximizes Tensor Core utilization through coalesced loads and to loop scheduling that eliminates redundant KV transfers. For decoding, the headline 11.6x at 64k is the expected speedup derived from KV-cache memory access in the table above, not a separate wall-clock benchmark. [1]

## Design alternatives the authors rejected

Section 6 of the arXiv preprint (moved to an appendix in the ACL version) describes approaches the team tried before settling on NSA. Key-clustering strategies such as ClusterKV were rejected because of the overhead of dynamic clustering, load imbalances between clusters (worse in mixture-of-experts systems with expert parallelism), and the need for periodic reclustering. Two other blockwise selection strategies were tested on a 3B-parameter model with a similar architecture: an auxiliary-loss-based learnable selector (conceptually similar to SeerAttention, supervised with KL divergence against mean-pooled attention scores) and a heuristic, parameter-free selector following Quest, including a variant that used full attention for the first 1,000 steps before switching. The paper reports that both had worse training loss than NSA. A visualization of the attention map of the authors' 27B full-attention model, showing blockwise clustering of attention scores, is presented as the inspiration for block-based selection. [1]

## Limitations and critiques

The ACL version of the paper adds a limitations section: the work considers sparsity within the attention map of a single layer, leaving cross-layer sparsity to future work, and the Triton implementation carries some abstraction overhead compared with native CUDA kernels. [2] As of the InfLLM-V2 paper (September 2025), the NSA authors had not published their code, so that team used a community Triton implementation for its NSA experiments. [15]

Later papers have raised several criticisms:

- **Extra parameters and short-context cost.** The MiniCPM4 technical report (June 2025) argues that NSA's three attention components add parameters, raising computation on short sequences and tripling key-value storage during pretraining. [16]
- **Short-to-long adaptation.** The InfLLM-V2 authors report that when they converted a model pretrained with dense attention on short sequences into an NSA model for long-context fine-tuning, NSA disrupted the training loss and gave unsatisfactory results, although they note that the original paper trained NSA from scratch. [15]
- **Kernel generality.** Flash Sparse Attention (August 2025) argues that NSA's kernel loop order is efficient only when each GQA group has many query heads, which limits its use on popular LLMs with smaller groups. [14]

## How does NSA compare to other sparse-attention methods?

NSA builds on the lineage of FlashAttention: it does not change attention's mathematics so much as supply blockwise, SRAM-tiling kernels that make its specific sparse pattern fast, and its full-attention baseline is itself a FlashAttention-2 implementation. Where FlashAttention computes dense attention exactly but efficiently, NSA computes a learned sparse subset efficiently. [1][6]

NSA appeared two days before [Mixture of Block Attention](https://aiwiki.ai/wiki/mixture_of_block_attention) (MoBA), posted to arXiv on February 18, 2025 by researchers at [Moonshot AI](https://aiwiki.ai/wiki/moonshot_ai), which says MoBA was already deployed to support [Kimi](https://aiwiki.ai/wiki/kimi)'s long-context requests. MoBA also makes block-sparse attention trainable, but it does so by applying the mixture-of-experts routing idea directly to attention: the context is partitioned into blocks and each query is routed to its top-k blocks according to the inner product between the query and the mean-pooled keys of each block, with no learned gating parameters. Compared with MoBA's single routing mechanism, NSA adds the explicit compression and sliding-window branches and the three-way gate, and builds its selection kernel around GQA groups. Both works demonstrated that sparsity learned during training can match full attention, a convergence from two laboratories in the same week. [1][4]

[DeepSeek Sparse Attention (DSA)](https://aiwiki.ai/wiki/deepseek_sparse_attention) is the sparse mechanism DeepSeek shipped in its experimental DeepSeek-V3.2-Exp model on September 29, 2025, and carried into [DeepSeek-V3.2](https://aiwiki.ai/wiki/deepseek_v3_2). DSA uses a lightweight "lightning indexer" to score the relevance of each preceding token to the current query, and a fine-grained selector then keeps only the top-k tokens for attention; it is instantiated on top of multi-head latent attention. DeepSeek's technical reports cite the NSA paper once, to support the point that at the kernel level each key-value entry must be shared across multiple queries for efficiency, which is why DSA uses the MQA mode of MLA. The reports do not otherwise describe how DSA relates to NSA. The two differ in granularity: NSA selects contiguous blocks and adds compression and window branches, while DSA selects individual tokens with a separate indexer, and DeepSeek's release README says DSA "achieves fine-grained sparse attention for the first time." With the V3.2-Exp release DeepSeek also announced API price cuts of more than 50 percent. [5][8][9][10]

In June 2026, [MiniMax](https://aiwiki.ai/wiki/minimax) published MiniMax Sparse Attention (MSA), the mechanism behind [MiniMax M3](https://aiwiki.ai/wiki/minimax_m3). The MSA paper describes natively trained sparse-attention designs, NSA among them, as its closest prior work and distinguishes MSA by combining per-GQA-group top-k sharing with block-level selection through a lightweight index branch trained with a KL alignment loss. [19] Together, NSA, MoBA, DSA and their successors mark a shift away from inference-only sparse attention toward sparsity that is trained into the model.

## Open implementations and follow-up work

Community implementations appeared within days of the preprint; the InfLLM-V2 authors later noted that the NSA code had not been published and used one of them. GitHub figures below were retrieved from the GitHub API on October 31, 2026. [11][12][13][14][15]

| Repository | Created | Description | License | Stars (Oct 31, 2026) |
| --- | --- | --- | --- | --- |
| lucidrains/native-sparse-attention-pytorch | February 19, 2025 | PyTorch implementation of the NSA sparse attention pattern | MIT | 816 |
| fla-org/native-sparse-attention | February 21, 2025 | Triton kernels; added an online top-k selection kernel (February 25, 2025) and full training support with the flame framework (March 15, 2025) | MIT | 1,026 |
| XunhaoLai/native-sparse-attention-triton | February 24, 2025 | Triton implementation; used for the NSA baseline in the InfLLM-V2 paper | Apache-2.0 | 285 |
| Relaxed-System-Lab/Flash-Sparse-Attention | August 17, 2025 | Alternative NSA kernels from the FSA paper | Apache-2.0 | 590 |

Selected follow-up research, by date of first arXiv posting:

| Date | Work | Relationship to NSA |
| --- | --- | --- |
| March 12, 2025 | TabNSA (Eslamian and Cheng) | Combines NSA with a TabMixer backbone for tabular data learning. [20] |
| June 9, 2025 | MiniCPM4 technical report ([OpenBMB](https://aiwiki.ai/wiki/openbmb)) | Introduces InfLLM v2 trainable sparse attention and positions it against NSA and MoBA; claims its top-k block selection cuts computation by 60 percent compared with NSA. [16] |
| June 14, 2025 | BSA: Ball Sparse Attention (Brita et al.) | Adapts NSA to unordered point sets using the Ball Tree structure of the Erwin Transformer. [21] |
| August 14, 2025 | Erwin NSA (Lapautre et al.) | Combines NSA with the Erwin architecture for physical-science point clouds (cosmology, molecular dynamics, air pressure). [22] |
| August 25, 2025 | FSA: Flash Sparse Attention (Yan et al.) | Alternative NSA kernel for small GQA groups; reports up to 3.5x (average 1.6x) kernel-level latency reduction, up to 1.25x end-to-end training speedup and up to 1.36x prefill speedup over the vanilla NSA kernel. [14] |
| September 29, 2025 | InfLLM-V2 (Zhao et al.) | Dense-sparse switchable attention that reuses dense attention parameters instead of NSA's extra key-value sets; follows NSA's block settings (compression block 32, stride 16, block 64) and GQA group size of 16; used to train the open-sourced [MiniCPM](https://aiwiki.ai/wiki/minicpm)4.1-8B. [15] |
| October 2, 2025 | VideoNSA (Song et al., ICLR 2026 per the arXiv listing) | Adapts [Qwen2.5-VL](https://aiwiki.ai/wiki/qwen2_5_vl) with NSA on video tokens and dense attention on text, trained end to end on a 216K video instruction dataset; reports scaling to 128K tokens. [17] |
| November 2, 2025 | Hu et al. | Alternates local and global (compression and selection) attention across layers and adds latent attention to NSA's branches; reports a 50 percent KV-cache memory reduction versus NSA, with experiments on 340M to 1.3B models. [18] |
| June 11, 2026 | MiniMax Sparse Attention (Lai et al.) | Cites NSA among the closest prior work; per-GQA-group block selection with a separate index branch. [19] |

## References

1. Yuan, Jingyang; Gao, Huazuo; Dai, Damai; Luo, Junyu; Zhao, Liang; Zhang, Zhengyan; Xie, Zhenda; Wei, Y. X.; Wang, Lean; Xiao, Zhiping; Wang, Yuqing; Ruan, Chong; Zhang, Ming; Liang, Wenfeng; Zeng, Wangding. "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention." arXiv:2502.11089, February 16, 2025. https://arxiv.org/abs/2502.11089
2. Yuan, Jingyang, et al. "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), Volume 1: Long Papers, pages 23078 to 23097, Vienna, Austria, July 2025. https://aclanthology.org/2025.acl-long.1126/
3. Wang, Hannah. "DeepSeek founder shares best paper award at top global AI research conference." South China Morning Post, July 31, 2025. https://www.scmp.com/tech/big-tech/article/3320255/deepseek-founder-shares-best-paper-award-top-global-ai-research-conference
4. Lu, Enzhe, et al. "MoBA: Mixture of Block Attention for Long-Context LLMs." arXiv:2502.13189, February 18, 2025. https://arxiv.org/abs/2502.13189
5. DeepSeek-AI. "DeepSeek-V3.2-Exp" (model release introducing DeepSeek Sparse Attention). September 29, 2025. https://github.com/deepseek-ai/DeepSeek-V3.2-Exp
6. Dao, Tri; Fu, Daniel Y.; Ermon, Stefano; Rudra, Atri; Re, Christopher. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." arXiv:2205.14135, May 2022. https://arxiv.org/abs/2205.14135
7. ACL 2025. "Awards" (Best Paper list). 63rd Annual Meeting of the Association for Computational Linguistics. https://2025.aclweb.org/program/awards/
8. DeepSeek-AI. "DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention." Technical report, September 2025. https://github.com/deepseek-ai/DeepSeek-V3.2-Exp/blob/main/DeepSeek_V3_2.pdf
9. DeepSeek-AI. "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models." arXiv:2512.02556, December 2, 2025. https://arxiv.org/abs/2512.02556
10. DeepSeek. "Introducing DeepSeek-V3.2-Exp." DeepSeek API Docs news, September 29, 2025. https://api-docs.deepseek.com/news/news250929
11. fla-org. "native-sparse-attention" (Efficient Triton implementations for Native Sparse Attention). GitHub repository, created February 21, 2025; accessed October 31, 2026. https://github.com/fla-org/native-sparse-attention
12. Wang, Phil (lucidrains). "native-sparse-attention-pytorch." GitHub repository, created February 19, 2025; accessed October 31, 2026. https://github.com/lucidrains/native-sparse-attention-pytorch
13. Lai, Xunhao. "native-sparse-attention-triton." GitHub repository, created February 24, 2025; accessed October 31, 2026. https://github.com/XunhaoLai/native-sparse-attention-triton
14. Yan, Ran; Jiang, Youhe; Chen, Zhuoming; Mai, Haohui; Chen, Beidi; Yuan, Binhang. "FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel." arXiv:2508.18224, August 25, 2025. https://arxiv.org/abs/2508.18224 (code: https://github.com/Relaxed-System-Lab/Flash-Sparse-Attention)
15. Zhao, Weilin, et al. "InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation." arXiv:2509.24663, September 29, 2025. https://arxiv.org/abs/2509.24663
16. MiniCPM Team. "MiniCPM4: Ultra-Efficient LLMs on End Devices." arXiv:2506.07900, June 9, 2025. https://arxiv.org/abs/2506.07900
17. Song, Enxin, et al. "VideoNSA: Native Sparse Attention Scales Video Understanding." arXiv:2510.02295, October 2, 2025. https://arxiv.org/abs/2510.02295
18. Hu, Yuxuan, et al. "Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies." arXiv:2511.00819, November 2, 2025. https://arxiv.org/abs/2511.00819
19. Lai, Xunhao, et al. "MiniMax Sparse Attention." arXiv:2606.13392, June 11, 2026. https://arxiv.org/abs/2606.13392
20. Eslamian, Ali; Cheng, Qiang. "TabNSA: Native Sparse Attention for Efficient Tabular Data Learning." arXiv:2503.09850, March 12, 2025. https://arxiv.org/abs/2503.09850
21. Brita, Catalin E., et al. "BSA: Ball Sparse Attention for Large-scale Geometries." arXiv:2506.12541, June 14, 2025. https://arxiv.org/abs/2506.12541
22. Lapautre, Nicolas, et al. "Natively Trainable Sparse Attention for Hierarchical Point Cloud Datasets." arXiv:2508.10758, August 14, 2025. https://arxiv.org/abs/2508.10758
23. DeepSeek-AI. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." arXiv:2405.04434, May 2024. https://arxiv.org/abs/2405.04434

