SparDA

SparDA (Sparse Decoupled Attention) is an add-on architecture for long-context large language model inference proposed by researchers at NVIDIA in a paper posted to arXiv on 3 June 2026. It modifies block-sparse attention by giving every layer a fourth linear projection, called the Forecast, alongside the usual query, key and value. The Forecast computed in layer l chooses which KV cache blocks layer l+1 will attend to, so block selection is decoupled from the attention query and runs one layer ahead. That lookahead lets a serving runtime prefetch the selected KV blocks from CPU memory while the current layer is still executing, and because the Forecast no longer has to mirror the query-head layout it can use one head per grouped-query attention (GQA) group, which makes selection itself cheaper. The authors report that on two sparse-pretrained 8B models, MiniCPM4.1-8B and NOSA-8B, SparDA adds 33.5M parameters (0.41 percent), trains only those projections, matches or slightly improves accuracy, and delivers up to 1.25x prefill and 1.7x decode speedup over a sparse-attention offload baseline, with up to 5.3x higher decode throughput than a non-offload sparse baseline because offloading allows larger batches on a single GPU.^[1]

Authors and publication

The paper, "SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference" (arXiv 2606.04511), lists five authors: Yaosheng Fu (NVIDIA, corresponding author), Guangxuan Xiao (Thinking Machines Lab), Xin Dong (ByteDance Seed), Song Han (NVIDIA and MIT) and Oreste Villa (NVIDIA). Xiao and Dong are marked with the footnote "Work done while working at NVIDIA". The PDF carries a 2 June 2026 date line and an NVIDIA copyright notice; the arXiv submission (version 1, categories cs.CL and cs.LG) was made by Fu on 3 June 2026.^[1] The training and evaluation code is published in the NVlabs/SparDA repository on GitHub under the Apache 2.0 license. The repository was created on 22 May 2026 and last pushed to on 4 June 2026; the README states that trained Forecast checkpoints are not included in the release and that the project is not accepting contributions.^[2]

Background: what sparse attention leaves unsolved

Sparse attention lowers the cost of long-context inference by having each query attend to a subset of the cached keys and values rather than all of them. In block-sparse designs such as InfLLM-V2, the backbone of the MiniCPM4 family, the attended set for a query is the union of a fixed set of initial blocks, a local sliding window and the top-k blocks chosen by a relevance score, so the per-query attention cost falls from O(T) to something proportional to the number of attended blocks.^[1][4] The SparDA paper argues that two costs remain.^[1]

The first is capacity. Even if attention reads only a few blocks per step, the KV cache itself still grows with sequence length. Offloading it to CPU memory and fetching only the selected blocks is the obvious remedy, but the fetch crosses PCI Express, which is far slower than GPU memory, and in standard designs the fetch cannot start until the layer's own query exists, because the query is what drives selection. The transfer therefore sits on the critical path of every layer of every decode step.^[1]

The second is the selection step. Sparse attention brings the attention computation down to O(T), but the block-selection scan that precedes it still scores every candidate block and remains O(T^2) in the sequence length, so at long contexts it can dominate the attention module. In a GQA model the standard InfLLM-V2 selector scores the compressed keys with every query head in a group, applies a softmax per head, sums the scores across the group and max-pools them to block level before taking the top-k.^[1]

The paper frames both problems as consequences of one design choice: the attention query drives selection, so selection is tied to the layer that needs it and inherits the query's multi-head layout.^[1]

How SparDA works

Decoupled selection

SparDA changes the per-layer projection so that it emits four tensors instead of three: (Q_l, K_l, V_l, F_l). The Forecast F_l is scored against the compressed keys of layer l+1, and the resulting top-k blocks are merged with the initial and local blocks to form layer l+1's attended set. Layer l+1's own query Q_l+1 is then used only for the sparse attention computation over that set. Because the sparse selection excludes the initial and local blocks, the block set for layer l+1 does not depend on the keys that layer l+1 will append during the current step, which is what makes the one-layer-ahead prediction well defined during decoding.^[1]

Two special cases are handled explicitly. For the first layer (l = 0), where no earlier Forecast exists, a separate projection produces a current-layer Forecast used for same-layer selection; it still uses one Forecast head per GQA group, so selection is cheaper than the baseline even without lookahead. For the final layer, the Forecast is computed but unused because there is no subsequent layer to predict for. During decoding, the layer-0 KV cache stays on the GPU, so the prefetch path begins at layer 1.^[1]

The paper notes that the benefit differs by phase. During prefill all keys are already on the GPU, so the only saving is the reduced selection cost. During decode with an offloaded KV cache the lookahead matters: F_l produces the block set for layer l+1 while layer l is still executing, and the runtime can overlap the CPU-to-GPU transfer with that execution. The compressed-key cache used for scoring stays on the GPU and is updated incrementally.^[1]

The Forecast indexer

Once selection is decoupled from the query, it no longer has to use the same head layout as attention. SparDA gives the Forecast one head per GQA group (that is, one per KV head) instead of one per query head, which removes the per-query-head scoring loop. It also skips the softmax that block-sparse selectors normally apply before top-k ranking, since there is no longer a need to sum normalized scores across several query heads within a group. The paper credits the lightning indexer in DeepSeek Sparse Attention (DSA) with already exploiting this kind of decoupling at the token level with fewer heads than the attention query, and describes SparDA as bringing the same design to block-sparse attention while operating one layer ahead.^[1][7]

Asynchronous prefetch with a persistent UVA kernel

Predicting the next layer's blocks early is only useful if the transfer can actually overlap with compute. SparDA fetches the selected blocks from pinned CPU memory on a dedicated CUDA stream using a persistent Triton kernel built on Unified Virtual Addressing (UVA). Rather than issuing many small, irregular memory copies, the kernel keeps a small fixed set of thread blocks (Cooperative Thread Arrays, or CTAs) resident and lets them process block-transfer tasks continuously within a single launch, which the authors say reduces launch overhead, avoids frequent synchronization and limits interference with the main compute stream.^[1]

The number of CTAs is chosen adaptively by batch size. More CTAs push transfer throughput toward the PCIe ceiling but occupy streaming multiprocessors that would otherwise run the attention and feed-forward kernels. At small batch sizes the GPU is underutilized and layer execution dominates, so a modest CTA budget suffices; as batch size grows, layer execution time scales sub-linearly while prefetch volume grows linearly, and past a crossover point extra CTAs are worth their cost. On the H100 the heuristic uses 16 CTAs below batch size 32 and 32 CTAs otherwise, which the paper's sweep (Table 7) shows matches or stays within 4 percent of the best fixed setting at every batch size; on the A100 the thresholds are 16 CTAs below batch size 64 and 32 otherwise.^[1]

Training the Forecast

SparDA is added to models that were already pretrained with sparse attention, so only the Forecast projections are trained and the backbone stays frozen. On both MiniCPM4.1-8B and NOSA-8B the projections add 33.5M parameters, which the paper gives as 0.41 percent of the 8B total.^[1]

The training objective follows DSA: the indexer is trained to match the original selector's block-attention distribution with a Kullback-Leibler divergence after top-k restriction and renormalization. For each target layer and GQA group, the target score is the shared importance score computed by the original InfLLM-V2 selector before its final max-pooling stage (max-pooling discards ranking information the indexer needs), while the predicted score comes from the previous layer's Forecast. The loss compares (k+1)-dimensional distributions that keep the k target-selected blocks individually and pool all remaining mass into one "rest" bucket, so out-of-set blocks still receive gradient through that bucket. Because the base models are already sparse-trained, the full-model sparse training stage of DSA's two-stage pipeline is skipped.^[1]

One training-time detail the paper singles out is a deliberate mismatch in compression windows. The predicted scores use the standard InfLLM-V2 mean-pooling window (kernel 32, stride 16) so that they match inference, but the target scores are computed with a much finer window (kernel 2, stride 1) and then max-pooled back down to the (32, 16) grid before the KL loss. The authors' explanation is that finer-grained targets give a higher-resolution supervision signal. Their ablation (Table 6) shows the finer target raising the average over four benchmark families from 60.4 to 61.7 on MiniCPM4.1-8B (RULER +3.0, reasoning +2.2) and from 50.1 to 51.7 on NOSA-8B (RULER +2.5, reasoning +1.8).^[1]

Training used 32 H100 GPUs; the MiniCPM4.1-8B indexer at 64K finished within 48 hours and the NOSA-8B indexer at 32K within 24 hours.^[1] The released code trains NOSA-8B at a 32,768-token sequence length and MiniCPM4.1-8B at 65,536.^[2]

Relation to prior work

The paper positions SparDA relative to three lines of work.^[1]

InfiniGen, published at OSDI 2024, is the method the paper calls closest to SparDA: it prefetches a subset of CPU-resident KV entries before each layer by using the current hidden state as a cross-layer proxy for the next layer's attention, overlapping DMA copies with the preceding layer.^[6] The SparDA authors argue that this proxy is inaccurate when adjacent-layer similarity breaks down, and they use InfiniGen as an accuracy and throughput baseline.^[1] The paper summarizes its own contribution as "bridging DSA's trained decoupled indexer with InfiniGen's lookahead" and co-designing the result with the persistent prefetch kernel.^[1]

Evaluation

Setup

Two 8B models were used. MiniCPM4.1-8B, released by OpenBMB in September 2025, uses InfLLM-V2 as its sparse backbone and natively supports 64K tokens.^[1][5][10] NOSA-8B adds a query-agnostic eviction head on top of InfLLM-V2 and natively supports 32K.^[1][3][11] Both use a block size of 64 tokens, one initial block, a compression kernel of 32 with stride 16, and the same overall top-k budget of 96 blocks. MiniCPM4.1-8B keeps its default local window of 2,048 tokens (32 blocks) with the rest of the budget for query-aware top-k; NOSA-8B uses a 1,024-token window (16 blocks) and allocates 24 blocks to query-aware top-k, following the NOSA paper.^[1]

Accuracy was measured on HELMET (seven categories, 100 samples each, greedy decoding, GPT-5.2 as judge where the original used GPT-4), LongBench (all 14 English tasks), RULER (all 13 synthetic tasks, 50 examples each) and a reasoning suite of MATH-500 (500 problems), AIME 2024 and AIME 2025 (30 problems each), the last judged by GPT-5.2. The reasoning suite used sampling with a long generation budget (up to 64K tokens for MiniCPM4.1-8B with thinking mode on, 8K for NOSA-8B, which has no thinking mode and was given a two-shot chain-of-thought prompt). HELMET, LongBench and RULER were run at each model's native maximum, and RULER was additionally run at 32K, 64K, 96K and 128K. Beyond the native maximum, MiniCPM4.1-8B used the 128K-validated LongRoPE factors shipped with its checkpoint and NOSA-8B had its RoPE base raised from 10,000 to 40,000; the same extension was applied to every configuration.^[1]

Four configurations were compared: Dense, Sparse (the InfLLM-V2 or NOSA sparse path), InfiniGen and SparDA. For efficiency, all methods were implemented on the NOSI inference engine from the NOSA project. A dagger in the throughput tables marks a no-offload configuration in which the whole KV cache stays on the GPU; the undecorated Sparse, InfiniGen and SparDA rows offload the full KV cache to pinned CPU memory, keeping only the compressed keys and the layer-0 cache on the GPU. NOSI caches each layer's previously selected top-k blocks, so a decode step only fetches newly selected blocks. Hardware was an H100 with 80 GB HBM3 and PCIe Gen5 x16, and an A100 with 80 GB HBM2e and PCIe Gen4 x16, each with 2 TB of CPU memory; each throughput number is the mean of two runs after one warm-up. The paper does not compare against IndexCache or HISA directly, since those target token-level DSA rather than the block-sparse backbones used here. InfiniGen, originally token-level, was adapted by max-pooling its approximate scores to block level and feeding them through the same 96-block top-k path.^[1]

Accuracy

Table 1 of the paper gives the aggregate averages.^[1]

On MiniCPM4.1-8B, SparDA is 0.3 points above Sparse on average, with RULER +0.5 and reasoning +1.1, LongBench flat and HELMET 0.6 lower. On NOSA-8B the average gain is +2.3, driven by reasoning (+6.5), RULER (+1.7) and HELMET (+1.2). The authors attribute the sizable gap between Dense and every sparse method on HELMET and RULER to evaluating at the models' maximum lengths (64K and 32K) when their sparse pretraining was done at shorter lengths (32K and 16K respectively). InfiniGen loses substantially on both models, which the paper ascribes to its reliance on hidden-state similarity across adjacent layers.^[1]

The reasoning column deserves care. Each AIME set has only 30 problems and is scored by sampling, so a single problem moves the score by 3.3 points; the per-dataset breakdown (Table 14) shows that NOSA-8B's +6.5 comes mostly from AIME 2025, where SparDA scored 53.3 against 40.0 for Sparse and 13.3 for Dense. On MiniCPM4.1-8B the Sparse configuration itself outscores Dense on AIME 2024 (90.0 versus 80.0), so these numbers measure variability as much as method quality.^[1]

Table 2 reports RULER averages at extended lengths, where SparDA leads Sparse at every length on both models.^[1]

Efficiency

A per-layer attention breakdown on MiniCPM4.1-8B at batch size 4 (Figure 3) shows where the gains come from. In prefill, block-sparse attention dominates and stays roughly constant with length while block selection grows until it is comparable to attention at 128K; SparDA cuts selection cost by up to 2.50x at 128K. In decode, with one query token per step, the attention itself is cheap and selection is the bottleneck; SparDA's indexer keeps decode-time selection nearly flat with length, cutting the overhead by more than 2x at 128K.^[1]

Prefill throughput on the H100 at batch size 4 (Table 3):^[1]

Dense leads at 32K but its quadratic scaling drops it behind as length grows; SparDA has the highest prefill throughput from 64K onward on MiniCPM4.1-8B and from 96K onward on NOSA-8B. At 128K on MiniCPM4.1-8B, SparDA is 1.25x faster than Sparse and 2.11x faster than Dense; on NOSA-8B it is 1.16x over Sparse and 1.40x over Dense. Offloading has almost no effect on prefill (the no-offload and offload Sparse rows are nearly identical) because the only offload-time transfer is an asynchronous writeback of new cache entries.^[1]

Decode is where the lookahead pays off. Table 4 sweeps batch sizes from 4 to 128; the 128K rows on the H100 are reproduced below (tokens per second, "-" is out of memory).^[1]

Within the offload regime at 128K, SparDA reaches 1.69x over Sparse on MiniCPM4.1-8B (at batch size 8, 471.2 versus 279.5 tokens per second) and 1.40x on NOSA-8B (399.3 versus 285.4); the paper's headline "1.7x" is this MiniCPM figure. The largest speedups occur at middle batch sizes (8 to 16), where prefetch and layer execution are roughly balanced. The 5.3x headline compares SparDA's best 128K throughput on MiniCPM4.1-8B (1000.1 tokens per second at batch 64) with the no-offload Sparse configuration, which runs out of memory above batch size 4 at that length (189.5 tokens per second), giving 5.28x; against no-offload Dense (108.6) the ratio is 9.21x. NOSA-8B shows smaller gains because its eviction head already reduces KV fetch traffic, leaving less transfer to overlap. InfiniGen is consistently slower than Sparse because it gathers the selected blocks on the CPU before transferring them, making the CPU-side gather the bottleneck. Because SparDA keeps the layer-0 cache on the GPU it runs out of memory slightly earlier than Sparse at long context (for example at batch 128 at 96K), though its peak throughput still exceeds Sparse's at every length.^[1]

The appendix separates the two sources of decode speedup (Table 10, H100, 128K). A "SparDA (no prefetch)" variant that uses the Forecast indexer but fetches blocks synchronously already beats Sparse at batch size 4 on both models (247.3 versus 167.8 on MiniCPM4.1-8B), and at that small batch is marginally faster than full SparDA because the prefetch pipeline adds slight overhead. By batch 16 full SparDA pulls ahead, and by batch 64 it is roughly 40 percent faster than the no-prefetch variant (1000.1 versus 696.9), while the no-prefetch variant falls below Sparse at batch 64 on both models. The authors read this as confirmation that prefetch overlap, not just cheaper selection, is what carries the advantage at high batch sizes.^[1]

On the A100 (Tables 8 and 9), SparDA reaches 1.23x over Sparse and 1.81x over Dense in prefill on MiniCPM4.1-8B at 128K, 1.16x over Sparse on NOSA-8B, and in decode 1.55x over Sparse on MiniCPM4.1-8B (128K, batch 8) and 1.33x on NOSA-8B (128K, batch 16), slightly below the H100 figures; InfiniGen's CPU-gather bottleneck is more pronounced there, with its throughput falling at larger batches.^[1]

Limitations

The paper's own limitations section is explicit that SparDA is not itself a sparse attention method. It replaces the selection path of an existing block-sparse backbone but changes neither the sparse attention computation nor the underlying sparse pattern, so its accuracy is bounded by the base method; the large Dense-to-Sparse gaps on HELMET and RULER in Table 1 are inherited rather than closed. The lookahead itself only matters when the cache is offloaded, which the paper ties to the decode phase with KV cache offloading; in prefill the only saving is cheaper selection. The authors leave extending the design to token-level DSA (as used in DeepSeek-V3.2 and GLM-5) and to DeepSeek-V4's Compressed Sparse Attention as future work, noting that those models are all significantly larger than the 8B models used here. In such a setting the Forecast would predict per-token rather than per-block scores.^[1]

Several caveats follow from the evaluation design rather than from the method. The throughput comparisons are all made inside the NOSI engine on single GPUs, the "5.3x" and "9.21x" figures compare different batch sizes (the no-offload baselines cannot run beyond batch 4 at 128K), and the reasoning results rest on 30-problem AIME sets scored by an LLM judge. The paper reports one warm-up and two measured runs per throughput setting and does not report variance.^[1]

Reception

Coverage of the paper included a long-form explainer post by the Daily Dose of Data Science account on X on 6 September 2026, which described it as "a recent paper from NVIDIA and MIT" and highlighted the 1.7x decode speedup and, in its headline, an unqualified "6.5 points" reasoning gain; the body of the post and the paper attribute that figure to NOSA-8B only, and the paper's author block lists Thinking Machines Lab and ByteDance Seed alongside NVIDIA and MIT.^[12][1] As of 16 September 2026 the GitHub repository had 72 stars.^[2]

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han, Oreste Villa. "SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference." arXiv:2606.04511, submitted 3 June 2026. arxiv.org/...2606.04511
2. ^1 ^2 ^3NVlabs. "SparDA: Sparse Decoupled Attention" (GitHub repository, README and license). github.com/...SparDA
3. ^Yuxiang Huang et al. "NOSA: Native and Offloadable Sparse Attention." arXiv:2510.13602, 2025. arxiv.org/...2510.13602
4. ^Weilin Zhao et al. "InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation." arXiv:2509.24663, 2025 (ICLR 2026). arxiv.org/...2509.24663
5. ^MiniCPM Team. "MiniCPM4: Ultra-Efficient LLMs on End Devices." arXiv:2506.07900, 2025. arxiv.org/...2506.07900
6. ^Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim. "InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management." USENIX OSDI 2024. usenix.org/...lee
7. ^DeepSeek-AI. "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models." arXiv:2512.02556, 2025. arxiv.org/...2512.02556
8. ^Yushi Bai et al. "IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse." arXiv:2603.12201, 2026. arxiv.org/...2603.12201
9. ^Yufei Xu et al. "HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention." arXiv:2603.28458, 2026. arxiv.org/...2603.28458
10. ^OpenBMB. "openbmb/MiniCPM4.1-8B" (Hugging Face model repository). huggingface.co/...MiniCPM4.1-8B
11. ^OpenBMB. "openbmb/NOSA-8B" (Hugging Face model repository). huggingface.co/...NOSA-8B
12. ^Daily Dose of Data Science (@DailyDoseOfDS_). "NVIDIA researchers built a new transformer variant..." X, 6 September 2026. x.com/...2096531369669394532