Kimi Linear is a hybrid linear attention architecture published by Moonshot AI on October 30, 2025, together with a 48-billion-parameter mixture-of-experts model that activates 3 billion parameters per token.[1][2][3] Its central component is Kimi Delta Attention (KDA), a linear attention mechanism that adds fine-grained, channel-wise gating to the gated delta rule of Gated DeltaNet.[1][9] Kimi Linear interleaves three KDA layers with one full multi-head latent attention layer and uses no positional encoding at all, a combination that later became the attention design of Kimi K3, Moonshot's 2.8-trillion-parameter flagship of July 2026.[19][20]

Field	Value
Developer	Moonshot AI
Released	October 30, 2025 (arXiv report, GitHub repository, and both checkpoints)
Total parameters	48 billion
Activated parameters	3 billion per token
Layers	27 (20 KDA, 7 full MLA)
Attention	KDA interleaved with full MLA at a 3:1 ratio
Positional encoding	None; NoPE in every full-attention layer
Experts	256 routed plus 1 shared, 8 routed experts per token, first layer dense
Hidden size	2,304; vocabulary 163,840
Context window	1,048,576 tokens
Pretraining tokens	5.7 trillion (released checkpoints); 1.4 trillion (matched-comparison runs)
Checkpoints	Kimi-Linear-48B-A3B-Base, Kimi-Linear-48B-A3B-Instruct
License	MIT

Why linear attention exists

Softmax attention, as introduced in Attention Is All You Need, compares every token with every earlier token. Two costs follow. Computing the attention map takes time proportional to the square of the sequence length, and generating text requires keeping a KV cache whose size grows linearly with the number of tokens processed so far. At a million tokens the cache, not the arithmetic, is usually what limits how many requests a server can hold in memory at once.[1]

Linear attention, formulated by Katharopoulos and colleagues at ICML 2020, replaces the softmax with a feature map so that the two matrix products can be reassociated. The model then keeps a single fixed-size matrix state instead of a growing cache, and decoding becomes a recurrent update whose cost per token does not depend on how much context has already been read.[1] The price is expressiveness: a fixed state cannot hold arbitrarily much history, so purely linear models have historically lost to softmax attention on exact copying and precise retrieval, a limitation the Kimi Linear report states plainly in its own related-work section.[2]

Background: from the delta rule to gated DeltaNet

Plain linear attention writes each new key-value pair into the state by addition and never removes anything. If a key appears twice with different values, both writes remain and interfere. The delta rule, connected to linear transformers by Schlag, Irie, and Schmidhuber at ICML 2021, changes the write into a correction: the model first reads what the state currently predicts for the incoming key, then stores only the difference between that prediction and the new value. The Kimi Linear report presents this as one step of online gradient descent on a reconstruction loss, which turns the state into a small associative memory that overwrites itself in place rather than piling up.[2]

Two follow-on results made the delta rule practical at scale. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim showed in "Parallelizing Linear Transformers with the Delta Rule over Sequence Length" (arXiv:2406.06484, 2024) that the sequential rank-1 updates can be batched into chunks and evaluated with dense matrix multiplications.[10] Songlin Yang, Jan Kautz, and Ali Hatamizadeh then combined the delta rule with a forget gate in "Gated Delta Networks: Improving Mamba2 with Delta Rule" (arXiv:2412.06464, ICLR 2025), giving the state a way to decay old associations instead of only correcting them.[9]

The gate in Gated DeltaNet is a single scalar per head per step, following the head-wise decay of Mamba-2 (Tri Dao and Albert Gu, "Transformers are SSMs", ICML 2024), which itself generalized the selective state space model of Mamba (Albert Gu and Tri Dao, 2023).[2] A scalar gate means every number in that head's state decays at exactly the same rate. Gated Linear Attention (Songlin Yang and colleagues, ICML 2024) had already shown that a diagonal, per-channel gate is more expressive, but at a cost in kernel speed.[2]

Kimi Delta Attention

KDA sits between those two. Its state update is the gated delta rule with the scalar forget gate replaced by a diagonal matrix, so that each of the 128 feature channels in a head keeps its own forgetting rate rather than sharing one.[2]

Why channel-wise gating matters

The report's argument for the change is a positional one. Much of what makes rotary position embedding effective is that it assigns a different rotation frequency to each pair of dimensions, so a single attention head can represent both short-range and long-range order at once. A per-head scalar decay has no such per-dimension diversity, which is why the authors describe KDA's learnable channel-wise gate as the component that lets the recurrent layer take over the job that positional encoding normally does.[2] Written out, a gated linear layer's output can be expressed as a query dotted with keys weighted by a cumulative product of transition matrices, exactly the shape of the equivalent formula for self-attention plus RoPE, except that the transition matrix is data dependent and learned instead of a fixed rotation.[2]

Empirically the difference shows up on small synthetic tasks. Using two-layer, two-head models trained for up to 20,000 steps, KDA reached higher accuracy than Gated DeltaNet and Mamba-2 on palindrome reversal, multi-query associative recall, and a 64-stack LIFO state-tracking task, and converged faster than Gated DeltaNet on the first two. Mamba-2, which uses multiplicative decay without any delta rule, "fails on all tasks in our model settings" according to the paper.[2]

The constrained DPLR form and the chunkwise algorithm

A diagonal gate combined with a rank-1 delta correction is an instance of the Diagonal-Plus-Low-Rank (DPLR) transition family used by structured state space models. General DPLR is expensive: the fine-grained decay forces divisions by cumulative decay terms, which are numerically unstable in half precision, so implementations fall back to secondary chunking in full precision and lose access to tensor cores.[2]

KDA constrains the DPLR form by tying both low-rank vectors to the key vector. In the general parameterization the update is written with independent vectors a and b; KDA sets a to the learning-rate-scaled key and b to the key modulated by the gate, which lets the gate factor out of the recurrence entirely. That removes two of the secondary chunking steps and roughly three matrix multiplications from the inter-chunk and output stages. Moonshot measured the resulting kernel at close to twice the speed of a general DPLR kernel for input lengths up to 64k, at batch size 1 with 16 heads.[2]

The chunkwise algorithm itself splits the sequence into chunks of 64, compresses the product of per-token Householder-style corrections into a compact WY representation, applies the UT transform to reduce non-matmul operations, and then runs recurrently between chunks while staying fully parallel inside each chunk.[2] Per attention head the training cost is 6Td^2 + 3TCd + TC^2 with head dimension d and chunk size C, against 2T^2d for full attention, so the KDA term is linear in sequence length where the full-attention term is quadratic.[2]

Each KDA head applies a short depthwise convolution with kernel size 4 followed by a Swish activation to the queries, keys, and values, L2-normalizes the queries and keys, computes the per-channel decay through a low-rank projection, and passes the output through a head-wise RMSNorm and a sigmoid output gate. Key and value head dimensions are both 128, so the recurrent state per head is a fixed 128 by 128 matrix regardless of context length.[2][5]

The 3:1 hybrid and no positional encoding

Kimi Linear does not use KDA alone. Full MLA layers, the compressed-KV attention introduced in DeepSeek-V3, appear at a fixed 3:1 ratio: three KDA layers, then one full attention layer, repeating.[1][2] The authors chose interleaving whole layers rather than mixing head types inside a layer because it is simpler to shard and more stable to train, and they note that the regular repeating pattern keeps KV cache management ordinary.[2] Because only one layer in four keeps a cache, the KV cache shrinks by up to 75 percent at long context purely by construction.[1]

The more unusual choice is that the full-attention layers carry no positional encoding at all. The configuration file sets mla_use_nope to true, and the report states that NoPE is applied to every MLA layer, leaving KDA as the model's only source of position and recency information.[2][5] This is the reverse of the pattern common in other hybrids, where the local or windowed layers keep RoPE and only the global layers go without: SWAN-GPT interleaves RoPE layers with NoPE full-attention layers, and Falcon-H pushes RoPE to a near-NoPE state by setting an extremely high base frequency.[2] The idea that a global NoPE layer needs a dedicated position-aware partner traces to the NoPE literature, in particular Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy, "The Impact of Positional Encoding on Length Generalization in Transformers" (NeurIPS 2023).[2]

The paper backs the choice with an ablation rather than an assertion. A variant called Kimi Linear (RoPE), identical except that the full-attention layers use rotary embeddings, scores about the same on short-context tasks but drops from 54.5 to 51.8 on the long-context average, which puts it behind the plain full-attention baseline.[2] The authors read this as a distribution problem: with RoPE in the global layer, positional bias concentrates there and over-weights short-range order, which hurts when the context window is later extended. Two practical benefits are also claimed. NoPE lets the MLA layers collapse to pure multi-query attention at inference time, and it removes the need for RoPE base-frequency retuning or YaRN-style rescaling during long-context training.[2]

Model configuration

The backbone follows Moonshot's earlier Moonlight MoE recipe, with sparsity raised to 32: 8 of 256 routed experts plus one shared expert are active per token, the first layer is dense, and the router uses a sigmoid activation.[2][5] The released 27-layer stack places full attention at layers 4, 8, 12, 16, 20, 24, and 27, with KDA everywhere else, so the pattern is a clean 3:1 except for the final group, where two KDA layers precede the last full-attention layer.[5] The MLA layers use a KV compression rank of 512 and 32 heads.[5]

Training and matched evaluation

The comparison at the heart of the paper is deliberately narrow. Kimi Linear, a full-attention MLA model, and GDN-H (the same 3:1 hybrid built on Gated DeltaNet instead of KDA) share architecture, parameter count, optimizer, and data: 1.4 trillion tokens from the Kimi K2 pretraining corpus, a 4,096-token training window, the MuonClip optimizer, a WSD learning rate schedule, a learning rate of 1.1e-3, and a 32-million-token global batch.[2] The GDN-H baseline was given the same sigmoid output gate that the ablations found best, so it is not handicapped on that axis.[2]

Short-context results

Benchmark (base, 1.4T tokens)	MLA	GDN-H	Kimi Linear
MMLU	71.6	72.2	73.8
MMLU-Pro	47.2	47.9	51.0
HellaSwag	81.7	82.2	82.9
ARC-Challenge	64.6	66.5	67.3
Winogrande	78.1	77.9	78.6
BBH	71.6	70.6	72.9
TriviaQA	68.9	70.1	71.7
GSM8K	83.7	81.7	83.9
MATH	54.7	54.1	54.7
EvalPlus	59.5	63.1	60.2
CRUXEval-I-cot	51.6	56.0	56.6
CRUXEval-O-cot	61.5	58.1	62.0
C-Eval	79.3	79.1	79.5
CMMLU	79.5	80.7	80.8

Kimi Linear leads on twelve of the fourteen base benchmarks, ties MLA on MATH, and loses to GDN-H on EvalPlus.[2] After identical supervised fine-tuning the pattern holds with three exceptions worth naming: LiveBench (45.2 against 46.4 for GDN-H), MATH500 (81.2 against 83.0), and EvalPlus again (61.0 against 62.6 for MLA).[2] The instruction-tuned gap is largest on GPQA-Diamond, where Kimi Linear scores 62.1 against 57.1 for MLA.[2]

Long-context results

All four models were evaluated at 128k context after the same 1.4T-token recipe.

Benchmark at 128k	MLA	GDN-H	Kimi Linear (RoPE)	Kimi Linear
RULER	81.3	80.5	78.8	84.3
MRCR	22.6	23.9	22.0	29.6
HELMET-ICL	88.0	85.5	88.0	90.0
LongBench V2	36.1	32.6	35.4	35.0
Frames	60.5	58.7	59.9	58.8
RepoQA	63.0	63.0	66.5	68.5
Long Code Arena (Lib)	32.8	34.7	31.3	37.1
Long Code Arena (Commit)	33.2	30.5	32.5	32.7
Average	52.2	51.2	51.8	54.5

The hybrid wins the average by 2.3 points over full attention, but it loses on LongBench V2, Frames, and the commit-level Long Code Arena split.[2] GDN-H is the clearest casualty: it beats MLA on short-context pretraining and then falls behind it at 128k, which the authors call out directly as a reversal of the short-context ordering.[2] A separate reinforcement-learning comparison on an in-house mathematics set found Kimi Linear improving faster than MLA on both training and held-out accuracy through RL training.[2]

Ablations

The hybrid ratio and two architectural details were ablated on a 16-layer, 16-head model at matched FLOPs, reported as training and validation perplexity.

Variant	Training PPL	Validation PPL
3:1 KDA to MLA	9.23	5.65
0:1 (pure full attention)	9.45	5.77
1:1	9.29	5.66
7:1	9.23	5.70
15:1	9.34	5.82
No output gate	9.25	5.67
Swish output gate	9.43	5.81
No convolution layer	9.29	5.70

The 7:1 row is the informative one: it matches 3:1 on training perplexity and is clearly worse on validation, so cutting the share of full-attention layers buys nothing and costs generalization.[2] A separate scaling law fit across five MoE models from 653M to 1.7B activated parameters, trained with the Muon optimizer, put Kimi Linear at roughly 1.16 times the compute efficiency of MLA. The authors note that the MLA baselines received a hyperparameter grid search while the KDA runs simply inherited the MLA configuration.[2]

Efficiency

The efficiency claims come from the paper's own matched 48B setup, and they are not all the same kind of measurement.

Measurement	Condition	Result
KV cache reduction	3:1 layout, long-sequence generation	Up to 75 percent
Prefill latency vs MLA	Batch size 1, 512k tokens	2.3x faster
Prefill latency vs MLA	Batch size 1, 1M tokens	2.9x faster
Time per output token vs MLA	Batch size 1, 1M tokens	About 2.3x faster
Time per output token vs MLA	Larger batches enabled by the memory saving, 1M tokens	6.3x faster (1.84 ms against 11.48 ms)
End-to-end decoding on RULER	128k context	3.98x acceleration at 84.3 accuracy
KDA kernel vs general DPLR kernel	Batch size 1, 16 heads, up to 64k tokens	About 2x

The distinction matters. At batch size 1 the hybrid's decoding advantage over full attention is roughly 2x at a million tokens, bounded by the 3:1 layer ratio itself. The headline 6x comes from spending the freed KV-cache memory on larger batches, and the paper labels it a "theoretical decoding speedup".[2] Prefill latency is indistinguishable from GDN-H, which is the point of the constrained DPLR design: the finer gate costs essentially nothing at the operator level.[2] Below about 16k tokens the hybrid and full attention perform similarly; the gap only opens from 128k onward.[2]

The released checkpoints

The published Base and Instruct checkpoints were trained on 5.7 trillion tokens rather than 1.4 trillion, to match Moonlight's budget, and support a 1M-token context.[2][3] Kimi-Linear-Instruct scores 95.4 on RULER at 128k and 94.8 at 1M, and beats Moonlight-Instruct across the reported suite, for example 72.7 against 43.8 on MMLU-Pro and 45.7 against 11.9 on LiveCodeBench v6.[2] That comparison is not parameter-matched: Moonlight has 16 billion total parameters against Kimi Linear's 48 billion at the same 3 billion active, so the architecture and the extra sparsity are confounded. The paper is explicit that the gain reflects "3x sparsity and a new attention architecture design" together.[2]

Both checkpoints and the reference kernels are MIT licensed; the GitHub repository was created on October 29, 2025 and the Hugging Face repositories received their first commits on October 30.[3][4][6] The KDA kernel shipped inside the flash-linear-attention library rather than as a standalone package, and the model card recommends vLLM with --max-model-len 1048576 for serving.[3]

Adoption and tooling

vLLM merged model support the same day the paper appeared, in pull request 27809 by Zhiyuan Li, and the model has been maintained there since, including AWQ quantization compatibility, ROCm paths, and a KDA state extension to the NIXL connector for prefill/decode disaggregation.[16] SGLang later shipped fused KDA kernels alongside gated DeltaNet kernels.[14] Support in llama.cpp did not arrive on the same timeline: a feature request opened on November 2, 2025 stalled behind the more general work of implementing gated DeltaNet for Qwen3-Next, since KDA is a variant of the same token mixer.[15]

In April 2026 Moonshot open-sourced FlashKDA, a CUTLASS-based CUDA implementation of the KDA kernels that auto-dispatches as a backend of flash-linear-attention. Moonshot's announcement reported a 1.72x to 2.22x prefill speedup over the Triton reference on H20 GPUs; the repository requires SM90 or newer hardware and constrains key and value head dimensions to 128.[11][12] FlashKDA predates the Kimi K3 launch by three months and was re-announced rather than introduced alongside it.[11][12]

Third-party coverage has been mostly explanatory. A DigitalOcean tutorial by Melani Maheswaran (February 13, 2026) walks through the WY representation and the tensor-core motivation without evaluating the claims.[13] On Hugging Face, the Instruct repository accumulated 568 likes and roughly 85,000 monthly downloads by mid-2026, with discussion threads dominated by vLLM deployment problems, requests for lower-precision variants, and questions about how the architecture compares with Qwen3-Next.[4][7]

Attention Residuals is a separate paper

Attention Residuals (AttnRes) is often listed among Kimi Linear's contributions. It is not one. The Kimi Linear technical report never uses the word residual, and the released Kimi Linear configuration contains no attention-residual field, whereas Kimi K3's configuration sets attn_res_block_size to 12.[2][5][19]

AttnRes comes from a separate Kimi Team paper, "Attention Residuals" (arXiv:2603.15031), submitted on March 16, 2026 with Guangyu Chen as first author.[18] It replaces the fixed-weight accumulation of the residual stream with an attention operation, so each layer aggregates earlier representations with learned, input-dependent weights, and a block variant limits the memory cost by attending over block-level summaries. The confusion is understandable: the AttnRes experiments were run on a 48B-total, 3B-activated Kimi Linear model trained on 1.4 trillion tokens, the same matched setup as the original paper.[18] When Sebastian Raschka wrote on July 28, 2026 that attention residuals were "also already part of Kimi Linear", that is accurate about the testbed and not about the October 2025 release.[20]

From Kimi Linear to Kimi K3

Kimi K3, released in July 2026, carries the architecture forward at roughly sixty times the total parameter count. Its configuration file lists 93 layers with 69 KDA layers and 24 full-attention layers, the same 3:1 interleave with full attention at every fourth layer through layer 92 plus a final layer 93, and mla_use_nope set to true throughout.[19] Raschka described K3 as "essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B to 2.8T)", identified the LatentMoE block as the one genuinely new component, and called K3 the first frontier-level model he knew of to drop positional embeddings everywhere.[20]

Limitations and open questions

The strongest caveats are the paper's own. It states that linear attention "still lags full attention on exact copying and fine-grained selection in extreme long-context retrieval", and that the 3:1 hybrid exists precisely because pure linear stacks struggle with precise memory retrieval.[2] The bounded 128 by 128 state per head is what buys the efficiency, and it is also what caps recall: no amount of gating makes a fixed-size associative memory hold unbounded history.

Several limits sit in the evidence rather than the architecture. Every matched comparison is at 48 billion total parameters with a 4,096-token pretraining window, so nothing in the paper demonstrates that the quality parity survives to frontier scale. The 5.7T comparison is against a smaller model. The scaling-law advantage is a modest 1.16x and was measured with the baseline better tuned than the treatment. And the losses are real, not rounding: full attention still wins LongBench V2, Frames, and one Long Code Arena split at 128k, and GDN-H wins EvalPlus.[2]

Independent commentary has pressed on the benchmark question. A July 2026 analysis by RayZ at Acing AI accepted the efficiency claims as structural and the research-scale quality parity as real, then argued that RULER-style needle tests favor bounded-state designs, that shortcut-free retrieval benchmarks remain untested at frontier scale, and that MiniMax M2 showed benchmark parity at development scale followed by deficits in multi-hop reasoning at production scale before reverting to full attention. The piece concludes that "the 3:1 layout itself concedes that periodic exact attention is still required".[17]

Implementation fragility is a related risk. In April 2026 a vLLM bug report found that KDA chunked prefill used the wrong recurrent state layout, which broke Kimi Linear's long-context retrieval while leaving short-context behavior intact.[8] A hybrid whose position information lives entirely inside a recurrent kernel has fewer independent checks than a transformer whose positional signal is an explicit rotation, and failures of that kind show up only in the tests the architecture is supposed to be good at.

See also

References

Kimi Team, "Kimi Linear: An Expressive, Efficient Attention Architecture," arXiv:2510.26692 (submitted October 30, 2025; v2 November 1, 2025). https://arxiv.org/abs/2510.26692
Kimi Team, "Kimi Linear: An Expressive, Efficient Attention Architecture," full technical report PDF, v2, November 1, 2025 (equations, ablation Table 1, results Tables 3 to 9, efficiency Figures 1, 2 and 7, and Sections 3, 4, 6 and 7). https://arxiv.org/pdf/2510.26692v2
Moonshot AI, "Kimi Linear" repository and README, GitHub, accessed July 31, 2026. https://github.com/MoonshotAI/Kimi-Linear
Moonshot AI, "moonshotai/Kimi-Linear-48B-A3B-Instruct" model card, Hugging Face, accessed July 31, 2026. https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct
Moonshot AI, "Kimi-Linear-48B-A3B-Instruct config.json," Hugging Face, accessed July 31, 2026. https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/raw/main/config.json
Moonshot AI, "Kimi-Linear-48B-A3B-Instruct commit history," Hugging Face, accessed July 31, 2026. https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/commits/main
Hugging Face community, "moonshotai/Kimi-Linear-48B-A3B-Instruct discussions," accessed July 31, 2026. https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/discussions
vLLM issue 41292, "KDA chunked prefill uses wrong recurrent state layout and breaks Kimi-linear long-context retrieval," opened April 29, 2026, closed April 30, 2026. https://github.com/vllm-project/vllm/issues/41292
Songlin Yang, Jan Kautz, Ali Hatamizadeh, "Gated Delta Networks: Improving Mamba2 with Delta Rule," arXiv:2412.06464 (December 9, 2024), ICLR 2025. https://arxiv.org/abs/2412.06464
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, Yoon Kim, "Parallelizing Linear Transformers with the Delta Rule over Sequence Length," arXiv:2406.06484 (June 10, 2024). https://arxiv.org/abs/2406.06484
Moonshot AI, "FlashKDA: high-performance Kimi Delta Attention kernels," GitHub, accessed July 31, 2026. https://github.com/MoonshotAI/FlashKDA
MarkTechPost, "Moonshot AI Open-Sources FlashKDA: CUTLASS Kernels for Kimi Delta Attention with Variable-Length Batching and H20 Benchmarks," April 30, 2026. https://www.marktechpost.com/2026/04/30/moonshot-ai-open-sources-flashkda-cutlass-kernels-for-kimi-delta-attention-with-variable-length-batching-and-h20-benchmarks/
Melani Maheswaran, "Designing Hardware-Aware Algorithms with Kimi Linear: Kimi Delta Attention," DigitalOcean Community, February 13, 2026. https://www.digitalocean.com/community/tutorials/kimi-linear-moonshot-ai
LMSYS Org, "SGLang and Miles Add Day-0 Support for Kimi K3," July 27, 2026. https://www.lmsys.org/blog/2026-07-27-kimi-k3-day0-support/
llama.cpp issue 16930, "Feature Request: Kimi Linear model (Kimi Delta Attention)," opened November 2, 2025. https://github.com/ggml-org/llama.cpp/issues/16930
vLLM pull request 27809, "[Model] Introduce Kimi Linear to vLLM," merged October 30, 2025. https://github.com/vllm-project/vllm/pull/27809
RayZ, "Linear Attention at Frontier Scale: Kimi K3's KDA Claim, Fact-Checked," Acing AI, July 19, 2026 (updated July 28, 2026). https://acingai.com/articles/linear-attention-kimi-k3
Kimi Team, "Attention Residuals," arXiv:2603.15031 (submitted March 16, 2026). https://arxiv.org/abs/2603.15031
Moonshot AI, "Kimi-K3 config.json," Hugging Face, accessed July 31, 2026. https://huggingface.co/moonshotai/Kimi-K3/raw/main/config.json
Sebastian Raschka (@rasbt), commentary on the Kimi K3 architecture, X, July 28, 2026. https://x.com/rasbt/status/2082098201247600765
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, Siva Reddy, "The Impact of Positional Encoding on Length Generalization in Transformers," arXiv:2305.19466, NeurIPS 2023. https://arxiv.org/abs/2305.19466
Tri Dao, Albert Gu, "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality," arXiv:2405.21060 (May 31, 2024), ICML 2024. https://arxiv.org/abs/2405.21060
Albert Gu, Tri Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces," arXiv:2312.00752 (December 1, 2023). https://arxiv.org/abs/2312.00752

Editing: Kimi Linear