Citation and evidence

State space model (deep learning)

45 min full readUpdated 52 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

Deep LearningMachine Learning

Cite this article

See also: Transformer, Recurrent neural network, Deep learning, Mamba, Mamba-2, Mamba-3

A state space model (SSM) in deep learning is a class of sequence model that maps an input sequence to an output sequence through a fixed-size latent state, using continuous-time state space equations that are discretized for neural networks. Because the state does not grow with the input, an SSM layer's cost grows linearly (or, when computed as a long convolution with FFTs, nearly linearly) with sequence length, compared with the quadratic cost of Transformer attention, and autoregressive generation needs only constant memory per step [1][2]. This makes SSMs attractive for very long sequences such as audio, genomics and long documents. The term most commonly refers to structured state space models, a family that began with the S4 model introduced by Albert Gu, Karan Goel, and Christopher Re at Stanford University in October 2021 [1]. Its most influential member is Mamba, introduced by Albert Gu and Tri Dao in December 2023, which made the SSM parameters depend on the input; these models are called selective state space models [2].

What is a state space model?

SSMs have attracted intense interest because they offer an alternative to the Transformer architecture that scales linearly with sequence length rather than quadratically. While Transformers rely on attention mechanisms that compute pairwise interactions between all tokens (yielding O(n^2) complexity in sequence length n), SSMs process sequences through a fixed-size recurrent state. Training cost is O(n) for recurrent or scan-based computation and O(n log n) for FFT-based convolution, and each generated token costs a constant amount of compute and memory [1][2]. This property makes SSMs particularly appealing for tasks involving very long sequences, such as genomics, audio modeling, and long-document understanding.

Mamba introduced selective state spaces, a mechanism that makes SSM parameters input-dependent, and combined this with a hardware-aware implementation optimized for modern GPUs. The original Mamba paper reports that the model "enjoys fast inference (5x higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences" [2]. Since 2024 most large deployments have been hybrids that keep a minority of attention layers, including AI21 Labs' Jamba, NVIDIA's Nemotron-H and Nemotron 3 families, IBM's Granite 4.0 and TII's Falcon-H1 [9][11][37][34][38]. Albert Gu is an assistant professor in the Machine Learning Department at Carnegie Mellon University and co-founder and chief scientist of the SSM startup Cartesia; Tri Dao is an assistant professor of computer science at Princeton University and co-founder and chief scientist of Together AI [19][45].

"What are State Space Models? Redefining AI & Machine Learning with Data", video from IBM Technology, published January 15, 2026. Watch on YouTube.

Mathematical Formulation

State space models are rooted in the classical state space representation from control theory. A continuous-time, linear, time-invariant (LTI) system is defined by four matrices (A, B, C, D) and two equations [1]:

  • State equation: x′(t)=Ax(t)+Bu(t)x'(t) = A x(t) + B u(t)
  • Output equation: y(t)=Cx(t)+Du(t)y(t) = C x(t) + D u(t)

Here, u(t) is the input signal, x(t) is the hidden state vector of dimension N, and y(t) is the output signal. The matrix A (of size N x N) governs how the state evolves over time. B (N x 1) controls how the input influences the state. C (1 x N) maps the state to the output. D (1 x 1) provides a direct skip connection from input to output; the S4 paper omits it from the exposition because "the term Du can be viewed as a skip connection and is easy to compute" [1]. A deep SSM layer applies one such single-input, single-output (SISO) system independently to each of the model's channels, with A, B, C (and the step size introduced below) learned by gradient descent [1]. The Mamba papers use a different letter convention for the same system: input x(t), state h(t), output y(t) [2].

Discretization

Since deep learning operates on discrete sequences rather than continuous signals, the continuous system must be discretized with a step size Δ\Delta that represents the resolution of the input, turning the continuous parameters into discrete ones written Aˉ,Bˉ\bar{A}, \bar{B} [1][2]. Two rules dominate the literature [4]:

  • Bilinear (Tustin) method, used by S4 and its predecessors [1]: Aˉ=(I−Δ2A)−1(I+Δ2A),Bˉ=(I−Δ2A)−1ΔB\bar{A} = (I - \tfrac{\Delta}{2} A)^{-1}(I + \tfrac{\Delta}{2} A), \quad \bar{B} = (I - \tfrac{\Delta}{2} A)^{-1} \Delta B
  • Zero-order hold (ZOH), used by DSS and given by the Mamba paper as its example [4][2]: Aˉ=exp⁡(ΔA),Bˉ=(ΔA)−1(exp⁡(ΔA)−I)⋅ΔB\bar{A} = \exp(\Delta A), \quad \bar{B} = (\Delta A)^{-1}(\exp(\Delta A) - I) \cdot \Delta B

The S4D paper reports "little empirical difference" between the two rules in its experiments [4]. The Mamba-3 paper later observed that the released Mamba-1 and Mamba-2 implementations actually use a simpler approximation for the input term, which it formalizes as "exponential-Euler" discretization [6].

The discrete recurrence then becomes:

xk=Aˉxk−1+Bˉuk,yk=Cxk+Dukx_k = \bar{A} x_{k-1} + \bar{B} u_k, \qquad y_k = C x_k + D u_k

This recurrence can be executed step by step like an RNN, which gives constant memory per generated token, or, when the parameters do not change over time, unrolled into a convolution for parallel training [1]. This dual view, convolution for training and recurrence for generation, is one of the key advantages of LTI SSMs over both Transformers and traditional RNNs.

Connection to Convolutions

By unrolling the discrete recurrence (with initial state zero), the output sequence y can be expressed as a single convolution of the input sequence u with a kernel Kˉ\bar{K} of the same length L as the sequence [1]:

Kˉ=(CBˉ,CAˉBˉ,CAˉ2Bˉ,…,CAˉL−1Bˉ),y=Kˉ∗u\bar{K} = (C\bar{B}, C\bar{A}\bar{B}, C\bar{A}^2\bar{B}, \ldots, C\bar{A}^{L-1}\bar{B}), \qquad y = \bar{K} * u

Once the kernel is known, the convolution can be computed with Fast Fourier Transforms (FFT) in O(L log L) time [1]. Computing the kernel itself is the hard part: done naively it needs L successive multiplications by Aˉ\bar{A}, or O(N^2 L) operations, which is the bottleneck S4 was designed to remove [1]. The convolutional shortcut exists only while (Δ,A,B,C)(\Delta, A, B, C) are fixed for all time steps; selective SSMs such as Mamba give it up and compute the recurrence with a parallel scan instead [2].

S4: Structured State Spaces for Sequence Modeling

The S4 (Structured State Space sequence model) architecture, posted by Gu, Goel, and Re on 31 October 2021 [1], was the breakthrough that made SSMs competitive with Transformers on a range of sequence modeling tasks. The paper was presented at ICLR 2022, where it received an Outstanding Paper Honorable Mention. Its abstract claims S4 is the first model "solving the challenging Path-X task of length 16k that all prior work fails on, while being as efficient as all competitors" [1]. S4 built on the Linear State-Space Layer (LSSL), an earlier model by the same group that showed the promise of deep SSMs but, in the S4 authors' words, was "computationally impractical as a general sequence modeling solution" [1].

The HiPPO Framework

A critical ingredient in S4 is the HiPPO (High-order Polynomial Projection Operators) framework, developed by Gu, Dao, Ermon, Rudra and Re and published at NeurIPS 2020 [3]. HiPPO is a framework "for the online compression of continuous signals and discrete time series by projection onto polynomial bases" [3]. Given a measure that specifies how much each past time step matters, it derives a linear update for a state vector holding the coefficients of the optimal polynomial approximation of the input history seen so far [3]. As special cases, the framework re-derives the Legendre Memory Unit and generalizes the gating mechanism of GRUs [3].

The HiPPO-LegS (scaled Legendre) variant assigns uniform weight to the entire history, so that it "scales through time to remember all history, avoiding priors on the timescale" [3]. The S4 paper calls its state matrix "the HiPPO matrix":

Ank=−{(2n+1)1/2(2k+1)1/2if n>kn+1if n=k0if n<kA_{nk} = -\begin{cases} (2n+1)^{1/2}(2k+1)^{1/2} & \text{if } n > k \\ n+1 & \text{if } n = k \\ 0 & \text{if } n < k \end{cases}

The initialization matters a great deal in practice. The S4 paper notes that the LSSL improved on sequential MNIST "from 60% to 98%" simply by switching from a random state matrix to the HiPPO matrix, and S4's own ablations on sequential CIFAR-10 concluded that "S4's effectiveness primarily comes from the HiPPO initialization, not the NPLR parameterization" [1].

Structured Parameterization

Naively computing the SSM convolution kernel requires repeated multiplication by the N x N matrix A. Diagonalizing A would make this cheap, but the S4 authors showed that the HiPPO matrix's diagonalization has entries exponentially large in N and is numerically infeasible [1]. S4 instead writes the HiPPO matrices as a normal plus low-rank (NPLR) matrix, equivalently diagonal plus low-rank (DPLR) after a change of basis. The low-rank term is corrected with the Woodbury identity and the diagonal term reduces to a Cauchy kernel, so computing the kernel takes O~(N+L)\tilde{O}(N + L) operations and O(N + L) memory, "essentially tight for sequence models" [1]. Compared to the LSSL, the authors report S4 is up to 30x faster with 400x less memory usage [1].

The results were strong. S4 reached 91% accuracy on sequential CIFAR-10 (processing images one pixel at a time) "with no data augmentation or auxiliary losses, on par with a larger 2-D ResNet": 91.12% against 89.46% for a ResNet-18 with 11.0M parameters versus S4's 7.9M [1]. It generated pixels and tokens 60x faster than standard autoregressive models on CIFAR-10 and WikiText-103, came within 0.8 perplexity of Transformers on WikiText-103, and was the first model to solve Path-X, a synthetic benchmark of length 16,384 (128 x 128 pixels), with 88% accuracy where all prior work scored the 50% of random guessing. It also reached state of the art on every task in the Long Range Arena benchmark [1].

S4 Variants

Several refinements followed S4 within a year, most of them simplifying its state matrix:

ModelPostedAuthors and venueKey ideaReported result
DSS (Diagonal State Spaces)March 2022Gupta, Gu, Berant; NeurIPS 2022Drops S4's low-rank correction and uses a purely diagonal state matrix with ZOH discretizationMatches S4 on Long Range Arena and Speech Commands [20]
S4DJune 2022Gu, Gupta, Goel, Re; NeurIPS 2022Systematic study of diagonal SSMs; explains why DSS works (the diagonal restriction of S4's matrix recovers the same kernel as state size grows) and shows initialization is criticalComparable to S4 in almost all settings; 85% average on Long Range Arena [4]
S5August 2022Smith, Warrington, Linderman; ICLR 2023Replaces S4's many independent SISO SSMs with one multi-input, multi-output (MIMO) SSM computed with parallel scans87.4% average on Long Range Arena, 98.5% on Path-X [21]

Expanded article table

The diagonal structure these papers established became standard: Mamba uses a diagonal state matrix initialized with the S4D-Real scheme [2], and S5's use of parallel scans anticipated the scan-based computation of selective SSMs [2][21].

Selective state space models (Mamba)

Mamba was introduced by Albert Gu (Carnegie Mellon University) and Tri Dao (Princeton University) on December 1, 2023, in the paper "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" [2]. The paper was submitted to ICLR 2024, where it was not accepted, and was published at the Conference on Language Modeling (COLM) 2024 [46]. It addressed a fundamental limitation of all prior structured SSMs: their linear time invariance. This section summarizes the selection mechanism at the concept level; the Mamba article covers the architecture, benchmarks and released models in depth.

Why are linear time-invariant SSMs limited?

In S4 and its variants, the matrices A, B, C, and the step size Δ\Delta are fixed parameters that do not change based on the input. This makes the system linear and time-invariant (LTI), which is precisely what allows the efficient convolutional computation. However, LTI systems treat all inputs identically: they cannot selectively attend to or ignore specific tokens based on content. The Mamba authors identified that "a key weakness of such models is their inability to perform content-based reasoning" [2].

Gu and Dao frame sequence modeling as a problem of compressing context into a state. Attention is effective but inefficient because it does not compress the context at all and must keep the whole KV cache; recurrent models are efficient because their state is finite, but are only as good as that state's compression of the context [2]. A selection mechanism is their answer to how a fixed-size state can keep what matters.

The paper makes the problem concrete with two synthetic tasks [2]:

  • Selective Copying modifies the classic Copying task by randomizing the positions of the tokens to be memorized among "noise" tokens. An LTI model can solve the ordinary Copying task just by tracking time (for example, with a convolution kernel of exactly the right length), but with random spacing it must recognize relevant tokens by content.
  • Induction Heads requires recalling the token that followed an earlier occurrence of the current token, an associative recall task that the paper describes as hypothesized to explain much of in-context learning in LLMs.
ArchitectureInner layerSelective Copying accuracy (%)
No gateS4 (LTI)18.3
No gateS6 (selective)97.0
H3S457.0
H3Hyena30.1
H3S699.7
MambaS456.4
MambaHyena28.4
MambaS699.8

Expanded article table

Selective Copying results from Table 1 of the Mamba paper; two-layer models on sequences of length 4,096 [2].

Swapping the LTI layer for the selective one raised accuracy in every architecture, while architectural gating alone (H3, Mamba with an LTI inner layer) did not solve the task. The authors argue that such gating "does not interact along the sequence axis" [2]. On Induction Heads, a two-layer Mamba trained on sequences of length 256 generalized perfectly to sequences of length 1,048,576, about 4,000x longer than it saw in training, while no other tested method, including attention with several positional encodings, went beyond 2x [2].

The selection mechanism (S6)

Mamba's core change is making three SSM parameters functions of the input. In the paper's notation, for an input x of shape (batch B, length L, channels D) [2]:

  • sB(x)=LinearN(x)s_B(x) = \text{Linear}_N(x), so BB gets batch and length dimensions;
  • sC(x)=LinearN(x)s_C(x) = \text{Linear}_N(x), likewise for CC;
  • Δ=softplus(Parameter+sΔ(x))\Delta = \text{softplus}(\text{Parameter} + s_\Delta(x)) with sΔ(x)=BroadcastD(Linear1(x))s_\Delta(x) = \text{Broadcast}_D(\text{Linear}_1(x)), a projection to one dimension broadcast across channels (the released code uses a low-rank projection).

Because Δ\Delta, BB and CC now vary by position, Aˉ\bar{A} and Bˉ\bar{B} do too, and "the model has changed from time-invariant to time-varying" [2]. The state matrix A is not made input-dependent; the authors explain that A affects the model only through Aˉ=exp⁡(ΔA)\bar{A} = \exp(\Delta A), so selectivity in Δ\Delta already makes (Aˉ,Bˉ)(\bar{A}, \bar{B}) selective [2]. The paper calls the resulting layer S6: an S4 model with a selection mechanism, computed with a scan [2].

The step size acts as the main gate. In the paper's words, "a large Δ\Delta resets the state h and focuses on the current input x, while a small Δ\Delta persists the state and ignores the current input" [2]. Theorem 1 of the paper makes the link to RNN gating exact: with state size N = 1, A = -1, B = 1 and a softplus step size, the selective recurrence reduces to ht=(1−gt)ht−1+gtxth_t = (1 - g_t) h_{t-1} + g_t x_t with gt=σ(Linear(xt))g_t = \sigma(\text{Linear}(x_t)), the interpolating update gate of GRU-style recurrent networks [2]. The authors list three mechanistic effects of selection [2]:

  • Variable spacing: filtering out irrelevant tokens between inputs of interest, as in Selective Copying or filler words such as "um" in speech.
  • Filtering context: ignoring irrelevant history, which they offer as a reason selective models can keep improving with longer context where many sequence models do not.
  • Boundary resetting: resetting the state when, for example, several documents are packed into one sequence.

The selective scan and hardware-aware implementation

Making the parameters input-dependent breaks the LTI property, so the convolutional mode of S4 can no longer be used. Mamba computes the time-varying recurrence with a work-efficient parallel scan instead [2]. The obstacle is memory: the expanded state has shape (B, L, D, N), N times larger than the input, and materializing it in GPU high-bandwidth memory (HBM) would be slow [2]. Gu and Dao's CUDA kernel uses three techniques [2]:

  1. Kernel fusion: load (Δ,A,B,C)(\Delta, A, B, C) into fast on-chip SRAM, discretize, scan and multiply by C there, and write only the (B, L, D) output back to HBM.
  2. Parallel scan: run the recurrence as an associative scan rather than one step at a time.
  3. Recomputation: recompute the intermediate states in the backward pass instead of storing them.

This approach is directly inspired by the IO-aware principles behind FlashAttention (also developed by Tri Dao). On an A100 80GB GPU, the authors report that their scan is faster than FlashAttention-2 beyond sequence length 2K and up to 20-40x faster than a standard scan implementation in PyTorch [2].

Mamba architecture

The full Mamba block combines the H3 block that most SSM architectures were based on with the gated MLP block of modern networks [2]. Each block:

  1. Expands the input dimension (by a factor of 2 in the paper) through two parallel linear projections.
  2. On one branch, applies a short 1D convolution, a SiLU activation and the selective SSM.
  3. On the other branch, applies a SiLU activation that acts as a multiplicative gate.
  4. Multiplies the two branches and projects back to the model dimension.

The authors describe the result as "a simplified end-to-end neural network architecture without attention or even MLP blocks" [2]. The model is a homogeneous stack of these blocks with normalization and residual connections; the released language model checkpoints use RMSNorm [2][47].

How well does Mamba perform?

The authors report that Mamba-3B outperforms Transformers of the same size and matches Transformers twice its size in both pretraining perplexity and downstream evaluation [2]. In their A100 inference benchmark, Mamba achieves 4-5x higher throughput than a Transformer of similar size, "since without the KV cache it can use much higher batch sizes" [2]. Its memory footprint during generation is constant regardless of sequence length, compared with the linearly growing KV cache of Transformers. On audio waveforms and DNA sequences, the authors report that Mamba outperforms prior state-of-the-art models such as SaShiMi (an S4-based audio model), Hyena and Transformers, with performance improving as context grows up to million-length sequences [2].

Mamba-2: State Space Duality

On 31 May 2024, Tri Dao and Albert Gu published "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" [5], introducing Mamba-2. The paper appeared at ICML 2024 and established a formal connection between SSMs and attention.

The SSD Framework

The central contribution is the Structured State Space Duality (SSD) framework. It shows that SSMs and a family of attention variants are different ways of multiplying by the same class of structured semiseparable matrices [5]. In particular, an SSM whose state matrix is a scalar times the identity computes the same function as masked self-attention without the softmax, Y=(L∘QK⊤)VY = (L \circ QK^\top) V, where L is an input-dependent 1-semiseparable causal mask built from products of scalar decays [5]. Linear attention is the special case where L is the plain causal mask, and RetNet the case of a fixed exponential decay mask [5].

This duality has both theoretical and practical implications. Theoretically, it places two seemingly disparate families of sequence models in one framework. Practically, SSD's chunked algorithm uses the quadratic, attention-like form inside short chunks and the linear recurrence between chunks, so most of the work becomes matrix multiplications that run on GPU tensor cores [5]. The authors describe the algorithm as "much easier to implement than the original selective scan of Mamba, coming in at just around 25 lines of code" [22].

Mamba-2 Architecture

Mamba-2 refines the selective SSM layer from Mamba-1 using insights from SSD. The key changes are [5]:

  • The state matrix A is restricted from diagonal to scalar times identity, and the layer uses multiple heads with a head dimension P of typically 64 or 128, analogous to multi-head attention; the authors describe this as "slightly decreasing the expressive power in return for significant training efficiency improvements."
  • Parallel parameter projections: A, X, B and C are produced by one projection at the start of the block, analogous to the parallel Q, K, V projections of attention, which also suits tensor parallelism.
  • Extra normalization: a normalization layer before the output projection, added to fix instabilities in larger models.
  • Much larger state sizes: the SSD implementation allows recurrent states 8x the size of Mamba's "or even higher, with minimal slowdown."

The authors report that the SSD core layer is 2-8x faster than Mamba's optimized selective scan, crosses over FlashAttention-2 at sequence length 2K and is 6x faster at 16K, while remaining competitive with Transformers on language modeling [5]. They also caution that a Mamba-2 model as a whole "might not be as efficient to train as Transformer at short sequence length (e.g. at 2K)" [5]. In hybrid experiments, around 10% attention layers worked best [5].

Mamba-3

On 16 March 2026, Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao and Albert Gu posted Mamba-3, presented as an oral paper at ICLR 2026 [6][23]. Framed around an "inference-first perspective," it improves the SSM along three axes rooted in SSM principles [6]:

  1. Exponential-trapezoidal discretization: the authors' framework formalizes the rule used in practice by Mamba-1 and Mamba-2 as "exponential-Euler," a first-order approximation, and generalizes it to a trapezoidal rule that is a second-order accurate approximation of the state-input integral and amounts to a data-dependent convolution of width two on the input. Combined with bias terms on B and C, this let Mamba-3 drop the short causal convolution used in earlier recurrent architectures [6].
  2. Complex-valued states: the authors note that restricting transitions to real, non-negative eigenvalues (as in Mamba-2) degrades performance on simple state-tracking tasks such as parity, citing work by Grazzi et al. and others. Mamba-3 reintroduces complex-valued state updates, proves they are equivalent to a data-dependent rotary embedding (RoPE) on the B and C projections, and reports near-perfect results on arithmetic state-tracking tasks that Mamba-2 cannot solve [6].
  3. MIMO formulation: instead of the standard single-input, single-output update, whose outer product has low arithmetic intensity at decode time, a multi-input, multi-output update uses matrix products, raising decoding FLOPs while keeping decode latency similar, "for better model performance without increasing decode latency" [6].

At the 1.5B parameter scale, the authors report that Mamba-3 improves average downstream accuracy by 0.6 points over the next best model, Gated DeltaNet, and that the MIMO variant adds another 1.2 points, for a total gain of 1.8 points. Across state-size experiments, Mamba-3 (MIMO) with state size 64 matches the perplexity of Mamba-2 with state size 128 [6].

What counts as a state space model?

The SSD framework shows that the boundary between SSMs and linear attention is a matter of parameterization rather than a sharp line: masked linear attention and Mamba-2's SSD layer are both semiseparable matrix multiplications [5]. In practice, papers and model cards still choose a label, and this article follows the authors' own descriptions:

  • Layers described by their authors as S4, Mamba, Mamba-2 or Mamba-3 layers (S4, DSS, S4D, S5, H3, Mamba and its successors) are SSMs in the sense used here.
  • Gated DeltaNet (Yang, Kautz and Hatamizadeh, ICLR 2025) is titled "Improving Mamba2 with Delta Rule" but presents itself as a "linear Transformer" built on a gated delta rule [24]. Models built from such layers, such as Qwen3-Next (whose model card describes its Gated DeltaNet layers as "linear attention heads") and Kimi Linear (whose Kimi Delta Attention is described as "an expressive linear attention module"), are linear-attention hybrids and are covered on their own pages, not counted as SSM hybrids here [25][26]. See linear attention, Qwen3-Next and Kimi Linear.
  • Gated linear recurrences such as Griffin's RG-LRU, which its authors describe as inspired by the Linear Recurrent Unit, and RWKV are related recurrent families, described in the section on other SSM and linear-complexity variants [16][14].

How do SSMs differ from Transformers?

The comparison between SSMs and Transformers involves trade-offs across multiple dimensions.

Computational Complexity

The most cited advantage of SSMs is their linear complexity. Standard Transformer self-attention compares every token with every other token, so its time per layer grows as O(n^2) in sequence length n, and its KV cache grows as O(n). An SSM layer updates a fixed-size state per token, so its time grows as O(n) (for a scan or recurrence) or O(n log n) (for an FFT convolution), with a per-token cost that depends on the model width and state size rather than on how many tokens came before [1][2]. Asymptotic counts hide large constant factors, so measured speed depends on kernels, hardware and sequence length.

In practice, Transformers can be faster at short sequence lengths because of highly optimized attention kernels such as FlashAttention, and the SSM advantage appears as sequences get longer. The Mamba authors report their selective scan overtaking FlashAttention-2 beyond 2K tokens on an A100, and the Mamba-2 authors report SSD crossing FlashAttention-2 at 2K and running 6x faster at 16K [2][5]. A 2025 benchmarking study by Mitra and colleagues of long-context inference on consumer and embedded GPUs found Transformers up to 1.9x faster below 8K tokens, while SSMs became up to 4x faster at around 57K tokens; it also found that custom SSM kernels such as selective scan accounted for over 55% of inference latency on edge platforms [7].

Memory Efficiency

During autoregressive generation, Transformers must store a KV cache that grows linearly with the number of tokens in the context. For a large model generating a long sequence, this cache can consume tens of gigabytes of GPU memory. SSMs maintain a fixed-size state regardless of sequence length; in the same consumer and embedded GPU study, this translated to a roughly 64% smaller memory footprint for SSMs in long-context inference [7].

In-Context Learning and Recall

Transformers excel at in-context learning, the ability to learn new tasks from examples provided in the prompt without any parameter updates. This capability depends on the attention mechanism's ability to look up any earlier token. SSMs, by contrast, must compress all past information into a fixed-size state, which limits precise retrieval of arbitrary facts from the context. Several lines of evidence document the gap:

  • Associative recall. The Zoology study (Arora et al., ICLR 2024) pretrained 17 attention and "gated-convolution" models, including H3 and Hyena, and found that 82% of the perplexity gap to attention was explained by recall of information mentioned earlier in context; a 70M-parameter attention model outperformed a 1.4B-parameter gated-convolution model on associative recall [27].
  • Copying. Jelassi and colleagues ("Repeat After Me," ICML 2024) proved that a two-layer Transformer can copy strings of exponential length while "generalized state space models" with a fixed-size latent state are fundamentally limited, and found that pretrained Transformers "dramatically outperform state space models at copying and retrieving information from context" [28].
  • Scale. NVIDIA's controlled study of 8B-parameter models trained on the same data found that pure Mamba and Mamba-2 models matched or exceeded Transformers on many tasks but lagged on tasks requiring strong copying or in-context learning (for example five-shot MMLU and phonebook lookup) and on long-context reasoning [8].

The selection mechanism was itself designed to improve recall (Mamba solves the synthetic Induction Heads task), so the remaining gap concerns how much information a fixed-size state can hold, not content-awareness as such [2][28].

Comparison Table

PropertyTransformerSelective SSM (e.g., Mamba)Hybrid (e.g., Jamba, Nemotron-H)
Training cost per layerO(n^2) in sequence lengthO(n) with a parallel scan [2]Dominated by the attention layers at long lengths, but with far fewer of them
Inference memoryKV cache grows linearly with contextFixed-size state [2]Smaller KV cache (only the attention layers keep one) [9]
Inference throughputLimited by KV cache at long contexts4-5x a similar-size Transformer in the Mamba authors' A100 benchmark [2]Up to 3x similar-size Transformers on long contexts in AI21 and NVIDIA reports [9][11]
Exact copying and associative recallStrongWeaker; limited by state size [27][28]Close to or better than Transformers in NVIDIA's 8B study [8]
Training parallelismFully parallel (matrix multiply)Parallel via scan (Mamba) or chunked matrix multiply (Mamba-2 SSD) [2][5]Parallel
Kernel maturityHighly optimized (FlashAttention)Custom scan and SSD kernels [2][5]Both kinds of kernel

Expanded article table

Hybrid Architectures

The complementary strengths and weaknesses of Transformers and SSMs have motivated hybrid architectures that combine both. The Mamba-2 authors hypothesize that the SSM layers act as a general sequence-to-sequence mapping while a few attention layers act as a retrieval mechanism [5].

Jamba (AI21 Labs, 2024)

Jamba was released by AI21 Labs in March 2024, which called it "the world's first production-grade Mamba based model" [9][29]. Its architecture interleaves Mamba layers and Transformer attention layers at a ratio of one attention layer to seven Mamba layers. It also incorporates Mixture of Experts (MoE) layers to increase parameter count without proportionally increasing compute. The base Jamba model has 52B total parameters but activates only 12B per token, and was released with open weights under the Apache 2.0 license [9][29].

AI21 reported a 256K-token context window, the ability to fit up to 140K tokens of context on a single GPU, and 3x the throughput on long contexts of Mixtral 8x7B, a Transformer model of comparable size [29]. AI21 released Jamba 1.5 in August 2024. Jamba 1.5 Large has 94B active and 398B total parameters in 9 blocks of 8 layers (72 layers), keeps the 1:7 attention-to-Mamba ratio, uses 16 MoE experts with top-2 routing, and uses 64 query heads with 8 key-value heads (grouped-query attention) [10]. The Jamba 1.5 paper reports that "in a hybrid architecture, the Mamba-1-Attention combination works better than Mamba-2-Attention," so Jamba 1.5 Large uses Mamba-1 layers [10].

NVIDIA Nemotron-H (2025)

NVIDIA's Nemotron-H family (April 2025) demonstrated hybrid Mamba-Transformer models at larger training scale [11]. About 8% of layers are self-attention, evenly dispersed, and the rest are split evenly between Mamba-2 and feed-forward layers: Nemotron-H-8B has 4 self-attention layers out of 52 (with 24 Mamba-2 and 24 feed-forward layers), and Nemotron-H-56B has 10 out of 118 [11]. NVIDIA trained the 8B model on a 15-trillion-token horizon and the 56B model on 20 trillion tokens, the latter fully in FP8 [11]. It also distilled the 56B model into Nemotron-H-47B-Base with a pruning-and-distillation method called MiniPuzzle; candidate architectures had to fit FP4 inference at a 1-million-token context within the 32 GiB memory of a single NVIDIA GeForce RTX 5090 GPU [11].

NVIDIA reports that Nemotron-H models offer accuracy better than or on par with similarly sized open Transformer models (such as Qwen-2.5 and Llama-3.1) while being up to 3x faster at inference, measured on H100 GPUs with 65,536 input and 1,024 output tokens [11]. Vision-language models based on Nemotron-H-8B and Nemotron-H-56B were used as base models for NVIDIA's Cosmos-Reason1 physical-AI reasoning models [11][30].

NVIDIA Mamba-2-Hybrid Study

In a systematic 2024 study, NVIDIA trained 8B-parameter Mamba, Mamba-2, Transformer, and hybrid models on up to 3.5 trillion tokens using identical datasets [8]. The hybrid model (43% Mamba-2 layers, 7% self-attention layers, 50% MLP layers) outperformed the pure Transformer on all 12 standard tasks evaluated, by an average of 2.65 points, and was predicted to be up to 8x faster at token generation during inference. On 23 additional long-context tasks, the hybrid continued to "closely match or exceed the Transformer on average" [8].

Other hybrid models

By late 2026, hybrids that keep a small share of attention had become the main form in which SSM layers reach released language models. The table lists models whose developers describe their sequence-mixing layers as Mamba or Mamba-2 (or SSM heads); see Mamba for pure-Mamba releases such as Codestral Mamba and Falcon Mamba.

ModelDeveloperDateSSM layer and layout
JambaAI21 LabsMarch 2024Mamba-1 and attention (7:1), MoE; 52B total / 12B active [9]
Zamba 7BZyphraMay 2024"a Mamba backbone with a single shared attention module"; 1T training tokens [31]
Samba 3.8BMicrosoftJune 2024 (ICLR 2025)Mamba combined layer-wise with sliding window attention; 3.2T training tokens [32]
Jamba 1.5 Mini / LargeAI21 LabsAugust 2024Mamba-1 and attention (7:1), MoE; Large has 398B total / 94B active [10]
HymbaNVIDIANovember 2024Attention heads and SSM heads in parallel within the same layer [33]
Zamba2 (1.2B, 2.7B, 7.4B)ZyphraJuly to September 2024 (report November 2024)Hybrid Mamba-2 and Transformer [48]
Bamba 9BIBM with Princeton, CMU and UIUCDecember 2024Hybrid Mamba-2 model [49]
Nemotron-H (8B, 56B, 47B)NVIDIAApril 2025Mamba-2 with about 8% attention layers [11]
Falcon-H1 (0.5B to 34B)TIIMay 2025Attention and Mamba-2 heads in parallel in each mixer block [34]
Hunyuan-TurboSTencentMay 2025128 layers of Mamba-2, attention and MoE feed-forward; 560B total / 56B active; described as "the first industry-deployed large-scale Mamba model" [35]
Nemotron Nano 2 (9B)NVIDIAAugust 2025Nemotron-H architecture: most attention layers replaced by Mamba-2 [36]
Granite 4.0 H modelsIBMOctober 2025Mamba-2 and Transformer blocks "sequentially in a 9:1 ratio" [38]
Nemotron 3 Nano (30B-A3B)NVIDIADecember 2025"Mixture-of-Experts hybrid Mamba-Transformer" [37][39]
Nemotron 3 Super (120B total / 12B active)NVIDIAMarch 2026Mamba-2 layers for most sequence processing, interleaved with attention, plus latent MoE [40]
Nemotron 3 Ultra (550B total / 55B active)NVIDIAJune 2026Hybrid Mamba-Transformer with latent MoE [41]

Expanded article table

Designs differ mainly in where attention goes. Jamba, Nemotron-H, Granite 4.0 and Hunyuan-TurboS interleave attention and Mamba layers; Hymba and Falcon-H1 run attention and SSM heads in parallel inside a layer; Zamba reuses one shared attention block [9][11][38][35][33][34][31].

Other SSM and Linear-Complexity Variants

The SSM family is part of a broader wave of architectures seeking to replace or complement Transformer attention with sub-quadratic alternatives.

H3 (2022)

H3 (Hungry Hungry Hippos), by Fu, Dao, Saab, Thomas, Rudra and Re of the Stanford group behind S4 (ICLR 2023), was designed after synthetic tasks showed that existing SSMs "struggle with two capabilities: recalling earlier tokens in the sequence and comparing tokens across the sequence" [12]. The H3 layer stacks two SSMs with multiplicative interactions, computing roughly Q⊙SSMdiag(SSMshift(K)⊙V)Q \odot \text{SSM}_{\text{diag}}(\text{SSM}_{\text{shift}}(K) \odot V): a shift SSM, which acts like a short convolution and can detect when an event occurs, and a diagonal SSM initialized from S4D, which can remember a token for the rest of the sequence [12]. H3 came within 0.4 perplexity of Transformers on OpenWebText, and a hybrid 125M-parameter model that kept two attention layers outperformed Transformers by 1.0 perplexity. The paper also introduced FlashConv, a fused FFT convolution kernel, and scaled hybrid H3-attention models to 2.7B parameters on the Pile [12].

Hyena (2023)

Hyena, also from the Stanford group (Poli et al., ICML 2023), replaced the SSM with "implicitly parametrized long convolutions and data-controlled gating": the convolution filters are generated by a small neural network rather than derived from state space equations [13]. The Mamba authors describe Hyena as using the same architecture as H3 with the S4 layer replaced by an MLP-parameterized global convolution [2]. Hyena's authors report reaching Transformer quality on language modeling with 20% less training compute at sequence length 2K, and operators twice as fast as optimized attention at 8K and 100x faster at 64K [13].

RWKV (2023-2025)

RWKV (Receptance Weighted Key Value) takes a different approach, combining the parallelizable training of Transformers with the efficient O(1) per-token inference of RNNs [14]. Developed primarily by Bo Peng (BlinkDL) and an open-source community, RWKV can be formulated as either a Transformer-like model (for parallel training) or an RNN (for efficient inference). RWKV uses a linear attention mechanism with learned decay factors rather than explicit state space equations.

The architecture has gone through multiple versions. RWKV-5 (Eagle) and RWKV-6 (Finch), described in a 2024 paper, introduced multi-headed matrix-valued states and a dynamic recurrence mechanism [14]. RWKV-7 (Goose) followed in March 2025 [42]. RWKV-X (April 2025) is a hybrid that combines RWKV for short-range modeling with a sparse attention mechanism for long-range context, while keeping linear-time training and constant-time decoding [43].

RetNet (2023)

Microsoft's RetNet (Retentive Network) frames efficient sequence modeling through a retention mechanism that can be computed in three equivalent ways: a parallel form (for training), a recurrent form (for O(1) inference), and a chunkwise recurrent form (for efficient long sequences) [15]. The Mamba authors describe RetNet as "based on Linear Attention and very similar to H3," with the inner S4 layer reduced to the special case of state dimension N = 1, so that it can be parallelized through a variant of multi-head attention rather than through convolutions [2]. In the SSD framework, RetNet corresponds to masked attention with a fixed exponential decay mask [5].

Griffin (Google DeepMind, 2024)

Google DeepMind's Griffin architecture mixes gated linear recurrences with local attention [16]. Its recurrent block uses the Real-Gated Linear Recurrent Unit (RG-LRU), a layer "inspired by the Linear Recurrent Unit" with gating motivated by LSTMs, for global sequence mixing, and sliding-window attention for local interactions [16]. The companion model Hawk uses only the gated linear recurrence; the authors report that Hawk exceeds the reported downstream performance of Mamba and that Griffin matches Llama-2 "despite being trained on over 6 times fewer tokens" [16]. Google released RecurrentGemma, open-weights models in 2B and 9B sizes based on the Griffin architecture [44].

Benchmarks and Empirical Results

The relative performance of SSMs, Transformers, and hybrids has been evaluated across multiple benchmarks and scales.

Language Modeling

At the 3B parameter scale, the Mamba authors report that Mamba matched or outperformed Transformers of equal size on perplexity and downstream tasks, while matching Transformers of twice the size [2]. At the 8B scale, NVIDIA's study found that pure Mamba-2 was competitive on many tasks but trailed Transformers on copying, in-context learning and long-context reasoning; the Mamba-2-Hybrid exceeded the Transformer on all 12 standard tasks and closely matched or exceeded it on average across 23 long-context tasks [8]. At 1.5B parameters, the Mamba-3 authors report that Mamba-3 (MIMO) beats a Transformer baseline by 2.2 points of average downstream accuracy [6].

Long-Range Arena

The Long-Range Arena (LRA) benchmark, designed specifically to test long-range dependency modeling, has been a standard evaluation suite for SSMs. S4 reached state of the art on every task in LRA, including the first solution to Path-X (sequence length 16,384) [1]. Later diagonal and MIMO variants reported LRA averages of 85% (S4D) and 87.4% (S5, with 98.5% on Path-X) [4][21].

Genomics and Audio

SSMs have done particularly well in domains with inherently long sequences. S4 halved the test error of specialized speech CNNs on raw speech classification with length-16,000 sequences, where RNN and Transformer baselines failed to learn [1]. The Mamba authors report that Mamba outperforms SaShiMi, Hyena and Transformers on audio waveform and DNA modeling, including more than halving FID on a challenging speech generation dataset, with gains continuing up to million-length contexts [2].

Inference Throughput

Mamba's inference advantage is most pronounced during autoregressive generation. Without a KV cache, Mamba can process much larger batch sizes on the same hardware; its authors report 4-5x higher throughput than a similarly sized Transformer on an A100 [2]. Hybrid models report smaller but still large gains, such as up to 3x for Nemotron-H at long contexts [11].

Limitations

Despite their advantages, SSMs have several notable limitations.

In-context learning and retrieval. The most significant limitation is weaker performance on tasks requiring precise information retrieval from context. Because SSMs compress all history into a fixed-size state vector, they struggle with tasks like phonebook lookup, where a specific fact must be retrieved from among many stored facts [8][28]. Transformers, with their O(n^2) attention, can directly access any position in the context.

Associative recall and copying. Related to in-context learning, SSMs and other gated-convolution models have difficulty with associative recall, where the model must remember arbitrary key-value pairs presented in context, and with copying long strings [27][28]. Jelassi and colleagues prove that copying with a fixed-size state fails unless the state grows linearly with sequence length [28]. Hybrid models that add a few attention layers largely close the gap in NVIDIA's 8B study, and the Zoology authors found that hybrids with input-dependent sparse attention closed 97.4% of the recall gap to attention in their experiments [8][27].

State tracking. Real-valued, non-negative transitions such as those of Mamba-2 cannot solve simple state-tracking problems such as parity; Mamba-3's complex-valued states and IBM's PD-SSM are two responses (see below) [6][51].

Ecosystem maturity. Training frameworks, inference engines and GPU kernels were designed first for Transformers. Support has broadened: as of October 2026, vLLM lists Mamba, Mamba-2, Jamba, Falcon-H1 and Granite 4.0 hybrid model classes among its supported models, and SSM layers depend on custom scan and SSD kernels rather than standard attention kernels [50][2][5].

Scaling. The Mamba paper noted in 2023 that its evaluation was limited to small models and that "it remains to assess whether Mamba still compares favorably" at larger sizes [2]. Pure SSM language models remain relatively small, but hybrids now reach large scales: Jamba 1.5 Large (398B total parameters, 2024), Hunyuan-TurboS (560B total, 2025) and Nemotron 3 Ultra (550B total, June 2026) [10][35][41].

Speed at short contexts. At short sequence lengths, optimized attention kernels can be faster than SSM kernels. The Mamba-2 authors caution that a Mamba-2 model may not be as efficient to train as a Transformer at 2K tokens, and a 2025 inference study found Transformers up to 1.9x faster below 8K tokens on consumer and embedded GPUs [5][7].

What are state space models used for?

SSMs have found applications across several domains:

  • Language modeling: Mamba, Mamba-2, and hybrid architectures serve as general-purpose language model backbones, in released models from AI21 Labs (Jamba), NVIDIA (Nemotron-H, Nemotron 3), IBM (Granite 4.0), TII (Falcon-H1) and others [9][11][38][34].
  • Genomics: DNA sequences can be hundreds of thousands of bases long, which makes linear scaling valuable; the Mamba authors report better pretraining quality than prior models on human-genome data, improving with context length up to a million bases [2].
  • Audio and speech: The continuous-signal origins of SSMs make them natural fits for raw audio. S4 was applied to raw speech classification, and Mamba to autoregressive audio waveform modeling and speech generation [1][2].
  • Computer vision: MambaVision (NVIDIA, CVPR 2025) introduced a hybrid Mamba-Transformer backbone that adds self-attention blocks in the final layers; its authors report state-of-the-art ImageNet-1K results in terms of both Top-1 accuracy and throughput for its variants [52].
  • Edge computing: SSMs' small, fixed state makes them attractive for resource-constrained hardware. At the May 2025 Embedded Vision Summit, BrainChip described TENN, a 1-billion-parameter model with 24 SSM layers that it said can run from read-only flash memory at under 0.5 watts, aimed at devices such as dashcams and medical devices [17].
  • Physical AI and robotics: NVIDIA used Nemotron-H-based vision-language models as base models for Cosmos-Reason1, its physical-AI reasoning models [11][30].

Current State (2025-2026)

As of October 2026, the main trend is hybridization. Pure SSM language models have not displaced Transformers, but hybrids that use SSM layers for most sequence mixing and keep a minority of attention layers have moved into large releases. NVIDIA's Nemotron 3 family, launched in December 2025, reached its largest size with Nemotron 3 Ultra (550B total, 55B active parameters) in June 2026, and IBM's Granite 4.0 (October 2025) uses a 9:1 Mamba-2 to Transformer ratio [37][41][38]. A parallel trend is linear-attention hybrids based on gated delta rules, such as Qwen3-Next and Kimi Linear, which are closely related to SSMs mathematically but are not described by their developers as SSMs [24][25][26].

The theoretical understanding of SSMs has deepened considerably. The SSD framework from Mamba-2 demonstrated a formal equivalence between structured SSMs and structured masked attention [5]. Mamba-3's complex-valued states and MIMO formulation addressed known expressivity and hardware-utilization gaps [6]. PD-SSM, by Aleksandar Terzić and colleagues at IBM Research - Zurich and ETH Zurich (a NeurIPS 2025 spotlight), parameterizes the transition matrix as the product of a column one-hot matrix and a complex-valued diagonal matrix, which keeps the cost of parallel scans linear in state size while letting one layer of dimension N emulate any N-state finite-state automaton [51].

A survey by Shriyank Somvanshi and colleagues, first posted in March 2025 and revised several times (most recently in August 2026 under the title "Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba"), traces the lineage from S4 through S4D, DSS, S5, Mamba and Mamba-2 to SSM-attention hybrids [18]. It concludes that SSMs "are competitive with Transformers and more efficient in specific long-context regimes, whereas attention retains an advantage on tasks dominated by exact retrieval and associative recall" [18].

The Transformer remains the dominant architecture for frontier AI systems. But SSMs have established themselves as a practical alternative, particularly for long-context, memory-constrained, or latency-sensitive applications, most often in combination with attention.

Explain Like I'm 5 (ELI5)

Imagine you are listening to a very long story. A Transformer is like writing down every single word of the story on sticky notes and spreading them all across a huge table so you can look at any two words at the same time. This works great, but it takes a lot of sticky notes and a really big table for a long story.

A state space model is more like keeping a small notebook. As you hear each word, you update your notes with the important stuff and erase what you do not need. You never need a bigger notebook no matter how long the story gets. The tricky part is deciding what to write down and what to erase. Mamba is a clever version that looks at each new word and decides on the spot whether it is important enough to remember.

Continue reading

More to read

Topics:Deep LearningMachine Learning

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25Gu, A., Goel, K., & Re, C. (2021). "Efficiently Modeling Long Sequences with Structured State Spaces." ICLR 2022 (Outstanding Paper Honorable Mention). arXiv:2111.00396. arxiv.org/...2111.00396
  2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48 ^49 ^50 ^51Gu, A. & Dao, T. (2023). "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." Conference on Language Modeling (COLM) 2024. arXiv:2312.00752. arxiv.org/...2312.00752
  3. ^1 ^2 ^3 ^4 ^5Gu, A., Dao, T., Ermon, S., Rudra, A., & Re, C. (2020). "HiPPO: Recurrent Memory with Optimal Polynomial Projections." NeurIPS 2020. arXiv:2008.07669. arxiv.org/...2008.07669
  4. ^1 ^2 ^3 ^4 ^5Gu, A., Gupta, A., Goel, K., & Re, C. (2022). "On the Parameterization and Initialization of Diagonal State Space Models." NeurIPS 2022. arXiv:2206.11893. arxiv.org/...2206.11893
  5. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18Dao, T. & Gu, A. (2024). "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality." ICML 2024. arXiv:2405.21060. arxiv.org/...2405.21060
  6. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026). "Mamba-3: Improved Sequence Modeling using State Space Principles." ICLR 2026 (oral). arXiv:2603.15569. arxiv.org/...2603.15569
  7. ^1 ^2 ^3Mitra, S., Karami, R., Xu, H., Huang, S., & Kwon, H. (2025). "Characterizing State Space Model and Hybrid Language Model Performance with Long Context." arXiv:2507.12442. arxiv.org/...2507.12442
  8. ^1 ^2 ^3 ^4 ^5 ^6 ^7Waleffe, R. et al. (2024). "An Empirical Study of Mamba-based Language Models." NVIDIA Research. arXiv:2406.07887. arxiv.org/...2406.07887
  9. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Lieber, O. et al. (2024). "Jamba: A Hybrid Transformer-Mamba Language Model." AI21 Labs. arXiv:2403.19887. arxiv.org/...2403.19887
  10. ^1 ^2 ^3 ^4AI21 Labs (Jamba Team). (2024). "Jamba-1.5: Hybrid Transformer-Mamba Models at Scale." arXiv:2408.12570. ai21.com/...brid-transformer-mamba-models-at-scale ; arxiv.org/...2408.12570
  11. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13NVIDIA. (2025). "Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models." arXiv:2504.03624. arxiv.org/...2504.03624
  12. ^1 ^2 ^3Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., & Re, C. (2022). "Hungry Hungry Hippos: Towards Language Modeling with State Space Models." ICLR 2023. arXiv:2212.14052. arxiv.org/...2212.14052
  13. ^1 ^2Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., & Re, C. (2023). "Hyena Hierarchy: Towards Larger Convolutional Language Models." ICML 2023. arXiv:2302.10866. arxiv.org/...2302.10866
  14. ^1 ^2 ^3Peng, B. et al. (2024). "Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence." arXiv:2404.05892. arxiv.org/...2404.05892
  15. ^Sun, Y. et al. (2023). "Retentive Network: A Successor to Transformer for Large Language Models." arXiv:2307.08621. arxiv.org/...2307.08621
  16. ^1 ^2 ^3 ^4De, S. et al. (2024). "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models." Google DeepMind. arXiv:2402.19427. arxiv.org/...2402.19427
  17. ^Farry, P. (2025). "State Space Models Can Enable AI in Low-Power Edge Computing." InfoQ, 24 July 2025 (report on BrainChip's presentation at the 2025 Embedded Vision Summit). infoq.com/...state-space-models-edge-compute
  18. ^1 ^2Somvanshi, S., Islam, M. M., Mimi, M. S., Polock, S. B. B., Chhetri, G., Dutta, A., Rafe, A., & Das, S. (2025, revised 2026). "Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba" (earlier versions titled "A Survey on Structured State Space Sequence (S4) Models" and "From S4 to Mamba: A Comprehensive Survey on Structured State Space Models"). arXiv:2503.18970. arxiv.org/...2503.18970
  19. ^Cartesia. "Company." cartesia.ai/company ; Tri Dao, personal website. tridao.me
  20. ^Gupta, A., Gu, A., & Berant, J. (2022). "Diagonal State Spaces are as Effective as Structured State Spaces." NeurIPS 2022. arXiv:2203.14343. arxiv.org/...2203.14343
  21. ^1 ^2 ^3Smith, J. T. H., Warrington, A., & Linderman, S. W. (2022). "Simplified State Space Layers for Sequence Modeling." ICLR 2023. arXiv:2208.04933. arxiv.org/...2208.04933
  22. ^Dao, T. & Gu, A. (2024). "State Space Duality (Mamba-2) Part III - The Algorithm." Blog post, 31 May 2024. tridao.me/...mamba2-part3-algorithm
  23. ^OpenReview. "Mamba-3: Improved Sequence Modeling using State Space Principles." ICLR 2026 Oral. openreview.net/forum
  24. ^1 ^2Yang, S., Kautz, J., & Hatamizadeh, A. (2024). "Gated Delta Networks: Improving Mamba2 with Delta Rule." ICLR 2025. arXiv:2412.06464. arxiv.org/...2412.06464
  25. ^1 ^2Qwen Team. "Qwen3-Next-80B-A3B-Instruct." Hugging Face model card. huggingface.co/...Qwen3-Next-80B-A3B-Instruct
  26. ^1 ^2Kimi Team. (2025). "Kimi Linear: An Expressive, Efficient Attention Architecture." arXiv:2510.26692. arxiv.org/...2510.26692
  27. ^1 ^2 ^3 ^4Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., & Re, C. (2023). "Zoology: Measuring and Improving Recall in Efficient Language Models." ICLR 2024. arXiv:2312.04927. arxiv.org/...2312.04927
  28. ^1 ^2 ^3 ^4 ^5 ^6Jelassi, S., Brandfonbrener, D., Kakade, S. M., & Malach, E. (2024). "Repeat After Me: Transformers are Better than State Space Models at Copying." ICML 2024. arXiv:2402.01032. arxiv.org/...2402.01032
  29. ^1 ^2 ^3AI21 Labs. (2024). "Introducing Jamba: AI21's Groundbreaking SSM-Transformer Model." AI21 blog, 28 March 2024. ai21.com/...announcing-jamba
  30. ^1 ^2NVIDIA. (2025). "Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning." arXiv:2503.15558. arxiv.org/...2503.15558
  31. ^1 ^2Glorioso, P., Anthony, Q., Tokpanov, Y., Whittington, J., Pilault, J., Ibrahim, A., & Millidge, B. (2024). "Zamba: A Compact 7B SSM Hybrid Model." arXiv:2405.16712. arxiv.org/...2405.16712
  32. ^Ren, L., Liu, Y., Lu, Y., Shen, Y., Liang, C., & Chen, W. (2024). "Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling." ICLR 2025. arXiv:2406.07522. arxiv.org/...2406.07522
  33. ^1 ^2Dong, X., Fu, Y., et al. (2024). "Hymba: A Hybrid-head Architecture for Small Language Models." arXiv:2411.13676. arxiv.org/...2411.13676
  34. ^1 ^2 ^3 ^4Technology Innovation Institute. (2025). "Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance." Falcon blog, 20 May 2025; technical report arXiv:2507.22448. falcon-lm.github.io/...falcon-h1
  35. ^1 ^2 ^3Tencent Hunyuan Team. (2025). "Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought." arXiv:2505.15431. arxiv.org/...2505.15431
  36. ^NVIDIA. (2025). "NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model." arXiv:2508.14444. arxiv.org/...2508.14444
  37. ^1 ^2 ^3NVIDIA. (2025). "NVIDIA Debuts Nemotron 3 Family of Open Models." NVIDIA Newsroom, 15 December 2025. nvidianews.nvidia.com/...n-3-family-of-open-models
  38. ^1 ^2 ^3 ^4 ^5IBM. (2025). "IBM Granite 4.0: hyper-efficient, high performance hybrid models for enterprise." IBM announcement, 2 October 2025. ibm.com/...fficient-high-performance-hybrid-models
  39. ^NVIDIA. (2025). "Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning." arXiv:2512.20848. arxiv.org/...2512.20848
  40. ^Alexiuk, C. & Patel, C. (2026). "Introducing Nemotron 3 Super: An Open Hybrid Mamba-Transformer MoE for Agentic Reasoning." NVIDIA Technical Blog, 11 March 2026. developer.nvidia.com/...-moe-for-agentic-reasoning
  41. ^1 ^2 ^3Alexiuk, C. & Patel, C. (2026). "NVIDIA Nemotron 3 Ultra Powers Faster, More Efficient Reasoning for Long-Running Agents." NVIDIA Technical Blog, 4 June 2026. developer.nvidia.com/...ng-for-long-running-agents
  42. ^Peng, B. et al. (2025). "RWKV-7 'Goose' with Expressive Dynamic State Evolution." arXiv:2503.14456. arxiv.org/...2503.14456
  43. ^Hou, H., Huang, Z., Tan, K., Lu, R., & Yu, F. R. (2025). "RWKV-X: A Linear Complexity Hybrid Language Model." arXiv:2504.21463. arxiv.org/...2504.21463
  44. ^Botev, A., De, S., Smith, S. L., et al. (2024). "RecurrentGemma: Moving Past Transformers for Efficient Open Language Models." arXiv:2404.07839. arxiv.org/...2404.07839
  45. ^Carnegie Mellon University Computer Science Department. "Albert Gu" (faculty profile, Assistant Professor, Machine Learning Department). csd.cmu.edu/...albert-gu
  46. ^OpenReview. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." COLM 2024 (openreview.net/forum) and ICLR 2024 submission (openreview.net/forum).
  47. ^state-spaces. Mamba model configurations (e.g. mamba-130m, mamba-2.8b), config.json. Hugging Face. huggingface.co/...mamba-2.8b
  48. ^Glorioso, P., Anthony, Q., Tokpanov, Y., et al. (2024). "The Zamba2 Suite: Technical Report." arXiv:2411.15242. arxiv.org/...2411.15242
  49. ^IBM, Princeton, CMU and UIUC. (2024). "Bamba: Inference-Efficient Hybrid Mamba2 Model." Hugging Face blog, 18 December 2024. huggingface.co/...bamba
  50. ^vLLM. "Supported Models." vLLM documentation (retrieved October 2026). docs.vllm.ai/...supported_models
  51. ^1 ^2Terzić, A., Menet, N., Hersche, M., Hofmann, T., & Rahimi, A. (2025). "Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models." NeurIPS 2025 (spotlight). arXiv:2509.22284. arxiv.org/...2509.22284
  52. ^Hatamizadeh, A. & Kautz, J. (2024). "MambaVision: A Hybrid Mamba-Transformer Vision Backbone." CVPR 2025. arXiv:2407.08083. arxiv.org/...2407.08083

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

10 revisions · v10 · 8,971 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independently verified 2026-10-31 against the HiPPO, S4, S4D, DSS, S5, H3, Hyena, Mamba, Mamba-2 and Mamba-3 papers, recall studies and hybrid-model reports; 7 minor verifier defects fixed and 2 unconfirmed RWKV venue claims removed before stamping.

Cite this page: AI Wiki. "State space model (deep learning)." aiwiki.ai, updated 31 Oct 2026, fact-checked 31 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/state_space_model

Edit article