# Mamba

> Source: https://aiwiki.ai/wiki/mamba
> Summary: Mamba is a neural network architecture for sequence modeling that uses selective state space models (SSMs) to process sequential data in linear time with respect to sequence length, offering an alternative to the transformer whose attention mechanism scales quadratically.
> Updated: 2026-10-31
> Fact-checked: 2026-10-31
> Categories: Large Language Models, Machine Learning, Model Architecture
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Mamba." aiwiki.ai, 31 Oct 2026. https://aiwiki.ai/wiki/mamba
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

*See also: [transformer](https://aiwiki.ai/wiki/transformers), [attention mechanism](https://aiwiki.ai/wiki/attention), [recurrent neural network](https://aiwiki.ai/wiki/recurrent_neural_network), [large language model](https://aiwiki.ai/wiki/large_language_model), [Mamba-2](https://aiwiki.ai/wiki/mamba_2), [state space model](https://aiwiki.ai/wiki/state_space_model)*

Mamba is a [neural network](https://aiwiki.ai/wiki/neural_network) architecture for [sequence modeling](https://aiwiki.ai/wiki/sequence_model) that uses selective state space models (SSMs) to process sequential data in linear time with respect to sequence length, offering an alternative to the transformer whose attention mechanism scales quadratically. It was introduced by [Albert Gu](https://aiwiki.ai/wiki/albert_gu) and [Tri Dao](https://aiwiki.ai/wiki/tri_dao) on 1 December 2023 in the paper "Mamba: Linear-Time Sequence Modeling with Selective State Spaces."[1] The paper was submitted to ICLR 2024, where OpenReview records it as a rejected submission, and was later published at the Conference on Language Modeling (COLM) 2024.[22][23] In benchmarks reported by its authors, Mamba has 5x higher generation throughput than Transformers of similar size (measured on an A100 GPU), scales linearly in sequence length, keeps improving on real data up to million-length sequences, and at the 3B-parameter scale outperforms Transformers of the same size while matching Transformers twice its size.[1]

The central idea behind Mamba is making the parameters of a state space model depend on the input, allowing the model to selectively propagate or forget information along the sequence depending on the current token.[1] In the words of Gu and Dao, "simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token."[1] Because input-dependent parameters rule out the convolutional shortcut used by earlier SSMs, the authors pair the selection mechanism with a hardware-aware parallel scan that keeps the expanded state in fast GPU memory.[1]

Mamba language models were pretrained at five sizes (130M, 370M, 790M, 1.4B, and 2.8B parameters) on 300 billion tokens of the Pile dataset.[1][12] The authors report that Mamba-3B (the 2.8B model) matches Transformers twice its size in both pretraining and downstream evaluation; for example, its average zero-shot accuracy is about 4 points higher than Pythia-2.8B and exceeds Pythia-6.9B.[1] As a general [sequence model](https://aiwiki.ai/wiki/sequence_model) backbone, Mamba was also evaluated on audio waveforms and DNA sequences.[1]

Since the original paper, Mamba layers have been used in released models from [Mistral AI](https://aiwiki.ai/wiki/mistral_ai), [AI21 Labs](https://aiwiki.ai/wiki/ai21_labs), the Technology Innovation Institute (TII), [Nvidia](https://aiwiki.ai/wiki/nvidia), IBM, [Zyphra](https://aiwiki.ai/wiki/zyphra) and [Tencent](https://aiwiki.ai/wiki/tencent), mostly in hybrid designs that keep a minority of attention layers. Libraries including [Hugging Face transformers](https://aiwiki.ai/wiki/transformers_library), [vLLM](https://aiwiki.ai/wiki/vllm), [TensorRT](https://aiwiki.ai/wiki/tensorrt)-LLM and [llama.cpp](https://aiwiki.ai/wiki/llama_cpp) support Mamba models. The authors followed the original with Mamba-2 (2024) and [Mamba-3](https://aiwiki.ai/wiki/mamba_3) (2026), and the architecture seeded many variants, including MambaByte and Vision Mamba.

## At a glance

| Property | Value |
|----------|-------|
| Architecture type | Selective state space model (SSM), called S6 in the paper[1] |
| Introduced | 1 December 2023 ([arXiv](https://aiwiki.ai/wiki/arxiv):2312.00752)[1] |
| Authors | [Albert Gu](https://aiwiki.ai/wiki/albert_gu) (Carnegie Mellon), [Tri Dao](https://aiwiki.ai/wiki/tri_dao) (Princeton)[1] |
| Publication | Rejected at ICLR 2024; published at COLM 2024[22][23] |
| Time complexity | $$O(L)$$ in sequence length $$L$$ (vs $$O(L^2)$$ for attention)[1] |
| Inference throughput | 4-5x higher than a Transformer of similar size in the authors' A100 benchmark[1] |
| Pretraining scales | 130M, 370M, 790M, 1.4B, 2.8B parameters on 300B Pile tokens[1][12] |
| Headline result | Mamba-3B matches Transformers twice its size (authors' evaluation)[1] |
| Reference implementation | `mamba-ssm` (state-spaces/mamba on GitHub), Apache 2.0; latest release v2.3.2.post1 (9 May 2026)[24] |
| Direct successors | [Mamba-2](https://aiwiki.ai/wiki/mamba_2) (May 2024), [Mamba-3](https://aiwiki.ai/wiki/mamba_3) (March 2026)[4][5] |

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/9dSkvxS2EB0" title="Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Paper Explained)"></iframe>
<figcaption>"Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Paper Explained)", video from Yannic Kilcher, published December 24, 2023. <a href="https://www.youtube.com/watch?v=9dSkvxS2EB0">Watch on YouTube</a>.</figcaption>
</figure>

## Background: state space models

This section summarizes only what is needed to follow Mamba; the general theory, S4 and its relatives are covered in [state space model](https://aiwiki.ai/wiki/state_space_model).

### Continuous-time formulation and discretization

The structured SSMs that Mamba builds on are inspired by a continuous system from [control theory](https://aiwiki.ai/wiki/control_theory) that maps a one-dimensional input $$x(t)$$ to an output $$y(t)$$ through an implicit latent state $$h(t) \in \mathbb{R}^N$$:[1]

- **State equation:** $$h'(t) = A h(t) + B x(t)$$
- **Output equation:** $$y(t) = C h(t)$$

To apply the model to discrete sequences such as tokens or audio samples, a step size $$\Delta$$ converts the continuous parameters $$(\Delta, A, B)$$ into discrete parameters $$(\bar{A}, \bar{B})$$. The Mamba paper uses the zero-order hold (ZOH) rule as its example:[1]

$$
\bar{A} = \exp(\Delta A)
$$

$$
\bar{B} = (\Delta A)^{-1} (\exp(\Delta A) - I) \cdot \Delta B
$$

after which the model runs as the recurrence $$h_t = \bar{A} h_{t-1} + \bar{B} x_t$$, $$y_t = C h_t$$.[1] The Mamba-3 paper later pointed out that, although Mamba-1 stated the ZOH formula, the released Mamba-1 and Mamba-2 implementations use a further approximation for the input term, $$h_t \approx e^{\Delta_t A_t} h_{t-1} + \Delta_t B_t x_t$$, which the Mamba-3 authors name "exponential-Euler" discretization.[5]

When $$(\Delta, A, B, C)$$ are fixed for all time steps, the model is linear time invariant (LTI). An LTI SSM can be computed either as a linear recurrence or as one global convolution with kernel $$\bar{K} = (C\bar{B}, C\bar{A}\bar{B}, \ldots)$$, so earlier models trained in the parallel convolutional mode and switched to the recurrent mode for autoregressive inference.[1] The Mamba authors note that all structured SSMs before Mamba were LTI "because of fundamental efficiency constraints," and argue that LTI models have fundamental limitations on certain kinds of data.[1]

### S4 and its successors

The direct ancestor of Mamba is S4 (Structured State Space sequence model), introduced by Albert Gu, Karan Goel, and Christopher Re in October 2021 and published at ICLR 2022, where it received an Outstanding Paper Honorable Mention.[2] S4 built on the HiPPO framework (Gu, Dao, [Stefano Ermon](https://aiwiki.ai/wiki/stefano_ermon), Atri Rudra, and Re, NeurIPS 2020), which derives state matrices for online compression of history by projection onto polynomial bases.[3] S4 conditioned the state matrix with a low-rank correction so that it could be diagonalized stably and computed through a Cauchy kernel. Its authors reported 91% accuracy on sequential [CIFAR-10](https://aiwiki.ai/wiki/cifar_10) with no [data augmentation](https://aiwiki.ai/wiki/data_augmentation) or auxiliary losses, generation 60x faster than Transformers, and state-of-the-art results on every Long Range Arena task, including the length-16k Path-X task "that all prior work fails on."[2]

| Model | First posted | Key contribution |
|-------|--------------|------------------|
| S4 | October 2021 | Low-rank-corrected (NPLR) parameterization of the HiPPO state matrix[2] |
| DSS (Diagonal State Spaces) | March 2022 | Showed S4's performance can be matched with purely diagonal state matrices[26] |
| S4D | June 2022 | Systematic study of diagonal SSM parameterization and initialization[27] |
| S5 | August 2022 | One multi-input, multi-output SSM computed with parallel scans[28] |
| H3 (Hungry Hungry Hippos) | December 2022 | SSM layer with multiplicative gating designed for recall in language modeling (ICLR 2023)[29] |
| Mamba | December 2023 | Input-dependent (selective) SSM parameters, computed with a hardware-aware scan[1] |

Mamba uses the diagonal structure popularized by these models, and its default real-valued initialization, S4D-Real, sets the $$n$$-th diagonal element of $$A$$ to $$-(n+1)$$, an initialization the paper describes as based on HiPPO theory.[1]

## How does Mamba's selective scan mechanism work?

### The selectivity problem

The core limitation of earlier SSM architectures was linear time invariance. Because the parameters stayed identical for every token regardless of content, these models could not perform content-aware filtering.[1] Consider a [language model](https://aiwiki.ai/wiki/language_model) processing the sentence "The cat sat on the mat": an LTI model applies the same dynamics to the word "the" as to the word "cat," even though they carry very different amounts of information.

The Mamba authors diagnosed this directly: "a key weakness of such models is their inability to perform content-based reasoning."[1] They frame sequence modeling as a problem of compressing context into a state: attention is effective but inefficient because it does not compress context at all (it keeps the whole [KV cache](https://aiwiki.ai/wiki/kv_cache)), while recurrent models are efficient because their state is finite but are limited by how well that state compresses the context.[1] [Self-attention](https://aiwiki.ai/wiki/self_attention) achieves content-based reasoning at $$O(L^2)$$ training cost; Mamba aims to do so while keeping $$O(L)$$ complexity.[1]

The paper demonstrates the problem with two synthetic tasks. In Selective Copying, the positions of the tokens to memorize vary randomly, so a model must recognize them by content. In [Induction Heads](https://aiwiki.ai/wiki/induction_heads), a model must recall the token that followed an earlier occurrence of the current token. On Selective Copying, an LTI S4 layer without gating reached 18.3% accuracy, while the selective layer (S6) reached 97.0% on its own and 99.8% inside the full Mamba architecture.[1] On Induction Heads, a two-layer Mamba trained on sequences of length 256 generalized perfectly to sequences of length 1,048,576, about 4,000 times longer than training, while no other tested method went beyond 2x.[1]

### Input-dependent parameters

Mamba makes three SSM parameters functions of the input:[1]

| Parameter | Role | How it becomes selective |
|-----------|------|-------------------------|
| $$B$$ (input matrix) | Controls how input enters the state | Linear projection of the input to dimension $$N$$; different for each token |
| $$C$$ (output matrix) | Controls how the state maps to output | Linear projection of the input to dimension $$N$$; different for each token |
| $$\Delta$$ (step size) | Controls discretization resolution | Low-rank projection of the input added to a learned bias, passed through softplus; different for each token and channel |

As a result $$\bar{A}$$ and $$\bar{B}$$ gain batch and length dimensions, which turns the model from time-invariant into time-varying.[1] The state matrix $$A$$ is not made input-dependent. The paper explains that $$A$$ "ultimately affects the model only through its interaction with $$\Delta$$ via $$\bar{A} = \exp(\Delta A)$$," so selectivity in $$\Delta$$ already makes $$(\bar{A}, \bar{B})$$ selective; the authors hypothesize that a selective $$A$$ would perform similarly and leave it out for simplicity.[1]

The step size $$\Delta$$ acts as the main selection mechanism. In the paper's words, "a large $$\Delta$$ resets the state $$h$$ and focuses on the current input $$x$$, while a small $$\Delta$$ persists the state and ignores the current input."[1] The authors prove (Theorem 1) that with $$N = 1$$, $$A = -1$$, $$B = 1$$ and a softplus step size, the selective recurrence reduces to the gated update $$h_t = (1 - g_t) h_{t-1} + g_t x_t$$, so the classical gating of RNNs such as [LSTMs](https://aiwiki.ai/wiki/lstm) is a special case of the selection mechanism.[1] Mamba's SSM state is real-valued by default; the paper switched to complex values only for its audio-waveform experiments.[1]

### Parallel scan algorithm

Making parameters input-dependent means the model can no longer be computed as a fixed convolution, so Mamba must use the recurrent form. Although the recurrence is time-varying, it can still be parallelized with a work-efficient parallel scan (prefix-sum) algorithm, following Blelloch (1990) and earlier SSM work such as S5.[1] The paper also observes that the recurrent computation uses $$O(BLDN)$$ FLOPs versus $$O(BLD \log L)$$ for the convolutional one, with a lower constant factor, so for long sequences and a moderate state dimension the recurrence can actually use fewer FLOPs.[1]

The paper abbreviates the selective SSM as "S6," because it is an S4 model with a selection mechanism, computed with a scan.[1]

## Hardware-aware implementation

The selective scan creates a memory problem. With $$D$$ channels and state size $$N$$, the latent state over a batch has shape $$(B, L, D, N)$$, a factor of $$N$$ larger than the input and output; materializing it in GPU high-bandwidth memory (HBM) would make the model slow and memory-hungry.[1] Mamba addresses this with three classical techniques the authors name explicitly: kernel fusion, parallel scan, and recomputation.[1] The approach follows the IO-aware design of [FlashAttention](https://aiwiki.ai/wiki/flash_attention), also co-developed by Tri Dao, and the repository describes the implementation as "in the spirit of FlashAttention."[24][10]

**Kernel fusion.** A standard implementation would prepare the discretized parameters of size $$(B, L, D, N)$$ in HBM, run a scan that writes outputs of the same size back to HBM, and then multiply by $$C$$. Mamba instead loads $$(\Delta, A, B, C)$$ from HBM into fast on-chip SRAM, performs the discretization, the scan and the multiplication by $$C$$ in SRAM within one fused kernel, and writes only the final $$(B, L, D)$$ output back to HBM. This reduces memory IO by a factor of $$O(N)$$, which the authors report speeds up the operation by 20 to 40 times.[1] Sequences too long to fit in SRAM are split into chunks, with the scan state carried from one chunk to the next.[1]

**Parallel scan.** The fused kernel runs a parallel associative scan over the sequence, so the recurrence does not have to be computed one step at a time.[1]

**Recomputation.** The intermediate states are needed for backpropagation but are not saved. They are recomputed in the backward pass when the inputs are reloaded from HBM to SRAM, which the authors note is faster than storing them and reading them back. Recomputation is also applied to other cheap-to-recompute activations in the block, so the fused selective SSM layer has about the same activation memory as an optimized Transformer layer using FlashAttention.[1]

The authors report that their scan is faster than FlashAttention-2 beyond sequence length 2K and up to 20-40x faster than a standard scan implementation in [PyTorch](https://aiwiki.ai/wiki/pytorch), measured on an A100 80GB GPU with state dimension $$N = 16$$.[1] Their introduction summarizes the resulting implementation as faster than previous SSM methods both in theory (linear rather than pseudo-linear scaling) and on hardware, "up to 3x faster on A100 GPUs."[1] The reference [CUDA](https://aiwiki.ai/wiki/cuda) kernel ships in the `mamba-ssm` package, which later releases extended with Mamba-2 and Mamba-3 modules.[24]

## Mamba architecture

The full Mamba architecture wraps the selective SSM in a single block that is stacked homogeneously. Prior SSM architectures such as H3 alternated an attention-like SSM block with an MLP block; Mamba merges the two into one block, a simplification the authors say was inspired by the gated attention unit (GAU), which did something similar for attention.[1] The paper describes the result as combining "the H3 block, which is the basis of most SSM architectures, with the ubiquitous MLP block of modern neural networks."[1] Each block works as follows:[1]

1. The input of dimension $$D$$ is expanded by a factor $$E$$ (fixed to $$E = 2$$ in the paper) through two parallel linear projections.
2. One branch passes through a short 1D convolution, a SiLU activation, and the selective SSM.
3. The other branch passes through a SiLU activation and acts as a multiplicative gate.
4. The two branches are multiplied element-wise.
5. A final linear projection maps back to dimension $$D$$.

Compared with the H3 block, Mamba replaces the first multiplicative gate with an activation function; compared with a gated MLP, it adds an SSM to the main branch, and the SiLU choice makes the gated MLP part the "SwiGLU" variant used in models such as [LLaMA](https://aiwiki.ai/wiki/llama).[1] Most of a block's parameters ($$3ED^2$$) sit in the linear projections, and two Mamba blocks roughly match the $$12D^2$$ parameters of one Transformer attention-plus-MLP pair.[1] The block also includes an optional normalization layer (the authors chose LayerNorm, following RetNet), and blocks are interleaved with standard normalization and residual connections.[1] The authors describe Mamba as "a simplified end-to-end neural network architecture without attention or even MLP blocks."[1]

Because two Mamba blocks replace one Transformer layer, the released language models have about twice as many layers as GPT-3-style Transformers of the same size. Their published configurations use 24 layers at width 768 (130M), 48 layers at widths 1024, 1536 and 2048 (370M, 790M and 1.4B), and 64 layers at width 2560 (2.8B), with [RMSNorm](https://aiwiki.ai/wiki/rmsnorm) enabled.[25]

## How does Mamba differ from a transformer?

| Aspect | [Transformer](https://aiwiki.ai/wiki/transformer) | Mamba |
|--------|-------------|-------|
| Core mechanism | [Self-attention](https://aiwiki.ai/wiki/self_attention) | Selective state space model |
| Training cost (sequence length $$L$$) | $$O(L^2)$$ per layer | $$O(L)$$ per layer |
| Training parallelism | Fully parallel (attention matrix) | Parallel (via parallel scan) |
| Inference mode | Autoregressive with [KV cache](https://aiwiki.ai/wiki/kv_cache) | Recurrent with fixed-size state |
| Inference memory | KV cache grows linearly with context | Fixed-size state (constant per step) |
| Content-aware reasoning | Yes (attention computes pairwise token interactions) | Yes (input-dependent SSM parameters) |
| [Context window](https://aiwiki.ai/wiki/context_window) | Fixed maximum (requires extension techniques) | No architectural maximum; quality depends on what the state retains |
| Long-range dependencies | Attention can directly connect any two positions | State carries compressed history |
| Copying and retrieval from context | Strong (direct access to all past tokens) | Weaker (information must survive state compression)[9] |
| Position encoding | Usually required (sinusoidal, RoPE, ALiBi, etc.) | Not used in the original language models |
| Inference throughput (similar size) | Baseline | 4-5x higher in the authors' benchmark[1] |

The fundamental trade-off is between expressiveness and efficiency. Transformers keep an uncompressed record of the sequence in the KV cache, which allows direct access to any past token. Mamba compresses the history into a fixed-size hidden state, which is far more memory-efficient but means information can only be used if it survived compression.[1] Later studies found that this makes pure SSMs weaker than Transformers on tasks that require copying or exact retrieval from context, such as looking up an entry in a long phone book.[9][8]

## What benchmark results did Mamba achieve?

All numbers in this section are the authors' own results from the Mamba paper.[1]

### Language modeling

In scaling-law experiments on the Pile (models from about 125M to 1.3B parameters, Chinchilla-style token budgets), the authors report that Mamba is "the first attention-free model to match the performance of a very strong Transformer recipe (Transformer++)," a LLaMA-style baseline with rotary embeddings, SwiGLU and RMSNorm, with the advantage growing at longer sequence length.[1]

For zero-shot evaluation, Mamba models trained on 300B Pile tokens were compared with Pythia and RWKV models trained on the same data, tokenizer and token count. Selected rows from the paper's Table 3:[1]

| Model | Pile perplexity (lower is better) | LAMBADA accuracy | HellaSwag | Average accuracy (6 tasks) |
|-------|-----------------|------------------|-----------|------------------|
| Pythia-1.4B | 7.51 | 61.7 | 52.1 | 55.2 |
| RWKV-1.5B | 7.70 | 56.4 | 52.5 | 54.3 |
| Mamba-1.4B | 6.80 | 64.9 | 59.1 | 59.7 |
| Pythia-2.8B | 6.73 | 64.7 | 59.3 | 59.1 |
| RWKV-3B | 7.00 | 63.9 | 59.6 | 59.6 |
| Mamba-2.8B | 6.22 | 69.2 | 66.1 | 63.3 |
| Pythia-6.9B | 6.51 | 67.1 | 64.0 | 61.7 |
| RWKV-7.4B | 6.31 | 67.2 | 65.5 | 62.5 |

The table caption states that Mamba is best-in-class on every evaluation at each model size and "generally matches baselines at twice the model size."[1]

### Audio

On YouTubeMix piano audio, Mamba improved on the SaShiMi (S4 plus MLP) baseline in autoregressive pretraining at every context length tested, up to minute-long (about one million sample) sequences, with the gap widening at longer lengths.[1] On the SC09 speech-generation benchmark (one-second spoken digits at 16 kHz), a 6.1M-parameter Mamba model outperformed larger GAN- and [diffusion](https://aiwiki.ai/wiki/diffusion_model)-based baselines:[1]

| Model | Parameters | FID (lower is better) | Inception score (higher is better) |
|-------|-----------|-----|----|
| [WaveNet](https://aiwiki.ai/wiki/wavenet) | 4.2M | 5.08 | 2.27 |
| SaShiMi | 5.8M | 1.99 | 5.13 |
| WaveGAN | 19.1M | 2.03 | 4.90 |
| DiffWave + SaShiMi | 23.0M | 1.42 | 5.94 |
| Mamba | 6.1M | 0.94 | 6.26 |
| Mamba | 24.3M | 0.67 | 7.33 |

The larger, parameter-matched Mamba cut FID by more than half relative to the best prior model, which the paper's introduction highlights.[1]

### DNA sequence modeling

On the HG38 human-genome dataset, Mamba's pretraining perplexity scaled better with model size than HyenaDNA and Transformer++; at about 40M parameters the authors estimate Mamba matches those baselines with roughly 3x to 4x fewer parameters.[1] When sequence length was increased from 1,024 to 1,048,576 at fixed model size, Mamba's perplexity kept improving while HyenaDNA's got worse.[1] On a harder version of a species-classification task, which asks the model to distinguish human, chimpanzee, gorilla, orangutan and bonobo DNA (species "known to share 99% of their DNA"), pretrained Mamba models were fine-tuned on sequences up to one million base pairs long.[1]

### Inference speed

The end-to-end throughput benchmark generated 128 tokens after a 2,048-token prompt, at batch sizes from 1 to 128, on an A100 80GB PCIe GPU, comparing Mamba-1.4B and an untrained Mamba-6.9B with GPT-3-style Transformers of 1.3B and 6.7B parameters using the Hugging Face implementation.[1] The authors report that Mamba achieves 4-5x higher inference throughput "since without the KV cache it can use much higher batch sizes," and that an untrained Mamba-6.9B would have higher throughput than a Transformer-1.3B five times smaller.[1] In a training-memory comparison of 125M models, Mamba used slightly more memory than a Transformer with FlashAttention-2 (for example 4.8 GB versus 4.6 GB at batch size 1).[1]

## Mamba-2: structured state space duality

In May 2024, Tri Dao and Albert Gu published "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality," introducing [Mamba-2](https://aiwiki.ai/wiki/mamba_2); the paper appeared at ICML 2024.[4] Its structured state space duality (SSD) framework shows that SSMs and a family of attention variants are different ways of multiplying by the same class of structured semiseparable matrices. In the dual form, the SSD layer computes $$(L \circ QK^\top) \cdot V$$, attention without the softmax and with an extra input-dependent mask $$L$$ whose entries are products of scalar decay terms $$a_i \in [0, 1]$$.[4]

Compared with Mamba-1's selective SSM, the SSD layer restricts $$A$$ from diagonal to scalar times identity and uses a head dimension $$P$$ of typically 64 or 128 instead of $$P = 1$$; the authors describe these changes as "slightly decreasing the expressive power in return for significant training efficiency improvements," chiefly the ability to use matrix-multiplication units.[4] They report that a dedicated SSD implementation is 2-8x faster than Mamba's optimized selective scan while allowing recurrent states 8x larger or more with minimal slowdown, and that it crosses over FlashAttention-2 at sequence length 2K and is 6x faster at 16K.[4] In hybrid experiments, the paper found that having around 10% of layers be attention worked best.[4] The `mamba-ssm` package added Mamba-2 in its v2.0.0 release on 3 June 2024.[24] Full details are in the [Mamba-2](https://aiwiki.ai/wiki/mamba_2) article.

## Mamba-3

Mamba-3 was posted on arXiv on 16 March 2026 by Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, [J. Zico Kolter](https://aiwiki.ai/wiki/zico_kolter), Tri Dao, and Albert Gu, and was presented as an oral paper at ICLR 2026.[5][30] Framed around an "inference-first perspective," it makes three changes:[5]

- **Exponential-trapezoidal discretization**, a second-order generalization of the exponential-Euler rule used in practice by Mamba-1 and Mamba-2, which the authors show is equivalent to applying a data-dependent convolution of size two.[5]
- **Complex-valued state updates**, which the authors prove are equivalent to applying a data-dependent rotary embedding ([RoPE](https://aiwiki.ai/wiki/rotary_position_embedding)) to the $$B$$ and $$C$$ projections, letting the model solve synthetic state-tracking tasks such as parity on which Mamba-2 performs no better than random guessing.[5]
- **A multi-input, multi-output (MIMO) formulation** that increases decoding FLOPs by up to 4x relative to Mamba-2 at a fixed state size while keeping similar wall-clock decode latency.[5]

At the 1.5B scale, the authors report that Mamba-3 improves average downstream accuracy by 0.6 points over the next best model, [Gated DeltaNet](https://aiwiki.ai/wiki/gated_deltanet), with the MIMO variant adding another 1.2 points, and that Mamba-3 (MIMO) with state size 64 matches the perplexity of Mamba-2 with state size 128.[5] The Mamba-3 code was released in `mamba-ssm` v2.3.2 on 9 May 2026.[24] See [Mamba-3](https://aiwiki.ai/wiki/mamba_3) for details.

## MambaByte and byte-level sequence modeling

MambaByte, by Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M. Rush (posted January 2024, published at COLM 2024), is a token-free adaptation of Mamba trained autoregressively on raw bytes.[13] Conventional large language models operate on subword tokens produced by a [BPE](https://aiwiki.ai/wiki/byte_pair_encoding) or [SentencePiece](https://aiwiki.ai/wiki/sentencepiece) tokenizer; removing the tokenizer removes that [inductive bias](https://aiwiki.ai/wiki/inductive_bias) but produces significantly longer sequences, which is costly for Transformers whose memory grows with sequence length.[13]

The authors report that MambaByte is competitive with, and in some settings outperforms, state-of-the-art subword Transformers on language modeling while keeping the noise robustness of token-free models.[13] Their largest model, MambaByte-972M, outperformed the other byte-level models they compared.[13] They also adapted [speculative decoding](https://aiwiki.ai/wiki/speculative_decoding) with tokenized drafting and byte-level verification, giving a 2.6x inference speedup over the standard MambaByte implementation.[13]

## Which production models use Mamba?

Within about two years of the original paper, Mamba-1 and Mamba-2 layers had moved from research code into openly released models from several organizations. Most of them are hybrids that keep a small share of attention layers; a few are pure SSMs. The models below are sourced to their vendors' announcements, model cards or technical reports.

### Codestral Mamba

[Mistral AI](https://aiwiki.ai/wiki/mistral_ai) released Codestral Mamba on 16 July 2024 under the Apache 2.0 license, as an instructed code model with 7,285,403,648 parameters designed "with help from Albert Gu and Tri Dao."[14] Mistral explained the choice by noting that "unlike Transformer models, Mamba models offer the advantage of linear time inference and the theoretical ability to model sequences of infinite length," which it called "especially relevant for code productivity use cases."[14] The Hugging Face model card describes it as "based on the Mamba2 architecture," and its configuration is a pure Mamba-2 stack of 64 layers with state size 128 and no attention layers.[31]

Mistral tested the model's in-context retrieval up to 256K tokens.[14] On the benchmarks Mistral published, Codestral Mamba scored 75.0% on [HumanEval](https://aiwiki.ai/wiki/humaneval) and 58.8% on Spider, against 61.0% (HumanEval) for CodeGemma 1.1 7B, 31.1% for CodeLlama 7B, 43.3% for CodeLlama 34B and 65.9% for "DeepSeek v1.5 7B."[31] It was offered on Mistral's la Plateforme as `codestral-mamba-2407`, with weights on [Hugging Face](https://aiwiki.ai/wiki/hugging_face) and deployment through the `mistral-inference` SDK and TensorRT-LLM.[14]

### Falcon Mamba 7B and Falcon3-Mamba

The Technology Innovation Institute (TII) in Abu Dhabi released Falcon Mamba 7B on 12 August 2024 under the "TII Falcon Mamba 7B License 1.0," and Hugging Face's launch post described it as the "first strong attention-free 7B model."[15] The launch post states that the model "can process sequences of arbitrary length without any increase in memory storage," fitting on a single A10 24GB GPU.[15] TII's technical report (October 2024) describes a pure Mamba model trained on 5.8 trillion tokens that "surpasses leading open-weight models based on Transformers, such as Mistral 7B, Llama3.1 8B, and Falcon2 11B" and is "on par with Gemma 7B" on the Open LLM Leaderboard.[32] On 17 December 2024 TII released Falcon3-Mamba-7B, Falcon Mamba 7B trained on an additional 1.5 trillion tokens with a 32K context length.[33]

TII then moved to hybrids. [Falcon-H1](https://aiwiki.ai/wiki/falcon_h1), announced in May 2025 in sizes from 0.5B to 34B, combines "attention and Mamba-2 heads in parallel within our hybrid mixer block," and Falcon H1R 7B (5 January 2026) is a reasoning model post-trained from the Falcon-H1 7B base.[34][35]

### AI21 Jamba family

[AI21 Labs](https://aiwiki.ai/wiki/ai21_labs) released Jamba on 28 March 2024 under the Apache 2.0 license, calling it "the world's first production-grade Mamba based model."[6][36] Jamba interleaves standard attention layers (with grouped-query attention) and Mamba layers at a ratio of one attention layer to seven Mamba layers, and adds [Mixture-of-Experts](https://aiwiki.ai/wiki/mixture_of_experts) MLPs so that 12B of its 52B parameters are active per token.[6] The paper reports strong results for contexts up to 256K tokens, and says the model fits on a single 80GB GPU with 8-bit weights "even with contexts of over 128K tokens"; AI21's announcement puts the single-GPU limit at up to 140K tokens.[6][36]

Jamba 1.5 Mini (12B active, 52B total) and Jamba 1.5 Large (94B active, 398B total) followed on 22 August 2024 under the Jamba Open Model License, with an effective context length of 256K tokens.[16][37] The Jamba 1.5 paper reports that "in a hybrid architecture, the Mamba-1-Attention combination works better than Mamba-2-Attention, so we use Mamba-1 in Jamba-1.5-Large," and introduces an ExpertsInt8 quantization that fits Jamba 1.5 Large on a machine with eight 80GB GPUs at 256K-token contexts.[37] At launch the models were available on Google Cloud Vertex AI, Hugging Face, Microsoft Azure and [NVIDIA NIM](https://aiwiki.ai/wiki/nvidia_nim), with other platforms announced as coming soon.[16]

| Version | Released | Architecture highlights | Context |
|---------|----------|------------------------|---------|
| Jamba (v0.1) | 28 March 2024 | 52B total / 12B active, 1:7 attention:Mamba, MoE; Apache 2.0[6][36] | Up to 256K |
| Jamba 1.5 Mini | 22 August 2024 | 52B total / 12B active[37] | 256K effective |
| Jamba 1.5 Large | 22 August 2024 | 398B total / 94B active, Mamba-1 layers[37] | 256K effective |
| Jamba 1.6 | 6 March 2025 | Hybrid SSM-Transformer, aimed at RAG and long-context grounded QA[17] | 256K |
| Jamba 1.7 | 3 July 2025 | Update aimed at more complete and accurate answers through better use of context[38] | 256K |
| Jamba Reasoning 3B | 8 October 2025 | Small on-device reasoning model on the SSM-Transformer architecture, Apache 2.0[39] | 256K |
| Jamba2 3B / Mini | 8 January 2026 | 3B dense and Mini MoE (12B active, 52B total), Apache 2.0[40] | 256K |

### Nvidia: Hymba and Nemotron

Nvidia researchers introduced Hymba in November 2024, a family of [small language models](https://aiwiki.ai/wiki/small_language_model) whose architecture "integrates attention heads and SSM heads within the same layer, offering parallel and complementary processing of the same inputs," rather than alternating attention and Mamba layers as Jamba does.[18] The authors report that Hymba-1.5B-Base "surpasses all sub-2B public models in performance and even outperforms Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size reduction, and 3.49x throughput."[18] See [Hymba](https://aiwiki.ai/wiki/hymba).

[Nemotron-H](https://aiwiki.ai/wiki/nemotron_h) (April 2025) is a family of 8B and 56B/47B hybrid Mamba-Transformer models in which about 8% of layers are self-attention (4 of 52 layers in the 8B model, 10 of 118 in the 56B model) and the rest are split evenly between Mamba-2 and feed-forward layers; Nvidia reports accuracy better than or on par with similarly sized open Transformer models such as Qwen-2.5 and Llama-3.1 while being up to 3x faster at inference.[41] [Nemotron Nano 2](https://aiwiki.ai/wiki/nemotron_nano_2) (August 2025) is a dense 9B hybrid, pruned and distilled from a 12B base model (62 layers: 6 attention, 28 Mamba-2 and 28 feed-forward) down to 56 layers, 4 of them attention, for which Nvidia reports up to 6x higher inference throughput than similarly sized models such as Qwen3-8B in reasoning settings like 8K input and 16K output tokens.[42] [Nemotron 3](https://aiwiki.ai/wiki/nemotron_3), announced on 15 December 2025, added Mixture-of-Experts: Nemotron 3 Nano 30B-A3B is described as "a Mixture-of-Experts hybrid Mamba-Transformer language model."[43][44]

### IBM Bamba and Granite 4.0

IBM released Bamba, a hybrid Mamba-2 model "trained by IBM, Princeton, CMU, and UIUC" on 2.2T tokens, on 18 December 2024, with Tri Dao, Albert Gu and Minjia Zhang among the authors of the announcement; Bamba v2 followed on 29 April 2025 with an additional 1T tokens of training.[45][46] On 2 October 2025 IBM launched [Granite 4.0](https://aiwiki.ai/wiki/granite_4), whose hybrid models combine "Mamba-2 layers and conventional transformer blocks sequentially in a 9:1 ratio."[19] IBM reports "over 70% reduction in RAM needed to handle long inputs and multiple concurrent batches" compared with conventional transformer models.[19] The hybrid line comprises Granite-4.0-H-Small (32B total, 9B active, MoE), H-Tiny (7B total, 1B active, MoE) and H-Micro (3B dense), while Granite-4.0-Micro is a conventional transformer; all are released under Apache 2.0.[19]

### Other hybrids

[Zyphra](https://aiwiki.ai/wiki/zyphra)'s Zamba (May 2024) is a 7B hybrid "combining a Mamba backbone with a single shared attention module," trained on 1T tokens, and the Zamba2 suite (models released July to September 2024, technical report November 2024) extended the idea to 1.2B, 2.7B and 7.4B Mamba-2 hybrids.[47][48] Microsoft's Samba (June 2024) "layer-wise combines Mamba" with [sliding window attention](https://aiwiki.ai/wiki/sliding_window_attention), scaled to 3.8B parameters trained on 3.2T tokens.[49] [Tencent](https://aiwiki.ai/wiki/tencent)'s Hunyuan-TurboS (technical report May 2025) is a hybrid Transformer-Mamba2 Mixture-of-Experts model with 56B activated and 560B total parameters over 128 layers, which Tencent calls "the first industry-deployed large-scale Mamba model."[50]

### Production model summary

| Model | Vendor | Released | Architecture |
|-------|--------|----------|--------------|
| Jamba | [AI21 Labs](https://aiwiki.ai/wiki/ai21_labs) | March 2024 | Hybrid Mamba-1 + attention (1:7) + MoE |
| Zamba 7B | [Zyphra](https://aiwiki.ai/wiki/zyphra) | May 2024 | Mamba backbone + one shared attention block |
| Samba 3.8B | [Microsoft](https://aiwiki.ai/wiki/microsoft_research) | June 2024 | Mamba + sliding window attention, layer-wise |
| Codestral Mamba 7B | [Mistral AI](https://aiwiki.ai/wiki/mistral_ai) | July 2024 | Pure Mamba-2 |
| Falcon Mamba 7B | TII | August 2024 | Pure Mamba |
| Jamba 1.5 Mini / Large | [AI21 Labs](https://aiwiki.ai/wiki/ai21_labs) | August 2024 | Hybrid Mamba-1 + attention + MoE |
| [Hymba](https://aiwiki.ai/wiki/hymba) 1.5B | [Nvidia](https://aiwiki.ai/wiki/nvidia) | November 2024 | Parallel attention and SSM heads in each layer |
| Falcon3-Mamba-7B | TII | December 2024 | Pure Mamba |
| Bamba 9B | IBM and partners | December 2024 | Hybrid Mamba-2 + attention |
| [Nemotron-H](https://aiwiki.ai/wiki/nemotron_h) | [Nvidia](https://aiwiki.ai/wiki/nvidia) | April 2025 | Hybrid Mamba-2 + attention (about 8%) |
| [Falcon-H1](https://aiwiki.ai/wiki/falcon_h1) | TII | May 2025 | Parallel attention and Mamba-2 heads |
| Hunyuan-TurboS | [Tencent](https://aiwiki.ai/wiki/tencent) | May 2025 | Hybrid Mamba-2 + attention + MoE |
| [Nemotron Nano 2](https://aiwiki.ai/wiki/nemotron_nano_2) | [Nvidia](https://aiwiki.ai/wiki/nvidia) | August 2025 | Dense hybrid Mamba-2 + attention |
| [Granite 4.0](https://aiwiki.ai/wiki/granite_4) (H models) | IBM | October 2025 | Mamba-2 : attention 9:1, some MoE |
| [Nemotron 3](https://aiwiki.ai/wiki/nemotron_3) | [Nvidia](https://aiwiki.ai/wiki/nvidia) | December 2025 | Hybrid Mamba-Transformer + MoE |
| Falcon H1R 7B | TII | January 2026 | Falcon-H1 hybrid, reasoning post-training |

## Hybrid architectures

The adoption list shows a clear pattern: most production uses of Mamba combine it with some attention. The Mamba-2 paper found that around 10% attention layers worked best in its experiments, and NVIDIA's 8B-parameter study of Mamba, Mamba-2 and Transformer models trained on up to 3.5T tokens found that a Mamba-2-Hybrid (43% Mamba-2, 7% attention, 50% MLP layers) exceeded the Transformer on all 12 standard tasks evaluated, by 2.65 points on average, and was predicted to be up to 8x faster at generating tokens.[4][8] The Mamba-2 authors hypothesize that SSM layers act as a general sequence-to-sequence mapping while the few attention layers act as a retrieval mechanism.[4]

Design choices differ in where attention goes. Jamba, Nemotron-H, Granite 4.0 and Hunyuan-TurboS interleave attention and Mamba layers sequentially; Hymba and Falcon-H1 run attention and SSM heads in parallel inside the same layer; Zamba reuses one shared attention block.[6][41][19][50][18][34][47] A systematic comparison by Sangmin Bae and colleagues (arXiv 2510.04800, October 2025) of inter-layer (sequential) and intra-layer (parallel) hybrids reported that both hybrid types, at up to 3B parameters and 32K context, outperform homogeneous architectures by up to 2.9% in few-shot accuracy, with intra-layer hybrids taking the Pareto frontier of quality and efficiency in their experiments.[20]

## What is Mamba used for?

### Natural language processing

Mamba and its variants have been applied to language modeling and [text generation](https://aiwiki.ai/wiki/text_generation_models), where linear-time training and constant-memory decoding make long contexts and high-throughput serving cheaper. Codestral Mamba for code, Falcon Mamba as a general pure-SSM model, and the Jamba, Nemotron and Granite hybrids for long-context enterprise and agent workloads are the main released examples (see above).

### Genomics and DNA modeling

The Mamba paper's DNA experiments showed better scaling than HyenaDNA and Transformer++ on the human genome and continued improvement up to million-length sequences.[1] Caduceus (Yair Schiff and colleagues, including Tri Dao and Albert Gu; ICML 2024) extends the Mamba block to a bidirectional "BiMamba" component and an "MambaDNA" block that is equivariant to the reverse complement of DNA; on a long-range variant-effect prediction task, its authors report that Caduceus "exceeds the performance of 10x larger models that do not leverage bi-directionality or equivariance."[21]

### Audio and speech

Mamba outperformed SaShiMi, Hyena and Transformers on audio waveform pretraining and SC09 speech generation in the original paper (see the benchmark section).[1] The ability to handle very long sequences efficiently suits raw audio, which at 16 kHz produces 16,000 samples per second.[1]

### Computer vision

Vision Mamba (Vim), posted in January 2024 and accepted at ICML 2024, adapts Mamba to images with bidirectional Mamba blocks over image-patch sequences marked with position embeddings.[7] On [ImageNet](https://aiwiki.ai/wiki/imagenet) classification, COCO [object detection](https://aiwiki.ai/wiki/object_detection), and ADE20k [semantic segmentation](https://aiwiki.ai/wiki/semantic_segmentation), the authors report higher performance than the DeiT [vision transformer](https://aiwiki.ai/wiki/vision_transformer); for example, Vim is 2.8x faster than DeiT and saves 86.8% of GPU memory when performing batch inference to extract features on 1248x1248 images.[7]

Other vision variants from early 2024 include VMamba (a NeurIPS 2024 spotlight), VideoMamba for video understanding, Mamba-ND for multi-dimensional data, and U-Mamba for biomedical [image segmentation](https://aiwiki.ai/wiki/image_segmentation).[51][52][53][54]

### Multimodal models

Cobra (March 2024, accepted at AAAI-25) extends Mamba to a multimodal large language model for efficient inference, and VL-Mamba (March 2024) explores state space models for multimodal learning.[55][56]

## Tooling and ecosystem

The reference implementation is the `mamba-ssm` Python package in the state-spaces/mamba GitHub repository, which as of 31 October 2026 had about 18,960 stars and 1,840 forks and is licensed under Apache 2.0.[24] It provides the selective-scan kernel and the Mamba, Mamba-2 and Mamba-3 modules for PyTorch, with the optimized short convolution supplied by the separate `causal-conv1d` package. Its README lists Linux, Python 3.10+ and PyTorch 1.12+ as core requirements, an NVIDIA GPU with CUDA 11.6+ for the CUDA selective-scan build, and separate instructions for AMD GPUs using ROCm.[24]

| Release | Date | Notes |
|---------|------|-------|
| v1.0.0 | 4 December 2023 | First tagged release, days after the paper[24] |
| v2.0.0 | 3 June 2024 | Release following the Mamba-2 paper (31 May 2024)[24][4] |
| v2.2.4 | 6 December 2024 | Last release of 2024[24] |
| v2.3.0 | 12 January 2026 | [24] |
| v2.3.2 | 9 May 2026 | Includes the "Mamba-3 code release"[24] |
| v2.3.2.post1 | 9 May 2026 | Latest release as of 31 October 2026[24] |

Major inference and training libraries support Mamba-family models:

| Framework | Mamba support |
|-----------|---------------|
| [Hugging Face transformers](https://aiwiki.ai/wiki/transformers_library) | `MambaForCausalLM` and `Mamba2ForCausalLM` classes[57] |
| [vLLM](https://aiwiki.ai/wiki/vllm) | Supported architectures include `MambaForCausalLM`, `Mamba2ForCausalLM`, `JambaForCausalLM`, `FalconMambaForCausalLM`, `FalconH1ForCausalLM`, `GraniteMoeHybridForCausalLM`, `NemotronHForCausalLM` and `Zamba2ForCausalLM`[58] |
| TensorRT-LLM | Mamba-1 and Mamba-2 examples (for example in release v0.20.0); current versions list `NemotronHForCausalLM`[59] |
| [llama.cpp](https://aiwiki.ai/wiki/llama_cpp) | Model architectures include `mamba`, `mamba2`, `jamba`, `falcon-h1`, `nemotron_h` and `granitehybrid`[60] |

## What are Mamba's limitations?

Despite its efficiency advantages, Mamba has several documented limitations compared to transformers:

**In-context learning.** In NVIDIA's controlled comparison of 8B-parameter models, pure Mamba and Mamba-2 lagged Transformers on tasks that need strong copying or in-context learning, such as five-shot MMLU and a phonebook lookup task.[8] After training on 1.1T tokens, both pure SSMs scored nearly 15 points lower than the Transformer on five-shot MMLU; training Mamba-2 on 3.5T tokens narrowed the gap to 1.37 points.[8]

**Copying and retrieval.** Jelassi and colleagues proved that a two-layer transformer can copy strings of exponential length, while any generalized state space model fails at copying "unless its latent state grows linearly with the sequence length."[9] Among pretrained models, even the smallest Pythia transformer (410M parameters) dramatically outperformed the largest Mamba (2.8B) at copying long natural-language strings, and also outperformed it on phone-book lookup once the phone book had 70 or more entries.[9] Ren, Li and Liu (2024) similarly argue that constant-size Mamba may struggle with copy operations, and that for solving arbitrary dynamic-programming problems with chain-of-thought its total cost is comparable to a standard Transformer.[11]

**State capacity.** These limitations share a root cause: Mamba compresses the whole sequence history into a fixed-size state, so information that does not survive compression cannot be recovered later. Transformers avoid this by keeping all past tokens in the KV cache, at the cost of memory that grows with context and quadratic attention compute.[1][9]

**State tracking.** Real-valued SSMs such as Mamba-2 cannot solve simple state-tracking tasks such as determining the parity of a bit sequence; the Mamba-3 authors report that Mamba-2 performs no better than random guessing on such arithmetic tasks, which Mamba-3's complex-valued (RoPE-like) state update solves almost perfectly.[5]

These limitations are the main motivation for the hybrid designs described above.[4][8]

## Explain like I'm 5 (ELI5)

Imagine you are listening to a very long story. A transformer is like writing down every single word of the story so you can look back at any word whenever you want. This works really well, but your notebook gets bigger and bigger, and it takes longer and longer to flip through all the pages.

Mamba is like keeping a summary in your head instead of writing everything down. As you hear each new word, you decide whether it is important enough to remember or whether you can forget it. You update your mental summary as you go. Your summary always stays the same size no matter how long the story gets, so you can listen to very, very long stories without running out of space. The downside is that if someone asks you to repeat word number 47 exactly, you might not remember it because you only kept a summary, not the full text.

The special trick Mamba uses is that it gets to decide what to remember based on what it is hearing right now. If it hears something important, it pays more attention and updates its summary. If it hears something less important, it mostly ignores it and keeps its existing summary. This is what "selective" means in "selective state space model."

Many recent models use a combination: mostly summary keeping (Mamba) with a few pages of detailed notes (attention) added where exact recall matters. That way you get much of the speed of summarizing and the precision of writing things down.

## References

1. Gu, A., & Dao, T. (2023). "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." arXiv:2312.00752 (v2, 31 May 2024). *Conference on Language Modeling (COLM) 2024*. https://arxiv.org/abs/2312.00752
2. Gu, A., Goel, K., & Re, C. (2021). "Efficiently Modeling Long Sequences with Structured State Spaces." *Proceedings of ICLR 2022* (Outstanding Paper Honorable Mention). arXiv:2111.00396. https://arxiv.org/abs/2111.00396
3. Gu, A., Dao, T., Ermon, S., Rudra, A., & Re, C. (2020). "HiPPO: Recurrent Memory with Optimal Polynomial Projections." *Proceedings of NeurIPS 2020*. arXiv:2008.07669. https://arxiv.org/abs/2008.07669
4. Dao, T., & Gu, A. (2024). "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality." *Proceedings of ICML 2024*. arXiv:2405.21060. https://arxiv.org/abs/2405.21060
5. Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026). "Mamba-3: Improved Sequence Modeling using State Space Principles." *ICLR 2026* (oral). arXiv:2603.15569. https://arxiv.org/abs/2603.15569
6. Lieber, O., Lenz, B., et al. (2024). "Jamba: A Hybrid Transformer-Mamba Language Model." arXiv:2403.19887. https://arxiv.org/abs/2403.19887
7. Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., & Wang, X. (2024). "Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model." *Proceedings of ICML 2024*. arXiv:2401.09417. https://arxiv.org/abs/2401.09417
8. Waleffe, R., Byeon, W., Riach, D., Norick, B., Korthikanti, V., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., Kulshreshtha, G., Singh, V., Casper, J., Kautz, J., Shoeybi, M., & Catanzaro, B. (2024). "An Empirical Study of Mamba-based Language Models." arXiv:2406.07887. https://arxiv.org/abs/2406.07887
9. Jelassi, S., Brandfonbrener, D., Kakade, S. M., & Malach, E. (2024). "Repeat After Me: Transformers are Better than State Space Models at Copying." *Proceedings of ICML 2024*. arXiv:2402.01032. https://arxiv.org/abs/2402.01032
10. Dao, T. (2024). "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." *Proceedings of ICLR 2024*. arXiv:2307.08691. https://arxiv.org/abs/2307.08691
11. Ren, R., Li, Z., & Liu, Y. (2024). "Exploring the Limitations of Mamba in COPY and CoT Reasoning." arXiv:2410.03810. https://arxiv.org/abs/2410.03810
12. Gao, L., et al. (2020). "The Pile: An 800GB Dataset of Diverse Text for Language Modeling." arXiv:2101.00027. https://arxiv.org/abs/2101.00027
13. Wang, J., Gangavarapu, T., Yan, J. N., & Rush, A. M. (2024). "MambaByte: Token-free Selective State Space Model." *Conference on Language Modeling (COLM) 2024*. arXiv:2401.13660. https://arxiv.org/abs/2401.13660
14. Mistral AI. (2024). "Codestral Mamba." Mistral AI news, 16 July 2024. https://mistral.ai/news/codestral-mamba
15. Hugging Face and Technology Innovation Institute. (2024). "Welcome Falcon Mamba: The first strong attention-free 7B model." Hugging Face blog, 12 August 2024. https://huggingface.co/blog/falconmamba
16. AI21 Labs. (2024). "The Jamba 1.5 Open Model Family: The Most Powerful and Efficient Long Context Models." AI21 blog, 22 August 2024. https://www.ai21.com/blog/announcing-jamba-model-family
17. AI21 Labs. (2025). "The Best Private LLM for Enterprise AI Deployment" (Jamba 1.6 announcement). AI21 blog, 6 March 2025. https://www.ai21.com/blog/introducing-jamba-1-6/
18. Dong, X., Fu, Y., Diao, S., Byeon, W., Chen, Z., Mahabaleshwarkar, A. S., et al. (2024). "Hymba: A Hybrid-head Architecture for Small Language Models." arXiv:2411.13676. https://arxiv.org/abs/2411.13676
19. IBM. (2025). "IBM Granite 4.0: hyper-efficient, high performance hybrid models for enterprise." IBM announcement, 2 October 2025. https://www.ibm.com/new/announcements/ibm-granite-4-0-hyper-efficient-high-performance-hybrid-models
20. Bae, S., Acun, B., Lin, C.-Y., Habeeb, H., Kim, S., Luo, L., Wang, J., & Wu, C.-J. (2025). "Hybrid Architectures for Language Models: Systematic Analysis and Design Insights." arXiv:2510.04800. https://arxiv.org/abs/2510.04800
21. Schiff, Y., Kao, C.-H., Gokaslan, A., Dao, T., Gu, A., & Kuleshov, V. (2024). "Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling." *Proceedings of ICML 2024*. arXiv:2403.03234. https://arxiv.org/abs/2403.03234
22. OpenReview. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." ICLR 2024 submission, listed under ICLR.cc/2024/Conference/Rejected_Submission. https://openreview.net/forum?id=AL1fq05o7H
23. OpenReview. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." COLM 2024 Conference. https://openreview.net/forum?id=tEYskw1VY2
24. state-spaces/mamba. GitHub repository, README and releases (retrieved 31 October 2026). https://github.com/state-spaces/mamba
25. state-spaces. Mamba model configurations (mamba-130m, mamba-370m, mamba-790m, mamba-1.4b, mamba-2.8b), config.json. Hugging Face. https://huggingface.co/state-spaces
26. Gupta, A., Gu, A., & Berant, J. (2022). "Diagonal State Spaces are as Effective as Structured State Spaces." arXiv:2203.14343. https://arxiv.org/abs/2203.14343
27. Gu, A., Gupta, A., Goel, K., & Re, C. (2022). "On the Parameterization and Initialization of Diagonal State Space Models." arXiv:2206.11893. https://arxiv.org/abs/2206.11893
28. Smith, J. T. H., Warrington, A., & Linderman, S. W. (2022). "Simplified State Space Layers for Sequence Modeling." arXiv:2208.04933. https://arxiv.org/abs/2208.04933
29. Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., & Re, C. (2022). "Hungry Hungry Hippos: Towards Language Modeling with State Space Models." *ICLR 2023*. arXiv:2212.14052. https://arxiv.org/abs/2212.14052
30. OpenReview. "Mamba-3: Improved Sequence Modeling using State Space Principles." ICLR 2026 Oral. https://openreview.net/forum?id=HwCvaJOiCj
31. Mistral AI. "Mamba-Codestral-7B-v0.1." Hugging Face model card and config.json. https://huggingface.co/mistralai/Mamba-Codestral-7B-v0.1
32. Zuo, J., Velikanov, M., Rhaiem, D. E., Chahed, I., Belkada, Y., Kunsch, G., & Hacid, H. (2024). "Falcon Mamba: The First Competitive Attention-free 7B Language Model." arXiv:2410.05355. https://arxiv.org/abs/2410.05355
33. Technology Innovation Institute. (2024). "Welcome to the Falcon 3 Family of Open Models!" Hugging Face blog, 17 December 2024. https://huggingface.co/blog/falcon3
34. Technology Innovation Institute. (2025). "Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance." Falcon blog, 20 May 2025. https://falcon-lm.github.io/blog/falcon-h1/
35. Technology Innovation Institute. (2026). "Introducing Falcon H1R 7B." Hugging Face blog, 5 January 2026. https://huggingface.co/blog/tiiuae/falcon-h1r-7b
36. AI21 Labs. (2024). "Introducing Jamba: AI21's Groundbreaking SSM-Transformer Model." AI21 blog, 28 March 2024. https://www.ai21.com/blog/announcing-jamba/
37. Jamba Team. (2024). "Jamba-1.5: Hybrid Transformer-Mamba Models at Scale." arXiv:2408.12570. https://arxiv.org/abs/2408.12570
38. AI21 Labs. "Changelog: Jamba 1.7" (release date 3 July 2025). AI21 documentation. https://docs.ai21.com/changelog
39. AI21 Labs. (2025). "Introducing Jamba Reasoning 3B: Tiny Model, Huge Possibilities." AI21 blog, 8 October 2025. https://www.ai21.com/blog/introducing-jamba-reasoning-3b/
40. AI21 Labs. (2026). "Jamba2: Open Source LLMs for Enterprise Reliability." AI21 blog, 8 January 2026. https://www.ai21.com/blog/introducing-jamba2/
41. NVIDIA. (2025). "Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models." arXiv:2504.03624. https://arxiv.org/abs/2504.03624
42. NVIDIA. (2025). "NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model." arXiv:2508.14444. https://arxiv.org/abs/2508.14444
43. NVIDIA. (2025). "NVIDIA Debuts Nemotron 3 Family of Open Models." NVIDIA Newsroom, 15 December 2025. https://nvidianews.nvidia.com/news/nvidia-debuts-nemotron-3-family-of-open-models
44. NVIDIA. (2025). "Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning." arXiv:2512.20848. https://arxiv.org/abs/2512.20848
45. IBM, Princeton, CMU and UIUC. (2024). "Bamba: Inference-Efficient Hybrid Mamba2 Model." Hugging Face blog, 18 December 2024. https://huggingface.co/blog/bamba
46. IBM. (2025). "Bamba-9B-v2 - Fast and powerful!" Hugging Face blog, 29 April 2025. https://huggingface.co/blog/ibm-ai-platform/bamba-9b-v2
47. Glorioso, P., Anthony, Q., Tokpanov, Y., Whittington, J., Pilault, J., Ibrahim, A., & Millidge, B. (2024). "Zamba: A Compact 7B SSM Hybrid Model." arXiv:2405.16712. https://arxiv.org/abs/2405.16712
48. Zyphra. (2024). "The Zamba2 Suite: Technical Report." arXiv:2411.15242. https://arxiv.org/abs/2411.15242
49. Ren, L., Liu, Y., Lu, Y., Shen, Y., Liang, C., & Chen, W. (2024). "Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling." arXiv:2406.07522. https://arxiv.org/abs/2406.07522
50. Tencent Hunyuan Team. (2025). "Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought." arXiv:2505.15431. https://arxiv.org/abs/2505.15431
51. Liu, Y., Tian, Y., Zhao, Y., Yu, H., et al. (2024). "VMamba: Visual State Space Model." NeurIPS 2024 (spotlight). arXiv:2401.10166. https://arxiv.org/abs/2401.10166
52. Li, K., Li, X., Wang, Y., He, Y., et al. (2024). "VideoMamba: State Space Model for Efficient Video Understanding." arXiv:2403.06977. https://arxiv.org/abs/2403.06977
53. Li, S., Singh, H., & Grover, A. (2024). "Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data." arXiv:2402.05892. https://arxiv.org/abs/2402.05892
54. Ma, J., Li, F., & Wang, B. (2024). "U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation." arXiv:2401.04722. https://arxiv.org/abs/2401.04722
55. Zhao, H., Zhang, M., Zhao, W., Ding, P., et al. (2024). "Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference." AAAI-25. arXiv:2403.14520. https://arxiv.org/abs/2403.14520
56. Qiao, Y., Yu, Z., Guo, L., Chen, S., et al. (2024). "VL-Mamba: Exploring State Space Models for Multimodal Learning." arXiv:2403.13600. https://arxiv.org/abs/2403.13600
57. Hugging Face. "Mamba" and "Mamba 2" model documentation, Transformers library. https://huggingface.co/docs/transformers/model_doc/mamba
58. vLLM. "Supported Models." vLLM documentation. https://docs.vllm.ai/en/latest/models/supported_models.html
59. NVIDIA. TensorRT-LLM Mamba example (tag v0.20.0) and supported-models list. https://github.com/NVIDIA/TensorRT-LLM
60. ggml-org. llama.cpp source, src/llama-arch.cpp. https://github.com/ggml-org/llama.cpp

