Citation and evidence

Mamba

41 min full readUpdated 60 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

Large Language ModelsMachine LearningModel Architecture

Cite this article

See also: transformer, attention mechanism, recurrent neural network, large language model, Mamba-2, state space model

Mamba is a neural network architecture for sequence modeling that uses selective state space models (SSMs) to process sequential data in linear time with respect to sequence length, offering an alternative to the transformer whose attention mechanism scales quadratically. It was introduced by Albert Gu and Tri Dao on 1 December 2023 in the paper "Mamba: Linear-Time Sequence Modeling with Selective State Spaces."[1] The paper was submitted to ICLR 2024, where OpenReview records it as a rejected submission, and was later published at the Conference on Language Modeling (COLM) 2024.[22][23] In benchmarks reported by its authors, Mamba has 5x higher generation throughput than Transformers of similar size (measured on an A100 GPU), scales linearly in sequence length, keeps improving on real data up to million-length sequences, and at the 3B-parameter scale outperforms Transformers of the same size while matching Transformers twice its size.[1]

The central idea behind Mamba is making the parameters of a state space model depend on the input, allowing the model to selectively propagate or forget information along the sequence depending on the current token.[1] In the words of Gu and Dao, "simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token."[1] Because input-dependent parameters rule out the convolutional shortcut used by earlier SSMs, the authors pair the selection mechanism with a hardware-aware parallel scan that keeps the expanded state in fast GPU memory.[1]

Mamba language models were pretrained at five sizes (130M, 370M, 790M, 1.4B, and 2.8B parameters) on 300 billion tokens of the Pile dataset.[1][12] The authors report that Mamba-3B (the 2.8B model) matches Transformers twice its size in both pretraining and downstream evaluation; for example, its average zero-shot accuracy is about 4 points higher than Pythia-2.8B and exceeds Pythia-6.9B.[1] As a general sequence model backbone, Mamba was also evaluated on audio waveforms and DNA sequences.[1]

Since the original paper, Mamba layers have been used in released models from Mistral AI, AI21 Labs, the Technology Innovation Institute (TII), Nvidia, IBM, Zyphra and Tencent, mostly in hybrid designs that keep a minority of attention layers. Libraries including Hugging Face transformers, vLLM, TensorRT-LLM and llama.cpp support Mamba models. The authors followed the original with Mamba-2 (2024) and Mamba-3 (2026), and the architecture seeded many variants, including MambaByte and Vision Mamba.

At a glance

PropertyValue
Architecture typeSelective state space model (SSM), called S6 in the paper[1]
Introduced1 December 2023 (arXiv:2312.00752)[1]
AuthorsAlbert Gu (Carnegie Mellon), Tri Dao (Princeton)[1]
PublicationRejected at ICLR 2024; published at COLM 2024[22][23]
Time complexityO(L)O(L) in sequence length LL (vs O(L2)O(L^2) for attention)[1]
Inference throughput4-5x higher than a Transformer of similar size in the authors' A100 benchmark[1]
Pretraining scales130M, 370M, 790M, 1.4B, 2.8B parameters on 300B Pile tokens[1][12]
Headline resultMamba-3B matches Transformers twice its size (authors' evaluation)[1]
Reference implementationmamba-ssm (state-spaces/mamba on GitHub), Apache 2.0; latest release v2.3.2.post1 (9 May 2026)[24]
Direct successorsMamba-2 (May 2024), Mamba-3 (March 2026)[4][5]

Expanded article table

"Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Paper Explained)", video from Yannic Kilcher, published December 24, 2023. Watch on YouTube.

Background: state space models

This section summarizes only what is needed to follow Mamba; the general theory, S4 and its relatives are covered in state space model.

Continuous-time formulation and discretization

The structured SSMs that Mamba builds on are inspired by a continuous system from control theory that maps a one-dimensional input x(t)x(t) to an output y(t)y(t) through an implicit latent state h(t)∈RNh(t) \in \mathbb{R}^N:[1]

  • State equation: h′(t)=Ah(t)+Bx(t)h'(t) = A h(t) + B x(t)
  • Output equation: y(t)=Ch(t)y(t) = C h(t)

To apply the model to discrete sequences such as tokens or audio samples, a step size Δ\Delta converts the continuous parameters (Δ,A,B)(\Delta, A, B) into discrete parameters (Aˉ,Bˉ)(\bar{A}, \bar{B}). The Mamba paper uses the zero-order hold (ZOH) rule as its example:[1]

Aˉ=exp⁡(ΔA)\bar{A} = \exp(\Delta A) Bˉ=(ΔA)−1(exp⁡(ΔA)−I)⋅ΔB\bar{B} = (\Delta A)^{-1} (\exp(\Delta A) - I) \cdot \Delta B

after which the model runs as the recurrence ht=Aˉht−1+Bˉxth_t = \bar{A} h_{t-1} + \bar{B} x_t, yt=Chty_t = C h_t.[1] The Mamba-3 paper later pointed out that, although Mamba-1 stated the ZOH formula, the released Mamba-1 and Mamba-2 implementations use a further approximation for the input term, ht≈eΔtAtht−1+ΔtBtxth_t \approx e^{\Delta_t A_t} h_{t-1} + \Delta_t B_t x_t, which the Mamba-3 authors name "exponential-Euler" discretization.[5]

When (Δ,A,B,C)(\Delta, A, B, C) are fixed for all time steps, the model is linear time invariant (LTI). An LTI SSM can be computed either as a linear recurrence or as one global convolution with kernel Kˉ=(CBˉ,CAˉBˉ,…)\bar{K} = (C\bar{B}, C\bar{A}\bar{B}, \ldots), so earlier models trained in the parallel convolutional mode and switched to the recurrent mode for autoregressive inference.[1] The Mamba authors note that all structured SSMs before Mamba were LTI "because of fundamental efficiency constraints," and argue that LTI models have fundamental limitations on certain kinds of data.[1]

S4 and its successors

The direct ancestor of Mamba is S4 (Structured State Space sequence model), introduced by Albert Gu, Karan Goel, and Christopher Re in October 2021 and published at ICLR 2022, where it received an Outstanding Paper Honorable Mention.[2] S4 built on the HiPPO framework (Gu, Dao, Stefano Ermon, Atri Rudra, and Re, NeurIPS 2020), which derives state matrices for online compression of history by projection onto polynomial bases.[3] S4 conditioned the state matrix with a low-rank correction so that it could be diagonalized stably and computed through a Cauchy kernel. Its authors reported 91% accuracy on sequential CIFAR-10 with no data augmentation or auxiliary losses, generation 60x faster than Transformers, and state-of-the-art results on every Long Range Arena task, including the length-16k Path-X task "that all prior work fails on."[2]

ModelFirst postedKey contribution
S4October 2021Low-rank-corrected (NPLR) parameterization of the HiPPO state matrix[2]
DSS (Diagonal State Spaces)March 2022Showed S4's performance can be matched with purely diagonal state matrices[26]
S4DJune 2022Systematic study of diagonal SSM parameterization and initialization[27]
S5August 2022One multi-input, multi-output SSM computed with parallel scans[28]
H3 (Hungry Hungry Hippos)December 2022SSM layer with multiplicative gating designed for recall in language modeling (ICLR 2023)[29]
MambaDecember 2023Input-dependent (selective) SSM parameters, computed with a hardware-aware scan[1]

Expanded article table

Mamba uses the diagonal structure popularized by these models, and its default real-valued initialization, S4D-Real, sets the nn-th diagonal element of AA to −(n+1)-(n+1), an initialization the paper describes as based on HiPPO theory.[1]

How does Mamba's selective scan mechanism work?

The selectivity problem

The core limitation of earlier SSM architectures was linear time invariance. Because the parameters stayed identical for every token regardless of content, these models could not perform content-aware filtering.[1] Consider a language model processing the sentence "The cat sat on the mat": an LTI model applies the same dynamics to the word "the" as to the word "cat," even though they carry very different amounts of information.

The Mamba authors diagnosed this directly: "a key weakness of such models is their inability to perform content-based reasoning."[1] They frame sequence modeling as a problem of compressing context into a state: attention is effective but inefficient because it does not compress context at all (it keeps the whole KV cache), while recurrent models are efficient because their state is finite but are limited by how well that state compresses the context.[1] Self-attention achieves content-based reasoning at O(L2)O(L^2) training cost; Mamba aims to do so while keeping O(L)O(L) complexity.[1]

The paper demonstrates the problem with two synthetic tasks. In Selective Copying, the positions of the tokens to memorize vary randomly, so a model must recognize them by content. In Induction Heads, a model must recall the token that followed an earlier occurrence of the current token. On Selective Copying, an LTI S4 layer without gating reached 18.3% accuracy, while the selective layer (S6) reached 97.0% on its own and 99.8% inside the full Mamba architecture.[1] On Induction Heads, a two-layer Mamba trained on sequences of length 256 generalized perfectly to sequences of length 1,048,576, about 4,000 times longer than training, while no other tested method went beyond 2x.[1]

Input-dependent parameters

Mamba makes three SSM parameters functions of the input:[1]

ParameterRoleHow it becomes selective
BB (input matrix)Controls how input enters the stateLinear projection of the input to dimension NN; different for each token
CC (output matrix)Controls how the state maps to outputLinear projection of the input to dimension NN; different for each token
Δ\Delta (step size)Controls discretization resolutionLow-rank projection of the input added to a learned bias, passed through softplus; different for each token and channel

Expanded article table

As a result Aˉ\bar{A} and Bˉ\bar{B} gain batch and length dimensions, which turns the model from time-invariant into time-varying.[1] The state matrix AA is not made input-dependent. The paper explains that AA "ultimately affects the model only through its interaction with Δ\Delta via Aˉ=exp⁡(ΔA)\bar{A} = \exp(\Delta A)," so selectivity in Δ\Delta already makes (Aˉ,Bˉ)(\bar{A}, \bar{B}) selective; the authors hypothesize that a selective AA would perform similarly and leave it out for simplicity.[1]

The step size Δ\Delta acts as the main selection mechanism. In the paper's words, "a large Δ\Delta resets the state hh and focuses on the current input xx, while a small Δ\Delta persists the state and ignores the current input."[1] The authors prove (Theorem 1) that with N=1N = 1, A=−1A = -1, B=1B = 1 and a softplus step size, the selective recurrence reduces to the gated update ht=(1−gt)ht−1+gtxth_t = (1 - g_t) h_{t-1} + g_t x_t, so the classical gating of RNNs such as LSTMs is a special case of the selection mechanism.[1] Mamba's SSM state is real-valued by default; the paper switched to complex values only for its audio-waveform experiments.[1]

Parallel scan algorithm

Making parameters input-dependent means the model can no longer be computed as a fixed convolution, so Mamba must use the recurrent form. Although the recurrence is time-varying, it can still be parallelized with a work-efficient parallel scan (prefix-sum) algorithm, following Blelloch (1990) and earlier SSM work such as S5.[1] The paper also observes that the recurrent computation uses O(BLDN)O(BLDN) FLOPs versus O(BLDlog⁡L)O(BLD \log L) for the convolutional one, with a lower constant factor, so for long sequences and a moderate state dimension the recurrence can actually use fewer FLOPs.[1]

The paper abbreviates the selective SSM as "S6," because it is an S4 model with a selection mechanism, computed with a scan.[1]

Hardware-aware implementation

The selective scan creates a memory problem. With DD channels and state size NN, the latent state over a batch has shape (B,L,D,N)(B, L, D, N), a factor of NN larger than the input and output; materializing it in GPU high-bandwidth memory (HBM) would make the model slow and memory-hungry.[1] Mamba addresses this with three classical techniques the authors name explicitly: kernel fusion, parallel scan, and recomputation.[1] The approach follows the IO-aware design of FlashAttention, also co-developed by Tri Dao, and the repository describes the implementation as "in the spirit of FlashAttention."[24][10]

Kernel fusion. A standard implementation would prepare the discretized parameters of size (B,L,D,N)(B, L, D, N) in HBM, run a scan that writes outputs of the same size back to HBM, and then multiply by CC. Mamba instead loads (Δ,A,B,C)(\Delta, A, B, C) from HBM into fast on-chip SRAM, performs the discretization, the scan and the multiplication by CC in SRAM within one fused kernel, and writes only the final (B,L,D)(B, L, D) output back to HBM. This reduces memory IO by a factor of O(N)O(N), which the authors report speeds up the operation by 20 to 40 times.[1] Sequences too long to fit in SRAM are split into chunks, with the scan state carried from one chunk to the next.[1]

Parallel scan. The fused kernel runs a parallel associative scan over the sequence, so the recurrence does not have to be computed one step at a time.[1]

Recomputation. The intermediate states are needed for backpropagation but are not saved. They are recomputed in the backward pass when the inputs are reloaded from HBM to SRAM, which the authors note is faster than storing them and reading them back. Recomputation is also applied to other cheap-to-recompute activations in the block, so the fused selective SSM layer has about the same activation memory as an optimized Transformer layer using FlashAttention.[1]

The authors report that their scan is faster than FlashAttention-2 beyond sequence length 2K and up to 20-40x faster than a standard scan implementation in PyTorch, measured on an A100 80GB GPU with state dimension N=16N = 16.[1] Their introduction summarizes the resulting implementation as faster than previous SSM methods both in theory (linear rather than pseudo-linear scaling) and on hardware, "up to 3x faster on A100 GPUs."[1] The reference CUDA kernel ships in the mamba-ssm package, which later releases extended with Mamba-2 and Mamba-3 modules.[24]

Mamba architecture

The full Mamba architecture wraps the selective SSM in a single block that is stacked homogeneously. Prior SSM architectures such as H3 alternated an attention-like SSM block with an MLP block; Mamba merges the two into one block, a simplification the authors say was inspired by the gated attention unit (GAU), which did something similar for attention.[1] The paper describes the result as combining "the H3 block, which is the basis of most SSM architectures, with the ubiquitous MLP block of modern neural networks."[1] Each block works as follows:[1]

  1. The input of dimension DD is expanded by a factor EE (fixed to E=2E = 2 in the paper) through two parallel linear projections.
  2. One branch passes through a short 1D convolution, a SiLU activation, and the selective SSM.
  3. The other branch passes through a SiLU activation and acts as a multiplicative gate.
  4. The two branches are multiplied element-wise.
  5. A final linear projection maps back to dimension DD.

Compared with the H3 block, Mamba replaces the first multiplicative gate with an activation function; compared with a gated MLP, it adds an SSM to the main branch, and the SiLU choice makes the gated MLP part the "SwiGLU" variant used in models such as LLaMA.[1] Most of a block's parameters (3ED23ED^2) sit in the linear projections, and two Mamba blocks roughly match the 12D212D^2 parameters of one Transformer attention-plus-MLP pair.[1] The block also includes an optional normalization layer (the authors chose LayerNorm, following RetNet), and blocks are interleaved with standard normalization and residual connections.[1] The authors describe Mamba as "a simplified end-to-end neural network architecture without attention or even MLP blocks."[1]

Because two Mamba blocks replace one Transformer layer, the released language models have about twice as many layers as GPT-3-style Transformers of the same size. Their published configurations use 24 layers at width 768 (130M), 48 layers at widths 1024, 1536 and 2048 (370M, 790M and 1.4B), and 64 layers at width 2560 (2.8B), with RMSNorm enabled.[25]

How does Mamba differ from a transformer?

AspectTransformerMamba
Core mechanismSelf-attentionSelective state space model
Training cost (sequence length LL)O(L2)O(L^2) per layerO(L)O(L) per layer
Training parallelismFully parallel (attention matrix)Parallel (via parallel scan)
Inference modeAutoregressive with KV cacheRecurrent with fixed-size state
Inference memoryKV cache grows linearly with contextFixed-size state (constant per step)
Content-aware reasoningYes (attention computes pairwise token interactions)Yes (input-dependent SSM parameters)
Context windowFixed maximum (requires extension techniques)No architectural maximum; quality depends on what the state retains
Long-range dependenciesAttention can directly connect any two positionsState carries compressed history
Copying and retrieval from contextStrong (direct access to all past tokens)Weaker (information must survive state compression)[9]
Position encodingUsually required (sinusoidal, RoPE, ALiBi, etc.)Not used in the original language models
Inference throughput (similar size)Baseline4-5x higher in the authors' benchmark[1]

Expanded article table

The fundamental trade-off is between expressiveness and efficiency. Transformers keep an uncompressed record of the sequence in the KV cache, which allows direct access to any past token. Mamba compresses the history into a fixed-size hidden state, which is far more memory-efficient but means information can only be used if it survived compression.[1] Later studies found that this makes pure SSMs weaker than Transformers on tasks that require copying or exact retrieval from context, such as looking up an entry in a long phone book.[9][8]

What benchmark results did Mamba achieve?

All numbers in this section are the authors' own results from the Mamba paper.[1]

Language modeling

In scaling-law experiments on the Pile (models from about 125M to 1.3B parameters, Chinchilla-style token budgets), the authors report that Mamba is "the first attention-free model to match the performance of a very strong Transformer recipe (Transformer++)," a LLaMA-style baseline with rotary embeddings, SwiGLU and RMSNorm, with the advantage growing at longer sequence length.[1]

For zero-shot evaluation, Mamba models trained on 300B Pile tokens were compared with Pythia and RWKV models trained on the same data, tokenizer and token count. Selected rows from the paper's Table 3:[1]

ModelPile perplexity (lower is better)LAMBADA accuracyHellaSwagAverage accuracy (6 tasks)
Pythia-1.4B7.5161.752.155.2
RWKV-1.5B7.7056.452.554.3
Mamba-1.4B6.8064.959.159.7
Pythia-2.8B6.7364.759.359.1
RWKV-3B7.0063.959.659.6
Mamba-2.8B6.2269.266.163.3
Pythia-6.9B6.5167.164.061.7
RWKV-7.4B6.3167.265.562.5

Expanded article table

The table caption states that Mamba is best-in-class on every evaluation at each model size and "generally matches baselines at twice the model size."[1]

Audio

On YouTubeMix piano audio, Mamba improved on the SaShiMi (S4 plus MLP) baseline in autoregressive pretraining at every context length tested, up to minute-long (about one million sample) sequences, with the gap widening at longer lengths.[1] On the SC09 speech-generation benchmark (one-second spoken digits at 16 kHz), a 6.1M-parameter Mamba model outperformed larger GAN- and diffusion-based baselines:[1]

ModelParametersFID (lower is better)Inception score (higher is better)
WaveNet4.2M5.082.27
SaShiMi5.8M1.995.13
WaveGAN19.1M2.034.90
DiffWave + SaShiMi23.0M1.425.94
Mamba6.1M0.946.26
Mamba24.3M0.677.33

Expanded article table

The larger, parameter-matched Mamba cut FID by more than half relative to the best prior model, which the paper's introduction highlights.[1]

DNA sequence modeling

On the HG38 human-genome dataset, Mamba's pretraining perplexity scaled better with model size than HyenaDNA and Transformer++; at about 40M parameters the authors estimate Mamba matches those baselines with roughly 3x to 4x fewer parameters.[1] When sequence length was increased from 1,024 to 1,048,576 at fixed model size, Mamba's perplexity kept improving while HyenaDNA's got worse.[1] On a harder version of a species-classification task, which asks the model to distinguish human, chimpanzee, gorilla, orangutan and bonobo DNA (species "known to share 99% of their DNA"), pretrained Mamba models were fine-tuned on sequences up to one million base pairs long.[1]

Inference speed

The end-to-end throughput benchmark generated 128 tokens after a 2,048-token prompt, at batch sizes from 1 to 128, on an A100 80GB PCIe GPU, comparing Mamba-1.4B and an untrained Mamba-6.9B with GPT-3-style Transformers of 1.3B and 6.7B parameters using the Hugging Face implementation.[1] The authors report that Mamba achieves 4-5x higher inference throughput "since without the KV cache it can use much higher batch sizes," and that an untrained Mamba-6.9B would have higher throughput than a Transformer-1.3B five times smaller.[1] In a training-memory comparison of 125M models, Mamba used slightly more memory than a Transformer with FlashAttention-2 (for example 4.8 GB versus 4.6 GB at batch size 1).[1]

Mamba-2: structured state space duality

In May 2024, Tri Dao and Albert Gu published "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality," introducing Mamba-2; the paper appeared at ICML 2024.[4] Its structured state space duality (SSD) framework shows that SSMs and a family of attention variants are different ways of multiplying by the same class of structured semiseparable matrices. In the dual form, the SSD layer computes (L∘QK⊤)⋅V(L \circ QK^\top) \cdot V, attention without the softmax and with an extra input-dependent mask LL whose entries are products of scalar decay terms ai∈[0,1]a_i \in [0, 1].[4]

Compared with Mamba-1's selective SSM, the SSD layer restricts AA from diagonal to scalar times identity and uses a head dimension PP of typically 64 or 128 instead of P=1P = 1; the authors describe these changes as "slightly decreasing the expressive power in return for significant training efficiency improvements," chiefly the ability to use matrix-multiplication units.[4] They report that a dedicated SSD implementation is 2-8x faster than Mamba's optimized selective scan while allowing recurrent states 8x larger or more with minimal slowdown, and that it crosses over FlashAttention-2 at sequence length 2K and is 6x faster at 16K.[4] In hybrid experiments, the paper found that having around 10% of layers be attention worked best.[4] The mamba-ssm package added Mamba-2 in its v2.0.0 release on 3 June 2024.[24] Full details are in the Mamba-2 article.

Mamba-3

Mamba-3 was posted on arXiv on 16 March 2026 by Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, and Albert Gu, and was presented as an oral paper at ICLR 2026.[5][30] Framed around an "inference-first perspective," it makes three changes:[5]

  • Exponential-trapezoidal discretization, a second-order generalization of the exponential-Euler rule used in practice by Mamba-1 and Mamba-2, which the authors show is equivalent to applying a data-dependent convolution of size two.[5]
  • Complex-valued state updates, which the authors prove are equivalent to applying a data-dependent rotary embedding (RoPE) to the BB and CC projections, letting the model solve synthetic state-tracking tasks such as parity on which Mamba-2 performs no better than random guessing.[5]
  • A multi-input, multi-output (MIMO) formulation that increases decoding FLOPs by up to 4x relative to Mamba-2 at a fixed state size while keeping similar wall-clock decode latency.[5]

At the 1.5B scale, the authors report that Mamba-3 improves average downstream accuracy by 0.6 points over the next best model, Gated DeltaNet, with the MIMO variant adding another 1.2 points, and that Mamba-3 (MIMO) with state size 64 matches the perplexity of Mamba-2 with state size 128.[5] The Mamba-3 code was released in mamba-ssm v2.3.2 on 9 May 2026.[24] See Mamba-3 for details.

MambaByte and byte-level sequence modeling

MambaByte, by Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M. Rush (posted January 2024, published at COLM 2024), is a token-free adaptation of Mamba trained autoregressively on raw bytes.[13] Conventional large language models operate on subword tokens produced by a BPE or SentencePiece tokenizer; removing the tokenizer removes that inductive bias but produces significantly longer sequences, which is costly for Transformers whose memory grows with sequence length.[13]

The authors report that MambaByte is competitive with, and in some settings outperforms, state-of-the-art subword Transformers on language modeling while keeping the noise robustness of token-free models.[13] Their largest model, MambaByte-972M, outperformed the other byte-level models they compared.[13] They also adapted speculative decoding with tokenized drafting and byte-level verification, giving a 2.6x inference speedup over the standard MambaByte implementation.[13]

Which production models use Mamba?

Within about two years of the original paper, Mamba-1 and Mamba-2 layers had moved from research code into openly released models from several organizations. Most of them are hybrids that keep a small share of attention layers; a few are pure SSMs. The models below are sourced to their vendors' announcements, model cards or technical reports.

Codestral Mamba

Mistral AI released Codestral Mamba on 16 July 2024 under the Apache 2.0 license, as an instructed code model with 7,285,403,648 parameters designed "with help from Albert Gu and Tri Dao."[14] Mistral explained the choice by noting that "unlike Transformer models, Mamba models offer the advantage of linear time inference and the theoretical ability to model sequences of infinite length," which it called "especially relevant for code productivity use cases."[14] The Hugging Face model card describes it as "based on the Mamba2 architecture," and its configuration is a pure Mamba-2 stack of 64 layers with state size 128 and no attention layers.[31]

Mistral tested the model's in-context retrieval up to 256K tokens.[14] On the benchmarks Mistral published, Codestral Mamba scored 75.0% on HumanEval and 58.8% on Spider, against 61.0% (HumanEval) for CodeGemma 1.1 7B, 31.1% for CodeLlama 7B, 43.3% for CodeLlama 34B and 65.9% for "DeepSeek v1.5 7B."[31] It was offered on Mistral's la Plateforme as codestral-mamba-2407, with weights on Hugging Face and deployment through the mistral-inference SDK and TensorRT-LLM.[14]

Falcon Mamba 7B and Falcon3-Mamba

The Technology Innovation Institute (TII) in Abu Dhabi released Falcon Mamba 7B on 12 August 2024 under the "TII Falcon Mamba 7B License 1.0," and Hugging Face's launch post described it as the "first strong attention-free 7B model."[15] The launch post states that the model "can process sequences of arbitrary length without any increase in memory storage," fitting on a single A10 24GB GPU.[15] TII's technical report (October 2024) describes a pure Mamba model trained on 5.8 trillion tokens that "surpasses leading open-weight models based on Transformers, such as Mistral 7B, Llama3.1 8B, and Falcon2 11B" and is "on par with Gemma 7B" on the Open LLM Leaderboard.[32] On 17 December 2024 TII released Falcon3-Mamba-7B, Falcon Mamba 7B trained on an additional 1.5 trillion tokens with a 32K context length.[33]

TII then moved to hybrids. Falcon-H1, announced in May 2025 in sizes from 0.5B to 34B, combines "attention and Mamba-2 heads in parallel within our hybrid mixer block," and Falcon H1R 7B (5 January 2026) is a reasoning model post-trained from the Falcon-H1 7B base.[34][35]

AI21 Jamba family

AI21 Labs released Jamba on 28 March 2024 under the Apache 2.0 license, calling it "the world's first production-grade Mamba based model."[6][36] Jamba interleaves standard attention layers (with grouped-query attention) and Mamba layers at a ratio of one attention layer to seven Mamba layers, and adds Mixture-of-Experts MLPs so that 12B of its 52B parameters are active per token.[6] The paper reports strong results for contexts up to 256K tokens, and says the model fits on a single 80GB GPU with 8-bit weights "even with contexts of over 128K tokens"; AI21's announcement puts the single-GPU limit at up to 140K tokens.[6][36]

Jamba 1.5 Mini (12B active, 52B total) and Jamba 1.5 Large (94B active, 398B total) followed on 22 August 2024 under the Jamba Open Model License, with an effective context length of 256K tokens.[16][37] The Jamba 1.5 paper reports that "in a hybrid architecture, the Mamba-1-Attention combination works better than Mamba-2-Attention, so we use Mamba-1 in Jamba-1.5-Large," and introduces an ExpertsInt8 quantization that fits Jamba 1.5 Large on a machine with eight 80GB GPUs at 256K-token contexts.[37] At launch the models were available on Google Cloud Vertex AI, Hugging Face, Microsoft Azure and NVIDIA NIM, with other platforms announced as coming soon.[16]

VersionReleasedArchitecture highlightsContext
Jamba (v0.1)28 March 202452B total / 12B active, 1:7 attention:Mamba, MoE; Apache 2.0[6][36]Up to 256K
Jamba 1.5 Mini22 August 202452B total / 12B active[37]256K effective
Jamba 1.5 Large22 August 2024398B total / 94B active, Mamba-1 layers[37]256K effective
Jamba 1.66 March 2025Hybrid SSM-Transformer, aimed at RAG and long-context grounded QA[17]256K
Jamba 1.73 July 2025Update aimed at more complete and accurate answers through better use of context[38]256K
Jamba Reasoning 3B8 October 2025Small on-device reasoning model on the SSM-Transformer architecture, Apache 2.0[39]256K
Jamba2 3B / Mini8 January 20263B dense and Mini MoE (12B active, 52B total), Apache 2.0[40]256K

Expanded article table

Nvidia: Hymba and Nemotron

Nvidia researchers introduced Hymba in November 2024, a family of small language models whose architecture "integrates attention heads and SSM heads within the same layer, offering parallel and complementary processing of the same inputs," rather than alternating attention and Mamba layers as Jamba does.[18] The authors report that Hymba-1.5B-Base "surpasses all sub-2B public models in performance and even outperforms Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size reduction, and 3.49x throughput."[18] See Hymba.

Nemotron-H (April 2025) is a family of 8B and 56B/47B hybrid Mamba-Transformer models in which about 8% of layers are self-attention (4 of 52 layers in the 8B model, 10 of 118 in the 56B model) and the rest are split evenly between Mamba-2 and feed-forward layers; Nvidia reports accuracy better than or on par with similarly sized open Transformer models such as Qwen-2.5 and Llama-3.1 while being up to 3x faster at inference.[41] Nemotron Nano 2 (August 2025) is a dense 9B hybrid, pruned and distilled from a 12B base model (62 layers: 6 attention, 28 Mamba-2 and 28 feed-forward) down to 56 layers, 4 of them attention, for which Nvidia reports up to 6x higher inference throughput than similarly sized models such as Qwen3-8B in reasoning settings like 8K input and 16K output tokens.[42] Nemotron 3, announced on 15 December 2025, added Mixture-of-Experts: Nemotron 3 Nano 30B-A3B is described as "a Mixture-of-Experts hybrid Mamba-Transformer language model."[43][44]

IBM Bamba and Granite 4.0

IBM released Bamba, a hybrid Mamba-2 model "trained by IBM, Princeton, CMU, and UIUC" on 2.2T tokens, on 18 December 2024, with Tri Dao, Albert Gu and Minjia Zhang among the authors of the announcement; Bamba v2 followed on 29 April 2025 with an additional 1T tokens of training.[45][46] On 2 October 2025 IBM launched Granite 4.0, whose hybrid models combine "Mamba-2 layers and conventional transformer blocks sequentially in a 9:1 ratio."[19] IBM reports "over 70% reduction in RAM needed to handle long inputs and multiple concurrent batches" compared with conventional transformer models.[19] The hybrid line comprises Granite-4.0-H-Small (32B total, 9B active, MoE), H-Tiny (7B total, 1B active, MoE) and H-Micro (3B dense), while Granite-4.0-Micro is a conventional transformer; all are released under Apache 2.0.[19]

Other hybrids

Zyphra's Zamba (May 2024) is a 7B hybrid "combining a Mamba backbone with a single shared attention module," trained on 1T tokens, and the Zamba2 suite (models released July to September 2024, technical report November 2024) extended the idea to 1.2B, 2.7B and 7.4B Mamba-2 hybrids.[47][48] Microsoft's Samba (June 2024) "layer-wise combines Mamba" with sliding window attention, scaled to 3.8B parameters trained on 3.2T tokens.[49] Tencent's Hunyuan-TurboS (technical report May 2025) is a hybrid Transformer-Mamba2 Mixture-of-Experts model with 56B activated and 560B total parameters over 128 layers, which Tencent calls "the first industry-deployed large-scale Mamba model."[50]

Production model summary

ModelVendorReleasedArchitecture
JambaAI21 LabsMarch 2024Hybrid Mamba-1 + attention (1:7) + MoE
Zamba 7BZyphraMay 2024Mamba backbone + one shared attention block
Samba 3.8BMicrosoftJune 2024Mamba + sliding window attention, layer-wise
Codestral Mamba 7BMistral AIJuly 2024Pure Mamba-2
Falcon Mamba 7BTIIAugust 2024Pure Mamba
Jamba 1.5 Mini / LargeAI21 LabsAugust 2024Hybrid Mamba-1 + attention + MoE
Hymba 1.5BNvidiaNovember 2024Parallel attention and SSM heads in each layer
Falcon3-Mamba-7BTIIDecember 2024Pure Mamba
Bamba 9BIBM and partnersDecember 2024Hybrid Mamba-2 + attention
Nemotron-HNvidiaApril 2025Hybrid Mamba-2 + attention (about 8%)
Falcon-H1TIIMay 2025Parallel attention and Mamba-2 heads
Hunyuan-TurboSTencentMay 2025Hybrid Mamba-2 + attention + MoE
Nemotron Nano 2NvidiaAugust 2025Dense hybrid Mamba-2 + attention
Granite 4.0 (H models)IBMOctober 2025Mamba-2 : attention 9:1, some MoE
Nemotron 3NvidiaDecember 2025Hybrid Mamba-Transformer + MoE
Falcon H1R 7BTIIJanuary 2026Falcon-H1 hybrid, reasoning post-training

Expanded article table

Hybrid architectures

The adoption list shows a clear pattern: most production uses of Mamba combine it with some attention. The Mamba-2 paper found that around 10% attention layers worked best in its experiments, and NVIDIA's 8B-parameter study of Mamba, Mamba-2 and Transformer models trained on up to 3.5T tokens found that a Mamba-2-Hybrid (43% Mamba-2, 7% attention, 50% MLP layers) exceeded the Transformer on all 12 standard tasks evaluated, by 2.65 points on average, and was predicted to be up to 8x faster at generating tokens.[4][8] The Mamba-2 authors hypothesize that SSM layers act as a general sequence-to-sequence mapping while the few attention layers act as a retrieval mechanism.[4]

Design choices differ in where attention goes. Jamba, Nemotron-H, Granite 4.0 and Hunyuan-TurboS interleave attention and Mamba layers sequentially; Hymba and Falcon-H1 run attention and SSM heads in parallel inside the same layer; Zamba reuses one shared attention block.[6][41][19][50][18][34][47] A systematic comparison by Sangmin Bae and colleagues (arXiv 2510.04800, October 2025) of inter-layer (sequential) and intra-layer (parallel) hybrids reported that both hybrid types, at up to 3B parameters and 32K context, outperform homogeneous architectures by up to 2.9% in few-shot accuracy, with intra-layer hybrids taking the Pareto frontier of quality and efficiency in their experiments.[20]

What is Mamba used for?

Natural language processing

Mamba and its variants have been applied to language modeling and text generation, where linear-time training and constant-memory decoding make long contexts and high-throughput serving cheaper. Codestral Mamba for code, Falcon Mamba as a general pure-SSM model, and the Jamba, Nemotron and Granite hybrids for long-context enterprise and agent workloads are the main released examples (see above).

Genomics and DNA modeling

The Mamba paper's DNA experiments showed better scaling than HyenaDNA and Transformer++ on the human genome and continued improvement up to million-length sequences.[1] Caduceus (Yair Schiff and colleagues, including Tri Dao and Albert Gu; ICML 2024) extends the Mamba block to a bidirectional "BiMamba" component and an "MambaDNA" block that is equivariant to the reverse complement of DNA; on a long-range variant-effect prediction task, its authors report that Caduceus "exceeds the performance of 10x larger models that do not leverage bi-directionality or equivariance."[21]

Audio and speech

Mamba outperformed SaShiMi, Hyena and Transformers on audio waveform pretraining and SC09 speech generation in the original paper (see the benchmark section).[1] The ability to handle very long sequences efficiently suits raw audio, which at 16 kHz produces 16,000 samples per second.[1]

Computer vision

Vision Mamba (Vim), posted in January 2024 and accepted at ICML 2024, adapts Mamba to images with bidirectional Mamba blocks over image-patch sequences marked with position embeddings.[7] On ImageNet classification, COCO object detection, and ADE20k semantic segmentation, the authors report higher performance than the DeiT vision transformer; for example, Vim is 2.8x faster than DeiT and saves 86.8% of GPU memory when performing batch inference to extract features on 1248x1248 images.[7]

Other vision variants from early 2024 include VMamba (a NeurIPS 2024 spotlight), VideoMamba for video understanding, Mamba-ND for multi-dimensional data, and U-Mamba for biomedical image segmentation.[51][52][53][54]

Multimodal models

Cobra (March 2024, accepted at AAAI-25) extends Mamba to a multimodal large language model for efficient inference, and VL-Mamba (March 2024) explores state space models for multimodal learning.[55][56]

Tooling and ecosystem

The reference implementation is the mamba-ssm Python package in the state-spaces/mamba GitHub repository, which as of 31 October 2026 had about 18,960 stars and 1,840 forks and is licensed under Apache 2.0.[24] It provides the selective-scan kernel and the Mamba, Mamba-2 and Mamba-3 modules for PyTorch, with the optimized short convolution supplied by the separate causal-conv1d package. Its README lists Linux, Python 3.10+ and PyTorch 1.12+ as core requirements, an NVIDIA GPU with CUDA 11.6+ for the CUDA selective-scan build, and separate instructions for AMD GPUs using ROCm.[24]

ReleaseDateNotes
v1.0.04 December 2023First tagged release, days after the paper[24]
v2.0.03 June 2024Release following the Mamba-2 paper (31 May 2024)[24][4]
v2.2.46 December 2024Last release of 2024[24]
v2.3.012 January 2026[24]
v2.3.29 May 2026Includes the "Mamba-3 code release"[24]
v2.3.2.post19 May 2026Latest release as of 31 October 2026[24]

Expanded article table

Major inference and training libraries support Mamba-family models:

FrameworkMamba support
Hugging Face transformersMambaForCausalLM and Mamba2ForCausalLM classes[57]
vLLMSupported architectures include MambaForCausalLM, Mamba2ForCausalLM, JambaForCausalLM, FalconMambaForCausalLM, FalconH1ForCausalLM, GraniteMoeHybridForCausalLM, NemotronHForCausalLM and Zamba2ForCausalLM[58]
TensorRT-LLMMamba-1 and Mamba-2 examples (for example in release v0.20.0); current versions list NemotronHForCausalLM[59]
llama.cppModel architectures include mamba, mamba2, jamba, falcon-h1, nemotron_h and granitehybrid[60]

Expanded article table

What are Mamba's limitations?

Despite its efficiency advantages, Mamba has several documented limitations compared to transformers:

In-context learning. In NVIDIA's controlled comparison of 8B-parameter models, pure Mamba and Mamba-2 lagged Transformers on tasks that need strong copying or in-context learning, such as five-shot MMLU and a phonebook lookup task.[8] After training on 1.1T tokens, both pure SSMs scored nearly 15 points lower than the Transformer on five-shot MMLU; training Mamba-2 on 3.5T tokens narrowed the gap to 1.37 points.[8]

Copying and retrieval. Jelassi and colleagues proved that a two-layer transformer can copy strings of exponential length, while any generalized state space model fails at copying "unless its latent state grows linearly with the sequence length."[9] Among pretrained models, even the smallest Pythia transformer (410M parameters) dramatically outperformed the largest Mamba (2.8B) at copying long natural-language strings, and also outperformed it on phone-book lookup once the phone book had 70 or more entries.[9] Ren, Li and Liu (2024) similarly argue that constant-size Mamba may struggle with copy operations, and that for solving arbitrary dynamic-programming problems with chain-of-thought its total cost is comparable to a standard Transformer.[11]

State capacity. These limitations share a root cause: Mamba compresses the whole sequence history into a fixed-size state, so information that does not survive compression cannot be recovered later. Transformers avoid this by keeping all past tokens in the KV cache, at the cost of memory that grows with context and quadratic attention compute.[1][9]

State tracking. Real-valued SSMs such as Mamba-2 cannot solve simple state-tracking tasks such as determining the parity of a bit sequence; the Mamba-3 authors report that Mamba-2 performs no better than random guessing on such arithmetic tasks, which Mamba-3's complex-valued (RoPE-like) state update solves almost perfectly.[5]

These limitations are the main motivation for the hybrid designs described above.[4][8]

Explain like I'm 5 (ELI5)

Imagine you are listening to a very long story. A transformer is like writing down every single word of the story so you can look back at any word whenever you want. This works really well, but your notebook gets bigger and bigger, and it takes longer and longer to flip through all the pages.

Mamba is like keeping a summary in your head instead of writing everything down. As you hear each new word, you decide whether it is important enough to remember or whether you can forget it. You update your mental summary as you go. Your summary always stays the same size no matter how long the story gets, so you can listen to very, very long stories without running out of space. The downside is that if someone asks you to repeat word number 47 exactly, you might not remember it because you only kept a summary, not the full text.

The special trick Mamba uses is that it gets to decide what to remember based on what it is hearing right now. If it hears something important, it pays more attention and updates its summary. If it hears something less important, it mostly ignores it and keeps its existing summary. This is what "selective" means in "selective state space model."

Many recent models use a combination: mostly summary keeping (Mamba) with a few pages of detailed notes (attention) added where exact recall matters. That way you get much of the speed of summarizing and the precision of writing things down.

Continue reading

More to read

Topics:Large Language ModelsMachine LearningModel Architecture

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48 ^49 ^50 ^51 ^52 ^53 ^54 ^55 ^56 ^57 ^58 ^59 ^60 ^61 ^62 ^63 ^64 ^65 ^66 ^67 ^68 ^69 ^70 ^71Gu, A., & Dao, T. (2023). "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." arXiv:2312.00752 (v2, 31 May 2024). *Conference on Language Modeling (COLM) 2024*. arxiv.org/...2312.00752
  2. ^1 ^2 ^3Gu, A., Goel, K., & Re, C. (2021). "Efficiently Modeling Long Sequences with Structured State Spaces." *Proceedings of ICLR 2022* (Outstanding Paper Honorable Mention). arXiv:2111.00396. arxiv.org/...2111.00396
  3. ^Gu, A., Dao, T., Ermon, S., Rudra, A., & Re, C. (2020). "HiPPO: Recurrent Memory with Optimal Polynomial Projections." *Proceedings of NeurIPS 2020*. arXiv:2008.07669. arxiv.org/...2008.07669
  4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Dao, T., & Gu, A. (2024). "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality." *Proceedings of ICML 2024*. arXiv:2405.21060. arxiv.org/...2405.21060
  5. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026). "Mamba-3: Improved Sequence Modeling using State Space Principles." *ICLR 2026* (oral). arXiv:2603.15569. arxiv.org/...2603.15569
  6. ^1 ^2 ^3 ^4 ^5Lieber, O., Lenz, B., et al. (2024). "Jamba: A Hybrid Transformer-Mamba Language Model." arXiv:2403.19887. arxiv.org/...2403.19887
  7. ^1 ^2Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., & Wang, X. (2024). "Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model." *Proceedings of ICML 2024*. arXiv:2401.09417. arxiv.org/...2401.09417
  8. ^1 ^2 ^3 ^4 ^5Waleffe, R., Byeon, W., Riach, D., Norick, B., Korthikanti, V., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., Kulshreshtha, G., Singh, V., Casper, J., Kautz, J., Shoeybi, M., & Catanzaro, B. (2024). "An Empirical Study of Mamba-based Language Models." arXiv:2406.07887. arxiv.org/...2406.07887
  9. ^1 ^2 ^3 ^4 ^5Jelassi, S., Brandfonbrener, D., Kakade, S. M., & Malach, E. (2024). "Repeat After Me: Transformers are Better than State Space Models at Copying." *Proceedings of ICML 2024*. arXiv:2402.01032. arxiv.org/...2402.01032
  10. ^Dao, T. (2024). "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." *Proceedings of ICLR 2024*. arXiv:2307.08691. arxiv.org/...2307.08691
  11. ^Ren, R., Li, Z., & Liu, Y. (2024). "Exploring the Limitations of Mamba in COPY and CoT Reasoning." arXiv:2410.03810. arxiv.org/...2410.03810
  12. ^1 ^2Gao, L., et al. (2020). "The Pile: An 800GB Dataset of Diverse Text for Language Modeling." arXiv:2101.00027. arxiv.org/...2101.00027
  13. ^1 ^2 ^3 ^4 ^5Wang, J., Gangavarapu, T., Yan, J. N., & Rush, A. M. (2024). "MambaByte: Token-free Selective State Space Model." *Conference on Language Modeling (COLM) 2024*. arXiv:2401.13660. arxiv.org/...2401.13660
  14. ^1 ^2 ^3 ^4Mistral AI. (2024). "Codestral Mamba." Mistral AI news, 16 July 2024. mistral.ai/...codestral-mamba
  15. ^1 ^2Hugging Face and Technology Innovation Institute. (2024). "Welcome Falcon Mamba: The first strong attention-free 7B model." Hugging Face blog, 12 August 2024. huggingface.co/...falconmamba
  16. ^1 ^2AI21 Labs. (2024). "The Jamba 1.5 Open Model Family: The Most Powerful and Efficient Long Context Models." AI21 blog, 22 August 2024. ai21.com/...announcing-jamba-model-family
  17. ^AI21 Labs. (2025). "The Best Private LLM for Enterprise AI Deployment" (Jamba 1.6 announcement). AI21 blog, 6 March 2025. ai21.com/...introducing-jamba-1-6
  18. ^1 ^2 ^3Dong, X., Fu, Y., Diao, S., Byeon, W., Chen, Z., Mahabaleshwarkar, A. S., et al. (2024). "Hymba: A Hybrid-head Architecture for Small Language Models." arXiv:2411.13676. arxiv.org/...2411.13676
  19. ^1 ^2 ^3 ^4IBM. (2025). "IBM Granite 4.0: hyper-efficient, high performance hybrid models for enterprise." IBM announcement, 2 October 2025. ibm.com/...fficient-high-performance-hybrid-models
  20. ^Bae, S., Acun, B., Lin, C.-Y., Habeeb, H., Kim, S., Luo, L., Wang, J., & Wu, C.-J. (2025). "Hybrid Architectures for Language Models: Systematic Analysis and Design Insights." arXiv:2510.04800. arxiv.org/...2510.04800
  21. ^Schiff, Y., Kao, C.-H., Gokaslan, A., Dao, T., Gu, A., & Kuleshov, V. (2024). "Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling." *Proceedings of ICML 2024*. arXiv:2403.03234. arxiv.org/...2403.03234
  22. ^1 ^2OpenReview. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." ICLR 2024 submission, listed under ICLR.cc/2024/Conference/Rejected_Submission. openreview.net/forum
  23. ^1 ^2OpenReview. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." COLM 2024 Conference. openreview.net/forum
  24. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13state-spaces/mamba. GitHub repository, README and releases (retrieved 31 October 2026). github.com/...mamba
  25. ^state-spaces. Mamba model configurations (mamba-130m, mamba-370m, mamba-790m, mamba-1.4b, mamba-2.8b), config.json. Hugging Face. huggingface.co/state-spaces
  26. ^Gupta, A., Gu, A., & Berant, J. (2022). "Diagonal State Spaces are as Effective as Structured State Spaces." arXiv:2203.14343. arxiv.org/...2203.14343
  27. ^Gu, A., Gupta, A., Goel, K., & Re, C. (2022). "On the Parameterization and Initialization of Diagonal State Space Models." arXiv:2206.11893. arxiv.org/...2206.11893
  28. ^Smith, J. T. H., Warrington, A., & Linderman, S. W. (2022). "Simplified State Space Layers for Sequence Modeling." arXiv:2208.04933. arxiv.org/...2208.04933
  29. ^Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., & Re, C. (2022). "Hungry Hungry Hippos: Towards Language Modeling with State Space Models." *ICLR 2023*. arXiv:2212.14052. arxiv.org/...2212.14052
  30. ^OpenReview. "Mamba-3: Improved Sequence Modeling using State Space Principles." ICLR 2026 Oral. openreview.net/forum
  31. ^1 ^2Mistral AI. "Mamba-Codestral-7B-v0.1." Hugging Face model card and config.json. huggingface.co/...Mamba-Codestral-7B-v0.1
  32. ^Zuo, J., Velikanov, M., Rhaiem, D. E., Chahed, I., Belkada, Y., Kunsch, G., & Hacid, H. (2024). "Falcon Mamba: The First Competitive Attention-free 7B Language Model." arXiv:2410.05355. arxiv.org/...2410.05355
  33. ^Technology Innovation Institute. (2024). "Welcome to the Falcon 3 Family of Open Models!" Hugging Face blog, 17 December 2024. huggingface.co/...falcon3
  34. ^1 ^2Technology Innovation Institute. (2025). "Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance." Falcon blog, 20 May 2025. falcon-lm.github.io/...falcon-h1
  35. ^Technology Innovation Institute. (2026). "Introducing Falcon H1R 7B." Hugging Face blog, 5 January 2026. huggingface.co/...falcon-h1r-7b
  36. ^1 ^2 ^3AI21 Labs. (2024). "Introducing Jamba: AI21's Groundbreaking SSM-Transformer Model." AI21 blog, 28 March 2024. ai21.com/...announcing-jamba
  37. ^1 ^2 ^3 ^4Jamba Team. (2024). "Jamba-1.5: Hybrid Transformer-Mamba Models at Scale." arXiv:2408.12570. arxiv.org/...2408.12570
  38. ^AI21 Labs. "Changelog: Jamba 1.7" (release date 3 July 2025). AI21 documentation. docs.ai21.com/changelog
  39. ^AI21 Labs. (2025). "Introducing Jamba Reasoning 3B: Tiny Model, Huge Possibilities." AI21 blog, 8 October 2025. ai21.com/...introducing-jamba-reasoning-3b
  40. ^AI21 Labs. (2026). "Jamba2: Open Source LLMs for Enterprise Reliability." AI21 blog, 8 January 2026. ai21.com/...introducing-jamba2
  41. ^1 ^2NVIDIA. (2025). "Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models." arXiv:2504.03624. arxiv.org/...2504.03624
  42. ^NVIDIA. (2025). "NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model." arXiv:2508.14444. arxiv.org/...2508.14444
  43. ^NVIDIA. (2025). "NVIDIA Debuts Nemotron 3 Family of Open Models." NVIDIA Newsroom, 15 December 2025. nvidianews.nvidia.com/...n-3-family-of-open-models
  44. ^NVIDIA. (2025). "Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning." arXiv:2512.20848. arxiv.org/...2512.20848
  45. ^IBM, Princeton, CMU and UIUC. (2024). "Bamba: Inference-Efficient Hybrid Mamba2 Model." Hugging Face blog, 18 December 2024. huggingface.co/...bamba
  46. ^IBM. (2025). "Bamba-9B-v2 - Fast and powerful!" Hugging Face blog, 29 April 2025. huggingface.co/...bamba-9b-v2
  47. ^1 ^2Glorioso, P., Anthony, Q., Tokpanov, Y., Whittington, J., Pilault, J., Ibrahim, A., & Millidge, B. (2024). "Zamba: A Compact 7B SSM Hybrid Model." arXiv:2405.16712. arxiv.org/...2405.16712
  48. ^Zyphra. (2024). "The Zamba2 Suite: Technical Report." arXiv:2411.15242. arxiv.org/...2411.15242
  49. ^Ren, L., Liu, Y., Lu, Y., Shen, Y., Liang, C., & Chen, W. (2024). "Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling." arXiv:2406.07522. arxiv.org/...2406.07522
  50. ^1 ^2Tencent Hunyuan Team. (2025). "Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought." arXiv:2505.15431. arxiv.org/...2505.15431
  51. ^Liu, Y., Tian, Y., Zhao, Y., Yu, H., et al. (2024). "VMamba: Visual State Space Model." NeurIPS 2024 (spotlight). arXiv:2401.10166. arxiv.org/...2401.10166
  52. ^Li, K., Li, X., Wang, Y., He, Y., et al. (2024). "VideoMamba: State Space Model for Efficient Video Understanding." arXiv:2403.06977. arxiv.org/...2403.06977
  53. ^Li, S., Singh, H., & Grover, A. (2024). "Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data." arXiv:2402.05892. arxiv.org/...2402.05892
  54. ^Ma, J., Li, F., & Wang, B. (2024). "U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation." arXiv:2401.04722. arxiv.org/...2401.04722
  55. ^Zhao, H., Zhang, M., Zhao, W., Ding, P., et al. (2024). "Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference." AAAI-25. arXiv:2403.14520. arxiv.org/...2403.14520
  56. ^Qiao, Y., Yu, Z., Guo, L., Chen, S., et al. (2024). "VL-Mamba: Exploring State Space Models for Multimodal Learning." arXiv:2403.13600. arxiv.org/...2403.13600
  57. ^Hugging Face. "Mamba" and "Mamba 2" model documentation, Transformers library. huggingface.co/...mamba
  58. ^vLLM. "Supported Models." vLLM documentation. docs.vllm.ai/...supported_models
  59. ^NVIDIA. TensorRT-LLM Mamba example (tag v0.20.0) and supported-models list. github.com/...TensorRT-LLM
  60. ^ggml-org. llama.cpp source, src/llama-arch.cpp. github.com/...llama.cpp

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

16 revisions · v16 · 8,157 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independently verified 2026-10-31 against the Mamba paper (arXiv 2312.00752 PDF), OpenReview/COLM records, mamba-ssm GitHub releases, and the papers, model cards and configs of every adoption-table model; 9 verifier defects fixed before stamping.

Cite this page: AI Wiki. "Mamba." aiwiki.ai, updated 31 Oct 2026, fact-checked 31 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/mamba

Edit article