# Nemotron-H

> Source: https://aiwiki.ai/wiki/nemotron_h
> Updated: 2026-07-06
> Categories: AI Models, Large Language Models, NVIDIA, Open Source AI
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution to "AI Wiki (aiwiki.ai)".

**Nemotron-H** is a family of open-weight [large language models](/wiki/large_language_model) released by [NVIDIA](/wiki/nvidia) in April 2025 that replace most of the self-attention layers of a standard [Transformer](/wiki/transformer) with [Mamba-2](/wiki/mamba_2) state-space layers, producing a hybrid architecture that is substantially faster and cheaper to run at long context while matching the accuracy of comparable pure-Transformer models. The family spans three base sizes, 8B, 56B, and 47B parameters, and is described in the technical report "Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models" (arXiv:2504.03624), credited to a team of 199 NVIDIA researchers and engineers.[1][2] Nemotron-H is part of NVIDIA's broader [Nemotron](/wiki/nemotron) program of open models, and its hybrid design became the architectural template for later NVIDIA releases including [Nemotron Nano 2](/wiki/nemotron_nano_2) and the [Nemotron 3](/wiki/nemotron_3) mixture-of-experts family.[1][3][4]

## Why a hybrid architecture

The dominant weakness of the Transformer is the cost of its attention mechanism. Self-attention compares every token against every other token, so both the compute and the memory required grow quadratically with sequence length. During autoregressive generation, the model must also keep a growing key-value (KV) cache holding the keys and values for every previous token, and that cache expands linearly with the length of the text. As the [context window](/wiki/context_window) reaches tens or hundreds of thousands of tokens, the KV cache dominates memory use and the per-token cost of generation keeps climbing.[1][5]

State-space models offer a different trade-off. A [state-space model](/wiki/state_space_model) (SSM) such as [Mamba](/wiki/mamba) summarizes the past in a fixed-size recurrent state rather than an ever-growing cache. Its successor, Mamba-2, performs a constant amount of computation and uses a constant amount of memory for each generated token, regardless of how long the preceding context is. That makes SSMs attractive for long-context inference, but pure SSMs tend to lag Transformers on tasks that require precise recall or in-context copying, which attention handles well.[1][5]

Nemotron-H takes the pragmatic middle path that several 2025 systems adopted (AI21's [Jamba](/wiki/jamba) being an earlier example): keep a small number of attention layers for the tasks they excel at, and let Mamba-2 layers carry the rest. The result is close to constant compute and memory per generated token during decoding, without giving up the accuracy that a handful of attention layers provides.[1][5]

## Architecture

Each Nemotron-H model interleaves three kinds of layers: Mamba-2 SSM layers, standard feed-forward (MLP) layers, and a deliberately small set of self-attention layers. In the report's design, self-attention accounts for only about 8 percent of the layers (roughly 92 percent of the attention that a pure Transformer would use is removed), and the attention layers are spread evenly through the depth of the network rather than clustered.[1][6] The attention layers use grouped-query attention with a small number of key-value heads to keep the residual KV cache tiny.

| Model | Total layers | Mamba-2 layers | MLP layers | Attention layers | Hidden size |
|-------|--------------|----------------|------------|------------------|-------------|
| Nemotron-H-8B | 52 | 24 | 24 | 4 | 4096 |
| Nemotron-H-56B | 118 | 54 | 54 | 10 | 8192 |
| Nemotron-H-47B | trimmed from 56B | reduced | reduced | 5 | 8192 |

The 8B model has 4 attention layers out of 52; the 56B model has 10 out of 118. In both cases the Mamba-2 and MLP layer counts are equal (24 and 24 for the 8B, 54 and 54 for the 56B). The 8B uses a Mamba-2 state dimension of 128 with 32 attention query heads and 8 key-value heads; the 56B scales the state dimension to 256 with 64 query heads and 8 key-value heads.[6] Keeping a few attention layers matters: NVIDIA's ablations found that an entirely attention-free model degraded on tasks needing exact retrieval, while a small, evenly spaced attention budget recovered that ability at a fraction of the inference cost.[1]

## The model family

Nemotron-H ships as base (pretrained) models plus aligned and multimodal variants.

| Model | Parameters | Training tokens | Base context | Notes |
|-------|-----------|-----------------|--------------|-------|
| Nemotron-H-8B-Base-8K | 8B | ~15 trillion | 8K | Smallest base model |
| Nemotron-H-56B-Base-8K | 56B | ~20 trillion | 8K | Flagship, trained in [FP8](/wiki/fp8) |
| Nemotron-H-47B-Base-8K | 47B | +63 billion (distillation) | 8K | Compressed from the 56B |

The base models were published on [Hugging Face](/wiki/hugging_face) on April 14, 2025. NVIDIA also produced instruction-aligned variants (for example Nemotron-H-8B-Instruct, tuned with two rounds of supervised fine-tuning followed by three rounds of offline reward-aware preference optimization) and vision-language variants, discussed below.[2][7] The base weights were released under NVIDIA's research-oriented license (the NVIDIA Internal Scientific Research and Development Model License) rather than the fully commercial open license NVIDIA later applied to the Nemotron 3 generation.[7]

## Training data and FP8 pretraining

The Nemotron-H base models were pretrained on a large multilingual and multi-domain corpus supporting English, German, Spanish, French, Italian, Korean, Portuguese, Russian, Japanese, and Chinese.[7] A major component is [Nemotron-CC](/wiki/nemotron_cc), NVIDIA's openly released Common Crawl derivative that contributes roughly 6.3 trillion English tokens, of which about 4.4 trillion are filtered "real" web tokens and about 1.9 trillion are rephrased [synthetic](/wiki/synthetic_data) tokens. The mixture also includes curated math, code, and general-knowledge data, with hundreds of billions of synthetic instruction-style tokens weighted toward math and code.[1][6] The 8B model saw about 15 trillion tokens and the 56B about 20 trillion.[1][6]

A defining engineering contribution of the report is training the flagship model largely in 8-bit floating point. [FP8](/wiki/fp8) pretraining used per-tensor current scaling: each tensor is quantized with a single scaling factor set from its maximum absolute value. The forward pass uses the E4M3 FP8 format and the gradient (backward) pass uses E5M2, while the first and last few matrix multiplications are kept in higher-precision BF16 for stability. NVIDIA reported that this recipe matched BF16 training, with the loss differing by less than about 0.1 percent, making Nemotron-H one of the first models of its scale trained end-to-end in FP8.[1][6]

The 56B model was trained on 6,144 [NVIDIA H100](/wiki/nvidia_h100) GPUs, using NVIDIA's resilience tooling (NVRx and the DGX Cloud Resilience service) to keep such a large job running through hardware faults.[1][6] Combining FP8 arithmetic, the Mamba-heavy architecture, and this large H100 cluster is what let NVIDIA push the model to 20 trillion tokens efficiently.

## Compressing the 56B into the 47B

The 47B model was not trained from scratch. It was derived from the 56B base model through a compression pipeline NVIDIA calls MiniPuzzle, which combines the distillation approach of [Minitron](/wiki/minitron) with the block-level [neural architecture search](/wiki/neural_architecture_search) of NVIDIA's Puzzle framework. The pipeline prunes layers and widths, then recovers quality with [knowledge distillation](/wiki/knowledge_distillation) from the larger teacher, using only about 63 billion tokens (again in FP8) rather than a full pretraining run.[1][6]

The most visible change is that the 47B keeps just 5 of the 56B's 10 attention layers, roughly halving the attention budget and further shrinking the model's memory footprint. The compressed model retains accuracy close to the 56B while running about 1.2 times faster (roughly 20 percent quicker). Crucially, its smaller footprint (about 31.7 GiB in FP4) fits within the 32 GB of a single consumer [Blackwell](/wiki/nvidia_blackwell)-generation NVIDIA RTX 5090, where it can serve context windows approaching one million tokens.[1][6] That "million-token context on a desktop GPU" claim is the headline demonstration of the hybrid design's memory advantage.

## Benchmarks and inference speed

On standard accuracy benchmarks the 56B model is competitive with or ahead of the leading open Transformers of its class, [Qwen-2.5](/wiki/qwen_2_5)-72B and [Llama-3.1](/wiki/llama_3_1)-70B. NVIDIA reported that Nemotron-H-56B beat Llama-3.1-70B on 16 of 17 evaluated tasks, and that the compressed 47B stays close to the 56B.[1][6]

| Benchmark | Nemotron-H-56B | Qwen-2.5-72B | Llama-3.1-70B |
|-----------|----------------|--------------|---------------|
| MMLU (5-shot) | 84.2 | 86.1 | 78.8 |
| MMLU-Pro (5-shot CoT) | 60.5 | 58.8 | 51.3 |
| GSM8K (8-shot CoT) | 93.7 | 90.9 | 83.9 |
| MATH | 59.4 | 64.6 | 42.9 |
| HumanEval (pass@1) | 60.4 | 56.7 | 57.3 |

The picture is honest rather than uniformly dominant: the 56B leads on MMLU-Pro, GSM8K, and HumanEval but trails Qwen-2.5-72B on MMLU and MATH. At the small end, Nemotron-H-8B scores 44.0 on MMLU-Pro (5-shot chain of thought), ahead of Llama-3.1-8B (35.9) but behind Qwen-2.5-7B (48.3), so it is broadly on par with the best open 8B models rather than a clean sweep.[6]

The efficiency gains are where the architecture pays off. Measured at a 65,536-token input with 1,024 output tokens on H100 GPUs, NVIDIA reported the following throughput advantages:[1][6]

- Nemotron-H-8B vs Llama-3.1-8B: about 3 times faster; vs Qwen-2.5-7B: about 1.8 times faster.
- Nemotron-H-56B vs Qwen-2.5-72B and Llama-3.1-70B: about 2.4 times higher throughput.
- Nemotron-H-47B vs Qwen-2.5-72B and Llama-3.1-70B: about 2.9 times higher throughput.

The advantage widens with context length, because the Transformer baselines pay a growing KV-cache cost that the Mamba-heavy Nemotron-H largely avoids. The report's summary figure of "up to 3 times faster at inference for similar or better accuracy" refers to this regime.[1]

## Long-context, vision, and reasoning variants

Beyond raw speed, the hybrid design is meant to hold up over long inputs. On the RULER long-context benchmark, the 8B-Instruct variant retained strong scores as context grew (roughly 91 at 16K tokens, 88 at 64K, and 82 at 128K), showing the SSM layers do not simply forget distant content.[1]

NVIDIA also built vision-language variants, Nemotron-H-8B-VLM and Nemotron-H-56B-VLM, by pairing the Nemotron-H language backbone with an InternViT-300M vision encoder that processes 448 by 448 pixel image tiles with dynamic tiling. The 56B VLM was reported to be competitive with Qwen-2.5-72B-Instruct on document and chart understanding (for example ChartQA 89.4 vs 88.9 and DocVQA 93.2 vs 92.0).[1][6] These hybrid backbones were subsequently used inside NVIDIA's [Cosmos Reason](/wiki/cosmos_reason) physical-AI reasoning models.[6]

The Instruct/aligned checkpoints extend the base models with supervised fine-tuning and preference optimization, and the Nemotron-H-8B-Instruct model reached parity with Qwen-2.5-7B-Instruct on an average of chat and reasoning benchmarks.[6] The architecture's suitability for reasoning, where models generate very long chains of thought and therefore benefit most from cheap per-token decoding, is what motivated its reuse in NVIDIA's later reasoning-focused releases.

## Place in the Nemotron lineage

Nemotron-H is best understood as the architectural pivot point in NVIDIA's Nemotron program. Earlier Nemotron models (such as the [Nemotron-4](/wiki/nemotron_4) and Llama-Nemotron lines) were conventional dense Transformers. Nemotron-H introduced the hybrid Mamba-Transformer recipe, and NVIDIA then carried that recipe forward:

- [Nemotron Nano 2](/wiki/nemotron_nano_2) (August 2025) states explicitly that it "builds on the Nemotron-H architecture, in which the majority of the self-attention layers in the common Transformer architecture are replaced with Mamba-2 layers." It is a reasoning-tuned hybrid model compressed from a larger base, extending Nemotron-H's design and its Minitron-style compression to the small, deployable end of the range.[3]
- [Nemotron 3](/wiki/nemotron_3) (December 2025) generalizes the same idea to a [mixture-of-experts](/wiki/mixture_of_experts) setting, described by NVIDIA as an "open hybrid Mamba-Transformer MoE" family (Nano, Super, and Ultra) built for agentic reasoning, and released under NVIDIA's fully open commercial license.[4]

In that sense Nemotron-H is the direct ancestor of NVIDIA's current open-model strategy: dense hybrid first, then reasoning-tuned and MoE hybrids layered on top.

## Reception and significance

Nemotron-H arrived during a 2025 wave of interest in attention/state-space hybrids, and it was widely cited as a demonstration that the approach works at genuinely large scale. Commentators highlighted that replacing about 92 percent of attention with Mamba-2 blocks delivered up to roughly 3 times the throughput of same-size Transformers precisely because the SSM layers carry no KV cache.[5][8] At the same time, analysts noted the measured, non-triumphalist framing: hybrids match or exceed pure Transformers on many benchmarks and win decisively on efficiency, but do not uniformly dominate accuracy, which pushed the field toward evaluating models by use-case trade-offs rather than treating any single architecture as a universal replacement.[5]

Three contributions of the report proved especially influential: the practical FP8 pretraining recipe at 20-trillion-token scale, the MiniPuzzle compression that turned an expensive 56B into a desktop-deployable 47B, and the open release of both the models and the Nemotron-CC data. Together they made Nemotron-H a reference design that later NVIDIA models (and outside researchers working on hybrid SSM compression, such as the group-aware pruning of Minitron-SSM) built upon.[6][9]

## ELI5

A regular AI language model (a Transformer) "re-reads" everything it has written so far every time it writes the next word, which gets slow and memory-hungry as the text grows long. Nemotron-H swaps most of that re-reading machinery for a different kind of layer (Mamba-2) that keeps a small running summary instead, so writing each new word costs about the same no matter how long the story already is. NVIDIA kept just a few of the old re-reading layers because they are still handy for looking up exact details. The result is a model that is about as smart as the best open models its size but runs up to three times faster, and a slimmed-down 47-billion-parameter version can even handle a million words of context on a single high-end gaming graphics card.

## See also

- [Mamba-2](/wiki/mamba_2) and [state-space models](/wiki/state_space_model)
- [Transformer](/wiki/transformer)
- [Nemotron](/wiki/nemotron) (hub) and [Nemotron Nano 2](/wiki/nemotron_nano_2), [Nemotron 3](/wiki/nemotron_3)
- [Minitron](/wiki/minitron) compression
- [Nemotron-CC](/wiki/nemotron_cc) training data
- [Jamba](/wiki/jamba), an earlier hybrid Mamba-Transformer model
- [FP8](/wiki/fp8) low-precision training

## References

1. NVIDIA. "Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models." arXiv:2504.03624, April 2025. https://arxiv.org/abs/2504.03624
2. NVIDIA ADLR. "Nemotron-H: A Family of Accurate, Efficient Hybrid Mamba-Transformer Models." research.nvidia.com, April 2025. https://research.nvidia.com/labs/adlr/nemotronh/
3. NVIDIA. "NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model." arXiv:2508.14444, August 2025. https://arxiv.org/abs/2508.14444
4. NVIDIA Developer Blog. "Introducing Nemotron 3 Super: An Open Hybrid Mamba-Transformer MoE for Agentic Reasoning." developer.nvidia.com, December 2025. https://developer.nvidia.com/blog/introducing-nemotron-3-super-an-open-hybrid-mamba-transformer-moe-for-agentic-reasoning/
5. AskAIBrain. "End of Transformers? Attention + State-Space Hybrids in 2025." askaibrain.com, 2025. https://www.askaibrain.com/en/posts/end-of-transformers-hybrids-attention-state-space-2025/
6. NVIDIA. "Nemotron-H Technical Report" (full text, HTML v3). arXiv:2504.03624, April 2025. https://arxiv.org/html/2504.03624v3
7. Hugging Face. "nvidia/Nemotron-H-56B-Base-8K" model card (and nvidia/Nemotron-H-47B-Base-8K). Released April 14, 2025. https://huggingface.co/nvidia/Nemotron-H-56B-Base-8K
8. EmergentMind. "Nemotron-H: Hybrid Mamba-Transformer LLM." emergentmind.com, 2025. https://www.emergentmind.com/topics/nemotron-h-architecture
9. NVIDIA. "Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning." arXiv:2504.11409, April 2025. https://arxiv.org/abs/2504.11409