Nemotron-4

Nemotron-4 is a family of decoder-only large language models developed by NVIDIA and documented in two technical reports released in 2024. The name covers two very different-sized flagship members: Nemotron-4 15B, a 15-billion-parameter multilingual model detailed in February 2024, and Nemotron-4 340B, a 340-billion-parameter model published in June 2024 as three coordinated checkpoints (Base, Instruct, and Reward). Both models were built around the same recipe: a dense Transformer trained on a heavily multilingual and code-rich corpus, sized so that inference fits neatly onto NVIDIA's own datacenter hardware. Nemotron-4 340B in particular became one of the largest open-weights language models of 2024, and NVIDIA positioned it less as a chatbot and more as a factory for generating synthetic data to train other models. ^[1][2][3]

The two models are the fourth-generation entries in NVIDIA's broader Nemotron line and the direct successors to the earlier Nemotron-3 8B model of 2023 (a distinct product from the December 2025 Nemotron 3 agentic family, which reused the number). They also seeded a large amount of downstream work: the 15B model was compressed into the Minitron small-model family, and the design lessons and data pipelines carried forward into later NVIDIA releases. ^[4][7] (This article covers the original 2024 Nemotron-4 models. NVIDIA later reused the name: in March 2026 it announced a separate, forthcoming Nemotron 4 family, co-developed with Mistral AI through its Nemotron Coalition.)

Background and place in the Nemotron line

NVIDIA had been building large models internally for years using its Megatron-LM and NeMo frameworks, and shipped the Nemotron-3 8B family in late 2023. Nemotron-4 raised both the ambition and the scale. The stated goal of the 15B model was to show that a single, moderately sized checkpoint could be simultaneously strong at English, strong across dozens of other natural languages, and competent at code, rather than trading one capability off against the others. The 340B model then pushed the same architecture to a size where the resulting checkpoints were good enough to serve as teachers, generating training data for smaller student models instead of being deployed directly to end users. ^[1][2]

That framing matters for understanding why NVIDIA, primarily a hardware company, invested in open foundation models at all. The models double as reference workloads and as demonstrations of what its GPUs, its NeMo software stack, and its data-curation tooling can produce. NVIDIA published enough of the method, and eventually the weights and the synthetic data pipeline, that outside teams could reproduce and build on the approach. ^[3][4]

Nemotron-4 15B

Nemotron-4 15B was introduced in a technical report submitted to arXiv on 26 February 2024. It is a standard decoder-only Transformer trained on 8 trillion text tokens, and it was engineered so that its weights (roughly 30 GB in 16-bit precision) fit comfortably on a single 80 GB A100 or H100 GPU, making it cheap to serve. ^[1]

Architecture

Scroll sideways for more →

The architecture uses grouped-query attention, in which the 48 query heads share only 8 key/value heads, to cut memory bandwidth during inference. The unusually large 256,000-token vocabulary is deliberate: a big tokenizer represents non-English text with fewer tokens, which improves both efficiency and quality across the many languages the model targets. NVIDIA trained the model on 384 DGX H100 nodes (about 3,072 H100 GPUs) over roughly 13 calendar days. ^[1]

Multilingual and code training

The pretraining blend was split roughly 70% English natural language, 15% multilingual natural language, and 15% source code. The multilingual slice spanned 53 additional natural languages beyond English, and the code slice covered 43 programming languages. This wide coverage, rather than raw parameter count, is the central design choice of the model. ^[1]

Benchmarks

On its own report card, Nemotron-4 15B posted the following representative scores:

Scroll sideways for more →

NVIDIA reported that the model outperformed all existing similarly sized open models on 4 of 7 evaluation areas, and that its multilingual ability was the best in its size class, in some multilingual tasks beating models more than four times larger. On English knowledge it was roughly on par with contemporaries such as Google's Gemma 7B (which scored about 64.3 on MMLU), while adding much broader language coverage. ^[1][8]

Legacy: seeding Minitron

The most visible downstream product of the 15B model was the Minitron family. NVIDIA researchers took Nemotron-4 15B and compressed it through structured pruning (shrinking the embedding size, the number of attention heads, and the MLP width) followed by continued training with knowledge distillation, producing Minitron 8B and Minitron 4B. The 4B model, for instance, was recovered with only about 94 billion tokens of distillation. NVIDIA reported that deriving the family this way required up to 40 times fewer training tokens per model than training each from scratch, for roughly a 1.8 times overall compute saving, while the distilled models matched or beat community models such as Mistral 7B, Gemma 7B, and Llama 3 8B on several tasks. Minitron-4B-Base was later fine-tuned into the small, deployment-oriented Nemotron-Mini-4B-Instruct. ^[7]

Nemotron-4 340B

Nemotron-4 340B was released on 14 June 2024 as three checkpoints under a single new license. Where the 15B model was a compact generalist, the 340B family was designed at a scale intended to serve as a data-generation engine. ^[2][3][6]

Scroll sideways for more →

Architecture

Nemotron-4 340B keeps the same building blocks as the 15B model, scaled up. It contains roughly 340 billion parameters in total, of which about 331.6 billion are non-embedding parameters and about 9.4 billion sit in the embedding and output layers. ^[2]

Scroll sideways for more →

The 4,096-token context window is modest by later standards; the model was optimized for high-throughput generation of training examples rather than long-document reasoning. NVIDIA trained the base model on 768 DGX H100 nodes, using tensor and pipeline parallelism to spread the 340-billion-parameter network across thousands of GPUs. ^[2]

Two-phase pretraining

The base model saw about 9 trillion tokens, split into two phases. A main phase of roughly 8 trillion tokens established general capability, followed by a continued-pretraining phase of about 1 trillion tokens on a higher-quality, up-weighted data blend to sharpen the model near the end of training. The overall data mix mirrored the 15B recipe at roughly 70% English, 15% multilingual (50-plus languages), and 15% code (40-plus programming languages), with a pretraining data cutoff of June 2023. ^[2][5]

Alignment pipeline

The alignment of Nemotron-4-340B-Instruct is the most influential part of the project, because of how little human labor it required. Over 98% of the data used to align the model was synthetically generated by NVIDIA's own models, leaving only about 20,000 human-annotated examples in the loop. The pipeline combined instruction tuning with preference optimization: a supervised fine-tuning stage (handled in two passes, first for code and then for general instructions), followed by preference-based tuning using Direct Preference Optimization (DPO) and a technique NVIDIA introduced called Reward-aware Preference Optimization (RPO), run over several iterations. This heavy reliance on model-generated data, bootstrapped from a small human seed, is precisely the workflow the release was meant to make reproducible for outside teams. ^[2][5]

The reward model and RewardBench

Nemotron-4-340B-Reward is a classifier built on the base model that scores a response on five separate attributes: helpfulness, correctness, coherence, complexity, and verbosity. It was trained on NVIDIA's HelpSteer2 preference dataset (roughly 10,000 human-annotated preference samples). At release it topped the Hugging Face RewardBench leaderboard with an overall score around 92 (the technical report cites 92.0, and NVIDIA's later-updated results cite 92.2), holding the top position for about two months and beating the next-best model in the difficult Chat-Hard category by more than 10 points. That put it ahead of strong proprietary reward signals derived from models such as GPT-4o and Gemini 1.5 Pro. A high-quality open reward model is the second half of a synthetic-data flywheel: the Instruct model proposes candidate answers, and the Reward model ranks and filters them so that only good samples become training data. ^[2][3][4]

Benchmarks

The base and instruct models were competitive with the strongest open-access models of mid-2024, including Meta's Llama 3 70B, Mistral's Mixtral 8x22B, and Alibaba's Qwen 2 72B.

Scroll sideways for more →

In a blind human evaluation against GPT-4-1106-preview, Nemotron-4-340B-Instruct's responses were preferred, tied, or dispreferred at rates of about 28.2%, 46.6%, and 25.2%, meaning it was roughly on par with a leading closed model of the day. ^[2][5]

Why 340B mattered: an engine for synthetic data

The reason NVIDIA shipped a 340-billion-parameter model and then told developers not to necessarily deploy it directly is the synthetic data thesis. High-quality human-labeled training data is expensive, slow to collect, and often restricted by privacy or licensing. NVIDIA's argument was that a sufficiently strong open model, paired with an open reward model, could manufacture large volumes of task-specific training data cheaply, which developers could then use to fine-tune smaller, cheaper models for their own domains such as healthcare, finance, retail, and manufacturing. ^[3][4]

To make this concrete, NVIDIA open-sourced the synthetic data generation pipeline used in its own alignment process, wired into the NeMo framework and NeMo Aligner and served efficiently through TensorRT-LLM, with packaging as an NVIDIA NIM microservice. The pipeline is the same two-role loop used internally: Nemotron-4-340B-Instruct generates candidate responses from seed prompts, and Nemotron-4-340B-Reward filters them by attribute score. This positioning connects Nemotron-4 to NVIDIA's later data efforts, including the large-scale Nemotron-CC pretraining corpus and subsequent Nemotron model generations, all of which lean on model-generated and model-refined data. ^[2][3][4]

Deployment and licensing

Both models were sized to their target hardware. The 15B model runs on a single high-end GPU. The 340B model was designed to fit on a single DGX H100 node (8 GPUs) when deployed in FP8 precision; in higher-precision BF16 it needs two nodes (16 GPUs), as reflected on the Hugging Face model card. ^[2][5]

Nemotron-4 340B was released under the NVIDIA Open Model License, a permissive license that allows commercial use, redistribution, and, importantly, free use of the models' outputs, including using generated synthetic data to train and distribute other models, without attribution requirements. That output clause is what makes the synthetic-data use case legally clean, and it was a notable difference from some other 2024 open-model licenses that restricted downstream training. The weights and cards were distributed through the NVIDIA NGC catalog and Hugging Face. ^[3][5]

Reception and significance

At 340 billion parameters, Nemotron-4-340B was among the largest open-weights models made available in 2024, at a time when most open releases topped out well below that scale. Coverage focused on two things: that its benchmark quality was competitive with the best open models rather than a mere size stunt, and that its reward model briefly led RewardBench ahead of signals from frontier closed models. The synthetic-data framing was the durable contribution. It helped normalize the now-common practice of bootstrapping alignment from a tiny human seed and a large volume of model-generated, model-filtered data, and it gave outside teams an open, end-to-end reference implementation of that loop. ^[3][4]

The Nemotron-4 architecture and philosophy fed directly into NVIDIA's later work. Beyond the Minitron compressed models pruned and distilled from the 15B checkpoint, the emphasis on open weights plus open data and the synthetic-data pipeline carried into subsequent Nemotron releases, including the hybrid Mamba-Transformer Nemotron-H models that pair state-space layers with attention for faster inference. In that sense Nemotron-4 marks the point where the Nemotron line shifted from "a set of NVIDIA foundation models" to "an open toolchain for making foundation models." ^[7]

ELI5

Think of Nemotron-4 as two tools from NVIDIA. The small one, 15B, is a well-rounded model that speaks lots of languages and can also read code, and it is small enough to run on one graphics card. The huge one, 340B, is less a chatbot you talk to and more a machine that writes practice worksheets: it can generate mountains of realistic training examples, and a companion "grader" model throws out the bad ones. Other teams can then take those clean examples and use them to teach their own, smaller AI, which is cheaper than paying humans to write all the examples by hand.

See also

• Nemotron (the model family hub)
• Minitron (models pruned and distilled from Nemotron-4 15B)
• Nemotron-H (later hybrid Mamba-Transformer Nemotron models)
• Nemotron-CC (NVIDIA's refined pretraining corpus)
• Synthetic data
• NVIDIA NeMo

References

1. Parmar, Jupinder, et al. "Nemotron-4 15B Technical Report." NVIDIA, arXiv:2402.16819, 26 February 2024. https://arxiv.org/abs/2402.16819 ↩
2. NVIDIA. "Nemotron-4 340B Technical Report." arXiv:2406.11704, June 2024. https://arxiv.org/abs/2406.11704 ↩
3. NVIDIA. "NVIDIA Releases Open Synthetic Data Generation Pipeline for Training Large Language Models." NVIDIA Blog, 14 June 2024. https://blogs.nvidia.com/blog/nemotron-4-synthetic-data-generation-llm-training/ ↩
4. NVIDIA. "Leverage Our Latest Open Models for Synthetic Data Generation with NVIDIA Nemotron-4 340B." NVIDIA Technical Blog, June 2024 (updated 16 August 2024). https://developer.nvidia.com/blog/leverage-our-latest-open-models-for-synthetic-data-generation-with-nvidia-nemotron-4-340b/ ↩
5. NVIDIA. "nvidia/Nemotron-4-340B-Instruct" (model card). Hugging Face, 2024. https://huggingface.co/nvidia/Nemotron-4-340B-Instruct ↩
6. NVIDIA Research. "Nemotron-4 340B." Publication page, 14 June 2024. https://research.nvidia.com/publication/2024-06_nemotron-4-340b ↩
7. Muralidharan, Saurav, et al. "Compact Language Models via Pruning and Knowledge Distillation" (Minitron). NVIDIA, arXiv:2407.14679, 2024; Minitron-4B-Base model card, Hugging Face. https://arxiv.org/abs/2407.14679 ↩
8. Nguyen, Tanya. "NVIDIA Researchers Introduce Nemotron-4 15B: A 15B Parameter Large Multilingual Language Model Trained on 8T Text Tokens." MarkTechPost, 6 March 2024. https://www.marktechpost.com/2024/03/06/nvidia-researchers-introduce-nemotron-4-15b-a-15b-parameter-large-multilingual-language-model-trained-on-8t-text-tokens/ ↩
9. NVIDIA. "Nemotron" model documentation. Hugging Face Transformers. https://huggingface.co/docs/transformers/model_doc/nemotron