Llama-3.1-Nemotron-70B-Instruct

Llama-3.1-Nemotron-70B-Instruct is a large language model released by NVIDIA in October 2024. It is a customized, alignment-tuned version of Meta's Llama 3.1 70B Instruct model, produced not by adding new pretraining data but by applying reinforcement learning from human feedback with a specially trained reward model. For a brief window after its release it topped three widely used automatic evaluation leaderboards, scoring ahead of much larger frontier systems including OpenAI's GPT-4o and Anthropic's Claude 3.5 Sonnet on those specific benchmarks ^[1][2]. The result drew attention because it suggested that careful reward modeling and preference optimization on an open Llama base could rival closed models many times its size, at least on chat-alignment evaluations. The model should not be confused with the later Llama Nemotron reasoning family that NVIDIA introduced in 2025; that is a separate, reasoning-focused line of models ^[3].

Background and place in the Nemotron family

Llama-3.1-Nemotron-70B-Instruct belongs to NVIDIA's Nemotron program, an ongoing line of open models and open datasets aimed at improving how large language models are aligned to human preferences. Rather than compete head-on with the largest frontier labs on raw pretraining scale, NVIDIA's alignment research has concentrated on the post-training stage: collecting high-quality human feedback, training strong reward models on it, and using those reward models to steer a base model toward more helpful behavior ^[4][5].

The October 2024 release paired two artifacts. The first was the instruction model itself, Llama-3.1-Nemotron-70B-Instruct. The second was its companion reward model, Llama-3.1-Nemotron-70B-Reward, which scores candidate responses and provides the training signal used to align the instruction model ^[2][6]. Both were built on top of Meta's Llama-3.1-70B-Instruct, so no new base architecture was involved. What NVIDIA contributed was the alignment recipe, the human-preference data, and the reward model.

This model is distinct from, and predates, the Llama Nemotron reasoning models (Nano, Super, and Ultra) that NVIDIA announced at GTC in March 2025. That later family consists of reasoning models post-trained specifically for multi-step math, coding, and agentic reasoning, and it exposes a toggleable "reasoning" mode ^[3]. The 2024 Nemotron-70B model, by contrast, was tuned for general helpfulness in ordinary conversation and does not use extended chain-of-thought tokens to reach its answers ^[1][7].

How it was built

The alignment pipeline behind Llama-3.1-Nemotron-70B-Instruct is the practical payoff of two NVIDIA research releases: the HelpSteer2 dataset and its extension, HelpSteer2-Preference.

HelpSteer2 and HelpSteer2-Preference

HelpSteer2, published in June 2024, is a permissively licensed (CC-BY-4.0) human-preference dataset built to train reward models ^[4][8]. It contains roughly 21,000 response-level ratings (20,324 training and 1,038 validation samples), organized as about 10,000 response pairs. Each response is scored by human annotators on five attributes, every one on a 0 to 4 Likert scale ^[8]:

Scroll sideways for more →

A notable feature of HelpSteer2 is its efficiency: at about ten thousand response pairs it is roughly an order of magnitude smaller than earlier preference datasets such as HH-RLHF, yet a reward model trained on it reached a then state-of-the-art 92.0 on the primary RewardBench dataset ^[4].

HelpSteer2-Preference, posted to arXiv on 2 October 2024 and later accepted to ICLR 2025, extended that dataset ^[5]. Reward models are typically trained under one of two paradigms. In the Bradley-Terry style, the model learns from pairwise preferences (annotators say which of two responses they prefer). In the regression style, used by NVIDIA's earlier SteerLM work, the model learns to predict absolute attribute scores. Before this work there was little clean evidence that either approach was better when the two were matched on the same data. HelpSteer2-Preference added dedicated preference annotations, including human-written justifications, on top of the existing ratings, allowing a direct comparison and, crucially, a way to combine both signals ^[5][6].

Reward modeling plus REINFORCE

The Llama-3.1-Nemotron-70B-Reward model was trained on this combined data using an approach that fuses the strengths of Bradley-Terry and SteerLM regression reward modeling ^[6]. Given an English multi-turn conversation of up to 4,096 tokens, it emits a single scalar reward score rating the quality of the final assistant turn ^[2][6].

That reward model was then used to align the instruction model through reinforcement learning from human feedback, a form of reinforcement learning. Specifically, NVIDIA used the REINFORCE algorithm, implemented in the NeMo Aligner toolkit (part of NVIDIA NeMo), with Llama-3.1-70B-Instruct as the initial policy and the HelpSteer2-Preference prompts as the training prompts ^[1][5]. During training, the policy generates responses, the reward model scores them, and REINFORCE updates the policy to increase the probability of higher-scoring outputs. This is a different route from supervised instruction tuning alone or from methods such as direct preference optimization; it keeps a separate reward model in the loop rather than optimizing preferences directly. The paper reports that this recipe drove the aligned model to the top of the automatic alignment leaderboards it was measured on ^[5].

The reward model and RewardBench

The reward model was itself a headline result. As of 1 October 2024, Llama-3.1-Nemotron-70B-Reward ranked first on RewardBench, a standard benchmark for reward-model quality, out of more than 140 reward models tracked at the time ^[5][6]. Its scores were:

Scroll sideways for more →

The strong Safety (95.1) and Reasoning (98.1) numbers indicate the reward model could reliably reject unsafe completions and could distinguish good from bad answers in structured domains such as math and code, which is what makes it useful as a training signal for RLHF ^[6]. Because the reward model is the component that judges every response during alignment, its accuracy sets a ceiling on how good the tuned policy can become, so topping RewardBench was a meaningful precondition for the instruction model's leaderboard performance.

Benchmark results

On its three headline evaluations, all measured as of 1 October 2024, Llama-3.1-Nemotron-70B-Instruct posted the following, alongside the frontier models NVIDIA compared it against ^[1][2]:

Scroll sideways for more →

The comparison against the unaligned base model is the clearest story in the table. Alignment lifted the same 70B Llama from 55.7 to 85.0 on Arena Hard and from 38.1 to 57.6 on AlpacaEval 2 length-controlled, without any change to the underlying weights' scale or architecture. The tuned 70B model also scored above Meta's own Llama-3.1-405B-Instruct, a model nearly six times larger, on all three metrics ^[1][2].

Caveats about automatic evals

These are three automatic alignment benchmarks, not a claim of general superiority. Arena Hard, AlpacaEval 2, and MT-Bench all use a strong LLM (typically GPT-4-Turbo) as an automatic judge of response quality on open-ended chat prompts. They correlate with human preference but are known to reward traits such as thoroughness and formatting, and Nemotron-70B's mean response length (about 2,200 characters) was the longest in the comparison ^[2]. NVIDIA itself framed the numbers narrowly, stating that "as of 1 Oct 2024" the model performed best on these specific benchmarks, and cautioning that the model was optimized for helpfulness and was not specifically tuned to excel at specialized tasks such as math or coding ^[1][2]. A useful reality check came from the human-voted Chatbot Arena: as of late October 2024 the model sat around 1267 Elo, and once style-control adjustments were applied (which discount verbosity and formatting), its rank dropped substantially, well outside the top of the board ^[2]. In other words, it genuinely led the automatic chat evaluations while remaining a mid-pack contender in blind human voting.

Reception and significance

The model appeared on Hugging Face in mid-October 2024 with almost no marketing, and the benchmark numbers were what drew coverage. Outlets including VentureBeat, MarkTechPost, and Beebom framed it as a "quiet" release that nonetheless outscored GPT-4o and Claude 3.5 Sonnet on the reported evaluations, and as a sign of NVIDIA's growing ambitions in AI software rather than only hardware ^[7][9][10].

A recurring talking point was the model card's demonstration on the "How many r's are in strawberry?" prompt, a question that famously trips up many language models because they operate over sub-word tokens rather than individual letters. Nemotron-70B answers "There are 3 R's in the word 'strawberry'" and walks through the letters, and NVIDIA noted that it does so without extra reasoning tokens (as OpenAI's o1 models use) and without special prompting ^[1][7]. Reviewers treated this as a fun illustration rather than proof of reasoning ability; some reported the model still answered inconsistently across attempts, and the strawberry demo was not a benchmark ^[7].

The broader significance was methodological. The release was a concrete demonstration that the alignment stage, not just pretraining scale, can move a model a long way on chat-quality evaluations. By open-sourcing both the HelpSteer2 and HelpSteer2-Preference datasets and by publishing the reward model and the recipe, NVIDIA gave the wider community a reproducible path to strong RLHF, part of the same open source AI posture that runs through the Nemotron program and its use of curated and synthetic data for post-training ^[4][5]. At the same time, the episode became a case study in reading benchmarks carefully: leading three automatic evals is not the same as being the best general-purpose assistant, a distinction NVIDIA's own caveats acknowledged.

Licensing and availability

Use of Llama-3.1-Nemotron-70B-Instruct is governed by two overlapping terms: Meta's Llama 3.1 Community License Agreement (because it is derived from Llama 3.1) and the NVIDIA Open Model License ^[1][2]. The model is ready for commercial use under those terms.

NVIDIA distributed the model in several forms. On Hugging Face it published both the native NeMo-format checkpoint (nvidia/Llama-3.1-Nemotron-70B-Instruct) and a Hugging Face Transformers-compatible conversion (the "-HF" variant), plus the corresponding reward-model checkpoints ^[1][2][6]. It is also offered as an NVIDIA NIM inference microservice and hosted for free trial use at build.nvidia.com behind an OpenAI-compatible API ^[2]. The model accepts up to 128,000 input tokens and produces up to about 4,000 output tokens ^[1].

ELI5

Imagine a smart student (Meta's Llama 3.1, 70 billion "brain cells") who already knows a lot but sometimes gives sloppy or unhelpful answers. NVIDIA hired a very good "grader" (a reward model) that was trained on thousands of human judgments about what makes an answer helpful, correct, and clear. Then NVIDIA had the student practice answering questions over and over, and every time the grader gave a high score, the student learned to answer more like that. After this coaching, the student did better than some much bigger and more famous models on a set of automatic answer-quality tests, even though its "brain" never got any bigger. The catch is that those tests are graded by another AI and mostly measure chat quality, so acing them does not mean the model is smarter at everything.

See also

• Nemotron (NVIDIA's model and dataset family)
• Llama Nemotron (the separate 2025 reasoning family)
• Llama 3.1 (the base model)
• RLHF and reinforcement learning
• Instruction tuning
• Arena Hard

References

1. "nvidia/Llama-3.1-Nemotron-70B-Instruct." NVIDIA / Hugging Face model card, October 2024. https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct ↩
2. "nvidia/Llama-3.1-Nemotron-70B-Instruct-HF." NVIDIA / Hugging Face model card, October 2024. https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF ↩
3. "NVIDIA Launches Family of Open Reasoning AI Models for Developers and Enterprises to Build Agentic AI Platforms." NVIDIA Newsroom, March 18, 2025. https://nvidianews.nvidia.com/news/nvidia-launches-family-of-open-reasoning-ai-models-for-developers-and-enterprises-to-build-agentic-ai-platforms ↩
4. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, Oleksii Kuchaiev. "HelpSteer2: Open-source dataset for training top-performing reward models." arXiv:2406.08673, June 12, 2024. https://arxiv.org/abs/2406.08673 ↩
5. Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, Yi Dong. "HelpSteer2-Preference: Complementing Ratings with Preferences." arXiv:2410.01257, October 2, 2024 (ICLR 2025). https://arxiv.org/abs/2410.01257 ↩
6. "nvidia/Llama-3.1-Nemotron-70B-Reward-HF." NVIDIA / Hugging Face model card, October 2024. https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward-HF ↩
7. Michael Nuñez. "Nvidia just dropped a new AI model that crushes OpenAI's GPT-4 (no big launch, just big results)." VentureBeat, October 15, 2024. https://venturebeat.com/ai/nvidia-just-dropped-a-new-ai-model-that-crushes-openais-gpt-4-no-big-launch-just-big-results ↩
8. "nvidia/HelpSteer2." NVIDIA dataset card, Hugging Face, 2024. https://huggingface.co/datasets/nvidia/HelpSteer2 ↩
9. "Nvidia AI Quietly Launches Nemotron 70B: Crushing OpenAI's GPT-4 on Various Benchmarks." MarkTechPost, October 16, 2024. https://www.marktechpost.com/2024/10/16/nvidia-ai-quietly-launches-nemotron-70b-crushing-openais-gpt-4-on-various-benchmarks/ ↩
10. "Nvidia Releases Nemotron 70B Model; Claims to Beat GPT-4o and Claude 3.5 Sonnet." Beebom, October 2024. https://beebom.com/nvidia-releases-nemotron-70b-model/ ↩
11. Akhiad Bercovich et al. "Llama-Nemotron: Efficient Reasoning Models." arXiv:2505.00949, 2025. https://arxiv.org/abs/2505.00949