OLMo 2

OLMo 2 is the second generation of fully open large language models released by the Allen Institute for AI (Ai2), spanning 1B, 7B, 13B, and 32B parameter sizes. Unlike open weight models such as Llama or Qwen, OLMo 2 is "fully open": Ai2 publishes not just the weights but the complete training data, training code, recipe (hyperparameters, mixture weights, schedule), training logs, and thousands of intermediate checkpoints, so the entire model can be inspected and reproduced ^[1][2]. The 7B and 13B variants were announced on November 26, 2024, and a 32B flagship followed on March 13, 2025, with a 1B model added on May 1, 2025 ^[12]; Ai2 describes the 32B as the first fully open model to outperform OpenAI's GPT-3.5 Turbo and GPT-4o mini on a broad academic benchmark suite ^[3].

The series (sometimes stylised 2 OLMo 2 Furious after the technical report of the same name) ships every artefact needed to reproduce the work, with weights and code under the permissive Apache 2.0 licence and the training data under ODC-BY ^[4][9][10]. Ai2 defines its openness tier precisely as "models released with weights, training data, code, and evaluation in full, and thus can be fully inspected and reproduced" ^[2]. The 7B and 13B base models were trained on up to 5 trillion tokens and the 32B on up to 6 trillion tokens, with post-training built on Ai2's own Tulu 3 recipe ^[2][3].

Ai2 pitched OLMo 2 as a response to the gradual closing of the so called open weight ecosystem. Llama, Qwen, Mistral and Gemma had become the default reference models for academic work by 2024, but their training data and recipes remained proprietary, which made it impossible to ablate dataset decisions or audit the provenance of training material. OLMo 2 attempts to close that gap at competitive performance levels, with the 7B and 13B variants matching or beating Meta's Llama 3.1 8B and Alibaba's Qwen 2.5 7B on several evaluation suites despite using fewer training FLOPs, and the 32B base model approaching, though still trailing, open-weight Qwen 2.5 32B and Mistral Small 24B on Ai2's aggregate of academic benchmarks ^[1][6].

ELI5: what is OLMo 2 in simple terms?

Most AI labs hand you a finished cake (the model weights) but keep the recipe secret. Ai2 hands you the cake, the recipe, every ingredient, the oven settings, and photos of the cake at every stage of baking. OLMo 2 is that fully open cake: anyone with enough computers can bake an identical one or swap an ingredient to see what changes. The biggest version, OLMo 2 32B, was the first such open cake good enough to beat some of OpenAI's older paid models on school-test-style questions ^[3].

What is OLMo 2 and how did it come about?

The original OLMo, released in February 2024, established Ai2's commitment to releasing models with their full training recipe. That first generation included 1B and 7B base models trained on the Dolma corpus, an open 3 trillion token web mixture also published by Ai2. Even at launch the 7B was competitive with Meta's Llama 2 7B on most academic benchmarks, but it lagged Mistral 7B and newer models such as Llama 3 and Qwen 1.5 7B. The original models also showed instability during long training runs, with intermittent loss spikes that forced engineers to manually intervene and restart from earlier checkpoints. A mid year refresh called OLMo 1.7 7B added a 24 point improvement on the MMLU benchmark by upgrading to a revised dataset (Dolma 1.7) and a longer 2 trillion token training run, but the architecture, optimiser and training hyperparameters were largely unchanged from the original release.

OLMo 2 was conceived as a clean redesign. The Ai2 OLMo team had spent most of 2024 cataloguing the sources of instability and inefficiency in the first generation, and used the new release to apply a coordinated set of fixes drawn from the wider open weights literature. The headline result was a family of base models that the team claimed sat on the Pareto frontier of performance versus training compute, beating contemporaneous open weight models at equivalent parameter counts while using fewer training FLOPs ^[1].

Which models are in the OLMo 2 family?

Each base model is accompanied by an SFT checkpoint (supervised fine tuning only), a DPO checkpoint, and a final Instruct release that adds reinforcement learning. The 7B and 13B Instruct models use PPO with verifiable rewards while the 32B Instruct model uses Group Relative Policy Optimisation (GRPO), the algorithm popularised by DeepSeek earlier in the year. Ai2 also released the reward models used in training, for example allenai/OLMo-2-1124-7B-RM, so that researchers can replicate or vary the preference tuning stage independently ^[7]. The naming convention encodes the release date in the suffix, so 1124 refers to November 2024 and 0325 to March 2025.

What architecture does OLMo 2 use?

The OLMo 2 base architecture is a decoder only Transformer that departs from the usual pre normalisation layout by normalising the outputs of the attention and feedforward sublayers rather than their inputs (a "reordered norm"), one of several focused changes intended to improve training stability and per token efficiency relative to the first OLMo generation ^[1].

The most important architectural change was the switch from the non parametric layer normalisation used in OLMo 1 to RMSNorm, a simpler and slightly cheaper alternative that omits the mean centering step. Ai2 also added QK normalisation, which applies RMSNorm to the query and key projections inside each attention head before computing attention scores. This combination was reported in the OLMo 2 technical report to substantially reduce the frequency of loss spikes during long training runs, which had been a recurring operational headache during OLMo 1 training ^[1].

The positional encoding scheme was upgraded to rotary positional embeddings, replacing the absolute learned positional embeddings used in OLMo 1. Z loss regularisation was added to discourage the final softmax from saturating, a trick borrowed from the PaLM family. The team also revisited weight initialisation, choosing a scheme that preserves activation and gradient magnitudes across the depth of the network. For tokenisation, OLMo 2 moved to a larger vocabulary, borrowing the pre-tokeniser and vocabulary of OpenAI's cl100k tokeniser (used for GPT-3.5 and GPT-4) in place of the first OLMo's GPT-NeoX-based tokeniser, while keeping the masking tokens of earlier Dolma data for backward compatibility ^[1].

Context length was held at 4096 tokens across all three variants for the initial release, a deliberately conservative choice that reflects Ai2's focus on reproducibility and stable evaluation rather than chasing long context benchmarks.

How was OLMo 2 trained?

Two stage curriculum

OLMo 2 introduced a two stage pretraining curriculum that became one of the most discussed contributions of the project. The first stage uses a broad web heavy mixture called OLMo Mix 1124, which comprises roughly 3.9 trillion tokens drawn from the DataComp Language Model (DCLM) corpus, Dolma, the StarCoder code corpus, and Proof Pile II for mathematics ^[1][9]. This stage consumes more than 90 percent of the total training budget and is run for about one epoch (4 trillion tokens) for the 7B, 1.2 epochs (5 trillion tokens) for the 13B, and around 1.5 epochs (6 trillion tokens) for the 32B ^[4][6].

The second stage applies an annealing curriculum on Dolmino Mix 1124, a roughly 843 billion token corpus composed of about 50 percent high quality filtered web data combined with academic content, question answering datasets, instruction style examples, and mathematics ^[1][10]. During this stage the learning rate is annealed from its first stage value down to zero. Ai2 trained multiple parallel runs on different slices of Dolmino Mix (the 7B souped three runs over the same 50 billion token sample in different data orders, while the 13B and 32B souped three runs on 100 billion token samples plus one run on a 300 billion token sample) and then merged the resulting checkpoints, a technique sometimes called model souping, to produce the final base weights. The Ai2 team argues this curriculum captures the benefit of staged training (sharpening the model on high quality data near the end) while keeping the bulk of the run on diverse web data where the optimisation dynamics are well understood.

Dolma 2 and the open data story

The pretraining corpora were released alongside the model weights under the ODC-BY licence (the weights and code use Apache 2.0), continuing the practice established by Dolma and the original OLMo ^[9][10]. OLMo Mix 1124 supersedes the older Dolma 1.7 corpus and is often referred to colloquially as Dolma 2, though Ai2's official naming uses the dated 1124 suffix. The mix is dominated by DCLM-Baseline web data (about 3.7 trillion of its 3.9 trillion tokens), with smaller StarCoder code, peS2o academic, arXiv, OpenWebMath, Algebraic Stack and Wikipedia components, and Ai2 published per source token counts for each ^[9].

Training infrastructure

The 7B and 13B models were each trained partly on Ai2's two clusters, with the bulk of the 7B run on Jupiter (a 128 node GPU cluster in Austin, Texas, operated by Cirrascale) and the bulk of the 13B run on Augusta ^[1]. The 32B was trained on the Augusta cluster, a 160 node deployment of 8 GPU H100 nodes on Google Cloud Engine connected by GPUDirect TCPXO ^[1]. Ai2 reported a sustained training throughput of more than 1800 tokens per second per GPU for the 32B run, corresponding to roughly 38 percent model FLOPs utilisation. The team has stated publicly that "OLMo 2 32B takes only one third of the cost of training Qwen 2.5 32B while reaching similar performance," a comparison that became a recurring talking point in coverage of the release ^[3].

How does Tulu 3 post training work?

The instruction tuned and aligned variants of OLMo 2 were produced by applying the Tulu 3 post training recipe, also developed and released by Ai2. Tulu 3 is a permissively licensed alignment pipeline built around three stages: supervised fine tuning on a curated instruction mixture, preference optimisation with DPO, and a final reinforcement learning stage that Ai2 calls Reinforcement Learning with Verifiable Rewards (RLVR) ^[11]. RLVR replaces the learned reward model used in conventional RLHF with deterministic checkers for tasks where correctness can be verified programmatically, such as mathematical problem solving against ground truth answers and instruction following against format constraints.

For the 7B and 13B Instruct models, RLVR was implemented using PPO. The 32B Instruct model switched to GRPO, which removes the value function critic used by PPO and replaces it with a baseline computed from a group of sampled completions. The 32B Instruct release also used a revised post training mixture called Tulu 3.1 with SFT and preference data filtered to remove synthetic responses that mentioned a date cutoff, and majority-vote filtering of its synthetic math data ^[3].

The Tulu 3 SFT mixture combines public datasets such as the FLAN collection, OpenAssistant conversations, Tulu's own persona based synthetic data, and competition mathematics datasets. Ai2 released variant SFT mixtures specifically constructed for OLMo 2 base models, distinct from the original Tulu 3 mixtures, because the strongest mixture turned out to depend on the base model's pretraining distribution.

How does OLMo 2 perform on benchmarks?

The headline benchmark numbers for the base models, reported by Ai2 in the OLMo 2 technical report and its accompanying blog posts, place the 7B and 13B variants ahead of similarly sized contemporaries on several core academic suites ^[1][2].

For the Instruct variants, Ai2's published comparison tables include side by side scores against Llama 3.1, Qwen 2.5, Gemma 2, and Ai2's own Tulu 3 8B baseline. The OLMo 2 7B Instruct numbers come from the model card on Hugging Face ^[7].

The 32B Instruct model targets a different weight class and is compared against larger or stronger reference models, including OpenAI's GPT-3.5 Turbo and GPT-4o mini ^[3].

According to Ai2, "OLMo 2 32B is the first fully-open model (all data, code, weights, and details are freely available) to outperform GPT3.5-Turbo and GPT-4o mini" on a suite of multi-skill academic benchmarks ^[3]. These numbers should be read with the usual caveats about contamination, prompting format, and the choice of evaluation harness. Ai2 evaluates on its own Open Language Model Evaluation System (OLMES), a 20 task harness specifically designed for the OLMo development cycle. OLMES distinguishes between a smaller pool of development tasks tracked during training and a larger pool of held out tasks used only for final reporting, in an attempt to limit overfitting to the eval suite.

How open is OLMo 2?

The term "fully open" is used by Ai2 in a specific and narrow sense. Ai2 reserves it for "models released with weights, training data, code, and evaluation in full, and thus can be fully inspected and reproduced" ^[2]. In the OLMo 2 vocabulary that includes the model weights, the complete training data, the training code, the training recipe (hyperparameters, mixture weights, schedule), training logs, and intermediate checkpoints. The intention is that an outside party with sufficient compute could reproduce the entire run and obtain a numerically similar model, or substitute in different data and rerun the recipe to ablate a specific choice. This standard sits noticeably above the "open weights" tier occupied by Llama, Qwen, Mistral and Gemma, all of which release weights but withhold the training data and the precise recipe.

For OLMo 2 the artefact list is unusually complete. The team published the full set of intermediate checkpoints saved every 1000 training steps, the merged Dolmino Mix files used in the annealing stage, the per source filtering scripts, the OLMES evaluation code, and the reward models used for preference tuning ^[1][9][10]. Even relatively minor artefacts such as the configuration files for the merging step were released.

This approach has trade offs. The data and checkpoint releases consume tens of terabytes of storage on Hugging Face; OLMo Mix 1124 alone is about 22.4 TB uncompressed ^[9]. The training data also exposes Ai2 to copyright and provenance questions that the open weights model providers can sidestep by keeping their corpora private, .

Is OLMo 2 open source, and what licence does it use?

OLMo 2 weights and source code are released under the Apache 2.0 licence, while the OLMo-Mix-1124 and Dolmino-Mix-1124 training datasets use the ODC-BY licence ^{[4][9][10][13]}. This is a permissive licence that allows commercial use, modification, and redistribution provided the licence notice is preserved. It places no restrictions on the use case or downstream user, unlike Meta's Llama Community Licence which imposes acceptable use policies and a 700 million monthly active user threshold above which the licensee must request a separate commercial agreement.

The Apache 2.0 choice was deliberate and is part of Ai2's broader policy argument for open source AI. The 32B release in particular was framed as proof that fully open models could match closed system performance at academic benchmark scales ^[3].

How does OLMo 2 compare to Llama, Qwen, and Gemma?

The OLMo 2 family slots into a 2024 to 2025 open model landscape where the main competition came from Meta, Alibaba, Google DeepMind, and the smaller Mistral and Cohere labs. At roughly equivalent scale the closest peers are listed below.

At the 7B parameter level the OLMo 2 7B is broadly competitive with Llama 3.1 8B on general knowledge benchmarks and ahead on GSM8K, but trails Qwen 2.5 7B on mathematics and MMLU. At 13B the OLMo 2 13B base model beats Llama 2 13B, Mistral Nemo 12B and StableLM 2 12B on Ai2's OLMES average, but trails Qwen 2.5 14B (72.2 versus 68.3) on most tasks, including mathematics ^[5]. The 32B Instruct model is the only fully open model in its weight class as of mid 2025; both Llama 3 and Qwen 2.5 use weights only licences, while Gemma never released a model above 27B in this generation.

In terms of pure benchmark headroom, OLMo 2 32B Instruct lands above the GPT-3.5 Turbo and GPT-4o mini line on aggregate scores but below Llama 3.3 70B and the 2025 frontier reasoning models ^[3]. The Ai2 framing is that this is the expected position for the first fully open run at the 32B scale, and that the gap should close with later iterations such as OLMo 3.

How was OLMo 2 received?

Reception in the AI research community was generally positive, with particular attention given to the two stage curriculum, the open release of intermediate checkpoints, and the QK normalisation result. The 32B release in March 2025 drew the most coverage because it crossed a symbolic threshold by beating OpenAI's GPT-3.5 Turbo and GPT-4o mini on Ai2's chosen aggregate, while using roughly one third the training compute of Qwen 2.5 32B ^[3][14].

The technical report "2 OLMo 2 Furious" became one of the more widely cited open language model papers of early 2025. A common observation was that the OLMo 2 13B was the first fully open model that could plausibly be used as a research baseline by labs that had previously relied on Llama 2 or Llama 3 for ablations, because it ships with enough surrounding artefacts to make controlled experiments tractable.

Criticisms were targeted rather than sweeping. The 4096 token context length was widely seen as too short for late 2024, when most other open models had moved to at least 32k tokens. Coding performance lagged the Qwen 2.5 Coder family, and the 7B Instruct trailed Llama 3.1 8B Instruct on MMLU and IFEval. The 32B Instruct was below Qwen 2.5 32B on the harder mathematics benchmarks despite leading on GSM8K. Ai2's later OLMo 3 family, released in November 2025, targeted several of these gaps.

Continue reading

• OLMoOLMo (Open Language Model) is a family of fully open large language models built by the Allen Institute for AI (Ai2) and first released on February 1, 2024.Mentioned 77 times above
• Tülu 3Tülu 3 is a fully open post-training recipe and a corresponding family of instruction-tuned language models released by the Allen Institute for AI (Ai2) on November 21, 2024.Mentioned 9 times above
• DolmaDolma is an open three-trillion-token English pretraining corpus released by the Allen Institute for AI (AI2) to power its fully open OLMo language models and to let researchers study how training data shapes…Mentioned 8 times above
• Llama 3.1Llama 3.1 is a family of open-weight large language models released by Meta on July 23, 2024, in three sizes, 8 billion, 70 billion, and 405 billion parameters, each shipped in both a pre-trained base form and…Mentioned 8 times above
• Gemma 2Gemma 2 is a family of open-weights large language models developed by Google DeepMind and released starting June 27, 2024, in three parameter sizes: 2 billion (2B), 9 billion (9B), and 27 billion (27B).Mentioned 3 times above
• RRLVRReinforcement Learning with Verifiable Rewards (RLVR) is a post-training paradigm for large language models in which the reward signal comes from a deterministicMentioned 3 times above

More from See also

• OLMoE
• Allen Institute for AI
• Open-source AI
• RMSNorm
• Rotary Position Embedding
• DPO
• Reinforcement Learning from Human Feedback (RLHF)

Topics:AI ModelsLarge Language ModelsOpen Source AIResearch Organizations

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12OLMo Team. "2 OLMo 2 Furious." arXiv preprint 2501.00656, January 2025. arxiv.org/...2501.00656
2. ^1 ^2 ^3 ^4 ^5Allen Institute for AI. "OLMo 2: The best fully open language model to date." November 26, 2024. allenai.org/...olmo2
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Allen Institute for AI. "OLMo 2 32B: First fully open model to outperform GPT 3.5 and GPT 4o mini." March 13, 2025. allenai.org/...olmo2-32B
4. ^1 ^2 ^3Allen Institute for AI. "allenai/OLMo-2-1124-7B." Hugging Face model card. huggingface.co/...OLMo-2-1124-7B
5. ^Allen Institute for AI. "allenai/OLMo-2-1124-13B." Hugging Face model card. huggingface.co/...OLMo-2-1124-13B
6. ^1 ^2Allen Institute for AI. "allenai/OLMo-2-0325-32B." Hugging Face model card. huggingface.co/...OLMo-2-0325-32B
7. ^1 ^2Allen Institute for AI. "allenai/OLMo-2-1124-7B-Instruct." Hugging Face model card. huggingface.co/...OLMo-2-1124-7B-Instruct
8. Allen Institute for AI. "allenai/OLMo-2-0325-32B-Instruct." Hugging Face model card. huggingface.co/...OLMo-2-0325-32B-Instruct
9. ^1 ^2 ^3 ^4 ^5 ^6 ^7Allen Institute for AI. "OLMo-Mix-1124 dataset." Hugging Face. huggingface.co/...olmo-mix-1124
10. ^1 ^2 ^3 ^4 ^5Allen Institute for AI. "Dolmino-Mix-1124 dataset." Hugging Face. huggingface.co/...dolmino-mix-1124
11. ^Lambert, Nathan et al. "Tulu 3: Pushing Frontiers in Open Language Model Post-Training." arXiv preprint 2411.15124, November 2024. arxiv.org/...2411.15124
12. ^Allen Institute for AI. "OLMo release notes." allenai.org/...release-notes
13. ^Allen Institute for AI GitHub. "allenai/OLMo." github.com/...OLMo
14. ^MarkTechPost. "Allen Institute for AI (AI2) Releases OLMo 32B." March 14, 2025. marktechpost.com/...uite-of-multi-skill-benchmarks

Show all 14 references