Olmo-core is an open-source Python library for modeling and large-scale distributed training with PyTorch. The Allen Institute for AI (Ai2) develops and maintains it under the Apache 2.0 license. It supplies training infrastructure for the OLMo ecosystem rather than a single pretrained model.[1][2]

Ai2 announced Olmo-core 3 on October 1, 2026, with a redesigned mixture-of-experts (MoE) training system that targets expert communication and memory costs.[3]

Overview

Field	Detail
Developer	Allen Institute for AI (Ai2)[1]
Software type	Language-model modeling and training library[1]
Implementation	Python and PyTorch[1]
License	Apache 2.0[1]
Package name	`ai2-olmo-core`[13]
Python namespace	`olmo_core`[2]
Major release covered here	Olmo-core 3, announced October 1, 2026[3]

The package identity, license, and implementation are documented in the repository and package metadata.[1][13] The library's version number is separate from the names of models trained with it: Olmo-core 3 is not the same subject as OLMo 3.[2][3]

Relationship to the OLMo research ecosystem

Olmo-core began as a major rewrite of the original OLMo modeling and training code, with an emphasis on performance and reusable research interfaces.[5] Its repository includes official scripts for OLMo 2 32B and OLMo 3 7B and 32B models.[2] The OLMo 2 research release included training code and recipes alongside weights, datasets, logs, and intermediate checkpoints, making the training system part of the published research artifacts.[15]

The stack builds on the dense OLMo 3 and sparse OLMoE work. It keeps experts resident on GPUs instead of gathering their weights for every microbatch.[3]

Configuring and running an experiment

The framework separates the experiment's configuration from the objects built at runtime. A typical setup defines a model, train module, dataset, data loader, and trainer. The script builds those components, calls Trainer.fit(), and can run through torchrun or the Beaker launch integration.[5]

Component	Responsibility
Model configuration	Describes the architecture and builds its modules
Train-module configuration	Connects the model to training operations and its optimizer
Dataset and data-loader configuration	Supplies training instances and each process's portion of a batch
`TrainerConfig`	Builds the trainer and configures the run
Launch configuration	Starts the experiment on the chosen execution platform

This separation allows a research script to describe an experiment before allocating its runtime objects.[5]

The trainer supports asynchronous checkpoint saving, asynchronous metric logging, and callbacks that extend the training loop. Its checkpoint settings distinguish loading model state from restoring optimizer or trainer state. A local checkpoint directory must be on a filesystem shared by all ranks; a remote URL is another supported save location.[6]

Data preparation and loading

The built-in text pipeline uses pre-tokenized one-dimensional NumPy arrays. Special tokens, including end-of-sequence tokens, belong in those arrays before loading; padding is handled separately.[7]

Strategy	Behavior
Concatenate and chunk	Joins documents and divides tokens into fixed-length instances
Intra-document masking	Uses document lengths to prevent attention across document boundaries
Packed fixed-length instances	Packs documents with limited padding; long documents can be truncated or fragmented
Padded documents	Keeps documents separate, padding or fragmenting them to the target length
Interleaved documents	Interleaves document chunks to place related tokens farther apart
Variable sequence length	Samples lengths through a curriculum while retaining the configured token budget per batch

The loading guide documents these alternatives and their tradeoffs. Custom loaders can implement DataLoaderBase, including saved state needed to resume their position in an epoch.[7]

Data mixing can use explicit file lists, predefined manifests, or ratio-based source sampling. The composable API mixes tokens, documents, or completed instances. Source-mixture configuration also controls repetition and the fraction of a source used, rather than only selecting a list of files.[8]

The Olmo-core 3 MoE training stack

Expert parallelism distributes experts across GPUs; pipeline parallelism divides layers between GPU groups; a distributed optimizer divides its state. These techniques reduce per-device storage requirements.[3]

The version 3 implementation includes the OLMoDDPModel stack and report-specific training entry points.[11][14]

Rowwise expert parallelism writes tokens into expert input buffers; routing metadata stays on GPUs. Ai2's preliminary eight-B300 comparison reported 52,000 tokens/second/GPU for a 47B MoE, versus 19,400 with its earlier system.[3] This compares complete stacks, not an isolated parallelism setting.[14]

Reported systems benchmarks

The October 2026 technical report measures training-system throughput, not the accuracy of completed models. Its headline configurations use random routing, sequence length 8,192, top-4 expert selection, one shared expert, and NVIDIA B300 GPUs with eight GPUs per node.[4]

Configuration	Total parameters	Active parameters per token	GPUs	Precision	Expert / pipeline parallelism	Highest observed useful-model TFLOP/s/GPU
Tiny-64E	12.91B	1.59B	16	BF16	1 / 1	903
Ultra-128E	1.200T	58.36B	512	MXFP8	8 / 8	858
Ultra-256E	2.380T	58.41B	512	MXFP8	32 / 8	689

These are selected operating points from Table 30, not a controlled scaling curve. The Ultra configurations use per-layer activation recomputation. The 2.380T test uses DeepEP v2 and lasted nine steps without stabilized throughput. The rates are the highest observed readings, not repetition averages or proof of a trained trillion-parameter model's quality.[4]

Reproducing the report's experiments

The tagged release's src/examples/olmo_ddp/ directory contains entry points for the technical report's throughput runs, transport profiles, precision experiments, and routing-loss experiments. Its README maps each configuration to a script and lists model size, routing, precision, and parallelism settings.[14]

The production scripts assume eight GPUs per node. They require the Beaker extra, FlashAttention 4, suitable checkpoint storage, and access to the specified data mix. DeepEP-based scripts require a separate DeepEP checkout.[14]

Not every experiment discussed in the report is available in that release. The README explicitly excludes reproducing the two-batch-overlap measurement from the tagged code. It also explains that its supported DDP-versus-FSDP comparison changes routers, optimizers, and sparse kernels as well as parallelization, so it cannot isolate DDP's effect alone.[14]

Checkpoints and inference

The checkpointing API can save and load local or remote checkpoints, restore them with a different distributed topology, and unshard model or optimizer state. It provides an interface for checkpoint handling separate from the training loop.[9]

Olmo-core also has native autoregressive generation. TransformerGenerationModule.from_checkpoint() loads a checkpoint, while generate_batch() produces token IDs and can return logits or log probabilities. The generation configuration controls sampling, output length, and caching.[10]

The interactive interface runs as:

python -m olmo_core.generate.chat path/to/checkpoint

The chat loop supports conversation history and chat templates; instruction-tuned models need the appropriate template.[10] The repository labels its native inference path beta and also documents running released OLMo checkpoints through Hugging Face Transformers or vLLM.[2]

Dense, hybrid, and multimodal components

Olmo-core is not limited to the redesigned MoE path. Version 2.6.0 added Qwen3.5 dense configurations with a hybrid Gated DeltaNet and full-attention architecture. It also added a configurable vision transformer, a vision-to-language connector, and MultimodalLM, which inserts image patch tokens into the language-model token stream. The vision encoders include CLIP, SigLIP, and SigLIP2 variants.[12]

That release also added an end-of-training Hugging Face conversion callback and learning-rate schedulers with power-law or piecewise schedules.[12]

Installation and version compatibility

The repository instructs users to install PyTorch for their operating system and hardware first, followed by the package:[2]

pip install ai2-olmo-core

The v3.0.0 metadata requires Python 3.10 or later and torch>=2.10.0,<2.14. Optional dependencies impose additional constraints: the fla extra, also included in all, requires torch>=2.13.0,<2.14. A minimum base requirement therefore does not mean every optional backend works with that same version.[13]

Version 3.0.0 removed the olmo_core.model_ladder API and its orchestration scripts. It replaced FusedAttention with FusedAttentionV2, retaining legacy fused-attention parameter names and shapes for checkpoint compatibility. Mapping legacy fused rotary embeddings to regular rotary embeddings can still change floating-point results, so resumed training is not numerically identical.[11]

Other release changes include explicit attention-backend selection, document-boundary metadata for packed datasets, and safeguards that reject unsupported combinations during MoE export to Hugging Face. These compatibility checks matter when loading an older configuration or converting a model, even when its weight tensors are accepted.[11]

The general API guides still display a v2.6.0 label. The tagged v3.0.0 source retains the documented trainer, data-loader, checkpoint, and generation interfaces; older examples remain subject to the version 3 compatibility changes.[1][16]

References

Ai2. "OLMo-core" documentation overview. Accessed October 13, 2026.
Ai2. Olmo-core repository and README. Accessed October 13, 2026.
Ai2. "Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs". October 1, 2026.
Tianhua Tao et al. "Supercharging Olmo-core for Efficient and Scalable MoE Training". Technical report, October 2026, especially Section 14 and Table 30.
Ai2. "Introduction," Olmo-core documentation. Accessed October 13, 2026.
Ai2. "train," Olmo-core API reference. Accessed October 13, 2026.
Ai2. "Datasets and Data loading". Accessed October 13, 2026.
Ai2. "Data Mixing". Accessed October 13, 2026.
Ai2. "distributed.checkpoint," Olmo-core API reference. Accessed October 13, 2026.
Ai2. "Native Generation and Chat". Accessed October 13, 2026.
Ai2. Olmo-core v3.0.0 release notes. October 1, 2026.
Ai2. Olmo-core v2.6.0 release notes. August 11, 2026.
Ai2. Olmo-core v3.0.0 package metadata. Accessed October 13, 2026.
Ai2. Olmo-core v3.0.0 DDP MoE examples. Accessed October 13, 2026.
Team OLMo et al. "2 OLMo 2 Furious". December 31, 2024; revised October 8, 2025.
Ai2. Olmo-core v3.0.0 Python source. Tagged implementation, accessed October 13, 2026.

Editing: Olmo-core