Olmo-core

Olmo-core is an open-source Python library for modeling and large-scale distributed training with PyTorch. The Allen Institute for AI (Ai2) develops and maintains it under the Apache 2.0 license. It supplies training infrastructure for the OLMo ecosystem rather than a single pretrained model.^[1][2]

Ai2 announced Olmo-core 3 on October 1, 2026, with a redesigned mixture-of-experts (MoE) training system that targets expert communication and memory costs.^[3]

Overview

The package identity, license, and implementation are documented in the repository and package metadata.^[1][13] The library's version number is separate from the names of models trained with it: Olmo-core 3 is not the same subject as OLMo 3.^[2][3]

Relationship to the OLMo research ecosystem

Olmo-core began as a major rewrite of the original OLMo modeling and training code, with an emphasis on performance and reusable research interfaces.^[5] Its repository includes official scripts for OLMo 2 32B and OLMo 3 7B and 32B models.^[2] The OLMo 2 research release included training code and recipes alongside weights, datasets, logs, and intermediate checkpoints, making the training system part of the published research artifacts.^[15]

The stack builds on the dense OLMo 3 and sparse OLMoE work. It keeps experts resident on GPUs instead of gathering their weights for every microbatch.^[3]

Configuring and running an experiment

The framework separates the experiment's configuration from the objects built at runtime. A typical setup defines a model, train module, dataset, data loader, and trainer. The script builds those components, calls Trainer.fit(), and can run through torchrun or the Beaker launch integration.^[5]

This separation allows a research script to describe an experiment before allocating its runtime objects.^[5]

The trainer supports asynchronous checkpoint saving, asynchronous metric logging, and callbacks that extend the training loop. Its checkpoint settings distinguish loading model state from restoring optimizer or trainer state. A local checkpoint directory must be on a filesystem shared by all ranks; a remote URL is another supported save location.^[6]

Data preparation and loading

The built-in text pipeline uses pre-tokenized one-dimensional NumPy arrays. Special tokens, including end-of-sequence tokens, belong in those arrays before loading; padding is handled separately.^[7]

The loading guide documents these alternatives and their tradeoffs. Custom loaders can implement DataLoaderBase, including saved state needed to resume their position in an epoch.^[7]

Data mixing can use explicit file lists, predefined manifests, or ratio-based source sampling. The composable API mixes tokens, documents, or completed instances. Source-mixture configuration also controls repetition and the fraction of a source used, rather than only selecting a list of files.^[8]

The Olmo-core 3 MoE training stack

Expert parallelism distributes experts across GPUs; pipeline parallelism divides layers between GPU groups; a distributed optimizer divides its state. These techniques reduce per-device storage requirements.^[3]

The version 3 implementation includes the OLMoDDPModel stack and report-specific training entry points.^[11][14]

Rowwise expert parallelism writes tokens into expert input buffers; routing metadata stays on GPUs. Ai2's preliminary eight-B300 comparison reported 52,000 tokens/second/GPU for a 47B MoE, versus 19,400 with its earlier system.^[3] This compares complete stacks, not an isolated parallelism setting.^[14]

Reported systems benchmarks

The October 2026 technical report measures training-system throughput, not the accuracy of completed models. Its headline configurations use random routing, sequence length 8,192, top-4 expert selection, one shared expert, and NVIDIA B300 GPUs with eight GPUs per node.^[4]

These are selected operating points from Table 30, not a controlled scaling curve. The Ultra configurations use per-layer activation recomputation. The 2.380T test uses DeepEP v2 and lasted nine steps without stabilized throughput. The rates are the highest observed readings, not repetition averages or proof of a trained trillion-parameter model's quality.^[4]

Reproducing the report's experiments

The tagged release's src/examples/olmo_ddp/ directory contains entry points for the technical report's throughput runs, transport profiles, precision experiments, and routing-loss experiments. Its README maps each configuration to a script and lists model size, routing, precision, and parallelism settings.^[14]

The production scripts assume eight GPUs per node. They require the Beaker extra, FlashAttention 4, suitable checkpoint storage, and access to the specified data mix. DeepEP-based scripts require a separate DeepEP checkout.^[14]

Not every experiment discussed in the report is available in that release. The README explicitly excludes reproducing the two-batch-overlap measurement from the tagged code. It also explains that its supported DDP-versus-FSDP comparison changes routers, optimizers, and sparse kernels as well as parallelization, so it cannot isolate DDP's effect alone.^[14]

Checkpoints and inference

The checkpointing API can save and load local or remote checkpoints, restore them with a different distributed topology, and unshard model or optimizer state. It provides an interface for checkpoint handling separate from the training loop.^[9]

Olmo-core also has native autoregressive generation. TransformerGenerationModule.from_checkpoint() loads a checkpoint, while generate_batch() produces token IDs and can return logits or log probabilities. The generation configuration controls sampling, output length, and caching.^[10]

The interactive interface runs as:

python -m olmo_core.generate.chat path/to/checkpoint

The chat loop supports conversation history and chat templates; instruction-tuned models need the appropriate template.^[10] The repository labels its native inference path beta and also documents running released OLMo checkpoints through Hugging Face Transformers or vLLM.^[2]

Dense, hybrid, and multimodal components

Olmo-core is not limited to the redesigned MoE path. Version 2.6.0 added Qwen3.5 dense configurations with a hybrid Gated DeltaNet and full-attention architecture. It also added a configurable vision transformer, a vision-to-language connector, and MultimodalLM, which inserts image patch tokens into the language-model token stream. The vision encoders include CLIP, SigLIP, and SigLIP2 variants.^[12]

That release also added an end-of-training Hugging Face conversion callback and learning-rate schedulers with power-law or piecewise schedules.^[12]

Installation and version compatibility

The repository instructs users to install PyTorch for their operating system and hardware first, followed by the package:^[2]

pip install ai2-olmo-core

The v3.0.0 metadata requires Python 3.10 or later and torch>=2.10.0,<2.14. Optional dependencies impose additional constraints: the fla extra, also included in all, requires torch>=2.13.0,<2.14. A minimum base requirement therefore does not mean every optional backend works with that same version.^[13]

Version 3.0.0 removed the olmo_core.model_ladder API and its orchestration scripts. It replaced FusedAttention with FusedAttentionV2, retaining legacy fused-attention parameter names and shapes for checkpoint compatibility. Mapping legacy fused rotary embeddings to regular rotary embeddings can still change floating-point results, so resumed training is not numerically identical.^[11]

Other release changes include explicit attention-backend selection, document-boundary metadata for packed datasets, and safeguards that reject unsupported combinations during MoE export to Hugging Face. These compatibility checks matter when loading an older configuration or converting a model, even when its weight tensors are accepted.^[11]

The general API guides still display a v2.6.0 label. The tagged v3.0.0 source retains the documented trainer, data-loader, checkpoint, and generation interfaces; older examples remain subject to the version 3 compatibility changes.^[1][16]

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7Ai2. "OLMo-core" documentation overview. Accessed October 13, 2026.
2. ^1 ^2 ^3 ^4 ^5 ^6Ai2. Olmo-core repository and README. Accessed October 13, 2026.
3. ^1 ^2 ^3 ^4 ^5 ^6Ai2. "Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs". October 1, 2026.
4. ^1 ^2Tianhua Tao et al. "Supercharging Olmo-core for Efficient and Scalable MoE Training". Technical report, October 2026, especially Section 14 and Table 30.
5. ^1 ^2 ^3Ai2. "Introduction," Olmo-core documentation. Accessed October 13, 2026.
6. ^Ai2. "train," Olmo-core API reference. Accessed October 13, 2026.
7. ^1 ^2Ai2. "Datasets and Data loading". Accessed October 13, 2026.
8. ^Ai2. "Data Mixing". Accessed October 13, 2026.
9. ^Ai2. "distributed.checkpoint," Olmo-core API reference. Accessed October 13, 2026.
10. ^1 ^2Ai2. "Native Generation and Chat". Accessed October 13, 2026.
11. ^1 ^2 ^3Ai2. Olmo-core v3.0.0 release notes. October 1, 2026.
12. ^1 ^2Ai2. Olmo-core v2.6.0 release notes. August 11, 2026.
13. ^1 ^2 ^3Ai2. Olmo-core v3.0.0 package metadata. Accessed October 13, 2026.
14. ^1 ^2 ^3 ^4 ^5Ai2. Olmo-core v3.0.0 DDP MoE examples. Accessed October 13, 2026.
15. ^Team OLMo et al. "2 OLMo 2 Furious". December 31, 2024; revised October 8, 2025.
16. ^Ai2. Olmo-core v3.0.0 Python source. Tagged implementation, accessed October 13, 2026.