# Zephon

> Source: https://aiwiki.ai/wiki/zephon
> Updated: 2026-10-16
> Fact-checked: 2026-10-16
> Categories: Data & Datasets, Developer Tools, Open Source AI, Training & Optimization
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Zephon." aiwiki.ai, 16 Oct 2026. https://aiwiki.ai/wiki/zephon
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Zephon** is an open-source data loader for training [foundation models](https://aiwiki.ai/wiki/foundation_models) on text and multimodal data, developed by [DatologyAI](https://aiwiki.ai/wiki/datology_ai) and released under the [Apache License 2.0](https://aiwiki.ai/wiki/apache_license) in October 2026.[2][4][5] It tokenizes, packs, shuffles and mixes training data inside the training job rather than in a separate offline preprocessing pass, and its central claim is **elastic determinism**: the sequence of global training batches stays the same when a run is resumed on a different number of GPUs, when the parallelism of individual preprocessing operators changes, or when the execution backend changes.[1][2] The system is described in an arXiv preprint by Maximilian Böther, Josh Wills, Ties Robroek and 14 co-authors, first posted on 2 October 2026, and a reference integration with [TorchTitan](https://aiwiki.ai/wiki/torchtitan) was presented on the official [PyTorch](https://aiwiki.ai/wiki/pytorch) blog on 16 October 2026.[1][3]

## Background

DatologyAI says it built Zephon for its own data curation experiments, in which it compares training runs that differ only in the training data.[2][4] Two problems motivated the project, according to the company and the paper.

The first is the cost of offline materialization. A common way to make training data reproducible is to tokenize, mix and pack the whole corpus ahead of time and store the finished training sequences on disk, which turns the training-time loader into a simple index lookup.[3] DatologyAI argues that each change to the data pipeline (a new filter, a different tokenizer, a different packing setting) then requires another full pass over the corpus and another stored copy, and that many ablation runs read that copy only once.[1][2] The paper adds that for some modalities, such as video, materializing fully decoded tensors is infeasible because of the size inflation.[3]

The second problem is that many data loaders change the data order when the GPU count changes. The PyTorch post illustrates this with a 1B-parameter model trained in TorchTitan on 16 GPUs, checkpointed at step 150 and resumed on 8, 32 and 64 GPUs. With a standard loader, each resumed run received a different sequence of samples and produced its own loss curve; with Zephon, the authors report that the SHA-256 hash of every global batch matched the original run and the loss curves overlapped.[1] The paper notes that training on varying GPU counts is common in practice: smaller allocations are used to prototype large runs via [gradient accumulation](https://aiwiki.ai/wiki/gradient_accumulation), and jobs can be preempted or scaled up as cluster priorities change.[3] DatologyAI's launch post says that more and more of its experiments are designed and run by AI agents, which, like human researchers, may not notice when a result depends on the GPU count; the PyTorch post makes the same point about agent-run experiments in general.[1][2]

## Elastic determinism

The paper defines elastic determinism as "a deterministic sequence of global training data batches despite changes to the GPU topology across runs (e.g., due to GPU scarcity), frequent checkpoint-resume cycles, and different data processing execution backends."[3] The term itself comes from MosaicML's Streaming library, which DatologyAI credits with coining it.[2][3]

What makes the property hard, according to the authors, is that modern foundation model pipelines are what the paper calls "stateful n-to-m" pipelines. [Tokenization](https://aiwiki.ai/wiki/tokenization) can split one document into several subsequences (1-to-n), sequence packing merges several short sequences into one fixed-length example (n-to-1), filtering drops records (1-to-0), and operators such as packing, shuffle buffers and mixture enforcement keep state that depends on every earlier input.[3] In such a pipeline there is no static index that answers "which sample is used at step t", so the usual approach of resuming from a sample offset, or of remapping an index onto a new set of GPUs, does not apply.[2][3]

The authors position Zephon against existing loaders as follows (their characterization):

| Loader | What the Zephon authors say it supports |
|---|---|
| Mosaic Streaming | Elastic determinism for indexable datasets, by partitioning the sample space into "canonical nodes" remapped to the physical topology[3] |
| Google Grain | Elastic determinism for indexable pipelines, and deterministic resumption for stateful pipelines only under a fixed execution configuration[3] |
| Megatron-Energon | Online packing and reproducible scaling, but only while the total number of data loader workers stays fixed[1][2] |
| Zephon | Elastic determinism and bounded-cost resumption for online, stateful n-to-m pipelines[3] |

The paper describes Zephon as, "to our knowledge, the first data loader to provide elastic determinism and efficient resumption for online, stateful n-to-m foundation model data pipelines."[3]

## Design

A Zephon pipeline has three user-facing parts: a **Dataset** describing stored data, a **WorkSource** that decides what to train on and in what order, and a **Pipeline** of operators that turns samples into training batches.[4] Under the hood, an **Engine** compiles the pipeline into concurrent stages connected by bounded queues.[4]

### WorkSource and operators

The WorkSource plans the training curriculum from dataset metadata (shard listings and sample counts) and emits lightweight sample pointers of the form (dataset, shard, sample) rather than the data itself.[3][4] A fetch operator, implicitly the first in every pipeline, resolves those pointers into payloads, after which records flow through user-defined operators.[2] Because the curriculum is separate from processing, the data mixture can be changed without touching the preprocessing code and vice versa.[1] A `TokenEstimation` utility measures the token-to-byte ratio on a small sample of each dataset so that mixture weights can target token proportions even when the data is stored as raw strings.[1] The paper notes that the WorkSource protocol could be backed by other systems, giving as an example a centralized sample management server such as Mixtera, a data plane for foundation model training by Böther and co-authors published at SIGMOD 2026.[3]

Unlike PyTorch's `DataLoader`, which gives each worker a full copy of the pipeline, Zephon treats the pipeline as a graph of operators, each with its own degree of parallelism. The company argues this lets an expensive step such as tokenization get more CPU without also multiplying the number of processes reading from storage.[2][3]

### Three mechanisms for determinism

The PyTorch post describes three design decisions that build on one another:[1]

| Mechanism | How it works | What it guarantees |
|---|---|---|
| Accumulators and stateless workers | Each operator is split into a single-threaded accumulator, which sees every record in order and makes all stateful decisions (for example, which sequences are packed together), and a stateless function that does the expensive work across parallel workers. Records are numbered before dispatch and a reorder buffer restores the order afterwards.[1][2] | Operator parallelism becomes "a pure performance knob": doubling the workers does not change the output.[1] |
| Lanes instead of ranks | The data is divided into a fixed number of logical streams called lanes, each with its own ordered sample sequence and its own copy of operator state. When the number of [data-parallel](https://aiwiki.ai/wiki/data_parallelism) ranks changes, lanes are redistributed across ranks. A lane multiplexer hands out microbatches round-robin across each GPU's lanes so every lane contributes the same amount of data to every optimizer step.[1][2] | Identical global batches across GPU counts.[1] |
| Flush and replay instead of saved state | A checkpoint records only which input chunks were still in flight, not the contents of packing bins or shuffle buffers. Zephon periodically sends a flush sentinel through the pipeline (every eight chunks per lane by default) telling stateful operators to emit what they hold and reset. On restore it replays input from the last flush and discards what training already consumed.[1][2] | Small checkpoints that grow only marginally with training progress (2.61 MiB at step 100 versus 2.97 MiB at step 10,000 in the paper).[1][3] |

The paper calls the lane abstraction a generalization of Mosaic Streaming's canonical nodes to non-indexable, stateful pipelines, and describes "per-offset contributor tracking" as the mechanism that handles operators which reorder, merge and spawn samples during checkpointing.[3] DatologyAI says flushing adds about 0.02 percentage points of padding tokens at default settings.[2]

### Runners

Operators run on a runner, the execution substrate. The process runner uses OS processes to get around Python's Global Interpreter Lock and coalesces large tensor payloads into shared memory so that only metadata crosses the inter-process pipe. The thread runner keeps workers in one process and can run Python code in parallel on free-threaded Python builds.[2][3] Because the ordering logic sits above the runner abstraction, switching runners does not change the output.[2] An experimental [Ray](https://aiwiki.ai/wiki/ray)-based runner is also included.[4]

### Example

The PyTorch post gives this pipeline, which mixes two [Hugging Face](https://aiwiki.ai/wiki/hugging_face) datasets at a 50:50 token ratio, then tokenizes, packs, shuffles and batches them:[1]

```python
from zephon import Pipeline
from zephon.io import Dataset
from zephon.work import StaticMixtureWorkSource, TokenEstimation
source = StaticMixtureWorkSource(
    datasets=[
        Dataset.from_path("wiki", "hf://Salesforce/wikitext/wikitext-2-raw-v1/train"),
        Dataset.from_path("news", "hf://fancyzhx/ag_news/default/train"),
    ],
    mixture={"wiki": 0.5, "news": 0.5},  # a token ratio, not a sample ratio
    token_estimation=TokenEstimation(),
    seed=42,
)
pipeline = (
    Pipeline(source)
    .options(io_options={"cache": {"enabled": True}})
    .tokenize(tokenizer_id="gpt2", field="text", parallelism=4)
    .pack_flat(max_length=2048, algorithm="wrap")
    .shuffle(buffer_size=1024, seed=42)
    .batch(microbatch_size=8)
)
for batch in pipeline:
    train_step(batch.to_training())
```

A pipeline is a plain Python iterable, so it can be used with any training loop; the README says it will also run inside a PyTorch `DataLoader` or torchdata `StatefulDataLoader`, although the developers recommend iterating over it directly. Position is saved with `pipeline.checkpoint()` and restored with `pipeline.restore(state)`.[4]

## Resume constraints and limits

Elastic determinism in Zephon holds only under conditions the developers spell out:[1][4][7]

| Constraint | Detail |
|---|---|
| Lane count fixed up front | The number of lanes (`canonical_replicas`) must be chosen at the start of the run. The developers suggest a common multiple of the expected data-parallel sizes: 32 lanes cover 8, 16 and 32 GPUs. More lanes than needed slow checkpoint restore.[1][4] |
| Even division | Lanes must divide evenly across data-parallel replicas, and each optimizer step's batch must split evenly across the lanes.[1][4] |
| Recipe fixed | Data, tokenizer, seed, mixture, context length and global batch size must stay the same when resuming on more or fewer ranks. Performance settings such as operator parallelism may change.[1] |
| Only data parallelism counts | Tensor, pipeline and context parallelism do not affect the lane calculation.[4] |

The developers stress that Zephon "guarantees the data, not the arithmetic": global batches are identical across GPU counts, but losses can still differ slightly because floating-point reduction order in the training loop depends on how many GPUs share the work.[1] The README makes the same point, noting that identical losses or weights also depend on kernel selection, which a data loader cannot control.[4] In the paper's vision-language experiment, the authors trained in fp32 with deterministic CUDA options enabled to isolate the effect of the input sequence, and note that with lower precision loss curves can differ more across GPU configurations even when the global batches are identical.[3]

The README describes Zephon as beta software and the Ray runner as experimental.[4]

## TorchTitan integration

DatologyAI maintains reference integrations for two training frameworks, each in its own repository: TorchTitan (`datologyai/torchtitan-zephon`) and [Megatron-LM](https://aiwiki.ai/wiki/megatron_lm) (`datologyai/Megatron-LM-zephon`).[4] The TorchTitan integration replaces the default data loader with a Zephon pipeline built from a data recipe (datasets, mixture weights, packing, shuffling and parallelism settings), and leaves model construction, optimization, distributed training and checkpoint coordination to TorchTitan.[1][7] Zephon's state is stored as opaque bytes in TorchTitan's distributed checkpoint alongside the model, and because that state covers every lane, a run can resume on a different GPU count from the same checkpoint.[1][7] At launch the integration checks that lanes divide evenly across data-parallel replicas and that each optimizer step's batch splits evenly across lanes, so a misconfigured run fails immediately.[1]

The integration is opt-in, and the integration README says it currently supports online raw text only.[7] Data recipes are TOML files listing sources (local, S3 or Hugging Face paths), weights, cache settings and per-stage parallelism.[7] The repository includes a CPU-only demo that checks a data stream resumed after a data-parallel resize (2 to 1 and 1 to 2) against an uninterrupted stream, and a GPU smoke test that trains TorchTitan's debug model for two steps, checkpoints, and resumes on a configurable GPU count.[1][7]

## Reported results

All results below are self-reported by the Zephon developers.

### Downstream evaluation spread across GPU counts

The developers trained a 1B-parameter model from scratch for 20B tokens on 8, 16, 32 and 64 GPUs and measured how much final evaluation scores varied with GPU count. The paper's setup used TorchTitan with sequence length 4,096, AdamW, a global batch size of 448 and a shared seed checkpoint on H100 GPUs.[3]

| Evaluation suite | Spread, standard loader | Spread, Zephon | Reference level used by authors |
|---|---|---|---|
| [DCLM](https://aiwiki.ai/wiki/dclm) Core v1 (blog figures) | up to 0.82 points | 0.05 points | 0.2 points[1][2] |
| [FineWeb](https://aiwiki.ai/wiki/fineweb) suite (blog figures) | up to 0.64 points | 0.011 points | 0.15 points[1][2] |
| DCLM Core v1 (paper v1, two-seed average) | 0.922 pp | 0.061 pp | 0.2[3] |
| FineWeb suite (paper v1, two-seed average) | 0.555 pp | 0.007 pp | about 0.15 pp[3] |

The authors say the standard loader's spread is roughly four times the thresholds that the DCLM and FineWeb authors used when evaluating data curation methods, meaning that an ablation comparing two data mixtures on different GPU allocations "could easily be measuring the GPU count instead of the data."[1] DatologyAI notes that these reference values come from published experiments and are not statistical significance thresholds.[2] It also notes that on DCLM Core v1 the single Zephon data stream happened to produce a slightly weaker model than the standard loader's streams, which it attributes to chance rather than to any trade-off between determinism and model quality.[2]

In a second test, a 2.1B-parameter vision-language model (frozen [SigLIP](https://aiwiki.ai/wiki/siglip) 2 encoder plus a Qwen3-1.7B backbone) trained for 300 steps on 8 to 64 GPUs, the paper reports that Zephon's loss curves had a 5.84 times lower mean absolute pairwise difference than an internal Streaming-based loader.[3]

### Throughput and checkpointing

| Benchmark (paper, single AWS r8id.48xlarge host) | Result reported by the authors |
|---|---|
| Text fetch-and-batch | Grain peaked at 201k samples/s; Zephon's process runner saturated at 59k samples/s. The authors say worker-based loaders have a structural advantage for small text payloads.[3] |
| Vision-language fetch-and-batch | Zephon process runner 16k samples/s versus 5.8k for Grain; LitData and Streaming did not scale beyond about 2k.[2][3] |
| Full vision-language pipeline | Zephon and Grain effectively tied at 32-way parallelism (29.8k vs 29.9k usable tokens/s/GPU); Grain led by about 8% at 64-way. Both exceeded the 16.9k usable tokens/s/GPU the model needed and delivered over 23 times the throughput of an internal Streaming-based loader.[2][3] |
| Free-threaded Python 3.14t | Thread runner rose from 10k to 59k samples/s on vision-language fetch-and-batch, which the PyTorch post describes as about 3.7 times faster than the process runner on standard Python.[1][3] |
| Cost of the guarantees | 16% (text) and 22% (vision-language) lower peak throughput than a stripped pipeline that can neither order its output nor checkpoint.[3] |
| Checkpoint size | 2.61, 2.92 and 2.97 MiB at 100, 1,000 and 10,000 steps.[3] |
| Restore time | 48.5 s to first batch after restore at step 100, versus 44.8 s for a cold start; 70.2 s and 73.8 s at 1,000 and 10,000 steps. With 64 lanes per rank, time to first batch after restore rose to 403 s.[3] |

## Release and availability

| Date | Event |
|---|---|
| 2 October 2026 | Preprint "Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines" posted to arXiv (2610.03087); the listing says it is under revision at VLDB 2027[3] |
| 7 October 2026 | Zephon 0.1.0 published on PyPI; DatologyAI launch post and public GitHub repository[2][6] |
| 16 October 2026 | PyTorch blog post on the TorchTitan integration[1][8] |

Zephon requires Python 3.10 or newer and is installed with `uv pip install zephon`. Optional extras add readers for cloud storage (S3, GCS, Azure), Hugging Face datasets, Parquet, MosaicML Streaming/MDS, LitData, Vortex and Lance, plus the experimental Ray runner.[4][6] The paper puts the codebase at about 20,000 lines of code excluding tests.[3] The paper's 17 authors are mostly from DatologyAI, with co-authors also affiliated with [ETH Zurich](https://aiwiki.ai/wiki/eth_zurich) (Ana Klimovic, and lead author Böther jointly with DatologyAI), ITU Copenhagen, [Stanford University](https://aiwiki.ai/wiki/stanford_university), [Carnegie Mellon University](https://aiwiki.ai/wiki/carnegie_mellon_university) and Arcee AI.[3]

DatologyAI says its next steps include dynamic curricula, in which a WorkSource changes mixture weights during a run while keeping the determinism guarantees, a Ray-based execution engine for compute-heavy audio and video preprocessing, and a more declarative interface for defining data mixes.[1][2]

## References

1. Josh Wills, Maximilian Böther, Ties Robroek and the Zephon team. "Bringing Elastically Deterministic Data Loading to TorchTitan with Zephon". PyTorch Blog, 16 October 2026. https://pytorch.org/blog/bringing-elastically-deterministic-data-loading-to-torchtitan-with-zephon/
2. DatologyAI. "Zephon: Fast, Flexible, Elastically Deterministic Data Loading". DatologyAI Blog, 7 October 2026. https://www.datologyai.com/blog/zephon
3. Maximilian Böther, Josh Wills, Ties Robroek, et al. "Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines". arXiv:2610.03087, 2 October 2026. https://arxiv.org/abs/2610.03087
4. datologyai/zephon. GitHub repository and README. https://github.com/datologyai/zephon
5. Zephon project site. https://zephon.io/
6. zephon. Python Package Index. https://pypi.org/project/zephon/
7. "Zephon + TorchTitan". datologyai/torchtitan-zephon, examples/zephon/README.md. https://github.com/datologyai/torchtitan-zephon/blob/main/examples/zephon/README.md
8. PyTorch (@PyTorch). Post on X, 16 October 2026. https://x.com/PyTorch/status/2111136074064118226