Dolma

Dolma is an open three-trillion-token English pretraining corpus released by the Allen Institute for AI (AI2) to power its fully open OLMo language models and to let researchers study how training data shapes large language models.^[1][2] First published on August 18, 2023 and described in a peer-reviewed paper that won the Best Resource Paper Award at ACL 2024 (Soldaini et al., arXiv:2402.00159), Dolma was distilled from roughly 200 terabytes of raw text down to an 11-terabyte curated dataset spanning web crawls, code, academic papers, books, social media, and encyclopedic sources.^[1][2][13] It ships not only as data but as an open source toolkit that lets any group rebuild or remix the corpus from scratch.^[1][4]

Dolma was built specifically to feed the OLMo family of fully open language models.^[5] AI2 released the data, the filtering and deduplication code, the training framework, intermediate checkpoints, and the model weights together, treating the corpus as one piece of a single reproducible artifact rather than a separate product.^[5] Among the first generation of open pretraining sets, Dolma stands out for the depth of its release: every step from raw HTML to final tokens has been documented, the toxicity and personal information filters are described in detail, and the per-document metadata is preserved so researchers can audit which sources actually fed the model.^[1] The Dolma team framed the project as a direct response to a structural problem, writing that the "lack of access to pretraining corpora alongside corresponding language models has been a major obstacle for the broader research community."^[2]

The corpus has since gone through several revisions (v1.5, v1.6, v1.7) and has been folded into the larger pretraining mixes used for OLMo 2 and later AI2 models.^[6] It has become a common reference point for studies of data quality, deduplication, and contamination, and it sits alongside The Pile, RefinedWeb, RedPajama, FineWeb, and DataComp-LM as one of the main openly available pretraining datasets used by the wider research community.

Why was Dolma created?

The project came out of an explicit complaint from AI2's OLMo team. By 2023, frontier language models were trained on multi-trillion token corpora that researchers outside the labs that built them could not see, study, or reproduce. Even nominally open models like LLaMA documented their data only at the level of broad source categories.^[1] Without access to the actual text, questions about memorization, contamination of evaluation sets, demographic bias in the corpus, or the effect of any specific filtering decision could not be answered from the outside.^[1]

The Dolma authors argued that this opacity slowed the field down. If a model produced a surprising behavior, no one could check whether it came from the architecture, the training recipe, or some quirk of the underlying text. If a deduplication trick or a quality classifier improved benchmark scores, no one could replicate the experiment without building a comparable corpus from scratch. AI2 framed Dolma as the data half of a fully open release: the corpus, the toolkit that produced it, the model trained on it, and the evaluation harness used to measure it would all live in public, under permissive terms, with enough documentation to let anyone repeat the work (the corpus itself was first released under a restricted license, see below).^[1] AI2 said it released Dolma so that researchers could "independently create better versions of this dataset, study the relationship between the data and any model trained on it, report any issues they observe when inspecting our data, and critique our curation practices."^[2]

The practical goal was a corpus large enough to train a competitive seven-billion-parameter model under modern compute budgets, broad enough in source coverage to mirror what closed labs were rumored to use, and clean enough that the resulting model would be worth studying rather than serving as a cautionary tale. On size, AI2 noted that Chinchilla scaling laws suggest a compute-optimal ratio of model size to training tokens, but that models such as LLaMA 2 showed room for improvement from more training tokens, so it set out to collect a dataset large enough to study the relationship between model and dataset size; the paper says it acquired enough Common Crawl shards for a target corpus of 2-3 trillion tokens.^[1][2]

What is Dolma made of?

Dolma draws on seven main source families. The web component is by far the largest, consistent with how most modern pretraining mixes are weighted, while books, code, papers, and reference text round out the corpus.^[1]

Exact totals shift slightly between releases. The original v1 announcement put the corpus at three trillion tokens; v1.6 holds about 3.06 trillion Llama tokens across roughly 4.4 billion documents, and v1.7, which adds new sources and applies more quality filtering and fuzzy deduplication, totals about 2.3 trillion tokens.^[2][3] The current default version, v1.7, breaks down to roughly 1,195.5 billion tokens from Dolma's own Common Crawl pipeline, 456.4 billion from RefinedWeb, 263.8 billion from StarCoder, 138.4 billion from C4, and 79.9 billion from Reddit, with the remainder drawn from arXiv, StackExchange, and other curated sources.^[3] The published paper covers v1.5 and v1.6 in detail; v1.7 was released alongside OLMo 1.7 with notes in the project's GitHub repository rather than a new paper.^[1][4]

A few choices shape the character of the corpus. The web tier is built from 25 Common Crawl snapshots (May 2020 to June 2023) in Common Crawl's WET plain-text format, processed through CCNet for language identification and initial paragraph deduplication.^[1] The code tier is restricted to permissively licensed repositories from The Stack rather than the full GitHub crawl, which keeps the legal status clearer.^[1] The Reddit tier treats submissions and comments as independent documents, drops comments shorter than 500 characters, submissions shorter than 400 characters and comments with fewer than 3 votes, and excludes 26,123 banned or NSFW subreddits plus any subreddit with over 10% of posts tagged NSFW.^[1] Books are limited to Project Gutenberg, which sidesteps the copyright disputes around the Books3 corpus that earlier projects relied on.^[1]

Language coverage is English-only in v1 through v1.7. AI2 has discussed multilingual extensions in later work, and the Dolma toolkit is language-agnostic, but the released corpus targets English alone.^[1]

How is Dolma filtered and deduplicated?

The pipeline that produced Dolma is published as the open source Dolma Toolkit.^[4] It is built around a series of independent filters that read and write the same JSONL format, which makes it easy to add or remove a step and rerun the corpus.^[4] The paper describes data ablations in which 1.2B-parameter models are trained on data with and without a given curation decision and compared on downstream tasks.^[1]

The stages, in roughly the order they run on the web tier, are:

1. Language identification. A fastText classifier tags each document with a language code; documents predicted to be English with confidence above a threshold are kept, and the rest are dropped.^[1]
2. Quality heuristics. Dolma keeps all of the Gopher heuristic rules (covering signals such as symbol density, short or bulleted lines, and repeated n-grams) plus a single C4 rule that removes paragraphs not ending in punctuation; the paper's ablations found this combination outperformed either full rule set alone.^[1]
3. Toxicity filtering. Two fastText classifiers trained on the Jigsaw toxic comment dataset, one for hate speech and one for NSFW content, score Common Crawl text sentence by sentence, and sentences above a threshold are removed. The paper compares a high and a low threshold and finds that the lower threshold, which removes more content, gives better downstream performance.^[1]
4. Personal information removal. Email addresses, phone numbers, and IP addresses are detected with regular expressions; documents with five or fewer such spans have them replaced with special tokens, and documents with more are removed. The paper says model-based PII detectors were impractical at Dolma's size and that its regexes "sacrifice some accuracy for significant speed-up."^[1]
5. Deduplication. The web tier goes through three exact-match stages run with a Bloom filter: URL deduplication (removing 53.2% of documents), document deduplication, and finally paragraph deduplication to strip boilerplate such as repeated bylines.^[1]

Non-web tiers go through analogous but lighter pipelines. peS2o is used as-is, code has data-heavy files such as JSON and CSV removed and goes through RedPajama v1 and StarCoder heuristics plus a secrets detector, and the Reddit tier reuses the web content filters but removes, rather than masks, documents containing PII.^[1]

Two design decisions deserve a note. First, the team chose not to apply a learned quality classifier on top of the heuristic filters in the released corpus, which differentiates Dolma from later projects like FineWeb-Edu and DCLM that lean heavily on classifier-based selection. The Dolma paper presents this as a deliberate choice, citing arguments in the Gopher and RefinedWeb papers against model-based quality filters.^[1] Second, the team logged not only the documents that survived but also the rules that dropped each one, so anyone using the toolkit can recover counts of how often each filter fired.^[1]

What versions of Dolma exist?

Dolma has been published as a sequence of incremental releases, each tied to an OLMo training run.^[1] Researchers using Dolma should be careful to specify which version they mean; results from one version do not always carry over to the next.

The ACL 2024 paper covers v1.5 and v1.6.^[1] v1.7, the OLMo 2 mixes and Dolma 3 are documented in the OLMo technical reports, dataset cards and the AI2 GitHub repositories rather than as new corpus papers.^[3][6][14] AI2 has been careful to keep older versions available so that prior OLMo checkpoints can still be reproduced from scratch.^[4]

Is Dolma open source, and how do you access it?

Dolma was originally released in August 2023 under AI2's tiered ImpACT license, which released the corpus as a medium-risk artifact: users had to provide contact information and state their intended use, disclose any derivatives, distribute derivatives under the same restrictions, and agree not to use Dolma for prohibited uses such as military surveillance or generating disinformation.^[2] On April 15, 2024, alongside the v1.7 release, AI2 relicensed the corpus to the fully permissive Open Data Commons Attribution License (ODC-BY), the same license used by C4, RefinedWeb and FineWeb.^[3] The dataset card states that "we have updated the license of Dolma to ODC-BY," which only requires attribution and lifts the use-restriction tiers that ImpACT imposed.^[3] Users still remain subject to the underlying licenses of the source datasets that Dolma was built from.^[3]

The data is hosted on the Hugging Face Hub under the allenai/dolma repository and mirrored on AI2's own infrastructure.^[3] Files are stored as gzipped JSONL with one document per line; per-document metadata records the source, the original URL or document identifier, and which filters were applied.^[3] The default v1.7 release is about 4.5 terabytes gzipped (v1.6 is 5.4 terabytes), while the curated dataset as a whole was distilled from about 200 terabytes of raw text down to roughly 11 terabytes, which makes streaming access through Hugging Face's datasets library the standard way to consume it for training.^[3][13] The Dolma Toolkit repository on GitHub (allenai/dolma) ships with a command-line interface, configuration files reproducing every released version, and example pipelines for tagging custom corpora.^[4]

How is Dolma used in OLMo and other models?

Dolma's primary user is the OLMo family of fully open models.^[5] OLMo 1B and OLMo 7B, released in February 2024, were trained on Dolma for 2 trillion and 2.46 trillion tokens respectively.^[5] OLMo 7B Twin 2T came from training on two different clusters, AMD GPUs on LUMI and NVIDIA GPUs from MosaicML, to show the codebase ran on both; the runs performed nearly identically by 2 trillion tokens.^[5] OLMo 1.7 7B (April 2024) used Dolma v1.7 and scored 52 on MMLU, which AI2 attributed to a combination of improved data quality, a new two-stage training procedure, and architectural improvements.^[15]

The November 2024 OLMo 2 release introduced a new pretraining mix that combined Dolma with other curated sources, including Dclm-Baseline-1.0.^[6] OLMo-Mix-1124 is a collection of about 3.9 trillion tokens drawn from DCLM, Dolma, StarCoder and Proof Pile II, and the older Dolma versions remain available.^[3][6] OLMo 2 32B, released in March 2025, used a further-refined version of this mix.^[6] In November 2025 AI2 released OLMo 3, pretrained on Dolma 3, a new ~9.3-trillion-token corpus from which it built the 5.9-trillion-token Dolma 3 Mix; OLMo 3 includes 7B and 32B models.^[14]

Beyond AI2, Dolma has been used in academic studies of data attribution, memorization, and bias. Because every document in the corpus carries its source metadata, researchers can ask questions like "how often does a specific subreddit appear in the training data of this model" or "which Wikipedia articles are most likely to be memorized" without needing private access to the lab that trained the model.^[1] This kind of study is mostly impossible with closed corpora and was a central motivation for the project.^[1]

How does Dolma compare with other open pretraining datasets?

Dolma is one of several large open corpora that became available between 2020 and 2025. The table below gathers the headline numbers for the main alternatives. Token counts are approximate and depend on the tokenizer; reported figures use the original release notes for each project.

Dolma's distinguishing feature is the depth of its release rather than its raw size. FineWeb is larger, RedPajama v2 and DCLM are larger still, .^[8][9][12] What Dolma offers is the toolkit, the per-document filter logs, the documented ablations, and the alignment with the OLMo training stack.^[1] For a researcher who wants to study how a specific filter changes downstream behavior, Dolma is one of the few corpora where the experiment can actually be run end to end on public data.

The trade-off is that Dolma's web tier is heuristically filtered rather than classifier-filtered, so models trained on Dolma alone tend to score below models trained on FineWeb-Edu or DCLM-Baseline at matched token counts.^[8][12] AI2's response in the OLMo 2 release was to mix Dolma with DCLM-Baseline rather than abandon it; the resulting corpus is treated as the operational successor.^[6]

Influence on open-data practice

The Dolma release set a baseline for what an open pretraining corpus should ship with. Before Dolma, most public corpora released the data and a paper describing it; the actual processing code was either missing or scattered across personal repositories.^[1] Dolma packaged the toolkit, the configurations, the metadata, and the corpus as a single artifact, which made it possible for other projects to inherit the pipeline rather than rebuild it.^[4]

The project also pushed forward the conversation about licensing for pretraining data. AI2's initial choice of the restrictive ImpACT license, followed by its 2024 switch to fully permissive ODC-BY, mirrored a broader move across the field toward attribution-only licensing for large web-scraped corpora.^[3][7] FineWeb and RedPajama used permissive licenses from the start.^[8][9] The debate is still live, and there is no consensus yet on the right license tier for a corpus that contains personal information, copyrighted material, and toxic content scraped from the public web.

Dolma's most lasting contribution may be its insistence on documentation. The published paper includes filter ablation tables, deduplication statistics, source breakdowns by token count and document count, and a discussion of the limits of the toxicity and PII pipelines.^[1] This level of detail helped the paper win a Best Resource Paper Award at ACL 2024.^[13]

Limitations and open questions

The Dolma authors are explicit about the corpus's limitations.^[1] The corpus is English-only, so it offers little for non-English tasks.^[1] The data ablations behind the pipeline used 1-billion-parameter models only, so some design decisions may not carry over to larger models, and the evaluation suite covers a limited set of tasks.^[1] The PII regexes target only email addresses, phone numbers and IP addresses, and deduplication is exact-match rather than semantic, so paraphrased duplicates remain.^[1] The paper's own audits found little dialect bias in its toxicity classifier and found that its language-ID threshold kept all English documents in the International Corpus of English.^[1]

A broader open question is how much further heuristic filtering can be pushed before classifier-based selection takes over. The OLMo 2 mixes already lean on DCLM-Baseline, which uses a learned classifier, and the gap between heuristic and classifier-filtered web data has widened over time.^[6][12] AI2's next corpus, Dolma 3 (November 2025), was described as using "extensive deduplication, quality filtering, and careful control over data mixing" in building its 5.9-trillion-token pretraining mix.^[14]

A second question is multilingual coverage. The current corpus is English-only, and there is real demand for non-English open pretraining data at the multi-trillion token scale.^[1] The Dolma Toolkit is language-agnostic and could in principle be applied to other languages, but no large multilingual Dolma release has been announced.^[4]

Continue reading

• OLMoOLMo (Open Language Model) is a family of fully open large language models built by the Allen Institute for AI (Ai2) and first released on February 1, 2024.Mentioned 32 times above
• Common CrawlCommon Crawl is a nonprofit 501(c)(3) organization that maintains a free, open repository of web crawl data, and it is the single largest publicly available source of text used to train large language models.Mentioned 12 times above
• DCLM (DataComp for Language Models)DCLM, short for DataComp for Language Models (also styled DataComp-LM), is an open benchmark, dataset, and software framework, released in June 2024Mentioned 11 times above
• FineWebFineWeb is a large-scale, open pretraining dataset for large language models (LLMs) created by Hugging Face.Mentioned 9 times above
• RRedPajamaRedPajama is a family of large-scale, openly available datasets for training large language models (LLMs), created by Together AI with academic and open-source partners to reproduce, in fully open formMentioned 6 times above
• Allen Institute for AIThe Allen Institute for AI (branded since the early 2020s as Ai2, and historically abbreviated AI2) is a Seattle-based non-profit artificial intelligence research institute, founded in 2014 by the late…Mentioned 4 times above

More from See also

• TxT360
• OpenOrca
• The Pile (dataset)
• C4 (Colossal Clean Crawled Corpus)
• Pre-training
• Tokenization
• Training Set

Topics:Data & DatasetsLarge Language ModelsOpen Source AI

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A. H., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O., Walsh, P., Zettlemoyer, L., Smith, N. A., Hajishirzi, H., Groeneveld, D., Beltagy, I., and Lo, K. (2024). "Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), pages 15725-15788. arXiv:2402.00159.
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7Allen Institute for AI. (2023). "Ai2 Dolma: 3 trillion token open corpus for language model pretraining." Blog post, August 18, 2023. Archived: web.archive.org/...ns-open-llm-corpus-9a0ff4b8da64.
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11Hugging Face. (2024). Dataset card for `allenai/dolma`. huggingface.co/datasets/allenai/dolma.
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Allen Institute for AI. "Dolma Toolkit." GitHub repository, github.com/allenai/dolma.
5. ^1 ^2 ^3 ^4 ^5Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Pyatkin, V., Ravichander, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L., Dodge, J., Lo, K., Soldaini, L., Smith, N. A., and Hajishirzi, H. (2024). "OLMo: Accelerating the Science of Language Models." arXiv:2402.00838.
6. ^1 ^2 ^3 ^4 ^5 ^6 ^7OLMo 2 Team, Allen Institute for AI. (2024). "OLMo 2: The best fully open language model to date." Technical report and blog post, allenai.org.
7. ^Allen Institute for AI. "AI2 ImpACT License." allenai.org/licenses/impact.
8. ^1 ^2 ^3Penedo, G., Kydlicek, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., and Wolf, T. (2024). "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale." NeurIPS 2024 Datasets and Benchmarks Track.
9. ^1 ^2Together AI. (2023). "RedPajama-Data-v2: an Open Dataset with 30 Trillion Tokens for Training Large Language Models." together.ai blog.
10. Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. (2023). "The RefinedWeb Dataset for Falcon LLM." arXiv:2306.01116.
11. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. (2020). "The Pile: An 800GB Dataset of Diverse Text for Language Modeling." arXiv:2101.00027.
12. ^1 ^2 ^3Li, J., Fang, A., Smyrnis, G., et al. (2024). "DataComp-LM: In Search of the Next Generation of Training Sets for Language Models." arXiv:2406.11794.
13. ^1 ^2 ^3Association for Computational Linguistics. (2024). "Best Paper Awards: ACL 2024." 2024.aclweb.org/program/best_papers. Dolma named Best Resource Paper.
14. ^1 ^2 ^3Allen Institute for AI. (2025). "Olmo 3: Charting a path through the model flow to lead open-source AI." Blog post, November 20, 2025. allenai.org/...olmo3.
15. ^Allen Institute for AI. (2024). "OLMo 1.7-7B: A 24 point improvement on MMLU." Blog post, April 17, 2024. allenai.org/...nt-improvement-on-mmlu-92b43f7d269d.

Show all 15 references