Common Corpus
Common Corpus is the largest fully open, multilingual dataset for pretraining large language models, assembled and released by the French AI research lab Pleias.
Data & Datasets
Explore Data & Datasets through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Open Source AI
Articles that also belong to these categories. Counts cover all of Data & Datasets.
All articles (116)Machine Learning (65)Natural Language Processing (15)Open Source AI (15)Computer Vision (13)AI Benchmarks (11)Large Language Models (11)Statistics (10)AI Ethics (6)Meta AI (6)Robotics (5)Embodied AI (4)AI Companies (3)Model Evaluation (3)NVIDIA (3)AI Research (2)Data Science (2)Deep Learning (2)MLOps (2)Multimodal AI (2)Training & Optimization (2)AI Code Generation (1)AI History (1)AI Infrastructure (1)AI Models (1)AI Policy & Regulation (1)AI Safety (1)AI Tools & Products (1)AI for Science (1)Algorithms (1)Artificial Intelligence (1)Chinese AI (1)Developer Tools (1)Enterprise AI (1)Generative AI (1)Google (1)Google DeepMind (1)Mathematics (1)Robotics Companies (1)
Showing 1-15 of 15 articles
Common Pile
Common Pile v0.1 is an 8 terabyte corpus of openly licensed and public domain text, released on June 5, 2025, by EleutherAI and a consortium of more than two dozen academic and industry collaborators.
Cosmopedia
Cosmopedia is an open synthetic pretraining dataset released by Hugging Face in February 2024, made up of textbooks, blog posts, stories, and WikiHow-style articles written entirely by a large language model.
DCLM (DataComp for Language Models)
DCLM, short for DataComp for Language Models (also styled DataComp-LM), is an open benchmark, dataset, and software framework, released in June 2024
Dolma
Dolma is an open three-trillion-token English pretraining corpus released by the Allen Institute for AI (AI2) to power its fully open OLMo language models and to let researchers study how training data shapes…
FineWeb
FineWeb is a large-scale, open pretraining dataset for large language models (LLMs) created by Hugging Face.
NuminaMath
NuminaMath is a family of openly licensed competition-mathematics resources developed by the non-profit Project Numina, spanning the largest public dataset of competition math problems and solutions, a set of…
OpenOrca
OpenOrca is a large open-source instruction-tuning dataset that augments the FLAN Collection with chain-of-thought responses generated by OpenAI's GPT-3.5 and GPT-4 APIs.
RedPajama
RedPajama is a family of large-scale, openly licensed datasets for training large language models (LLMs), created by Together AI with academic and open-source partners to reproduce, in fully open form
RefinedWeb
RefinedWeb is a large-scale English pretraining dataset for large language models, built from filtered and deduplicated Common Crawl web data alone and released in June 2023 by the Technology Innovation…
SlimPajama
SlimPajama is a 627-billion-token English-language pre-training corpus for large language models, produced by Cerebras Systems in collaboration with the Opentensor Foundation by extensively cleaning and…
Snorkel
Snorkel is an open-source software framework and methodology for programmatic data labeling that started at Stanford University in 2015 and is licensed under Apache 2.0 .
The Pile (dataset)
The Pile is an 825.18 GiB (approximately 886 GB) English text corpus designed for training large language models, assembled from 22 diverse, high-quality subsets spanning academic, professional, internet…
The Stack (BigCode dataset)
The Stack is a family of large, permissively-licensed source-code datasets built by the BigCode project, an open scientific collaboration jointly led by Hugging Face and ServiceNow Research, to train and…
TxT360
TxT360 is an open large-scale pretraining corpus for large language models, released in October 2024 by the LLM360 project, a collaboration led by Petuum and the Mohamed bin Zayed University of Artificial…