Citation and evidence

FineWeb

23 min full readUpdated 17 references

This article's verification

Report a problem with this article

Edit article
More

Cite this article

FineWeb is a large-scale, open pretraining dataset for large language models (LLMs) created by Hugging Face. Released in April 2024, it originally contained approximately 15 trillion tokens extracted and cleaned from 96 Common Crawl snapshots spanning from the summer of 2013 to April 2024.[1][2] Hugging Face has since added newer snapshots: as of version 1.4.0 (July 2025), the dataset card lists more than 18.5 trillion tokens from 110 snapshots running through June 2025.[2] At roughly 44 terabytes of disk space at release, FineWeb was described by Hugging Face as the largest publicly available clean LLM pretraining dataset.[2][5] The dataset was introduced by Guilherme Penedo, Hynek Kydlicek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf in a paper titled "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale," submitted on 25 June 2024 and accepted to the NeurIPS 2024 Datasets and Benchmarks Track.[1] FineWeb is released under the Open Data Commons Attribution License (ODC-By) v1.0.[2]

Hugging Face logo
Hugging Face logo, from the official website huggingface.co. Trademark of its owner.

The paper frames FineWeb as a transparency response to closed industrial datasets, stating that "the pretraining datasets for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly available and very little is known about how they were created," and that FineWeb "produces better-performing LLMs than other open pretraining datasets."[1] Alongside the base dataset, the authors released FineWeb-Edu, a 1.3 trillion token subset filtered for educational content using a classifier trained on annotations generated by Llama 3-70B-Instruct.[1][3] The authors report that models pretrained on FineWeb-Edu outperformed those trained on the other openly accessible web datasets they compared on knowledge-intensive and reasoning benchmarks, including MMLU, ARC (AI2 Reasoning Challenge), and OpenBookQA.[1][3]

Both datasets are freely available on the Hugging Face Hub.[2][3]

Why was FineWeb created?

The quality of pretraining data has a direct effect on the downstream performance of large language models.[1] While Common Crawl provides petabytes of raw web data, the unprocessed crawl contains enormous amounts of boilerplate text, duplicate content, low-quality pages, and non-English material. Converting this raw data into a corpus suitable for LLM pretraining requires extensive filtering, cleaning, and deduplication.[1]

Before FineWeb, several open datasets attempted to address this problem. C4 (Colossal Clean Crawled Corpus), introduced by Raffel et al. (2020) for training the T5 model, applied heuristic filters to a single Common Crawl snapshot and produced roughly 172 billion tokens (as counted by the FineWeb authors).[5][7] RefinedWeb, released by the Technology Innovation Institute (TII) in 2023, used more sophisticated filtering and deduplication to produce a 600 billion token public English web corpus (the full internal version was estimated at 5 trillion tokens).[9] The Pile, assembled by EleutherAI, combined web data with curated sources like books and academic papers to reach approximately 340 billion tokens.[12] RedPajama v2, created by Together AI, reported 30 trillion filtered and deduplicated tokens (from more than 100 trillion raw tokens) across five languages; the deduplicated version the FineWeb authors compared against holds about 20 trillion tokens.[5][11] Dolma, released by the Allen Institute for AI (AI2) for the OLMo project, provided roughly 3 trillion tokens in its version 1.6 Common Crawl portion.[10]

Despite this progress, existing datasets had clear limitations. Some were too small for modern training runs, others used outdated or opaque filtering methods, and few documented their curation decisions in enough detail for the research community to reproduce or improve upon them. FineWeb was designed to address all of these shortcomings by building the largest open web corpus with a fully documented, empirically validated processing pipeline.[1]

FineWeb started as an effort to openly replicate the RefinedWeb dataset, which was used to train the Falcon family of models.[9] During development, the Hugging Face team found that by carefully adding extra filtering and deduplication steps, they could push model performance well above what RefinedWeb achieved.[5] The project grew into a full-scale empirical study of data curation practices, with over 70 ablation models trained to test different processing choices.[1][5] The motivation was twofold: to lower the barrier for open-source model training by providing a high-quality dataset that most groups could not afford to produce from scratch, and to document every design decision and release the full curation codebase so other researchers could reproduce and extend the work.[5]

Where does FineWeb get its data?

FineWeb draws its raw material from Common Crawl, a nonprofit organization that regularly crawls the web and makes its archives publicly available.[1] Common Crawl publishes data in three formats:

FormatFull nameContents
WARCWeb ARChiveFull HTTP response including HTML source code
WETWARC Encapsulated TextPre-extracted plain text from each page
WATWeb Archive TransformationMetadata about crawled pages

Expanded article table

Most prior datasets (including C4 and parts of RedPajama) relied on WET files for convenience, since the text has already been extracted. However, the FineWeb team discovered that WET files retain too much boilerplate content, including navigation menus, sidebar text, and cookie notices, which degrades downstream model performance.[1][5]

A core innovation of FineWeb was processing the raw WARC files directly and using the open-source trafilatura library for text extraction.[1][13] Trafilatura is specifically designed to identify and extract the main content of a web page while discarding boilerplate elements.[13] In ablation experiments, models trained on trafilatura-extracted text consistently outperformed those trained on the default WET text.[1][5]

The version described in the paper processes 96 Common Crawl snapshots covering web crawls from mid-2013 through April 2024 (CC-MAIN-2013-20 through CC-MAIN-2024-18), and later releases added snapshots through June 2025 (CC-MAIN-2025-26), giving the dataset broad temporal coverage.[1][2]

How is FineWeb processed?

The FineWeb processing pipeline was built using datatrove, an open-source data processing library developed by Hugging Face.[6] The pipeline applies a sequential series of steps, each empirically validated through ablation experiments. The overall approach was primarily empirical: the team performed "data ablation" experiments to test different methods at each stage, training models and measuring benchmark performance to determine which steps helped and which did not.[1][5]

Text extraction

Raw HTML from Common Crawl WARC files is processed using the trafilatura library.[13] Trafilatura identifies the main content region of each web page and extracts clean text while removing navigation bars, headers, footers, advertisements, and other boilerplate.[13] This approach produces notably cleaner text than the pre-extracted WET files provided by Common Crawl. Ablation experiments confirmed that trafilatura WARC extraction substantially outperformed default WET extraction in terms of downstream model performance.[1][5]

URL filtering

Documents originating from malicious or adult content websites are removed using a URL blocklist. This blocklist contains known domains hosting pornographic material, malware, and spam. The filtering uses both exact domain matching and pattern detection in URLs.[5]

Language identification

A fastText language classifier is applied to each document. Only English-language documents with a confidence score of 0.65 or higher are retained.[1][5] This threshold was chosen to balance recall (keeping documents that are primarily English) against precision (excluding non-English or mixed-language content).[5]

Quality and repetition filters (Gopher/MassiveText rules)

FineWeb applies the quality and repetition filters from the MassiveText dataset (Rae et al., 2022), originally developed for training DeepMind's Gopher model.[8] These filters target common patterns in low-quality web text:

Filter categoryDescription
Word countDocuments with too few or too many words are removed
Mean word lengthDocuments with unusually short or long average word lengths are filtered
Symbol-to-word ratioDocuments with excessive special characters are removed
Stop word presenceDocuments lacking common English stop words are discarded
Line-level repetitionDocuments where a high fraction of lines are duplicates are removed
Paragraph-level repetitionDocuments with excessive duplicate paragraphs are removed
N-gram repetitionDocuments with high proportions of repeated 2-grams, 3-grams, or 4-grams are filtered
Bullet point ratioDocuments dominated by bullet points or list markers are removed
Ellipsis ratioDocuments with excessive ellipsis usage are filtered

Expanded article table

MinHash deduplication

Deduplication is one of the most consequential steps in the pipeline. FineWeb uses MinHash-based locality-sensitive hashing to identify and remove near-duplicate documents efficiently at web scale.[1][5]

The specific parameters are:

ParameterValue
Shingle size5-grams (using English word tokenizer)
Number of hash functions112
Band configuration14 buckets of 8 hashes each
Similarity threshold75%
Match probability at 75% similarity~77%

Expanded article table

Each 5-gram in a document is hashed with each of the 112 hash functions, and a document signature is obtained by taking the minimum hash value across all 5-grams for each hash function.[5] Two documents sharing at least 8 identical MinHash values in any single band are flagged as near-duplicates, and one copy is removed.[5]

A notable finding from the FineWeb research is that individual per-snapshot deduplication outperformed global deduplication across all snapshots.[1][5] Global deduplication removed roughly 90% of content from older snapshots, which turned out to hurt model performance.[5] The authors hypothesized that deduplication of clusters with a small number of duplicates can actually harm data quality, since these documents are often legitimately similar (for example, different editions of reference pages) rather than true spam duplicates.[1][5] Per-snapshot deduplication retained approximately 20 trillion tokens before additional filtering brought the total to 15 trillion.[5]

C4-style filters

After deduplication, heuristic filters originally developed for the C4 dataset are applied.[7] These include checks for curly brackets (common in code or template artifacts), lorem ipsum placeholder text, JavaScript mentions (often indicating error pages), and policy/cookie statement boilerplate.[7] However, the terminal punctuation filter from C4 (which requires sentences to end with standard punctuation marks) was deliberately excluded because ablation experiments showed it removed approximately 30% of all tokens without a corresponding improvement in model quality.[5]

Custom heuristic filters

The Hugging Face team evaluated over fifty candidate filtering heuristics from prior work and selected three custom filters based on their impact during ablation studies:[5]

FilterThresholdTokens removed
Fraction of lines ending with punctuationRemove if <= 0.1210.14%
Fraction of characters in duplicated linesRemove if >= 0.112.47%
Fraction of lines shorter than 30 charactersRemove if >= 0.673.73%

Expanded article table

These filters target list-heavy documents, documents with repeated boilerplate lines, and documents with corrupted line formatting. Together, the additional heuristic filters removed roughly 22% of tokens while improving aggregate benchmark scores.[5]

PII removal

As a final step, basic personally identifiable information (PII) removal is applied. Email addresses are replaced with anonymized placeholders and public IP addresses are masked.[2][5] The authors acknowledge this is not a comprehensive PII scrubbing solution.[2]

What is FineWeb-Edu?

FineWeb-Edu is a 1.3 trillion token subset of FineWeb that retains only web pages with high educational value.[3] The motivation behind FineWeb-Edu was the observation that filtering for educational content can dramatically improve model performance on knowledge-intensive and reasoning benchmarks, even when the resulting dataset is much smaller than the original.[1][3] The paper reports that "LLMs pretrained on FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC."[1]

Annotation process

To generate educational quality annotations, Llama 3-70B-Instruct was prompted to rate approximately 460,000 randomly sampled FineWeb documents on a scale from 0 to 5 based on their educational quality:[1]

ScoreMeaning
0Not educational at all
1Minimally educational
2Somewhat educational
3Moderately educational, suitable for learning
4Highly educational with clear explanations
5Outstanding educational quality, textbook-level

Expanded article table

The annotation prompt was designed to focus on grade-school and middle-school level knowledge, intentionally avoiding bias toward highly technical content like arXiv papers or academic submissions.[3][5] Of the total annotations, 410,000 were used for training and 50,000 were held out for validation, according to the paper; the public classifier card describes 450,000 training annotations and a hold-out set of 46,867 samples.[1][4]

Classifier architecture and training

Rather than using the LLM directly to score all 15 trillion tokens (which would be prohibitively expensive), the team trained a lightweight classifier to replicate the LLM's annotations at scale:[3][4]

ComponentDetail
Base embedding modelSnowflake Arctic Embed M (snowflake-arctic-embed-m), 109M parameters
Classification headSingle linear regression layer (score 0-5)
Training samples410,000 (with 50,000 held out for validation)
Training procedureEmbedding layers frozen; only regression head trained
Epochs20
Learning rate3e-4
Inference compute~6,000 H100 GPU hours for full FineWeb

Expanded article table

When evaluated as a binary classifier (keeping documents with score >= 3 versus discarding them), the model achieved an F1 score of 82% on the held-out validation set.[4][5]

Filtering threshold and results

Documents scoring below 3 were removed. This threshold eliminated roughly 92% of the original FineWeb data, leaving 1.3 trillion tokens of educational content.[3][5] The resulting FineWeb-Edu dataset is notable for its efficiency: models trained on FineWeb-Edu can match the MMLU performance of models trained on roughly 10 times as many tokens from other datasets like C4 or Dolma.[1][5]

Each document in FineWeb-Edu includes both the continuous score and a rounded integer score (int_score), allowing users to apply their own thresholds.[3] A variant called FineWeb-Edu Score 2 retains documents with scores of 2 or higher, providing a larger but less selective alternative.[3]

The trained classifier is publicly released as HuggingFaceFW/fineweb-edu-classifier on the Hugging Face Hub, enabling researchers to apply educational quality scoring to their own datasets.[4]

Bias analysis

Analysis of FineWeb-Edu showed that educational filtering reduced some categories of bias found in the broader FineWeb dataset.[5] While FineWeb contained problematic associations between religious terms and online dating terminology (based on TF-IDF analysis), FineWeb-Edu exhibited more educationally appropriate associations, such as "woman" co-occurring with "pregnancy" and "man" with "king."[1] However, common web data biases persist: the word "man" appears significantly more often than "woman" or "non-binary," and "Christian" is the dominant religious term by frequency.[1]

How is FineWeb evaluated?

All evaluation during FineWeb's development used a consistent setup to ensure fair comparisons across datasets.[1][5]

Model architecture

ParameterValue
ArchitectureLlama-style transformer
Total parameters1.82 billion (including embeddings)
Sequence length2,048 tokens
TokenizerGPT-2 tokenizer
Global batch size~2 million tokens
Short ablation runs~28 billion tokens
Long confirmation runs350 billion tokens
Evaluation frameworklighteval (Hugging Face)

Expanded article table

This model size was chosen as a practical compromise: large enough to show meaningful differences between training datasets, but small enough to train many variants within a reasonable compute budget.[5]

Benchmark suite

Eight benchmarks were used for evaluation:[1][5]

BenchmarkCategory
MMLUMultitask knowledge (57 subjects, truncated to 1,000 samples)
ARC (AI2 Reasoning Challenge)Grade-school and high-school science questions
HellaSwagCommonsense sentence completion
OpenBookQAElementary science with open-book reasoning
PIQAPhysical intuition and reasoning
SIQA (Social IQa)Social interaction reasoning
WinoGrandePronoun resolution and commonsense
CommonsenseQAGeneral commonsense reasoning

Expanded article table

All evaluations were run using the lighteval framework.[5]

Total ablation compute

Over 70 models were trained during the development process.[1][5] Filtering ablations used approximately 28 billion tokens of training data, while deduplication and final confirmation ablations used 350 billion tokens.[5] The total compute for all ablation experiments was estimated at 80,000 H100 GPU hours.[1]

How does FineWeb compare with other datasets?

FineWeb was benchmarked against several widely used open pretraining datasets. For fair comparison, a 1.82 billion parameter model was trained on 350 billion tokens sampled from each dataset (or the full dataset when its size was smaller than 350 billion tokens).[1][5]

Dataset overview

DatasetOrganizationTokens (approx.)SourceYearLicense
FineWebHugging Face15T96 Common Crawl snapshots2024ODC-By 1.0
FineWeb-EduHugging Face1.3TEducational subset of FineWeb2024ODC-By 1.0
C4Google172BSingle Common Crawl snapshot2019ODC-By 1.0
RefinedWeb (public)TII600BCommon Crawl2023ODC-By 1.0
Dolma v1.6 (CC portion)AI2~3TCommon Crawl portion2024ODC-By 1.0 (relicensed April 2024; originally AI2 ImpACT)
Dolma v1.7AI2~2.3T (CC portion ~1.2T)Mixed (improved filtering)2024ODC-By 1.0
The PileEleutherAI340B22 diverse sources2020Mixed (per-component terms)
SlimPajamaCerebras627BDeduplicated RedPajama2023Apache 2.0
RedPajama-V2 (dedup)Together AI20TCommon Crawl + other2023Apache 2.0

Expanded article table

Benchmark results

At the 350 billion token training scale, models trained on FineWeb outperformed models trained on C4, Dolma v1.6, The Pile, SlimPajama, and RedPajama-V2 on the aggregate benchmark score across all eight tasks.[1][5] FineWeb's aggregate score was competitive with or exceeded RefinedWeb on most individual benchmarks.[1][5]

FineWeb-Edu showed even stronger results, particularly on knowledge-intensive and reasoning benchmarks:[1][3]

BenchmarkFineWeb (350B tokens)FineWeb-Edu (350B tokens)Relative improvement
MMLU~33%~37%~12% relative
ARC~46%~57%~24% relative
Aggregate (all 8 tasks)BaselineHigherBest among the open datasets compared

Expanded article table

A separate 2025 study, FinerWeb-10BT, applied LLM-based line-level filtering to a 10-billion-token FineWeb sample and reported that small GPT-2-style models trained on its cleaned versions reached the original data's peak HellaSwag score about 32% sooner, even with up to 25% less data.[16]

The most striking finding concerns data efficiency. FineWeb-Edu can match the final MMLU performance of models trained on roughly 10 times as many tokens from C4 or Dolma.[1][5] This suggests that filtering for educational quality is one of the highest-leverage interventions available for improving pretraining data.[1]

A 2024 study by Youssef Mansour and Reinhard Heckel found that a small transformer classifier could tell FineWeb text apart from C4 text with about 87% accuracy in binary classification, and from other web datasets with accuracies ranging from about 75% (RefinedWeb) to 99% (DCLM), indicating that filtered web datasets carry distinctive fingerprints.[17]

Technical infrastructure

The datatrove library

The entire FineWeb pipeline was implemented using datatrove, Hugging Face's open-source Python library for large-scale data processing.[6] Datatrove provides modular, composable pipeline blocks:[6]

Block typeFunction
ReadersLoad data from various formats (WARC, Parquet, JSONL)
ExtractorsExtract text from raw formats (HTML via trafilatura)
FiltersRemove documents based on configurable criteria
WritersSave processed data to disk or cloud storage
DeduplicatorsPerform MinHash or exact deduplication

Expanded article table

Datatrove is designed to scale across computing clusters using frameworks like Slurm, making it practical to process the hundreds of terabytes of raw Common Crawl data.[6] The FineWeb pipeline code is publicly available in the datatrove repository as a reference implementation.[6]

How much did it cost to build FineWeb?

Processing 96 Common Crawl snapshots through the full pipeline required substantial resources. The final dataset occupies about 44 TB of disk space.[5] The educational quality classification step alone required approximately 6,000 H100 GPU hours.[5] All ablation model training added an estimated 80,000 H100 GPU hours.[1]

Data schema

Each document in FineWeb is stored as a Parquet record with the following fields:[2]

FieldTypeDescription
textstringExtracted and cleaned web page content
idstringUnique document identifier
dumpstringCommon Crawl snapshot identifier (e.g., CC-MAIN-2024-10)
urlstringSource URL of the web page
datestringDate the page was crawled
file_pathstringInternal file path within the dataset
languagestringDetected language code
language_scorefloat64fastText language identification confidence score
token_countint64Number of tokens (GPT-2 tokenizer)

Expanded article table

FineWeb-Edu includes two additional fields:[3]

FieldTypeDescription
scorefloat64Continuous educational quality score (0.0 to 5.0)
int_scoreint64Rounded integer score (0 to 5)

Expanded article table

What has FineWeb been used for?

FineWeb and FineWeb-Edu have been widely adopted since their release, with downstream effects across multiple areas of the open-source AI ecosystem.

Lowering the barrier for LLM training

Before FineWeb, researchers training open LLMs often had to choose between smaller curated datasets (like C4 or RefinedWeb) or larger but noisier options (like raw RedPajama v2). FineWeb offered both scale and quality in a single package, reducing the need for each research group to build its own data pipeline from scratch. The release of all processing code, the educational classifier, the LLM annotations, and all 70+ ablation model checkpoints enabled full reproducibility.[1][5]

Downstream datasets and models

FineWeb-Edu has served as a foundation for multilingual training initiatives. The TransWeb-Edu project produced balanced document-level corpora from FineWeb-Edu, and the resulting CuatroLLM and TransWebLLM models demonstrated that a high-quality English educational seed corpus can match or exceed the performance of models like Llama 3.2, Gemma, and Qwen 2.5 with far less training data.

The dataset's curation methodology influenced several other projects. Stack-Edu adapted the educational filtering approach for code data. FineWeb-Edu-Chinese applied similar techniques to Chinese web text. The Zyda-2 dataset incorporated FineWeb-Edu as a component, and Ultra-FineWeb explored additional quality verification methods on top of FineWeb's pipeline. Domain-specific subsets have also been created; OnlySports, for instance, extracted approximately 600 billion tokens of sports-related content from FineWeb.

Methodological influence

The idea that filtering for educational content could so dramatically improve benchmark performance (while discarding over 90% of the data) challenged the common assumption that more data is always better for LLM training.[1][5] This shifted attention in the field toward data quality over data quantity. The synthetic annotation approach (using an LLM to generate labels, then distilling into a lightweight classifier for efficient inference) has been widely adopted for other data filtering and scoring tasks.[3]

What is FineWeb-2?

In December 2024, Hugging Face released FineWeb-2, a multilingual successor to FineWeb, later accompanied by a paper, "FineWeb2: One Pipeline to Scale Them All, Adapting Pre-Training Data Processing to Every Language," posted to arXiv in June 2025.[14][15] FineWeb-2 extends the curation pipeline to over 1,000 languages using nearly 100 Common Crawl snapshots.[14][15] The resulting dataset contains approximately 20 TB of text across roughly 5 billion documents, covering 1,868 language-script pairs (1,226 of which have more than 100 documents, 474 more than 1,000 documents, and 203 at least 10,000 documents).[14][15] FineWeb-2 was built from the portion of Common Crawl that did not pass FineWeb's English language identification threshold.[14]

FineWeb-2 adapts the pipeline by tuning individual thresholds and stop-word lists for each language.[14][15] In benchmark evaluations, models trained on FineWeb-2 outperformed those trained on prior multilingual datasets (CC-100, mC4, CulturaX, HPLT, HPLT2, and raw Common Crawl) on 11 out of 14 tested languages.[14][15]

Limitations

The authors acknowledge several limitations of FineWeb:

  • English only: The base FineWeb dataset contains only English text. Non-English content was later addressed by FineWeb-2.[2][14]
  • Web-only source: FineWeb does not include books, academic papers, code repositories, or other curated text sources. It is intended to be combined with other data sources in a training mixture.[2]
  • PII concerns: While basic PII removal is applied, the dataset may still contain personal information found on the open web.[2]
  • Bias and toxicity: As a web-scraped dataset, FineWeb inherits the biases and potentially harmful content present on the internet. Gender bias analysis found that male-associated terms appear far more frequently than female or non-binary terms.[5] The URL blocklist removes some problematic domains, but no content-level toxicity filtering is applied.[2]
  • Snapshot-level deduplication only: Deduplicating within each snapshot rather than globally means some near-duplicate content appears across different snapshots. This was a deliberate trade-off to preserve older high-quality content.[1][5]

Continue reading

More from See also

More to read

Topics:Data & DatasetsMachine LearningNatural Language ProcessingOpen Source AI

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41Penedo, G., Kydlicek, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., & Wolf, T. (2024). "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale." *Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track.* arXiv:2406.17557. arxiv.org/...2406.17557
  2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13Hugging Face. (2024). "FineWeb Dataset Card." Hugging Face Hub. huggingface.co/...fineweb
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13Hugging Face. (2024). "FineWeb-Edu Dataset Card." Hugging Face Hub. huggingface.co/...fineweb-edu
  4. ^1 ^2 ^3 ^4Hugging Face. (2024). "FineWeb-Edu Classifier." Hugging Face Hub. huggingface.co/...fineweb-edu-classifier
  5. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45Hugging Face. (2024). "FineWeb: Decanting the Web for the Finest Text Data at Scale." Blog post. huggingface.co/...blogpost-fineweb-v1
  6. ^1 ^2 ^3 ^4 ^5Hugging Face. (2024). "datatrove: Freeing data processing from scripting madness." GitHub. github.com/...datatrove
  7. ^1 ^2 ^3Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." *Journal of Machine Learning Research*, 21(140), 1-67.
  8. ^Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., et al. (2022). "Scaling Language Models: Methods, Analysis & Insights from Training Gopher." arXiv:2112.11446.
  9. ^1 ^2Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Launay, J., & Srinivasan, B. V. (2023). "The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only." *NeurIPS 2023.* arXiv:2306.01116.
  10. ^Soldaini, L., et al. (2024). "Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research." arXiv:2402.00159.
  11. ^Together AI. (2023). "RedPajama-Data-v2: An Open Dataset with 30 Trillion Tokens for Training Large Language Models." together.ai/...redpajama-data-v2
  12. ^Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., et al. (2020). "The Pile: An 800GB Dataset of Diverse Text for Language Modeling." arXiv:2101.00027.
  13. ^1 ^2 ^3 ^4Barbaresi, A. (2021). "Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction." *Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics.*
  14. ^1 ^2 ^3 ^4 ^5 ^6 ^7Hugging Face. (2025). "FineWeb2: One Pipeline to Scale Them All." Hugging Face Hub. huggingface.co/...fineweb-2
  15. ^1 ^2 ^3 ^4 ^5Penedo, G., Kydlicek, H., Sabolcec, V., Messmer, B., Foroutan, N., Kargaran, A. H., Raffel, C., Jaggi, M., Von Werra, L., & Wolf, T. (2025). "FineWeb2: One Pipeline to Scale Them All, Adapting Pre-Training Data Processing to Every Language." arXiv:2506.20920. arxiv.org/...2506.20920
  16. ^Henriksson, E., Tarkka, O., & Ginter, F. (2025). "FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering." arXiv:2501.07314. arxiv.org/...2501.07314
  17. ^Mansour, Y., & Heckel, R. (2024). "Measuring Fingerprints of Web-filtered Text Datasets and Fingerprint Propagation Through Training." arXiv:2412.02857. arxiv.org/...2412.02857

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

11 revisions · v11 · 4,544 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check of the full article against its cited sources (120 claims, 16 sources fetched); 9 material + 21 minor fixed

Cite this page: AI Wiki. "FineWeb." aiwiki.ai, updated 1 Nov 2026, fact-checked 2 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/fineweb

Edit article