Common Corpus
Common Corpus is the largest fully open, multilingual dataset for pretraining large language models, assembled and released by the French AI research lab Pleias.
Natural Language Processing
Explore Natural Language Processing through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Open Source AI
Articles that also belong to these categories. Counts cover all of Natural Language Processing.
All articles (239)Machine Learning (92)Large Language Models (58)AI Benchmarks (42)Deep Learning (37)Information Retrieval (29)AI Models (24)Artificial Intelligence (24)Open Source AI (23)Data & Datasets (15)AI Companies (9)Model Evaluation (9)Transformer Models (9)Meta AI (8)Speech & Audio AI (8)OpenAI (7)Training & Optimization (7)AI Ethics (6)AI Safety (6)AI Tools & Products (6)Computer Vision (6)Generative AI (6)People (6)AI Agents (5)Prompt Engineering (5)Developer Tools (4)Neural Networks (4)AI Code Generation (3)Algorithms (3)Anthropic (3)Mathematics (3)Reasoning Models (3)Statistics (3)AI Alignment (2)AI History (2)AI Research (2)Chinese AI (2)Diffusion Models (2)Google DeepMind (2)Healthcare AI (2)Model Architecture (2)Multimodal AI (2)AI Inference (1)Computer Science (1)Education AI (1)Enterprise AI (1)Legal AI (1)Microsoft (1)Mixture of Experts (1)NVIDIA (1)Research Organizations (1)Voice AI (1)
Showing 1-23 of 23 articles
Common Pile
Common Pile v0.1 is an 8 terabyte corpus of openly licensed and public domain text, released on June 5, 2025, by EleutherAI and a consortium of more than two dozen academic and industry collaborators.
DCLM (DataComp for Language Models)
DCLM, short for DataComp for Language Models (also styled DataComp-LM), is an open benchmark, dataset, and software framework, released in June 2024
Falcon (language model)
Falcon is a family of open-source large language models built by the Technology Innovation Institute (TII)
FineWeb
FineWeb is a large-scale, open pretraining dataset for large language models (LLMs) created by Hugging Face.
Haystack (framework)
Haystack is an open-source AI orchestration framework developed by deepset, a Berlin-based company, for building production-ready natural language processing (NLP), retrieval-augmented generation (RAG), and AI…
Jina Embeddings v3
Jina Embeddings v3 is a multilingual text embedding model released by Jina AI on September 18, 2024, with 570 million parameters, support for 89 languages, an 8,192 token context window, and a stack of…
LLaMA
LLaMA is the original name of a family of large language models developed by Meta AI.
Llama 3
Llama 3 is a family of open-weight large language models developed by Meta. Meta released the original Llama 3 checkpoints on April 18, 2024, in 8-billion-parameter and 70-billion-parameter sizes.
LlamaIndex
LlamaIndex is an open-source data framework for building large language model (LLM) applications, with a particular focus on retrieval-augmented generation (RAG) and document processing.
MTEB (Massive Text Embedding Benchmark)
MTEB, short for Massive Text Embedding Benchmark, is the standard public leaderboard for evaluating text embedding models across many task types at once.
No Language Left Behind (NLLB)
No Language Left Behind (NLLB) is a machine translation research project and model family from Meta AI, announced in July 2022.
Pleias
Pleias (stylized PleIAs) is a Paris based artificial intelligence laboratory and small company that designs, pretrains, and releases large language models trained exclusively on public domain and permissively…
Qwen
Qwen is a family of large language models and multimodal models developed by the Qwen Team at Alibaba Cloud.
RedPajama
RedPajama is a family of large-scale, openly licensed datasets for training large language models (LLMs), created by Together AI with academic and open-source partners to reproduce, in fully open form
Sentence-transformers/all-MiniLM-L6-v2 model
Sentence-transformers/all-MiniLM-L6-v2 model
Sentence-transformers/all-mpnet-base-v2 model
sentence-transformers/all-mpnet-base-v2
SentencePiece
SentencePiece is an open-source, language-agnostic subword tokenization library and detokenizer developed at Google and introduced by Taku Kudo and John Richardson in their 2018 EMNLP system demonstration…
The Pile (dataset)
The Pile is an 825.18 GiB (approximately 886 GB) English text corpus designed for training large language models, assembled from 22 diverse, high-quality subsets spanning academic, professional, internet…
fastText
fastText is an open-source library for learning word embeddings and performing text classification, developed by Facebook AI Research (FAIR) and released to the public on August 18, 2016 .
mT5
mT5 (multilingual T5) is a transformer-based encoder-decoder language model released by Google Research in October 2020 that covers 101 languages in a single model, pre-trained on a Common Crawl corpus called…
olmOCR
olmOCR is an open toolkit and vision-language model from the Allen Institute for AI (Ai2) that turns PDFs and document images into clean, structured plain text and Markdown.
tiktoken
tiktoken is an open source byte pair encoding (BPE) tokenizer library released by OpenAI in December 2022 that converts text into the integer token sequences its language models read and write.