Common Corpus
Common Corpus is the largest fully open, multilingual dataset for pretraining large language models, assembled and released by the French AI research lab Pleias.
Open Source AI
Explore Open Source AI through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Natural Language Processing
Articles that also belong to these categories. Counts cover all of Open Source AI.
All articles (444)Large Language Models (165)AI Models (120)Developer Tools (109)Chinese AI (77)AI Agents (53)AI Companies (42)Mixture of Experts (40)AI Infrastructure (37)Multimodal AI (28)NVIDIA (28)Robotics (28)Machine Learning (25)AI Code Generation (23)Natural Language Processing (23)AI Tools & Products (18)Generative AI (18)Meta AI (17)Small Language Models (16)Data & Datasets (15)Video Generation (13)Computer Vision (12)Diffusion Models (12)Image Generation (12)Information Retrieval (12)Google (11)Reasoning Models (11)AI Research (10)Artificial Intelligence (10)AI Hardware (9)AI Inference (9)Microsoft (9)Research Organizations (9)Speech & Audio AI (9)Embodied AI (8)Google DeepMind (8)MLOps (7)Training & Optimization (7)AI for Science (6)Reinforcement Learning (6)Robot Hardware (6)Deep Learning (5)Drug Discovery (5)OpenAI (5)Voice AI (5)AI Benchmarks (4)AI Safety (4)Healthcare AI (4)Model Evaluation (4)Algorithms (3)Conversational AI (3)AI Ethics (2)Agentic Commerce (2)Anthropic (2)Enterprise AI (2)Humanoid Robots (2)Interpretability (2)Mathematics (2)Model Architecture (2)Neural Networks (2)People (2)Physical AI (2)Programming Languages (2)Software Development (2)Transformer Models (2)AI Incidents & Controversies (1)AI Policy (1)AI Policy & Regulation (1)Data Science (1)Music & Audio Generation (1)Quantum Computing (1)Robotics Companies (1)World Models (1)
Showing 1-23 of 23 articles
Common Pile
Common Pile v0.1 is an 8 terabyte corpus of openly licensed and public domain text, released on June 5, 2025, by EleutherAI and a consortium of more than two dozen academic and industry collaborators.
DCLM (DataComp for Language Models)
DCLM, short for DataComp for Language Models (also styled DataComp-LM), is an open benchmark, dataset, and software framework, released in June 2024
Falcon (language model)
Falcon is a family of open-source large language models built by the Technology Innovation Institute (TII)
FineWeb
FineWeb is a large-scale, open pretraining dataset for large language models (LLMs) created by Hugging Face.
Haystack (framework)
Haystack is an open-source AI orchestration framework developed by deepset, a Berlin-based company, for building production-ready natural language processing (NLP), retrieval-augmented generation (RAG), and AI…
Jina Embeddings v3
Jina Embeddings v3 is a multilingual text embedding model released by Jina AI on September 18, 2024, with 570 million parameters, support for 89 languages, an 8,192 token context window, and a stack of…
LLaMA
LLaMA is the original name of a family of large language models developed by Meta AI.
Llama 3
Llama 3 is a family of open-weight large language models developed by Meta. Meta released the original Llama 3 checkpoints on April 18, 2024, in 8-billion-parameter and 70-billion-parameter sizes.
LlamaIndex
LlamaIndex is an open-source data framework for building large language model (LLM) applications, with a particular focus on retrieval-augmented generation (RAG) and document processing.
MTEB (Massive Text Embedding Benchmark)
MTEB, short for Massive Text Embedding Benchmark, is the standard public leaderboard for evaluating text embedding models across many task types at once.
No Language Left Behind (NLLB)
No Language Left Behind (NLLB) is a machine translation research project and model family from Meta AI, announced in July 2022.
Pleias
Pleias (stylized PleIAs) is a Paris based artificial intelligence laboratory and small company that designs, pretrains, and releases large language models trained exclusively on public domain and permissively…
Qwen
Qwen is a family of large language models and multimodal models developed by the Qwen Team at Alibaba Cloud.
RedPajama
RedPajama is a family of large-scale, openly licensed datasets for training large language models (LLMs), created by Together AI with academic and open-source partners to reproduce, in fully open form
Sentence-transformers/all-MiniLM-L6-v2 model
Sentence-transformers/all-MiniLM-L6-v2 model
Sentence-transformers/all-mpnet-base-v2 model
sentence-transformers/all-mpnet-base-v2
SentencePiece
SentencePiece is an open-source, language-agnostic subword tokenization library and detokenizer developed at Google and introduced by Taku Kudo and John Richardson in their 2018 EMNLP system demonstration…
The Pile (dataset)
The Pile is an 825.18 GiB (approximately 886 GB) English text corpus designed for training large language models, assembled from 22 diverse, high-quality subsets spanning academic, professional, internet…
fastText
fastText is an open-source library for learning word embeddings and performing text classification, developed by Facebook AI Research (FAIR) and released to the public on August 18, 2016 .
mT5
mT5 (multilingual T5) is a transformer-based encoder-decoder language model released by Google Research in October 2020 that covers 101 languages in a single model, pre-trained on a Common Crawl corpus called…
olmOCR
olmOCR is an open toolkit and vision-language model from the Allen Institute for AI (Ai2) that turns PDFs and document images into clean, structured plain text and Markdown.
tiktoken
tiktoken is an open source byte pair encoding (BPE) tokenizer library released by OpenAI in December 2022 that converts text into the integer token sequences its language models read and write.