Showing 121-180 of 220 articles
MBPP
MBPP (Mostly Basic Python Problems) is a code generation benchmark of 974 crowd-sourced Python programming tasks designed to be solvable by entry-level...
AI Code GenerationLarge Language Models
METR
METR (Model Evaluation and Threat Research) is a nonprofit research organization based in Berkeley, California, that develops scientific methods for measuring...
AI SafetyResearch Organizations
MGSM (Multilingual Grade School Math)
Multilingual Grade School Math Abbreviation A multilingual benchmark evaluating mathematical reasoning across 10 typologically diverse languages using...
Natural Language Processing
MLE-bench
MLE-bench is a benchmark for measuring how well autonomous AI agents perform real-world machine learning engineering, released by OpenAI's Preparedness team on...
AI AgentsOpenAI
MLPerf
MLPerf is the AI industry's standard suite of benchmarks for measuring how fast machine learning systems train and run models, developed and maintained by...
AI HardwareMachine Learning
MM-BrowseComp
MM-BrowseComp is a benchmark for evaluating multimodal web-browsing AI agents, introduced in August 2025 by researchers from ByteDance, Nanjing University,...
AI AgentsMultimodal AI
MM-Vet
MM-Vet is an AI benchmark for evaluating large multimodal models (LMMs, also called multimodal large language models or MLLMs) on tasks that require combining...
Computer Vision
MMLU
Measuring Massive Multitask Language Understanding Abbreviation A comprehensive benchmark evaluating large language models across 57 diverse academic...
MMLU-Pro
Massive Multitask Language Understanding Professional Abbreviation A more robust and challenging multi-task language understanding benchmark with 10-choice...
Large Language ModelsMachine Learning
MMLU-ProX
MMLU-ProX is a multilingual benchmark for evaluating reasoning and knowledge in large language models, extending the English-only MMLU-Pro test set to 29...
Large Language Models
MMLU-Redux
MMLU-Redux is a manually re-annotated, error-corrected subset of the Massive Multitask Language Understanding (MMLU) benchmark, intended to serve as a cleaner...
Model Evaluation
MMMLU
Multilingual Massive Multitask Language Understanding Abbreviation Professional human translations of the MMLU test set into 14 languages, released by...
Data & Datasets
MMMU
MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark) is a multimodal AI benchmark of 11,550 college-level questions that pairs text...
Machine LearningMultimodal AI
MMMU-Pro
MMMU-Pro is a rigorous benchmark for evaluating multimodal AI systems on college-level, expert questions that genuinely require seeing an image, built as a...
Computer VisionLarge Language Models
MMStar
MMStar (Multi-modal Star) is a vision-language model evaluation benchmark consisting of 1,500 multimodal samples that were filtered from six pre-existing...
Multimodal AI
MMTEB
MMTEB (Massive Multilingual Text Embedding Benchmark) is a large, community-built suite for evaluating text embedding models across more than 500...
Information RetrievalNatural Language Processing
MRCR
MRCR (Multi-Round Co-reference Resolution) is a synthetic long-context evaluation that tests whether a large language model can locate and disambiguate among...
Large Language ModelsModel Evaluation
MT-Bench
MT-Bench (Multi-Turn Benchmark) is a benchmark of 80 hand-written, two-turn questions that evaluates large language models (LLMs) on multi-turn conversation...
Large Language ModelsNatural Language Processing
MTEB (Massive Text Embedding Benchmark)
MTEB, short for Massive Text Embedding Benchmark, is the standard public leaderboard for evaluating text embedding models across many task types at once. It...
Artificial IntelligenceInformation Retrieval
MathArena
MathArena: Evaluating LLMs on Uncontaminated Math Competitions Description March 2025 (USAMO 2025 evaluation), May 2025 (paper) Latest expansion Mislav...
Artificial IntelligenceNatural Language Processing
MathVista
MathVista is a benchmark for evaluating the mathematical reasoning capabilities of foundation models in visual contexts.[1] It was introduced by Pan Lu, Hritik...
Multimodal AI
MedHELM
MedHELM (Holistic Evaluation of Large Language Models for Medical Tasks) is a benchmark and evaluation framework that measures how well large language models...
AI Safety
MedQA
MedQA is a large-scale, open-domain medical question answering benchmark of multiple-choice questions taken from real medical licensing examinations,...
Healthcare AINatural Language Processing
Mike Knoop
Mike Knoop is an American technology entrepreneur and artificial intelligence researcher who co-founded the workflow-automation company Zapier in 2011 and, in...
People
Mind2Web
Mind2Web is the first large-scale dataset and benchmark for building and evaluating generalist AI agents that follow natural language instructions to complete...
AI Agents
MuSR
MuSR (Multistep Soft Reasoning) is a benchmark for evaluating multistep reasoning in large language models, built around long free-text narratives such as...
Model EvaluationReasoning Models
Multi-SWE-bench
Multi-SWE-bench is a multilingual benchmark for evaluating the ability of large language model based coding systems to resolve real-world software issues...
AI Code GenerationModel Evaluation
MultiChallenge
MultiChallenge is an AI benchmark for evaluating large language models on realistic multi-turn conversations. It was introduced in the paper "MultiChallenge: A...
AI Code Generation
Natural language inference (NLI)
Natural language inference (NLI), also known as recognising textual entailment (RTE), is the natural language processing task of deciding whether a hypothesis...
Natural Language ProcessingReasoning Models
Needle in a Haystack (NIAH)
Needle in a Haystack (NIAH) is a long-context evaluation that measures whether a large language model can retrieve a single fact (the "needle") inserted at a...
Large Language Models
NoLiMa
NoLiMa, short for "No Literal Matching," is a long-context benchmark for large language models that measures how well a model can find and use a single...
Large Language ModelsModel Evaluation
OSWorld
OSWorld is a benchmark for evaluating multimodal AI agents on open-ended tasks performed inside real computer environments. Published at NeurIPS 2024 by...
AI Agents
OlympiadBench
OlympiadBench is an AI benchmark of 8,476 Olympiad-level mathematics and physics problems, designed to test the advanced scientific reasoning of large language...
Model Evaluation
Omni-MATH
Omni-MATH is an AI benchmark of Olympiad-level competition mathematics, introduced in October 2024 to measure the mathematical reasoning ability of large...
Model Evaluation
Online-Mind2Web
Online-Mind2Web is an AI benchmark for evaluating AI agents that operate web browsers, run against live, real websites rather than cached page snapshots. It...
AI Code Generation
OpenAI MRCR (Multi-Round Co-reference Resolution)
OpenAI MRCR (Multi-Round Co-reference Resolution) is a long-context evaluation dataset published by OpenAI that measures a language model's ability to...
AI Safety
PASCAL VOC
PASCAL VOC (Pattern Analysis, Statistical Modelling and Computational Learning Visual Object Classes) is a long-running benchmark dataset and annual challenge...
Computer VisionData & Datasets
PIQA
PIQA (Physical Interaction Question Answering) is a benchmark dataset of roughly 21,000 binary multiple-choice questions that evaluates the physical...
Natural Language Processing
Paper2Video
Paper2Video (full title: Paper2Video: Automatic Video Generation from Scientific Papers) is a research project from Show Lab at the National University of...
AI ResearchMultimodal AI
PaperBench
PaperBench is a benchmark released by OpenAI in April 2025 that measures whether AI agents can replicate cutting-edge machine learning research papers from...
AI AgentsOpenAI
Pass@k
Pass@k is the standard metric for evaluating code generation models: it measures the probability that at least one of k generated candidate solutions passes...
AI Code GenerationMachine Learning
ProcessBench
ProcessBench is a benchmark for step-level verification of mathematical reasoning, built by the Qwen Team at Alibaba and released in December 2024. [1] It...
Model EvaluationReasoning Models
PubMedQA
PubMedQA is a biomedical question answering dataset and benchmark that evaluates whether machine learning models can answer yes/no/maybe research questions...
Healthcare AINatural Language Processing
PutnamBench
PutnamBench is a benchmark for evaluating automated theorem proving systems, especially neural and large-language-model-based provers, on competition...
Model Evaluation
RE-Bench
RE-Bench (short for Research Engineering Benchmark) is a benchmark for evaluating the frontier AI research-and-development capabilities of large language model...
Model Evaluation
RULER (benchmark)
RULER: What's the Real Context Size of Your Long-Context Language Models? Abbreviation A synthetic benchmark for evaluating long-context large language...
Large Language Models
RepoBench
RepoBench is an AI benchmark for repository-level code auto-completion, introduced in the 2023 paper "RepoBench: Benchmarking Repository-Level Code...
AI Code Generation
RewardBench
RewardBench is a benchmark and public leaderboard for evaluating reward models, the scoring functions that sit at the center of reinforcement learning from...
Model EvaluationReinforcement Learning
SIMPLER
SIMPLER (Simulated Manipulation Policy Evaluation for Real Robot Setups) is a collection of simulated robot manipulation environments, released in 2024, that...
Robotics
SQuAD
SQuAD (the Stanford Question Answering Dataset) is a large-scale reading comprehension benchmark from Stanford University in which a model must answer a...
Natural Language Processing
SUPERB
SUPERB, which stands for Speech processing Universal PERformance Benchmark, is a comprehensive evaluation framework designed to measure how well...
Machine LearningSpeech & Audio AI
SWE-Atlas
SWE-Atlas is a benchmark for evaluating AI coding agents on professional software-engineering work that goes beyond fixing bugs and resolving issues. Built by...
AI AgentsAI Code Generation
SWE-Bench Pro
SWE-Bench Pro (stylized SWE-BENCH PRO) is a contamination-resistant benchmark, released by Scale AI in September 2025, that measures whether an AI coding agent...
AI AgentsAI Code Generation
SWE-Lancer
SWE-Lancer is a benchmark released by OpenAI in February 2025 that evaluates the ability of frontier large language models to perform real-world freelance...
AI Code GenerationOpenAI
SWE-bench
Software Engineering Benchmark Abbreviation A benchmark for evaluating large language models and AI agents on real-world software engineering tasks from...
Software Development
SWE-bench Multilingual
SWE-bench Multilingual is an AI benchmark of 300 real-world software bug-fixing tasks drawn from 42 open-source repositories across nine programming languages,...
AI Code Generation
SWE-bench Multimodal
SWE-bench Multimodal (also written SWE-bench M) is a benchmark that measures whether autonomous software-engineering systems can resolve bugs in visual,...
AI Code GenerationMultimodal AI
SWE-bench Verified
Software Engineering Benchmark, Verified subset Abbreviation A 500-task, human-validated subset of SWE-bench for evaluating AI agents on real-world GitHub...
AI Code GenerationModel Evaluation
SWE-rebench
SWE-rebench is a continuously refreshed, contamination-resistant AI benchmark and public leaderboard for evaluating AI agents on real-world software...
AI Code Generation
SciCode
--- Full name SciCode Description July 2024 (arXiv); NeurIPS 2024 Datasets and Benchmarks Track Authors University of Illinois Urbana-Champaign,...