AI Benchmarks

Explore AI Benchmarks through related topics and the articles other pages reference most.

Explore articles

Browse subtopics (56)

Articles that also belong to these categories. Counts cover all of AI Benchmarks.

Showing 181-240 of 268 articles

MultiLangBench

MultiLangBench is an AI coding benchmark introduced by OpenAI on October 19, 2026. It evaluates implementations of a SQLite-compatible database engine across 20 programming languages.

Model Evaluation

Needle in a Haystack (NIAH)

Needle in a Haystack (NIAH) is a long-context evaluation that measures whether a large language model can retrieve a single fact (the "needle") inserted at a controlled position inside a long body of text (the…

Large Language Models

NoLiMa

NoLiMa, short for "No Literal Matching," is a long-context benchmark for large language models that measures how well a model can find and use a single relevant fact buried in a long document when that fact…

Large Language ModelsModel Evaluation

OSWorld

OSWorld is a benchmark for evaluating multimodal AI agents on open-ended tasks performed inside real computer environments.

AI Agents

OlympiadBench

OlympiadBench is an AI benchmark of 8,476 Olympiad-level mathematics and physics problems, designed to test the advanced scientific reasoning of large language models and large multimodal models.

Model Evaluation

Omni-MATH

Omni-MATH is an AI benchmark of Olympiad-level competition mathematics, introduced in October 2024 to measure the mathematical reasoning ability of large language models on problems far harder than those in…

Model Evaluation

Online-Mind2Web

Online-Mind2Web is a live-web benchmark for measuring whether browser-based AI agents can complete multi-step tasks on real websites.

AI Agents

PASCAL VOC

PASCAL VOC (Pattern Analysis, Statistical Modelling and Computational Learning Visual Object Classes) is a long-running benchmark dataset and annual challenge for object recognition, object detection…

Computer VisionData & Datasets

PIQA

PIQA (Physical Interaction Question Answering) is a benchmark dataset of roughly 21,000 binary multiple-choice questions that evaluates the physical commonsense reasoning abilities of natural language…

Natural Language Processing

Paper2Video

Paper2Video (full title: Paper2Video: Automatic Video Generation from Scientific Papers) is a research project from Show Lab at the National University of Singapore that formalizes and evaluates automatic…

AI ResearchMultimodal AI

PaperBench

PaperBench is a benchmark released by OpenAI in April 2025 that measures whether AI agents can replicate cutting-edge machine learning research papers from scratch.

AI AgentsOpenAI

Pass@k

Pass@k is the standard metric for evaluating code generation models: it measures the probability that at least one of k generated candidate solutions passes all of a problem's unit tests.

AI Code GenerationMachine Learning

Physics-IQ

Physics-IQ is a benchmark that tests whether generative video models understand physical principles, by asking them to continue real videos of physical experiments and comparing the result with what actually…

Google DeepMindVideo Generation

PubMedQA

PubMedQA is a biomedical question answering dataset and benchmark that evaluates whether machine learning models can answer yes/no/maybe research questions using evidence from PubMed abstracts.

Healthcare AINatural Language Processing

PutnamBench

PutnamBench is a benchmark for evaluating automated theorem proving systems, especially neural and large-language-model-based provers, on competition mathematics.

Model Evaluation

RE-Bench

RE-Bench (short for Research Engineering Benchmark) is a benchmark for evaluating the frontier AI research-and-development capabilities of large language model agents, developed by the nonprofit METR (Model…

Model Evaluation

RSI-Exam

RSI-Exam is a benchmark that tests whether an AI agent can take a working but weak executable method, improve it through hours of autonomous experimentation, and produce a version that still performs better on…

AI Agents

RULER (benchmark)

RULER is a synthetic benchmark from NVIDIA that measures the real, usable context window of large language models (LLMs) by testing them on 13 tasks across four categories (retrieval, multi-hop tracing…

Large Language Models

Real-SWE

Real-SWE is a coding-agent benchmark published in September 2026 by Specific Labs, a San Francisco company that licenses operational data and source code from businesses and packages them as training and…

AI Code GenerationModel Evaluation

RepoBench

RepoBench is an AI benchmark for repository-level code auto-completion, introduced in the 2023 paper "RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems" by Tianyang Liu, Canwen Xu, and…

AI Code Generation

RoboArena

RoboArena is a distributed, real-world benchmark for generalist robot policies in which volunteer evaluators at different laboratories run double-blind, pairwise (A/B) comparisons of policies on tasks and…

Embodied AIModel Evaluation

RoboChallenge

RoboChallenge is an online benchmarking platform that evaluates robot control policies, mainly vision-language-action models (VLAs), on physical robots instead of in simulation.

Embodied AIModel Evaluation

RoboDojo

RoboDojo is a benchmark for generalist robot manipulation policies that combines a simulation suite of 42 tasks with a real-robot suite of 18 tasks, a standardized physical test platform called…

Embodied AIModel Evaluation

Robocurve

Robocurve (Robocurve, Inc., which styles itself Robocurve PBC) is a San Francisco public benefit corporation that evaluates how well frontier AI models and robot-control models perform on real robots, and…

AI CompaniesAI Safety

SIMPLER

SIMPLER (Simulated Manipulation Policy Evaluation for Real Robot Setups) is a collection of simulated robot manipulation environments, released in 2024

Robotics

SQuAD

SQuAD (the Stanford Question Answering Dataset) is a large-scale reading comprehension benchmark from Stanford University in which a model must answer a question by extracting the exact span of text that…

Natural Language Processing

SUPERB

SUPERB, which stands for Speech processing Universal PERformance Benchmark, is a comprehensive evaluation framework designed to measure how well self-supervised learning (SSL) models generalize across a…

Machine LearningSpeech & Audio AI

SWE-Atlas

SWE-Atlas is a benchmark for evaluating AI coding agents on professional software-engineering work that goes beyond fixing bugs and resolving issues.

AI AgentsAI Code Generation

SWE-Bench Pro

SWE-Bench Pro (stylized SWE-BENCH PRO) is a contamination-resistant benchmark, released by Scale AI in September 2025, that measures whether an AI coding agent can resolve long-horizon

AI AgentsAI Code Generation

SWE-Lancer

SWE-Lancer is a benchmark released by OpenAI in February 2025 that evaluates the ability of frontier large language models to perform real-world freelance software-engineering work.

AI Code GenerationOpenAI

SWE-bench

SWE-bench is an execution-based benchmark for evaluating whether a language-model system can resolve real software issues.

Software Development

SWE-rebench

SWE-rebench is a continuously refreshed, contamination-resistant AI benchmark and public leaderboard for evaluating AI agents on real-world software engineering tasks.

AI Code Generation

Samaya AI

Samaya AI is an American artificial intelligence company that builds AI agents for financial research and analysis, used by Morgan Stanley and a number of hedge funds.

AI AgentsAI Companies

SciCode

SciCode is a research coding benchmark that asks large language models to write Python code for realistic, PhD level scientific problems drawn from working scientists' day to day workflows.

ScienceAgentBench

ScienceAgentBench is an AI benchmark for evaluating whether language model agents can perform real, data-driven scientific analysis by writing and executing code.

Model Evaluation

SimpleQA

SimpleQA is a factuality benchmark released by OpenAI on October 30, 2024 that measures whether large language models can answer short, fact-seeking questions correctly instead of producing hallucinations.

AI SafetyNatural Language Processing

SimpleQA Verified

SimpleQA Verified is a short-form factuality benchmark released by Google DeepMind and Google Research in September 2025 that measures the parametric knowledge of large language models using roughly 1,000…

Google DeepMindModel Evaluation

SkillsBench

SkillsBench is a benchmark for measuring whether Agent Skills, the structured packages of procedural knowledge that augment AI agents at inference time, actually improve how well those agents do real work.

AI Agents

Specific Labs

Specific Labs is a San Francisco company that buys and licenses operational data and source code from businesses, and packages it as training and evaluation material for AI labs.

AI CompaniesData & Datasets

SuperGLUE

SuperGLUE is a public benchmark for evaluating general-purpose English language understanding in machine learning systems, consisting of eight challenging reasoning and reading-comprehension tasks, two…

Data & DatasetsNatural Language Processing

SuperGPQA

SuperGPQA is a large graduate-level knowledge and reasoning benchmark for evaluating large language models across 285 academic disciplines.

Model Evaluation

T2I-CompBench

T2I-CompBench is an AI benchmark for evaluating compositional text-to-image generation, introduced in the paper "T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation"…

Computer Vision

Tau2-bench

τ²-bench (also written Tau2-bench or τ^2-bench) is a benchmark for evaluating conversational AI agents in dual-control environments, where both the agent and a simulated user can call tools to read from and…

AI Agents

TheAgentCompany

TheAgentCompany is an AI benchmark that evaluates AI agents on long-horizon, economically valuable knowledge work inside a self-hosted simulation of a small software company.

AI Code Generation