RE-Bench
RE-Bench (short for Research Engineering Benchmark) is a benchmark for evaluating the frontier AI research-and-development capabilities of large language model agents, developed by the nonprofit METR (Model…
Explore Model Evaluation through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of Model Evaluation.
Showing 121-162 of 162 articles
RE-Bench (short for Research Engineering Benchmark) is a benchmark for evaluating the frontier AI research-and-development capabilities of large language model agents, developed by the nonprofit METR (Model…
A Receiver Operating Characteristic (ROC) curve is a graph that measures how well a binary classification system separates two classes by plotting its true positive rate (TPR, also called sensitivity or…
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of automatic metrics that score the quality of a machine-generated text summary by counting how many overlapping units (n-grams, word…
Ragas is an open-source Python framework for evaluating applications built with large language models. It provides evaluation metrics, test-data generation, and tools for comparing experiments.
Real-SWE is a coding-agent benchmark published in September 2026 by Specific Labs, a San Francisco company that licenses operational data and source code from businesses and packages them as training and…
Recall is a classification and retrieval metric that measures the proportion of actual positive instances a model correctly identifies, defined as TP / (TP + FN)
RewardBench is a benchmark and public leaderboard for evaluating reward models, the scoring functions that sit at the center of reinforcement learning from human feedback (RLHF).
RoboArena is a distributed, real-world benchmark for generalist robot policies in which volunteer evaluators at different laboratories run double-blind, pairwise (A/B) comparisons of policies on tasks and…
RoboChallenge is an online benchmarking platform that evaluates robot control policies, mainly vision-language-action models (VLAs), on physical robots instead of in simulation.
RoboDojo is a benchmark for generalist robot manipulation policies that combines a simulation suite of 42 tasks with a real-robot suite of 18 tasks, a standardized physical test platform called…
Root Mean Squared Error (RMSE), also known as root mean square deviation (RMSD), is a regression evaluation metric equal to the square root of the average of the squared differences between predicted and…
SWE-bench Verified is a 500-problem, human-validated subset of the SWE-bench software engineering benchmark, released on August 13
Sabotage evaluations are a set of tests, introduced by Anthropic in October 2024, that try to measure whether a frontier language model is capable of covertly subverting human oversight, decision-making, and…
Sandbagging, in the context of AI safety, refers to the strategic and intentional underperformance of an AI system on a capability evaluation or specific task, typically to hide a capability from human…
The SEAL Leaderboards are a set of expert-curated, contamination-resistant evaluation leaderboards for frontier large language models, produced by the Safety, Evaluations and Alignment Lab (SEAL) at Scale AI.
ScienceAgentBench is an AI benchmark for evaluating whether language model agents can perform real, data-driven scientific analysis by writing and executing code.
Self-Taught Evaluator is a method for training a strong LLM-as-a-judge without any human preference annotations, using synthetic training data and an iterative self-improvement loop.
SimpleQA Verified is a short-form factuality benchmark released by Google DeepMind and Google Research in September 2025 that measures the parametric knowledge of large language models using roughly 1,000…
SkillEvaluator is an open-source command-line framework developed by NVIDIA for evaluating artifacts used by AI agents.
Spider 2.0 is a benchmark for evaluating large language models on real-world enterprise text-to-SQL workflows.
A splitter is a term used in two distinct senses in machine learning. The first and most common sense is a utility that partitions a dataset into subsets such as training, validation, and test sets, or into…
The AI Index Report is an annual, independent, data-driven publication that tracks, distills, and visualizes trends in artificial intelligence across research and development, technical performance, the…
The State of AI Report is a free, annual review of progress in artificial intelligence, published every year since 2018.
StrongREJECT is a benchmark and automated evaluator for measuring how well jailbreaking attacks actually work against large language models.
SuperGPQA is a large graduate-level knowledge and reasoning benchmark for evaluating large language models across 285 academic disciplines.
The task-completion time horizon is a metric for AI capability proposed by METR that expresses a model's ability in units of human time: it is the length of task
Terminal-Bench is an open benchmark for evaluating AI agents on complex, real-world tasks performed through command-line terminal interfaces.
A test set is a collection of examples reserved from model fitting and model selection so that it can evaluate a fixed machine learning model or a fully specified learning procedure.
The Leaderboard Illusion is a 2025 research paper, led by Cohere and Cohere Labs with academic collaborators, that argues the most influential public ranking of large language models, Chatbot Arena
A true negative (TN) is a case that a binary classification model correctly predicts as belonging to the negative class: the true label is negative and the predicted label is also negative.
A true positive (TP) is a prediction that is correctly positive: the model predicts the positive class and the true label is also positive.
The true positive rate (TPR) is the proportion of actual positive cases that a classifier correctly identifies as positive, computed as TPR = TP / (TP + FN), where TP is the number of true positives and FN the…
The UK AI Security Institute (AISI) is a UK government research organization (part of the Department for Science, Innovation and Technology until July 2026, when responsibility for it moved to the Cabinet…
A validation set (also called a development set or dev set) is a subset of labeled data that is held out from the training set and used to evaluate a model's performance during development
Vals AI is an independent, third-party AI model evaluation company based in San Francisco that builds and publishes domain-specific benchmarks measuring how well large language models perform on real…
Vending-Bench is a family of benchmarks from Andon Labs that measures the long-term coherence of large language model-based AI agents by tasking them with running a simulated vending-machine business.
VulcanBench is an open-source benchmark for AI coding agents, created and run by Morgan Linton, co-founder and CTO of the apparel-sizing AI company Bold Metrics.
WebArena is a realistic, self-hosted web environment and benchmark designed for developing and evaluating autonomous AI agents that perform tasks on the web.
WebVoyager is an end-to-end web agent and its companion benchmark, introduced by Hongliang He and seven coauthors in a paper accepted to ACL 2024.
WildBench is an automated evaluation framework for large language models (LLMs) introduced by the Allen Institute for AI (AI2) in 2024.
Word error rate (WER) is the standard metric for measuring the accuracy of an automatic speech recognition (ASR) system
chrF is a machine translation evaluation metric that scores a candidate translation by counting the character n-grams it shares with one or more reference translations, then combining character n-gram…