Showing 61-120 of 121 articles
Langfuse
Langfuse is an open-source LLM engineering platform that provides observability, tracing, prompt management, evaluation, and dataset tooling for applications...
AI CompaniesDeveloper Tools
LongBench v2
LongBench v2 is a benchmark for evaluating how well large language models understand and reason over long contexts. It consists of 503 challenging...
AI BenchmarksLarge Language Models
Loss Curve
See also: Machine learning terms, Loss function A loss curve is a plot that shows the value of a loss function over the course of training a machine learning...
Deep LearningMachine Learning
MATH
MATH is a benchmark of 12,500 competition mathematics problems used to evaluate the mathematical problem-solving ability of machine learning systems,...
AI BenchmarksReasoning Models
METEOR (metric)
METEOR (Metric for Evaluation of Translation with Explicit ORdering) is an automatic evaluation metric for machine translation and other text-generation tasks...
Machine LearningNatural Language Processing
MMLU-Redux
MMLU-Redux is a manually re-annotated, error-corrected subset of the Massive Multitask Language Understanding (MMLU) benchmark, intended to serve as a cleaner...
AI Benchmarks
MRCR
MRCR (Multi-Round Co-reference Resolution) is a synthetic long-context evaluation that tests whether a large language model can locate and disambiguate among...
AI BenchmarksLarge Language Models
Mean Absolute Error (MAE)
Mean Absolute Error (MAE) is a regression accuracy metric and loss function that measures the average absolute difference between predicted values and actual...
Machine LearningStatistics
Mean Squared Error (MSE)
See also: Machine learning terms Mean Squared Error (MSE), also called mean squared deviation (MSD), is the average of the squared differences between...
Machine LearningStatistics
Model Capacity
Model capacity is the size and richness of the family of functions a machine learning model can represent and learn, which determines how complex a pattern the...
Machine Learning
MuSR
MuSR (Multistep Soft Reasoning) is a benchmark for evaluating multistep reasoning in large language models, built around long free-text narratives such as...
AI BenchmarksReasoning Models
Multi-SWE-bench
Multi-SWE-bench is a multilingual benchmark for evaluating the ability of large language model based coding systems to resolve real-world software issues...
AI BenchmarksAI Code Generation
NIST ARIA
NIST ARIA (Assessing Risks and Impacts of AI) is a testing, evaluation, validation, and verification (TEVV) program operated by the United States National...
AI Policy & RegulationAI Safety
NoLiMa
NoLiMa, short for "No Literal Matching," is a long-context benchmark for large language models that measures how well a model can find and use a single...
AI BenchmarksLarge Language Models
OlympiadBench
OlympiadBench is an AI benchmark of 8,476 Olympiad-level mathematics and physics problems, designed to test the advanced scientific reasoning of large language...
AI Benchmarks
Omni-MATH
Omni-MATH is an AI benchmark of Olympiad-level competition mathematics, introduced in October 2024 to measure the mathematical reasoning ability of large...
AI Benchmarks
Out-of-bag evaluation (OOB evaluation)
See also: Machine learning terms, Bagging, Random forest, Cross-validation Out-of-bag (OOB) evaluation, sometimes called out-of-bag estimation or OOB error, is...
Machine Learning
Overfitting
Overfitting is when a machine learning model fits its training set so closely that it learns the noise and quirks of those specific examples instead of the...
Deep LearningMachine Learning
PR AUC
See also: precision, recall, ROC curve, AUC, F1 score, confusion matrix, precision-recall curve PR AUC (Precision-Recall Area Under the Curve), also referred...
Machine Learning
Pass@k
Pass@k is the standard metric for evaluating code generation models: it measures the probability that at least one of k generated candidate solutions passes...
AI BenchmarksAI Code Generation
Patronus AI
Patronus AI is an automated LLM evaluation, observability, and guardrails platform founded in 2023[1] and headquartered in San Francisco. The company builds...
AI CompaniesAI Safety
Precision
Precision is a classification metric defined as the fraction of positive predictions that are correct: Precision = TP / (TP + FP), where TP is the number of...
Machine Learning
Precision-Recall Curve
A precision-recall curve (PR curve) is a graph that plots precision on the y-axis against recall on the x-axis at every possible classification threshold for a...
Machine Learning
Prediction Bias
Prediction bias is the difference between the average of a machine learning model's predictions and the average of the ground-truth labels in a dataset. Stated...
Machine Learning
Process reward model (PRM)
A process reward model (PRM), also called a process-supervised reward model or step-level verifier, is a learned scoring model that evaluates the correctness...
AI SafetyMachine Learning
ProcessBench
ProcessBench is a benchmark for step-level verification of mathematical reasoning, built by the Qwen Team at Alibaba and released in December 2024. [1] It...
AI BenchmarksReasoning Models
PutnamBench
PutnamBench is a benchmark for evaluating automated theorem proving systems, especially neural and large-language-model-based provers, on competition...
AI Benchmarks
RE-Bench
RE-Bench (short for Research Engineering Benchmark) is a benchmark for evaluating the frontier AI research-and-development capabilities of large language model...
AI Benchmarks
ROC (Receiver Operating Characteristic) Curve
A Receiver Operating Characteristic (ROC) curve is a graph that measures how well a binary classification system separates two classes by plotting its true...
Machine Learning
ROUGE
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of automatic metrics that score the quality of a machine-generated text summary by counting...
Machine LearningNatural Language Processing
Recall (metric)
Recall is a classification and retrieval metric that measures the proportion of actual positive instances a model correctly identifies, defined as TP / (TP +...
Machine Learning
RewardBench
RewardBench is a benchmark and public leaderboard for evaluating reward models, the scoring functions that sit at the center of reinforcement learning from...
AI BenchmarksReinforcement Learning
Root Mean Squared Error (RMSE)
Root Mean Squared Error (RMSE), also known as root mean square deviation (RMSD), is a regression evaluation metric equal to the square root of the average of...
Machine LearningStatistics
SWE-bench Verified
Software Engineering Benchmark, Verified subset Abbreviation A 500-task, human-validated subset of SWE-bench for evaluating AI agents on real-world GitHub...
AI BenchmarksAI Code Generation
Sabotage evaluations
Sabotage evaluations are a set of tests, introduced by Anthropic in October 2024, that try to measure whether a frontier language model is capable of covertly...
AI SafetyAnthropic
Sandbagging (artificial intelligence)
Sandbagging, in the context of AI safety, refers to the strategic and intentional underperformance of an AI system on a capability evaluation or specific task,...
AI Safety
Scale SEAL Leaderboards
The SEAL Leaderboards are a set of expert-curated, contamination-resistant evaluation leaderboards for frontier large language models, produced by the Safety,...
AI Companies
ScienceAgentBench
ScienceAgentBench is an AI benchmark for evaluating whether language model agents can perform real, data-driven scientific analysis by writing and executing...
AI Benchmarks
Self-Taught Evaluator
Self-Taught Evaluator is a method for training a strong LLM-as-a-judge without any human preference annotations, using synthetic training data and an iterative...
Meta AITraining & Optimization
SimpleQA Verified
SimpleQA Verified is a short-form factuality benchmark released by Google DeepMind and Google Research in September 2025 that measures the parametric knowledge...
AI BenchmarksGoogle DeepMind
Spider 2.0
Spider 2.0 is a benchmark for evaluating large language models on real-world enterprise text-to-SQL workflows. Introduced by Fangyu Lei and colleagues in the...
AI BenchmarksAI Code Generation
Splitter
A splitter is a term used in two distinct senses in machine learning. The first and most common sense is a utility that partitions a dataset into subsets such...
Machine Learning
Stanford HAI AI Index Report
The AI Index Report is an annual, independent, data-driven publication that tracks, distills, and visualizes trends in artificial intelligence across research...
AI Companies
State of AI Report
The State of AI Report is a free, annual review of progress in artificial intelligence, published every year since 2018. It is compiled by the investor Nathan...
AI Companies
StrongREJECT
StrongREJECT is a benchmark and automated evaluator for measuring how well jailbreaking attacks actually work against large language models. It was introduced...
AI BenchmarksAI Safety
SuperGPQA
SuperGPQA is a large graduate-level knowledge and reasoning benchmark for evaluating large language models across 285 academic disciplines. It contains 26,529...
AI Benchmarks
Task-completion time horizon (METR)
The task-completion time horizon is a metric for AI capability proposed by METR that expresses a model's ability in units of human time: it is the length of...
AI BenchmarksAI Safety
Terminal-Bench
Terminal-Bench is an open benchmark for evaluating AI agents on complex, real-world tasks performed through command-line terminal interfaces. Developed as a...
AI AgentsAI Benchmarks
Test Set
A test set is a portion of data held back from model development and used only once, after all training and tuning is complete, to give an unbiased estimate of...
Machine Learning
The Leaderboard Illusion
The Leaderboard Illusion is a 2025 research paper, led by Cohere and Cohere Labs with academic collaborators, that argues the most influential public ranking...
AI Companies
True negative
A true negative (TN) is a case that a binary classification model correctly predicts as belonging to the negative class: the true label is negative and the...
Machine LearningStatistics
True positive
A true positive (TP) is a prediction that is correctly positive: the model predicts the positive class and the true label is also positive.[^1][^3][^17] It is...
Machine LearningStatistics
True positive rate (TPR)
The true positive rate (TPR) is the proportion of actual positive cases that a classifier correctly identifies as positive, computed as TPR = TP / (TP + FN),...
Machine Learning
UK AI Security Institute
The UK AI Security Institute (AISI) is a research organization within the United Kingdom's Department for Science, Innovation and Technology (DSIT) that...
AI Policy & RegulationAI Safety
Validation Set
A validation set (also called a development set or dev set) is a subset of labeled data that is held out from the training set and used to evaluate a model's...
Machine Learning
Vals AI
Vals AI is an independent, third-party AI model evaluation company based in San Francisco that builds and publishes domain-specific benchmarks measuring how...
AI Companies
WebArena
WebArena is a realistic, self-hosted web environment and benchmark designed for developing and evaluating autonomous AI agents that perform tasks on the web....
AI AgentsAI Benchmarks
WebVoyager
WebVoyager is an end-to-end web agent and its companion benchmark, introduced by Hongliang He and seven coauthors in a paper accepted to ACL 2024 [1][2]. It is...
AI AgentsAI Benchmarks
WildBench
WildBench is an automated evaluation framework for large language models (LLMs) introduced by the Allen Institute for AI (AI2) in 2024. It evaluates models on...
AI Benchmarks
Word error rate
Word error rate (WER) is the standard metric for measuring the accuracy of an automatic speech recognition (ASR) system, expressed as the fraction of words the...
Machine LearningNatural Language Processing