AA-Briefcase
AA-Briefcase is a private benchmark from Artificial Analysis that tests AI agents on long-horizon office work.
Explore AI Benchmarks through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of AI Benchmarks.
Showing 1-60 of 79 articles
AA-Briefcase is a private benchmark from Artificial Analysis that tests AI agents on long-horizon office work.
AGIEval is an AI benchmark for evaluating foundation models on tasks that were originally designed for, and taken by, humans.
ARC-AGI-2 (Abstraction and Reasoning Corpus for Artificial General Intelligence 2) is an abstract reasoning benchmark for artificial intelligence, released on March 24, 2025 by the ARC Prize Foundation
Agent Arena is a public leaderboard run by Arena (the evaluation company formerly called LMArena
Agent evaluation is the systematic measurement of how well AI agents (LLM-based systems that plan and act over multiple steps using tools) perform on real-world tasks, using benchmarks, metrics, and testing…
Arena-Hard (and its evaluation tool Arena-Hard-Auto) is an automatic large language model (LLM) benchmark developed by the team behind Chatbot Arena that scores instruction-tuned models on 500 challenging
AutomationBench is a benchmark for AI agents, built by the automation company Zapier, that tests whether a model can finish a multi-step business workflow across several simulated software applications.
BABILong is a benchmark for testing how well a large language model can reason over facts scattered through very long text.
In artificial intelligence and machine learning, a benchmark is a specified evaluation used to compare systems under common conditions.
BigFinanceBench (BFB) is a benchmark for evaluating how well AI agents perform the research work of professional financial analysts, developed by the finance-AI company Rogo and announced on May 27, 2026.
BlenderWorld is a benchmark for visual AI agents that build editable 3D scenes in Blender, the open-source 3D creation suite, from text briefs, reference images, reference videos and structured specification…
BridgeBench is a website that publishes ratings and tests of large language models used for software development, describing itself as "independent benchmarks for AI coding models." It is built and operated by…
Build-A-Chip-Bench is a benchmark for AI agents that tests whether an agent can carry a digital chip design from a workload specification all the way to a routed, verified GDS layout
CaliBench is a benchmark for image-to-video generative models that asks whether a model reproduces the correct distribution of physical outcomes across many generations from the same starting frame
Chartography is a benchmark for professional chart understanding created by Surge AI.
ChemBench is an automated AI benchmark that measures the chemical knowledge, reasoning, and safety judgment of large language models and compares their performance against expert human chemists.
Coreference resolution is the natural language processing task of finding all the expressions in a text that refer to the same entity and grouping them together.
CritPt (Complex Research using Integrated Thinking - Physics Test, pronounced "critical point") is a benchmark that tests large language models on unpublished, research-level physics problems.
Crusader Bench is an AI benchmark that tests AI agents by having them develop a settlement's economy in Stronghold Crusader: Definitive Edition.
CursorBench is an agentic coding benchmark built and run by Cursor, the AI code editor made by Anysphere.
Cybench (short for Cybersecurity benchmark) is an open-source evaluation framework for measuring the cybersecurity capabilities and risks of large language model agents.
Data contamination is the presence of benchmark evaluation data in the corpus used to train a model
DeepSWE is a benchmark for AI coding agents made by Datacurve, a San Francisco data company.
DesignArena (also written Design Arena) is a crowdsourced benchmark and consumer creation platform for AI-generated design, built by the San Francisco startup Intelligence.
The Elo rating system, as applied to AI models, is a method for turning a pile of head-to-head preference votes into a single number per model, so that large language models and chatbots can be ranked on a…
EnigmaEval is an AI benchmark of long, complex multimodal puzzles drawn from real-world puzzle hunts, designed to measure the unstructured, creative, multi-step reasoning abilities of frontier AI models.
FACTS Grounding is a factuality benchmark from Google DeepMind and Google Research that measures whether a large language model answers a request using only the information in a provided source document
FRAMES is an evaluation dataset for retrieval-augmented generation that tests factual accuracy, retrieval, and reasoning together rather than one at a time.
FormulaOne is an AI benchmark for evaluating whether a model can design and implement dynamic programming algorithms for graph problems.
FrontierCode is an agentic coding benchmark created by Cognition, the company behind the Devin coding agent.
GDP.pdf is a benchmark from Surge AI that tests whether multimodal language models can answer realistic professional questions about real PDF documents, such as benefits packets, leases, datasheets, clinical…
GDPx is an AI benchmark published by SemiAnalysis on November 2, 2026 to evaluate agents working in simulated professional environments.
Global-MMLU is a multilingual evaluation benchmark that extends the MMLU question-answering dataset across 42 languages, with designated subsets labeled culturally sensitive (CS) and culturally agnostic (CA).
HELM (Holistic Evaluation of Language Models) is an open-source benchmark framework created by the Center for Research on Foundation Models (CRFM) at Stanford University for the reproducible and transparent…
HELMET (How to Evaluate Long-context Models Effectively and Thoroughly) is a benchmark for evaluating long-context language models introduced by researchers at Princeton University and Intel Labs in 2024.
HalluLens is a large language model hallucination benchmark introduced by researchers at Meta AI's Fundamental AI Research (FAIR) lab, together with collaborators at the Hong Kong University of Science and…
Harness, in artificial intelligence, is the software scaffolding that wraps a machine learning model, most often a large language model
KernelBench is an AI benchmark and open-source evaluation environment that measures how well large language models can write fast and correct GPU kernels.
LAB-Bench (the Language Agent Biology Benchmark) is an AI benchmark of more than 2,400 multiple-choice questions built to measure how well language models and AI agents can perform practical biology research…
As of July 2026, no single model wins every LLM benchmark, but Anthropic's Claude Fable 5 tops the most: it currently leads MMLU-Pro (91.5%), SWE-bench Verified (95.0%), Humanity's Last Exam in both the…
LLM evaluation is the practice of measuring what a large language model can do, how reliably it does it, and how it behaves under adversarial or high-stakes conditions.
LLM-as-a-judge is the practice of using a strong large language model to evaluate the outputs of other models, or of itself, in place of a human annotator.
LM Evaluation Harness (the Language Model Evaluation Harness, often abbreviated lm-eval or written lm-evaluation-harness) is an open-source software framework for measuring the performance of large language…
LMArena is a crowdsourced artificial intelligence evaluation platform and company that ranks large language models by having anonymous users vote on which of two blind, side-by-side model responses they prefer
LiveNerf (styled livenerf in its repository) is an open-source, pre-registered benchmark that tracks whether a frontier large language model gets worse after it is released.
LongBench v2 is a benchmark for evaluating how well large language models understand and reason over long contexts.
MATH is a benchmark of 12,500 competition mathematics problems used to evaluate the mathematical problem-solving ability of machine learning systems, particularly large language models.
MMLU-Redux is a manually re-annotated, error-corrected subset of the Massive Multitask Language Understanding (MMLU) benchmark
MRCR (Multi-Round Co-reference Resolution) is a synthetic long-context evaluation that tests whether a large language model can locate and disambiguate among several near-identical "needles" buried inside a…
MuSR (Multistep Soft Reasoning) is a benchmark for evaluating multistep reasoning in large language models, built around long free-text narratives such as murder mysteries, object-placement scenarios, and…
Multi-SWE-bench is a multilingual benchmark for evaluating the ability of large language model based coding systems to resolve real-world software issues across seven programming languages.
MultiLangBench is an AI coding benchmark introduced by OpenAI on October 19, 2026. It evaluates implementations of a SQLite-compatible database engine across 20 programming languages.
NoLiMa, short for "No Literal Matching," is a long-context benchmark for large language models that measures how well a model can find and use a single relevant fact buried in a long document when that fact…
OlympiadBench is an AI benchmark of 8,476 Olympiad-level mathematics and physics problems, designed to test the advanced scientific reasoning of large language models and large multimodal models.
Omni-MATH is an AI benchmark of Olympiad-level competition mathematics, introduced in October 2024 to measure the mathematical reasoning ability of large language models on problems far harder than those in…
Pass@k is the standard metric for evaluating code generation models: it measures the probability that at least one of k generated candidate solutions passes all of a problem's unit tests.
PhyGenBench (Physics Generation Benchmark) is a benchmark for testing whether text-to-video models produce videos that follow basic physical commonsense.
Physics-IQ Verified is an audited and re-scored version of Physics-IQ, the Google DeepMind benchmark that tests whether video generation models can predict how real physical experiments unfold.
ProcessBench is a benchmark for step-level verification of mathematical reasoning, built by the Qwen Team at Alibaba and released in December 2024.
PutnamBench is a benchmark for evaluating automated theorem proving systems, especially neural and large-language-model-based provers, on competition mathematics.