# Benchmark (AI)

> Source: https://aiwiki.ai/wiki/benchmark
> Updated: 2026-07-29
> Fact-checked: 2026-08-04
> Categories: AI Benchmarks, Model Evaluation
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Benchmark (AI)." aiwiki.ai, 29 Jul 2026. https://aiwiki.ai/wiki/benchmark
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

In [artificial intelligence](https://aiwiki.ai/wiki/artificial_intelligence) and [machine learning](https://aiwiki.ai/wiki/machine_learning), a **benchmark** is a specified evaluation used to compare systems under common conditions. A benchmark normally combines a target capability or property, a set of tasks or test items, an evaluation protocol, one or more scoring rules, and a reporting procedure. A score is therefore evidence about performance under that particular design, not a direct measurement of intelligence or of every property needed in deployment.[1][2][3]

Benchmarks make empirical results easier to compare, expose model weaknesses, and help track progress. They also shape research incentives because the choice of tasks, metrics, and aggregation rules determines which improvements are visible. A useful benchmark states what it is intended to measure, what it leaves out, how examples were sampled and checked, how systems may interact with the test, and how uncertainty should affect comparisons.[1][2]

## Definition and scope

The word benchmark is used for several related objects. It can mean a fixed dataset with a conventional metric, a competition with a hidden test server, a suite that combines many datasets, or an interactive environment in which an agent acts and receives observations. It can also refer informally to the reported result itself. Keeping these meanings separate matters because a dataset alone does not specify an evaluation.

A benchmark differs from a general [model evaluation](https://aiwiki.ai/wiki/model_evaluation). Evaluation is the broader process of determining whether a system is suitable for a purpose. It may include benchmark results, safety tests, red-team exercises, user studies, cost measurements, monitoring, and application-specific validation. A benchmark supplies a repeatable measurement procedure within that process.

A benchmark also differs from a scientific claim. A result such as "82 percent accuracy" describes performance only after the population of items, response format, model configuration, and scoring code have been fixed. A broader statement such as "the model reasons well" additionally requires an argument that the measured task represents reasoning and is not primarily measuring memorization, formatting compliance, language familiarity, or another confounder. This link between the intended phenomenon and the evidence produced by a test is called construct validity.[1][3]

## Anatomy of a benchmark

### Target phenomenon and intended use

Design begins with the phenomenon or decision the benchmark is meant to inform. Examples include recognizing objects in photographs, transferring linguistic knowledge across tasks, generating functionally correct programs, following a policy while using tools, or satisfying human preferences. Broad labels such as reasoning, helpfulness, safety, or general capability need an operational definition because different definitions imply different tasks and interpretations.[1]

The intended use should identify the relevant system and setting. A test designed to compare pretrained language models without tools does not automatically support claims about assistants with retrieval, and a test of an interactive agent in a sandbox does not by itself establish reliability in a changing production environment. Benchmark documentation should name the covered population, languages, modalities, domains, and deployment conditions, as well as excluded cases.

### Tasks, items, and data

The task specifies what the evaluated system receives and what it must produce or do. An item can be an image and class label, a question and accepted answer, a source sentence and reference translations, a repository state and issue description, or an initial computer environment and goal. Interactive benchmarks may also define allowed actions, tools, observations, time limits, and termination conditions.

Items are samples from a larger task space. Their source and selection process affect what can be inferred from a score. Random or stratified sampling can support claims about a defined population when the sampling frame is appropriate. Targeted sampling can probe known failure modes. Expert-written or adversarial items can increase difficulty, but may represent a narrower or deliberately unusual distribution. Convenience samples are easy to collect but may omit important cases. Recent research reviewing 445 large-language-model benchmarks emphasized representative sampling, item-quality checks, and explicit coverage boundaries as central to construct validity.[1]

Annotations can include class labels, acceptable answers, rationales, preferences, unit tests, or environment states. Their reliability depends on the instructions, annotator qualifications, adjudication process, and treatment of ambiguous cases. A benchmark that discards disagreement may hide genuine ambiguity, while one that treats every annotator vote as equally authoritative may add noise. Where a task has multiple valid outputs, a single reference answer can understate performance unless the metric or adjudication process recognizes alternatives.

### Evaluation protocol

The protocol determines how a system is run. It includes preprocessing, prompt templates, examples shown in context, decoding settings, random seeds, tool access, retrieval sources, time and compute budgets, retry rules, and any post-processing or answer parser. For an agent, the protocol also includes the environment version, initial state, available interfaces, action limits, and reset procedure.

These details are part of the measurement. A model can receive different scores from semantically equivalent prompt formats, different few-shot examples, or different decoding temperatures. Tool access can change the object being measured from unaided recall to information-seeking or execution. Allowing many attempts measures a different operating point from allowing one attempt. Results should therefore identify the exact model version and configuration rather than attaching a score only to a product family.

### Scoring and aggregation

A scoring function maps an output or trajectory to an item-level result. The [metric](https://aiwiki.ai/wiki/metric) may be binary success, accuracy, F1, an overlap measure, execution success, a continuous quality score, a human preference, or a cost-adjusted quantity. Aggregation then combines items, tasks, annotators, or repeated runs. Macro averages give each group equal weight; micro averages weight observations; geometric means and minimum scores impose different trade-offs. A single aggregate can conceal important variation across domains and demographic or linguistic groups.

The direction, range, units, and threshold of every score should be explicit. If multiple metrics are reported, the benchmark should explain whether one is primary and whether trade-offs are expected. HELM, for example, was designed to evaluate common language-model scenarios with multiple measures including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency rather than treating accuracy as the only relevant dimension.[11]

### Reporting and comparison

A complete report includes per-task or subgroup results, not only a headline average. It should also include baselines, uncertainty estimates, evaluation dates, code and data versions, exclusions, failed runs, and the number of attempts. When results depend on a paid API or changing service, the access date and returned model identifier are important because later calls may not reproduce the same system.

The 2024 BetterBench study assessed lifecycle practices across a sample of AI benchmarks. It found large differences in quality and reported that most assessed benchmarks did not report statistical significance or make results easy to reproduce. Its checklist treats clear metric definitions, uncertainty reporting, licensing, maintenance, and feedback channels as parts of benchmark quality rather than optional presentation details.[2]

## Data partitions and information boundaries

### Training, development, and test sets

In a conventional supervised benchmark, a training split may be used to fit model parameters, a development or validation split to choose models and settings, and a [test set](https://aiwiki.ai/wiki/test_set) to estimate final performance. The test set should not participate in training or model selection if it is intended to estimate performance on unseen examples. If a team tunes repeatedly against test feedback, the test set gradually acts like development data.

Not every benchmark provides training data. Zero-shot and few-shot evaluations may release only examples or instructions. Agent benchmarks can supply an environment rather than a labeled training split. A benchmark should state what data may be used, whether external data or retrieval is allowed, and whether demonstrations from the evaluation distribution may appear in the prompt.

### Public and hidden tests

Public test items make inspection, error analysis, and independent implementation easier. They also make exact item exposure more likely, especially when training corpora are assembled from the public web. Hidden labels or hidden items reduce direct access, but do not guarantee validity. Items can still overlap with training sources, and repeated server queries can reveal information about a holdout.

Hidden evaluation is most useful when paired with submission limits, audited code execution, delayed or coarse feedback, and a final private test that was not used during development. Conversely, secrecy can make it difficult to identify annotation errors or demographic gaps. The appropriate design depends on the threat model, the desired reproducibility, and whether participants or model developers can be trusted with the test.

### Cross-validation and nested selection

When data are limited, cross-validation estimates performance by rotating held-out folds. Hyperparameters must be selected without using the same held-out observations that are later treated as an unbiased final evaluation. Cawley and Talbot showed that optimizing a noisy model-selection criterion can overfit that criterion, producing selection bias comparable in magnitude to reported differences between learning algorithms in some settings.[25] Nested cross-validation or a separate final test can separate selection from estimation.

The unit of splitting also matters. Randomly separating rows is insufficient when related observations share a patient, document, speaker, code repository, time period, or source template. Grouped or temporal splits are needed when the intended generalization target is a new group or future period. Near-duplicates across partitions can make an evaluation easier even when no labels were explicitly reused.

## Common metrics

No metric is universally appropriate. A metric must correspond to the intended property, respond predictably to meaningful changes, and avoid rewarding irrelevant behavior.

### Classification and retrieval

Accuracy is the fraction of items assigned the accepted label. It is interpretable when labels are reliable and classes or error costs are reasonably balanced. With severe imbalance, high accuracy can coexist with poor detection of a rare class. Precision, recall, and the [F1 score](https://aiwiki.ai/wiki/f1_score) separate different error types, but still depend on class definitions and thresholds.

Area-under-curve metrics summarize performance across thresholds. Calibration metrics ask whether stated probabilities correspond to observed frequencies. Ranking and retrieval tasks can use mean reciprocal rank, recall at a cutoff, or normalized discounted cumulative gain. Each measure embeds a view of which positions and errors matter. Reporting several complementary measures is often more informative than collapsing them prematurely.

### Text generation

[BLEU](https://aiwiki.ai/wiki/bleu_bilingual_evaluation_understudy) compares a candidate translation with reference translations using modified n-gram precision and a brevity penalty.[12] [ROUGE](https://aiwiki.ai/wiki/rouge_score) is a family of overlap-oriented measures developed for summary evaluation.[13] METEOR aligns candidate and reference words and can use exact matches, stems, and synonym resources; it is not simply another n-gram precision score.[14]

Reference-overlap metrics are reproducible and inexpensive, but a low-overlap output can still be valid and a high-overlap output can still be factually wrong. Their usefulness depends on the task, language, tokenization, reference set, and level of semantic variation. Human evaluation or task-specific factuality and utility checks may be needed for open-ended outputs.

Exact match is suitable when a canonical short answer can be normalized reliably. The normalization procedure is part of the metric: case folding, punctuation removal, unit conversion, numerical tolerances, and accepted aliases can change results. Automated parsers should be tested for false rejection and false acceptance, particularly when a model is required to emit JSON or another structured format.[1]

### Code and verifiable outputs

Execution-based metrics run generated code, proofs, database queries, or actions against an evaluator. HumanEval introduced hand-written programming problems and measured functional correctness with unit tests.[15] The associated [pass@k](https://aiwiki.ai/wiki/pass_at_k) family estimates how often at least one of several sampled solutions passes, so the value of k, the number of generated samples, and the sampling distribution must accompany the score.

Execution is a stronger criterion than text similarity for functional tasks, but it is only as complete as the evaluator. Inadequate unit tests can accept incorrect implementations, nondeterministic dependencies can change results, and unsafe code requires isolation. Efficiency, security, maintainability, and conformance to the issue intent may remain unmeasured even when tests pass.

### Human preference and pairwise comparison

Open-ended assistants are often compared through blinded pairwise choices. Chatbot Arena presented anonymous responses from two models and aggregated crowdsourced preferences, with the 2024 paper analyzing more than 240,000 votes collected at that stage.[21] Pairwise data can be converted into ratings or estimated win probabilities, but the statistical model, sampling policy, treatment of ties, user population, and confidence intervals all affect the ranking.

Preference is not a single objective fact. Raters can disagree because they value correctness, style, safety, concision, or creativity differently. Prompts supplied by voluntary users may not represent a deployment population. A preference leaderboard should document its rater and prompt distribution and should avoid interpreting a small rating gap as a universal capability difference.

### Automated judges

A language model can grade or compare open-ended responses using a rubric. This reduces cost and supports large-scale experiments, but introduces another model into the measurement chain. The NeurIPS 2023 study of LLM judges documented position and verbosity biases and reasoning limitations, while also finding useful agreement with human preferences under the tested conditions. It examined possible self-enhancement effects but said its limited data and small differences did not establish that bias.[22]

Judge evaluations should identify the judge version, prompt, rubric, sampling settings, reference information, and tie policy. Order swapping can test position sensitivity. Human auditing can estimate agreement and reveal systematic errors. A judge should not be treated as ground truth merely because its scores are numerical or reproducible.

### Agent success, reliability, and cost

Interactive agents are often scored by final state or task success. GAIA combines questions that may require reasoning, multimodal inputs, browsing, and tools.[17] WebArena provides functional websites and checks whether web tasks are completed.[18] [OSWorld](https://aiwiki.ai/wiki/osworld) defines computer tasks with initial-state configurations and execution-based evaluators across desktop applications.[19] Tau-bench evaluates tool-using agents in policy-constrained conversations by comparing the final database state with an annotated goal and introduced a repeated-trial reliability measure.[20]

An agent score is inseparable from its environment and resource budget. Browser versions, site data, credentials, network access, latency, tool errors, and reset behavior can affect success. Results should report token use, tool calls, elapsed time, monetary cost, retries, and failure categories when those quantities matter. A system that succeeds once after many attempts answers a different question from one that succeeds consistently under a fixed budget.

## Uncertainty and statistical comparison

A benchmark score is an estimate, even when the scoring program is deterministic. Uncertainty can come from the finite item sample, random training, stochastic inference, annotators, changing environments, and model or service updates. A difference in displayed decimals is not automatically a meaningful difference.

For an item-level mean, bootstrap intervals can resample the appropriate independent unit. Paired methods are usually preferable when the same items are evaluated by both systems because they preserve the item-level relationship. If examples are grouped, the resampling unit should reflect that grouping. For stochastic systems, repeated runs can estimate run-to-run variability, while repeated training seeds address a different source of variation.

Comparing many systems, tasks, or metrics increases the chance of finding an apparent advantage by chance. The analysis plan should distinguish exploratory from confirmatory comparisons and account for multiplicity where formal inference is claimed. Demsar reviewed nonparametric methods for comparing classifiers over multiple datasets and recommended paired tests for two methods and Friedman-style procedures with post-hoc comparisons for multiple methods in that setting.[31] These recommendations do not replace task-specific statistical judgment, but illustrate why a leaderboard ordering alone is not an uncertainty analysis.

Practical significance also matters. A statistically detectable improvement on a very large test may be too small to affect users, while a wide interval on a rare safety outcome may conceal a deployment-relevant risk. Benchmark reports should pair estimates with intervals, effect sizes, costs, and domain-specific consequences.

## Historical development

### From task datasets to community challenges

Early machine-learning comparisons often used shared datasets such as [MNIST](https://aiwiki.ai/wiki/mnist), which accompanied work on gradient-based handwritten document recognition.[4] Such datasets lowered the cost of comparing algorithms, but a fixed set of images and labels left many choices, including preprocessing and model selection, to individual studies.

Large community challenges made the protocol more explicit. The ImageNet Large Scale Visual Recognition Challenge evaluated classification and detection over hundreds of categories and millions of images, with annual competitions beginning in 2010.[5] The 2012 deep convolutional network by Krizhevsky, Sutskever, and Hinton reported substantially lower ImageNet classification error than previous approaches and became an influential demonstration of GPU-trained [deep learning](https://aiwiki.ai/wiki/deep_learning).[6] The durable historical point is the change in methods and results under a common challenge, not a claim that one score alone created an entire research era.

### Multi-task language evaluation

The [GLUE benchmark](https://aiwiki.ai/wiki/glue_benchmark) combined nine natural-language-understanding tasks, a diagnostic dataset, and an online evaluation platform. Four component tasks used private test data.[7] Rapid gains reduced the available headroom, and SuperGLUE followed with a more difficult task set, toolkit, and leaderboard.[8] This sequence illustrates both the value and the limited lifespan of a fixed suite: a benchmark can organize comparison and later cease to discriminate among strong systems.

[MMLU](https://aiwiki.ai/wiki/mmlu) broadened multiple-choice evaluation to 57 academic and professional subjects in zero-shot and few-shot settings.[9] [BIG-Bench](https://aiwiki.ai/wiki/big_bench) assembled 204 diverse tasks contributed by a large collaboration and studied performance across model scales.[10] HELM shifted emphasis from one aggregate toward standardized, multi-scenario and multi-metric evaluation, while publishing prompts and completions for analysis.[11]

These suites do not measure identical constructs. MMLU emphasizes performance on subject questions, BIG-Bench samples many researcher-contributed tasks, and HELM is an evaluation framework organized around scenarios and desiderata. Reporting them together can reveal complementary behavior, but averaging arbitrary suites does not create a theory-free measure of general intelligence.

### Interactive and application-level benchmarks

As models became components of assistants and agents, benchmarks began to include repositories, browsers, tools, and operating systems. SWE-bench consists of software-engineering problems derived from real GitHub issues and corresponding pull requests, evaluated in repository environments.[16] GAIA asks questions that may require information seeking and tool use.[17] WebArena and OSWorld evaluate sequences of actions in reproducible web or desktop environments.[18][19]

These evaluations move closer to some applications, but they also add sources of variance and new design choices. A task can fail because of perception, planning, tool syntax, environment state, policy interpretation, or an evaluator bug. Meaningful progress therefore requires trajectory analysis and component diagnostics in addition to a final success rate.

## Representative benchmark families

| Family | Typical item or environment | Common scoring approach | Main interpretation limit |
| --- | --- | --- | --- |
| Image classification | Labeled image | Top-1 or top-k accuracy | Sensitive to label set and image distribution |
| Language classification | Sentence or passage with label | Accuracy, F1, correlation | Format and dataset artifacts can substitute for intended understanding |
| Knowledge and exam questions | Multiple-choice or short answer | Accuracy or exact match | Mixes knowledge, language, reasoning, and test-taking strategy |
| Generation | Prompt plus one or more references | Overlap metric, rubric, human rating | Many valid outputs and reference incompleteness |
| Code generation | Specification plus executable tests | Unit-test success or pass@k | Tests may not cover all functional and nonfunctional requirements |
| Software engineering | Repository state plus issue | Patch accepted by tests | Environment construction and test quality constrain validity |
| Human preference | Two blinded outputs | Win rate or fitted rating | Depends on prompts, raters, and rating model |
| Tool-using assistant | Question plus tools | Exact answer, state match, or rubric | Tool access and budget change the evaluated system |
| Web or desktop agent | Initial environment plus goal | Final-state or execution success | Environment drift and long-horizon error compounding |
| Safety or fairness probe | Scenario, demographic variation, or adversarial prompt | Rate, disparity, severity, or rubric | Coverage is necessarily incomplete and context dependent |

The table describes common patterns, not a ranking of benchmark quality. A narrow benchmark can be highly valid for a precise claim, while a broad suite can be weak evidence for an undefined general claim.

## Validity, reliability, and coverage

### Construct validity

Construct validity asks whether a score supports the intended interpretation. Bean and colleagues describe a chain from phenomenon to task to metric to claim. A failure at any link weakens the conclusion.[1] For example, exact-match arithmetic questions might support a claim about correct answers under a specified prompt, but a claim about general mathematical reasoning would also require evidence that recall, parsing, and answer formatting are not driving the result.

Evidence for validity can include expert analysis of the task space, correlations with other measures of the same phenomenon, weak correlations with unrelated properties, prediction of performance in a relevant setting, and controlled tests of plausible confounders. No single check proves validity for every use. The strength of a claim should match the strength and scope of the evidence.

### Reliability

Reliability concerns the stability of measurement. A test-retest design can show whether a score changes when the same system is evaluated again. Inter-rater agreement measures consistency among annotators or judges. Internal-consistency measures address whether items behave as a coherent scale, although a diverse benchmark need not represent a single latent trait.

A measure can be reliable without being valid. A deterministic answer key may reproduce the same score while measuring an unintended shortcut. Conversely, a valuable open-ended property can be difficult to measure reliably. Benchmark design must address both rather than treating repeatability as proof of meaning.

### Coverage and subgroup performance

An overall score describes the mixture of cases selected by the benchmark. Changing that mixture changes the estimand. Results should be disaggregated when performance across languages, domains, difficulty levels, demographic groups, modalities, or task types is relevant. A [fairness metric](https://aiwiki.ai/wiki/fairness_metric) can expose a disparity, but different fairness definitions answer different questions and may conflict.

Coverage statements should be concrete. "Multilingual" can mean two languages or many, and performance in translated exam questions may not represent naturally occurring tasks. "Multimodal" can refer to a narrow image-question format without covering audio, video, spatial interaction, or accessibility contexts. Missing populations should be reported rather than silently generalized over.

## Failure modes

### Data contamination

[Data contamination](https://aiwiki.ai/wiki/data_contamination) occurs when information from evaluation items reaches training or development in a way that compromises the intended test. Exact test-item inclusion is the clearest case, but near-duplicates, answer explanations, benchmark repositories, translated variants, and synthetic reconstructions can also transfer useful information. Sainz and colleagues distinguish levels of contamination and note that its extent is difficult to measure for large language models trained on broad, partly undisclosed corpora.[24]

High performance is not by itself evidence of contamination, and a failed string-matching detector does not prove cleanliness. Evidence can come from training-data records, temporal cutoffs, canary strings, membership or exposure tests, performance on paraphrases, and comparison with genuinely post-training items. Each method has false positives and false negatives. Reports should state what was checked and what remains unknown.

### Adaptive reuse and leaderboard overfitting

Repeated use of a holdout can make development decisions increasingly dependent on it. Blum and Hardt formalized the adaptive leaderboard problem and proposed the Ladder mechanism to limit the information released while retaining useful rankings.[26] The risk is strongest when participants can make many submissions and receive detailed feedback.

The effect is not identical in every benchmark. Feldman, Frostig, and Hardt found theoretical and empirical reasons why multiclass settings can reduce some attacks and noted limited evidence of severe reuse overfitting in popular multiclass benchmarks at that time.[27] Fresh-test studies also separate adaptivity from distribution shift. A new ImageNet test set produced substantial absolute accuracy drops while largely preserving relative ordering, and its authors attributed the drops to differences in image difficulty rather than adaptive overfitting.[28] A SQuAD replication found no evidence of adaptive overfitting on a fresh same-domain set but found larger drops under natural shifts to other text domains.[29]

The appropriate conclusion is not that reused tests are always invalid or always safe. Submission history, feedback precision, number of classes, item count, model-development process, and distribution shift all matter.

### Benchmark selection and the benchmark lottery

Researchers can choose among tasks, datasets, preprocessing rules, metrics, seeds, and aggregation schemes. If only favorable combinations are reported, the selected result overstates the typical advantage. The "benchmark lottery" study demonstrated that relative algorithm rankings could change substantially with the selected tasks and argued that benchmark choice directs what the community treats as progress.[30]

Pre-registration, complete reporting, common evaluation harnesses, and sensitivity analysis can reduce selective presentation. A claimed method improvement is stronger when it appears across justified tasks and configurations rather than on one convenient suite.

### Shortcuts, artifacts, and confounding

Models may exploit correlations that predict labels without performing the intended task. Examples include annotation patterns, answer-position imbalance, image backgrounds, lexical overlap, source metadata, and rigid templates. A benchmark can be difficult while still rewarding a shortcut, and adversarial difficulty does not automatically imply better construct validity.

Counterfactual examples, perturbation tests, artifact baselines, and error analysis can reveal some shortcuts. However, a perturbation must preserve the intended label and task. If it changes what a reasonable system should do, lower performance does not demonstrate a flaw.

### Saturation

A benchmark saturates when scores cluster near a ceiling or otherwise lose the ability to distinguish systems for its intended use. Saturation can arise from genuine progress, item exposure, narrow coverage, annotation noise, easy items, or a coarse metric. It does not mean the broader problem is solved.

Bowman and Dahl argued that useful language-understanding benchmarks need robust in-domain validity, accurate and unambiguous annotations, adequate statistical power, and attention to harmful social biases.[33] A 2026 study of 60 language-model benchmarks defined and analyzed saturation, finding nearly half of its sample saturated and saturation increasing with age. In its analysis, expert-curated benchmarks were more resistant at comparable ages, while hidden test data alone did not show protection from saturation.[36] These are findings about the study's sample, not a universal law for every benchmark.

### Distribution shift

Benchmark data are collected at a time and place under particular procedures. Deployment data can differ in content, users, sensors, policies, or environment state. Same-distribution test performance estimates performance under a narrower assumption than robustness to natural or adversarial shift.

Fresh and shifted test sets help distinguish these questions. The ImageNet and SQuAD replication studies show why a performance drop must be interpreted with the new distribution in view rather than automatically labeled test overfitting.[28][29] Temporal, geographic, institutional, or source-based splits can better represent some deployment shifts, but no finite suite covers all future conditions.

### Annotation and evaluator error

Labels can be wrong, ambiguous, outdated, or based on contested expertise. Unit tests can be incomplete. State checkers can miss side effects. Human raters can be inconsistent, and automated judges can introduce systematic biases. As models approach the evaluator's error rate, additional score gains may reflect fitting the evaluator rather than improving the underlying property.

Item audits, duplicate annotation, expert adjudication, public error reporting, and versioned corrections improve quality. Corrections should preserve result provenance: changing an answer key or removing an item creates a new benchmark version, so old and new scores should not be mixed without recomputation.

### Aggregation and rank instability

An aggregate score embeds weights. Equal task weighting, equal item weighting, and equal domain weighting can yield different rankings. Normalizing against a human score, random baseline, or previous state of the art introduces further assumptions. Missing results can create survivorship bias if only tasks a system can run are averaged.

Reports should publish the component scores and formula so readers can recompute justified alternatives. Rank uncertainty should be shown where possible. A leaderboard should allow ties or uncertainty bands when the data do not distinguish adjacent systems.

### Judge and preference bias

Human and automated preference evaluations can be affected by response length, order, style, familiarity, and rater identity. The LLM-judge study found that swapping answer order changed some judgments and that repetitive lengthening could attract higher ratings from some judges.[22] Such findings motivate order randomization, blinded identity, calibrated rubrics, rater-quality checks, and human audit samples.

Crowdsourced voting systems also need security controls. A 2025 ICML study showed in offline experiments on historical Chatbot Arena votes that targeted vote manipulation could change rankings with a relatively small number of rigged votes, highlighting the need for abuse detection and robust aggregation.[23] This does not imply that every observed leaderboard result was manipulated; it identifies an attack surface that operators must address.

## Designing and maintaining a benchmark

### Specify the claim before collecting data

A benchmark should begin with a decision or scientific question, not with a convenient dataset. Designers should define the target phenomenon, relevant population, unit of analysis, expected sources of variation, and acceptable uses. A claim about a specific task can be supported by a narrow test. A broad capability claim requires representative coverage and evidence that nuisance factors are controlled.

The benchmark should include negative scope statements. For example, a code benchmark may measure whether generated patches pass provided tests but not whether they are secure or maintainable. An agent benchmark may measure success in a sandbox but not resilience to malicious websites. Clear boundaries prevent a precise result from being inflated into an unsupported conclusion.

### Build and audit the item set

Data collection should document provenance, consent or licensing, sampling, filtering, deduplication, and annotation. Items should be checked for solvability, leakage, ambiguity, and harmful content. Baseline systems can reveal trivial artifacts, while domain experts can assess whether items represent the intended work.

Datasheets for datasets propose structured documentation of motivation, composition, collection, preprocessing, uses, distribution, and maintenance.[34] These records help benchmark users identify mismatches between a dataset and their application. Documentation does not cure a flawed sample, but it makes limitations inspectable.

### Freeze the protocol and evaluator

Before comparison, designers should freeze prompts, preprocessing, decoding, allowed tools, resource limits, score code, and dependency versions. Evaluators should be tested on known positive, negative, and edge cases. For execution benchmarks, containers or reproducible environments reduce variation, while logs make failures auditable.

If a benchmark intentionally changes over time, it should define the update mechanism. Dynamic evaluation can collect new examples against current systems; Dynabench demonstrated a human-and-model-in-the-loop approach in which annotators seek examples that fool a target model while remaining valid to another person.[32] Dynamic designs reduce some forms of staleness but can shift the sampled distribution and create round-to-round comparability problems. Versioned rounds and anchor items can help.

### Establish baselines and uncertainty

Baselines can include random, majority, retrieval, heuristic, earlier model, and human performance where each is meaningful. Human baselines need the same instructions and access conditions as models if they are used for direct comparison. A non-expert speed-limited human estimate is not a universal human ceiling.

The evaluation plan should set the number of items and repeated runs using the precision needed for the intended comparison. Confidence intervals, paired differences, subgroup estimates, and power calculations should be considered before expensive model runs. Reporting all planned conditions reduces the temptation to select favorable variants.

### Release, govern, and version

A public release should include a license, data card or equivalent documentation, evaluator code, environment specification, baseline outputs, known issues, and a contact channel. Private-test benchmarks can still publish the protocol and item-generation process while protecting exact test content.

Maintenance includes reviewing error reports, checking code against updated dependencies, monitoring leakage, and deciding when to retire or refresh items. Every change should receive a version. A leaderboard should bind each submission to benchmark version, model version, protocol, date, and compute or tool budget.

### Document evaluated systems

A [model card](https://aiwiki.ai/wiki/model_card) can record intended uses, limitations, training context, and performance across relevant groups and conditions.[35] Benchmark results are most useful when connected to such system documentation. A score without a model identifier, training cutoff, prompt, or tool configuration is difficult to reproduce and easy to misuse.

## Improving benchmark resilience

No single intervention solves all failure modes. Several complementary designs are common:

- **Hidden final tests** limit direct item access, especially when submission feedback is restricted.
- **Temporal tests** evaluate on material created after a documented model or data cutoff, though later reuse can eventually contaminate them.
- **Dynamic collection** adds items over rounds and can target current failure modes.[32]
- **Fresh replications** recreate the collection process to test whether results transfer to new samples.[28][29]
- **Adversarial or contrast sets** probe known sensitivities, provided the transformed items remain valid.
- **Execution-based scoring** checks observable behavior for code, tools, or environments, but depends on evaluator completeness.[15][16][19]
- **Multi-metric reporting** exposes accuracy, robustness, fairness, efficiency, and other trade-offs instead of hiding them in one number.[11]
- **Submission controls** reduce adaptive leakage from a leaderboard.[26]
- **Independent audits** examine items, code, annotations, and security assumptions.

The choice should follow the threat model. A hidden static exam may protect answers but fail to represent an application. A dynamic benchmark may stay difficult while changing what its score means. An execution benchmark may verify final state while missing unsafe intermediate actions. Resilience is therefore a lifecycle property, not a label guaranteed by one feature.

## How to interpret benchmark results

A responsible comparison asks:

1. What exact system, configuration, and access conditions were evaluated?
2. What phenomenon and population does the benchmark claim to represent?
3. How were items sampled, annotated, split, and versioned?
4. Could the model or its developers have seen the items or detailed feedback?
5. Does the metric reward the behavior that matters and penalize important failures?
6. Are component and subgroup results available?
7. What uncertainty surrounds the difference between systems?
8. Were compute, tools, retries, and cost held comparable?
9. Has the evaluator been audited for errors, bias, or incomplete coverage?
10. Does performance transfer to a relevant independent or deployment-like test?

These questions are more informative than asking which benchmark is "hardest." Difficulty depends on the evaluated systems, protocol, metric, and date. A benchmark can be hard because it tests a valuable capability, because its labels are noisy, because its instructions are obscure, or because its distribution is deliberately adversarial. Low scores alone do not distinguish those explanations.

Similarly, a first-place rank is not a complete assessment. The gap may be within uncertainty, driven by one task, obtained with a larger sampling budget, or irrelevant to a target application. For broad language-model comparisons, the dedicated [LLM evaluation](https://aiwiki.ai/wiki/llm_evaluation) process should combine multiple benchmarks, application tests, qualitative analysis, and resource reporting. For interactive systems, [agent evaluation](https://aiwiki.ai/wiki/agent_evaluation) should additionally examine trajectories, environment assumptions, reliability, and side effects.

## Leaderboards and evaluation infrastructure

A leaderboard is a reporting interface, not a benchmark by itself. It can improve transparency by preserving submissions and configurations, but it can also encourage optimization for rank, conceal uncertainty, and amplify small or incomparable differences. The [Leaderboard Illusion](https://aiwiki.ai/wiki/the_leaderboard_illusion) describes broader concerns about interpreting public rankings without their evaluation context.

An evaluation harness can standardize data loading, prompts, inference, and scoring across models. The [LM Evaluation Harness](https://aiwiki.ai/wiki/lm_evaluation_harness) is one example for language-model tasks. Standardization reduces accidental variation but does not make the included tasks valid for every claim. Harness versions, task implementations, and model adapters must be recorded because a bug fix or prompt change can alter scores.

Leaderboards are strongest when they:

- require reproducible configuration and model-version disclosure;
- separate benchmark versions and evaluation tracks;
- show intervals, ties, and component scores;
- distinguish public, audited, and self-reported submissions;
- publish resource budgets and tool access;
- enforce rate limits and monitor abuse;
- retain corrected and superseded results with provenance;
- provide a process for item and evaluator error reports.

## Benchmarking as an industry

Alongside its scientific role, benchmarking developed into a commercial industry during the mid-2020s. Evaluation moved beyond free academic datasets and volunteer-run leaderboards into paid products, as model developers, enterprise buyers, and investors began to pay for independent measurement, and several evaluation projects converted into venture-funded companies.

[LMArena](https://aiwiki.ai/wiki/lmarena_org), the operator of Chatbot Arena, is the most prominent example. The platform began in 2023 as a research project run primarily by UC Berkeley-affiliated researchers and was incorporated as a company in April 2025. In May 2025 it raised a US$100 million seed round co-led by Andreessen Horowitz and UC Investments at a reported US$600 million valuation.[37] In January 2026 it announced a US$150 million Series A at a post-money valuation of US$1.7 billion, reporting more than 5 million monthly users across 150 countries and disclosing that a paid evaluation product launched in September 2025 had exceeded a US$30 million annualized consumption run rate by December 2025.[38] In June 2026 the company, by then rebranded as Arena, said the product had reached a US$100 million annual revenue run rate eight months after launch and that it was extending its evaluation work to long-running agents.[39]

Other companies sell evaluation from different starting points. [Scale AI](https://aiwiki.ai/wiki/scale_ai), primarily a training-data company, launched its [SEAL leaderboards](https://aiwiki.ai/wiki/seal_leaderboards) in May 2024, ranking frontier models on private, held-out prompt sets written and graded by vetted domain experts so that test items cannot be incorporated into training corpora.[40] [Artificial Analysis](https://aiwiki.ai/wiki/artificial_analysis), founded in 2023, runs its own standardized evaluations of models and API providers rather than republishing vendor-reported scores. TechCrunch reported in April 2025, citing the firm's data, that evaluating OpenAI's o1 across a suite of seven benchmarks cost US$2,767.05, against US$108.85 for the non-reasoning GPT-4o, and that the firm had spent roughly US$5,200 evaluating about a dozen reasoning models, nearly twice what it spent on more than 80 non-reasoning models.[41] Rising per-model costs of this kind are themselves a commercial force, because they make independent verification of published scores harder for academics to afford and easier to buy than to reproduce.[41]

The largest sums have gone to companies supplying expert humans rather than test items. [Mercor](https://aiwiki.ai/wiki/mercor), a marketplace that connects AI laboratories with scientists, physicians, lawyers, and other domain experts who train and evaluate models, raised a US$350 million Series C in October 2025 at a US$10 billion valuation, five times the US$2 billion valuation from its Series B earlier that year.[42] A common thread across these businesses is paid evaluation by humans: Arena aggregates crowdsourced preference votes, Scale grades with vetted experts, and Mercor supplies expert evaluators, reflecting demand for testing models against people rather than only against fixed test sets.[39][40][42]

Commercialization has sharpened questions about evaluator independence. The nonprofit [Epoch AI](https://aiwiki.ai/wiki/epoch_ai) disclosed in December 2024 that OpenAI had funded the creation of its FrontierMath benchmark and had visibility into many of its problems and solutions; contributors criticized the late disclosure, and Epoch acknowledged it had made a mistake, saying OpenAI had verbally agreed not to train on the problems and that a separate holdout set supported independent verification.[43] In June 2025 Meta paid approximately US$14.3 billion for a 49 percent stake in Scale AI, valuing it at US$29 billion; Scale founder Alexandr Wang left to join Meta, and OpenAI and Google subsequently said they would stop working with the data provider.[44][45] [The Leaderboard Illusion](https://aiwiki.ai/wiki/the_leaderboard_illusion), a 2025 paper analyzing Chatbot Arena, argued that undisclosed private testing, selective score disclosure, and asymmetric data access had produced "a distorted playing field" rewarding "overfitting to Arena-specific dynamics rather than general model quality".[46] LMArena disputed several of the paper's figures in a public response on its blog.[47]

## See also

- [Goodhart's law](https://aiwiki.ai/wiki/goodharts_law)
- [HELM](https://aiwiki.ai/wiki/helm)
- [HumanEval](https://aiwiki.ai/wiki/humaneval)
- [Chatbot Arena](https://aiwiki.ai/wiki/lmsys_chatbot_arena)
- [LMArena](https://aiwiki.ai/wiki/lmarena_org)
- [Epoch AI](https://aiwiki.ai/wiki/epoch_ai)
- [GAIA benchmark](https://aiwiki.ai/wiki/gaia_benchmark)

## References

1. Bean, A. M., Kearns, R. O., Romanou, A., et al. (2025). "Measuring what Matters: Construct Validity in Large Language Model Benchmarks." *Advances in Neural Information Processing Systems 38, Datasets and Benchmarks Track*. [NeurIPS proceedings](https://papers.neurips.cc/paper_files/paper/2025/hash/1967e0fc3aa6cbbace562f5cb8e3954e-Abstract-Datasets_and_Benchmarks_Track.html)
2. Reuel, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M., & Kochenderfer, M. J. (2024). "BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices." *Advances in Neural Information Processing Systems 37, Datasets and Benchmarks Track*. [NeurIPS proceedings](https://proceedings.neurips.cc/paper_files/paper/2024/hash/26889e8359e7ef8a7f5d77457364ca55-Abstract-Datasets_and_Benchmarks_Track.html)
3. Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021). "AI and the Everything in the Whole Wide World Benchmark." *Advances in Neural Information Processing Systems 34, Datasets and Benchmarks Track*. [https://arxiv.org/abs/2111.15366](https://arxiv.org/abs/2111.15366)
4. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). "Gradient-Based Learning Applied to Document Recognition." *Proceedings of the IEEE*, 86(11), 2278-2324. [Author-hosted record](https://bottou.org/papers/lecun-98h)
5. Russakovsky, O., Deng, J., Su, H., et al. (2015). "ImageNet Large Scale Visual Recognition Challenge." *International Journal of Computer Vision*, 115, 211-252. [https://arxiv.org/abs/1409.0575](https://arxiv.org/abs/1409.0575)
6. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). "ImageNet Classification with Deep Convolutional Neural Networks." *Advances in Neural Information Processing Systems 25*. [NeurIPS proceedings](https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html)
7. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018). "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding." *Proceedings of the 2018 EMNLP Workshop BlackboxNLP*, 353-355. [ACL Anthology](https://aclanthology.org/W18-5446/)
8. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019). "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems." *Advances in Neural Information Processing Systems 32*. [NeurIPS proceedings](https://papers.nips.cc/paper_files/paper/2019/hash/4496bf24afe7fab6f046bf4923da8de6-Abstract.html)
9. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). "Measuring Massive Multitask Language Understanding." *International Conference on Learning Representations*. [OpenReview](https://openreview.net/forum?id=d7KBjmI3GmQ)
10. Srivastava, A., Rastogi, A., Rao, A., et al. (2022). "Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models." *Transactions on Machine Learning Research*. [https://arxiv.org/abs/2206.04615](https://arxiv.org/abs/2206.04615)
11. Liang, P., Bommasani, R., Lee, T., et al. (2023). "Holistic Evaluation of Language Models." *Transactions on Machine Learning Research*. [https://arxiv.org/abs/2211.09110](https://arxiv.org/abs/2211.09110)
12. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). "Bleu: a Method for Automatic Evaluation of Machine Translation." *Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics*, 311-318. [ACL Anthology](https://aclanthology.org/P02-1040/)
13. Lin, C.-Y. (2004). "ROUGE: A Package for Automatic Evaluation of Summaries." *Text Summarization Branches Out*, 74-81. [ACL Anthology](https://aclanthology.org/W04-1013/)
14. Banerjee, S., & Lavie, A. (2005). "METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments." *Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization*, 65-72. [ACL Anthology](https://aclanthology.org/W05-0909/)
15. Chen, M., Tworek, J., Jun, H., et al. (2021). "Evaluating Large Language Models Trained on Code." arXiv:2107.03374. [https://arxiv.org/abs/2107.03374](https://arxiv.org/abs/2107.03374)
16. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" *International Conference on Learning Representations*. [ICLR proceedings](https://proceedings.iclr.cc/paper_files/paper/2024/hash/edac78c3e300629acfe6cbe9ca88fb84-Abstract-Conference.html)
17. Mialon, G., Fourrier, C., Wolf, T., LeCun, Y., & Scialom, T. (2024). "GAIA: a benchmark for General AI Assistants." *International Conference on Learning Representations*. [ICLR proceedings](https://proceedings.iclr.cc/paper_files/paper/2024/hash/25ae35b5b1738d80f1f03a8713e405ec-Abstract-Conference.html)
18. Zhou, S., Xu, F. F., Zhu, H., et al. (2024). "WebArena: A Realistic Web Environment for Building Autonomous Agents." *International Conference on Learning Representations*. [https://arxiv.org/abs/2307.13854](https://arxiv.org/abs/2307.13854)
19. Xie, T., Zhang, D., Chen, J., et al. (2024). "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments." *Advances in Neural Information Processing Systems 37*. [https://arxiv.org/abs/2404.07972](https://arxiv.org/abs/2404.07972)
20. Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2025). "Tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." *International Conference on Learning Representations*. [ICLR proceedings](https://proceedings.iclr.cc/paper_files/paper/2025/hash/1b126cc38b8638e07bef37e7b2bb72bf-Abstract-Conference.html)
21. Chiang, W.-L., Zheng, L., Sheng, Y., et al. (2024). "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference." *Proceedings of the 41st International Conference on Machine Learning*, 8359-8388. [PMLR](https://proceedings.mlr.press/v235/chiang24b.html)
22. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." *Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track*. [NeurIPS proceedings](https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html)
23. Min, R., Pang, T., Du, C., Liu, Q., Cheng, M., & Lin, M. (2025). "Improving Your Model Ranking on Chatbot Arena by Vote Rigging." *Proceedings of the 42nd International Conference on Machine Learning*, 44252-44271. [PMLR](https://proceedings.mlr.press/v267/min25a.html)
24. Sainz, O., Campos, J., Garcia-Ferrero, I., Etxaniz, J., Lopez de Lacalle, O., & Agirre, E. (2023). "NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark." *Findings of the Association for Computational Linguistics: EMNLP 2023*, 10776-10787. [ACL Anthology](https://aclanthology.org/2023.findings-emnlp.722/)
25. Cawley, G. C., & Talbot, N. L. C. (2010). "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation." *Journal of Machine Learning Research*, 11, 2079-2107. [JMLR](https://jmlr.org/papers/v11/cawley10a.html)
26. Blum, A., & Hardt, M. (2015). "The Ladder: A Reliable Leaderboard for Machine Learning Competitions." *Proceedings of the 32nd International Conference on Machine Learning*, 1006-1014. [PMLR](https://proceedings.mlr.press/v37/blum15.html)
27. Feldman, V., Frostig, R., & Hardt, M. (2019). "The Advantages of Multiple Classes for Reducing Overfitting from Test Set Reuse." *Proceedings of the 36th International Conference on Machine Learning*, 1892-1900. [PMLR](https://proceedings.mlr.press/v97/feldman19a.html)
28. Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019). "Do ImageNet Classifiers Generalize to ImageNet?" *Proceedings of the 36th International Conference on Machine Learning*, 5389-5400. [PMLR](https://proceedings.mlr.press/v97/recht19a.html)
29. Miller, J., Krauth, K., Recht, B., & Schmidt, L. (2020). "The Effect of Natural Distribution Shift on Question Answering Models." *Proceedings of the 37th International Conference on Machine Learning*, 6905-6916. [PMLR](https://proceedings.mlr.press/v119/miller20a.html)
30. Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., & Vinyals, O. (2021). "The Benchmark Lottery." arXiv:2107.07002. [https://arxiv.org/abs/2107.07002](https://arxiv.org/abs/2107.07002)
31. Demsar, J. (2006). "Statistical Comparisons of Classifiers over Multiple Data Sets." *Journal of Machine Learning Research*, 7, 1-30. [JMLR](https://www.jmlr.org/papers/v7/demsar06a.html)
32. Kiela, D., Bartolo, M., Nie, Y., et al. (2021). "Dynabench: Rethinking Benchmarking in NLP." *Proceedings of NAACL-HLT 2021*, 4110-4124. [ACL Anthology](https://aclanthology.org/2021.naacl-main.324/)
33. Bowman, S. R., & Dahl, G. (2021). "What Will it Take to Fix Benchmarking in Natural Language Understanding?" *Proceedings of NAACL-HLT 2021*, 4843-4855. [ACL Anthology](https://aclanthology.org/2021.naacl-main.385/)
34. Gebru, T., Morgenstern, J., Vecchione, B., et al. (2021). "Datasheets for Datasets." *Communications of the ACM*, 64(12), 86-92. [https://arxiv.org/abs/1803.09010](https://arxiv.org/abs/1803.09010)
35. Mitchell, M., Wu, S., Zaldivar, A., et al. (2019). "Model Cards for Model Reporting." *Proceedings of the Conference on Fairness, Accountability, and Transparency*, 220-229. [https://arxiv.org/abs/1810.03993](https://arxiv.org/abs/1810.03993)
36. Akhtar, M., Reuel, A., Soni, P., et al. (2026). "When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation." Accepted at the 43rd International Conference on Machine Learning (ICML 2026). [https://arxiv.org/abs/2602.16763](https://arxiv.org/abs/2602.16763)
37. TechCrunch (Zeff, M.). "LM Arena, the organization behind popular AI leaderboards, lands $100M." May 21, 2025. [https://techcrunch.com/2025/05/21/lm-arena-the-organization-behind-popular-ai-leaderboards-lands-100m/](https://techcrunch.com/2025/05/21/lm-arena-the-organization-behind-popular-ai-leaderboards-lands-100m/)
38. LMArena. "LMArena Raises $150 Million to Build the World's Most Trusted AI Evaluation Platform." PR Newswire, January 6, 2026. [https://www.prnewswire.com/news-releases/lmarena-raises-150-million-to-build-the-worlds-most-trusted-ai-evaluation-platform-302653012.html](https://www.prnewswire.com/news-releases/lmarena-raises-150-million-to-build-the-worlds-most-trusted-ai-evaluation-platform-302653012.html)
39. Arena (official X account). "Arena reached a $100M annual revenue run rate just 8 months after launching our evaluation product." June 29, 2026. [https://x.com/arena/status/2071630464583151727](https://x.com/arena/status/2071630464583151727)
40. Scale AI. "Scale's SEAL Leaderboards." May 29, 2024. [https://scale.com/blog/leaderboard](https://scale.com/blog/leaderboard)
41. TechCrunch. "The rise of AI 'reasoning' models is making benchmarking more expensive." April 10, 2025. [https://techcrunch.com/2025/04/10/the-rise-of-ai-reasoning-models-is-making-benchmarking-more-expensive/](https://techcrunch.com/2025/04/10/the-rise-of-ai-reasoning-models-is-making-benchmarking-more-expensive/)
42. TechCrunch. "Mercor quintuples valuation to $10B with $350M Series C." October 27, 2025. [https://techcrunch.com/2025/10/27/mercor-quintuples-valuation-to-10b-with-350m-series-c/](https://techcrunch.com/2025/10/27/mercor-quintuples-valuation-to-10b-with-350m-series-c/)
43. TechCrunch. "AI benchmarking organization criticized for waiting to disclose funding from OpenAI." January 19, 2025. [https://techcrunch.com/2025/01/19/ai-benchmarking-organization-criticized-for-waiting-to-disclose-funding-from-openai/](https://techcrunch.com/2025/01/19/ai-benchmarking-organization-criticized-for-waiting-to-disclose-funding-from-openai/)
44. TechCrunch. "Scale AI confirms 'significant' investment from Meta, says CEO Alexandr Wang is leaving." June 13, 2025. [https://techcrunch.com/2025/06/13/scale-ai-confirms-significant-investment-from-meta-says-ceo-alexandr-wang-is-leaving/](https://techcrunch.com/2025/06/13/scale-ai-confirms-significant-investment-from-meta-says-ceo-alexandr-wang-is-leaving/)
45. TechCrunch. "Cracks are forming in Meta's partnership with Scale AI." August 29, 2025. [https://techcrunch.com/2025/08/29/cracks-are-forming-in-metas-partnership-with-scale-ai/](https://techcrunch.com/2025/08/29/cracks-are-forming-in-metas-partnership-with-scale-ai/)
46. Singh, S., Nan, Y., Wang, A., et al. (2025). "The Leaderboard Illusion." arXiv:2504.20879. [https://arxiv.org/abs/2504.20879](https://arxiv.org/abs/2504.20879)
47. LMArena. (2025). "Our Response to 'The Leaderboard Illusion' Writeup." Arena blog. [https://arena.ai/blog/our-response](https://arena.ai/blog/our-response)