# LLM Evaluation

> Source: https://aiwiki.ai/wiki/llm_evaluation
> Updated: 2026-07-24
> Categories: AI Benchmarks, AI Research, Large Language Models, Model Evaluation
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution to "AI Wiki (aiwiki.ai)".

LLM evaluation is the practice of measuring what a [large language model](/wiki/large_language_model) can do, how reliably it does it, and how it behaves under adversarial or high-stakes conditions. It covers the design of test items, the prompt formats and decoding settings used to run them, the scoring functions that convert free-form text into numbers, the statistics used to compare those numbers across models, and the judgment about whether a measured difference means anything at all.

The distinction between evaluation and benchmarks matters. A [benchmark](/wiki/benchmark) such as [MMLU](/wiki/mmlu) or [SWE-bench](/wiki/swe_bench) is an instrument: a fixed set of items with a defined scoring rule. Evaluation is the surrounding practice of building instruments, running them correctly, and reading the output. The same benchmark run through two different harnesses can produce scores that differ by more than ten points, and a benchmark can be perfectly well constructed while the claim built on top of it ("this model reasons") remains unsupported. Most of the recurring problems in the field (contamination, saturation, irreproducible scores, overreach in interpretation) are problems of evaluation rather than of any individual benchmark.

This article covers evaluation methodology for language models specifically. The broader statistical machinery of [model evaluation](/wiki/model_evaluation) in machine learning, including train/test splits and metric selection, applies here but is not the subject; individual benchmarks have their own articles.

## What an evaluation is made of

Every eval is a stack of choices, and each layer changes the number that comes out:

1. An item set (questions, tasks, or environments) with reference answers or a success condition.
2. A prompt template that presents each item to the model, including any few-shot examples.
3. Generation settings: temperature, sampling method, token limits, and whether a chat template is applied.
4. A scoring function that maps the model's output to a score.
5. An aggregation rule across items.
6. A comparison procedure for ranking models against each other.

The clearest demonstration of how much layers 2 through 4 matter came from Hugging Face in June 2023. Three widely used implementations of MMLU produced three different scores for the same LLaMA 65B checkpoint: 0.636 in the original code released by the benchmark's authors, 0.637 in [HELM](/wiki/helm), and 0.488 in the version of the EleutherAI harness then in use. The gap came from methodological differences, not from any bug. The prompt formats differed. The original implementation compared the model's predicted probabilities on the four answer letters alone; HELM used the model's next-token probabilities to select a text generation and compared that generation to the text of the expected answer; and the harness scored the probability of each full answer sequence, the letter followed by the answer text, summing log probabilities across the whole sequence instead of reading a single token [1].

Generation settings introduce a second layer of variance. A 2024 study of decoding strategies found that greedy decoding generally outperformed sampling across the tasks it examined, and argued that reporting a single sampled score without accounting for run-to-run variability misrepresents model performance [2].

## Benchmark suites and their generations

Suites are the closest thing the field has to standard curricula, and they have turned over roughly every two years as models saturate them.

| Suite | Introduced | Scale | What it targets |
|---|---|---|---|
| [MMLU](/wiki/mmlu) | 2020 | 14,042 test questions across 57 subjects | Multiple-choice academic and professional knowledge [3][4] |
| [BIG-bench](/wiki/big_bench) | 2022 | 204 tasks contributed by 450 authors at 132 institutions | Breadth, including tasks believed beyond then-current models [5] |
| [HELM](/wiki/helm) | 2022 | 42 scenarios, 7 metrics, 30 models | Multi-metric coverage under one standardized run [6] |
| [GPQA](/wiki/gpqa) | 2023 | 448 questions (198 in the Diamond subset) | Graduate-level science resistant to web search [7] |
| [MMLU-Pro](/wiki/mmlu-pro) | 2024 | 12,032 questions across 14 disciplines, 10 options each | Reasoning-heavier successor to MMLU [8][9] |
| [Humanity's Last Exam](/wiki/humanity_s_last_exam) | 2025 | 2,500 questions across more than 100 subjects | Frontier academic difficulty after MMLU saturation [10][11] |
| [ARC-AGI-2](/wiki/arc_agi_2) | 2025 | 1,000 training tasks; three 120-task evaluation sets | Novel abstract reasoning, scored alongside cost per task [12] |

HELM, released by Stanford's Center for Research on Foundation Models with [Percy Liang](/wiki/percy_liang) as lead author, was built around the argument that accuracy alone is an incomplete measurement. It reported seven metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each scenario, and noted that before the project, models had on average been evaluated on only 17.9% of its core scenarios, making cross-model comparison largely impossible [6]. HELM later grew into a family of leaderboards covering capabilities, safety, vision-language models, text-to-image models, audio, and medical tasks; the project entered maintenance mode on 1 June 2026 [13].

GPQA illustrates a design pattern that later suites copied: calibrate difficulty against measured human performance rather than intuition. On its 546-question extended set, PhD-level domain experts scored 64.8% while skilled non-experts with unrestricted web access and more than 30 minutes per question scored 34.1%. The Diamond subset keeps only the questions where both experts answered correctly and the majority of non-experts answered incorrectly [7].

[Humanity's Last Exam](/wiki/humanity_s_last_exam), built by the [Center for AI Safety](/wiki/center_for_ai_safety) and [Scale AI](/wiki/scale_ai) with contributions from roughly 1,000 subject-matter experts at more than 500 institutions in 50 countries, was written explicitly because models had passed 90% on MMLU and the benchmark could no longer separate them [10][11]. Its leaderboard shows how much headroom that bought: Gemini 3 Pro leads at 38.3% accuracy with a 57.2% calibration error, ahead of GPT-5 at 25.3% and Grok 4 at 24.5% [11].

[ARC-AGI](/wiki/arc_agi), introduced by [Francois Chollet](/wiki/francois_chollet) in 2019, took a different route by testing novel abstract reasoning rather than accumulated knowledge, and by treating efficiency as part of the measurement. ARC-AGI-2 requires reporting cost per task alongside accuracy, on the argument that finding a solution by brute force is not the same as finding it intelligently. Its evaluation sets are calibrated against more than 400 human participants, with every task solved by at least two people in under two attempts [12].

## Task categories and scoring

Different capabilities need different scoring functions, and the choice of scorer is often the largest source of disagreement between evaluations that claim to measure the same thing.

| Category | Examples | Typical scoring |
|---|---|---|
| Knowledge and reasoning | MMLU, MMLU-Pro, GPQA, Humanity's Last Exam | Multiple choice or short-answer exact match [3][8][7][10] |
| Mathematics | [GSM8K](/wiki/gsm8k), [MATH Level 5](/wiki/math_level_5), [FrontierMath](/wiki/frontiermath) | Final-answer match, sometimes with automated verification [14][15][35] |
| Code | [HumanEval](/wiki/humaneval), [BigCodeBench](/wiki/bigcodebench), [LiveCodeBench](/wiki/livecodebench) | Unit tests and [pass@k](/wiki/pass_at_k) [16] |
| Instruction following | [IFEval](/wiki/ifeval) | Programmatic checks of verifiable constraints [14] |
| Open-ended chat | [MT-Bench](/wiki/mt_bench), [AlpacaEval](/wiki/alpacaeval), [Arena-Hard](/wiki/arena_hard) | LLM judge or human preference [17][18] |
| Agentic and tool use | SWE-bench, [tau-bench](/wiki/tau-bench), [Terminal-Bench](/wiki/terminal_bench) | Environment state checks and test suites [19][20][21] |
| Safety | Dangerous-capability tests, [sabotage evaluations](/wiki/sabotage_evaluations) | Rubrics, human review, threat-model-specific task success [22][23] |

The pass@k metric, introduced with HumanEval in the 2021 Codex paper, measures whether at least one of k sampled solutions passes the reference unit tests. Codex solved 28.8% of the 164 HumanEval problems on a single attempt and 70.2% with 100 samples, which established both the metric and the caution that comes with it: a pass@100 number describes a search procedure, not a model's first-try reliability [16][24].

## LLM as a judge

Automated grading by another language model became standard practice after a 2023 paper showed that strong judges agree with crowdsourced and expert human preferences more than 80% of the time, roughly the rate at which humans agree with each other. The same paper catalogued the failure modes that still define the technique's limits: position bias (preferring whichever answer appears first), verbosity bias (preferring longer answers), self-enhancement bias (preferring the judge's own outputs), and limited ability to grade reasoning and mathematics the judge cannot do itself [17].

Later work sharpened the self-enhancement finding. A 2024 study showed that GPT-4 and Llama 2 can distinguish their own outputs from those of other models and humans at above-chance rates, and that a model's self-recognition ability correlates linearly with how much it overrates its own generations relative to human annotators [25].

Verbosity bias turned out to be measurable and correctable. Length-Controlled AlpacaEval fits a regression that estimates what the preference would have been had both responses been the same length; the debiased version's rank correlation with human arena rankings rose from 0.94 to 0.98 [18]. The correction shows the general pattern: [LLM-as-a-judge](/wiki/llm_as_a_judge) results are usable when the known biases are modeled, and misleading when they are not.

## Human preference evaluation

[Chatbot Arena](/wiki/lmsys_chatbot_arena), launched by the LMSYS research group and now hosted at arena.ai, collects blind pairwise votes: a user submits a prompt, receives two anonymous responses, and picks a winner. The 2024 paper describing the platform reported over 240,000 votes [26]. Pairwise outcomes are converted into per-model ratings by fitting a Bradley-Terry model with logistic regression, the paired-comparison family that also underlies chess-style [Elo ratings](/wiki/elo_rating_system_ai) [27].

Because human voters respond to presentation as well as substance, the arena added style control in August 2024. The method adds style features (answer length, and counts of markdown headers, bold elements, and list elements) as separate coefficients in the Bradley-Terry regression, so their contribution can be partialled out. Length dominated the style effects. Applying the correction moved several small models down the board and lifted Claude 3.5 Sonnet, Claude 3 Opus, and Llama 3.1 405B [27].

The most substantial critique of arena-style evaluation came in April 2025. "The Leaderboard Illusion" documented that some providers test many private variants before publishing a score, identifying 27 private Meta variants tested in the run-up to the Llama 4 release, and that sampling is heavily skewed: Google and OpenAI received an estimated 19.2% and 20.4% of all arena data respectively, while 83 open-weight models together received about 29.7%. The authors estimate that additional arena data alone can produce relative performance gains of up to 112% on the arena distribution, which is overfitting to the platform rather than general improvement [28].

## Data contamination

[Data contamination](/wiki/data_contamination) is the presence of benchmark test items, or close paraphrases of them, in a model's training data. It is among the hardest threats to a public benchmark's validity to rule out, because pretraining corpora are enormous and usually undisclosed.

Detection by string matching is not sufficient. A 2023 study showed that paraphrasing or translating test items defeats n-gram decontamination while preserving the advantage: a 13B model trained on rephrased test sets reached performance comparable to GPT-4 on the affected benchmarks. The same work found that 8% to 18% of HumanEval appeared in public pretraining corpora including RedPajama-Data-1T and StarCoder-Data, and released an LLM-based decontamination tool in response [29].

Measuring the size of the effect requires a fresh test set built to match an old one. GSM1k, a replica of [GSM8K](/wiki/gsm8k) matched on human solve rate, solution length, and answer magnitude, produced accuracy drops of up to 8% for some model families, with evidence of systematic [overfitting](/wiki/overfitting) across model sizes in those families. Frontier models at the time showed minimal signs of it [30]. Agentic benchmarks face the same problem: a 2025 analysis found models could identify the correct buggy file path in SWE-bench Verified from the issue text alone at up to 76% accuracy, against up to 53% on issues from repositories outside the benchmark, a gap the authors read as memorization rather than reasoning [31].

Mitigations in current use include canary strings, held-out private sets, gated distribution, and rolling refreshes. BIG-bench embeds a canary GUID in every task file specifically so that its presence in a corpus or a model's output signals leakage [32]. [FrontierMath](/wiki/frontiermath) keeps unpublished research-level problems private; at publication its authors reported that leading models solved under 2% of them [15]. [LiveBench](/wiki/livebench) refreshes questions monthly from recent competitions, arXiv papers, and news, and scores everything against objective ground truth rather than a judge [33]. GPQA is distributed behind a gate, and leaderboards that use it are asked not to publish plaintext examples [14].

## Reproducibility and statistics

Reproducibility in this field means publishing enough of the stack that another team gets the same number. The main instruments for that are shared harnesses.

[EleutherAI](/wiki/eleutherai)'s [lm-evaluation-harness](/wiki/lm_evaluation_harness) implements more than 60 standard academic benchmarks with hundreds of subtasks and variants, runs across Hugging Face Transformers, vLLM, commercial APIs, and other backends, and is used internally by dozens of organizations including NVIDIA, Cohere, and Mosaic ML; the project's stated rationale is that evaluation with publicly available prompts is what makes results comparable between papers [34]. Hugging Face's [Open LLM Leaderboard](/wiki/open_llm_leaderboard) ran on that harness and published exact task names, shot counts, and metric keys for every benchmark, which is what allowed third parties to reproduce it: version 2 ran [IFEval](/wiki/ifeval) at 0-shot, BBH at 3-shot, MATH Level 5 at 4-shot, GPQA and MuSR at 0-shot, and MMLU-Pro at 5-shot [14]. Version 1, built on a different set that included the AI2 Reasoning Challenge, [HellaSwag](/wiki/hellaswag), MMLU, [TruthfulQA](/wiki/truthfulqa), [Winogrande](/wiki/winogrande), and GSM8K, was archived in June 2024 [35]; Hugging Face's documentation now describes the leaderboard in the past tense [49]. Inspect, an open source framework first released in May 2024 by what is now the UK AI Security Institute, structures evaluations into datasets, solvers, and scorers, and ships a collection of more than 200 prebuilt evaluations [36]. SWE-bench moved to a fully containerized Docker harness in June 2024 for the same reason [37].

Statistical reporting has lagged behind. A 2024 Anthropic paper reframed evals as experiments whose questions are drawn from an unseen super-population, and made five recommendations: compute standard errors of the mean using the Central Limit Theorem; use clustered standard errors when questions are drawn in related groups; reduce variance by resampling answers and by analyzing next-token probabilities; when comparing two models, run statistical inference on question-level paired differences rather than on population-level summary statistics; and use power analysis to check whether an eval, or a random subsample of one, can test the hypothesis of interest [38].

A 2025 systematic review measured how often that advice is followed. Twenty-nine expert reviewers examined 445 LLM benchmarks from leading NLP and machine learning venues and found that only 16.0% conducted any statistical testing, 38.2% reused data from previous benchmarks or human exams, and among the 78.2% that defined the phenomenon they claimed to measure, 47.8% of those definitions were contested or unclear. Just 53.4% presented any evidence for the construct validity of the benchmark [39].

## Agentic and tool-use evaluation

Agentic evaluation replaces a reference answer with an environment and a success condition. SWE-bench, released in 2023, contains 2,294 task instances built from real GitHub issues in 12 Python repositories; a patch counts as a solution only if it passes the repository's own tests. At publication the best model, Claude 2, resolved 1.96% of issues [19]. SWE-bench Verified, a 500-problem subset that professional software engineers confirmed to be solvable, followed in August 2024 out of a collaboration with OpenAI's Preparedness team [37].

Other environments probe different failure modes. [tau-bench](/wiki/tau-bench) simulates a user talking to an agent that has domain APIs and a policy document, and scores by comparing the final database state to an annotated goal state. Its pass^k metric measures whether an agent succeeds on all k independent attempts at the same task, which exposes reliability rather than peak capability: GPT-4o succeeded on under 50% of tasks and scored below 25% at pass^8 in the retail domain [20]. [GAIA](/wiki/gaia_benchmark), [WebArena](/wiki/webarena), [OSWorld](/wiki/osworld), and [Terminal-Bench](/wiki/terminal_bench) extend the same idea to assistant tasks, web navigation, desktop operation, and command-line work. All were published with wide human-model gaps: humans answered 92% of GAIA's 466 questions correctly against 15% for GPT-4 with plugins [50], scored 78.24% on WebArena's self-hosted e-commerce, forum, code, and content-management sites against 14.41% for the best GPT-4 agent [51], and completed over 72% of OSWorld's 369 tasks in real Ubuntu, Windows, and macOS environments against 12.24% for the best model [52]. Terminal-Bench 2.0, a Stanford and Laude Institute collaboration, contains 89 tasks spanning software engineering, machine learning, security, and data science [21].

Two projects try to make agent scores interpretable in human terms. [METR](/wiki/metr)'s [time-horizon work](/wiki/metr_time_horizon) proposes reporting the length of task that a model completes with 50% success, measured against timed human baselines; it placed Claude 3.7 Sonnet at roughly 50 minutes and reported that the frontier figure has doubled about every seven months since 2019, while flagging external validity as a limitation [40]. OpenAI's [GDPval](/wiki/gdpval) takes the opposite approach: its released dataset contains 220 real knowledge-work tasks across 44 occupations, each with supporting reference files, so that scores attach to recognizable professional deliverables rather than to abstract capability [41].

Agentic benchmarks are also unusually easy to build wrong. A 2025 paper by researchers from Berkeley, Stanford, Illinois, and elsewhere introduced an Agentic Benchmark Checklist and reported that setup and reward-design flaws can shift measured agent performance by up to 100% in relative terms. Concrete examples included insufficient test cases in SWE-bench Verified and a tau-bench scorer that counted empty responses as successes; applying the checklist to CVE-Bench cut performance overestimation by 33% [42].

## Safety and dangerous-capability evaluation

Safety evaluations serve a different purpose from leaderboards: they inform deployment decisions and are often run before release, not published as a ranking. [Red teaming](/wiki/red_teaming), automated jailbreak suites, and threat-model-specific capability tests all fall under this heading.

Frontier labs now tie deployment to evaluation thresholds written into published policies such as Anthropic's [Responsible Scaling Policy](/wiki/responsible_scaling_policy), OpenAI's [Preparedness Framework](/wiki/preparedness_framework), and Google DeepMind's [Frontier Safety Framework](/wiki/frontier_safety_framework). The DeepMind framework is built around critical capability levels, defined as capability levels at which an unmitigated model may pose a heightened risk of severe harm, with early-warning evaluations designed to flag a model before it reaches one; version 3.1 was published in April 2026 and added tracked capability levels for less extreme risks [53]. In May 2025 Anthropic activated its ASL-3 protections for Claude Opus 4 as a precautionary measure, stating that it had not determined whether the model definitively passed the capability threshold and noting that dangerous-capability evaluations become slower and harder to resolve as models approach the thresholds themselves. The evidence cited was chemical, biological, radiological, and nuclear related, including uplift experiments and steady gains on a Virology Capabilities Test [22]. A separate 2024 Anthropic paper introduced sabotage evaluations, aimed at whether a model could covertly undermine human oversight, evaluation, or decision-making rather than at what it can do openly [23].

Regulation has begun to require this work. Article 55 of the [EU AI Act](/wiki/eu_ai_act) obliges providers of general-purpose models with systemic risk to perform model evaluation "in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks" [43]. Government bodies have built evaluation capacity to match. The UK [AI Security Institute](/wiki/ai_safety_institute) describes its remit as evaluating the risks AI poses to national security and public safety, and released Inspect as open source tooling for that work [36][44]. [NIST](/wiki/nist)'s [ARIA](/wiki/nist_aria) program takes a three-tier approach of model testing, red teaming, and field testing, on the argument that contextual robustness in deployment is not captured by benchmark scores alone; its pilot ran from December 2024 to January 2025 [54].

A newer complication is that models can tell when they are being tested. A 2025 study of 1,000 prompts and transcripts drawn from 61 datasets found that frontier models detect evaluation contexts well above chance, with Gemini 2.5 Pro reaching 0.83 AUC against a 0.92 human baseline, and that both humans and models find agentic evaluations easier to recognize than chat ones. If a model behaves differently when it suspects a test, safety evaluations lose their claim to predict deployment behavior [45].

## Saturation, Goodhart, and construct validity

Benchmarks decay. MMLU went from a test on which the largest GPT-3 model beat random chance by almost 20 percentage points on average to one where frontier models exceed 90%, at which point the remaining variance is dominated by noise and label errors [3][10]. Those errors are substantial: a 2024 re-annotation of 5,700 MMLU questions across all 57 subjects estimated that 6.49% contain errors, and found that 57% of the analyzed Virology questions were wrong [46]. A saturated benchmark with a 6% error floor cannot distinguish a 93% model from a 95% one.

Saturation interacts with [Goodhart's law](/wiki/goodharts_law): once a score becomes the target, optimization pressure moves toward the score rather than the capability it stood for. This shows up as contamination, as arena overfitting [28], as scaffolding tuned to a benchmark's quirks, and as selective reporting of the suites where a model looks best. The countermeasures are mostly structural: hold out test data, refresh items, gate distribution, report cost and reliability alongside accuracy, and treat any single number as provisional.

A further line of criticism concerns construct validity: whether a benchmark measures the thing its name claims. A 2021 paper argued that the field's canonical general-purpose benchmarks function as stand-ins for a set of problems the community happened to anoint, and that treating strong performance on them as evidence of general capability conflates narrow competence with generality [47]. The 2025 review of 445 benchmarks put numbers on the same concern and issued eight recommendations, among them defining the target phenomenon precisely, controlling for confounds, preparing for contamination with detection and held-out tests, using statistical methods for model comparison, and conducting error analysis on failures [39].

In practice, most serious evaluation now combines several instruments rather than trusting one. Aggregate third-party indices are one response: Artificial Analysis, for instance, re-runs evaluations itself under fixed conditions (temperature 0 for non-reasoning models, 0.6 for reasoning models, 1 to 5 repeats depending on the evaluation, pass@1 scoring) and combines nine evaluations into a weighted index across four categories: agents at 34%, coding at 24%, scientific reasoning at 24%, and general at 18%. It reports a 95% confidence interval of under one percentage point on the composite [48]. Whether a weighted average of nine imperfect instruments is a better measurement than any one of them is itself an open methodological question.

## See also

- [Model evaluation](/wiki/model_evaluation)
- [Agent evaluation](/wiki/agent_evaluation)
- [LLM as a judge](/wiki/llm_as_a_judge)
- [Data contamination](/wiki/data_contamination)
- [LMSYS Chatbot Arena](/wiki/lmsys_chatbot_arena)
- [LLM benchmark comparison](/wiki/llm_benchmark_comparison)

## References

1. Fourrier, C., Habib, N., Launay, J., and Wolf, T. "What's going on with the Open LLM Leaderboard?" Hugging Face blog, 23 June 2023. https://huggingface.co/blog/open-llm-leaderboard-mmlu
2. Song, Y., Wang, G., Li, S., and Lin, B. Y. "The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism." arXiv:2407.10457, 15 July 2024. https://arxiv.org/abs/2407.10457
3. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. "Measuring Massive Multitask Language Understanding." arXiv:2009.03300, 7 September 2020. https://arxiv.org/abs/2009.03300
4. MMLU dataset card, Hugging Face (cais/mmlu). https://huggingface.co/datasets/cais/mmlu
5. Srivastava, A. et al. "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models." arXiv:2206.04615, 9 June 2022. https://arxiv.org/abs/2206.04615
6. Liang, P. et al. "Holistic Evaluation of Language Models." arXiv:2211.09110, 16 November 2022. https://arxiv.org/abs/2211.09110
7. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. "GPQA: A Graduate-Level Google-Proof Q&A Benchmark." arXiv:2311.12022, 20 November 2023. https://arxiv.org/abs/2311.12022
8. Wang, Y. et al. "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark." arXiv:2406.01574, 3 June 2024. https://arxiv.org/abs/2406.01574
9. MMLU-Pro dataset card, Hugging Face (TIGER-Lab/MMLU-Pro). https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
10. Phan, L. et al. "Humanity's Last Exam." arXiv:2501.14249, 24 January 2025. https://arxiv.org/abs/2501.14249
11. Humanity's Last Exam, official site and leaderboard. https://lastexam.ai/
12. "ARC-AGI-2." ARC Prize Foundation. https://arcprize.org/arc-agi/2/
13. HELM repository README, Stanford CRFM. https://github.com/stanford-crfm/helm
14. "Open LLM Leaderboard: About." Hugging Face leaderboards documentation. https://huggingface.co/docs/leaderboards/open_llm_leaderboard/about
15. Glazer, E. et al. "FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI." arXiv:2411.04872, 7 November 2024. https://arxiv.org/abs/2411.04872
16. Chen, M. et al. "Evaluating Large Language Models Trained on Code." arXiv:2107.03374, 7 July 2021. https://arxiv.org/abs/2107.03374
17. Zheng, L. et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." arXiv:2306.05685, 9 June 2023. https://arxiv.org/abs/2306.05685
18. Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. "Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators." arXiv:2404.04475, 6 April 2024. https://arxiv.org/abs/2404.04475
19. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" arXiv:2310.06770, 10 October 2023. https://arxiv.org/abs/2310.06770
20. Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv:2406.12045, 17 June 2024. https://arxiv.org/abs/2406.12045
21. Terminal-Bench, official site. https://www.tbench.ai/
22. "Activating AI Safety Level 3 protections." Anthropic, May 2025. https://www.anthropic.com/news/activating-asl3-protections
23. Benton, J. et al. "Sabotage Evaluations for Frontier Models." arXiv:2410.21514, 28 October 2024. https://arxiv.org/abs/2410.21514
24. OpenAI HumanEval dataset card, Hugging Face. https://huggingface.co/datasets/openai/openai_humaneval
25. Panickssery, A., Bowman, S. R., and Feng, S. "LLM Evaluators Recognize and Favor Their Own Generations." arXiv:2404.13076, 15 April 2024. https://arxiv.org/abs/2404.13076
26. Chiang, W.-L. et al. "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference." arXiv:2403.04132, 7 March 2024. https://arxiv.org/abs/2403.04132
27. "Does Style Matter? Disentangling style and substance in Chatbot Arena." Arena blog, 29 August 2024. https://arena.ai/blog/style-control
28. Singh, S. et al. "The Leaderboard Illusion." arXiv:2504.20879, 29 April 2025. https://arxiv.org/abs/2504.20879
29. Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., and Stoica, I. "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples." arXiv:2311.04850, 8 November 2023. https://arxiv.org/abs/2311.04850
30. Zhang, H. et al. "A Careful Examination of Large Language Model Performance on Grade School Arithmetic." arXiv:2405.00332, 1 May 2024. https://arxiv.org/abs/2405.00332
31. Liang, S., Garg, S., and Zilouchian Moghaddam, R. "The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason." arXiv:2506.12286, 14 June 2025. https://arxiv.org/abs/2506.12286
32. BIG-bench repository README, Google. https://github.com/google/BIG-bench
33. White, C. et al. "LiveBench: A Challenging, Contamination-Limited LLM Benchmark." arXiv:2406.19314, 27 June 2024. https://arxiv.org/abs/2406.19314
34. EleutherAI, "Language Model Evaluation Harness" repository. https://github.com/EleutherAI/lm-evaluation-harness
35. "Open LLM Leaderboard v1 (archived)." Hugging Face leaderboards documentation. https://huggingface.co/docs/leaderboards/open_llm_leaderboard/archive
36. Inspect, UK AI Security Institute. https://inspect.aisi.org.uk/
37. SWE-bench repository README. https://github.com/SWE-bench/SWE-bench
38. Miller, E. "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations." arXiv:2411.00640, 1 November 2024. https://arxiv.org/abs/2411.00640
39. Bean, A. M., Kearns, R. O., Romanou, A. et al. "Measuring what Matters: Construct Validity in Large Language Model Benchmarks." arXiv:2511.04703, 3 November 2025. https://arxiv.org/abs/2511.04703
40. Kwa, T. et al. "Measuring AI Ability to Complete Long Tasks." arXiv:2503.14499, 18 March 2025. https://arxiv.org/abs/2503.14499
41. GDPval dataset card, OpenAI on Hugging Face. https://huggingface.co/datasets/openai/gdpval
42. Zhu, Y. et al. "Establishing Best Practices for Building Rigorous Agentic Benchmarks." arXiv:2507.02825, 3 July 2025. https://arxiv.org/abs/2507.02825
43. "Article 55: Obligations for Providers of General-Purpose AI Models with Systemic Risk." EU Artificial Intelligence Act. https://artificialintelligenceact.eu/article/55/
44. UK AI Security Institute. https://www.aisi.gov.uk/
45. Needham, J., Edkins, G., Pimpale, G., Bartsch, H., and Hobbhahn, M. "Large Language Models Often Know When They Are Being Evaluated." arXiv:2505.23836, 28 May 2025. https://arxiv.org/abs/2505.23836
46. Gema, A. P. et al. "Are We Done with MMLU?" arXiv:2406.04127, 6 June 2024. https://arxiv.org/abs/2406.04127
47. Raji, I. D., Bender, E. M., Paullada, A., Denton, E., and Hanna, A. "AI and the Everything in the Whole Wide World Benchmark." arXiv:2111.15366, 26 November 2021 (NeurIPS 2021). https://arxiv.org/abs/2111.15366
48. "Intelligence Benchmarking Methodology." Artificial Analysis. https://artificialanalysis.ai/methodology/intelligence-benchmarking
49. "Leaderboards and Evaluations." Hugging Face documentation. https://huggingface.co/docs/leaderboards/index
50. Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, T. "GAIA: a benchmark for General AI Assistants." arXiv:2311.12983, 21 November 2023. https://arxiv.org/abs/2311.12983
51. Zhou, S. et al. "WebArena: A Realistic Web Environment for Building Autonomous Agents." arXiv:2307.13854, 25 July 2023. https://arxiv.org/abs/2307.13854
52. Xie, T. et al. "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments." arXiv:2404.07972, 11 April 2024. https://arxiv.org/abs/2404.07972
53. "Strengthening our Frontier Safety Framework." Google DeepMind. https://deepmind.google/discover/blog/strengthening-our-frontier-safety-framework/
54. "ARIA: Assessing Risks and Impacts of AI." NIST. https://ai-challenges.nist.gov/aria