LLM Benchmarks Timeline: Difference between revisions

From AI Wiki
(Created page with "= LLM Benchmarks Timeline = A memorial to the benchmarks that defined—and were defeated by—AI progress === All Time === * [#2024 2024] * [#2023 2023] * [#pre-2023 Pre-2023] ---- == 2024 == === ARC-AGI (2019 - 2024) === ; Category : Reasoning ; Killed by : Saturation ; Details : Killed 1 month ago, Abstract reasoning challenge consisting of visual pattern completion tasks. Each task presents a sequence of abstract visual patterns and requires selecting the correct...")
 
No edit summary
 
(21 intermediate revisions by the same user not shown)
Line 1: Line 1:
= LLM Benchmarks Timeline =
{{see also|LLM Comparisons|LLM Rankings}}
A memorial to the benchmarks that defined—and were defeated by—AI progress
Timeline of [[benchmarks]] surpassed by [[large language models]] (LLMs).


=== All Time ===
==2024==
* [#2024 2024]
{| class="wikitable sortable"
* [#2023 2023]
|-
* [#pre-2023 Pre-2023]
! Benchmark
! Category
! Date Created
! Date Defeated
! Killed By
! Defeated By Model
! Original Score
! Final Score
! Details
! Links
|-
| [[ARC-AGI]]
| Reasoning
| 2019-11
| 2024-12
| Saturation
| [[O3]]
| Human Baseline: ~80%
| O3: 87.5%
| Abstract reasoning challenge with visual pattern completion tasks created by François Chollet.
| [https://arxiv.org/abs/1911.01547 Paper], [https://arcs-benchmark.org Website]
|-
| [[MATH]]
| Mathematics
| 2021-03
| 2024-09
| Saturation
| [[O1]]
| Average CS PhD: ~40%
| O1: 94.8%
| 12K challenging competition math problems from AMC/AIME, requiring complex multi-step reasoning.
| [https://arxiv.org/abs/2103.03874 Paper], [https://github.com/hendrycks/math GitHub]
|-
| [[BIG-Bench-Hard]]
| Multi-task
| 2022-10
| 2024-06
| Saturation
| [[Sonnet 3.5]]
| Average Human: 67.7%
| Sonnet 3.5: 93.1%
| A curated suite of 23 challenging tasks from BIG-Bench.
| [https://arxiv.org/abs/2210.09261 Paper], [https://github.com/suzgunmirac/BIG-Bench-Hard GitHub], [https://assets.anthropic.com/m/1cd9d098ac3e6467/original/Claude-3-Model-Card-October-Addendum.pdf Evidence]
|-
| [[HumanEval]]
| Coding
| 2021-07
| 2024-05
| Saturation
| [[GPT-4o]]
| Unspecified
| GPT-4o: 90.2%
| 164 Python programming problems testing coding abilities.
| [https://arxiv.org/abs/2107.03374 Paper], [https://github.com/openai/human-eval GitHub], [https://openai.com/index/hello-gpt-4o/ Evidence]
|-
| [[IFEval]]
| Instruction Following
| 2023-11
| 2024-03
| Saturation
| [[LLama 3.3 70B]]
| Unspecified
| LLama 3.3 70B: 92.1%
| Evaluation suite testing multi-step instruction-following capabilities.
| [https://arxiv.org/abs/2311.07911 Paper], [https://github.com/google-research/google-research/tree/master/instruction_following_eval GitHub], [https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD_VISION.md Evidence]
|}
==2023==
{| class="wikitable sortable"
|-
! Benchmark
! Category
! Date Created
! Date Defeated
! Killed By
! Defeated By Model
! Original Score
! Final Score
! Details
! Links
|-
| [[GSM8K]]
| Mathematics
| 2021-10
| 2023-11
| Saturation
| [[GPT-4]]
| Unspecified
| GPT-4: 92.0%
| 8.5K grade school math word problems requiring step-by-step solutions.
| [https://arxiv.org/abs/2110.14168 Paper], [https://github.com/openai/grade-school-math GitHub], [https://cdn.openai.com/papers/gpt-4.pdf Evidence]
|-
| [[Turing Test]]
| Conversation
| 1950-10
| 2023-03
| Saturation
| [[GPT-4]]
| Interrogator > 50%
| Interrogator 46%
| The original AI benchmark proposed by Alan Turing in 1950 (the "imitation game").
| [https://courses.cs.umbc.edu/471/papers/turing.pdf Paper], [https://arxiv.org/pdf/2405.08007 Evidence]
|-
| [[ARC (AI2)]]
| Reasoning
| 2018-03
| 2023-03
| Saturation
| [[GPT-4]]
| Unspecified
| GPT-4: 96.3%
| Grade-school multiple-choice reasoning tasks testing logical, spatial, temporal reasoning.
| [https://arxiv.org/abs/1803.05457 Paper], [https://leaderboard.allenai.org/arc/submissions/get-started Website], [https://cdn.openai.com/papers/gpt-4.pdf Evidence]
|-
| [[HellaSwag]]
| Common Sense
| 2019-05
| 2023-03
| Saturation
| [[GPT-4]]
| Human: 95.6%
| GPT-4: 95.3%
| Multiple-choice questions about everyday scenarios with adversarial filtering.
| [https://arxiv.org/abs/1905.07830 Paper], [https://rowanzellers.com/hellaswag/ Website], [https://cdn.openai.com/papers/gpt-4.pdf Evidence]
|-
| [[MMLU]]
| Knowledge
| 2020-09
| 2023-03
| Saturation
| [[GPT-4]]
| 95th pct Human: 87.0%
| GPT-4: 87.3%
| 57 subjects from real-world sources (professional exams) testing breadth and depth of knowledge.
| [https://arxiv.org/abs/2009.03300 Paper], [https://github.com/hendrycks/test GitHub], [https://cdn.openai.com/papers/gpt-4.pdf Evidence]
|-
| [[WinoGrande]]
| Common Sense
| 2019-07
| 2023-03
| Saturation
| [[GPT-4]]
| Human: 94%
| GPT-4: 87.5%
| Enhanced WSC with 44K problems testing common-sense pronoun resolution.
| [https://arxiv.org/abs/1907.10641 Paper], [https://winogrande.allenai.org/ Website], [https://cdn.openai.com/papers/gpt-4.pdf Evidence]
|}


----
==Pre-2023==
===2022===
{| class="wikitable sortable"
|-
! Benchmark
! Category
! Date Created
! Date Defeated
! Killed By Model
! Defeated By
! Original Score
! Final Score
! Details
! Links
|-
| [[BIG-Bench]]
| Multi-task
| 2021-06
| 2022-04
| Saturation
| [[Palm 540B]]
| Human: 49.8%
| Palm 540B: 61.4%
| 204 tasks spanning linguistics, math, common-sense reasoning, and more.
| [https://arxiv.org/abs/2206.04615 Paper], [https://github.com/google/BIG-bench GitHub], [https://arxiv.org/pdf/2204.02311 Evidence]
|}


== 2024 ==
===2019===
=== ARC-AGI (2019 - 2024) ===
{| class="wikitable sortable"
; Category
|-
: Reasoning
! Benchmark
; Killed by
! Category
: Saturation
! Date Created
; Details
! Date Defeated
: Killed 1 month ago, Abstract reasoning challenge consisting of visual pattern completion tasks. Each task presents a sequence of abstract visual patterns and requires selecting the correct completion. Created by François Chollet as part of a broader investigation into measuring intelligence. It was 5 years and 1 month old.
! Killed By
; Defeated by
! Defeated By Model
: O3
! Original Score
; Original Score
! Final Score
: Human Baseline: ~80%
! Details
; Final Score
! Links
: O3: 87.5%
|-
| [[SuperGLUE]]
| Language
| 2019-05
| 2019-10
| Saturation
| [[T5]]
| Human: 89.8%
| T5: 89.3%
| More challenging language understanding tasks (word sense, causal reasoning, RC).
| [https://arxiv.org/abs/1905.00537 Paper], [https://super.gluebenchmark.com/ Website]
|-
| [[WSC]]
| Common Sense
| 2012-05
| 2019-07
| Saturation
| [[ROBERTA (w SFT)]]
| Human: 96.5%
| ROBERTA (w SFT): 90.1%
| Carefully crafted sentence pairs with ambiguous pronoun references.
| [https://cdn.aaai.org/ocs/4492/4492-21843-1-PB.pdf Paper], [https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html Website]
|-
| [[GLUE]]
| Language
| 2018-05
| 2019-06
| Saturation
| [[XLNet]]
| Human: 87.1%
| XLNet: 88.4%
| Nine tasks for evaluating NLU (inference, paraphrase, similarity, etc.).
| [https://arxiv.org/abs/1804.07461 Paper], [https://gluebenchmark.com/ Website]
|-
| [[TriviaQA]]
| Knowledge
| 2017-05
| 2019-06
| Saturation
| [[SpanBERT]]
| Human: 79.7%
| SpanBERT: 83.6%
| 650K QA-evidence triples requiring cross-sentence reasoning.
| [https://arxiv.org/abs/1705.03551 Paper], [http://nlp.cs.washington.edu/triviaqa/ Website]
|-
| [[SQuAD v2.0]]
| Language
| 2018-05
| 2019-04
| Saturation
| [[BERT]]
| Human: 89.5%
| BERT: 89.5%
| Extension of SQuAD adding unanswerable questions.
| [https://arxiv.org/abs/1806.03822 Paper], [https://rajpurkar.github.io/SQuAD-explorer/ Website]
|-
| [[SQuAD]]
| Language
| 2016-05
| 2019-03
| Saturation
| [[BERT]]
| Human: 91.2%
| BERT: 93.2%
| 100,000+ QA tasks on Wikipedia articles.
| [https://arxiv.org/abs/1606.05250 Paper], [https://rajpurkar.github.io/SQuAD-explorer/ Website]
|}


=== MATH (2021 - 2024) ===
===2018===
; Category
{| class="wikitable sortable"
: Mathematics
|-
; Killed by
! Benchmark
: Saturation
! Category
; Details
! Date Created
: Killed 4 months ago, A dataset of 12K challenging competition mathematics problems from AMC, AIME, and other math competitions. Problems range from pre-algebra to olympiad-level and require complex multi-step reasoning. Each problem has a detailed solution that tests mathematical reasoning capabilities. It was 3 years and 6 months old.
! Date Defeated
; Defeated by
! Killed By
: O1
! Defeated By Model
; Original Score
! Original Score
: Average CS PhD: ~40%
! Final Score
; Final Score
! Details
: O1: 94.8%
! Links
|-
| [[SWAG]]
| Common Sense
| 2018-05
| 2018-10
| Saturation
| [[BERT]]
| Human: 88%
| BERT: 86%
| 113K multiple-choice questions about grounded situations (common sense “next step”).
| [https://arxiv.org/abs/1808.05326 Paper], [https://rowanzellers.com/swag/ Website]
|}


=== BIG-Bench-Hard (2022 - 2024) ===
==References==
; Category
[https://r0bk.github.io/killedbyllm/ website]
: Multi-task
[https://github.com/R0bk/killedbyllm github]
; Killed by
: Saturation
; Details
: Killed 7 months ago, A curated suite of 23 challenging tasks from BIG-Bench where language models initially performed below average human level. Selected to measure progress on particularly difficult capabilities. It was 1 year and 8 months old.
; Defeated by
: Sonnet 3.5
; Original Score
: Average Human: 67.7%
; Final Score
: Sonnet 3.5: 93.1%


=== HumanEval (2021 - 2024) ===
[[Category:Benchmarks]] [[Category:Timelines]] [[Category:Aggregate pages]]
; Category
: Coding
; Killed by
: Saturation
; Details
: Killed 8 months ago, A collection of 164 Python programming problems designed to test language models' coding abilities. Each problem includes a function signature, docstring, and unit tests. Models must generate complete, correct function implementations that pass all test cases. It was 2 years and 10 months old.
; Defeated by
: GPT-4o
; Original Score
: Unspecified
; Final Score
: GPT-4o: 90.2%
 
=== IFEval (2023 - 2024) ===
; Category
: Instruction Following
; Killed by
: Saturation
; Details
: Killed 10 months ago, A comprehensive evaluation suite testing instruction following capabilities across coding, math, roleplay, and other tasks. Measures ability to handle complex multi-step instructions and constraints. It was 4 months old.
; Defeated by
: LLama 3.3 70B
; Original Score
: Unspecified
; Final Score
: LLama 3.3 70B: 92.1%
 
----
 
== 2023 ==
=== GSM8K (2021 - 2023) ===
; Category
: Mathematics
; Killed by
: Saturation
; Details
: Killed 1 year ago, A collection of 8.5K grade school math word problems requiring step-by-step solutions. Problems test both numerical computation and natural language understanding through multi-step mathematical reasoning. It was 2 years and 1 month old.
; Defeated by
: GPT-4
; Original Score
: Unspecified
; Final Score
: GPT-4: 92.0%
 
=== Turing Test (1950 - 2023) ===
; Category
: Conversation
; Killed by
: Saturation
; Details
: Killed 1 year ago, The original AI benchmark proposed by Alan Turing in 1950. In this “imitation game,” a computer must convince human judges it is human through natural conversation. The test sparked decades of debate about machine intelligence and consciousness. It was 73 years and 5 months old.
; Defeated by
: GPT-4
; Original Score
: Interrogator > 50%
; Final Score
: Interrogator 46%
 
=== ARC (AI2) (2018 - 2023) ===
; Category
: Reasoning
; Killed by
: Saturation
; Details
: Killed 1 year ago, AI2 Reasoning Challenge (ARC) – A collection of grade-school level multiple-choice reasoning tasks testing logical deduction, spatial reasoning, and temporal reasoning. Each task requires applying abstract reasoning skills to solve multi-step problems. It was 5 years old.
; Defeated by
: GPT-4
; Original Score
: Unspecified
; Final Score
: GPT-4: 96.3%
 
=== HellaSwag (2019 - 2023) ===
; Category
: Common Sense
; Killed by
: Saturation
; Details
: Killed 1 year ago, A challenging dataset of multiple-choice questions about everyday scenarios. Uses adversarial filtering to test models' ability to understand and reason about real-world situations and their likely outcomes. It was 3 years and 10 months old.
; Defeated by
: GPT-4
; Original Score
: Human: 95.6%
; Final Score
: GPT-4: 95.3%
 
=== MMLU (2020 - 2023) ===
; Category
: Knowledge
; Killed by
: Saturation
; Details
: Killed 1 year ago, A comprehensive benchmark covering 57 subjects including mathematics, history, law, computer science, and more. Questions are drawn from real-world sources like professional exams to test both breadth and depth of knowledge across diverse academic domains. It was 2 years and 6 months old.
; Defeated by
: GPT-4
; Original Score
: 95th pct Human: 87.0%
; Final Score
: GPT-4: 87.3%
 
=== WinoGrande (2019 - 2023) ===
; Category
: Common Sense
; Killed by
: Saturation
; Details
: Killed 1 year ago, An enhanced version of WSC with 44K problems testing common-sense reasoning through pronoun resolution. Uses adversarial filtering to ensure problems require real-world understanding. It was 3 years and 8 months old.
; Defeated by
: GPT-4
; Original Score
: Human: 94%
; Final Score
: GPT-4: 87.5%
 
----
 
== Pre-2023 ==
=== 2022 ===
==== BIG-Bench (2021 - 2022) ====
; Category
: Multi-task
; Killed by
: Saturation
; Details
: Killed 2 years ago, A collaborative collection of 204 tasks spanning linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, and software development. Tests diverse capabilities of language models. It was 10 months old.
; Defeated by
: Palm 540B
; Original Score
: Human: 49.8%
; Final Score
: Palm 540B: 61.4%
 
=== 2019 ===
==== SuperGLUE (2019 - 2019) ====
; Category
: Language
; Killed by
: Saturation
; Details
: Killed 5 years ago, A collection of more challenging language understanding tasks including word sense disambiguation, causal reasoning, and reading comprehension. Designed as a more difficult successor to GLUE. It was 5 months old.
; Defeated by
: T5
; Original Score
: Human: 89.8%
; Final Score
: T5: 89.3%
 
==== WSC (2012 - 2019) ====
; Category
: Common Sense
; Killed by
: Saturation
; Details
: Killed 5 years ago, A collection of carefully crafted sentence pairs with ambiguous pronoun references that resolve differently based on small changes. Designed to test genuine language understanding over statistical patterns. It was 7 years and 3 months old.
; Defeated by
: ROBERTA (w SFT)
; Original Score
: Human: 96.5%
; Final Score
: ROBERTA (w SFT): 90.1%
 
==== GLUE (2018 - 2019) ====
; Category
: Language
; Killed by
: Saturation
; Details
: Killed 5 years ago, A collection of nine tasks for evaluating natural language understanding, including single-sentence tasks, similarity and paraphrase tasks, and inference tasks. The primary NLU benchmark before SuperGLUE. It was 1 year and 1 month old.
; Defeated by
: XLNet
; Original Score
: Human: 87.1%
; Final Score
: XLNet: 88.4%
 
==== TriviaQA (2017 - 2019) ====
; Category
: Knowledge
; Killed by
: Saturation
; Details
: Killed 5 years ago, A large-scale dataset of 650K question-answer-evidence triples authored by trivia enthusiasts. Requires cross-sentence reasoning and synthesis of information from multiple sources. It was 2 years and 1 month old.
; Defeated by
: SpanBERT
; Original Score
: Human: 79.7%
; Final Score
: SpanBERT: 83.6%
 
==== SQuAD v2.0 (2018 - 2019) ====
; Category
: Language
; Killed by
: Saturation
; Details
: Killed 5 years ago, An extension of SQuAD that adds unanswerable questions. Models must both answer questions when possible and determine when no answer is supported by the passage. It was 11 months old.
; Defeated by
: BERT
; Original Score
: Human: 89.5%
; Final Score
: BERT: 89.5%
 
==== SQuAD (2016 - 2019) ====
; Category
: Language
; Killed by
: Saturation
; Details
: Killed 5 years ago, A reading comprehension dataset of 100,000+ questions posed by crowdworkers on Wikipedia articles. Answers must be text segments from the corresponding reading passage. It was 2 years and 10 months old.
; Defeated by
: BERT
; Original Score
: Human: 91.2%
; Final Score
: BERT: 93.2%
 
=== 2018 ===
==== SWAG (2018 - 2018) ====
; Category
: Common Sense
; Killed by
: Saturation
; Details
: Killed 6 years ago, A dataset of 113K multiple choice questions about grounded situations. Given a partial description of a situation, models must predict what happens next from 4 choices using common sense reasoning. It was 5 months old.
; Defeated by
: BERT
; Original Score
: Human: 88%
; Final Score
: BERT: 86%

Latest revision as of 21:01, 13 January 2025

See also: LLM Comparisons and LLM Rankings

Timeline of benchmarks surpassed by large language models (LLMs).

2024

Benchmark Category Date Created Date Defeated Killed By Defeated By Model Original Score Final Score Details Links
ARC-AGI Reasoning 2019-11 2024-12 Saturation O3 Human Baseline: ~80% O3: 87.5% Abstract reasoning challenge with visual pattern completion tasks created by François Chollet. Paper, Website
MATH Mathematics 2021-03 2024-09 Saturation O1 Average CS PhD: ~40% O1: 94.8% 12K challenging competition math problems from AMC/AIME, requiring complex multi-step reasoning. Paper, GitHub
BIG-Bench-Hard Multi-task 2022-10 2024-06 Saturation Sonnet 3.5 Average Human: 67.7% Sonnet 3.5: 93.1% A curated suite of 23 challenging tasks from BIG-Bench. Paper, GitHub, Evidence
HumanEval Coding 2021-07 2024-05 Saturation GPT-4o Unspecified GPT-4o: 90.2% 164 Python programming problems testing coding abilities. Paper, GitHub, Evidence
IFEval Instruction Following 2023-11 2024-03 Saturation LLama 3.3 70B Unspecified LLama 3.3 70B: 92.1% Evaluation suite testing multi-step instruction-following capabilities. Paper, GitHub, Evidence

2023

Benchmark Category Date Created Date Defeated Killed By Defeated By Model Original Score Final Score Details Links
GSM8K Mathematics 2021-10 2023-11 Saturation GPT-4 Unspecified GPT-4: 92.0% 8.5K grade school math word problems requiring step-by-step solutions. Paper, GitHub, Evidence
Turing Test Conversation 1950-10 2023-03 Saturation GPT-4 Interrogator > 50% Interrogator 46% The original AI benchmark proposed by Alan Turing in 1950 (the "imitation game"). Paper, Evidence
ARC (AI2) Reasoning 2018-03 2023-03 Saturation GPT-4 Unspecified GPT-4: 96.3% Grade-school multiple-choice reasoning tasks testing logical, spatial, temporal reasoning. Paper, Website, Evidence
HellaSwag Common Sense 2019-05 2023-03 Saturation GPT-4 Human: 95.6% GPT-4: 95.3% Multiple-choice questions about everyday scenarios with adversarial filtering. Paper, Website, Evidence
MMLU Knowledge 2020-09 2023-03 Saturation GPT-4 95th pct Human: 87.0% GPT-4: 87.3% 57 subjects from real-world sources (professional exams) testing breadth and depth of knowledge. Paper, GitHub, Evidence
WinoGrande Common Sense 2019-07 2023-03 Saturation GPT-4 Human: 94% GPT-4: 87.5% Enhanced WSC with 44K problems testing common-sense pronoun resolution. Paper, Website, Evidence

Pre-2023

2022

Benchmark Category Date Created Date Defeated Killed By Model Defeated By Original Score Final Score Details Links
BIG-Bench Multi-task 2021-06 2022-04 Saturation Palm 540B Human: 49.8% Palm 540B: 61.4% 204 tasks spanning linguistics, math, common-sense reasoning, and more. Paper, GitHub, Evidence

2019

Benchmark Category Date Created Date Defeated Killed By Defeated By Model Original Score Final Score Details Links
SuperGLUE Language 2019-05 2019-10 Saturation T5 Human: 89.8% T5: 89.3% More challenging language understanding tasks (word sense, causal reasoning, RC). Paper, Website
WSC Common Sense 2012-05 2019-07 Saturation ROBERTA (w SFT) Human: 96.5% ROBERTA (w SFT): 90.1% Carefully crafted sentence pairs with ambiguous pronoun references. Paper, Website
GLUE Language 2018-05 2019-06 Saturation XLNet Human: 87.1% XLNet: 88.4% Nine tasks for evaluating NLU (inference, paraphrase, similarity, etc.). Paper, Website
TriviaQA Knowledge 2017-05 2019-06 Saturation SpanBERT Human: 79.7% SpanBERT: 83.6% 650K QA-evidence triples requiring cross-sentence reasoning. Paper, Website
SQuAD v2.0 Language 2018-05 2019-04 Saturation BERT Human: 89.5% BERT: 89.5% Extension of SQuAD adding unanswerable questions. Paper, Website
SQuAD Language 2016-05 2019-03 Saturation BERT Human: 91.2% BERT: 93.2% 100,000+ QA tasks on Wikipedia articles. Paper, Website

2018

Benchmark Category Date Created Date Defeated Killed By Defeated By Model Original Score Final Score Details Links
SWAG Common Sense 2018-05 2018-10 Saturation BERT Human: 88% BERT: 86% 113K multiple-choice questions about grounded situations (common sense “next step”). Paper, Website

References

website github

Retrieved from "https://aiwiki.ai/index.php?title=LLM_Benchmarks_Timeline&oldid=13776"