LLM Benchmarks Timeline: Difference between revisions

From AI Wiki
No edit summary
 
(6 intermediate revisions by the same user not shown)
Line 1: Line 1:
== 2024 ==
{{see also|LLM Comparisons|LLM Rankings}}
{| class="wikitable"
Timeline of [[benchmarks]] surpassed by [[large language models]] (LLMs).
 
==2024==
{| class="wikitable sortable"
|-
|-
! Benchmark
! Benchmark
! Category
! Category
! Time Span
! Date Created
! Date Created
! Date Defeated
! Date Defeated
! Killed By
! Killed By
! Defeated By
! Defeated By Model
! Original Score
! Original Score
! Final Score
! Final Score
Line 16: Line 18:
| [[ARC-AGI]]
| [[ARC-AGI]]
| Reasoning
| Reasoning
| 2019-11 – 2024-12
| 2019-11
| 2019-11
| 2024-12
| 2024-12
| Saturation
| Saturation
| O3
| [[O3]]
| Human Baseline: ~80%
| Human Baseline: ~80%
| O3: 87.5%
| O3: 87.5%
Line 28: Line 29:
| [[MATH]]
| [[MATH]]
| Mathematics
| Mathematics
| 2021-03 – 2024-09
| 2021-03
| 2021-03
| 2024-09
| 2024-09
| Saturation
| Saturation
| O1
| [[O1]]
| Average CS PhD: ~40%
| Average CS PhD: ~40%
| O1: 94.8%
| O1: 94.8%
Line 40: Line 40:
| [[BIG-Bench-Hard]]
| [[BIG-Bench-Hard]]
| Multi-task
| Multi-task
| 2022-10 – 2024-06
| 2022-10
| 2022-10
| 2024-06
| 2024-06
| Saturation
| Saturation
| Sonnet 3.5
| [[Sonnet 3.5]]
| Average Human: 67.7%
| Average Human: 67.7%
| Sonnet 3.5: 93.1%
| Sonnet 3.5: 93.1%
Line 52: Line 51:
| [[HumanEval]]
| [[HumanEval]]
| Coding
| Coding
| 2021-07 – 2024-05
| 2021-07
| 2021-07
| 2024-05
| 2024-05
| Saturation
| Saturation
| GPT-4o
| [[GPT-4o]]
| Unspecified
| Unspecified
| GPT-4o: 90.2%
| GPT-4o: 90.2%
Line 64: Line 62:
| [[IFEval]]
| [[IFEval]]
| Instruction Following
| Instruction Following
| 2023-11 – 2024-03
| 2023-11
| 2023-11
| 2024-03
| 2024-03
| Saturation
| Saturation
| LLama 3.3 70B
| [[LLama 3.3 70B]]
| Unspecified
| Unspecified
| LLama 3.3 70B: 92.1%
| LLama 3.3 70B: 92.1%
Line 74: Line 71:
| [https://arxiv.org/abs/2311.07911 Paper], [https://github.com/google-research/google-research/tree/master/instruction_following_eval GitHub], [https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD_VISION.md Evidence]
| [https://arxiv.org/abs/2311.07911 Paper], [https://github.com/google-research/google-research/tree/master/instruction_following_eval GitHub], [https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD_VISION.md Evidence]
|}
|}
== 2023 ==
==2023==
{| class="wikitable"
{| class="wikitable sortable"
|-
|-
! Benchmark
! Benchmark
! Category
! Category
! Time Span
! Date Created
! Date Created
! Date Defeated
! Date Defeated
! Killed By
! Killed By
! Defeated By
! Defeated By Model
! Original Score
! Original Score
! Final Score
! Final Score
Line 91: Line 87:
| [[GSM8K]]
| [[GSM8K]]
| Mathematics
| Mathematics
| 2021-10 – 2023-11
| 2021-10
| 2021-10
| 2023-11
| 2023-11
| Saturation
| Saturation
| GPT-4
| [[GPT-4]]
| Unspecified
| Unspecified
| GPT-4: 92.0%
| GPT-4: 92.0%
Line 103: Line 98:
| [[Turing Test]]
| [[Turing Test]]
| Conversation
| Conversation
| 1950-10 – 2023-03
| 1950-10
| 1950-10
| 2023-03
| 2023-03
| Saturation
| Saturation
| GPT-4
| [[GPT-4]]
| Interrogator > 50%
| Interrogator > 50%
| Interrogator 46%
| Interrogator 46%
Line 115: Line 109:
| [[ARC (AI2)]]
| [[ARC (AI2)]]
| Reasoning
| Reasoning
| 2018-03 – 2023-03
| 2018-03
| 2018-03
| 2023-03
| 2023-03
| Saturation
| Saturation
| GPT-4
| [[GPT-4]]
| Unspecified
| Unspecified
| GPT-4: 96.3%
| GPT-4: 96.3%
Line 127: Line 120:
| [[HellaSwag]]
| [[HellaSwag]]
| Common Sense
| Common Sense
| 2019-05 – 2023-03
| 2019-05
| 2019-05
| 2023-03
| 2023-03
| Saturation
| Saturation
| GPT-4
| [[GPT-4]]
| Human: 95.6%
| Human: 95.6%
| GPT-4: 95.3%
| GPT-4: 95.3%
Line 139: Line 131:
| [[MMLU]]
| [[MMLU]]
| Knowledge
| Knowledge
| 2020-09 – 2023-03
| 2020-09
| 2020-09
| 2023-03
| 2023-03
| Saturation
| Saturation
| GPT-4
| [[GPT-4]]
| 95th pct Human: 87.0%
| 95th pct Human: 87.0%
| GPT-4: 87.3%
| GPT-4: 87.3%
Line 151: Line 142:
| [[WinoGrande]]
| [[WinoGrande]]
| Common Sense
| Common Sense
| 2019-07 – 2023-03
| 2019-07
| 2019-07
| 2023-03
| 2023-03
| Saturation
| Saturation
| GPT-4
| [[GPT-4]]
| Human: 94%
| Human: 94%
| GPT-4: 87.5%
| GPT-4: 87.5%
Line 162: Line 152:
|}
|}


== Pre-2023 ==
==Pre-2023==
=== 2022 ===
===2022===
{| class="wikitable"
{| class="wikitable sortable"
|-
|-
! Benchmark
! Benchmark
! Category
! Category
! Time Span
! Date Created
! Date Created
! Date Defeated
! Date Defeated
! Killed By
! Killed By Model
! Defeated By
! Defeated By
! Original Score
! Original Score
Line 180: Line 169:
| [[BIG-Bench]]
| [[BIG-Bench]]
| Multi-task
| Multi-task
| 2021-06 – 2022-04
| 2021-06
| 2021-06
| 2022-04
| 2022-04
| Saturation
| Saturation
| Palm 540B
| [[Palm 540B]]
| Human: 49.8%
| Human: 49.8%
| Palm 540B: 61.4%
| Palm 540B: 61.4%
Line 191: Line 179:
|}
|}


=== 2019 ===
===2019===
{| class="wikitable"
{| class="wikitable sortable"
|-
|-
! Benchmark
! Benchmark
! Category
! Category
! Time Span
! Date Created
! Date Created
! Date Defeated
! Date Defeated
! Killed By
! Killed By
! Defeated By
! Defeated By Model
! Original Score
! Original Score
! Final Score
! Final Score
Line 208: Line 195:
| [[SuperGLUE]]
| [[SuperGLUE]]
| Language
| Language
| 2019-05 – 2019-10
| 2019-05
| 2019-05
| 2019-10
| 2019-10
| Saturation
| Saturation
| T5
| [[T5]]
| Human: 89.8%
| Human: 89.8%
| T5: 89.3%
| T5: 89.3%
Line 220: Line 206:
| [[WSC]]
| [[WSC]]
| Common Sense
| Common Sense
| 2012-05 – 2019-07
| 2012-05
| 2012-05
| 2019-07
| 2019-07
| Saturation
| Saturation
| ROBERTA (w SFT)
| [[ROBERTA (w SFT)]]
| Human: 96.5%
| Human: 96.5%
| ROBERTA (w SFT): 90.1%
| ROBERTA (w SFT): 90.1%
Line 232: Line 217:
| [[GLUE]]
| [[GLUE]]
| Language
| Language
| 2018-05 – 2019-06
| 2018-05
| 2018-05
| 2019-06
| 2019-06
| Saturation
| Saturation
| XLNet
| [[XLNet]]
| Human: 87.1%
| Human: 87.1%
| XLNet: 88.4%
| XLNet: 88.4%
Line 244: Line 228:
| [[TriviaQA]]
| [[TriviaQA]]
| Knowledge
| Knowledge
| 2017-05 – 2019-06
| 2017-05
| 2017-05
| 2019-06
| 2019-06
| Saturation
| Saturation
| SpanBERT
| [[SpanBERT]]
| Human: 79.7%
| Human: 79.7%
| SpanBERT: 83.6%
| SpanBERT: 83.6%
Line 256: Line 239:
| [[SQuAD v2.0]]
| [[SQuAD v2.0]]
| Language
| Language
| 2018-05 – 2019-04
| 2018-05
| 2018-05
| 2019-04
| 2019-04
| Saturation
| Saturation
| BERT
| [[BERT]]
| Human: 89.5%
| Human: 89.5%
| BERT: 89.5%
| BERT: 89.5%
Line 268: Line 250:
| [[SQuAD]]
| [[SQuAD]]
| Language
| Language
| 2016-05 – 2019-03
| 2016-05
| 2016-05
| 2019-03
| 2019-03
| Saturation
| Saturation
| BERT
| [[BERT]]
| Human: 91.2%
| Human: 91.2%
| BERT: 93.2%
| BERT: 93.2%
Line 279: Line 260:
|}
|}


=== 2018 ===
===2018===
{| class="wikitable"
{| class="wikitable sortable"
|-
|-
! Benchmark
! Benchmark
! Category
! Category
! Time Span
! Date Created
! Date Created
! Date Defeated
! Date Defeated
! Killed By
! Killed By
! Defeated By
! Defeated By Model
! Original Score
! Original Score
! Final Score
! Final Score
Line 296: Line 276:
| [[SWAG]]
| [[SWAG]]
| Common Sense
| Common Sense
| 2018-05 – 2018-10
| 2018-05
| 2018-05
| 2018-10
| 2018-10
| Saturation
| Saturation
| BERT
| [[BERT]]
| Human: 88%
| Human: 88%
| BERT: 86%
| BERT: 86%
Line 306: Line 285:
| [https://arxiv.org/abs/1808.05326 Paper], [https://rowanzellers.com/swag/ Website]
| [https://arxiv.org/abs/1808.05326 Paper], [https://rowanzellers.com/swag/ Website]
|}
|}
==References==
[https://r0bk.github.io/killedbyllm/ website]
[https://github.com/R0bk/killedbyllm github]
[[Category:Benchmarks]] [[Category:Timelines]] [[Category:Aggregate pages]]

Latest revision as of 21:01, 13 January 2025

See also: LLM Comparisons and LLM Rankings

Timeline of benchmarks surpassed by large language models (LLMs).

2024

Benchmark Category Date Created Date Defeated Killed By Defeated By Model Original Score Final Score Details Links
ARC-AGI Reasoning 2019-11 2024-12 Saturation O3 Human Baseline: ~80% O3: 87.5% Abstract reasoning challenge with visual pattern completion tasks created by François Chollet. Paper, Website
MATH Mathematics 2021-03 2024-09 Saturation O1 Average CS PhD: ~40% O1: 94.8% 12K challenging competition math problems from AMC/AIME, requiring complex multi-step reasoning. Paper, GitHub
BIG-Bench-Hard Multi-task 2022-10 2024-06 Saturation Sonnet 3.5 Average Human: 67.7% Sonnet 3.5: 93.1% A curated suite of 23 challenging tasks from BIG-Bench. Paper, GitHub, Evidence
HumanEval Coding 2021-07 2024-05 Saturation GPT-4o Unspecified GPT-4o: 90.2% 164 Python programming problems testing coding abilities. Paper, GitHub, Evidence
IFEval Instruction Following 2023-11 2024-03 Saturation LLama 3.3 70B Unspecified LLama 3.3 70B: 92.1% Evaluation suite testing multi-step instruction-following capabilities. Paper, GitHub, Evidence

2023

Benchmark Category Date Created Date Defeated Killed By Defeated By Model Original Score Final Score Details Links
GSM8K Mathematics 2021-10 2023-11 Saturation GPT-4 Unspecified GPT-4: 92.0% 8.5K grade school math word problems requiring step-by-step solutions. Paper, GitHub, Evidence
Turing Test Conversation 1950-10 2023-03 Saturation GPT-4 Interrogator > 50% Interrogator 46% The original AI benchmark proposed by Alan Turing in 1950 (the "imitation game"). Paper, Evidence
ARC (AI2) Reasoning 2018-03 2023-03 Saturation GPT-4 Unspecified GPT-4: 96.3% Grade-school multiple-choice reasoning tasks testing logical, spatial, temporal reasoning. Paper, Website, Evidence
HellaSwag Common Sense 2019-05 2023-03 Saturation GPT-4 Human: 95.6% GPT-4: 95.3% Multiple-choice questions about everyday scenarios with adversarial filtering. Paper, Website, Evidence
MMLU Knowledge 2020-09 2023-03 Saturation GPT-4 95th pct Human: 87.0% GPT-4: 87.3% 57 subjects from real-world sources (professional exams) testing breadth and depth of knowledge. Paper, GitHub, Evidence
WinoGrande Common Sense 2019-07 2023-03 Saturation GPT-4 Human: 94% GPT-4: 87.5% Enhanced WSC with 44K problems testing common-sense pronoun resolution. Paper, Website, Evidence

Pre-2023

2022

Benchmark Category Date Created Date Defeated Killed By Model Defeated By Original Score Final Score Details Links
BIG-Bench Multi-task 2021-06 2022-04 Saturation Palm 540B Human: 49.8% Palm 540B: 61.4% 204 tasks spanning linguistics, math, common-sense reasoning, and more. Paper, GitHub, Evidence

2019

Benchmark Category Date Created Date Defeated Killed By Defeated By Model Original Score Final Score Details Links
SuperGLUE Language 2019-05 2019-10 Saturation T5 Human: 89.8% T5: 89.3% More challenging language understanding tasks (word sense, causal reasoning, RC). Paper, Website
WSC Common Sense 2012-05 2019-07 Saturation ROBERTA (w SFT) Human: 96.5% ROBERTA (w SFT): 90.1% Carefully crafted sentence pairs with ambiguous pronoun references. Paper, Website
GLUE Language 2018-05 2019-06 Saturation XLNet Human: 87.1% XLNet: 88.4% Nine tasks for evaluating NLU (inference, paraphrase, similarity, etc.). Paper, Website
TriviaQA Knowledge 2017-05 2019-06 Saturation SpanBERT Human: 79.7% SpanBERT: 83.6% 650K QA-evidence triples requiring cross-sentence reasoning. Paper, Website
SQuAD v2.0 Language 2018-05 2019-04 Saturation BERT Human: 89.5% BERT: 89.5% Extension of SQuAD adding unanswerable questions. Paper, Website
SQuAD Language 2016-05 2019-03 Saturation BERT Human: 91.2% BERT: 93.2% 100,000+ QA tasks on Wikipedia articles. Paper, Website

2018

Benchmark Category Date Created Date Defeated Killed By Defeated By Model Original Score Final Score Details Links
SWAG Common Sense 2018-05 2018-10 Saturation BERT Human: 88% BERT: 86% 113K multiple-choice questions about grounded situations (common sense “next step”). Paper, Website

References

website github

Retrieved from "https://aiwiki.ai/index.php?title=LLM_Benchmarks_Timeline&oldid=13776"