LLM Benchmarks Timeline

2024

Benchmark	Category	Time Span	Date Created	Date Defeated	Killed By	Defeated By	Original Score	Final Score	Links	Details
ARC-AGI	Reasoning	2019-11 – 2024-12	2019-11	2024-12	Saturation	O3	Human Baseline: ~80%	O3: 87.5%	Paper, Website	Abstract reasoning challenge with visual pattern completion tasks created by François Chollet.
MATH	Mathematics	2021-03 – 2024-09	2021-03	2024-09	Saturation	O1	Average CS PhD: ~40%	O1: 94.8%	Paper, GitHub	12K challenging competition math problems from AMC/AIME, requiring complex multi-step reasoning.
BIG-Bench-Hard	Multi-task	2022-10 – 2024-06	2022-10	2024-06	Saturation	Sonnet 3.5	Average Human: 67.7%	Sonnet 3.5: 93.1%	Paper, GitHub, Evidence	A curated suite of 23 challenging tasks from BIG-Bench.
HumanEval	Coding	2021-07 – 2024-05	2021-07	2024-05	Saturation	GPT-4o	Unspecified	GPT-4o: 90.2%	Paper, GitHub, Evidence	164 Python programming problems testing coding abilities.
IFEval	Instruction Following	2023-11 – 2024-03	2023-11	2024-03	Saturation	LLama 3.3 70B	Unspecified	LLama 3.3 70B: 92.1%	Paper, GitHub, Evidence	Evaluation suite testing multi-step instruction-following capabilities.

Benchmark	Category	Time Span	Date Created	Date Defeated	Killed By	Defeated By	Original Score	Final Score	Links	Details
GSM8K	Mathematics	2021-10 – 2023-11	2021-10	2023-11	Saturation	GPT-4	Unspecified	GPT-4: 92.0%	Paper, GitHub, Evidence	8.5K grade school math word problems requiring step-by-step solutions.
Turing Test	Conversation	1950-10 – 2023-03	1950-10	2023-03	Saturation	GPT-4	Interrogator > 50%	Interrogator 46%	Paper, Evidence	The original AI benchmark proposed by Alan Turing in 1950 (the "imitation game").
ARC (AI2)	Reasoning	2018-03 – 2023-03	2018-03	2023-03	Saturation	GPT-4	Unspecified	GPT-4: 96.3%	Paper, Website, Evidence	Grade-school multiple-choice reasoning tasks testing logical, spatial, temporal reasoning.
HellaSwag	Common Sense	2019-05 – 2023-03	2019-05	2023-03	Saturation	GPT-4	Human: 95.6%	GPT-4: 95.3%	Paper, Website, Evidence	Multiple-choice questions about everyday scenarios with adversarial filtering.
MMLU	Knowledge	2020-09 – 2023-03	2020-09	2023-03	Saturation	GPT-4	95th pct Human: 87.0%	GPT-4: 87.3%	Paper, GitHub, Evidence	57 subjects from real-world sources (professional exams) testing breadth and depth of knowledge.
WinoGrande	Common Sense	2019-07 – 2023-03	2019-07	2023-03	Saturation	GPT-4	Human: 94%	GPT-4: 87.5%	Paper, Website, Evidence	Enhanced WSC with 44K problems testing common-sense pronoun resolution.

Benchmark	Category	Time Span	Date Created	Date Defeated	Killed By	Defeated By	Original Score	Final Score	Links	Details
BIG-Bench	Multi-task	2021-06 – 2022-04	2021-06	2022-04	Saturation	Palm 540B	Human: 49.8%	Palm 540B: 61.4%	Paper, GitHub, Evidence	204 tasks spanning linguistics, math, common-sense reasoning, and more.

Benchmark	Category	Time Span	Date Created	Date Defeated	Killed By	Defeated By	Original Score	Final Score	Links	Details
SuperGLUE	Language	2019-05 – 2019-10	2019-05	2019-10	Saturation	T5	Human: 89.8%	T5: 89.3%	Paper, Website	More challenging language understanding tasks (word sense, causal reasoning, RC).

Benchmark	Category	Time Span	Date Created	Date Defeated	Killed By	Defeated By	Original Score	Final Score	Links	Details
SWAG	Common Sense	2018-05 – 2018-10	2018-05	2018-10	Saturation	BERT	Human: 88%	BERT: 86%	[Paper](https://arxiv.org/abs/1808.05326), [Website](https://rowanzellers.com/swag/)	113K multiple-choice questions about grounded situations (common sense “next step”).