AIME 2024
Last reviewed
Sources
No citations yet
Review status
Needs citations
Revision
v4 · 4,131 words
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation.
Suggest edit
CtrlK
Last reviewed
Sources
No citations yet
Review status
Needs citations
Revision
v4 · 4,131 words
Add missing citations, update stale details, or suggest a clearer explanation.
| AIME 2024 | |
|---|---|
| Overview | |
| Full name | American Invitational Mathematics Examination 2024 |
| Abbreviation | AIME 2024 |
| Description | A challenging mathematical reasoning benchmark based on the American Invitational Mathematics Examination 2024 problems, designed to evaluate AI models' ability to solve complex high school mathematics problems requiring multi-step reasoning |
| Release date | 2024-02-01 |
| Latest version | 1.0 |
| Benchmark updated | 2024-02-07 |
| Authors | Mathematical Association of America |
| Organization | Mathematical Association of America (MAA), Art of Problem Solving (AoPS) |
| Technical Details | |
| Type | Mathematical Reasoning, Problem Solving |
| Modality | Text |
| Task format | Open-ended problem solving |
| Number of tasks | 30 (15 from AIME I + 15 from AIME II) |
| Total examples | 30 |
| Evaluation metric | Exact Match, Pass@1, Cons@N |
| Domains | Algebra, Geometry, Number Theory, Combinatorics, Probability |
| Languages | English |
| Performance | |
| Human performance | ~26% (median high-scoring qualifier solves 4 of 15) |
| Baseline | ~12% (GPT-4o pass@1) |
| SOTA score | 95.8% (Grok 3 Mini, cons@64) |
| SOTA model | Grok 3 Mini (xAI) |
| SOTA date | 2025-02-17 |
| Saturated | Yes (top reasoning models exceed 90%) |
| Resources | |
| Website | Official website |
| GitHub | Repository |
| Dataset | Download |
| Predecessor | AIME 2023 |
| Successor | AIME 2025 |
AIME 2024 is an AI benchmark of 30 problems drawn from the 2024 American Invitational Mathematics Examination that has become the standard yardstick for measuring the mathematical reasoning of large language models such as OpenAI o1, o3, and DeepSeek R1.[1][3] Each problem has an integer answer between 0 and 999, the competition was originally written by the Mathematical Association of America for top high school mathematicians in the United States, and the 30 questions combine the 15 problems of AIME I with the 15 of AIME II.[1][2] Because the answer space is small, the questions resist guessing, and the problems demand layered reasoning, AIME 2024 became one of the most cited tests of mathematical reasoning ability for reasoning models released in late 2024 and 2025.[3] By 2026 the benchmark is widely considered saturated: most frontier reasoning models score above 90% and several public leaderboards have stopped weighting it in their composite rankings.[15][19]
The AIME 2024 benchmark is built on the 2024 cycle of the American Invitational Mathematics Examination, an invitational round of the AMC competition series. AIME is administered to students who score in roughly the top 5% of the AMC 10 or top 2.5% of the AMC 12.[2] Strong AIME performance is the gateway to the USA Mathematical Olympiad (USAMO) and USA Junior Mathematical Olympiad (USAJMO), and the index used for those competitions combines AMC and AIME scores.
The AIME 2024 problems were administered in two sittings: AIME I on January 31 to February 1, 2024, and AIME II on February 7, 2024.[2] Each sitting contains 15 problems and runs for three hours, scored 1 point per correct answer with no penalty for wrong or blank answers, so a perfect score is 15 per form and 30 across both.[2] After the contest closed, the problems and full solutions were posted publicly through the Art of Problem Solving wiki and other community archives, which is exactly why AIME 2024 became a popular target for AI researchers, and also why later contamination concerns emerged.[12]
The machine-readable benchmark most often used in papers is the Hugging Face dataset published by Maxwell Jia, which packages the 30 official problems with reference answers in JSON Lines format.[11] Several alternative packagings exist, including HuggingFaceH4/aime_2024 and math-ai/aime24, but they all wrap the same 30 MAA problems. Some early evaluations, including OpenAI's blog post for o1, restricted the benchmark to the 15 AIME I problems only, which is one source of confusion when comparing scores across papers.[3]
The official MAA answer for AIME I 2024 Problem 12 was initially published as 384 but was corrected to 385 within days of the contest.[20] The Maxwell-Jia dataset and most modern evaluation harnesses use the corrected 385; some early scripts that copied the original answer key produce systematically lower scores for any model that arrives at the correct 385. This is a small but real source of cross-paper discrepancy for late-2024 results.
A few features made AIME 2024 the right shape for evaluating reasoning models:
Each AIME 2024 problem requires the model to output a single integer between 000 and 999. Problems are presented as plain text and may include LaTeX. Common task patterns include:
The difficulty curve is steep. Problems 1 to 5 are typically tractable for an experienced AMC solver. Problems 11 to 15 frequently require nonobvious construction or clever invariants and are roughly USAMO entry difficulty. Problem 15 on AIME I 2024, for example, required setting up surface-area equations with Vieta's relations on a degree-three polynomial.[20]
The benchmark is run in several ways depending on the paper:
| Evaluation method | Description | Implementation |
|---|---|---|
| Exact match | Model output must equal the ground truth integer | Final answer extracted from \boxed{} or last line of model output |
| Pass@1 (greedy) | Single deterministic attempt | Temperature 0, no sampling |
| Pass@1 (averaged) | Average correctness across many samples | Common setup is 64 samples per problem with temperature 0.6 and top-p 0.95 |
| Cons@N (consensus) | Majority vote over N samples | Often N=64; reduces variance from sampling |
| Best-of-N | Re-rank N samples with a learned scorer | Used by OpenAI for the 93% o1 result with N=1000 |
| Tool-augmented | Model can call a Python interpreter | Pushes scores to ~99-100% on frontier models |
DeepSeek's R1 paper, for example, fixes temperature to 0.6, top-p to 0.95, and reports pass@1 averaged over 64 sampled responses, which is now a common reference setup.[6] Models are usually prompted with something close to: "Please reason step by step, and put your final answer within \boxed{}."
Because there are only 30 problems, single-run scores have high variance. A model that gets 24 right one run and 21 right the next has shifted by 10 percentage points without any change in capability. That is why most credible scores either average over many seeds or report cons@N.
By 2025 it became standard to publish two AIME numbers for any model that supports tool use: pure reasoning (no tools) and with a Python interpreter. Giving a frontier reasoning model a Python sandbox typically pushes AIME 2024 accuracy to ~99-100%, with errors confined to a small number of problems whose translation to code is itself the hard step.[17][21] At OpenAI's GPT-5 launch, the "Pro" variant with parallel test-time compute and tool use reportedly solved 100% of the AIME 2025 set, and similar numbers are routine on the older 2024 problems. These open-book results have shifted attention away from AIME as a pure reasoning yardstick and toward benchmarks like FrontierMath, HMMT, and USAMO 2025 where tool use is either disallowed or not enough.[15][16]
The table below collects widely-cited AIME 2024 scores from primary sources. Where scores were originally reported on the 15-problem AIME I subset (as in OpenAI's September 2024 blog post), that is noted in the methodology column.
| Model | AIME 2024 score | Methodology | Source / date |
|---|---|---|---|
| GPT-5 | ~95.7% | pass@1 | OpenAI / aggregator, 2026[16][22] |
| Grok 3 Mini (Think, high) | 95.8% | cons@64, test-time compute scaling | xAI, February 2025[9] |
| Grok 4 | ~94.0% | pass@1 | xAI, 2026[22] |
| OpenAI o4-mini | 93.4% | pass@1, no tools | OpenAI, April 2025[4] |
| Grok 3 (Think) | 93.3% | cons@64 | xAI, February 2025[9] |
| LongCat-Flash-Thinking (Meituan) | 93.3% | self-reported | Meituan, 2025[15] |
| OpenAI o1 (re-ranked) | 93% (13.9/15) | re-ranking 1000 samples on AIME I | OpenAI, September 2024[3] |
| Gemini 2.5 Pro | 92.0% | pass@1 | Google DeepMind, March 2025[10] |
| OpenAI o3 | 91.6% (96.7% later report) | pass@1 | OpenAI, April 2025[4][16] |
| DeepSeek-R1-0528 | 91.4% | pass@1 | DeepSeek, May 2025[23] |
| GLM-4.5 (Zhipu AI) | 91.0% | self-reported | Zhipu AI, 2025[15] |
| Ministral 3 (14B Reasoning) | 89.8% | self-reported | Mistral AI, 2025[15] |
| Gemini 2.5 Flash | 88.0% | pass@1 | Google, 2025[15] |
| OpenAI o3-mini (high) | 87.3% | pass@1 | OpenAI, January 2025[5] |
| DeepSeek R1-Zero (cons@64) | 86.7% | majority vote over 64 samples | DeepSeek, January 2025[6] |
| DeepSeek R1 Distill Llama 70B | 86.7% | self-reported | DeepSeek, 2025[15] |
| Qwen3-235B-A22B | 85.7% | self-reported | Alibaba, 2025[15] |
| OpenAI o1-pro | 86.0% | pass@1 | OpenAI, 2025[15] |
| OpenAI o1 (cons@64) | 83% (12.5/15) | consensus on AIME I | OpenAI, September 2024[3] |
| Claude 3.7 Sonnet (extended thinking) | 80.0% | parallel extended thinking, 64K token budget | Anthropic, February 2025[7] |
| DeepSeek R1 | 79.8% | pass@1 averaged over 64 samples | DeepSeek paper, January 2025[6] |
| QwQ-32B-Preview (Alibaba) | 79.5% | pass@1 averaged | Alibaba, November 2024[16] |
| OpenAI o1 (single sample) | 74.4% | pass@1 on AIME I (11.1/15) | OpenAI, September 2024[3] |
| Gemini 2.0 Flash Thinking | 73.3% | pass@1 | Google, December 2024[18] |
| OpenAI o1-mini | 63.6% | pass@1 averaged | DeepSeek paper / OpenAI[6] |
| Grok 3 (base, non-reasoning) | 52.2% | pass@1 | xAI, February 2025[9] |
| Gemini 2.0 Flash (experimental) | 35.5% | pass@1 | Google, December 2024[18] |
| Claude 3.7 Sonnet (standard mode) | 23.3% | pass@1, no extended thinking | Anthropic, February 2025[7] |
| Gemini 1.5 Pro | 19.3% | pass@1 | Google, 2024[10] |
| GPT-4o | ~12% (1.8/15) | pass@1 on AIME I | OpenAI, September 2024[3] |
| Claude 3.5 Sonnet | ~10% | pass@1 | Anthropic, 2024[14] |
A few notes on this table:
The most striking pattern in the table is the gap between reasoning-trained models and conventional chat models. GPT-4o and the original Claude 3.5 Sonnet both sit around 10 to 13% on AIME 2024. Models trained with reinforcement learning on chains of thought, including OpenAI o1, DeepSeek R1, and Claude 3.7 Sonnet with extended thinking, jumped to 70-90%+. The same Claude 3.7 Sonnet weights score 23.3% in standard mode and 80.0% with extended thinking enabled, which is the cleanest demonstration of how much of the gain comes from inference-time reasoning rather than raw capability.[7][8] OpenAI's o1 blog also reports a roughly log-linear scaling relationship between accuracy and test-time compute on AIME, a pattern most labs have since adopted in their evaluation reports.[3] In follow-on analyses, scaling test-time compute by 100x typically lifts AIME 2024 accuracy from around 20% to around 80%, while 100x more RL training only moves it from about 33% to 66%, indicating inference scaling is the larger lever on this dataset.[16]
AIME 2024 has been the headline number for nearly every open-weight reasoning model release in 2025-2026. The fast-moving open-weight stack of late 2025 - DeepSeek-R1-0528 at 91.4%, Qwen3-235B-A22B at 85.7%, GLM-4.5 at 91.0%, MiniMax M1 at 86.0%, LongCat-Flash-Thinking at 93.3%, and the Ministral 3 reasoning family - has effectively closed the AIME 2024 gap with proprietary frontier models, even if it has not closed the gap on more contamination-resistant benchmarks like AIME 2025 or FrontierMath.[15][23] DeepSeek's R1-0528 release notes specifically called out AIME 2024 as moving from 79.8% to 91.4% as the chain-of-thought training matured, with average reasoning trace length nearly doubling from roughly 12K to 23K tokens on harder problems.[23]
Separating model scores from human performance is messier than people often present. AIME is taken only by AMC qualifiers, so the typical AIME taker is already strong:
When OpenAI claimed o1 at 93% placed it "among the top 500 students in the United States," that comparison is to AIME I only, not the AIME I and II combined dataset most papers use today.[3]
The AIME 2024 benchmark tests proficiency across the standard secondary math contest domains:
| Domain | Example topics | Approximate share of problems |
|---|---|---|
| Algebra | Polynomial equations, systems, inequalities, sequences | ~27% |
| Geometry | Euclidean geometry, coordinate geometry, transformations | ~27% |
| Number theory | Divisibility, modular arithmetic, primes, Diophantine | ~20% |
| Combinatorics | Counting, probability, recursion | ~20% |
| Complex numbers | Roots of unity, complex algebra | ~6% |
Topic boundaries are fuzzy. A typical AIME 12 problem might mix coordinate geometry with number theory and a touch of combinatorics, and many problems are deliberately built so that the obvious approach is intractable and a clever observation cuts the work down. The published Areteem and AoPS solution sets for AIME I 2024 cover precisely these blended patterns: AIME I Problem 3 is a game-theory/modular arithmetic mash-up, Problem 10 mixes circle geometry with Stewart's theorem, Problem 14 is a 3D volume calculation, and Problem 15 reduces to applying Vieta's formulas to an unknown polynomial.[20]
Data contamination is the largest caveat hanging over AIME 2024 as a benchmark. The 2024 problems were posted in full, with detailed solutions, on the Art of Problem Solving wiki and elsewhere within hours of the contest. By the time models were trained or fine-tuned in late 2024 and 2025, those pages were almost certainly part of the public web crawls feeding pretraining and instruction tuning datasets.
Researchers building MathArena, a contamination-resistant evaluation framework, found strong signs of contamination on AIME 2024.[12] They report that "widely available problems from AIME 2024 have contaminated the pretraining data of multiple LLMs," inflating scores by 10 to 20 points relative to fresh, uncontaminated competitions.[12] Most leading models scored 10-20 points above what their performance on freshly released contests (AIME 2025, BRUMO 2025) would predict, and QwQ-32B-Preview was estimated to score around 60 percentage points above its human-aligned expectation, the most extreme contamination signal in the paper.[12]
The same project released VAR-AIME24, which substitutes symbolic parameters for the fixed numeric constants in each AIME 2024 problem to test whether models actually solve the problem or recall the answer. Frontier models drop 7-18 percentage points under this perturbation, while smaller RL-tuned models lose 40-75% of their symbolic consistency, suggesting they were memorizing surface form more than reasoning to a solution.[12]
The usual remedy now is to evaluate on AIME 2025, which is administered after the cutoff for most current models, in addition to AIME 2024. When a model that scored 90% on AIME 2024 drops to 75% on AIME 2025, that gap is a useful contamination signal even if neither score is perfectly clean.[12]
Thirty problems is a small sample. A single problem worth 1 of 30 is 3.3% of the score. A model that solves 24 problems correctly scores 80%, but its true skill could plausibly produce anywhere between 22 and 26 on a different draw of similarly hard problems. Confidence intervals on AIME 2024 scores are wide, which is part of why pass@1 averaged over 64 samples and cons@64 are now the standard reporting modes.[6][17] Some recent reports, including Microsoft's inference-time scaling analyses, show that averaging across diverse temperature settings adds another ~7 points on top of conventional same-temperature test-time scaling, providing yet another non-capability lever that can shift headline AIME numbers.[24]
AIME 2024 only tests one style of math: short-answer, integer-output, contest-flavored problems. It says nothing about whether a model can write a real proof, formalize an argument in Lean, or do open-ended exploration of a research-style question. Benchmarks like the USAMO, Putnam competitions, FrontierMath, and Humanity's Last Exam are designed to fill those gaps.
By May 2026 several public leaderboards have stopped weighting AIME 2024 in their composite scores. The CodeSOTA and BenchLM math leaderboards continue to display AIME 2024 numbers for reference but explicitly mark the benchmark as saturated and exclude it from headline rankings, redirecting attention to BRUMO 2025, USAMO 2025, and HMMT 2025.[15][19] Anthropic, Google, and OpenAI have all moved their flagship math results in 2026 launches to a mix of AIME 2025 with tools, FrontierMath, and Putnam, often relegating AIME 2024 to an appendix or omitting it entirely.[21]
AIME 2024 and AIME 2025 share an identical structure, 30 integer-answer problems split across two forms, but they differ in the one dimension that matters most for benchmarking: training-data exposure. The 2024 problems had been on the public web for more than a year before most 2025-2026 models finished pretraining, while the 2025 problems were released in February 2025, after the knowledge cutoff of many of those models.[12] In practice this means AIME 2024 scores tend to run higher and less reliable, and a large drop from AIME 2024 to AIME 2025 for the same model is read as a contamination signal rather than a genuine difference in problem difficulty.[12] Most 2025-2026 model cards now report both numbers side by side for this reason.
AIME 2024 sits inside a broader ecosystem of mathematical reasoning benchmarks that AI researchers run alongside it:
AIME 2024 is probably the single benchmark most responsible for the 2024-2025 reasoning model wave entering public consciousness. The September 2024 OpenAI o1 announcement leaned on AIME 2024 as its headline reasoning result, which set the framing that DeepSeek directly attacked four months later when R1 matched o1's score at a fraction of the inference cost.[3][6] Anthropic, Google, and xAI followed with their own reasoning launches, and AIME 2024 was on every comparison chart.[7][9][10]
The benchmark also accelerated two reporting habits. First, test-time compute became a first-class axis: almost every 2025 reasoning model release included a chart of accuracy against thinking budget, with AIME 2024 the most common dataset for that x-axis.[3][16] Second, cons@N and best-of-N now appear alongside pass@1 in most releases, since 30 problems and large test-time budgets together make any single number too noisy. For education, contamination concerns mean current AIME 2024 scores probably overstate how well frontier models reason on truly novel contest problems, and the more sobering AIME 2025 numbers tend to support that.[12] In 2026 the benchmark functions less as a frontier yardstick than as a baseline floor: any new reasoning model expected to be taken seriously is implicitly required to clear 90% on it before its other benchmark numbers will be read at all.[15][22]