ARC-AGI 1

ARC-AGI 1
Overview
Full name	Abstraction and Reasoning Corpus for Artificial General Intelligence, version 1
Abbreviation	ARC-AGI-1
Description	A benchmark testing fluid intelligence and abstract reasoning through colored grid puzzles solved from a handful of input output examples
Release date	November 5, 2019
Creator	François Chollet
Original affiliation	Google
Current steward	ARC Prize Foundation
Source paper	On the Measure of Intelligence (arXiv:1911.01547)
License	Apache License 2.0
Repository	github.com/fchollet/ARC-AGI
Successor	ARC-AGI 2 (2025), ARC-AGI 3 (2026)
Technical Details
Task type	Visual program induction from input output grid pairs
Modality	Visual, language agnostic
Grid range	1x1 to 30x30 cells
Color palette	10 discrete values (integers 0 to 9)
Demonstration pairs	Typically 2 to 5 per task (around 3 on average)
Total tasks	1,000
Public training set	400 tasks
Public evaluation set	400 tasks
Semi-private evaluation set	100 tasks
Private evaluation set	100 tasks
Evaluation metric	Pass at 2 (top 2 attempts), exact grid match
Performance
Average human (crowd)	64.2% on public eval, 76.2% on training (H-ARC, 2024)
At least one human solver	790 of 800 public tasks (98.75%)
ARC Prize Grand Prize threshold	85% under cost limits
GPT-3 (2020)	0%
GPT-4o (early 2024)	~5%
Greenblatt with GPT-4o (June 2024)	50% on public eval
Claude 3.5 Sonnet (Sept 2024)	~21%
OpenAI o1 preview (Sept 2024)	~21%
MIT TTT 8B (Nov 2024)	53% public, 61.9% with ensemble
ARChitects, Kaggle 2024 winner	53.5% private
MindsAI (2024 best)	55.5% private (ineligible, closed source)
OpenAI o3 low compute (Dec 2024)	75.7% semi private
OpenAI o3 high compute (Dec 2024)	87.5% semi private
Resources
Official site	arcprize.org/arc-agi/1
Leaderboard	arcprize.org/leaderboard
Live human test	arcprize.org/play

ARC-AGI 1, short for Abstraction and Reasoning Corpus for Artificial General Intelligence, version 1, is a visual reasoning benchmark introduced by François Chollet in his November 2019 paper On the Measure of Intelligence^[1]. The test consists of 1,000 unique grid puzzles. For each puzzle the system is shown a handful of input output example pairs, must figure out the underlying transformation rule, and must then produce the correct output for one or more held out test inputs. ARC-AGI 1 was specifically designed to resist memorization, brute statistical pattern matching, and the type of training data scaling that drives most progress on conventional AI benchmarks. It stayed largely unsolved for five years until OpenAI's o3 model reached 87.5% on the semi private evaluation set in December 2024^[2].

Overview

Most AI benchmarks score systems on tasks that look like the data they were trained on. ARC-AGI 1 takes a different position. Each task is novel, the problem definition is given through only a few examples, and the rule the solver must discover is meant to come from general human cognitive priors rather than from memorized facts about the world. Chollet calls these priors "core knowledge": objectness, basic geometry and topology, simple counting, simple agentness, and a few similar building blocks present in young children.

The practical setup looks deceptively simple. A solver sees three or so colored grids on the left labeled as inputs, and the same number of grids on the right labeled as outputs. It then sees a fresh test grid and is asked for the matching output. A four year old can usually figure out tasks like "copy the blue shape, fill its inside with red" without being told the rule. A standard large language model trained on every webpage in existence, in 2020, scored zero. That gap is the entire point of the benchmark.

ARC-AGI 1 is also notable for what it is not. It is not a measure of crystallized knowledge, it is not a multilingual test, and it is not a measure of writing quality, factuality, or instruction following. It targets one specific capability, the ability to acquire a new skill from a handful of demonstrations, which Chollet argues is the operative definition of fluid intelligence.

Origin: Chollet's 2019 paper

The benchmark was first published as part of Chollet's paper On the Measure of Intelligence, posted to arXiv on November 5, 2019 under the identifier 1911.01547^[1]. At the time Chollet was a Senior Staff Engineer at Google, best known as the creator of the Keras deep learning library and as the author of Deep Learning with Python.

The paper makes two main moves. First, it argues that almost every existing intelligence test, whether for humans or for machines, conflates skill (the output) with intelligence (the process that converts experience into skill). Second, it offers a formal alternative grounded in algorithmic information theory. Intelligence is defined as skill acquisition efficiency, that is, the rate at which a system can convert priors and experience into general purpose skill across a scope of unknown future tasks. A high IQ system reaches usable skill on a new task with very little experience and very little compute. A brute memorizer can reach the same skill, but only after seeing massive amounts of relevant data.

Four formal concepts hold the framework together.

Scroll sideways for more →

From these definitions Chollet derives several practical desiderata for an intelligence test, including controlled priors, controlled experience, novelty, and resistance to test set leakage. The Abstraction and Reasoning Corpus is the concrete instantiation of those desiderata.

The paper has since become a touchstone for the AGI research community and has been cited thousands of times. It is the conceptual foundation not only for ARC-AGI 1 but also for ARC-AGI 2 (2025) and ARC-AGI 3 (2026)^[3][4].

Dataset structure

ARC-AGI 1 contains 1,000 hand authored tasks, distributed across four splits. The training and public evaluation splits are openly published in the github.com/fchollet/ARC-AGI repository under the Apache 2.0 license. The semi private and private splits are kept secret and are used to score competition entries.

Scroll sideways for more →

The semi private split was added by the ARC Prize team in mid 2024 so that closed source commercial models could be evaluated without exposing the truly secret private set used for the Kaggle Grand Prize^[5].

Grids and colors

A task's grids are rectangular arrays of integers in the range 0 through 9, where each integer renders as a fixed color in the official viewer. The smallest possible grid is 1x1 and the largest is 30x30. Heights and widths can change between input and output, which is itself part of the puzzle. There is no fixed grid size. The 10 color values are arbitrary labels and have no built in semantics, although the same value always renders as the same color across an entire task^[6].

Example pairs

Most ARC-AGI 1 tasks present 2 to 5 demonstration pairs (3 is the most common count) followed by one or two test inputs. The solver must produce the corresponding test outputs by figuring out the rule the demonstrations all share. Rules in the dataset include things like:

Scroll sideways for more →

Individual tasks can chain several of these ideas together, which is part of why purely template matching approaches have failed.

Example task description

A classic ARC-AGI 1 task, often shown in introductory material, gives the solver three demonstration pairs. In each input, a single small colored shape sits inside a larger black rectangle. In each output, the same shape appears tiled at four corners of the larger rectangle. The implicit rule is something like "locate the shape and stamp four copies of it at the rectangle's corners". A test input then shows a new color and a new shape. The solver must produce the correct corner stamped output, and the answer is graded by exact pixel match.

Nothing in the task statement spells the rule out in words. The solver has to infer it. Different humans will describe the rule differently, but the actual answer grid is unambiguous.

Evaluation methodology

Scoring rule

A task is solved when at least one of the solver's submitted attempts produces an exact match for every test output in that task. The official ARC-AGI 1 protocol allows two attempts per test input, so the leaderboard metric is sometimes called pass at 2^[6]. Older Kaggle protocols allowed three attempts, which is why a few publications report pass at 3 numbers.

Scroll sideways for more →

No partial credit is given. A grid that gets one pixel wrong scores zero on that task. This makes scores brittle but also unambiguous, an important property for a benchmark designed to resist statistical fudging.

Cost limits

The ARC-AGI Pub leaderboard caps total compute spend per evaluation run at $10,000 in API credits^[5]. The ARC Prize Kaggle competition, which targets the secret private set, additionally requires solutions to run inside a 12 hour Kaggle notebook with no external API calls. Together those constraints push researchers toward solutions that are both accurate and efficient, not solutions that simply throw arbitrary amounts of compute at each puzzle.

The Grand Prize threshold of 85% accuracy must be reached inside these efficiency limits. That is why o3's 87.5% high compute score in December 2024, which used roughly 172 times more inference compute than the high efficiency configuration and cost about $4,560 per task on the semi private set, did not unlock the Grand Prize even though the accuracy was above 85%^[2].

History of attempts

Scroll sideways for more →

The 2020 Kaggle competition

The first wide open competition on ARC-AGI 1 was the 2020 "Abstraction and Reasoning Challenge" hosted on Kaggle. The contest used the private 100 task evaluation set and ran for several months. The winner, posting under the handle icecuber, achieved 20% accuracy and won $8,000^[7].

Icecuber's solution was a brute force discrete program search. The author hand built a small domain specific language (DSL) of grid manipulation primitives and then enumerated programs over that DSL to find ones consistent with the demonstration pairs. The approach scaled poorly because the search space grows combinatorially with program length, but it produced the first measurable AI progress on the benchmark and validated Chollet's hypothesis that pure statistical learning was not enough^[7]. Icecuber's open sourced code became a starting point for many later program search systems.

For about three years afterward, AI scores on the private set hovered around 20% to 30%, almost entirely from various refinements of icecuber style DSL search.

Lab42 ARCathons (2022 and 2023)

While OpenAI and other large labs largely ignored ARC-AGI 1 in 2021 and 2022, the Swiss nonprofit research lab Lab42 (founded by Pascal Kaufmann) kept the benchmark alive by running annual community competitions called ARCathons. The 2022 ARCathon attracted 118 teams from 47 countries; Michael Hodel won and went on to release one of the most widely used DSLs for ARC. The 2023 ARCathon expanded to 265 teams from 65 countries and was jointly won by Team SM (Somayyeh Gholami and Mehran Kazeminia) and Jack Cole's MindsAI, both reaching 30% on the private evaluation^[8].

MindsAI's submission was the first credible attempt to apply large language model fine tuning to ARC-AGI 1, foreshadowing the test time training methods that would dominate 2024.

ARC Prize 2024

In June 2024, Mike Knoop, the co founder of automation startup Zapier, partnered with Chollet to launch ARC Prize 2024, a $1 million open competition designed to break the long stagnation on ARC-AGI 1^[9]. Knoop personally funded much of the prize pool. The competition ran on Kaggle through November 2024 and attracted 1,430 teams that submitted 17,789 entries.

Prize structure

Scroll sideways for more →

When the Kaggle phase closed, the state of the art on the private evaluation set had risen from 33% to 55.5%, the largest single year jump in the benchmark's history^[10]. The ARC Prize Foundation published a full technical report on December 5, 2024 (arXiv:2412.04604) with co authors François Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers^[10].

Top winning teams

Scroll sideways for more →

MindsAI achieved a higher score (55.5%) than the ARChitects but was ineligible for prizes because they declined to open source their solution^[10]. All eligible winning solutions are published on Kaggle.

Three winning techniques

The technical report identifies three families of approaches that drove 2024 progress:

1. Deep learning guided program synthesis. Specialized code language models generate candidate Python programs for each task, which are then verified against the demonstration pairs. Ryan Greenblatt's GPT-4o approach is the most widely cited example.
2. Test time training (TTT). A pretrained language model is fine tuned at inference time on synthetic variants of the current task, producing a temporary task specific model. The MIT and ARChitects approaches are the canonical examples.
3. Hybrid systems that combine program synthesis with transductive neural models, recognizing that the two approaches solve different subsets of tasks.

Greenblatt's GPT-4o approach (March to June 2024)

In March 2024, Redwood Research's Ryan Greenblatt announced that he had reached 50% on the public evaluation set using only GPT-4o^[11]. At the time the previous best public eval score was about 34%, so the result was a significant jump. Greenblatt's recipe became a template for several later systems.

The procedure works roughly as follows^[11]:

1. For each task, prompt GPT-4o with both an image rendering and several text encodings of the demonstration grids, plus textual diffs between inputs and outputs.
2. Ask the model to (a) describe the transformation in words, (b) describe how to implement it in code, and (c) emit a Python function that performs the transformation.
3. Sample roughly 5,000 to 8,000 candidate programs per task.
4. Run each program against the demonstration pairs and discard programs whose outputs do not match the demonstrations exactly.
5. Take the most promising 12 programs and prompt GPT-4o again with the diff between actual and expected output to revise them.
6. Take a majority vote across surviving programs on the test input to choose the final answer.

The result demonstrated that frontier LLMs already had enough latent reasoning ability to solve a large fraction of ARC-AGI 1, provided they were used as program search engines rather than as direct grid generators. This was an important conceptual unlock for the rest of the 2024 work.

Test time training approaches

MIT and Cornell

In November 2024, researchers Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, and Jacob Andreas at MIT, with Cornell collaborators, released The Surprising Effectiveness of Test-Time Training for Few-Shot Learning (arXiv:2411.07279)^[12]. They reported a 6x improvement in accuracy over a strong fine tuned baseline, reaching 53.0% on the public validation set with an 8B parameter language model and 61.9% when ensembled with a program synthesis solver^[12]. The combined MIT and Cornell submission scored 47.5% on the semi private set and won 2nd place in the ARC Prize 2024 Paper Award.

MindsAI and the AIRV pattern

Jack Cole's MindsAI team had been refining a similar idea since 2023. Their pipeline pretrains a base LLM on a massive set of synthetically generated ARC like puzzles, then at inference time creates augmented variants of each new task (by rotating, reflecting, recoloring, or removing pairs) and fine tunes the model briefly on those variants. They call the inference time procedure AIRV, for Augment, Inference, Reverse Augment, Vote. MindsAI reports that test time fine tuning alone provides roughly a 300% accuracy boost over their unmodified base model^[13].

The ARChitects

The Kaggle 2024 winners, the ARChitects, applied test time training on top of the open source NeMo-Minitron-8B foundation model. Their key contribution was a stability based selection criterion: rather than picking the most likely answer under the model, they picked the answer that remained the most stable under a battery of input augmentations^[10]. This solved a known failure mode of TTT, where the model overfits to spurious features of the demonstrations.

The o3 breakthrough (December 2024)

On December 20, 2024, OpenAI announced the o3 reasoning model and shared scores on ARC-AGI 1 verified by the ARC Prize team^[2]. The headline numbers:

Scroll sideways for more →

The high efficiency configuration spent roughly $2,680 to evaluate all 100 semi private tasks. The low efficiency configuration spent roughly $456,000 across the same 100 tasks, used about 172 times more inference compute, and crossed the 85% Grand Prize threshold for the first time in the benchmark's history^[2]. Because the cost was orders of magnitude beyond the ARC Prize efficiency limits, the Grand Prize itself remained unclaimed in 2024.

The model was trained on roughly 75% of the public ARC-AGI 1 training set, which is permitted under the leaderboard rules. OpenAI did not publish full architectural details, but Chollet's analysis described the system as a "natural language program search and execution within token space", with chain of thought sequences serving as candidate programs that are scored and refined in a manner roughly analogous to Monte Carlo Tree Search^[2]. He emphasized that the result was not just scaling: "You couldn't throw more compute at GPT-4 and get these results."

The practical effect of the o3 result was that ARC-AGI 1 was effectively retired as a frontier benchmark. The ARC Prize team began work on ARC-AGI 2 shortly afterward.

Human performance

The most carefully measured human baseline on ARC-AGI 1 is the H-ARC study published by Hodel and collaborators in September 2024 (arXiv:2409.01374)^[14]. The team recruited 1,729 humans through online crowd platforms and had them attempt the full set of 400 training and 400 public evaluation tasks. Headline results:

Scroll sideways for more →

In other words, almost every ARC-AGI 1 task is solvable by some ordinary human within three attempts, but no single ordinary human solves all of them. Expert solvers given unlimited time score noticeably higher than the crowd average. The often quoted "73% to 85%" range for human performance combines several earlier smaller studies and the H-ARC numbers.

This baseline is a deliberate part of the benchmark's design. ARC-AGI is not meant to require superhuman ability. It is meant to require human level ability, which is exactly what makes the long stretch of 0% to 30% AI scores so striking.

ARC-AGI 2 and ARC-AGI 3

ARC-AGI 2

ARC-AGI 2 was released on March 24, 2025, in a paper by Chollet and the ARC Prize team (arXiv:2505.11831)^[3]. It uses the same grid format as ARC-AGI 1 (1x1 to 30x30 grids of integers 0 through 9, a few demonstration pairs per task) but introduces a new curated task set specifically designed to be easy for humans and very hard for the methods that solved ARC-AGI 1. Headline results released alongside the launch:

Scroll sideways for more →

The ARC Prize 2025 Kaggle competition used ARC-AGI 2 as its target. The top private score reached 24% during the 2025 contest, far short of the 85% Grand Prize threshold^[15].

ARC-AGI 3

ARC-AGI 3 was previewed in 2025 and launched as the basis of the 2026 ARC Prize, which carries a $2 million prize pool^[16]. ARC-AGI 3 is the first major format change since 2019. Instead of static input output grid pairs, each task is an interactive mini environment. The system must explore, plan, build memory of what it has discovered, set its own subgoals, and remain aligned with the underlying task objective. ARC-AGI 1 and 2 measure abstract reasoning over fixed inputs; ARC-AGI 3 measures agentic reasoning over an unfolding interaction.

Impact and critiques

Influence on the field

For the first half of the 2020s, ARC-AGI 1 served as the most cited counterexample to the claim that ever larger transformer models would automatically reach general intelligence. Its persistence at near zero scores while other benchmarks fell was a major argument used by skeptics of the scaling thesis. Conversely, the 2024 jump from 5% to 87.5% within a single year became Exhibit A for advocates of test time compute and reasoning models.

The benchmark also pushed serious work on:

Scroll sideways for more →

Common critiques

ARC-AGI 1 has attracted several substantive criticisms.

Scroll sideways for more →

Chollet's own framing

In his commentary on the o3 result, Chollet was careful to separate scoring on ARC-AGI 1 from claiming AGI. His statement, that "o3 still fails on some very easy tasks, indicating fundamental differences with human intelligence", reflects his view that benchmarks bound progress from above but cannot certify it^[2]. The point of ARC-AGI 1, in his framing, was to expose a specific weakness of the 2019 era deep learning paradigm. Once that weakness was addressed, the benchmark had served its purpose, and a next generation test was needed. Hence ARC-AGI 2 in 2025 and ARC-AGI 3 in 2026.

Chollet on AGI measurement

Chollet's broader position on measuring AGI, articulated in the 2019 paper and elaborated in interviews since, can be summarized in a few claims.

Scroll sideways for more →

This framework is what makes ARC-AGI 1 unusual. It is not just a hard puzzle dataset, it is the operational instantiation of a specific definition of intelligence. The benchmark's design choices (small handcrafted task pool, secret evaluation splits, exact match scoring, cost limits) are direct consequences of that definition.

In November 2024, Chollet left Google after more than nine years. He and Mike Knoop formalized the ARC Prize Foundation as a US nonprofit in early 2025 with Greg Kamradt as president, and announced the for profit AGI lab Ndea, which uses program synthesis and search style methods aligned with the ideas in On the Measure of Intelligence^[17][18].

See also

• ARC-AGI 2
• ARC-AGI 3
• ARC-AGI (series overview)
• Artificial General Intelligence
• OpenAI o3
• OpenAI o1
• GPT-4o
• Keras
• Few-Shot Learning
• In-context learning
• Chain-of-Thought Prompting
• Monte Carlo Tree Search
• Kaggle
• Benchmark

References

1. Chollet, François. "On the Measure of Intelligence." arXiv preprint, arXiv:1911.01547, November 5, 2019. https://arxiv.org/abs/1911.01547 ↩
2. ARC Prize Foundation. "OpenAI o3 Breakthrough High Score on ARC-AGI-Pub." December 20, 2024 (updated April 16, 2025). https://arcprize.org/blog/oai-o3-pub-breakthrough ↩
3. Chollet, François et al. "ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems." arXiv:2505.11831, 2025. https://arxiv.org/abs/2505.11831 ↩
4. ARC Prize Foundation. "ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence." Technical report, 2026. https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf ↩
5. ARC Prize Foundation. "Introducing the ARC-AGI Public Leaderboard." 2024. https://arcprize.org/blog/introducing-arc-agi-public-leaderboard ↩
6. Chollet, François. "ARC-AGI README and dataset." GitHub repository fchollet/ARC-AGI, Apache License 2.0. https://github.com/fchollet/ARC-AGI ↩
7. icecuber. "1st place solution + code and official documentation." Kaggle Abstraction and Reasoning Challenge writeup, 2020. https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge/writeups/icecuber-1st-place-solution-code-and-official-docu ↩
8. Lab42. "About ARC and ARCathon history." https://lab42.global/arc/ ↩
9. ARC Prize Foundation. "ARC Prize 2024 competition page." 2024. https://arcprize.org/competitions/2024 ↩
10. Chollet, F., Knoop, M., Kamradt, G., Landers, B. "ARC Prize 2024: Technical Report." arXiv:2412.04604, December 5, 2024. https://arxiv.org/abs/2412.04604 ↩
11. Greenblatt, Ryan. "Getting 50% (SoTA) on ARC-AGI with GPT-4o." Redwood Research blog, June 17, 2024. https://blog.redwoodresearch.org/p/getting-50-sota-on-arc-agi-with-gpt ↩
12. Akyürek, E., Damani, M., Qiu, L., Guo, H., Kim, Y., Andreas, J. "The Surprising Effectiveness of Test-Time Training for Abstract Reasoning." arXiv:2411.07279, November 11, 2024. https://arxiv.org/abs/2411.07279 ↩
13. Lab42. "Community Interview: Jack Cole." 2024. https://lab42.global/community-interview-jack-cole/ ↩
14. Lebrun, S., Spies, R., Hodel, M., Ellis, K. et al. "H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark." arXiv:2409.01374, September 2024. https://arxiv.org/abs/2409.01374 ↩
15. ARC Prize Foundation. "ARC Prize 2025 Results and Analysis." 2026. https://arcprize.org/blog/arc-prize-2025-results-analysis ↩
16. ARC Prize Foundation. "ARC Prize history page." https://arcprize.org/history ↩
17. Wiggers, K. "AI pioneer François Chollet leaves Google." TechCrunch, November 14, 2024. https://techcrunch.com/2024/11/14/ai-pioneer-francois-chollet-leaves-google/ ↩
18. Wiggers, K. "AI researcher François Chollet is co-founding a nonprofit to build benchmarks for AGI." TechCrunch, January 8, 2025. https://techcrunch.com/2025/01/08/ai-researcher-francois-chollet-is-co-founding-a-nonprofit-to-build-benchmarks-for-agi/ ↩