ARC-AGI, Explained: Why There Are Three of Them

The same model, on three benchmarks with the same name, in the same week: 97.5%, 90.4%, 30.2%.

That's Claude Opus 5 on ARC-AGI-1, -2 and -3 respectively — all three verified by the ARC Prize Foundation, all three measuring "abstract reasoning." The spread isn't noise or difficulty tuning. Each version exists because the previous one turned out to be measuring something narrower than its authors intended, and the story of how that happened is the most honest account we have of the gap between benchmark progress and actual progress.

Grouped bar chart comparing Claude Opus 5's verified score against the human baseline on ARC-AGI-1 (97.5% vs 98%), ARC-AGI-2 (90.4% vs 60%), and ARC-AGI-3 (30.2% vs 100%)

What ARC-AGI is trying to measure

In November 2019, François Chollet — creator of Keras — published On the Measure of Intelligence, arguing that the field had been measuring the wrong thing. A system that plays superhuman chess isn't demonstrating intelligence; it's demonstrating chess. Chollet proposed defining intelligence instead as skill-acquisition efficiency: how effectively a system picks up genuinely new capabilities, relative to the prior knowledge and experience it started with.

The Abstraction and Reasoning Corpus was released as a concrete instrument for that definition. Each task shows a few example pairs — a small colored input grid, its transformed output — and asks you to infer the rule and apply it to a fresh grid. Solving one requires noticing something like "objects fall to the bottom," or "the small shape gets recolored to match the large one," from two or three demonstrations.

The design is deliberately hostile to memorization. There is no vast internet corpus of grid-transformation puzzles to have absorbed during pretraining, and the tasks were built to resist being reduced to a learnable statistical pattern. Humans find them easy — most people solve them in seconds. That combination is the whole point: a benchmark where the human-machine gap can't be closed by having read more of the internet.

Chollet co-founded ARC Prize with Mike Knoop in June 2024, later formalized as a nonprofit foundation, offering substantial cash prizes for beating human-level performance — with open-sourcing required for eligibility.

How the scoring actually works

Worth understanding, because it's unusually transparent and it explains the axes on every chart in this post. ARC Prize's evaluation harness for v1 and v2 is public on GitHub under an MIT license. Reading the source: each task allows two scored attempts, and counts as solved if either attempt exactly matches the target grid. "Cost per task" — the x-axis on the official leaderboards — is nothing more exotic than the summed API cost of every attempt divided by the number of tasks. It's what you'd pay to run the thing yourself.

The anti-contamination guarantee lives in the datasets rather than the code. There are four: a public training set (1,000 tasks), a public evaluation set for open experimentation, a semi-private set (~120 tasks, necessarily transmitted to commercial APIs in order to score them, so carrying some leakage risk over time), and a fully private set (~120 tasks) that isn't sent anywhere avoidable and backs the official leaderboard. That layering is what makes "the model already saw this puzzle" an answerable objection rather than a permanent asterisk.

ARC-AGI-1, and what saturating it revealed

Version 1 stood essentially unbeaten for five years. Then in December 2024, OpenAI's o3 scored 75.7% at low compute and 87.5% at high compute — clearing the human-level threshold ARC Prize had set.

Chollet's response on the day was notably two-sided. He called it "a significant breakthrough in getting AI to adapt to novel tasks," and in nearly the same breath cautioned that "ARC-AGI is not an acid test for AGI. Passing ARC-AGI does not equate to achieving AGI, and, as a matter of fact, I don't think o3 is AGI yet."

That caveat turned out to be the important half. The low-compute run cost about $20 per task; the high-compute run that produced 87.5% cost roughly **$ 3,460 per task** — around $346,000 for the evaluation. The score came from massive test-time search, effectively brute-forcing the space of candidate programs, rather than from the cheap, fluid rule-inference the benchmark was built to reward.

So a test designed to measure efficient generalization got beaten by spending enormous compute on exhaustive enumeration. That's a genuine engineering achievement and simultaneously not the thing being measured. The benchmark had a loophole, and the loophole was money.

ARC-AGI-2: closing the loophole

Launched in March 2025, version 2 was explicitly built against that failure mode: tasks resistant to brute-force program search, requiring compositional reasoning and context-dependent rule application that doesn't yield to enumeration.

The reset was brutal and immediate. o3, fresh off 87.5% on v1, scored roughly 3% on v2. Whatever the v1 result had demonstrated, it transferred almost not at all.

ARC Prize established v2's human baseline with a panel of 400+ testers: every task was solvable by at least two people, and the average individual tester scored around 60%. That number matters for reading the chart above — Opus 5's 90.4% sits well above the average human, but the human ceiling on v2 is 100% by construction, since every task was verified solvable. The honest summary is that v2 is no longer a wall for frontier models, though it isn't fully conquered either.

ARC-AGI-3: not a puzzle set at all

Version 3, launched March 25, 2026 at a Y Combinator fireside with Chollet and Sam Altman, abandons the format entirely.

Where v1 and v2 are single-shot — here's a grid, return a grid — ARC-AGI-3 is a collection of hundreds of turn-based interactive environments, each handcrafted by human game designers, closer to small video game levels than puzzles. An agent gets a 64×64 color grid, a small fixed set of discrete actions, and critically: no instructions, no stated goal, no explanation of the rules. It has to act, observe what changed, form hypotheses about the mechanics, test them, build a working model of the environment, and work out what winning even means — entirely through interaction. Formally, these are POMDPs: partially observable environments requiring exploration, planning and memory.

Scoring is by RHAE — Relative Human Action Efficiency. Per level, an agent's score is (human actions ÷ agent actions)², capped at 1.15, normalized across environments. The squaring matters: fumbling toward a solution with extra actions collapses your score even if you eventually get there. Efficiency, not just success.

Humans handle this natively; it's roughly what playing an unfamiliar game is. At launch, essentially every frontier model scored under 1% — Gemini 3.1 Pro at 0.37%, GPT-5.4 at 0.26%, Opus 4.6 at 0.25%, Grok 4.20 at 0%.

That gap is a category difference, not a difficulty setting. Static grid tasks ask whether a model can infer a fixed rule from examples. Interactive environments ask whether it can build and act on a world model under genuine uncertainty — the thing an agent actually has to do. Nothing about solving v1 or v2 implied the capacity to do that, and the near-zero scores made the point unambiguously.

Where things actually stand

This is where the first version of this post went wrong — it named a leader on v1 and v2 from numbers I couldn't source — so let me be precise about what's actually known, and about what "leading" even means here.

On ARC-AGI-1 and -2 there is no clean single leader: a tight cluster sits near the top and the real competition is on cost. ARC Prize's own verdict on Opus 5's 97.5% and 90.4% was that these are "competitive with previous SOTA performance for slightly higher cost" — matching the frontier, not pulling away from it. GPT-5.5 at high effort is the closest rival on v2, at or near 90% depending on the source; Gemini 3.1 Pro sits clearly below both, somewhere in the high-60s to high-70s. I'm hedging those rival figures deliberately: third-party trackers and ARC Prize's own leaderboard disagree by several points, because they're measuring different eval sets at different effort settings. When a benchmark is this close to its ceiling, the meaningful question stops being who scores highest and becomes who gets there for $0.70 a task instead of$ 10.

ARC-AGI-3 is the exception — the only one of the three with a decisive leader. Opus 5's 30.2% is roughly four times the previous state of the art, GPT-5.6 Sol at maximum reasoning effort at 7.8%. Everything else remains near the floor.

The catch: the harness is part of the score

Except that "Opus 5 scores 30.2%" is shorthand for something more specific — Opus 5, running inside ARC Prize's harness, scores 30.2%. That distinction turns out to matter enormously.

In July 2026 a team introduced Schema, a harness that has frontier models write each environment's mechanics as an executable program, verify that program against recorded history, and then plan by searching inside it. Their reported result: ~99% RHAE on the ARC-AGI-3 public set — and in a controlled comparison, lifting a Claude Code baseline from 42.83% to 98.98% on the 25 public games. Notably, they got there with Opus 4.8 and Fable 5, not Opus 5.

Two caveats do a lot of work: those numbers are self-reported and not verified by ARC Prize, and they're on the public set rather than the semi-private evaluation that backs the official leaderboard. Public games are known to the harness's authors in a way private ones aren't, and the gap between a 25-game public set and the private set is exactly where benchmark results usually go to die.

But strip the caveats and the shape is unmistakable, because we've seen it twice already. On ARC-AGI-1, external search machinery plus enormous compute converted into a score that the raw model couldn't reach. Schema is the same move in a new costume: not a better model, but better scaffolding — world-model synthesis plus program search — bolted around models that score far worse on their own. If that result survives verification on the private set, the honest conclusion is that ARC-AGI-3 is partially susceptible to the same class of attack v1 was, and that the v3 leaderboard is ranking model-harness pairs rather than models.

The most interesting detail isn't the score, though. ARC Prize's analysis observed Opus 5 doing something they hadn't seen before: translating visual puzzle layouts into explicit algebra. On action 23 it described a scene as 4_center = 2×axis − 5_center — a reflection equation, derived unprompted. By action 248 it had generalized to two dimensions: "ghost = reflection about axis strips: br' = 2·br_axis − br, bc' = 2·bc_axis − bc." A model formalizing a geometric transformation mid-task, in an environment where it was never told the rules, is a qualitatively different observation than a benchmark number.

One outlier worth naming for scale-skeptics: the Tiny Recursion Model, roughly 7 million parameters, outperforms models thousands of times its size on v1 and v2 through pure recursive latent refinement. Architecture matched to task can beat brute scale by orders of magnitude on this particular kind of problem.

What it means

The three-version history is really one lesson, learned twice.

Saturating ARC-AGI-1 did not mean what the headlines said it meant. It meant a specific benchmark had a specific exploitable structure, and enough test-time compute could exploit it. Chollet said as much on day one and was largely talked over. Version 2 closed that hole; version 3 changed the question entirely — and each reset revealed that the previous "solved" result had been narrower than it looked.

Which is why ARC-AGI-3 is the one worth watching now — and why the Schema result is worth watching just as closely. Right now v3 is the least saturated benchmark in this family, and Opus 5 quadrupling the ceiling while spontaneously deriving reflection equations is genuinely hard to explain away as memorization. But a harness reaching ~99% on the public set, using older models, is precisely the early signature of the v1 story repeating: capability arriving from the scaffolding rather than the model, and a number going up faster than understanding does.

That's not a reason for cynicism about the benchmark. It's an argument for reading it correctly. What ARC-AGI actually measures well is the gap between a system and a human on tasks nobody could have memorized — and its three-version history shows that gap closing repeatedly for reasons that turned out to be less profound than the headline suggested. The useful discipline is to ask, every time a score jumps: is this the model, or the machinery around it? On v1 the answer was machinery. On v3 we don't know yet.

Chollet's 2024 caveat still stands, and now applies to his own newest benchmark: it is not an acid test for AGI. But an instrument that keeps catching us mistaking machinery for intelligence is doing something valuable — even if what it mostly measures is our own eagerness to declare victory.