Prime Intellect ran 153 autonomous runs across 18 frontier models on the nanoGPT optimizer speedrun — a live test of whether models can conduct genuine AI research, not just apply a known recipe. Each run got a shared 124M-parameter training script, a rulebook, one message, an 8xH200 GPU node, and up to eight days of wall-clock time, with zero internet access. The task: beat a human-tuned baseline of 3,290 training steps (verified under Prime Intellect's own statistical bar; the leaderboard's version is 3,250) toward a public human record of 2,600, changing only optimizer-related hyperparameters. This directly answers the open question we flagged when Prime Intellect's Prime Agent harness first crossed the ARC-AGI-3 human line by a coincidental 0.1 points: whether the gain generalizes past one benchmark. It does — and what it reveals along the way is more interesting than the leaderboard.
Scale, for context
This is a meaningfully larger experiment than anything comparable published so far: Anthropic's internal automated AI R&D eval optimizes on a CPU node; OpenAI's GPT-5.6 Sol system card reports nanoGPT Track 1 on a single H100 for under a day; METR ran six agents against a wall-clock speedrun capped at five days and $10K each; Intology's NanoGPT-Bench gives agents about 2.7 days on one 8xH100 node and finds they recover less than 10% of five months of human progress. Prime Intellect's runs went up to eight days on 8xH200s, across 18 models, with multiple seeds per model — and published everything: traces, scratchpads, reasoning streams for the open-weights models, hourly monitor reports, and the harness itself.
The headline result, and the equal-budget check that makes it credible
Fable 5 closed 81.7% of the human-record gap, reaching 2,726 steps — the clear leader by a wide margin. Opus 5 and Kimi K3 followed at roughly 52–54%. The rest of the field — GPT-5.6 Sol, Grok 4.5, GLM 5.2, DeepSeek V4 Pro, Qwen3.8 Max, Muse Spark, Kimi K2.7 — clustered well below 40%. What makes this more than a raw leaderboard artifact is Prime Intellect's equal-budget comparison: giving every model's best run the same fixed budget, in agent-hours, in experiment count, or in output tokens, barely reorders the ranking. Fable 5 and Opus 5 lead under all three framings. That rules out the easy objection that the top models simply burned more compute — the gap is in what they did with a given budget, not how much budget they had.
The finding that matters more than the ranking: nobody found anything new
Prime Intellect states this plainly rather than burying it: none of the 153 runs produced a fundamentally new optimization method. The winning ingredients — better preconditioning, caps and floors on weight/update magnitude, learning-rate schedules held hot longer, late-training weight averaging — are all techniques already in the literature. Every model, strong or weak, converged on roughly the same idea space. The entire gap between tiers came from research process, not idea generation: which experiments to prioritize, how carefully to run them, and how to interpret a noisy result. That's a genuinely uncomfortable finding for anyone treating "autonomous AI research" claims as evidence of emerging novel discovery — the models here demonstrate research execution skill without research creativity, and the report is honest that it can't tell whether that's a property of this specific speedrun setup or a real capability ceiling.
What actually separated the tiers: noise modeling and revisiting old failures
The clearest evidence of genuine research skill is a detail Prime Intellect deliberately withheld: the noise estimate given to every model in the rulebook was slightly too large on purpose. 62 of roughly 100 runs measured the actual noise themselves rather than trusting the stated figure, and those runs cluster at the top of the results table. 42 went further and discovered, unprompted, that rerunning the identical recipe on the identical seed still moves the loss — GPU non-determinism, smaller than seed-to-seed noise but real — and several models rebuilt their entire screening protocol around comparing recipes on a shared seed to resolve differences a normal screen couldn't detect for the same compute cost. Nobody told them this existed.
The second differentiator is how models treated negative results. Weaker models killed entire idea families off a single noisy seed, treated their own crashes as proof an idea was bad, and discarded small gains that didn't individually clear significance. Grok 4.5 lost row normalization twice because of its own scaling bugs — a specific, attributable failure of blaming the method for a bug in the implementation. The strongest models did the opposite: testing borderline results on three seeds before committing to eight, re-ablating the full stack after every merge to catch ideas that stopped helping, and explicitly revisiting old negatives once the recipe changed, on the reasoning that something inert before might matter now. Opus 5 re-opened β2 tuning under a new recipe and it became a new record. Kimi K3 deleted two mechanisms that had produced its own previous record, once a new normalization made them redundant. That's the actual skill being measured here — not "more ideas," but knowing which negative results still apply after the ground shifts.
Prime Agent's persistent kernel let models build their own tooling
Several runs used Prime Agent, Prime Intellect's own harness — a persistent IPython kernel as the model's only tool, the same Recursive Language Model design we covered when it first shipped. Kimi K3 built its own functions for constructing controlled optimizer variants, launching runs, comparing loss curves, and restoring a clean baseline — then, later in the same persistent session, built a separate numerical laboratory to retune Newton–Schulz coefficients cheaply on CPU before spending a GPU run to test them, and revised its hypothesis when the theoretically cleaner update lost in actual training. DeepSeek V4 Pro 0813 did something similar: screening candidate PSGD recursions in a synthetic covariance simulation across several self-driven revisions before deciding a full run was worth the compute. That's a genuinely different behavior from "the harness makes the model better at the benchmark" — it's the model using a general-purpose tool to construct its own research infrastructure mid-task, unprompted.
The anti-cheating design is worth naming
Claiming a record requires an eight-seed run beating a mean of 3.27859 against the 3.28 bar — a margin that makes passing by luck alone roughly one in a thousand. Each run is sandboxed (bwrap plus a network namespace) with the only outside route being a logging proxy that permits the model's own API and nothing else — no general internet access, deliberately, after earlier experiments showed models got less creative when they could search for existing PRs to imitate rather than deriving methods themselves. An independent LLM monitor audited every run hourly; after hundreds of reports with no cheating or sandbox escapes found, Prime Intellect stopped running it continuously. Worth flagging plainly: some models in the results table — Qwen3.8 Max, DeepSeek V4 Pro, Grok 4.6, Muse Spark 1.2, GLM 5.3 — are marked "RUNNING" or show no final record at publication time, so treat their current standings as provisional, not final rankings.
What to expect next
- Watch for the "still running" models' final numbers. Grok 4.6, GLM 5.3, and others were incomplete at publication — the current tier ordering for those specific models isn't settled yet.
- Watch whether the no-new-ideas finding replicates on a different research task. This is one speedrun with one clear reward signal; whether the same "execution beats creativity" pattern holds on a less well-trodden research problem is the real test of what's actually being measured here.
- Watch for partial-internet-access follow-ups. Prime Intellect explicitly flags this as a planned next step, having found zero internet access produced more creative (if not more novel) results than earlier runs with arXiv search enabled — worth testing directly rather than assumed.
References: Prime Intellect — Measuring Autonomous Research · Prime Intellect — NanoGPT Speedrun Frontier results · @PrimeIntellect on X — announcement · PrimeIntellect-ai/prime-agent · related coverage: Prime Agent Crossed the Human Line by 0.1 Points · Two Harnesses Solved the Same Benchmark by Disagreeing About Memory · Frontier Arcade: trends & predictions