Researchers from Tsinghua University, MIT, Harvard, Carnegie Mellon, Microsoft Research, and a dozen other institutions released ASI-Bench — "At the Dawn of Artificial Superintelligence" — detailed in a paper on arXiv, with code on GitHub and a summary from Synced (机器之心), which first flagged the release on X. It's a benchmark of 60 project-level research tasks across 11 scientific domains, from mathematics and chemistry to robotics and electrical engineering. Built by more than 40 domain experts at a reported cost of over 31,000 human-hours, it's designed to answer a narrower and harder question than most science benchmarks ask: not whether a model knows the right answer, but whether it can independently run an entire research project when the amount of human guidance is progressively taken away.
What B1 through B4 actually strip away
Each ASI-Bench task keeps the underlying research problem, data, and scoring criteria fixed, and varies only how much methodological guidance the agent receives. In B1, the governing method and procedure are handed over explicitly — the agent mostly has to implement and execute a prescribed approach. In B2, the full procedure is removed and the agent gets only the name of the method, needing to work out how to actually carry it out. In B3, even that's gone: the agent sees only the raw data and the research objective, and has to determine the underlying model or method itself. B4 repeats the B3 conditions but adds plausible, task-irrelevant information, testing whether agents stay on task under distraction. Completing one of these tasks isn't quick, either — the paper reports an average of over 2,600 interaction turns and 2,400 execution steps per task, spanning more than 35 hours of agent execution.
The real bottleneck: turning a method into a procedure, not choosing one
Across 18 evaluated model-and-harness combinations, average scores drop from 50.91 in B1 to 29.10 in B2 to 26.62 in B3. The size of each drop is the actual finding: losing the detailed procedure (B1→B2) costs 21.82 points, while losing the method name itself on top of that (B2→B3) costs only a further 2.48 points. Adding irrelevant distractor information in B4 barely moved the average at all (26.99 versus 26.62). In the authors' own words, this identifies "method operationalization, rather than method selection or distraction, as the primary bottleneck to autonomous scientific research" — current systems aren't mainly failing at deciding what to try, they're failing at turning a stated approach into a working, executable research procedure.
No configuration clears 52%, and cost doesn't reliably buy more
The strongest setup tested — Codex paired with GPT-5.6 Sol (ultra) — is "the sole evaluated system to exceed 50" in the fully autonomous B3 setting, at 51.60%. Claude Opus 5 with the Claude Code harness leads by a wide margin under full guidance (72.29 in B1) but falls to 40.70 once guidance disappears, a steeper relative decline than the strongest GPT-5.6 configuration shows. Cost doesn't track performance cleanly either: Codex with GPT-5.6 (xhigh reasoning) reaches a similar B3 score, 40.86%, for roughly $684 per run — about a quarter of Claude Opus 5's $2,728 per-run cost for a near-identical result. Spending more can still buy real gains, though: pushing GPT-5.6 Sol to its highest ("ultra") reasoning tier is what produces the benchmark's best B3 score, at roughly $1,550 per run.
The most expensive setting isn't the most autonomous one
The token and time costs behave counterintuitively across guidance levels. B1 (full procedure given) is the cheapest, at 4.35 million tokens and 37.8 minutes per task on average. B3 and B4 — where the agent must determine its own method — cost 25% and 30% more tokens respectively. But B2, not B3 or B4, is the most expensive setting of all: 6.91 million tokens and 49.7 minutes per task, 59% more tokens and 32% more time than B1. Being told which method to use without being told how to execute it, the authors argue, is structurally the worst combination — the agent is constrained to someone else's chosen approach while still having to reconstruct every procedural detail from scratch.
The agent harness matters almost as much as the model
One more result worth flagging on its own: the same backbone model can score very differently depending on the agent scaffolding wrapped around it. MiMo V2.5 Pro improves from 16.17 under its own MiMo Code harness to 23.25 under Claude Code; Kimi K2.7 rises from 19.72 to 27.34 the same way. But the effect isn't universal — Kimi K3 barely moves between harnesses (36.22 versus 37.09). That's a smaller-scale echo of the same pattern this blog just covered in GPT-6 Astra's ARC-AGI-3 results: as models get more capable, more of what used to require external scaffolding starts happening inside the model itself, and the harness stops being the deciding factor.
The construction process is unusually rigorous for a new benchmark
ASI-Bench's own account of how it was built is worth taking at face value as a differentiator: more than 1,300 candidate research problems were screened down to 60 through five rounds of expert review, over 1,100 review assignments, more than 2,000 task revisions, and over 1,500 end-to-end sandbox validation runs to catch scientific inconsistencies, unreliable scoring, or unintended shortcuts before a task made the final cut. That's a heavier quality-control pipeline than most new benchmarks disclose, though it's still one research consortium's judgment calls about task design and scoring — worth independent replication over time rather than taken as a settled standard on release day.
What to expect next
- Watch whether frontier labs start reporting ASI-Bench B3 scores specifically, rather than the easier B1 setting most existing benchmarks resemble. The authors frame B3 as the actual measure of autonomous research capability — if that framing sticks, it becomes the number worth citing.
- Watch whether the B2 cost spike holds up as more agents get tested against ASI-Bench, or turns out to be specific to how these particular 18 configurations handle partial guidance.
- Watch for outside task contributions. ASI-Bench is explicitly soliciting new tasks from the community at its submission site — whether that actually broadens the benchmark's scientific coverage, or stays mostly the work of the founding consortium, will shape how much it's trusted as shared infrastructure rather than one lab's leaderboard.