The Best AI Agent Reproduces Half a Biology Paper's Analysis — Edison's Own Data Says the Other Half Is the Hard Part

Edison Scientific — the commercial AI-for-science spinout of the nonprofit lab FutureHouse, which built the original BixBench — introduced BixBench3, a benchmark that asks AI agents to reproduce the complete analysis pipeline behind a published computational biology paper — starting from raw experimental data and producing the same structured artifacts (read-count matrices, differential-expression tables, pathway-enrichment results) the original researchers generated, graded programmatically against the published reference. Edison calls it "the longest-horizon biology benchmark reported to date," and the scale backs that up: an average attempt ran 6.8 hours, processed 102 million tokens, and cost $43; the largest single run consumed 1.07 billion tokens over 24 hours for $525.

What the benchmark actually asks for

Each of the 20 tasks is built from a real bioRxiv-style paper, decomposed into a directed acyclic graph of artifacts — 138 in total across the benchmark, averaging 4–14 per task. Of those, 56 come directly from raw data (a read-count matrix from raw .fastq files), 44 require a second analysis stage (testing for differential gene expression), and 38 require three or more stages (pathway enrichment). Datasets range from 7 to 241 GB. This is a meaningfully different design than most agent benchmarks: the scientist supplies the research question and a general methodological plan, and the agent is graded on executing that plan correctly through every downstream stage, not on choosing what to investigate in the first place — a scope limit Edison states explicitly rather than leaving implicit.

The scores, and how tight the top actually is

The top four models — spanning OpenAI, Moonshot, Zhipu, and Anthropic — land within 0.02 of each other, worth naming directly: this isn't one lab pulling ahead on a genuinely novel long-horizon task, it's a near-tie across very differently built models. But Edison's own breakdown shows that closeness hides real divergence underneath: GLM 5.2 and GPT 5.5 differ by only 0.02 overall, yet scored 0.5 and 0.25 respectively on one specific task combining ChIP-seq and gene-expression data to study Polycomb/RING1 effects in Drosophila — a 2x gap the aggregate number completely erases. Same average, different actual competence.

Where performance actually breaks down

The clearest, most specific finding in Edison's own data: performance declines as analysis moves further downstream. Artifacts produced directly from raw data scored 0.30 on average across all models; artifacts requiring three or more analysis stages scored 0.24. That's the real shape of the result — agents are more reliable at the first step of a pipeline than at the steps that depend on it, which is exactly the failure mode you'd expect from errors compounding across a long chain of dependent analyses rather than being independent per-step.

Edison's own failure-mode tagging (via a separate LLM judge reviewing each run's trace and outputs) backs this up with numbers: the number of failure tags per run correlates strongly with low scores (Spearman's ρ = −0.92), and premature termination and repetitive retry loops specifically were roughly 2.0x and 2.2x more common in the worst-scoring quarter of runs than across all runs. Among the best-scoring quarter, only one run out of 65 ended prematurely and none entered a retry loop. That's a genuinely useful, specific diagnostic — not just "the model failed," but which failure modes actually predict it.

Cost varied 367-fold, and cheap wasn't worse

Average cost per task ranged from $0.35 to $129.14 across models — a 367-fold spread — and the relationship between cost and score wasn't linear. Kimi K3 scored just 0.01 below the top model while costing roughly 55% less.

Worth crediting directly: Edison caught and disclosed a real measurement artifact in its own data rather than reporting a misleading number unqualified. GLM 5.2 was accessed via OpenRouter, which spreads requests across multiple hosting providers with separate caches — so GLM 5.2's cache-hit rate came out to 41.4% against 96.0% for every other model, inflating its measured cost through a routing artifact rather than a real inefficiency in the model itself. Edison reports both the raw paid cost and an adjusted estimate at the same cache-hit rate as the other models, and states plainly that this "reflects our OpenRouter configuration rather than an inherent limitation of GLM 5.2." That's the kind of self-caught measurement caveat that's easy to omit and genuinely useful to disclose.

The prediction is Edison's, not the benchmark's

Edison's own announcement states: "Based on current performance and the rate of improvement, we expect AI agents will be able to write papers on their own (given data and a general plan) in as little as 6 months." Worth being precise about what that claim rests on and what it doesn't. The measured result is that the best model reproduces 48% of requested artifacts on average, with accuracy specifically declining at the deeper analytical stages that a fully autonomous paper would depend on most — the exact part of the pipeline where BixBench3's own data shows the most room to close. A six-month timeline to full autonomous paper-writing is Edison's extrapolation from a trend line, not a figure the benchmark itself measured, and it's worth reading as a forecast made by a company whose own products — Kosmos, a world-model system aimed at exactly this kind of long-research-horizon coherence, and ether0, a chemistry-focused model — have a direct stake in that trend continuing.

What to expect next

• Watch for independent replication of the top-four clustering. Four different labs landing within 0.02 of each other on a genuinely hard, novel benchmark is worth confirming isn't an artifact of this specific task set before treating it as a real capability ceiling.
• Watch the artifact-depth gap specifically. The 0.30-vs-0.24 split between shallow and deep artifacts is the most concrete number in this release for tracking whether agents actually close the downstream-reliability gap Edison's own six-month prediction depends on.
• Watch whether BixBench3 becomes a recurring benchmark with tracked scores over time, the way SWE-bench and similar coding benchmarks have — a single snapshot can't confirm or refute a rate-of-improvement claim; a second run in six months actually could.

References: Edison Labs — Benchmarks · FutureHouse — Announcing BixBench (the original benchmark BixBench3 extends) · content relayed from Edison Scientific's own announcement text and charts, provided directly rather than fetched (this environment could not independently retrieve the source page) · related coverage: Frontier Arcade: trends & predictions