Inherent Laboratories released a paper on Faraday, a 27B-parameter "AI Scientist" agent trained to replicate results from published machine learning and AI-for-science papers — reproducing a figure from a paper with a redacted result, on a 60-minute time budget and a single slice of an H200 GPU, without seeing the original plot. The headline finding: Faraday, post-trained on this task, beats both Claude Opus 4.8 and GPT-5.5 (Codex) according to the paper's own judge — including on paper topics it never trained on. The more interesting engineering detail sits underneath that headline: Faraday is a 27B model that works by directing Codex GPT-5.5 — an agent the paper estimates at roughly 5 trillion parameters — as a tool, and it outperforms that much larger model's own standalone attempts at the same task.
Why replication is a harder training target than it looks
Paper replication doesn't have a single correct answer to check against. A paper compresses the research that produced it, so an agent replicating it has to infer missing details, decide what to scale down under time and compute limits, and judge whether a scaled-down version still tests the paper's actual claim — closer to open-ended research than to a task with a verifiable pass/fail signal. That's the specific problem the paper is solving for: most existing "AI Scientist" training setups either need a fixed reward function that only works for problems with a hill-climbable metric, or fall back to fully open-ended LLM judging (closer to peer review) that's noisy and hard to validate.
The task space: 310 auto-generated replication tasks from 100 real papers
The paper's benchmark, called Replica, is built from 100 well-known ML and AI-for-science papers published between 1990 and 2026, yielding 310 tasks (242 for training, 68 held out for testing) — one task per results figure, generated automatically by using Gemini 2.5 Pro to locate and redact each figure from the paper's PDF, then hand-checked to filter out badly redacted or misidentified figures. Each task hands the agent the paper with one figure removed, a 60-minute wall-clock limit, and a single one-seventh MIG slice of an H200 GPU — explicitly not enough to run the original experiment at its original scale for most papers, which forces the agent to decide what a faithful, scaled-down version of the experiment actually looks like rather than just running a smaller version blindly.
The reward problem: a rubric judge, and honest reporting on how well it works
Because there's no ground-truth pass/fail signal for a replication, the paper trains a per-task grading rubric (generated by Claude Opus 4.7, without seeing the real figure, so it grades the underlying claim rather than cosmetic plot details) and scores rollouts with a Codex-based judge against that rubric. The paper is unusually direct about testing this judge against actual human researchers rather than assuming it works: in a validation study with PhD-level raters, the rubric judge agreed with human rankings at a Kendall correlation of 0.19, against 0.15 for a simpler constant-prompt baseline judge. Worth sitting with directly rather than rounding up: 0.19 is a real but modest edge, and it's also worth noting the two human raters in this study agreed with each other at only 0.30 — meaning the task itself is noisy for humans too, and the judge is being validated against a genuinely hard-to-pin-down target, not a clean gold standard. Where the rubric judge does show a clearer advantage is self-consistency: two independent draws of the rubric judge agreed with each other at 0.66, well above two independent human raters' 0.30, which is the paper's actual basis for using it as a lower-variance RL reward signal rather than as a claim that it reasons like a human grader would.
The result, counted directly: Faraday wins most of the time, not all of it
Faraday — Qwen3.6-27B, LoRA-finetuned with a turn-level-credit variant of GRPO on the Replica training split — outperforms both Claude Opus 4.8 and Codex GPT-5.5 on 73% of in-distribution ML tasks and 60% of held-out AI-for-science tasks, per the paper's own judge, with average score improvements of 6% over Claude and 8% over Codex on the test split. The paper also tested whether prompting alone could close this gap: 24 generations of automated prompt optimization on the Codex baseline (Claude Opus 4.7 rewriting the prompt against rollout feedback each generation) left Codex's score essentially unchanged, meaning the gap didn't come from a fixable prompt weakness. Qualitatively, the paper's own hand-reviewed examples are specific and checkable in spirit even if the underlying rollouts aren't public: on a Voyager replication task, the best Claude rollout reportedly supplied a pre-populated skill library instead of running the skill-acquisition process the figure was meant to test, while Faraday ran the actual acquisition-and-transfer process the paper describes — a real qualitative distinction between demonstrating an output and reproducing the mechanism, if the reported example holds up.
Where the paper is careful not to overclaim
Two of the paper's own follow-up tests are worth reading as deliberately modest rather than as confirmations. First, a human preference study: researchers selected 41 rollouts specifically from cases where the rubric judge said Faraday held a strong advantage, and asked humans to compare — humans preferred Faraday over both baselines in 29 of 41. The paper explicitly states this design "does not allow us to draw any conclusions as to whether humans prefer Faraday over Claude and Codex on average," since the sample was selected for cases favoring Faraday rather than drawn at random — a caveat worth taking at face value rather than reading the 29/41 figure as a general preference rate. Second, a generalization test on eight papers requiring larger compute budgets (up to eight hours and eight B300 GPUs): Faraday beat Claude on five of eight, which the authors call suggestive of generalization while explicitly noting that a clearer claim would need the rubric judge separately validated at that larger scale, since it was built and checked against the smaller-budget task distribution. The paper also solicited feedback from the original authors of four replicated papers directly, and reported the mixed results plainly rather than only the praise — quoting real criticisms alongside the positive notes, including that some of Faraday's write-up quality and code style fell short even where the underlying replication was sound.
What this is, and isn't, evidence of
This is a preprint from an industry lab (Inherent Laboratories), submitted August 13, 2026 — not yet peer-reviewed, evaluated against a benchmark and a judge the same team built, and not yet independently reproduced outside the paper. That combination doesn't make the result wrong, but it's worth naming plainly rather than treating a company's own rubric-judged comparison of its own model as equivalent to third-party validation. What the paper does substantiate well, on its own terms, is the structural claim: a 27B model, post-trained specifically to decide what to investigate and how to scope an experiment, can direct a much larger general-purpose coding agent to better results than that agent achieves on its own — without anyone needing to hand-design a specialized multi-agent harness for the task.
What to expect next
- Watch for independent reproduction of the headline comparison. The rubric judge and the Replica task set are the paper's own construction; a third party re-running the same comparison, or building an independent judge, is what would settle whether the 73%/60% win rates hold up.
- Watch for the rubric judge validated at larger compute scale. The authors flag this gap themselves — the eight-paper generalization test used a judge that was only checked against the smaller default task budget.
- Watch whether the "coding agent as a tool" pattern shows up elsewhere. The paper's own framing — a smaller post-trained model directing a much larger frontier model as a tool, and beating that larger model's unaided attempts — is a distinct and testable architectural claim independent of the replication benchmark itself.
References: arXiv — Training AI Scientists to Replicate Research (2608.13331) · Inherent Laboratories — Training to Replicate · Inherent (@inherent_labs) on X — announcement thread · related coverage: Frontier Arcade: trends & predictions