Inherent Laboratories released a paper on Faraday, a 27B-parameter "AI Scientist" agent trained to replicate results from published machine learning and AI-for-science papers. The task: reproduce a results figure from a paper with that result redacted, on a 60-minute time budget and a single slice of an H200 GPU, without ever seeing the original plot. The headline finding: Faraday, post-trained on this task, beats both Claude Opus 4.8 and GPT-5.5 (Codex) according to the paper's own judge — including on paper topics it never trained on. The more interesting engineering detail sits underneath that headline. Faraday is a 27B model that works by directing Codex GPT-5.5 — an agent the paper estimates at roughly 5 trillion parameters — as a tool, and it outperforms that much larger model's own standalone attempts at the same task.
Why replication is a harder training target than it looks
Paper replication doesn't have a single correct answer to check against. A paper compresses the research that produced it, so an agent replicating it has to infer missing details, decide what to scale down under time and compute limits, and judge whether a scaled-down version still tests the paper's actual claim. That is closer to open-ended research than to a task with a verifiable pass/fail signal. That's the specific problem the paper is solving for: most existing "AI Scientist" training setups either need a fixed reward function that only works for problems with a hill-climbable metric, or fall back to fully open-ended LLM judging (closer to peer review) that's noisy and hard to validate.
The task space: 310 auto-generated replication tasks from 100 real papers
The paper's benchmark, called Replica, is built from 100 well-known ML and AI-for-science papers published between 1990 and 2026, yielding 310 tasks (242 for training, 68 held out for testing). Each task corresponds to one results figure, generated automatically by using Gemini 2.5 Pro to locate and redact that figure from the paper's PDF, then hand-checked to filter out badly redacted or misidentified figures. The agent gets the paper with one figure removed, a 60-minute wall-clock limit, and a single one-seventh MIG slice of an H200 GPU. For most papers that is explicitly not enough to run the original experiment at its original scale, which forces the agent to decide what a faithful, scaled-down version of the experiment actually looks like rather than just running a smaller version blindly.
The reward problem: a rubric judge, and honest reporting on how well it works
Because there's no ground-truth pass/fail signal for a replication, the paper trains a per-task grading rubric and scores rollouts with a Codex-based judge against that rubric. The rubric is generated by Claude Opus 4.7 without seeing the real figure, so it grades the underlying claim rather than cosmetic plot details. The paper is unusually direct about testing this judge against actual human researchers rather than assuming it works: in a validation study with PhD-level raters, the rubric judge agreed with human rankings at a Kendall correlation of 0.19, against 0.15 for a simpler constant-prompt baseline judge. That 0.19 is a real but modest edge, worth sitting with rather than rounding up. It's also worth noting the two human raters in this study agreed with each other at only 0.30 — the task itself is noisy for humans too, so the judge is being validated against a genuinely hard-to-pin-down target, not a clean gold standard. Where the rubric judge does show a clearer advantage is self-consistency: two independent draws of the rubric judge agreed with each other at 0.66, well above two independent human raters' 0.30. That, not any claim that it reasons like a human grader would, is the paper's actual basis for using it as a lower-variance RL reward signal.
The result, counted directly: Faraday wins most of the time, not all of it
Faraday — Qwen3.6-27B, LoRA-finetuned with a turn-level-credit variant of GRPO on the Replica training split — outperforms both Claude Opus 4.8 and Codex GPT-5.5 on 73% of in-distribution ML tasks and 60% of held-out AI-for-science tasks, per the paper's own judge. Average score improvements are 6% over Claude and 8% over Codex on the test split. The paper also tested whether prompting alone could close this gap: 24 generations of automated prompt optimization on the Codex baseline (Claude Opus 4.7 rewriting the prompt against rollout feedback each generation) left Codex's score essentially unchanged, meaning the gap didn't come from a fixable prompt weakness. Qualitatively, the paper's own hand-reviewed examples are specific and checkable in spirit even if the underlying rollouts aren't public. On a Voyager replication task, the best Claude rollout reportedly supplied a pre-populated skill library instead of running the skill-acquisition process the figure was meant to test, while Faraday ran the actual acquisition-and-transfer process the paper describes. If the reported example holds up, that is a real qualitative distinction between demonstrating an output and reproducing the mechanism.
Where the paper is careful not to overclaim
Two of the paper's own follow-up tests are worth reading as deliberately modest rather than as confirmations. First, a human preference study. Researchers selected 41 rollouts specifically from cases where the rubric judge said Faraday held a strong advantage, and asked humans to compare; humans preferred Faraday over both baselines in 29 of 41. The paper explicitly states this design "does not allow us to draw any conclusions as to whether humans prefer Faraday over Claude and Codex on average," since the sample was selected for cases favoring Faraday rather than drawn at random. That caveat is worth taking at face value rather than reading the 29/41 figure as a general preference rate. Second, a generalization test on eight papers requiring larger compute budgets, up to eight hours and eight B300 GPUs. Faraday beat Claude on five of eight, which the authors call suggestive of generalization — while explicitly noting that a clearer claim would need the rubric judge separately validated at that larger scale, since it was built and checked against the smaller-budget task distribution. The paper also solicited feedback from the original authors of four replicated papers directly, and reported the mixed results plainly rather than only the praise. It quotes real criticisms alongside the positive notes, including that some of Faraday's write-up quality and code style fell short even where the underlying replication was sound.
What this is, and isn't, evidence of
This is a preprint from an industry lab (Inherent Laboratories), submitted August 13, 2026 — not yet peer-reviewed, evaluated against a benchmark and a judge the same team built, and not yet independently reproduced outside the paper. That combination doesn't make the result wrong, but it's worth naming plainly rather than treating a company's own rubric-judged comparison of its own model as equivalent to third-party validation. What the paper does substantiate well, on its own terms, is the structural claim: a 27B model, post-trained specifically to decide what to investigate and how to scope an experiment, can direct a much larger general-purpose coding agent to better results than that agent achieves on its own — without anyone needing to hand-design a specialized multi-agent harness for the task.
What to expect next
- Watch for independent reproduction of the headline comparison. The rubric judge and the Replica task set are the paper's own construction; a third party re-running the same comparison, or building an independent judge, is what would settle whether the 73%/60% win rates hold up.
- Watch for the rubric judge validated at larger compute scale. The authors flag this gap themselves — the eight-paper generalization test used a judge that was only checked against the smaller default task budget.
- Watch whether the "coding agent as a tool" pattern shows up elsewhere. The paper's own framing — a smaller post-trained model directing a much larger frontier model as a tool, and beating that larger model's unaided attempts — is a distinct and testable architectural claim independent of the replication benchmark itself.