OpenMOSS released SWE-bench Science, a benchmark that asks a different question than the SWE-bench family has asked so far. Where SWE-bench Verified and its successors check whether an agent's patch makes a held-out test suite pass, SWE-bench Science checks whether an agent's patch preserves the domain-specific correctness a scientific-software repository actually depends on — units, coordinate systems, numerical invariants, file-format semantics, physical assumptions, and the workflows that connect them. A patch can pass every test in a repository and still be wrong in a way that would quietly corrupt a research result, and that's the gap this benchmark is built to expose.
What's being measured, and why it's a different axis
Software that computes physical quantities, transforms coordinate systems, or parses domain-specific file formats has failure modes generic software doesn't: a unit conversion silently dropped, an axis order swapped, a numerical tolerance loosened past the point where it still means anything. A test suite written before those constraints were made explicit won't necessarily catch a change that violates them. That is the argument for treating "does it pass CI" and "does it preserve the scientific contract" as two different questions, both worth measuring. That framing is worth taking on its own terms regardless of where the specific pass rates land, since it targets a failure mode that a pure test-pass metric structurally can't see.
Scale and construction
The benchmark comprises 119 release tasks drawn from 98 GitHub repositories across 20 scientific domains. Of those, 96 ship under unrestricted licenses as the default evaluation set, with 23 additional tasks under GPL/LGPL/AGPL-family licenses available only through explicit opt-in. That's a deliberate, disclosed choice to keep the default set legally uncomplicated rather than silently including copyleft-licensed code in a default benchmark run. A separate 91-task subset is marked for a "science-knowledge ablation" — testing what happens to agent performance when domain-specific scientific context is deliberately supplied or withheld — though the project's repository doesn't spell out the exact protocol.
Each task pins to a fixed baseline commit and ships as two separate Docker images: an environment image carrying the baseline source, fixtures, and dependencies an agent works against, and a verifier image that applies the agent's candidate patch to a clean rebuild of that baseline and runs held-out tests against it. Evaluation runs through a tool called Pier, with pinned image digests and recorded run configurations for reproducibility. That two-image separation matters methodologically: the agent never has direct access to the verification environment while working, which closes off the easiest way a harness could otherwise game a fixed test suite it can see.
The one number available, and what couldn't be confirmed
The headline result reported for this benchmark: the best-performing configuration found so far, Claude Code running Opus 5 at maximum reasoning effort, scores below 50% pass@1. Worth being precise about the limits of that statement: the sub-50% figure rests on secondary corroboration rather than a direct read of the leaderboard. A full leaderboard breakdown across other agents and models (Codex, mini-swe-agent, and other frontier models the project's tooling explicitly supports) could not be verified, so it isn't reported here rather than guessed at.
One more small inconsistency is worth flagging — the kind this coverage checks for on every self-published benchmark. The project's own top-level README states the default/restricted split as 96/23, while a separate documentation file in the same repository states it as 97/22. The two figures don't reconcile against a stated primary total — exactly the kind of precision gap that's easy for a reader to overlook and easy for a project to fix.
Not yet peer-reviewed
This is an arXiv preprint, submitted this month — an important distinction for a paper making methodological claims about how scientific software should be evaluated. Preprint status doesn't make the benchmark design or the reported result wrong, but it means the task construction, the invariant-checking methodology, and the headline pass rate haven't yet gone through independent peer review. That step is worth waiting for before treating any of this as a settled measurement rather than a proposed one.
What to expect next
- Watch for peer review. A benchmark whose entire premise is precision about scientific correctness should be held to that same precision itself — peer review is the mechanism for that.
- Watch for the full leaderboard becoming independently readable. The per-model comparison chart is published as an image; a text-based leaderboard table would make results like this independently checkable.
- Watch whether other labs report scores on this benchmark. A single self-reported "best configuration" number is a starting point, not a ranking — third-party runs against the same fixed baselines and verifier images are what would make this comparable to the rest of the SWE-bench family.