Chamath Palihapitiya published Model Benchmarks Are Worthless, an article built on a study he asked the 8090 Labs team to run. Leaderboards score models alone, so 8090 Labs measured the model and its harness together: 7 models in 5 harnesses (Codex, Claude Code, OpenCode, Pi and the DeepSeek harness) on Datacurve's DeepSWE v1.1, 113 real bug fixes and features from open-source projects, with 2 trials per task and data as of September 27. The headline: one model scores 24% or 59% on the same 113 tasks depending on the harness around it.
The spread by model
From Chamath's article (8090 Labs, Datacurve DeepSWE v1.1).
Each model ran at one fixed reasoning setting. The table gives its worst and best harness.
| Model | Worst harness | Best harness | Spread (pts) |
|---|---|---|---|
| GLM 5.3 Flash | Codex, 24% | DeepSeek harness, 59% | 35 |
| Claude Fable 5.1 | Claude Code, 41% | Codex, 61% | 20 |
| DeepSeek V4 Flash | OpenCode, 22% | Codex, 39% | 17 |
| GPT-5.6 Luna | OpenCode, 53% | Pi, 64% | 11 |
| GPT-5.6 Terra | OpenCode, 60% | DeepSeek harness, 69% | 9 |
| GPT-5.6 Sol | OpenCode, 69% | Pi, 75% | 6 |
| Grok 4.6 | Pi, 64% | Claude Code, 69% | 5 |
GLM 5.3 Flash is the most harness-sensitive model, with a 35-point swing that amounts to a 2.5x success rate (59% against 24%). Claude Fable 5.1 follows at 20 points: 41% in Claude Code against 61% in Codex, with a median of 28 tool calls in one harness and 66 in the other. GPT-5.6 Sol and Grok 4.6 barely cared, at 6 and 5 points.
No harness wins everywhere. Codex is the best harness for Fable 5.1 and DeepSeek V4 Flash but the worst for GLM 5.3 Flash. Claude Code is the worst for Fable 5.1 and the best for Grok 4.6. OpenCode is the worst for four of the seven models (DeepSeek V4 Flash, Luna, Terra and Sol), while Pi tops Luna and Sol and the DeepSeek harness tops GLM and Terra.
GLM 5.3 Flash across five harnesses
GLM 5.3 Flash (high reasoning), share of runs passed. From Chamath's article.
At high reasoning, GLM passes 24% of runs in Codex, 26% in Claude Code and 27% in OpenCode, then jumps to 58% in Pi and 59% in the DeepSeek harness. The harnesses split into two clusters 31 to 35 points apart, and the model never changes.
Tasks passed in both, one or neither trial. From Chamath's article.
Consistency follows the same pattern. Tasks passed on both attempts number 16 in Codex, 14 in Claude Code and 16 in OpenCode, against 50 in Pi and 51 in the DeepSeek harness. In Codex the split across 113 tasks is 16 passed both, 22 passed one and 75 passed neither; in Pi it is 50, 32 and 31; in the DeepSeek harness 51, 31 and 31.
What a passing run costs
GLM 5.3 Flash (high reasoning), at Z.AI's promotional rate. From Chamath's article.
The cost chart prices GLM at Z.AI's promotional rate, which ended September 9; list rates are twice as high. A verified task includes the cost of failed runs.
| Harness | Success | Per attempt | Per verified task |
|---|---|---|---|
| Codex | 24% | $0.11 | $0.46 |
| Claude Code | 26% | $0.29 | $1.10 |
| OpenCode | 27% | $0.44 | $1.64 |
| Pi | 58% | $0.13 | $0.22 |
| DeepSeek harness | 59% | $0.19 | $0.32 |
Chamath's text quotes the same measurements at list prices: Codex costs $0.22 a run and $0.93 per success, Pi $0.25 a run and $0.43 per success. That is exactly double the chart ($0.11 becomes $0.22, $0.46 becomes about $0.93, $0.13 becomes about $0.25 and $0.22 becomes about $0.43).
The ratios hold at either price. Pi costs slightly more per attempt than Codex ($0.13 against $0.11 promotional, $0.25 against $0.22 at list) yet is about 2.1x cheaper per verified task ($0.22 against $0.46, or $0.43 against $0.93 at list). OpenCode costs about 7.5x more than Pi per verified task on the chart ($1.64 against $0.22). Cheap attempts that mostly fail are not cheap work.
Chamath's conclusion
Chamath writes that he has increasingly ignored model-only leaderboard results, partly from exhaustion and partly because a rise in a ranking does not show up in the actual work. His bottom line: choosing a model and harness combination from a model-only leaderboard means making a critical decision on half the information.
Context on this blog:
- DeepSeek Harness: the open-source harness that tops the GLM score chart here.
- Meta's Muse Wonder launch: DeepSWE 1.1 appears as the long-horizon coding row in its benchmark table.
- Claude Fable 5.1 and Mythos 5.1: the model that moves from 41% to 61% between harnesses.
- GPT-6 Sol and Luna: the successors to the GPT-5.6 Sol and Luna measured here.