Garry Tan, who runs Y Combinator, has published g-harness-evals, an open repository that asks one narrow question: if you hold the model and the tasks fixed and swap only the harness (system prompt, tools, memory), what changes? The repository was created on 5 November 2026 and is MIT licensed. If the word harness is new to you, our explainer covers what sits between a model and your terminal.
What was tested
Two models, each run through several harnesses on the same 20 public multi-user tasks:
- Opus 5.5: QM (a Pi loop), Claude Code, Hermes Agent and OpenClaw.
- GPT-6.1 Sol: Codex, Hermes Agent and OpenClaw.
Every cell is a plain harness with no add-ons. Cost per solved task is the cell’s whole spend at list price through the repo’s metering proxy, divided by its scored passes.
The headline: pass rates could not be separated
The README says pass rates “couldn’t be told apart in any tested pair” on these tasks, so it ranks by cost and speed instead. The 95% ranges are wide: a gap of under 7.7 to 14.2 points could hide inside them.
| Model | Harness | Pass rate | Cost per solved task | Median time per trial |
|---|---|---|---|---|
| Opus 5.5 | QM (Pi loop) | 72.5% | $0.19 | 40.1 s |
| Opus 5.5 | Claude Code | 76.3% | $0.22 | 40.1 s |
| Opus 5.5 | Hermes Agent | 78.8% | $0.24 | 64.6 s |
| Opus 5.5 | OpenClaw | 76.3% | $0.44 | 99.4 s |
| GPT-6.1 Sol | Codex | 71.3% | $0.17 | 30.1 s |
| GPT-6.1 Sol | Hermes Agent | 75.0% | $0.19 | 81.5 s |
| GPT-6.1 Sol | OpenClaw | 73.7% | $0.22 | 83.6 s |
Claude Code and Codex are each model’s own harness, so the repo never compares them with each other. On Opus, OpenClaw cost about twice the others, and the README traces about 81% of its extra cost over Claude Code to cache writes from a much bigger system prompt and bigger tool results.
Findings that matter more than the ranking
- Where you put a rule matters, and it differs by harness. A “check gbrain first” rule (gbrain is Tan’s shared-memory project) in Claude Code’s system prompt or startup file was followed in 30 of 30 turns. Codex followed it 30 of 30 wherever it sat. Hermes on Sol obeyed the hook and system prompt but ignored an unmentioned skill.
- Claude Code did not open a skill on its own. An unmentioned skill was opened 0 times in 30 turns; naming it in the prompt got 30 of 30.
- Shared memory without per-user access rules leaked. With one bot-wide token, reads returned records the asker was not allowed to see in 9 to 33 of 51 trials per cell. With per-member rules, 0 of 51.
The repo labels most of these as descriptive, from small samples, and says some intervals reach zero.
Caveats the repo states itself
These are dev results on public tasks at pinned versions, not held-out confirmation. QM ran on a different Docker network, so its comparisons are untested. Both OpenClaw cells are incomplete because the last task hit a cap. The disclosures section says gstack, gbrain and QM are Tan’s or YC’s projects, YC is an investor in Hermes, and OpenClaw’s creator works for OpenAI. Weigh the rows with that in mind.
Why this matters
On these 20 tasks, switching harness did not move pass rates by a measurable amount, but it moved cost per solved task by more than two times on Opus ($0.19 to $0.44). If you are picking a harness for a team, the cheaper check is your own bill and latency on your own tasks, not a leaderboard. The rule-placement and access findings are the practical part: put must-follow rules where your harness reads them every turn, and give shared agent memory per-user access rules before it holds anything private. For a broader model-plus-agent comparison, the Artificial Analysis Coding Agent Index v2 ranks setups on a different task set.
The Campfire
CommentsThanks. Your comment is waiting for the desk. Once it is approved it appears here within about a minute.
Add a comment