FrontierCS 2 asks coding agents to rebuild a paper's result, and beat it on every metric

Most coding benchmarks ask whether an agent’s patch makes the tests pass. FrontierCS 2, posted to arXiv on 21 October by Qiuyang Mang and 60 co-authors, including Ion Stoica, Matei Zaharia, Joseph Gonzalez and Ofir Press, asks something harder: can an agent rediscover a research result and do better than the people who published it?

How a task is built

Each of the 48 tasks starts from a paper’s released code, in 13 domains from operating systems and high-performance computing to databases, security and robotics. The authors delete the paper’s key contribution; the deleted code becomes the human reference. The agent inherits the rest of the repository and the problem statement, can submit a patch at any time, and gets a score on a development suite.

The evaluator, the held-out final suite and the reference stay private. A run passes only if its final submission beats the reference on every metric and on every hidden workload. Winning on average is not enough.

What the agents scored

The authors ran 12 agents, each once per task, in the harness each vendor ships: Claude Opus 5.5 in Claude Code 2.1.296, GPT-6 Astra and GPT-6.1 Sol in Codex CLI 0.162.1, Gemini 4 Argon in Pi 1.2.0, and open models such as GLM 5.3, DeepSeek V4.1 Flash, Kimi K3 and Qwen 3.8 Max in their own coding tools.

• Claude Opus 5.5 passes 58% of its runs, the best result. Every other agent passes fewer than half.
• No agent passes 16 of the 48 tasks.
• A higher score is not a pass. The authors say most runs that beat the reference on the combined final score still fall short on some metric or workload.
• The leader depends on budget. In the authors’ cost analysis, DeepSeek V4.1 Flash leads at the smallest budgets (its costs are lower bounds) and Opus 5.5 at the largest; GPT-6.1 Sol passes less often than GPT-6 Astra but costs far less.

The paper also places agents and tasks on one scale, the FrontierCS Capabilities Index, adapted from Epoch AI’s Capabilities Index and anchored at Kimi K2.7. On it, the median task is about as hard as what the strongest agent can handle, so the benchmark is not close to saturated.

Caveats

One run per task means a gap of a task or two is within noise. Refusals count as failures: the leaderboard’s notes say the model provider’s content filter ended 10 of Gemini 4 Argon’s 48 runs in Pi (the paper’s text mentions two refused runs, so check the site for the current count), and in one Opus 5.5 run a refusal switched the rest of the run to Claude Opus 4.8. The site also lists a second Argon run in Google’s Antigravity harness, where refused requests are retried.

The authors keep a private set of more than 100 further tasks for later. The leaderboard, last updated 25 October, takes new model scores by email.