DrivingBench (Aditya Ramabadran, Simon Mahns, Tobias Gessler — equal contribution) gives frontier models control of a Toyota Corolla’s steering, accelerator, and brakes via tools, then scores them on a fixed cone course (site, report).
The results are self-published and have not been replicated by third parties. They do not show that a model “can drive,” that autonomous driving is solved, or road-ready AV competence: cone course only, low speed, empty lot, human ready to brake (the authors’ own caveats).
Protocol
Up to 3 attempts in one continuous chat. Progress = share of course centerline completed while staying within 4 m; a collision keeps pre-collision progress. Stack notes in the report include MCP tools observe, set_motion, stop_now, comma four + openpilot, and a tool/controller speed hard-limit (~0.5–3.5 m/s). Cone-course trial dated 2026-09-17 (report).
Live leaderboard (fetched 2026-09-24)
From drivingbench.com (Codex / Claude Code / Cursor · medium):
| Model | Best | Attempts |
|---|---|---|
| GPT-6 Astra (Codex · medium) | 100% in 5:22 | 49% (DNF), 100% / 5:22, third empty |
| Claude Fable 5.1 (Claude Code · medium) | 45% | 9%, 10%, 45% (all DNF) |
| Grok 4.6 (Cursor · medium) | 11% | 8%, 11%, 10% (all DNF) |
| GPT-5.6 Sol (Codex · medium) | 6% | 6%, 6%, 6% (all DNF) |
Astra was the only model to complete the course (attempt 2). Trace viewer: gpt-6-astra / 2. Numbers can change—as of this fetch.
Limits (authors)
Single multi-attempt trial per model in the same chat (not independent replicates). Reproducibility needs a Corolla + comma four + similar lot. Authors plan v2 with more runs / models / harder course. Open artifacts: harness, traces.
Who should care
Embodied-agent and eval teams watching real-car, human-supervised demos should read the site and report. Read it as self-published and limited to a cone course, not as evidence of an “ability to drive.”
The Campfire
CommentsThanks. Your comment is waiting for the desk. Once it is approved it appears here within about a minute.
Add a comment