Agents finished database sign-up in 16.6% of 1,035 tries; captchas were the main wall

Scope, a Y Combinator startup that sells “agent experience” audits, has published a benchmark of a task many developers now hand to coding agents: sign up for a hosted database and start using it, with no human help.

The test

Each session gave an agent its own cloud sandbox, a fresh email inbox and a password, then one line of instructions: create a database with a named product, add a table with one specific row, and return the connection string. A session only counted if Scope could read that row back with the credentials the agent returned. An account or API key alone did not count, and neither did the agent saying it had worked. Scope ran 1,035 scored sessions from 2 to 6 November across 18 providers and 20 agent configurations, including Claude Code, Codex, Cursor, OpenCode and OpenClaw with various models, up to three runs per product and agent.

The results

Only 172 sessions, 16.6%, ended with a working database. On the leaderboard, Prisma Postgres led at 88.3%, then Upstash at 63.3%, Neon and Nile at 43.3% and ClickHouse Cloud at 38.3%. Turso got 5.0%, Cloudflare D1 and MongoDB Atlas 3.3%, and Supabase, Appwrite Cloud and Nhost 1.7%. Seven products had no successful session at all: Aiven, CockroachDB Cloud, Convex, Render, SingleStore Helios, Tiger Cloud and Xata.

Scope says the product mattered more than the agent. Success ranged from 0% to 88.3% across products but only from 3.7% to 29.6% across agent setups. It attributes 67% of failed sessions to a wall the product put up, against 16% to agent mistakes. Captchas stopped 373 sessions and bot checks another 105. Other blocks were sign-ups that rejected the agent’s email domain (51), Google or GitHub-only login (32) and verification links that never got used (31).

The biggest help was a database the agent could use before anyone created an account. Fifty of the 172 successes came that way. Prisma’s npx create-db makes a temporary Postgres database with no sign-up, and in one transcript Scope shows an agent that hit Neon’s captcha switching to Neon’s own account-less instant database instead.

The caveats

Scope sells this kind of testing, so it has a reason to show that sign-ups are broken. Samples are small: at most three sessions per agent on each product, and Scope says most of the agent ranking is not statistically ordered. Agents ran from a cloud sandbox with an agent email domain, which some products refuse. And sign-up flows change often, so this is a snapshot of one week.

Why this matters

If your team lets coding agents provision infrastructure, pick providers with an API or CLI path and a claimable trial database, or expect to step in at the captcha. If you build a developer tool, this is a cheap test to copy: give an agent an inbox and your sign-up page, and see where it stops. Agent-created databases are already happening at scale, as Tencent Cloud’s TDSQL-C numbers show, and the Supabase and Turso deal was pitched at the same agent market.