agent-desktop 1.0 lets coding agents drive desktop apps through the accessibility tree

agent-desktop from Vercel Labs is a command-line tool, written in Rust, that lets an AI agent read and operate native desktop apps through the operating system’s accessibility tree rather than through screenshots. Version 1.0.0 was tagged on 28 October, followed by 1.0.1 the same day and 1.0.2 on 30 October. It is Apache-2.0.

How it works

• Snapshots with refs: the agent asks for a snapshot of an app’s accessibility tree and gets back elements with short references. It then clicks, types or reads by ref. Refs belong to the snapshot they came from; if the app has changed, the tool returns errors such as STALE_REF or AMBIGUOUS_TARGET instead of guessing.
• Headless by default: actions go through accessibility APIs, without moving your pointer or stealing focus. --headed switches to physical input for apps that need it.
• Platforms: macOS, Windows and Linux (through AT-SPI).
• Install: npm install -g agent-desktop. There is also an FFI library for calling it from other languages.
• Retry rules: the README tells agents to retry only when an error says retry is safe, which cuts down on repeated clicks.

The 1.0 tags themselves are mostly about packaging; 1.0.2 prepares a move to a new GitHub organization, so expect the repository address to change.

The privacy catch

The README warns that snapshots can contain private content from whatever is on screen, such as messages, documents or account details. Everything in a snapshot can end up in the model’s context and with your model provider, and a bundled helper sends screen descriptions to a provider too. Run it against a test account or a clean desktop first.

Desktop control is a busy area: GitHub shipped Copilot computer use in preview, and Microsoft released Mage-VL-Cowork, a small model that clicks the desktop. agent-desktop takes the structured route instead of pixels.

Why this matters

Reading the accessibility tree gives an agent named buttons and fields, which is usually more reliable and cheaper than guessing from screenshots. agent-desktop wraps that in one CLI across three operating systems, with clear errors when the screen changes under the agent. The same access means everything on screen can reach the model, so it needs the care you would give any tool that reads your desktop.