AWS and Berkeley open-source Infinity, a Rust agent runtime that suspends agents at every tool call

Most agent harnesses run a simple loop: call the model, call a tool, wait for the tool, repeat. While the agent waits for a twenty-minute build or a reviewer’s reply, it holds a process and its memory. Infinity, released on 23 October under Apache-2.0, does it differently. When an agent calls a tool, the runtime saves the conversation and stops the agent. When the result arrives, it loads the conversation and carries on.

Infinity is part of Hydro, a Rust distributed-systems project run jointly by AWS and UC Berkeley. Shadaj Laddad introduced it on the AWS Open Source Blog as a research project and “an early research release”, shared to get feedback.

Agents as coroutines

The idea comes from how programming languages handle async functions. A stackless coroutine can be paused at an await and resumed later, because everything it needs is saved. AWS argues an agent works the same way: at the moment it calls a tool, its conversation history holds everything needed to continue.

So Infinity runs each agent as a series of short slices. A slice loads the history from a store, runs one model completion, saves the output and sends off at most one tool call. Then it ends. Tool calls are the await points, and a slice never blocks on a tool, a timer or anything else.

Tools are written to match. In Infinity, a tool’s execute method starts the work and returns at once. When the work is done, the tool puts its result on the agent’s input queue, which starts the next slice. If the lookup takes 100 milliseconds, the agent carries on almost at once. If it takes three days, the agent is only stored history in the meantime.

The numbers

AWS measured memory with a benchmark of 80,000 agents, each running 20 turns against a simulated model, for 3.2 million completions on one thread of an AMD EPYC server. Memory grew by about 100 KB per idle agent, mostly the conversation histories. At that rate, about 75,000 agents fit in 8 GB, the memory of a Raspberry Pi 5. That Pi figure is arithmetic from the server run, not a test on a Pi, and with a real model the histories, and so the memory, will be larger.

Serverless without waiting

Because a slice loads all its state from storage and never waits, it can run on serverless platforms. The reference AWS deployment uses:

• an SQS FIFO queue as the input queue, with one message group per conversation thread, so each thread’s slices run in order without distributed locks
• a Lambda function that runs one step per delivery and exits
• Aurora DSQL for conversation histories and DynamoDB for deduplication
• SQS delivery delays and EventBridge Scheduler for timers

The repository includes CDK constructs for all of it. Messages carry stable IDs so a redelivered message is skipped, and streamed model output is only saved once the turn completes. The point, in AWS’s words, is that “an agent waiting on a three-day CI pipeline uses no compute until the pipeline reports its result.”

Events as fake tool calls

Model APIs expect every tool call to get exactly one result, which makes ongoing events awkward, for example a subscription that reports every new pull request. Infinity’s answer is a synthetic tool call. When an event arrives, the runtime adds a tool call the model never made, named receive_event__injected, that points back at the original call, followed by a result holding the event. The model never sees that name as a tool it can call, and the runtime rejects it if the model tries.

The same mechanism handles sub-agents. A built-in spawn_thread tool creates a child agent, and the child’s reports come back to the parent as events on that call. By default each event is even handled in a short-lived child thread, so a hundred events do not fill the parent’s context.

What is in the repo

Besides the runtime, the repository has the Reactive Agent Protocol (RAP), a tool protocol that delivers results and events even to agents that have no running process. Existing MCP servers work through a compatibility layer. There is also Infinity Code, a coding agent that runs builds and tests in the background and edits in parallel threads.

AWS lists open questions itself. Models are mostly trained on traces where tool calls are synchronous, so it is not yet clear how well they reason about events arriving out of order. Scheduling is first-come, first-served for now. The repo had 25 stars when we checked. For a Rust runtime aimed at very large numbers of waiting agents, the design is worth reading even if you never deploy it.