llm-d reaches 1.0: Kubernetes-native distributed inference gets v1 APIs, flow control and safe upgrades

llm-d 1.0 is out. The project turns a cluster of model servers (vLLM, SGLang) into one coordinated serving pool on Kubernetes, and it is now a Cloud Native Computing Foundation Sandbox project. IBM’s release names Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA as founders and counts more than 500 contributors from over 25 organizations. Google announced it at the PyTorch Conference. The code is Apache-2.0.

What 1.0 changes

The release notes group the work into four themes:

• Multi-tenant quality of service. Priority- and fairness-aware flow control with controlled load shedding, so latency-sensitive requests and bursty agent loops can share a fleet. Config is consolidated under flowControl.
• Zero-downtime operations. Router high availability with a hot standby, and DisaggregatedSet, which manages prefill and decode replicas as one unit with lockstep upgrades.
• A stable base. The router configuration API graduates to v1, and the Responses API supports previous_response_id.
• Agent-aware serving. Session affinity and turn-priority scheduling keep multi-turn conversations near their cached context.

There is also a new async processor that pulls latency-tolerant work from queues such as Pub/Sub or Redis and runs it in spare capacity, and Prism, a dashboard for exploring published benchmark runs.

Two housekeeping items matter if you upgrade. All components now share one version number (v1.0.0 across the org). And the project is deprecating its own inference images in favour of upstream vLLM, SGLang and other images. The ROCm image is already deprecated.

The numbers, and whose they are

The release notes carry no performance figures. The benchmarks come from the companies:

• Google says that on an agentic coding workload on Ironwood TPUs, prefix-aware routing plus host-memory offload cut median time to first token by more than 85% and raised throughput 2.4x against a plain Kubernetes Service. It also says its Gemini platform doubled its prefix-cache hit rate from 35% to 70% after moving traffic through GKE Inference Gateway, which runs the llm-d router.
• IBM’s release cites demonstrations of five to 10 times lower cost for context-heavy agentic workloads, including a GLM 5.2 deployment on NVIDIA GPUs that served up to 3,000 concurrent coding agents.

Each links to its own write-up. None is an independent test, and each is tuned for its author’s hardware.

Why it matters

The hard part of serving agents is not one fast server. It is keeping each agent’s next step on the replica that already holds its context, and stopping one bursty loop from starving everyone else. llm-d has offered pieces of that since 2025. What 1.0 adds is a stable API and upgrades that do not drop sessions, the parts an operations team asks for before putting it in front of production traffic.