Buoyant open-sources Kinference, a Rust inference control plane for Kubernetes and vLLM

Buoyant, the company behind the Linkerd service mesh, launched Kinference on 4 November. It is an open-source control plane for running large language models on your own GPUs in Kubernetes, sitting on top of vLLM. The repository is almost entirely Rust and licensed Apache-2.0. The vLLM project covered the launch on its own blog the same day.

What it does

Kinference does not serve models itself. The README describes it as a component that receives an inference request, for example in OpenAI chat-completions format, and returns which vLLM endpoint should handle it, along with a possibly rewritten request body. It is designed to sit behind a proxy such as Buoyant’s own gateway. Clients point at one HTTP endpoint, much like an LLM gateway.

To pick an endpoint it watches vLLM’s KV-cache events and engine metrics. Cache events are handled with NVIDIA’s dynamo-kv-router crate, the same project family we covered in Dynamo 1.6’s Rust endpoint picker. The point is to send a request to the worker that already holds its prompt prefix in cache, which is why the KV cache is a memory budget worth planning around.

Configuration lives in Kubernetes custom resources. The central one is an inference policy: a rule that decides how a request is routed or rejected based on the tenant, the model asked for and the state of the GPU pool. Buoyant’s examples include sending easy prompts to a cheaper model, running batch work only when GPUs are idle, bursting to a third-party provider when the pool is saturated, and keeping prompts with personal data inside your environment. A kinfctl CLI validates policy YAML without a cluster, which Buoyant says is meant for coding agents that write the config.

It also exports token rates, time to first token, inter-token and end-to-end latency, cached versus uncached prompt tokens, and GPU saturation, most of them per tenant.

The numbers and the gaps

Buoyant says that against standard Kubernetes Service load balancing, Kinference cut median time to first token by 95% and median end-to-end latency by 45% on one workload type. That is Buoyant’s benchmark, and the baseline is ordinary Kubernetes routing, not another cache-aware router such as llm-d.

The launch post lists token rate limiting and Anthropic-compatible traffic among its features. The README is more careful: it calls Kinference alpha software that “has not seen sustained production exposure”, and says Anthropic-style /v1/messages, rate limiting, semantic classifiers and multiple replicas are not implemented yet. Trust the README.

Why this matters

If you already run vLLM on Kubernetes for several teams, Kinference is a Rust option for the missing middle layer: who gets which GPUs, and when a request should go elsewhere. Policies as CRDs mean routing rules can be reviewed in git like any other config. Before you rely on it, note the single-replica limit and the missing rate limiting, and compare it with llm-d on your own traffic rather than against the plain Service baseline.