LiteLLM swaps its 18,000-line Python router for a Rust core in a dev pre-release

LiteLLM, the open-source LLM gateway and SDK, has moved its router onto Rust. Three pull requests merged on 5 and 6 November add a Rust router core, delete the Python one and move the proxy’s Redis traffic to native Rust stores. All three ship in the v1.110.0-dev.1 pre-release from 7 November. The latest stable release, v1.108.2, does not list them.

What moved to Rust

PR #49513 adds a litellm-router-core crate that owns the deployment registry, model selection, retries, fallbacks, cooldowns and health tracking. Python stays as the “host”: it shapes each request, makes every provider call and runs callbacks. The core calls out to the host for each attempt and nothing else.

PR #50178 then deleted router_python_core.py, about 18,000 lines. Router now always builds the Rust core, the LITELLM_RUST switch for the router is gone, and a configuration the Rust core declines raises an error instead of falling back to Python. The author reports about 670 tests that reached into Python internals were deleted, about 160 rewritten, and 8,954 router tests passing.

PR #50170 moves router cooldowns and usage, spend counters, the auth cache and the v3 rate limiter onto native Rust Redis stores. Keys, values and commands stay the same, so pods on old and new versions can share one Redis during a rollout.

What it does not claim

The PRs do not promise faster routing: loading 2,000 deployments took 1.7 s on the Rust core against 1.5 to 1.8 s on the Python core. LiteLLM’s X account said “routing is fully in Rust”, which holds for routing decisions on the dev build, not for provider calls. The separate Rust AI Gateway docs still describe an opt-in, mostly Python proxy, because they cover provider translation, a different track.

Why this matters

This follows the same pattern as Anthropic’s Rust port of the Messages API: keep Python at the edges, move the hot, stateful core to Rust behind parity tests. If you run LiteLLM with custom routing code or tests that touch router internals, try v1.110.0-dev.1 in staging now, since the Python fallback is gone. For why the failover path matters as much as model choice, see Routing is two systems.