NVIDIA's Flare prepares models in PyTorch and serves them from a Rust runtime

NVIDIA has a new public repository, flare-local-inference, created on 20 October under Apache-2.0. Its README describes Flare as “a local inference server for NVIDIA SM120/SM121 GPUs, built for long conversations, coding agents, and repeated tool use,” aimed at workstations and compact systems “such as DGX Spark.” We found no NVIDIA announcement; the repository is the source.

Python to prepare, Rust to serve

The design splits the work in two. A preparation tool called Ignition uses PyTorch, Transformers and FlashInfer/b12x to load a model, tune kernels, compile forwards and capture CUDA graphs into a runtime package. Helios, which the README calls “the Rust runtime, serves that package without Python.” All compilation and graph capture finish before serving starts.

The model preparation guide adds detail:

• A runtime package is a signed bundle of content-addressed blobs: weights, graphs, kernels and frontend assets. The final format uses a signed envelope with encrypted cubins and graph records, and unsigned staging packages cannot load in Helios.
• Every recipe defaults to FP8 E4M3 KV-cache storage. The guide names Qwen, GLM and DeepSeek recipes, each keeping some model-specific formats.
• Each tensor-parallel size needs its own package. The guide says Qwen at TP1 and TP2 and GLM at TP4 have been run end to end, and Qwen at TP4 still needs validation.
• Conversation state, including recurrent state alongside KV caches, is treated as one unit so reuse, branching and resuming keep it intact. That is the part aimed at agents that come back to the same long context.

What it does not claim

The README is unusually plain about limits. “vLLM covers a much broader range of models, hardware, and deployment patterns. Flare trades that breadth for a focused serving contract around selected models and local workloads.” It adds that Rust gives the runtime explicit resource ownership, but “performance still depends on the GPU work, memory traffic, and scheduling,” and tells readers to “compare the engines on your actual model and workload.”

Two cautions. The hardware scope is SM120 and SM121 only. And GitHub lists Python as the repository’s main language: the Rust is the serving runtime, not the whole project. For another take on local agent inference, see Magnitude.