Topic · 8 stories
8 stories tagged vllm on AI Tamer, newest first.
Linkerd's maker released Kinference: Apache-2.0 Rust control plane for vLLM on Kubernetes, with KV-cache-aware routing and policies. It is alpha.
IBM Research's vLLM-Hook MIA plugin captures attention and activations, and steers them, inside vLLM 0.29 with CUDA graphs left on. Overhead is a few percent.
EAGLE-4 adapts its draft tree to batch size and hardware. It needs patched engine builds, and commercial use stops being free above $20M in revenue.
TorchTPU, built with Meta's PyTorch team, is open source. Google adds TPU backends for vLLM, SGLang, Ray and VeRL, and device_map="tpu" in Transformers.
llm-d 1.0, a CNCF Sandbox project started by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, adds flow control, router HA and DisaggregatedSet.
JFrog rates CVE-2026-105192 at 9.8 for pickle on LMCache's multiprocess ZeroMQ socket when that socket is routable. No release removes the path. Pull request 5644 is open and unmerged.
A response can be slow to start or slow to continue. vLLM exposes separate metrics for those delays, making each easier to investigate.
A practical guide to choosing a local model runtime, understanding GGUF and memory needs, and deciding when a desktop setup is enough or a server makes sense.
