Topic · 5 stories
5 stories tagged local-inference on AI Tamer, newest first.
The Rust local inference server now transcribes and labels speakers on the CPU behind OpenAI-style routes. Its 0.9.0 and 0.9.1 downloads shipped without CUDA.
Open Linux modules lift GeForce FP16 and BF16 tensor issue rates toward 2x on RTX 30 and 40 cards. Owners report 14 to 36% faster LLM prefill, not decode.
hipfire 0.4.3 adds a new mq4rs quant for Qwen3.6-35B-A3B with much faster prefill on gfx1201. 0.4.3.1 fixes a 27B MTP load bug and lifts decode about 48%.
Magnitude is an Apache-2.0 on-device inference engine for local agents: desktop app plus magnitude CLI, on-hardware kernel compile and tune, shared prefix cache, one-click agents or an OpenAI-compatible API.
A practical guide to choosing a local model runtime, understanding GGUF and memory needs, and deciding when a desktop setup is enough or a server makes sense.
