Topic · 5 stories
5 stories tagged local-llm on AI Tamer, newest first.
A source-available Rust engine for Qwen GGUF models. Its author measures 113 tok/s decode vs 66.5 for tuned llama.cpp on an RTX 3060; prefill is level.
A 9B multimodal model that stores 32 layers but executes 56, under Apache 2.0. Transformers only for now, with trust_remote_code. What the card says.
Prospero is a Dear ImGui coding harness that brings no model of its own: point it at llama.cpp or an API. Unlicense on GitHub, $25 Pro build. What it does.
Mirai published @trymirai/uzu-pi-extension 0.6.8 on 15 October. It adds uzu as a local provider for Pi on Apple Silicon Macs.
Mirai's open-source Rust engine uzu uses tree speculative decoding tuned for M5 Macs. A creator run shows 92 tokens/s on a 9B model versus 22 for llama.cpp, with caveats on quants, chips and workload.
