Topic · 5 stories
5 stories tagged speculative-decoding on AI Tamer, newest first.
Velo Research open-sourced Kinetic, a Rust drop-in for llama.cpp's tools. Its big speedups are on file-rewrite tasks; plain generation is about level.
EAGLE-4 adapts its draft tree to batch size and hardware. It needs patched engine builds, and commercial use stops being free above $20M in revenue.
Mirai's open-source Rust engine uzu uses tree speculative decoding tuned for M5 Macs. A creator run shows 92 tokens/s on a 9B model versus 22 for llama.cpp, with caveats on quants, chips and workload.
A small model can draft tokens for a larger model to verify. The speed gain depends on how quickly it drafts and how often its proposals are accepted.
A small model can draft the next words while a larger model checks them. Speculative decoding can shorten the wait when enough drafts survive the check.
