Topic · 36 stories · page 1 of 2
36 stories tagged inference on AI Tamer, newest first.
TokenGate, from Stanford and Berkeley researchers, trains a gate that drops unneeded context tokens early, cutting prefill time and memory on small models.
Dynamo 1.6.0 adds pluggable router queues, multi-replica Rust EPP and coding-agent session routing. One default change touches Claude Code traffic.
One developer rewrote NVIDIA's streaming RE-USE model in Rust and measured 8.4 ms per 20 ms frame on an M3 CPU, against 16.9 ms through WebGPU.
NVIDIA posted Flare, an Apache-2.0 local inference server for SM120/SM121 GPUs such as DGX Spark. Python prepares each model; a Rust runtime serves it.
EAGLE-4 adapts its draft tree to batch size and hardware. It needs patched engine builds, and commercial use stops being free above $20M in revenue.
TorchTPU, built with Meta's PyTorch team, is open source. Google adds TPU backends for vLLM, SGLang, Ray and VeRL, and device_map="tpu" in Transformers.
doubleAI's Maxwell optimizes programs while they run and says it formally proves each rewrite equivalent. It launched with a $200M Series A disclosure.
llm-d 1.0, a CNCF Sandbox project started by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, adds flow control, router HA and DisaggregatedSet.
DeepInfer is Apache-2.0 and runs V4.1-Flash on NVIDIA SM100 or Huawei Ascend 950. This release is offline inference only, without KV cache or load balancing.
Respan's 15 October post releases Span-02. The card says the weights run only on its Rust and CUDA engine, on Blackwell GPUs.
Thinking Machines' 15 October note marks a year of Tinker. Compatible sampling is already in beta. Three models retire on 23 October 2026.
At OCP, Intel named Crescent Island the GPU D-series. GPU D270 is a 350 W air-cooled PCIe card. Specs are Intel's. No price or ship date.
NVIDIA and Tri Dao's lab describe ReplaySSM, which caches recent SSM inputs instead of writing full state every step. The speedups are the authors' benchmarks.
Scott Guthrie says Azure Maia 300, Microsoft's third in-house inference chip, begins production deployments in 2027 and is meant to be the company's main inference system. The efficiency figures are Microsoft's.
River Inference is serving DeepSeek V4.1 Flash and GLM 5.3 Flash. Its public card is 20% under DeepSeek's off-peak rates and Z.ai's list prices. A second homepage table is much higher.
MIG cuts a supported NVIDIA GPU into up to seven instances with their own memory, cache and compute. How to create them and pick a profile for a model server.
Dropout changes a model’s forward pass according to module mode. Learn how masking, scaling, evaluation mode and gradient recording affect a reliable inference path.
Mamba changes a recurrent state according to each input. That keeps generation memory bounded by the model state, while making recall depend on what the state retained.
Grouped-query attention gives several query heads one key-value head pair. That shrinks decoding caches, with speed and quality gains that depend on the model and workload.
Mirai's open-source Rust engine uzu uses tree speculative decoding tuned for M5 Macs. A creator run shows 92 tokens/s on a 9B model versus 22 for llama.cpp, with caveats on quants, chips and workload.
Long prompts can fill GPU memory through the key-value cache. Moving that cache to the CPU saves GPU space, with a possible speed cost.
A small model can draft tokens for a larger model to verify. The speed gain depends on how quickly it drafts and how often its proposals are accepted.
A model’s weights are only part of its memory needs. The attention cache grows with the conversation, and saving space can slow generation.
Continuous batching lets a serving system replace finished requests while longer responses keep generating. Here is how that changes GPU use and what limits it.
