Topic · 5 stories
5 stories tagged kv-cache on AI Tamer, newest first.
Long prompts to GLM-5.3 on Workers AI could return text from other users' requests from 27 Aug to 9 Sep. Cloudflare and SGLang explain the bug and the fix.
A ByteDance Seed preprint says token offset can swing DeepSeek-V4 retrieval. The 40-point gap is the base checkpoint. Post-trained gaps are smaller.
Grouped-query attention gives several query heads one key-value head pair. That shrinks decoding caches, with speed and quality gains that depend on the model and workload.
Long prompts can fill GPU memory through the key-value cache. Moving that cache to the CPU saves GPU space, with a possible speed cost.
A model’s weights are only part of its memory needs. The attention cache grows with the conversation, and saving space can slow generation.
