Topic · 8 stories
8 stories tagged latency on AI Tamer, newest first.
Anthropic's message threads beta keeps a conversation server-side for 5 minutes, so agent loops send only new messages. Prices unchanged; not ZDR eligible.
A Lua script can keep a Redis read-and-update decision together, but every millisecond of execution holds up other server activity. Bound the work before using EVAL.
A small model can draft tokens for a larger model to verify. The speed gain depends on how quickly it drafts and how often its proposals are accepted.
Continuous batching lets a serving system replace finished requests while longer responses keep generating. Here is how that changes GPU use and what limits it.
Averaging latency percentiles across replicas can misstate an endpoint’s performance. Histograms let you combine observations before calculating p95.
A response can be slow to start or slow to continue. vLLM exposes separate metrics for those delays, making each easier to investigate.
A small model can draft the next words while a larger model checks them. Speculative decoding can shorten the wait when enough drafts survive the check.
A practical map of speech recognition, language models, speech synthesis, direct speech-to-speech models, streaming, and the latency measurements that matter.
