Topic · 36 stories · page 2 of 2
36 stories tagged inference on AI Tamer, newest first.
Memory requests place a Pod. Limits constrain its containers. Learn how to tell a container out-of-memory kill from node-pressure eviction and size inference workloads accordingly.
Use a Kubernetes init container and a shared volume to put model files in place before an inference server starts. Plan for retries, storage use, and model loading.
A PodDisruptionBudget can limit how many inference pods a node drain evicts at once. Its allowed-disruptions count shows when the workload has room for another eviction.
Inference requests can wait while CPU utilization stays below its scaling target. Measure pending work and queue delay to decide when more workers are useful.
A sudden inference surge meets several gates before new Pods serve traffic. Follow the autoscaler's measurement loop, replica rules, Node capacity, and readiness.
Both PyTorch modes skip gradient recording. Inference mode also changes the tensors it creates, which matters when later code needs autograd.
A small model can draft the next words while a larger model checks them. Speculative decoding can shorten the wait when enough drafts survive the check.
Modal (Oct 1, 2026) made Clusters generally available via @modal.clustered for multi-node GPU jobs. RDMA is optional; Volumes, Cloud Bucket Mounts, and Queues stay in the loop. No dollar rates on the post.
EricLBuehler/mistral.rs (not Mistral AI) merged PR #2462 on 2026-10-01: Qwen3.8-Flash-Next from safetensors and GGUF with built-in MTP. Not a new release tag. Paged attention requires CUDA; FP8 KV / TP / GGUF MTP / external drafters unsupported.
warpfront tagged hipfire v0.4.0 (2026-09-30): Rust LLM inference over HIP kernels for AMD RDNA. Self-reported ≈5,120 tok/s pp8192 on R9700; Qwen3.8-Flash-Next full 262K on one R9700. Serve defaults to 127.0.0.1. Apache-2.0 (residual MIT files).
A practical guide to model routers, inference providers, media APIs and model makers, with the trade-offs to check before sending production data.
A practical guide to model weight precision, memory estimates, common quantization formats, quality checks and choosing a quant for your workload.
