ReplaySSM caches SSM inputs so hybrid decoding can skip a full state write

On 12 October 2026, NVIDIA’s technical blog published ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding. The paper it points to, ReplaySSM: Cache SSM Inputs, Not State, is arXiv:2610.13778, submitted 9 October 2026 at 15:59:40 UTC. The PDF marks Ze-Wei Liou and Tri Dao at Princeton University and Harris Nover, Amey Naik, Andrii Skliar, Igor Shovkun, and Ido Shahaf at NVIDIA. The NVIDIA post calls it joint work with the Dao AI Lab at Princeton.

What changes in the decode step

A hybrid model is mostly state-space layers, such as Mamba-2 or Gated DeltaNet, plus a smaller number of attention layers. The NVIDIA post uses Nemotron 3 Super as the example: five times as many SSM layers as attention layers. Each SSM step keeps a fixed-size state. Conventional decoding loads that state from high-bandwidth memory, updates it, computes the output, and writes the state back. At serving batch sizes the post says that traffic, not the attention layers, dominates the step out to long contexts, because the state is large and the arithmetic on it is small.

Speculative decoding makes the same design awkward. A drafter proposes several tokens. Attention can drop the rejected ones by moving a cache pointer. An SSM has already folded the drafts into its state, and that update has no cheap inverse. The post says engines such as vLLM keep a state snapshot after every draft token. Those snapshots add memory traffic and limit how many requests fit on the GPU. At serving batch sizes, the post says, speculative decoding barely helps these hybrids.

ReplaySSM stores a checkpoint plus a buffer of recent SSM inputs, and rebuilds what it needs from those instead of writing the full state every step. On most steps it appends the new inputs and computes the output from the checkpoint and the buffer. When the buffer fills, a flush folds the inputs into the checkpoint and writes that checkpoint once. Rollback of a rejected draft is a pointer move on the buffer, the post says, rather than a restore from per-token state snapshots. The Dao AI Lab post describes the same idea: cache recent inputs, reconstruct state when needed, write back when the buffer is full. A matching walkthrough is on Tri Dao’s site.

The paper and the NVIDIA post also describe a dual-precision checkpoint. A higher-precision copy is what the recurrence updates. A lower-precision copy is what most steps read to produce the output. The low-precision copy is not fed back into the next update, so its rounding error stays in the current window. The NVIDIA post says that on GPQA-Diamond this kept three of four models within about 2 percentage points of FP32 accuracy. That is the authors’ table.

The speedups, and which page says which

The numbers below are the authors’ benchmarks, on one GPU per model.

The NVIDIA post’s table covers four hybrids: Nemotron-3-Nano-4B (Mamba-2, BF16, one H100), Nemotron-3-Super-120B-A12B (Mamba-2, NVFP4, one B300), Qwen3.5-4B (Gated DeltaNet, BF16, one H100), and Qwen3.5-122B-A10B (Gated DeltaNet, NVFP4, one B300). It says that at batch size 256 the kernel runs 1.46x to 1.80x faster and the decode step runs 1.20x to 1.47x faster than a tuned vLLM baseline, and that batch size 1 shows no speedup in these tests. For speculative decoding, it says that at batch size 512 ReplaySSM delivers 1.85x and 2.10x the throughput of vLLM’s speculative decoding on the two 120B-class models, with matched acceptance lengths.

The abstract of the paper, for speculative decoding, says about 1.8x the throughput of vLLM’s speculative decoding on those two large models at batch size 256, and 1.20x to 1.47x on standard decoding at that batch across the four models. The Dao lab figure caption says up to 1.43x on standard decoding and 1.87x to 1.96x on speculative decoding, window 4, NVFP4 models on a B300. Same project, three summaries. Match the sentence to the batch size, and treat all three as the authors’ reports.

What NVIDIA says is already integrated

The NVIDIA post’s opening says ReplaySSM is integrated into TensorRT-LLM, vLLM, and FlashInfer, and that it “runs in production in GLM-5.3-Flash and Kimi-K3.” A later section is narrower. It says GLM-5.3-Flash serves with ReplaySSM, and that Kimi-K3 caches the SSM inputs of draft tokens for verification and cites ReplaySSM as a concurrent approach. That later section also names SGLang alongside TensorRT-LLM, vLLM, and FlashInfer. The Dao lab post and the Tri Dao post do not say ReplaySSM is serving GLM-5.3-Flash or Kimi-K3. They describe hybrid models already in production, including Nemotron 3, Qwen3.5, and Kimi Linear, as the reason the problem matters.

The paper’s code link is github.com/Johnny-Liou/ReplaySSM. GitHub’s pull-request API, checked 12 October 2026, listed the TensorRT-LLM, vLLM, and FlashInfer pull requests from the NVIDIA post as merged. The latest of those vLLM pulls merged on 30 September 2026. A merged pull request is a code landing. The production sentence remains NVIDIA’s, and the same post qualifies the Kimi half of it.