DeepSeek open-sources DeepInfer, the serving engine behind V4.1-Flash, minus the cluster parts

DeepSeek has open-sourced DeepInfer, which it describes as the inference framework behind its large-scale DeepSeek-V4.1-Flash service. The code is under Apache-2.0. DeepSeek announced it on X on 20 October, saying it is sharing its production stack “to help the community lower compute costs, reduce energy use, and make AI more affordable.”

This is more than another kernel library. DeepSeek has published kernels before (FlashMLA, DeepGEMM, DeepEP). DeepInfer is the engine that sits on top of them.

What the release covers

The README is precise about the scope, and the scope is narrow:

• One model. DeepSeek-V4.1-Flash. The default checkpoint is DeepSeek-V4.1-Flash-1017, the open weights DeepSeek posted last week.
• Two hardware targets. NVIDIA SM100 GPUs (CUDA 13) and Huawei Ascend 950 NPUs (CANN 9.2), on Linux with Python 3.12.
• Offline inference only. You run it from a script, not as a finished service.

Inside that scope the feature list is long: attention data parallelism and balanced context parallelism, MoE expert parallelism with expert load balancing (EPLB), draft-token decoding DeepSeek calls DSpark with confidence-scheduled verification, a bounded replay for the sliding-window decoder, an asynchronous batch scheduler, and structured outputs (regex, JSON schema or EBNF). Batch-invariant inference and deterministic sampling are NVIDIA only.

What is missing

The README lists three things “required for a complete inference service” that are not in this release: a KV cache system, prefill/decode disaggregation (it says most of the engine side is already implemented), and load balancing across data-parallel replicas. The deployment Q&A says the KV cache and load-balancing components are “tightly coupled to our internal infrastructure,” so that code is not yet fully open-sourced. It describes their design instead, down to a recommended KV cache storage server.

The same Q&A puts the smallest efficient unit at 64 GPUs of prefill plus 64 of decode. 1P + 1D is deployable, but this is cluster software.

The numbers DeepSeek gives

The README’s performance section is two charts (prefill and decode, measured on 4 and 64 SM100 GPUs) with a sweep script to reproduce them. It gives no single headline figure, so we quote none. In the Q&A, DeepSeek says a single cluster serves 155 trillion tokens a day, with overall revenue of $38 per GPU-hour “at valley pricing.” Those are DeepSeek’s operating claims, not benchmarks anyone has checked.

The second repo

Alongside DeepInfer, DeepSeek pointed to deepseek-recipe, a Rust library (MIT) with Python bindings. It converts Messages, Chat Completions and Responses requests into DeepSeek V4/V4.1 prompts and token IDs and back. DeepSeek says its logic matches the official API, so self-hosted deployments behave like api.deepseek.com.

Why it matters

Most open serving stacks are built by people who do not run a frontier model at scale. This one comes from a lab that does, with an Ascend path alongside NVIDIA’s. For teams self-hosting V4.1-Flash, the Q&A alone is worth reading. For US federal civilian IT, note that the DHS order published this week also covers “third-party products and services developed using DeepSeek implementations.”