Glonce

DeepSeek open-sources DeepInfer inference framework for V4.1-Flash

infrastructureThe Glonce editorsHacker News🔥86Glonce rating86Impact78Novelty70Relevance100Surprise60Players80Credibility88
DeepSeek open-sources DeepInfer inference framework for V4.1-Flash

DeepSeek-AI has published DeepInfer, an inference framework, on GitHub under the Apache License 2.0. The README describes it as the high-performance inference framework that powers DeepSeek's large-scale production services. The open-source release is narrower than that sounds: it provides offline inference with the core features needed for serving at scale, and the only supported model is DeepSeek-V4.1-Flash. Supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

The feature list covers optimized kernels with a model and sampler runtime; parallelism through Attention DP / balanced CP and MoE EP with EPLB; DSpark and confidence-scheduled verification; SWA bounded replay; asynchronous scheduling; batch-invariant inference; deterministic sampling; and structured outputs. The kernels named are FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.

The README gives configuration details for several features. DSpark accelerates decoding with draft tokens and uses confidence estimates to schedule verification; it is switched on with DEEPINFER_ENABLE_MTP_VARLEN=1, and DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length, which defaults to 5. Decoder SWA bounded replay cuts decoder prefill work by replaying only a bounded suffix; the default V4.1 settings are DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140, and the retained suffix must cover the SWA window plus the maximum backtracking length. Asynchronous scheduling overlaps CPU batch scheduling with device execution and is on by default.

Batch-invariant inference keeps results identical across different batch compositions; it is enabled with DEEPINFER_BATCH_INVARIANT=1 and works on NVIDIA only. Deterministic sampling produces identical outputs for identical input tokens and sampling parameters. It needs DEEPINFER_BATCH_INVARIANT=1, DEEPINFER_ENABLE_MTP_VARLEN=0 and a fixed seq_seed passed to SamplingParams, and is also NVIDIA only. Structured outputs constrain generation to a regex, a JSON schema, or an EBNF grammar, passed to SamplingParams through so_args.

Several pieces are missing. The README lists what a complete inference service still needs and says is not included in this release: a KV cache system; VE/Prefill/Decode disaggregation (the README adds that most of the required engine functionality is already implemented); and load balancing across data-parallel replicas for both prefill and decode.

To run it, you need Linux, GCC 13.0 or newer, and Python 3.12, plus CUDA 13 on NVIDIA or CANN 9.2 on Ascend. You create an empty Python 3.12 environment, set DEEPINFER_ACCELERATOR (nvidia, or ascend for the Ascend platform) and run ./build.sh. The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, downloads into the Hugging Face cache on first run; DEEPINFER_HF_MODEL_DIR points to a local copy instead. The entry example is examples/offline_inference.py.

The README says the benchmarks were measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to run_sweep.sh for reproducing them. An evaluation guide sits in eval/README.md. The credits say DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM, and thank their developers. The citation entry lists the author as DeepSeek-AI and the year as 2026.

Key facts

  • DeepSeek-AI released DeepInfer on GitHub under Apache 2.0; the README calls it the framework that powers DeepSeek's large-scale production services.
  • This release is offline inference only and supports a single model, DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
  • Features include DSpark with confidence-scheduled verification (default maximum draft length 5), SWA bounded replay, asynchronous scheduling, batch-invariant inference, deterministic sampling and structured outputs (regex, JSON schema, EBNF).
  • A KV cache system, VE/Prefill/Decode disaggregation and data-parallel load balancing are not included, so it is not a complete serving stack.
  • Build requirements: Linux, GCC 13.0 or newer, Python 3.12, and CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). The code was originally built primarily on vLLM.

Why it matters

DeepSeek has opened the engine it says runs its own large-scale production services, including kernels such as FlashMLA, DeepGEMM and DeepEP and techniques like DSpark and SWA bounded replay. Support for Huawei Ascend 950 NPUs alongside NVIDIA SM100 GPUs puts both hardware families in one codebase. The code also shows which of vLLM's ideas a production team kept, since the README says DeepInfer was originally built primarily on vLLM.

Who it affects

Teams that run or study DeepSeek-V4.1-Flash on NVIDIA SM100 or Huawei Ascend 950 hardware, inference engineers who want to read how the kernels and scheduling features are put together, and anyone who needs deterministic or batch-invariant outputs or grammar-constrained generation. Operators on other models or other hardware are not covered by this release.

How to use it

Use Linux with GCC 13.0 or newer and Python 3.12, and install CUDA 13 for NVIDIA or CANN 9.2 for Ascend. Create an empty Python 3.12 environment, export DEEPINFER_ACCELERATOR=nvidia (use ascend for Ascend), run ./build.sh, then run python examples/offline_inference.py. The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, downloads into the Hugging Face cache on first run, or set DEEPINFER_HF_MODEL_DIR to a local copy. The licence is Apache 2.0. To reproduce the benchmarks, the README points to run_sweep.sh.

How solid is it

This is the official deepseek-ai repository, and the README is the only material described here. The production claim is DeepSeek's own statement. The text does not say that this open-source release is identical to the internal production code. No benchmark results (throughput, latency, cost) are given in the text; only the measurement setup of 4 SM100 GPUs for prefill and 64 for decode, plus a pointer to run_sweep.sh. No comparison with vLLM, SGLang or TensorRT-LLM performance is given.

Risks and caveats

The release is offline inference only and supports one model. By the README's own list, a complete inference service still needs a KV cache system, VE/Prefill/Decode disaggregation and load balancing across data-parallel replicas, none of which is included. Batch-invariant inference and deterministic sampling work on NVIDIA only. Details of DSpark are limited to a one-line description. The toolchain is demanding: CUDA 13 or CANN 9.2, GCC 13.0 or newer, and Python 3.12.

“DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services.”

— DeepInfer README, deepseek-ai on GitHub