Run DeepSeek-V4.1-Flash offline inference on SM120 GPUs (RTX PRO 6000 Blackwell) with the same entry points as SM100. SM100 and Ascend paths are unchanged. - Build: SM120 is detected from the first visible GPU (scripts/nvidia_arch.sh). It builds DeepEP and DeepGEMM from its nv_dev branch (vendor/DeepGEMM-sm120), and no FlashMLA or DeepSelect; FlashInfer is installed from a pinned commit (requirements-nvidia-sm120.txt). - Attention: V4.1 sparse MLA runs on FlashInfer's SM120 sparse MLA kernels, reading the inline-scale KV records as stored; KV blocks are aligned to 512 bytes. - Indexer: FlashInfer's fused DSA indexer scores the index keys and selects their top-k in one call, for decode on every indexer layer (candidate source and consumer layers included) and for prefill outside the candidate layers. The remaining selections (prefill of the candidate layers, the sampler) use FlashInfer's top-k behind DeepSelect's interface (model_executor/layers/topk.py), in deterministic mode. - MoE EP: DeepEP over NCCL GIN by default, or NCCL collectives (DEEPINFER_SM120_EP_BACKEND=nccl); NCCL settings known to break the EP paths are rejected. EPLB, including decode-only draft-layer EPLB, and per-layer YoCo MoE dispatch. - Engram tables live in page-locked host shared memory; killed workers leave nothing in /dev/shm. - Features: attention DP + MoE EP, DSpark/nextn speculative decoding, CUDA graphs, async scheduling, decoder SWA bounded replay, structured outputs, image inputs, multi-node deployments. Features without an SM120 implementation (CP > 1, MTP varlen, batch-invariant mode, GPU-resident Engram tables) are refused at engine start. - Eval: chat GSM8K and CV-Bench; tests for the SM120 kernels and their DeepInfer call sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DeepInfer
DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.
Supported models:
- DeepSeek-V4.1-Flash
Supported features:
- Optimized kernels, model and sampler runtime
- Parallelism: Attention DP / balanced CP, MoE EP + EPLB
- DSpark and confidence-scheduled verification
- SWA bounded replay
- Asynchronous scheduling (zero-overhead batch scheduler)
- Batch-invariant inference
- Deterministic sampling
- Structured outputs
Feature details and configuration
- Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.
- DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set
DEEPINFER_ENABLE_MTP_VARLEN=1;DEEPINFER_USE_NEXTN_DECODEcontrols the maximum draft length (default: 5). - Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set
DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compactandDEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length. - Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default.
- Batch-invariant inference: keep model results identical across different batch compositions. Enable with
DEEPINFER_BATCH_INVARIANT=1(NVIDIA only). - Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set
DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0and pass a fixedseq_seedtoSamplingParams(NVIDIA only). - Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass
so_args={"schema_type": ..., "schema": ...}toSamplingParams; see examples.
Supported hardware:
- NVIDIA SM100 GPUs
- Huawei Ascend 950 NPUs
Required for a complete inference service (not included in this release):
- A KV cache system
- VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)
- Load balancing across data-parallel replicas for both prefill and decode
Getting started
Prerequisites
- Linux
- GCC >= 13.0
- Python 3.12
- CUDA 13 (NVIDIA only)
- CANN 9.2 (Ascend only)
Build from source
Before building DeepInfer, please create and activate an empty Python 3.12 environment.
# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
Offline inference
The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is
downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to
run from a local copy instead.
python examples/offline_inference.py
See examples for details.
Performance benchmarks
| Prefill | Decode |
|---|---|
![]() |
![]() |
Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.
Evaluation benchmarks
See eval/README.md.
License
DeepInfer is released under the Apache License 2.0. See LICENSE for details.
Acknowledgments
DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.
Citation
@misc{deepinfer2026,
title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},
author = {DeepSeek-AI},
year = {2026},
publisher = {GitHub},
howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}},
}

