SM120 has no DeepEP feature left (no deepep backend, Engram in host memory,
no context parallelism, no NVLink domain), yet five modules imported deep_ep
at the top, so every SM120 process needed a DeepEP build, one compiled
against an NCCL newer than the one the process loads.
- mlp, engram, cp, userbuffer: deep_ep names used only in annotations move
under TYPE_CHECKING; EPBuffer, EngramBuffer, BucketBuffer and the CP
alignment helpers are imported in the functions that build them.
- parallel_state: is_within_nvl_domain imports deep_ep after the SM120 and
single-rank returns.
No behavior change where DeepEP is installed: the same code imports it,
later. tests/test_sm120_without_deep_ep.py imports each module in a fresh
interpreter that cannot import deep_ep, and runs the SM120 paths (nccl EP
buffer, no CP buffer, no NVLink domain) with deep_ep blocked; the nccl
backend allocation test blocks deep_ep too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One persistent kernel per MoE call dispatches the tokens, runs the routed
MXFP4 experts and the FP8 shared expert, and combines, moving data over
PCIe peer load/store between reachable ranks and NCCL GIN elsewhere. The
DeepEP path keeps all peer traffic on the NICs (NCCL_LSA_TEAM_SIZE=1).
- cake_moe.py: the routed and shared expert weights convert to Cake's
operands (bit-exact against DeepInfer's dequantized layouts) and its
expert-major kernel layout, owned by DeepInfer so EPLB moves them in
place. Every call is bucketed with one row bound on all EP ranks:
decode one call bounded by the decode capacity (graph-capturable;
the bound does not change the cost), prefill chunks of
DEEPINFER_SM120_CAKE_CHUNK_TOKENS rows (default 2048, which bounds the
workspace) covering the EP group's largest batch, found once per step
by a CPU all-reduce. Warmup at userbuffer allocation creates the pooled
workspace and checks the kernel's protocol flag.
- EPLB: Cake does not report what each rank received, so its layers count
the routes they send and the load balancer all-reduces them once per
step into the received counts (count_on_sender).
- Runtime: the cake backend leaves the LSA team to NCCL and warns without
NCCL_TOPO_FILE; is_within_nvl_domain is False on SM120 (no NVLink).
- build.sh: SM120 installs NCCL 2.30.7, which Cake requires exactly.
Tests: tests/kernels/test_sm120_cake_moe.py (operand conversion, EPLB
locality of the kernel layout) and tests/distributed/test_sm120_cake_moe.py
(8 GPUs: against a reference with Cake's quantization steps, chunked
calls bitwise equal to one call, uneven and empty ranks, CUDA graph
replay). examples/offline_inference.py runs end to end on 8 GPUs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run DeepSeek-V4.1-Flash offline inference on SM120 GPUs (RTX PRO 6000
Blackwell) with the same entry points as SM100. SM100 and Ascend paths
are unchanged.
- Build: SM120 is detected from the first visible GPU
(scripts/nvidia_arch.sh). It builds DeepEP and DeepGEMM from its nv_dev
branch (vendor/DeepGEMM-sm120), and no FlashMLA or DeepSelect;
FlashInfer is installed from a pinned commit
(requirements-nvidia-sm120.txt).
- Attention: V4.1 sparse MLA runs on FlashInfer's SM120 sparse MLA
kernels, reading the inline-scale KV records as stored; KV blocks are
aligned to 512 bytes.
- Indexer: FlashInfer's fused DSA indexer scores the index keys and
selects their top-k in one call, for decode on every indexer layer
(candidate source and consumer layers included) and for prefill outside
the candidate layers. The remaining selections (prefill of the
candidate layers, the sampler) use FlashInfer's top-k behind
DeepSelect's interface (model_executor/layers/topk.py), in
deterministic mode.
- MoE EP: DeepEP over NCCL GIN by default, or NCCL collectives
(DEEPINFER_SM120_EP_BACKEND=nccl); NCCL settings known to break the EP
paths are rejected. EPLB, including decode-only draft-layer EPLB, and
per-layer YoCo MoE dispatch.
- Engram tables live in page-locked host shared memory; killed workers
leave nothing in /dev/shm.
- Features: attention DP + MoE EP, DSpark/nextn speculative decoding,
CUDA graphs, async scheduling, decoder SWA bounded replay, structured
outputs, image inputs, multi-node deployments. Features without an SM120
implementation (CP > 1, MTP varlen, batch-invariant mode, GPU-resident
Engram tables) are refused at engine start.
- Eval: chat GSM8K and CV-Bench; tests for the SM120 kernels and their
DeepInfer call sites.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>