The first Cake MoE layer found the EP group's largest prefill row count with
a gloo all-reduce and stored it as an ad-hoc attribute on the metadata. The
YoCo consumer view is a shallow copy (get_consumer_view), so it inherited the
full-token maximum: under compact truncation every consumer layer made as
many collective Cake calls as a full-token layer, the extra ones empty.
- DeepseekModel.forward reduces the full-token and, under compaction, the
consumer row counts in one CPU all-reduce before the layer loop (like the
per-layer YoCo dispatch bound) and sets each on its own metadata, so
consumer layers make calls for their own rows.
- InputMetadata declares cake_moe_max_tokens_per_rank. Layers reached with
metadata the forward did not prepare (e.g. draft layers) still compute it
once on the first layer.
Tests: tests/models/test_cake_moe_prefill_calls.py (CPU).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run DeepSeek-V4.1-Flash offline inference on SM120 GPUs (RTX PRO 6000
Blackwell) with the same entry points as SM100. SM100 and Ascend paths
are unchanged.
- Build: SM120 is detected from the first visible GPU
(scripts/nvidia_arch.sh). It builds DeepEP and DeepGEMM from its nv_dev
branch (vendor/DeepGEMM-sm120), and no FlashMLA or DeepSelect;
FlashInfer is installed from a pinned commit
(requirements-nvidia-sm120.txt).
- Attention: V4.1 sparse MLA runs on FlashInfer's SM120 sparse MLA
kernels, reading the inline-scale KV records as stored; KV blocks are
aligned to 512 bytes.
- Indexer: FlashInfer's fused DSA indexer scores the index keys and
selects their top-k in one call, for decode on every indexer layer
(candidate source and consumer layers included) and for prefill outside
the candidate layers. The remaining selections (prefill of the
candidate layers, the sampler) use FlashInfer's top-k behind
DeepSelect's interface (model_executor/layers/topk.py), in
deterministic mode.
- MoE EP: DeepEP over NCCL GIN by default, or NCCL collectives
(DEEPINFER_SM120_EP_BACKEND=nccl); NCCL settings known to break the EP
paths are rejected. EPLB, including decode-only draft-layer EPLB, and
per-layer YoCo MoE dispatch.
- Engram tables live in page-locked host shared memory; killed workers
leave nothing in /dev/shm.
- Features: attention DP + MoE EP, DSpark/nextn speculative decoding,
CUDA graphs, async scheduling, decoder SWA bounded replay, structured
outputs, image inputs, multi-node deployments. Features without an SM120
implementation (CP > 1, MTP varlen, batch-invariant mode, GPU-resident
Engram tables) are refused at engine start.
- Eval: chat GSM8K and CV-Bench; tests for the SM120 kernels and their
DeepInfer call sites.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>