SM120 has no DeepEP feature left (no deepep backend, Engram in host memory,
no context parallelism, no NVLink domain), yet five modules imported deep_ep
at the top, so every SM120 process needed a DeepEP build, one compiled
against an NCCL newer than the one the process loads.
- mlp, engram, cp, userbuffer: deep_ep names used only in annotations move
under TYPE_CHECKING; EPBuffer, EngramBuffer, BucketBuffer and the CP
alignment helpers are imported in the functions that build them.
- parallel_state: is_within_nvl_domain imports deep_ep after the SM120 and
single-rank returns.
No behavior change where DeepEP is installed: the same code imports it,
later. tests/test_sm120_without_deep_ep.py imports each module in a fresh
interpreter that cannot import deep_ep, and runs the SM120 paths (nccl EP
buffer, no CP buffer, no NVLink domain) with deep_ep blocked; the nccl
backend allocation test blocks deep_ep too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DeepEP needs NCCL 2.31 or newer and Cake MegaMoE exactly 2.30.7, and a
process loads one NCCL: since the SM120 build installs 2.30.7, the deepep
backend could only fail at startup. Remove it so SM120 can stop building
DeepEP:
- DEEPINFER_SM120_EP_BACKEND accepts auto, cake and nccl; deepep is refused
by the choice check.
- userbuffer drops the SM120 EPBuffer path (_create_sm120_ep_buffer and its
notify-thread and LSA-team checks): on SM120 only the nccl backend
allocates an EP buffer, and allocate_ep asserts that.
- sm120_support drops the DeepEP NCCL check; runtime_config keeps the nccl
backend's NCCL settings (single-rank LSA team, no graph registration).
- Tests: the deepep cases go; the environment tests use nccl or cake, and
the NCCL fixture defaults to 2.30.7.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build.sh builds DeepEP against NCCL 2.32.3 and then replaces NCCL with 2.30.7
for Cake. deepinfer imports deep_ep on every platform, so if that build no
longer loaded with 2.30.7 every SM120 run would fail at import, with nothing
pointing at the NCCL swap; and if torch loaded another libnccl.so.2, auto
would quietly pick the slower nccl backend.
- scripts/check_sm120_nccl.py runs after the pin: it fails the build when
deep_ep has an undefined symbol under the pinned NCCL, and warns when it
cannot import deep_ep for another reason (e.g. no GPU on the build host)
or when torch loads an NCCL other than the pin.
- The pin is CAKE_NCCL_VERSION in build.sh; tests/test_build_nccl_pin.py
keeps it equal to sm120_support.CAKE_NCCL_VERSION and covers the check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cake's requirements on the model and settings (an integer SwiGLU clamp, at
most one shared expert) were asserts, which `python -O` strips: the layer
would then build with a truncated clamp or a dropped shared expert.
check_cake_config raises ValueError naming the nccl fallback, and also
refuses a non-positive DEEPINFER_SM120_CAKE_CHUNK_TOKENS and experts that do
not split evenly over the EP group. DeepseekMoE checks before converting the
weights; CakeMoE checks again for direct use.
expert_major_views notes why slicing L of L + 1 scale segments is right
without a shared expert too: FlashInfer fills the shared segment with
neutral scales then.
Tests: tests/models/test_cake_moe_config.py (CPU).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first Cake MoE layer found the EP group's largest prefill row count with
a gloo all-reduce and stored it as an ad-hoc attribute on the metadata. The
YoCo consumer view is a shallow copy (get_consumer_view), so it inherited the
full-token maximum: under compact truncation every consumer layer made as
many collective Cake calls as a full-token layer, the extra ones empty.
- DeepseekModel.forward reduces the full-token and, under compaction, the
consumer row counts in one CPU all-reduce before the layer loop (like the
per-layer YoCo dispatch bound) and sets each on its own metadata, so
consumer layers make calls for their own rows.
- InputMetadata declares cake_moe_max_tokens_per_rank. Layers reached with
metadata the forward did not prepare (e.g. draft layers) still compute it
once on the first layer.
Tests: tests/models/test_cake_moe_prefill_calls.py (CPU).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cake MegaMoE runs only with NCCL 2.30.7, but only deepep had a startup NCCL
check: with any other NCCL the default backend failed inside FlashInfer.
- DEEPINFER_SM120_EP_BACKEND gains auto, the new default: cake when the
loaded NCCL is 2.30.7 (what build.sh installs), otherwise nccl with a
warning. sm120_support.ep_backend() resolves it once per process and
writes the result back to the environment variable, so the worker
settings, the MoE layers and the userbuffer manager (and child processes)
all see the same backend.
- sm120_support.check_ep_backend_nccl() refuses an explicit cake with
another NCCL, and moves the deepep >= 2.31 check out of runtime_config;
nccl_runtime_version() moves to sm120_support with it.
- MXFP4 x FP8 stays required for cake, as an error rather than a fallback:
the worker's NCCL settings (LSA team, graph registration) are made for
the backend before the model's expert format is known. The message now
says the backend may have come from auto.
Tests: auto resolution (once, written back), explicit backends untouched,
cake refused with other NCCL, auto falling back to nccl in the worker
environment (tests/worker/test_nccl_ep.py).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Cake kernel reports an invalid call (e.g. its lossless return codec ran
out of overflow blocks) in a fail-closed flag, but DeepInfer read it only
once at warmup, so such a call silently returned wrong MoE outputs. The flag
is per call (the next launch clears it) and lives in the workspace all
layers share, so it has to be captured right after each call:
- CakeMoE ORs the flag into its own protocol_errors tensor after every
call, on the stream (no host sync, also inside CUDA graphs), and names
itself after its MoE layer.
- userbuffer.check_cake_moe_errors(), called once per step from
DeepseekForCausalLM.sample, copies the layers' flags to pinned memory
behind the step and checks the previous step's copy, so an error stops
the run one step later without a host synchronization.
- check_protocol reads the same flag tensor.
Tests: tests/worker/test_cake_moe_errors.py (CPU).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PR #291 files failed `ruff check` (ANN001 on cake_moe.expert_major_views,
userbuffer.register_cake_moe and the test helpers; E741 on the tests' `I`)
and `ruff format --check`. Annotate the arguments (CakeMoE under
TYPE_CHECKING in userbuffer), rename the intermediate size to INTER in the
tests, and apply ruff format. No behavior change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SM120 prefill gathered the paged compressed (CA) KV into BF16 and then
re-quantized the whole gathered buffer into temporary row caches, for every
layer and every prefill chunk: two extra full passes over total_k_compressed
tokens plus a temporary cache, although decode already reads the same paged
records in place.
Prefill now reads the compressed cache in place: the compressed indices
count positions in the gathered order of the prefill load plan
(load_segments_ca), so load_plan_cache_rows maps those positions to cache
rows and gathered_to_cache_rows applies the map to the indices. The values
read are the stored records the re-quantization reproduced, so results do
not change. V4Attention skips the BF16 gather of the compressed part (and the
share-KV gather buffer) where prefill_reads_compressed_cache_in_place().
The SWA part is still gathered and quantized: the current chunk's stores
overwrite ring slots that earlier queries of the chunk still attend to.
prefill_attn_sparse_swa_ca now takes the gathered SWA KV, the compressed
cache and its load plan instead of one gathered buffer. The SM120 kernel
tests follow the new API, compare the compressed part against the stored
records, and check that the mapped rows address those records; CPU tests
cover the row mapping, zero-filled positions and out-of-range entries past
a length.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Call sites tested `not PLATFORM.is_sm120` to mean "has the SM100 fused
kernels", hard-coded SM120 constants next to their use, and treated any GPU
that is not 12.x as SM100. A new family (or 12.x variant) would need every
site revisited, and an unsupported GPU silently took SM100 paths.
- deepinfer.platform.NVIDIA_ARCHS maps compute capability major to an
NvidiaArch entry: sm100_fused_kernels, grouped_gemm_zero_padding,
kv_block_alignment and default_comm_sms, with the same values SM100 and
SM120 used before.
- Platform exposes them as capabilities (has_sm100_fused_kernels, ...), with
the previous values on Ascend; the mega MoE / gate / mHC, fused
norm-rope-attn-rope-cast, FP8 einsum output, grouped GEMM zero padding, KV
block alignment and default EP comm SM sites ask for the capability.
- PlatformGPU.nvidia_arch selects the entry from the first visible GPU. No
visible GPU still means SM100; a GPU of an unsupported family now raises
instead of running SM100 kernels. is_sm120 stays for the SM120 backends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nccl_ep.py and host_engram.py each defined a private _align identical to the
existing tilelang_ops.common.align (an import-free constants module, so the
communication modules stay lightweight).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_topk_decode returns through _sm120_topk_decode on SM120 before it reaches
_compute_decode_p_sum, so the SM120-only `weights.float()` and
_dense_logits_kwargs() there never applied. Restore the call to its SM100 /
Ascend form so readers do not assume SM120 takes this path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every call shape not memoized yet started its scratch search from empty
(0, 0) views, so it always failed FlashInfer's capacity check and retried
(up to three wrapper.run calls). Prefill shapes change almost every step, so
the first SM120 attention call of each prefill step paid one or two thrown
ValueErrors plus FlashInfer's validate/prepare for each failed attempt, and
those attempts filled the per-wrapper plan cache with dead entries.
A new shape now first tries views of the whole current arenas, which fit
most shapes once a few calls have grown them; only a larger need pays a
failed attempt before the arena grows. Memoized shapes keep their views, so
CUDA graph captures are unaffected.
Adds a CPU test of the sizing with a stand-in wrapper (skipped without
flashinfer).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Draft-layer schedules went through the main-model validation, so a bad
DEEPINFER_DRAFT_EXPERT_COUNT_WINDOW_SIZE was reported as
"DEEPINFER_EXPERT_COUNT_WINDOW_SIZE=... must be between 1 and
DEEPINFER_EXPERT_REARRANGEMENT_STEP_INTERVAL=...", naming variables the user
never set. Draft settings could also be dropped without a word.
- ExpertLoadBalancer takes setting_names for its error messages; the draft
schedule passes the draft variables (or says the interval is inherited, or
the window is the default).
- Warn when DEEPINFER_DRAFT_EXPERT_COUNT_WINDOW_SIZE is set but no draft
interval resolves, and when draft layers share an EPLB layout group with
main-model layers and so cannot follow the draft schedule.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
summarize_extra averaged ADE20K/COCO/Omni3D with .get(source, 0.0), so a run
that scored no rows of a source (e.g. --limit 200, which keeps only ADE20K
rows because records are taken in file order) reported an overall CV-Bench
score of at most a quarter of the real accuracy.
The overall/2D/3D scores are now reported only when every source they
average has scored rows; otherwise summary.json lists
cvbench_missing_sources and the scorer warns. Per-source and per-task
accuracies are reported either way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build.sh tested `[ "$(nvidia_arch.sh)" = sm120 ]`, which discards the
script's exit status, so a bad DEEPINFER_NVIDIA_ARCH only surfaced in
build_vendor.sh after the long pip install. nvidia_arch.sh also mapped every
GPU that is not 12.x (and hosts without nvidia-smi) to sm100 silently, so an
SM120 build container without nvidia-smi installed the SM100 dependency set.
- build.sh runs detection as its own assignment (set -e stops before pip) and
exports the result so build_vendor.sh builds for the same family.
- DEEPINFER_NVIDIA_ARCH is case-insensitive.
- A detected GPU of another family (e.g. compute capability 9.0) is an error
that points at DEEPINFER_NVIDIA_ARCH; no visible GPU still builds for sm100
(GPU-less SM100 build hosts keep working) but warns on stderr.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_eval.py forced DEEPINFER_ENABLE_MTP_VARLEN=1, which the engine refuses on
SM120, so `run_eval.py --bench aime|hle|gpqa` (and their driver scripts)
failed at LLMEngine init unless the user remembered --env ...=0. meta.json
also assumed mtp_varlen=True whenever --env did not mention it.
- One _default_engine_env() supplies the defaults for both the rank processes
and meta.json: MTP varlen on, except on SM120.
- meta.json's mtp / mtp_varlen and bench run_metadata now see the same
defaults-then---env environment the ranks run with.
- README.sm120.md no longer asks for --env DEEPINFER_ENABLE_MTP_VARLEN=0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host-resident Engram tables are one /dev/shm mapping shared through
/proc/<pid>/fd, so every Engram group must sit on one host. The default
engram_parallel_size (0 = every rank) spans all nodes of a multi-node
deployment, which only failed on a bare assert in HostEngramBuffer after the
weights were loaded (and under python -O failed later with a misleading
"cannot open /proc/..." error).
- sm120_support.check_rank_placement (was check_distinct_devices) now also
checks, from the hostnames it already gathers, that each Engram group
(contiguous rank blocks) stays on one host, and names the offending ranks.
- HostEngramBuffer raises ValueError instead of asserting.
- README.sm120.md says multi-node deployments must set engram_parallel_size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compute_capability() only understood digit indices and GPU- UUIDs, so the first
CUDA_VISIBLE_DEVICES entry being a MIG instance made is_sm_major_10/12 return
False: SM120 then imported flash_mla/deep_select (not built there) and SM100
lost its GPU-type-dependent paths. Unresolvable entries also silently read as
"not this architecture".
- Resolve "MIG-<uuid>" (via NVML's MIG lookups) and the legacy
"MIG-GPU-<uuid>/<gi>/<ci>" form to the capability of the physical GPU.
- Raise LookupError when NVML is present but the entry names no GPU, instead
of returning None.
- Digit entries are NVML (PCI bus) indices, but CUDA orders devices fastest
first unless CUDA_DEVICE_ORDER=PCI_BUS_ID. On hosts whose GPUs differ in
capability that is ambiguous, so ask for CUDA_DEVICE_ORDER=PCI_BUS_ID.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cake needs NCCL 2.30.7 in the process, and the pinned DeepEP needs NCCL 2.31+
(ginMinStride, ginSupport, ncclDevCommRequirements.ginType, ...), so the two
cannot share a process. On SM120:
- DEEPINFER_SM120_EP_BACKEND defaults to cake; nccl stays the fallback for
expert formats other than MXFP4 x FP8, named in the cake error.
- build.sh builds the vendored packages against NCCL 2.32.3 (DeepEP no longer
compiles against 2.30.7) and installs 2.30.7 afterwards; deep_ep is still
imported but its backend is unavailable.
- deepep is rejected at worker setup when the loaded NCCL is older than 2.31,
read with ncclGetVersion (torch.cuda.nccl.version() reports torch's build).
- README.sm120.md documents the backends, the NCCL pin, NCCL_TOPO_FILE, the
chunk size, and the 8-GPU results.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cake's layers own collective NCCL state (symmetric windows, the GIN device
communicator) and must be destroyed on every EP rank, in the same order,
before the process group. Register them with the userbuffer manager, which
destroys them first in close(); left to garbage collection at exit, a rank
could fail after the run had finished.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One persistent kernel per MoE call dispatches the tokens, runs the routed
MXFP4 experts and the FP8 shared expert, and combines, moving data over
PCIe peer load/store between reachable ranks and NCCL GIN elsewhere. The
DeepEP path keeps all peer traffic on the NICs (NCCL_LSA_TEAM_SIZE=1).
- cake_moe.py: the routed and shared expert weights convert to Cake's
operands (bit-exact against DeepInfer's dequantized layouts) and its
expert-major kernel layout, owned by DeepInfer so EPLB moves them in
place. Every call is bucketed with one row bound on all EP ranks:
decode one call bounded by the decode capacity (graph-capturable;
the bound does not change the cost), prefill chunks of
DEEPINFER_SM120_CAKE_CHUNK_TOKENS rows (default 2048, which bounds the
workspace) covering the EP group's largest batch, found once per step
by a CPU all-reduce. Warmup at userbuffer allocation creates the pooled
workspace and checks the kernel's protocol flag.
- EPLB: Cake does not report what each rank received, so its layers count
the routes they send and the load balancer all-reduces them once per
step into the received counts (count_on_sender).
- Runtime: the cake backend leaves the LSA team to NCCL and warns without
NCCL_TOPO_FILE; is_within_nvl_domain is False on SM120 (no NVLink).
- build.sh: SM120 installs NCCL 2.30.7, which Cake requires exactly.
Tests: tests/kernels/test_sm120_cake_moe.py (operand conversion, EPLB
locality of the kernel layout) and tests/distributed/test_sm120_cake_moe.py
(8 GPUs: against a reference with Cake's quantization steps, chunked
calls bitwise equal to one call, uneven and empty ranks, CUDA graph
replay). examples/offline_inference.py runs end to end on 8 GPUs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run DeepSeek-V4.1-Flash offline inference on SM120 GPUs (RTX PRO 6000
Blackwell) with the same entry points as SM100. SM100 and Ascend paths
are unchanged.
- Build: SM120 is detected from the first visible GPU
(scripts/nvidia_arch.sh). It builds DeepEP and DeepGEMM from its nv_dev
branch (vendor/DeepGEMM-sm120), and no FlashMLA or DeepSelect;
FlashInfer is installed from a pinned commit
(requirements-nvidia-sm120.txt).
- Attention: V4.1 sparse MLA runs on FlashInfer's SM120 sparse MLA
kernels, reading the inline-scale KV records as stored; KV blocks are
aligned to 512 bytes.
- Indexer: FlashInfer's fused DSA indexer scores the index keys and
selects their top-k in one call, for decode on every indexer layer
(candidate source and consumer layers included) and for prefill outside
the candidate layers. The remaining selections (prefill of the
candidate layers, the sampler) use FlashInfer's top-k behind
DeepSelect's interface (model_executor/layers/topk.py), in
deterministic mode.
- MoE EP: DeepEP over NCCL GIN by default, or NCCL collectives
(DEEPINFER_SM120_EP_BACKEND=nccl); NCCL settings known to break the EP
paths are rejected. EPLB, including decode-only draft-layer EPLB, and
per-layer YoCo MoE dispatch.
- Engram tables live in page-locked host shared memory; killed workers
leave nothing in /dev/shm.
- Features: attention DP + MoE EP, DSpark/nextn speculative decoding,
CUDA graphs, async scheduling, decoder SWA bounded replay, structured
outputs, image inputs, multi-node deployments. Features without an SM120
implementation (CP > 1, MTP varlen, batch-invariant mode, GPU-resident
Engram tables) are refused at engine start.
- Eval: chat GSM8K and CV-Bench; tests for the SM120 kernels and their
DeepInfer call sites.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DeepEP now records an HTTPS DeepJIT URL, so build_vendor no longer needs the git@github.com rewrite.
Co-authored-by: Shaoyuan Chen <chensy@deepseek.com>