Commit Graph
12 Commits
Author SHA1 Message Date
shunyadandClaude Opus 5.5 f009131253 refactor: import deep_ep where it is used
SM120 has no DeepEP feature left (no deepep backend, Engram in host memory,
no context parallelism, no NVLink domain), yet five modules imported deep_ep
at the top, so every SM120 process needed a DeepEP build, one compiled
against an NCCL newer than the one the process loads.

- mlp, engram, cp, userbuffer: deep_ep names used only in annotations move
  under TYPE_CHECKING; EPBuffer, EngramBuffer, BucketBuffer and the CP
  alignment helpers are imported in the functions that build them.
- parallel_state: is_within_nvl_domain imports deep_ep after the SM120 and
  single-rank returns.

No behavior change where DeepEP is installed: the same code imports it,
later. tests/test_sm120_without_deep_ep.py imports each module in a fresh
interpreter that cannot import deep_ep, and runs the SM120 paths (nccl EP
buffer, no CP buffer, no NVLink domain) with deep_ep blocked; the nccl
backend allocation test blocks deep_ep too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:43:19 -08:00
shunyadandClaude Opus 5.5 a2d45e4f22 refactor(sm120): drop the deepep MoE backend on SM120
DeepEP needs NCCL 2.31 or newer and Cake MegaMoE exactly 2.30.7, and a
process loads one NCCL: since the SM120 build installs 2.30.7, the deepep
backend could only fail at startup. Remove it so SM120 can stop building
DeepEP:

- DEEPINFER_SM120_EP_BACKEND accepts auto, cake and nccl; deepep is refused
  by the choice check.
- userbuffer drops the SM120 EPBuffer path (_create_sm120_ep_buffer and its
  notify-thread and LSA-team checks): on SM120 only the nccl backend
  allocates an EP buffer, and allocate_ep asserts that.
- sm120_support drops the DeepEP NCCL check; runtime_config keeps the nccl
  backend's NCCL settings (single-rank LSA team, no graph registration).
- Tests: the deepep cases go; the environment tests use nccl or cake, and
  the NCCL fixture defaults to 2.30.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:34:48 -08:00
shunyadandClaude Opus 5.5 2f18193bb5 fix(sm120): check NCCL for the cake backend; default to auto
Cake MegaMoE runs only with NCCL 2.30.7, but only deepep had a startup NCCL
check: with any other NCCL the default backend failed inside FlashInfer.

- DEEPINFER_SM120_EP_BACKEND gains auto, the new default: cake when the
  loaded NCCL is 2.30.7 (what build.sh installs), otherwise nccl with a
  warning. sm120_support.ep_backend() resolves it once per process and
  writes the result back to the environment variable, so the worker
  settings, the MoE layers and the userbuffer manager (and child processes)
  all see the same backend.
- sm120_support.check_ep_backend_nccl() refuses an explicit cake with
  another NCCL, and moves the deepep >= 2.31 check out of runtime_config;
  nccl_runtime_version() moves to sm120_support with it.
- MXFP4 x FP8 stays required for cake, as an error rather than a fallback:
  the worker's NCCL settings (LSA team, graph registration) are made for
  the backend before the model's expert format is known. The message now
  says the backend may have come from auto.

Tests: auto resolution (once, written back), explicit backends untouched,
cake refused with other NCCL, auto falling back to nccl in the worker
environment (tests/worker/test_nccl_ep.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
shunyadandClaude Opus 5.5 d2755d60a1 fix(sm120): check Cake MegaMoE protocol errors at runtime
The Cake kernel reports an invalid call (e.g. its lossless return codec ran
out of overflow blocks) in a fail-closed flag, but DeepInfer read it only
once at warmup, so such a call silently returned wrong MoE outputs. The flag
is per call (the next launch clears it) and lives in the workspace all
layers share, so it has to be captured right after each call:

- CakeMoE ORs the flag into its own protocol_errors tensor after every
  call, on the stream (no host sync, also inside CUDA graphs), and names
  itself after its MoE layer.
- userbuffer.check_cake_moe_errors(), called once per step from
  DeepseekForCausalLM.sample, copies the layers' flags to pinned memory
  behind the step and checks the previous step's copy, so an error stops
  the run one step later without a host synchronization.
- check_protocol reads the same flag tensor.

Tests: tests/worker/test_cake_moe_errors.py (CPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
shunyadandClaude Opus 5.5 f7da10314b style(sm120): fix ruff errors in the Cake MegaMoE change
The PR #291 files failed `ruff check` (ANN001 on cake_moe.expert_major_views,
userbuffer.register_cake_moe and the test helpers; E741 on the tests' `I`)
and `ruff format --check`. Annotate the arguments (CakeMoE under
TYPE_CHECKING in userbuffer), rename the intermediate size to INTER in the
tests, and apply ruff format. No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
shunyadandClaude Opus 5.5 9da41ad18e refactor(platform): describe NVIDIA families with a capability table
Call sites tested `not PLATFORM.is_sm120` to mean "has the SM100 fused
kernels", hard-coded SM120 constants next to their use, and treated any GPU
that is not 12.x as SM100. A new family (or 12.x variant) would need every
site revisited, and an unsupported GPU silently took SM100 paths.

- deepinfer.platform.NVIDIA_ARCHS maps compute capability major to an
  NvidiaArch entry: sm100_fused_kernels, grouped_gemm_zero_padding,
  kv_block_alignment and default_comm_sms, with the same values SM100 and
  SM120 used before.
- Platform exposes them as capabilities (has_sm100_fused_kernels, ...), with
  the previous values on Ascend; the mega MoE / gate / mHC, fused
  norm-rope-attn-rope-cast, FP8 einsum output, grouped GEMM zero padding, KV
  block alignment and default EP comm SM sites ask for the capability.
- PlatformGPU.nvidia_arch selects the entry from the first visible GPU. No
  visible GPU still means SM100; a GPU of an unsupported family now raises
  instead of running SM100 kernels. is_sm120 stays for the SM120 backends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:28:43 -08:00
shunyadandClaude Opus 5.5 c23e8c3a97 refactor(parallel_utils): reuse tilelang_ops.common.align instead of local _align copies
nccl_ep.py and host_engram.py each defined a private _align identical to the
existing tilelang_ops.common.align (an import-free constants module, so the
communication modules stay lightweight).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:28:43 -08:00
shunyadandClaude Opus 5.5 8898e97395 fix(sm120): refuse multi-host Engram groups right after rendezvous
Host-resident Engram tables are one /dev/shm mapping shared through
/proc/<pid>/fd, so every Engram group must sit on one host. The default
engram_parallel_size (0 = every rank) spans all nodes of a multi-node
deployment, which only failed on a bare assert in HostEngramBuffer after the
weights were loaded (and under python -O failed later with a misleading
"cannot open /proc/..." error).

- sm120_support.check_rank_placement (was check_distinct_devices) now also
  checks, from the hostnames it already gathers, that each Engram group
  (contiguous rank blocks) stays on one host, and names the offending ranks.
- HostEngramBuffer raises ValueError instead of asserting.
- README.sm120.md says multi-node deployments must set engram_parallel_size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:22:14 -08:00
Zihua WuandClaude Opus 5.5 eb4bb125f2 fix(sm120): destroy Cake MegaMoE layers before the process group
Cake's layers own collective NCCL state (symmetric windows, the GIN device
communicator) and must be destroyed on every EP rank, in the same order,
before the process group. Register them with the userbuffer manager, which
destroys them first in close(); left to garbage collection at exit, a rank
could fail after the run had finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 09:39:24 -08:00
Zihua WuandClaude Opus 5.5 2155f3eb46 feat(sm120): MoE on FlashInfer's Cake MegaMoE (DEEPINFER_SM120_EP_BACKEND=cake)
One persistent kernel per MoE call dispatches the tokens, runs the routed
MXFP4 experts and the FP8 shared expert, and combines, moving data over
PCIe peer load/store between reachable ranks and NCCL GIN elsewhere. The
DeepEP path keeps all peer traffic on the NICs (NCCL_LSA_TEAM_SIZE=1).

- cake_moe.py: the routed and shared expert weights convert to Cake's
  operands (bit-exact against DeepInfer's dequantized layouts) and its
  expert-major kernel layout, owned by DeepInfer so EPLB moves them in
  place. Every call is bucketed with one row bound on all EP ranks:
  decode one call bounded by the decode capacity (graph-capturable;
  the bound does not change the cost), prefill chunks of
  DEEPINFER_SM120_CAKE_CHUNK_TOKENS rows (default 2048, which bounds the
  workspace) covering the EP group's largest batch, found once per step
  by a CPU all-reduce. Warmup at userbuffer allocation creates the pooled
  workspace and checks the kernel's protocol flag.
- EPLB: Cake does not report what each rank received, so its layers count
  the routes they send and the load balancer all-reduces them once per
  step into the received counts (count_on_sender).
- Runtime: the cake backend leaves the LSA team to NCCL and warns without
  NCCL_TOPO_FILE; is_within_nvl_domain is False on SM120 (no NVLink).
- build.sh: SM120 installs NCCL 2.30.7, which Cake requires exactly.

Tests: tests/kernels/test_sm120_cake_moe.py (operand conversion, EPLB
locality of the kernel layout) and tests/distributed/test_sm120_cake_moe.py
(8 GPUs: against a reference with Cake's quantization steps, chunked
calls bitwise equal to one call, uneven and empty ranks, CUDA graph
replay). examples/offline_inference.py runs end to end on 8 GPUs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 08:55:46 -08:00
Zihua WuandClaude Opus 5.5 efecbb5ba6 feat(sm120): offline inference on NVIDIA Blackwell SM120
Run DeepSeek-V4.1-Flash offline inference on SM120 GPUs (RTX PRO 6000
Blackwell) with the same entry points as SM100. SM100 and Ascend paths
are unchanged.

- Build: SM120 is detected from the first visible GPU
  (scripts/nvidia_arch.sh). It builds DeepEP and DeepGEMM from its nv_dev
  branch (vendor/DeepGEMM-sm120), and no FlashMLA or DeepSelect;
  FlashInfer is installed from a pinned commit
  (requirements-nvidia-sm120.txt).
- Attention: V4.1 sparse MLA runs on FlashInfer's SM120 sparse MLA
  kernels, reading the inline-scale KV records as stored; KV blocks are
  aligned to 512 bytes.
- Indexer: FlashInfer's fused DSA indexer scores the index keys and
  selects their top-k in one call, for decode on every indexer layer
  (candidate source and consumer layers included) and for prefill outside
  the candidate layers. The remaining selections (prefill of the
  candidate layers, the sampler) use FlashInfer's top-k behind
  DeepSelect's interface (model_executor/layers/topk.py), in
  deterministic mode.
- MoE EP: DeepEP over NCCL GIN by default, or NCCL collectives
  (DEEPINFER_SM120_EP_BACKEND=nccl); NCCL settings known to break the EP
  paths are rejected. EPLB, including decode-only draft-layer EPLB, and
  per-layer YoCo MoE dispatch.
- Engram tables live in page-locked host shared memory; killed workers
  leave nothing in /dev/shm.
- Features: attention DP + MoE EP, DSpark/nextn speculative decoding,
  CUDA graphs, async scheduling, decoder SWA bounded replay, structured
  outputs, image inputs, multi-node deployments. Features without an SM120
  implementation (CP > 1, MTP varlen, batch-invariant mode, GPU-resident
  Engram tables) are refused at engine start.
- Eval: chat GSM8K and CV-Bench; tests for the SM120 kernels and their
  DeepInfer call sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:32:01 -08:00
Jiashi Li d585e6542c initial commit 2026-10-20 12:09:42 +08:00