Files
DeepInfer/deepinfer/model_executor/parallel_utils
Zihua WuandClaude Opus 5.5 2155f3eb46 feat(sm120): MoE on FlashInfer's Cake MegaMoE (DEEPINFER_SM120_EP_BACKEND=cake)
One persistent kernel per MoE call dispatches the tokens, runs the routed
MXFP4 experts and the FP8 shared expert, and combines, moving data over
PCIe peer load/store between reachable ranks and NCCL GIN elsewhere. The
DeepEP path keeps all peer traffic on the NICs (NCCL_LSA_TEAM_SIZE=1).

- cake_moe.py: the routed and shared expert weights convert to Cake's
  operands (bit-exact against DeepInfer's dequantized layouts) and its
  expert-major kernel layout, owned by DeepInfer so EPLB moves them in
  place. Every call is bucketed with one row bound on all EP ranks:
  decode one call bounded by the decode capacity (graph-capturable;
  the bound does not change the cost), prefill chunks of
  DEEPINFER_SM120_CAKE_CHUNK_TOKENS rows (default 2048, which bounds the
  workspace) covering the EP group's largest batch, found once per step
  by a CPU all-reduce. Warmup at userbuffer allocation creates the pooled
  workspace and checks the kernel's protocol flag.
- EPLB: Cake does not report what each rank received, so its layers count
  the routes they send and the load balancer all-reduces them once per
  step into the received counts (count_on_sender).
- Runtime: the cake backend leaves the LSA team to NCCL and warns without
  NCCL_TOPO_FILE; is_within_nvl_domain is False on SM120 (no NVLink).
- build.sh: SM120 installs NCCL 2.30.7, which Cake requires exactly.

Tests: tests/kernels/test_sm120_cake_moe.py (operand conversion, EPLB
locality of the kernel layout) and tests/distributed/test_sm120_cake_moe.py
(8 GPUs: against a reference with Cake's quantization steps, chunked
calls bitwise equal to one call, uneven and empty ranks, CUDA graph
replay). examples/offline_inference.py runs end to end on 8 GPUs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 08:55:46 -08:00
..
2026-10-20 12:09:42 +08:00
2026-10-20 12:09:42 +08:00
2026-10-20 12:09:42 +08:00
2026-10-20 12:09:42 +08:00