mirror of
https://github.com/deepseek-ai/DeepInfer.git
synced 2026-11-20 04:22:27 +00:00
One persistent kernel per MoE call dispatches the tokens, runs the routed MXFP4 experts and the FP8 shared expert, and combines, moving data over PCIe peer load/store between reachable ranks and NCCL GIN elsewhere. The DeepEP path keeps all peer traffic on the NICs (NCCL_LSA_TEAM_SIZE=1). - cake_moe.py: the routed and shared expert weights convert to Cake's operands (bit-exact against DeepInfer's dequantized layouts) and its expert-major kernel layout, owned by DeepInfer so EPLB moves them in place. Every call is bucketed with one row bound on all EP ranks: decode one call bounded by the decode capacity (graph-capturable; the bound does not change the cost), prefill chunks of DEEPINFER_SM120_CAKE_CHUNK_TOKENS rows (default 2048, which bounds the workspace) covering the EP group's largest batch, found once per step by a CPU all-reduce. Warmup at userbuffer allocation creates the pooled workspace and checks the kernel's protocol flag. - EPLB: Cake does not report what each rank received, so its layers count the routes they send and the load balancer all-reduces them once per step into the received counts (count_on_sender). - Runtime: the cake backend leaves the LSA team to NCCL and warns without NCCL_TOPO_FILE; is_within_nvl_domain is False on SM120 (no NVLink). - build.sh: SM120 installs NCCL 2.30.7, which Cake requires exactly. Tests: tests/kernels/test_sm120_cake_moe.py (operand conversion, EPLB locality of the kernel layout) and tests/distributed/test_sm120_cake_moe.py (8 GPUs: against a reference with Cake's quantization steps, chunked calls bitwise equal to one call, uneven and empty ranks, CUDA graph replay). examples/offline_inference.py runs end to end on 8 GPUs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>