Commit Graph
5 Commits
Author SHA1 Message Date
shunyadandClaude Opus 5.5 f7da10314b style(sm120): fix ruff errors in the Cake MegaMoE change
The PR #291 files failed `ruff check` (ANN001 on cake_moe.expert_major_views,
userbuffer.register_cake_moe and the test helpers; E741 on the tests' `I`)
and `ruff format --check`. Annotate the arguments (CakeMoE under
TYPE_CHECKING in userbuffer), rename the intermediate size to INTER in the
tests, and apply ruff format. No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
shunyadandClaude Opus 5.5 2a5ba049cc perf(sm120): read the compressed KV in place in prefill attention
SM120 prefill gathered the paged compressed (CA) KV into BF16 and then
re-quantized the whole gathered buffer into temporary row caches, for every
layer and every prefill chunk: two extra full passes over total_k_compressed
tokens plus a temporary cache, although decode already reads the same paged
records in place.

Prefill now reads the compressed cache in place: the compressed indices
count positions in the gathered order of the prefill load plan
(load_segments_ca), so load_plan_cache_rows maps those positions to cache
rows and gathered_to_cache_rows applies the map to the indices. The values
read are the stored records the re-quantization reproduced, so results do
not change. V4Attention skips the BF16 gather of the compressed part (and the
share-KV gather buffer) where prefill_reads_compressed_cache_in_place().

The SWA part is still gathered and quantized: the current chunk's stores
overwrite ring slots that earlier queries of the chunk still attend to.

prefill_attn_sparse_swa_ca now takes the gathered SWA KV, the compressed
cache and its load plan instead of one gathered buffer. The SM120 kernel
tests follow the new API, compare the compressed part against the stored
records, and check that the mapped rows address those records; CPU tests
cover the row mapping, zero-filled positions and out-of-range entries past
a length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:28:43 -08:00
shunyadandClaude Opus 5.5 704094524d perf(sm120): size new sparse-MLA shapes from the current scratch arenas
Every call shape not memoized yet started its scratch search from empty
(0, 0) views, so it always failed FlashInfer's capacity check and retried
(up to three wrapper.run calls). Prefill shapes change almost every step, so
the first SM120 attention call of each prefill step paid one or two thrown
ValueErrors plus FlashInfer's validate/prepare for each failed attempt, and
those attempts filled the per-wrapper plan cache with dead entries.

A new shape now first tries views of the whole current arenas, which fit
most shapes once a few calls have grown them; only a larger need pays a
failed attempt before the arena grows. Memoized shapes keep their views, so
CUDA graph captures are unaffected.

Adds a CPU test of the sizing with a stand-in wrapper (skipped without
flashinfer).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:28:43 -08:00
Zihua WuandClaude Opus 5.5 2155f3eb46 feat(sm120): MoE on FlashInfer's Cake MegaMoE (DEEPINFER_SM120_EP_BACKEND=cake)
One persistent kernel per MoE call dispatches the tokens, runs the routed
MXFP4 experts and the FP8 shared expert, and combines, moving data over
PCIe peer load/store between reachable ranks and NCCL GIN elsewhere. The
DeepEP path keeps all peer traffic on the NICs (NCCL_LSA_TEAM_SIZE=1).

- cake_moe.py: the routed and shared expert weights convert to Cake's
  operands (bit-exact against DeepInfer's dequantized layouts) and its
  expert-major kernel layout, owned by DeepInfer so EPLB moves them in
  place. Every call is bucketed with one row bound on all EP ranks:
  decode one call bounded by the decode capacity (graph-capturable;
  the bound does not change the cost), prefill chunks of
  DEEPINFER_SM120_CAKE_CHUNK_TOKENS rows (default 2048, which bounds the
  workspace) covering the EP group's largest batch, found once per step
  by a CPU all-reduce. Warmup at userbuffer allocation creates the pooled
  workspace and checks the kernel's protocol flag.
- EPLB: Cake does not report what each rank received, so its layers count
  the routes they send and the load balancer all-reduces them once per
  step into the received counts (count_on_sender).
- Runtime: the cake backend leaves the LSA team to NCCL and warns without
  NCCL_TOPO_FILE; is_within_nvl_domain is False on SM120 (no NVLink).
- build.sh: SM120 installs NCCL 2.30.7, which Cake requires exactly.

Tests: tests/kernels/test_sm120_cake_moe.py (operand conversion, EPLB
locality of the kernel layout) and tests/distributed/test_sm120_cake_moe.py
(8 GPUs: against a reference with Cake's quantization steps, chunked
calls bitwise equal to one call, uneven and empty ranks, CUDA graph
replay). examples/offline_inference.py runs end to end on 8 GPUs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 08:55:46 -08:00
Zihua WuandClaude Opus 5.5 efecbb5ba6 feat(sm120): offline inference on NVIDIA Blackwell SM120
Run DeepSeek-V4.1-Flash offline inference on SM120 GPUs (RTX PRO 6000
Blackwell) with the same entry points as SM100. SM100 and Ascend paths
are unchanged.

- Build: SM120 is detected from the first visible GPU
  (scripts/nvidia_arch.sh). It builds DeepEP and DeepGEMM from its nv_dev
  branch (vendor/DeepGEMM-sm120), and no FlashMLA or DeepSelect;
  FlashInfer is installed from a pinned commit
  (requirements-nvidia-sm120.txt).
- Attention: V4.1 sparse MLA runs on FlashInfer's SM120 sparse MLA
  kernels, reading the inline-scale KV records as stored; KV blocks are
  aligned to 512 bytes.
- Indexer: FlashInfer's fused DSA indexer scores the index keys and
  selects their top-k in one call, for decode on every indexer layer
  (candidate source and consumer layers included) and for prefill outside
  the candidate layers. The remaining selections (prefill of the
  candidate layers, the sampler) use FlashInfer's top-k behind
  DeepSelect's interface (model_executor/layers/topk.py), in
  deterministic mode.
- MoE EP: DeepEP over NCCL GIN by default, or NCCL collectives
  (DEEPINFER_SM120_EP_BACKEND=nccl); NCCL settings known to break the EP
  paths are rejected. EPLB, including decode-only draft-layer EPLB, and
  per-layer YoCo MoE dispatch.
- Engram tables live in page-locked host shared memory; killed workers
  leave nothing in /dev/shm.
- Features: attention DP + MoE EP, DSpark/nextn speculative decoding,
  CUDA graphs, async scheduling, decoder SWA bounded replay, structured
  outputs, image inputs, multi-node deployments. Features without an SM120
  implementation (CP > 1, MTP varlen, batch-invariant mode, GPU-resident
  Engram tables) are refused at engine start.
- Eval: chat GSM8K and CV-Bench; tests for the SM120 kernels and their
  DeepInfer call sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:32:01 -08:00