Files
DeepInfer/deepinfer/model_executor/input_metadata
shunyadandClaude Opus 5.5 c6e0f1b067 perf(sm120): size Cake prefill calls before the layer loop, per YoCo view
The first Cake MoE layer found the EP group's largest prefill row count with
a gloo all-reduce and stored it as an ad-hoc attribute on the metadata. The
YoCo consumer view is a shallow copy (get_consumer_view), so it inherited the
full-token maximum: under compact truncation every consumer layer made as
many collective Cake calls as a full-token layer, the extra ones empty.

- DeepseekModel.forward reduces the full-token and, under compaction, the
  consumer row counts in one CPU all-reduce before the layer loop (like the
  per-layer YoCo dispatch bound) and sets each on its own metadata, so
  consumer layers make calls for their own rows.
- InputMetadata declares cake_moe_max_tokens_per_rank. Layers reached with
  metadata the forward did not prepare (e.g. draft layers) still compute it
  once on the first layer.

Tests: tests/models/test_cake_moe_prefill_calls.py (CPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
..
2026-10-20 12:09:42 +08:00
2026-10-20 12:09:42 +08:00