Commit Graph
32 Commits
Author SHA1 Message Date
shunyadandClaude Opus 5.5 f009131253 refactor: import deep_ep where it is used
SM120 has no DeepEP feature left (no deepep backend, Engram in host memory,
no context parallelism, no NVLink domain), yet five modules imported deep_ep
at the top, so every SM120 process needed a DeepEP build, one compiled
against an NCCL newer than the one the process loads.

- mlp, engram, cp, userbuffer: deep_ep names used only in annotations move
  under TYPE_CHECKING; EPBuffer, EngramBuffer, BucketBuffer and the CP
  alignment helpers are imported in the functions that build them.
- parallel_state: is_within_nvl_domain imports deep_ep after the SM120 and
  single-rank returns.

No behavior change where DeepEP is installed: the same code imports it,
later. tests/test_sm120_without_deep_ep.py imports each module in a fresh
interpreter that cannot import deep_ep, and runs the SM120 paths (nccl EP
buffer, no CP buffer, no NVLink domain) with deep_ep blocked; the nccl
backend allocation test blocks deep_ep too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:43:19 -08:00
shunyadandClaude Opus 5.5 a2d45e4f22 refactor(sm120): drop the deepep MoE backend on SM120
DeepEP needs NCCL 2.31 or newer and Cake MegaMoE exactly 2.30.7, and a
process loads one NCCL: since the SM120 build installs 2.30.7, the deepep
backend could only fail at startup. Remove it so SM120 can stop building
DeepEP:

- DEEPINFER_SM120_EP_BACKEND accepts auto, cake and nccl; deepep is refused
  by the choice check.
- userbuffer drops the SM120 EPBuffer path (_create_sm120_ep_buffer and its
  notify-thread and LSA-team checks): on SM120 only the nccl backend
  allocates an EP buffer, and allocate_ep asserts that.
- sm120_support drops the DeepEP NCCL check; runtime_config keeps the nccl
  backend's NCCL settings (single-rank LSA team, no graph registration).
- Tests: the deepep cases go; the environment tests use nccl or cake, and
  the NCCL fixture defaults to 2.30.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:34:48 -08:00
shunyad 7eb9f52897 Merge pull request #293 from shunyad-ai/shunyad/sm120-cake
fix(sm120): follow-ups to #291 (Cake protocol errors, NCCL check and auto backend, prefill call sizing)
2026-11-19 15:27:43 +08:00
shunyadandClaude Opus 5.5 fe13eb0251 fix(build): check the SM120 NCCL pin after installing it
build.sh builds DeepEP against NCCL 2.32.3 and then replaces NCCL with 2.30.7
for Cake. deepinfer imports deep_ep on every platform, so if that build no
longer loaded with 2.30.7 every SM120 run would fail at import, with nothing
pointing at the NCCL swap; and if torch loaded another libnccl.so.2, auto
would quietly pick the slower nccl backend.

- scripts/check_sm120_nccl.py runs after the pin: it fails the build when
  deep_ep has an undefined symbol under the pinned NCCL, and warns when it
  cannot import deep_ep for another reason (e.g. no GPU on the build host)
  or when torch loads an NCCL other than the pin.
- The pin is CAKE_NCCL_VERSION in build.sh; tests/test_build_nccl_pin.py
  keeps it equal to sm120_support.CAKE_NCCL_VERSION and covers the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
shunyadandClaude Opus 5.5 4c668001b5 fix(sm120): refuse unsupported Cake MegaMoE configs with ValueError
Cake's requirements on the model and settings (an integer SwiGLU clamp, at
most one shared expert) were asserts, which `python -O` strips: the layer
would then build with a truncated clamp or a dropped shared expert.
check_cake_config raises ValueError naming the nccl fallback, and also
refuses a non-positive DEEPINFER_SM120_CAKE_CHUNK_TOKENS and experts that do
not split evenly over the EP group. DeepseekMoE checks before converting the
weights; CakeMoE checks again for direct use.

expert_major_views notes why slicing L of L + 1 scale segments is right
without a shared expert too: FlashInfer fills the shared segment with
neutral scales then.

Tests: tests/models/test_cake_moe_config.py (CPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
shunyadandClaude Opus 5.5 c6e0f1b067 perf(sm120): size Cake prefill calls before the layer loop, per YoCo view
The first Cake MoE layer found the EP group's largest prefill row count with
a gloo all-reduce and stored it as an ad-hoc attribute on the metadata. The
YoCo consumer view is a shallow copy (get_consumer_view), so it inherited the
full-token maximum: under compact truncation every consumer layer made as
many collective Cake calls as a full-token layer, the extra ones empty.

- DeepseekModel.forward reduces the full-token and, under compaction, the
  consumer row counts in one CPU all-reduce before the layer loop (like the
  per-layer YoCo dispatch bound) and sets each on its own metadata, so
  consumer layers make calls for their own rows.
- InputMetadata declares cake_moe_max_tokens_per_rank. Layers reached with
  metadata the forward did not prepare (e.g. draft layers) still compute it
  once on the first layer.

Tests: tests/models/test_cake_moe_prefill_calls.py (CPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
shunyadandClaude Opus 5.5 2f18193bb5 fix(sm120): check NCCL for the cake backend; default to auto
Cake MegaMoE runs only with NCCL 2.30.7, but only deepep had a startup NCCL
check: with any other NCCL the default backend failed inside FlashInfer.

- DEEPINFER_SM120_EP_BACKEND gains auto, the new default: cake when the
  loaded NCCL is 2.30.7 (what build.sh installs), otherwise nccl with a
  warning. sm120_support.ep_backend() resolves it once per process and
  writes the result back to the environment variable, so the worker
  settings, the MoE layers and the userbuffer manager (and child processes)
  all see the same backend.
- sm120_support.check_ep_backend_nccl() refuses an explicit cake with
  another NCCL, and moves the deepep >= 2.31 check out of runtime_config;
  nccl_runtime_version() moves to sm120_support with it.
- MXFP4 x FP8 stays required for cake, as an error rather than a fallback:
  the worker's NCCL settings (LSA team, graph registration) are made for
  the backend before the model's expert format is known. The message now
  says the backend may have come from auto.

Tests: auto resolution (once, written back), explicit backends untouched,
cake refused with other NCCL, auto falling back to nccl in the worker
environment (tests/worker/test_nccl_ep.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
shunyadandClaude Opus 5.5 d2755d60a1 fix(sm120): check Cake MegaMoE protocol errors at runtime
The Cake kernel reports an invalid call (e.g. its lossless return codec ran
out of overflow blocks) in a fail-closed flag, but DeepInfer read it only
once at warmup, so such a call silently returned wrong MoE outputs. The flag
is per call (the next launch clears it) and lives in the workspace all
layers share, so it has to be captured right after each call:

- CakeMoE ORs the flag into its own protocol_errors tensor after every
  call, on the stream (no host sync, also inside CUDA graphs), and names
  itself after its MoE layer.
- userbuffer.check_cake_moe_errors(), called once per step from
  DeepseekForCausalLM.sample, copies the layers' flags to pinned memory
  behind the step and checks the previous step's copy, so an error stops
  the run one step later without a host synchronization.
- check_protocol reads the same flag tensor.

Tests: tests/worker/test_cake_moe_errors.py (CPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
shunyadandClaude Opus 5.5 f7da10314b style(sm120): fix ruff errors in the Cake MegaMoE change
The PR #291 files failed `ruff check` (ANN001 on cake_moe.expert_major_views,
userbuffer.register_cake_moe and the test helpers; E741 on the tests' `I`)
and `ruff format --check`. Annotate the arguments (CakeMoE under
TYPE_CHECKING in userbuffer), rename the intermediate size to INTER in the
tests, and apply ruff format. No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:46:30 -08:00
shunyad 861b2a81c4 Merge pull request #292 from shunyad-ai/shunyad/sm120
fix(sm120): follow-ups to #289 (hardware detection, Engram placement, prefill KV in place, capability table)
2026-11-19 14:45:30 +08:00
shunyadandClaude Opus 5.5 2a5ba049cc perf(sm120): read the compressed KV in place in prefill attention
SM120 prefill gathered the paged compressed (CA) KV into BF16 and then
re-quantized the whole gathered buffer into temporary row caches, for every
layer and every prefill chunk: two extra full passes over total_k_compressed
tokens plus a temporary cache, although decode already reads the same paged
records in place.

Prefill now reads the compressed cache in place: the compressed indices
count positions in the gathered order of the prefill load plan
(load_segments_ca), so load_plan_cache_rows maps those positions to cache
rows and gathered_to_cache_rows applies the map to the indices. The values
read are the stored records the re-quantization reproduced, so results do
not change. V4Attention skips the BF16 gather of the compressed part (and the
share-KV gather buffer) where prefill_reads_compressed_cache_in_place().

The SWA part is still gathered and quantized: the current chunk's stores
overwrite ring slots that earlier queries of the chunk still attend to.

prefill_attn_sparse_swa_ca now takes the gathered SWA KV, the compressed
cache and its load plan instead of one gathered buffer. The SM120 kernel
tests follow the new API, compare the compressed part against the stored
records, and check that the mapped rows address those records; CPU tests
cover the row mapping, zero-filled positions and out-of-range entries past
a length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:28:43 -08:00
shunyadandClaude Opus 5.5 9da41ad18e refactor(platform): describe NVIDIA families with a capability table
Call sites tested `not PLATFORM.is_sm120` to mean "has the SM100 fused
kernels", hard-coded SM120 constants next to their use, and treated any GPU
that is not 12.x as SM100. A new family (or 12.x variant) would need every
site revisited, and an unsupported GPU silently took SM100 paths.

- deepinfer.platform.NVIDIA_ARCHS maps compute capability major to an
  NvidiaArch entry: sm100_fused_kernels, grouped_gemm_zero_padding,
  kv_block_alignment and default_comm_sms, with the same values SM100 and
  SM120 used before.
- Platform exposes them as capabilities (has_sm100_fused_kernels, ...), with
  the previous values on Ascend; the mega MoE / gate / mHC, fused
  norm-rope-attn-rope-cast, FP8 einsum output, grouped GEMM zero padding, KV
  block alignment and default EP comm SM sites ask for the capability.
- PlatformGPU.nvidia_arch selects the entry from the first visible GPU. No
  visible GPU still means SM100; a GPU of an unsupported family now raises
  instead of running SM100 kernels. is_sm120 stays for the SM120 backends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:28:43 -08:00
shunyadandClaude Opus 5.5 c23e8c3a97 refactor(parallel_utils): reuse tilelang_ops.common.align instead of local _align copies
nccl_ep.py and host_engram.py each defined a private _align identical to the
existing tilelang_ops.common.align (an import-free constants module, so the
communication modules stay lightweight).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:28:43 -08:00
shunyadandClaude Opus 5.5 7fee1340a7 refactor(index_attn): drop SM120 arguments from the unreachable decode logits path
_topk_decode returns through _sm120_topk_decode on SM120 before it reaches
_compute_decode_p_sum, so the SM120-only `weights.float()` and
_dense_logits_kwargs() there never applied. Restore the call to its SM100 /
Ascend form so readers do not assume SM120 takes this path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:28:43 -08:00
shunyadandClaude Opus 5.5 704094524d perf(sm120): size new sparse-MLA shapes from the current scratch arenas
Every call shape not memoized yet started its scratch search from empty
(0, 0) views, so it always failed FlashInfer's capacity check and retried
(up to three wrapper.run calls). Prefill shapes change almost every step, so
the first SM120 attention call of each prefill step paid one or two thrown
ValueErrors plus FlashInfer's validate/prepare for each failed attempt, and
those attempts filled the per-wrapper plan cache with dead entries.

A new shape now first tries views of the whole current arenas, which fit
most shapes once a few calls have grown them; only a larger need pays a
failed attempt before the arena grows. Memoized shapes keep their views, so
CUDA graph captures are unaffected.

Adds a CPU test of the sizing with a stand-in wrapper (skipped without
flashinfer).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:28:43 -08:00
shunyadandClaude Opus 5.5 cb1d0b3ee2 fix(eplb): name the draft-layer settings in EPLB validation errors
Draft-layer schedules went through the main-model validation, so a bad
DEEPINFER_DRAFT_EXPERT_COUNT_WINDOW_SIZE was reported as
"DEEPINFER_EXPERT_COUNT_WINDOW_SIZE=... must be between 1 and
DEEPINFER_EXPERT_REARRANGEMENT_STEP_INTERVAL=...", naming variables the user
never set. Draft settings could also be dropped without a word.

- ExpertLoadBalancer takes setting_names for its error messages; the draft
  schedule passes the draft variables (or says the interval is inherited, or
  the window is the default).
- Warn when DEEPINFER_DRAFT_EXPERT_COUNT_WINDOW_SIZE is set but no draft
  interval resolves, and when draft layers share an EPLB layout group with
  main-model layers and so cannot follow the draft schedule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:28:43 -08:00
shunyadandClaude Opus 5.5 addd6e6c3e fix(eval/cvbench): do not count missing sources as 0% in the overall score
summarize_extra averaged ADE20K/COCO/Omni3D with .get(source, 0.0), so a run
that scored no rows of a source (e.g. --limit 200, which keeps only ADE20K
rows because records are taken in file order) reported an overall CV-Bench
score of at most a quarter of the real accuracy.

The overall/2D/3D scores are now reported only when every source they
average has scored rows; otherwise summary.json lists
cvbench_missing_sources and the scorer warns. Per-source and per-task
accuracies are reported either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:22:15 -08:00
shunyadandClaude Opus 5.5 7b4322294e fix(build): stop on GPU family detection errors instead of building for sm100
build.sh tested `[ "$(nvidia_arch.sh)" = sm120 ]`, which discards the
script's exit status, so a bad DEEPINFER_NVIDIA_ARCH only surfaced in
build_vendor.sh after the long pip install. nvidia_arch.sh also mapped every
GPU that is not 12.x (and hosts without nvidia-smi) to sm100 silently, so an
SM120 build container without nvidia-smi installed the SM100 dependency set.

- build.sh runs detection as its own assignment (set -e stops before pip) and
  exports the result so build_vendor.sh builds for the same family.
- DEEPINFER_NVIDIA_ARCH is case-insensitive.
- A detected GPU of another family (e.g. compute capability 9.0) is an error
  that points at DEEPINFER_NVIDIA_ARCH; no visible GPU still builds for sm100
  (GPU-less SM100 build hosts keep working) but warns on stderr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:22:15 -08:00
shunyadandClaude Opus 5.5 bdb6014ca1 fix(eval): default MTP varlen off on SM120 and record the effective value
run_eval.py forced DEEPINFER_ENABLE_MTP_VARLEN=1, which the engine refuses on
SM120, so `run_eval.py --bench aime|hle|gpqa` (and their driver scripts)
failed at LLMEngine init unless the user remembered --env ...=0. meta.json
also assumed mtp_varlen=True whenever --env did not mention it.

- One _default_engine_env() supplies the defaults for both the rank processes
  and meta.json: MTP varlen on, except on SM120.
- meta.json's mtp / mtp_varlen and bench run_metadata now see the same
  defaults-then---env environment the ranks run with.
- README.sm120.md no longer asks for --env DEEPINFER_ENABLE_MTP_VARLEN=0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:22:14 -08:00
shunyadandClaude Opus 5.5 8898e97395 fix(sm120): refuse multi-host Engram groups right after rendezvous
Host-resident Engram tables are one /dev/shm mapping shared through
/proc/<pid>/fd, so every Engram group must sit on one host. The default
engram_parallel_size (0 = every rank) spans all nodes of a multi-node
deployment, which only failed on a bare assert in HostEngramBuffer after the
weights were loaded (and under python -O failed later with a misleading
"cannot open /proc/..." error).

- sm120_support.check_rank_placement (was check_distinct_devices) now also
  checks, from the hostnames it already gathers, that each Engram group
  (contiguous rank blocks) stays on one host, and names the offending ranks.
- HostEngramBuffer raises ValueError instead of asserting.
- README.sm120.md says multi-node deployments must set engram_parallel_size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:22:14 -08:00
shunyadandClaude Opus 5.5 05cd50f21a fix(hardware): resolve MIG entries and ambiguous device order in compute_capability
compute_capability() only understood digit indices and GPU- UUIDs, so the first
CUDA_VISIBLE_DEVICES entry being a MIG instance made is_sm_major_10/12 return
False: SM120 then imported flash_mla/deep_select (not built there) and SM100
lost its GPU-type-dependent paths. Unresolvable entries also silently read as
"not this architecture".

- Resolve "MIG-<uuid>" (via NVML's MIG lookups) and the legacy
  "MIG-GPU-<uuid>/<gi>/<ci>" form to the capability of the physical GPU.
- Raise LookupError when NVML is present but the entry names no GPU, instead
  of returning None.
- Digit entries are NVML (PCI bus) indices, but CUDA orders devices fastest
  first unless CUDA_DEVICE_ORDER=PCI_BUS_ID. On hosts whose GPUs differ in
  capability that is ambiguous, so ask for CUDA_DEVICE_ORDER=PCI_BUS_ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:22:14 -08:00
shunyad f29e7209a4 Merge pull request #291 from lucifer1004/sm120-cake-moe
feat(sm120): MoE on FlashInfer's Cake MegaMoE, the new SM120 default
2026-11-19 14:14:46 +08:00
Zihua WuandClaude Opus 5.5 4bc0ced7d0 feat(sm120): make Cake MegaMoE the default MoE backend
Cake needs NCCL 2.30.7 in the process, and the pinned DeepEP needs NCCL 2.31+
(ginMinStride, ginSupport, ncclDevCommRequirements.ginType, ...), so the two
cannot share a process. On SM120:

- DEEPINFER_SM120_EP_BACKEND defaults to cake; nccl stays the fallback for
  expert formats other than MXFP4 x FP8, named in the cake error.
- build.sh builds the vendored packages against NCCL 2.32.3 (DeepEP no longer
  compiles against 2.30.7) and installs 2.30.7 afterwards; deep_ep is still
  imported but its backend is unavailable.
- deepep is rejected at worker setup when the loaded NCCL is older than 2.31,
  read with ncclGetVersion (torch.cuda.nccl.version() reports torch's build).
- README.sm120.md documents the backends, the NCCL pin, NCCL_TOPO_FILE, the
  chunk size, and the 8-GPU results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:16:27 -08:00
Zihua WuandClaude Opus 5.5 eb4bb125f2 fix(sm120): destroy Cake MegaMoE layers before the process group
Cake's layers own collective NCCL state (symmetric windows, the GIN device
communicator) and must be destroyed on every EP rank, in the same order,
before the process group. Register them with the userbuffer manager, which
destroys them first in close(); left to garbage collection at exit, a rank
could fail after the run had finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 09:39:24 -08:00
Zihua WuandClaude Opus 5.5 2155f3eb46 feat(sm120): MoE on FlashInfer's Cake MegaMoE (DEEPINFER_SM120_EP_BACKEND=cake)
One persistent kernel per MoE call dispatches the tokens, runs the routed
MXFP4 experts and the FP8 shared expert, and combines, moving data over
PCIe peer load/store between reachable ranks and NCCL GIN elsewhere. The
DeepEP path keeps all peer traffic on the NICs (NCCL_LSA_TEAM_SIZE=1).

- cake_moe.py: the routed and shared expert weights convert to Cake's
  operands (bit-exact against DeepInfer's dequantized layouts) and its
  expert-major kernel layout, owned by DeepInfer so EPLB moves them in
  place. Every call is bucketed with one row bound on all EP ranks:
  decode one call bounded by the decode capacity (graph-capturable;
  the bound does not change the cost), prefill chunks of
  DEEPINFER_SM120_CAKE_CHUNK_TOKENS rows (default 2048, which bounds the
  workspace) covering the EP group's largest batch, found once per step
  by a CPU all-reduce. Warmup at userbuffer allocation creates the pooled
  workspace and checks the kernel's protocol flag.
- EPLB: Cake does not report what each rank received, so its layers count
  the routes they send and the load balancer all-reduces them once per
  step into the received counts (count_on_sender).
- Runtime: the cake backend leaves the LSA team to NCCL and warns without
  NCCL_TOPO_FILE; is_within_nvl_domain is False on SM120 (no NVLink).
- build.sh: SM120 installs NCCL 2.30.7, which Cake requires exactly.

Tests: tests/kernels/test_sm120_cake_moe.py (operand conversion, EPLB
locality of the kernel layout) and tests/distributed/test_sm120_cake_moe.py
(8 GPUs: against a reference with Cake's quantization steps, chunked
calls bitwise equal to one call, uneven and empty ranks, CUDA graph
replay). examples/offline_inference.py runs end to end on 8 GPUs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 08:55:46 -08:00
shunyad fc099cd21e Merge pull request #289 from lucifer1004/sm120-offline
Offline inference on NVIDIA Blackwell SM120 (RTX PRO 6000)
2026-11-18 16:07:33 +08:00
Zihua WuandClaude Opus 5.5 efecbb5ba6 feat(sm120): offline inference on NVIDIA Blackwell SM120
Run DeepSeek-V4.1-Flash offline inference on SM120 GPUs (RTX PRO 6000
Blackwell) with the same entry points as SM100. SM100 and Ascend paths
are unchanged.

- Build: SM120 is detected from the first visible GPU
  (scripts/nvidia_arch.sh). It builds DeepEP and DeepGEMM from its nv_dev
  branch (vendor/DeepGEMM-sm120), and no FlashMLA or DeepSelect;
  FlashInfer is installed from a pinned commit
  (requirements-nvidia-sm120.txt).
- Attention: V4.1 sparse MLA runs on FlashInfer's SM120 sparse MLA
  kernels, reading the inline-scale KV records as stored; KV blocks are
  aligned to 512 bytes.
- Indexer: FlashInfer's fused DSA indexer scores the index keys and
  selects their top-k in one call, for decode on every indexer layer
  (candidate source and consumer layers included) and for prefill outside
  the candidate layers. The remaining selections (prefill of the
  candidate layers, the sampler) use FlashInfer's top-k behind
  DeepSelect's interface (model_executor/layers/topk.py), in
  deterministic mode.
- MoE EP: DeepEP over NCCL GIN by default, or NCCL collectives
  (DEEPINFER_SM120_EP_BACKEND=nccl); NCCL settings known to break the EP
  paths are rejected. EPLB, including decode-only draft-layer EPLB, and
  per-layer YoCo MoE dispatch.
- Engram tables live in page-locked host shared memory; killed workers
  leave nothing in /dev/shm.
- Features: attention DP + MoE EP, DSpark/nextn speculative decoding,
  CUDA graphs, async scheduling, decoder SWA bounded replay, structured
  outputs, image inputs, multi-node deployments. Features without an SM120
  implementation (CP > 1, MTP varlen, batch-invariant mode, GPU-resident
  Engram tables) are refused at engine start.
- Eval: chat GSM8K and CV-Bench; tests for the SM120 kernels and their
  DeepInfer call sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:32:01 -08:00
Inference_ 553741557a fix: validate boolean environment variables with strict parsing (#218) 2026-10-25 19:59:27 +08:00
Shaoyuan CHENandShaoyuan Chen 9bd9cf243f Bump DeepEP to f6aa38fe and drop the SSH->HTTPS rewrite (#157)
DeepEP now records an HTTPS DeepJIT URL, so build_vendor no longer needs the git@github.com rewrite.

Co-authored-by: Shaoyuan Chen <chensy@deepseek.com>
2026-10-21 12:55:43 +08:00
hkeee21 774ce244b8 update Q&A EN doc 2026-10-20 19:03:13 +08:00
Jiashi Li 8b16149741 update Q&A documentation 2026-10-20 17:06:57 +08:00
Jiashi Li d585e6542c initial commit 2026-10-20 12:09:42 +08:00