Files
DeepInfer/eval/cvbench
shunyadandClaude Opus 5.5 addd6e6c3e fix(eval/cvbench): do not count missing sources as 0% in the overall score
summarize_extra averaged ADE20K/COCO/Omni3D with .get(source, 0.0), so a run
that scored no rows of a source (e.g. --limit 200, which keeps only ADE20K
rows because records are taken in file order) reported an overall CV-Bench
score of at most a quarter of the real accuracy.

The overall/2D/3D scores are now reported only when every source they
average has scored rows; otherwise summary.json lists
cvbench_missing_sources and the scorer warns. Per-source and per-task
accuracies are reported either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:22:15 -08:00
..