gensokyo-devandgensokyo-dev
8084916859
* perf(eval): measure MCP catalog usage through real Codex sessions * fix(eval): control the active Codex code mode host * fix(eval): reject unavailable host controls without hiding client errors * test(eval): cover the supported default and actionable client errors * docs(eval): record pinned Codex catalog and real package measurements * fix(eval): pin matrix identity and retain failed trial observations --------- Co-authored-by: gensokyo-dev <gensokyo-dev@users.noreply.github.com>
1.6 KiB
Analysis verification test support
The maintained catalog-eval.py entrypoint (npm run eval:mcp-catalog) uses the
real local Codex CLI and the retained resource/agent runners. See
MCP catalog evaluation for its matrix,
observed-token accounting, isolation and completed-task limits.
These files support the existing comparison, independent-oracle, receipt and
cleanup regressions. The TypeScript consumers live in tests/boundary/,
tests/evaluation/ and tests/conformance/; npm run test:python discovers
the Python checks here. Build first for checks that use the compiled runtime.
The agent-harness tests inject command outcomes and own temporary workspaces. They do not invoke a real model or establish real-provider behavior. The retained resource observer keeps its process-family CPU, memory and cleanup guards. Public corpus identities and original receipt provenance stay intact.
The historical allocation report and receipt JSON belong to documentation. Their numerical sections are produced and checked by:
node tests/support/analysis-efficiency/mcp-integer-report.mjs --check
The former broad experiment suite was retired. For historical reproduction, use the recorded runtime and producer revisions; the archived source tree preserves the former layout. Current deterministic checks do not requalify those historical measurements.