mirror of
https://github.com/morluto/rea.git
synced 2026-11-08 07:46:28 +00:00
gensokyo-devandgensokyo-dev
8084916859
* perf(eval): measure MCP catalog usage through real Codex sessions * fix(eval): control the active Codex code mode host * fix(eval): reject unavailable host controls without hiding client errors * test(eval): cover the supported default and actionable client errors * docs(eval): record pinned Codex catalog and real package measurements * fix(eval): pin matrix identity and retain failed trial observations --------- Co-authored-by: gensokyo-dev <gensokyo-dev@users.noreply.github.com>
29 lines
1.6 KiB
Markdown
29 lines
1.6 KiB
Markdown
# Analysis verification test support
|
|
|
|
The maintained `catalog-eval.py` entrypoint (`npm run eval:mcp-catalog`) uses the
|
|
real local Codex CLI and the retained resource/agent runners. See
|
|
[MCP catalog evaluation](../../../docs/mcp-catalog-evaluation.md) for its matrix,
|
|
observed-token accounting, isolation and completed-task limits.
|
|
|
|
These files support the existing comparison, independent-oracle, receipt and
|
|
cleanup regressions. The TypeScript consumers live in `tests/boundary/`,
|
|
`tests/evaluation/` and `tests/conformance/`; `npm run test:python` discovers
|
|
the Python checks here. Build first for checks that use the compiled runtime.
|
|
|
|
The agent-harness tests inject command outcomes and own temporary workspaces.
|
|
They do not invoke a real model or establish real-provider behavior. The
|
|
retained resource observer keeps its process-family CPU, memory and cleanup
|
|
guards. Public corpus identities and original receipt provenance stay intact.
|
|
|
|
The historical [allocation report and receipt JSON](../../../docs/verification/analysis-efficiency/mcp-integer-allocation.md)
|
|
belong to documentation. Their numerical sections are produced and checked by:
|
|
|
|
```sh
|
|
node tests/support/analysis-efficiency/mcp-integer-report.mjs --check
|
|
```
|
|
|
|
The former broad experiment suite was retired. For historical reproduction, use the
|
|
recorded runtime and producer revisions; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency)
|
|
preserves the former layout. Current deterministic checks do not requalify
|
|
those historical measurements.
|