Files
reverse-engineering-anything/docs/mcp-catalog-evaluation.md
T
morluto 5c53bcaed8 feat(mcp)!: retire the full schema profiles and declare schema facts as typed annotations (#10693)
* test(mcp): check advertised schemas against a portable dialect in every profile

* fix(mcp): re-emit advertised patterns in a portable regex grammar

* fix(mcp): present discover_tools input through the selected profile

The hot-cold catalog registered discover_tools directly with the SDK, so
the default compact input profile still advertised its canonical schema,
including page-size defaults clients apply on their own (#5322). Route
every listed input schema, transport discovery included, through one
server-owned presentation, and pass the selected presentation to the
deferred server, which serves recorded tool schemas but not discovery.

* fix(mcp): keep the Evidence union on described borrowed inputs

borrowEvidenceInput installed its advertised projection on one schema
instance. A call site's .describe() rebuilds the schema from its
definition, so match_native_functions advertised left and right as
unconstrained values in every input profile. Install the projection in
the borrowed parser's constructor so derived schemas keep it, for both
borrowed inputs and borrowed arrays.

* fix(mcp): validate advertised dialect by selected profile

* fix(mcp): preserve schema widening through references

* feat(mcp): project compact input schemas into the portable advertised dialect

One projection (advertisedDialectProjection.ts) owns the compact input
advertisement: one type per node, items on every array, string literals
as enums, oneOf widened to anyOf, allOf merged, tuples as element unions,
exclusive bounds made inclusive, and every keyword outside the dialect
dropped. Each rewrite is exact or widening; server validation is
unchanged. The root description and sentences for dropped cross-field
constraints move to the tool description, which Codex and Zed keep.

The compact baseline falls to nullable values only (29), which the
dialect cannot express without narrowing an input. The gateway shape
repairs (typeArrayItemUnions, hasGatewayIncompatibleShape), the second
dependentRequired-to-prose copy and the strict-gateway test denylist are
removed; the dialect checker gates the inline presentation instead.

* feat(mcp): omit output schemas from tools/list by default

No surveyed model API reads an output schema, MCP client validators
repeat REA's own result validation, and complete output schemas were
about 92% of the default tools/list. REA_MCP_OUTPUT_SCHEMA_PROFILE now
resolves to off through one owner (resolveMcpOutputSchemaProfile);
compact and full remain selectable. structuredContent is still
delivered, and the deferred server serves both off and full from the
recorded catalog with the matching recorded identity.

Tests and artifact verifiers that validate results against complete
advertised output contracts select full explicitly.

* docs: describe omitted output schemas as the default in README translations

* fix(mcp): select schemas in verification workflows

* fix(mcp): refuse zero-padded repeat bounds and record host-specific path patterns

* fix(mcp): keep projected schemas permissive

* fix(mcp): keep references valid when presentation widens their targets

* fix(mcp): keep a single union branch's closure scoped to its properties

* fix(mcp): select complete output schemas in the Electron verifier

* fix(mcp): select complete output schemas in standalone verifier probes

* fix(mcp): keep an allOf member's tuple tail with its own positions

* fix(mcp): relax references into negations and conditionals presentation may remove

* refactor(contracts): declare root discriminators as typed schema annotations

* test(mcp): select the borrowed Evidence contract from its family

* fix(mcp): keep astral literals and dynamic references widening in every mode

* fix(mcp): omit astral literals and relax untracked references when presentation widens

* fix(mcp): select complete output schemas in every verifier transport

* fix(mcp): read reference pointers by schema position, not segment spelling

* fix(mcp): protect references into collapsing boolean unions and keep referencing branches out of the collapse

* fix(mcp): keep reduced schemas open when their root admits names by pattern

* test(mcp): keep the reduced-schema pattern case outside the size-limited block

* fix(mcp): omit patterns whose classes name astral characters

* fix(mcp): decide union collapse and pointer protection from canonical branches

* docs(mcp): document non-u astral matching and pin BMP agreement across V8 modes

* fix(mcp): keep prototype-named properties as own properties in the dialect projection

* test(mcp): select full output schemas in packet and signal schema checks

* test(mcp): expect the advertised base64 schema to omit the unbounded pad-bit probe

* fix(mcp): protect references into unevaluated keywords lost evaluation removes

* fix(mcp): validate the doctor probe reply against the canonical output contract (#10655)

* fix(mcp): validate the doctor probe reply against the canonical output contract

* test(mcp): replay doctor probe validation under every output profile

* test(mcp): accept typed-branch nullable unions in the advertised dialect (#10649)

* test(mcp): select full output schemas in HTTP session workflows

* fix(mcp): emit stack-safe classes with non-ASCII members and protect draft-07 reference siblings

* fix(mcp): protect references into literal siblings of relaxed draft-07 references

* feat(mcp): omit output schemas from tools/list by default (#10646)

* feat(mcp): omit output schemas from tools/list by default

No surveyed model API reads an output schema, MCP client validators
repeat REA's own result validation, and complete output schemas were
about 92% of the default tools/list. REA_MCP_OUTPUT_SCHEMA_PROFILE now
resolves to off through one owner (resolveMcpOutputSchemaProfile);
compact and full remain selectable. structuredContent is still
delivered, and the deferred server serves both off and full from the
recorded catalog with the matching recorded identity.

Tests and artifact verifiers that validate results against complete
advertised output contracts select full explicitly.

* fix(mcp): select schemas in verification workflows

* fix(mcp): select complete output schemas in the Electron verifier

* fix(mcp): select complete output schemas in standalone verifier probes

* fix(mcp): select complete output schemas in every verifier transport

* test(mcp): select full output schemas in packet and signal schema checks

* fix(mcp): validate the doctor probe reply against the canonical output contract (#10655)

* fix(mcp): validate the doctor probe reply against the canonical output contract

* test(mcp): replay doctor probe validation under every output profile

* test(mcp): accept typed-branch nullable unions in the advertised dialect (#10649)

* test(mcp): select full output schemas in HTTP session workflows

* fix(mcp): keep the Evidence union on described borrowed inputs (#10597)

* fix(mcp): keep the Evidence union on described borrowed inputs

borrowEvidenceInput installed its advertised projection on one schema
instance. A call site's .describe() rebuilds the schema from its
definition, so match_native_functions advertised left and right as
unconstrained values in every input profile. Install the projection in
the borrowed parser's constructor so derived schemas keep it, for both
borrowed inputs and borrowed arrays.

* test(mcp): select the borrowed Evidence contract from its family

* docs(mcp): admit baseline growth only for restored contract declarations

* fix(mcp): state reduced multi-type properties as typed branches, not type lists

The reduced fallback derived a property's types and emitted them as a type
list, which Gemini, Moonshot and Ark reject. A property whose canonical
schema is anyOf string and number therefore traded a typed union for a
rejected construct when the inline projection fell back. Emit one type, or
anyOf typed branches, so [T, null] becomes the dialect's nullable union.

* feat(mcp)!: retire the full schema profiles and declare complete observations as a typed annotation (#10718)

* refactor(mcp): declare complete observations as a typed annotation applied only on the wire

* test(mcp): compare complete-observation declarations with their owning text

* refactor(schema): couple Zod checks with their JSON Schema constraint keywords (#10725)

* refactor(schema): couple Zod checks with their JSON Schema constraint keywords

* refactor(schema): declare pattern approximations and object output roots as typed annotations (#10754)

* refactor(schema): declare pattern approximations and object output roots as typed annotations

* fix(schema): refuse pattern declarations JSON Schema cannot state

* test(schema): select guided_workflows from its contract family

* feat(mcp)!: retire the full input and output schema profiles (#10841)

* feat(mcp)!: retire the full input and output schema profiles

The compact input presentation is now the only advertised input schema, and
output schemas are either omitted (`off`, the default) or bounded
(`compact`). Catalog identity, the build-recorded digests and the recorded
startup catalog collapse to those presentations: two digest profiles instead
of six, one recorded catalog instead of two, and no input dialect baseline.

Tests and verifiers that validated replies against listed full output
schemas now validate against the canonical output contract, which REA
already enforces server-side; tests that asserted canonical input
restrictions read the canonical input schema instead of a full listing.

BREAKING CHANGE: `REA_MCP_INPUT_SCHEMA_PROFILE=full` and
`REA_MCP_OUTPUT_SCHEMA_PROFILE=full` are refused at startup and by `rea setup`
with a configuration error naming the setting. Remove the setting from the
client registration (or rerun setup with `REA_MCP_INPUT_SCHEMA_PROFILE=compact`
/ `REA_MCP_OUTPUT_SCHEMA_PROFILE=compact`). `REA_MCP_INPUT_SCHEMA_PROFILE`
still accepts `compact`. Server-side validation is unchanged.

* test(mcp): validate replies against canonical output contracts at every listed-schema reader

tools/list omits output schemas by default, so fixtures, verifiers and the
Android analysis runners that read a listed outputSchema now bind the
canonical contract instead. connectLocalToolsMcp and advertisedMcpToolSchemas
hand the SDK a definition carrying the canonical output schema, so the SDK
client and caller-supplied validators check every successful reply.

* test(mcp): resolve Android runners' output contracts from the selected repository

The Android efficiency runners imported the canonical output-schema helper from the harness checkout while --repository selects the revision under test. Import it from the selected repository, as the runners already do for the SDK and Ajv.

* test(mcp): let Android runners compare revisions before the canonical output helper

A selected revision without scripts/lib/canonical-output-schema.mjs could not start the Android runners. Choose the output contracts in one harness helper: the selected revision's canonical contracts when it has them, otherwise its listed schemas under the full output profile it still supports.

* fix(scripts): keep caller launch settings when the real Hopper verifier builds its server environment

The verifier deleted REA_VERIFY_SERVER_COMMAND and REA_VERIFY_SERVER_ARGS_JSON from process.env itself before reading them, so a caller selecting an installed REA command silently ran the checkout's dist/main.js. Copy the environment, as main does, so only the child's environment changes.

* test(mcp): validate whole replies against canonical output contracts wherever a listed definition is held

tools/list omits output schemas by default and the full output profile is gone, so tests that handed the SDK a listed definition no longer checked complete replies. Attach the canonical output schema with withCanonicalOutputSchema at every held REA tool definition. Environment-served tools (not REA contracts), and the deferred-runtime and doctor tests that exercise per-profile listings, keep their listed definitions.

* test(mcp): record the advertised catalog summary without the retired full profiles

Retiring the full input and output profiles leaves only the compact input and compact output listings, so the summary drops the full-profile digests and every pattern only the full listings carried. Regenerated with npm run verify:advertised-schemas -- --update-summary.
2026-11-15 21:51:18 +08:00

8.5 KiB

Measure MCP catalog costs in Codex

Use actual harness usage to evaluate catalog changes. A tools/list JSON byte count measures transport size. It does not establish how many tokens a client submits to a model, whether it defers schemas, or the total cost of an analysis. The pinned Codex observations include actual loading controls, an unavailable-host negative control and a completed real-package pilot.

Run a controlled loading comparison

Build the selected source checkout with the pinned Node and npm versions first:

npm ci
npm run build:cached
npm run eval:mcp-catalog -- --model gpt-6.1-sol \
  --output "$HOME/.cache/rea-catalog-run" --repetitions 3 --plan

Remove --plan to execute the matrix. This makes real model requests using the local Codex authentication. The output directory must not already exist. No account configuration is changed. The runner copies only authentication into an external disposable profile and removes that profile after each trial. Execution stays outside both source checkouts even when results are saved inside one, so ancestor repository instructions and skills do not pollute the trial. The npm entrypoint holds the driver's artifact lock throughout the matrix. A selected external checkout has its own lock, so the matrix also pins one runtime fingerprint and binds its catalog receipt to that manifest. Freshness checks stop further model calls if the runtime changes between trials. Each trial's before/after fingerprints must match the matrix pin; different builds never form a valid pair. Matrix admission/freshness audits sit outside individual guarded trial resource measurements. Do not rebuild the selected runtime concurrently.

This lane requires Linux, Python 3, a working user systemd manager with cgroup v2, and an installed Codex CLI supporting the recorded flags. It reuses the existing resource observer: one CPU, 4 GiB hard memory limit, 3 GiB early stop, no swap, and a 192-task ceiling. Runs are sequential. Unsupported clients, resource stops, failed startup, missing usage, and unverified cleanup remain failed measurements. They do not become zero-cost successes.

Trials disable account apps and plugins explicitly. A fresh CODEX_HOME alone does not prevent account-connected tools from appearing in the client executor catalog. For a deliberate mixed-catalog investigation, run agent-trial.py with --include-account-tools; keep those results separate from REA-only trials. The settings receipt records the same feature policy written to Codex config.

Each repetition compares no server, all REA tools, and a client-selected subset (binary_session by default), in the supported client-code host mode. Alternate repetitions reverse ordering. All variants receive the same developer instructions and exact-response prompt. A valid loading trial completes one turn, returns CATALOG_OK, and performs no tool calls. REA is marked required in every server-enabled configuration. The runner explicitly disables/enables the stable code_mode_host feature for direct/client-code modes. The old code_mode and code_mode_only switches do not override Codex 0.165's enabled host default. The requested flags and CLI version are retained; verify the actual call shape in completed-task traces too. --modes direct client-code adds a diagnostic host-disabled control. This installed Codex reports that disabling the host makes tool dispatch unavailable; it does not provide a qualified eager/direct comparison. An empty response alone cannot qualify tool availability. Errors invalidate that profile even when the model still emits the expected marker.

Useful controls:

# Choose exact tool names in the working host mode.
npm run eval:mcp-catalog -- --model gpt-6.1-sol \
  --output "$HOME/.cache/rea-catalog-selected" --modes client-code \
  --selected-tools analyze_javascript_application inspect_analysis_view

enabled_tools filters the catalog after client discovery. It does not reduce the server's original tools/list transport. Server tool-family scope is a separate existing control; do not attribute client filtering results to it. The output-schema profile remains REA's default in these experiments.

Read the evidence

report.json retains every attempted trial and compares each enabled-server trial with its same-mode, same-repetition no-server control. It reports valid pair counts and the median/range of observed input-token differences. Failed pairs are explicit and excluded from the median. A small or negative delta can reflect client framing, randomized workspace paths, and measurement variation; it does not establish a negative schema cost.

Usage, cleanup and resource receipts are collected independently of the verdict. A failed host, malformed event or wrong marker can retain observed usage and resource costs. Stage-specific issues explain invalidity; missing observations remain unknown. A changed matrix records its unattempted cells explicitly.

Each trial retains its exact prompt, runtime file digests, requested model alias, Codex version, settings/source hashes, JSONL events, actual usage, answer, cleanup receipt, and process-family resource receipt. The selected rollout capture keeps per-response usage and tool exchanges; authentication, ambient environment, account metadata, and encrypted reasoning are excluded. Tool output and local paths remain evidence and should be reviewed before sharing the raw files.

The metrics have distinct meanings:

Metric Meaning and limit
Catalog bytes and SHA-256 Built server advertisement, including output schemas; not model input.
turn.completed.usage.input_tokens Observed whole-turn usage reported by Codex; includes cached input.
Cached/uncached input Report both. Cache hits do not remove tokens from the context.
Per-response usage Retained usage records help locate discovery and continuation costs. Do not add cumulative thread counts again.
Paired input delta Incremental loading observation including instructions/client framing; not an exact schema-only token count.
Elapsed and family CPU Whole guarded trial, including local setup and client/network waits; remote inference CPU is unmeasured.
Family peak memory Kernel cgroup peak at payload exit, including observer and file cache; not final whole-scope counters.

These are fresh client/server sessions. OS caches and API prefix caches can already be warm. No cold-cache claim follows from a new session. The requested model alias is recorded; a resolved server-side model version is not inferred. The harness usage is not an independent billing reconciliation. The usage event shape follows the documented Codex non-interactive JSON output.

Evaluate completed investigations too

A no-tool response can avoid deferred discovery entirely. Before changing the catalog based on this lane, compare completed tasks through the real harness. Use npm run verify:agent and its known-answer rubric, or run a selected real artifact through tests/support/analysis-efficiency/agent-trial.py under resource-run.py --observe-payload-exit. Use the same task, artifact digest, model/reasoning, authority, result coverage, and correctness oracle in every arm. The existing runner accepts exact --enabled-tools and defaults to client-code host mode. Its direct option is a host-disabled diagnostic, not a claim of working native dispatch in the installed client. Use --capture-tool-context to locate per-response costs. Its task summary explicitly leaves factual correctness unassessed until an independent oracle or transcript review establishes the requested facts.

Count discovery, arguments, replies, retries, and continuations. Check the requested facts against source bytes or another independent oracle, real produced Evidence IDs and source links, partial/unknown coverage, and cleanup. Preserve failed attempts. A successful small-package task is a pilot, not proof of broad application performance or provider compatibility.