diff --git a/docs/specs/agent-environments/fidelity-compare/plan.md b/docs/specs/agent-environments/fidelity-compare/plan.md new file mode 100644 index 0000000000..7f3e1b5548 --- /dev/null +++ b/docs/specs/agent-environments/fidelity-compare/plan.md @@ -0,0 +1,12 @@ +# Plan + +1. Export the capture parser from `environmentSource.ts` so a probe loads like a capture. +2. `environmentCompareMatch.ts`: a probe request to a tool and its arguments. +3. `environmentCompareIds.ts`: the id map, natural-key pairing, learning from two answers, translating arguments. +4. `environmentCompareDiff.ts`: normalisation and categorised differences. +5. `environmentCompare.ts`: per probe, reset the state, answer each request, compare and learn; the report. +6. `rea environment-compare` in `src/cli/environmentCommands.ts`, documented in `docs/agent-environments.md` and + `docs/cli.md`. +7. Tests: matching, pairing, normalisation and categories as unit tests; the command over a fixture capture and probes. +8. Verification on VMs: six applications brought up and seeded as for their capture, reset from a snapshot per sequence, + driven by a script that records each sequence as a HAR; baseline reports per environment. diff --git a/docs/specs/agent-environments/fidelity-compare/spec.md b/docs/specs/agent-environments/fidelity-compare/spec.md new file mode 100644 index 0000000000..ef99470ef0 --- /dev/null +++ b/docs/specs/agent-environments/fidelity-compare/spec.md @@ -0,0 +1,49 @@ +# Compare an environment with the application it came from + +Issue: https://github.com/morluto/rea/issues/4933 + +## Outcome + +An environment is only as useful as it is faithful: an agent trained or graded on it should meet the behaviour of the +application it was captured from. REA checks an environment only against its own capture, so nothing says how far it +drifts from the application on calls the capture did not make, after writes, or on errors. + +`rea environment-compare` measures that. It takes an environment (a manifest and its capture Evidence) and one or more +**probe captures**, each a recording of one call sequence made against the live application, and reports how many of the +probe's calls the environment answers as the application did, which tools disagree and how. + +The command only reads Evidence: it makes no network call. Whoever records the probes drives the application with its own +credentials and resets it between sequences. + +## Requirements + +- Inputs: the manifest Evidence, its capture Evidence (`--evidence`), and one or more probe capture Evidence files + (`--probe`, repeatable), each `inspect-web-network-capture` or `capture-browser-scenario` Evidence of one sequence. +- Each probe is one sequence run against a freshly reset application. Before each probe the environment's state is reset, + so sequences are independent on both sides. +- Requests are taken in the probe's order (start time, then ordinal). A request is matched to the manifest tool of the + same method whose path template it fits with the most literal segments; format-rule parameters must fit their format. + Path values, query values the tool takes and a JSON body become the tool's arguments. +- A request no tool matches is reported `no-tool`; a matched tool the environment does not offer, such as a write it + withholds, is reported `not-served`. Neither is answered. +- Ids differ between a re-seeded application and the capture. The comparison maps them: + - seeded entities pair across the two answers by a natural key: a field present on every element of both arrays whose + string values are unique on each side and shared between them; + - identifier fields (`id`, `uuid`, names ending in `id` or `_id`) of paired objects, and of the two answers to the same + call, map the live value to the environment's value; a create's two answers map the new ids; + - path, query and body values equal to a mapped live id are translated before the environment answers. + A path value the map does not know, and the environment's state does not hold, is reported `unmapped-id`. +- Answers compare after normalisation: mapped ids compare equal; two clock values (ISO-8601 strings, or epoch numbers under + time-named fields) compare by format only; two different generated values (both UUIDs, or both of one length and the + same character classes with a digit) that neither side maps compare equal and are counted. +- A step agrees when the statuses are equal and the normalised bodies are. Differences are categorised: `status`, + `missing-field` (the application had it), `extra-field`, `value`, `length`, `order`, `clock-format`, `refused` (the + environment refused the call), `not-served`, `no-tool` and `unmapped-id`, each with a JSON pointer and short values. +- The report gives agreement overall, per probe and per tool, the counts per category, the ids mapped and the values + normalised, and each step's differences (at most 20 per step). + +## Non-goals + +- No network access, no credentials, no reset of the application: probes are recorded outside REA. +- No generation of call sequences; a later change generates them where the environment disagrees. +- No change to how environments serve. diff --git a/docs/specs/agent-environments/fidelity-compare/tasks.md b/docs/specs/agent-environments/fidelity-compare/tasks.md new file mode 100644 index 0000000000..b3ad14b0d0 --- /dev/null +++ b/docs/specs/agent-environments/fidelity-compare/tasks.md @@ -0,0 +1,12 @@ +# Tasks + +- [x] Record the outcome, requirements and non-goals. +- [ ] Load probes as captures. +- [ ] Match probe requests to tools. +- [ ] Map ids and translate arguments. +- [ ] Normalise and categorise differences. +- [ ] Compare per probe and report. +- [ ] Add the command and its documentation. +- [ ] Add unit and command tests. +- [ ] Run the gates. +- [ ] Measure six environments against their live applications and record the results.