docs(specs): plan a comparison of an environment with its application

Record the outcome, requirements, non-goals, plan and tasks for
environment-compare, which measures how often an environment answers a
live application's recorded calls as the application did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Edgar ArakelyanandClaude Opus 5.5 committed 2026-10-20 14:48:50 -07:00
1 parent 7056f5ff13
commit d337fc5b6a
3 files changed
+73

No files matched your search

@@ -0,0 +1,12 @@
# Plan
1. Export the capture parser from `environmentSource.ts` so a probe loads like a capture.
2. `environmentCompareMatch.ts`: a probe request to a tool and its arguments.
3. `environmentCompareIds.ts`: the id map, natural-key pairing, learning from two answers, translating arguments.
4. `environmentCompareDiff.ts`: normalisation and categorised differences.
5. `environmentCompare.ts`: per probe, reset the state, answer each request, compare and learn; the report.
6. `rea environment-compare` in `src/cli/environmentCommands.ts`, documented in `docs/agent-environments.md` and
`docs/cli.md`.
7. Tests: matching, pairing, normalisation and categories as unit tests; the command over a fixture capture and probes.
8. Verification on VMs: six applications brought up and seeded as for their capture, reset from a snapshot per sequence,
driven by a script that records each sequence as a HAR; baseline reports per environment.
@@ -0,0 +1,49 @@
# Compare an environment with the application it came from
Issue: https://github.com/morluto/rea/issues/4933
## Outcome
An environment is only as useful as it is faithful: an agent trained or graded on it should meet the behaviour of the
application it was captured from. REA checks an environment only against its own capture, so nothing says how far it
drifts from the application on calls the capture did not make, after writes, or on errors.
`rea environment-compare` measures that. It takes an environment (a manifest and its capture Evidence) and one or more
**probe captures**, each a recording of one call sequence made against the live application, and reports how many of the
probe's calls the environment answers as the application did, which tools disagree and how.
The command only reads Evidence: it makes no network call. Whoever records the probes drives the application with its own
credentials and resets it between sequences.
## Requirements
- Inputs: the manifest Evidence, its capture Evidence (`--evidence`), and one or more probe capture Evidence files
(`--probe`, repeatable), each `inspect-web-network-capture` or `capture-browser-scenario` Evidence of one sequence.
- Each probe is one sequence run against a freshly reset application. Before each probe the environment's state is reset,
so sequences are independent on both sides.
- Requests are taken in the probe's order (start time, then ordinal). A request is matched to the manifest tool of the
same method whose path template it fits with the most literal segments; format-rule parameters must fit their format.
Path values, query values the tool takes and a JSON body become the tool's arguments.
- A request no tool matches is reported `no-tool`; a matched tool the environment does not offer, such as a write it
withholds, is reported `not-served`. Neither is answered.
- Ids differ between a re-seeded application and the capture. The comparison maps them:
- seeded entities pair across the two answers by a natural key: a field present on every element of both arrays whose
string values are unique on each side and shared between them;
- identifier fields (`id`, `uuid`, names ending in `id` or `_id`) of paired objects, and of the two answers to the same
call, map the live value to the environment's value; a create's two answers map the new ids;
- path, query and body values equal to a mapped live id are translated before the environment answers.
A path value the map does not know, and the environment's state does not hold, is reported `unmapped-id`.
- Answers compare after normalisation: mapped ids compare equal; two clock values (ISO-8601 strings, or epoch numbers under
time-named fields) compare by format only; two different generated values (both UUIDs, or both of one length and the
same character classes with a digit) that neither side maps compare equal and are counted.
- A step agrees when the statuses are equal and the normalised bodies are. Differences are categorised: `status`,
`missing-field` (the application had it), `extra-field`, `value`, `length`, `order`, `clock-format`, `refused` (the
environment refused the call), `not-served`, `no-tool` and `unmapped-id`, each with a JSON pointer and short values.
- The report gives agreement overall, per probe and per tool, the counts per category, the ids mapped and the values
normalised, and each step's differences (at most 20 per step).
## Non-goals
- No network access, no credentials, no reset of the application: probes are recorded outside REA.
- No generation of call sequences; a later change generates them where the environment disagrees.
- No change to how environments serve.
@@ -0,0 +1,12 @@
# Tasks
- [x] Record the outcome, requirements and non-goals.
- [ ] Load probes as captures.
- [ ] Match probe requests to tools.
- [ ] Map ids and translate arguments.
- [ ] Normalise and categorise differences.
- [ ] Compare per probe and report.
- [ ] Add the command and its documentation.
- [ ] Add unit and command tests.
- [ ] Run the gates.
- [ ] Measure six environments against their live applications and record the results.