mirror of
https://github.com/morluto/rea.git
synced 2026-11-09 00:07:18 +00:00
docs(specs): plan a comparison of an environment with its application
Record the outcome, requirements, non-goals, plan and tasks for environment-compare, which measures how often an environment answers a live application's recorded calls as the application did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7056f5ff13
commit
d337fc5b6a
3 files changed
+73
No files matched your search
@@ -0,0 +1,12 @@
|
||||
# Plan
|
||||
|
||||
1. Export the capture parser from `environmentSource.ts` so a probe loads like a capture.
|
||||
2. `environmentCompareMatch.ts`: a probe request to a tool and its arguments.
|
||||
3. `environmentCompareIds.ts`: the id map, natural-key pairing, learning from two answers, translating arguments.
|
||||
4. `environmentCompareDiff.ts`: normalisation and categorised differences.
|
||||
5. `environmentCompare.ts`: per probe, reset the state, answer each request, compare and learn; the report.
|
||||
6. `rea environment-compare` in `src/cli/environmentCommands.ts`, documented in `docs/agent-environments.md` and
|
||||
`docs/cli.md`.
|
||||
7. Tests: matching, pairing, normalisation and categories as unit tests; the command over a fixture capture and probes.
|
||||
8. Verification on VMs: six applications brought up and seeded as for their capture, reset from a snapshot per sequence,
|
||||
driven by a script that records each sequence as a HAR; baseline reports per environment.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Compare an environment with the application it came from
|
||||
|
||||
Issue: https://github.com/morluto/rea/issues/4933
|
||||
|
||||
## Outcome
|
||||
|
||||
An environment is only as useful as it is faithful: an agent trained or graded on it should meet the behaviour of the
|
||||
application it was captured from. REA checks an environment only against its own capture, so nothing says how far it
|
||||
drifts from the application on calls the capture did not make, after writes, or on errors.
|
||||
|
||||
`rea environment-compare` measures that. It takes an environment (a manifest and its capture Evidence) and one or more
|
||||
**probe captures**, each a recording of one call sequence made against the live application, and reports how many of the
|
||||
probe's calls the environment answers as the application did, which tools disagree and how.
|
||||
|
||||
The command only reads Evidence: it makes no network call. Whoever records the probes drives the application with its own
|
||||
credentials and resets it between sequences.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Inputs: the manifest Evidence, its capture Evidence (`--evidence`), and one or more probe capture Evidence files
|
||||
(`--probe`, repeatable), each `inspect-web-network-capture` or `capture-browser-scenario` Evidence of one sequence.
|
||||
- Each probe is one sequence run against a freshly reset application. Before each probe the environment's state is reset,
|
||||
so sequences are independent on both sides.
|
||||
- Requests are taken in the probe's order (start time, then ordinal). A request is matched to the manifest tool of the
|
||||
same method whose path template it fits with the most literal segments; format-rule parameters must fit their format.
|
||||
Path values, query values the tool takes and a JSON body become the tool's arguments.
|
||||
- A request no tool matches is reported `no-tool`; a matched tool the environment does not offer, such as a write it
|
||||
withholds, is reported `not-served`. Neither is answered.
|
||||
- Ids differ between a re-seeded application and the capture. The comparison maps them:
|
||||
- seeded entities pair across the two answers by a natural key: a field present on every element of both arrays whose
|
||||
string values are unique on each side and shared between them;
|
||||
- identifier fields (`id`, `uuid`, names ending in `id` or `_id`) of paired objects, and of the two answers to the same
|
||||
call, map the live value to the environment's value; a create's two answers map the new ids;
|
||||
- path, query and body values equal to a mapped live id are translated before the environment answers.
|
||||
A path value the map does not know, and the environment's state does not hold, is reported `unmapped-id`.
|
||||
- Answers compare after normalisation: mapped ids compare equal; two clock values (ISO-8601 strings, or epoch numbers under
|
||||
time-named fields) compare by format only; two different generated values (both UUIDs, or both of one length and the
|
||||
same character classes with a digit) that neither side maps compare equal and are counted.
|
||||
- A step agrees when the statuses are equal and the normalised bodies are. Differences are categorised: `status`,
|
||||
`missing-field` (the application had it), `extra-field`, `value`, `length`, `order`, `clock-format`, `refused` (the
|
||||
environment refused the call), `not-served`, `no-tool` and `unmapped-id`, each with a JSON pointer and short values.
|
||||
- The report gives agreement overall, per probe and per tool, the counts per category, the ids mapped and the values
|
||||
normalised, and each step's differences (at most 20 per step).
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No network access, no credentials, no reset of the application: probes are recorded outside REA.
|
||||
- No generation of call sequences; a later change generates them where the environment disagrees.
|
||||
- No change to how environments serve.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Tasks
|
||||
|
||||
- [x] Record the outcome, requirements and non-goals.
|
||||
- [ ] Load probes as captures.
|
||||
- [ ] Match probe requests to tools.
|
||||
- [ ] Map ids and translate arguments.
|
||||
- [ ] Normalise and categorise differences.
|
||||
- [ ] Compare per probe and report.
|
||||
- [ ] Add the command and its documentation.
|
||||
- [ ] Add unit and command tests.
|
||||
- [ ] Run the gates.
|
||||
- [ ] Measure six environments against their live applications and record the results.
|
||||
Reference in new issue
Block a user