mirror of
https://github.com/morluto/rea.git
synced 2026-10-28 10:25:42 +00:00
377 lines
27 KiB
Markdown
377 lines
27 KiB
Markdown
# Verification
|
||
|
||
Verified base: cbb7b613 (this branch before this file). Executed on macOS with
|
||
Node v22.23.1 (the repository pins 24.18.0 in `.nvmrc`, which was not
|
||
installed); the AgentEnv runs and the live comparison ran on a disposable Linux
|
||
VM. Every check below ran as written after the last source change.
|
||
|
||
## Gates
|
||
|
||
- `npm run build:cached`, `npm run check:fast` (typecheck and lint),
|
||
`npm run format:check`, `npx knip`, `npm run scan:todos`, `npm run docs:check`
|
||
and `npm run verify:test-discovery`: passed.
|
||
- `npm run jscpd`: 3.43% duplicated lines over 3538 files, under the 3.5% gate.
|
||
- `npx vitest run --project domain --project services`: 7407 passed, 5
|
||
pre-existing skips.
|
||
- `npm run test:prepare`, then `npm run test:focused --` the new write MCP
|
||
boundary, the serve MCP, CLI and process boundaries, the export CLI
|
||
boundary, the analysis-view CLI and MCP boundaries, the compact input schema
|
||
boundary and the contract description tests: 10 files, 72 passed.
|
||
- `npm run mcp-catalog:check`: passed.
|
||
|
||
New tests: `environmentStateWriteValues` (94), `environmentStateWriteExplain`
|
||
(101), `environmentStateWriteApply` (52), `environmentStateWrites` (16, the gate
|
||
through the view), `environmentStateWriteAnswer` (14, serving) and
|
||
`environmentServeWritesMcp` (1, the built CLI over MCP). The replay card pinned
|
||
by the serve MCP boundary changed with the response summaries tool descriptions
|
||
carry; its digest now matches the combined branch.
|
||
|
||
## Real captures
|
||
|
||
The `environment-state` view ran over the 50 showcase captures (45 HAR
|
||
inspections and 5 `capture_browser_scenario` captures), each within its budget in
|
||
8–110 ms. It served 33 write tools in 16 captures (11 creates, 12 updates, 10
|
||
deletes). Of the other 150 state-changing tools, 102 name no state type or
|
||
document (most write to literal single-entity paths the manifest left
|
||
untemplated, or to collections whose reads are not state-backed), and 48 are
|
||
modelled but not served:
|
||
|
||
- 15 unconfirmed (no later read checked them);
|
||
- 13 no-observation;
|
||
- 8 contradicted, for example a deleted Airbyte source and EspoCRM task still
|
||
readable, Immich and BookStack item reads showing fields the create
|
||
response lacked, and Sonarr's tag list ordering;
|
||
- 7 unexplained-change, such as BookStack's description_html and Paperless's
|
||
derived text colour;
|
||
- 4 side-effect, for example a Sonarr tag also listed by `/tag/detail`;
|
||
- 1 incomplete-entities: WordPress answers a delete with the post in its edit
|
||
context, which the reads never showed for seeded posts.
|
||
|
||
| Capture | Served (create / update / delete) | Not served |
|
||
| ---------------- | --------------------------------- | ----------------------------------------------------------------------------------------------- |
|
||
| airbyte | 2 / 2 / 0 | create contradicted, delete contradicted, delete unconfirmed |
|
||
| baserow | none | no-state-target ×3 |
|
||
| bookstack | 0 / 0 / 1 | create contradicted, create side-effect, update unexplained-change |
|
||
| directus | none | no-state-target ×2 |
|
||
| espocrm | none | create unconfirmed, delete contradicted, update unexplained-change |
|
||
| ghost | none | no-state-target ×2 |
|
||
| gitea | 1 / 1 / 0 | no-state-target ×2 |
|
||
| grafana | none | create unconfirmed, no-state-target ×5 |
|
||
| grist | none | create no-observation, no-state-target ×5 |
|
||
| grocy | none | create no-observation, delete no-observation, update no-observation, no-state-target ×2 |
|
||
| healthchecks | none | create no-observation, delete no-observation, update no-observation, no-state-target ×2 |
|
||
| homebox | none | document no-observation, no-state-target ×3 |
|
||
| immich | 0 / 0 / 1 | create contradicted, update contradicted |
|
||
| jellyfin | none | no-state-target ×2 |
|
||
| keycloak | none | no-state-target ×1 |
|
||
| komga | none | create unconfirmed ×2, no-state-target ×4 |
|
||
| linkding | 2 / 2 / 2 | create unconfirmed, delete unconfirmed, update unconfirmed, no-state-target ×2 |
|
||
| listmonk | 1 / 1 / 1 | — |
|
||
| lychee | none | no-state-target ×4 |
|
||
| mailpit | none | document no-observation, no-state-target ×5 |
|
||
| mattermost | 1 / 1 / 1 | no-state-target ×3 |
|
||
| mealie | none | create no-observation, no-state-target ×2 |
|
||
| memos | none | no-state-target ×3 |
|
||
| metabase | 0 / 0 / 1 | create side-effect, update unexplained-change |
|
||
| miniflux | 0 / 1 / 0 | no-state-target ×3 |
|
||
| n8n | none | create unconfirmed, no-state-target ×6 |
|
||
| navidrome | none | create no-observation, no-state-target ×2 |
|
||
| netbox | 1 / 1 / 1 | — |
|
||
| nocodb | none | create no-observation, no-state-target ×4 |
|
||
| nomad | none | no-state-target ×4 |
|
||
| paperless | 0 / 0 / 1 | create unconfirmed, update unexplained-change, no-state-target ×3 |
|
||
| photoprism | 1 / 1 / 0 | delete unconfirmed |
|
||
| pinry | none | create unconfirmed, delete unconfirmed, update unexplained-change |
|
||
| pocketbase | none | no-state-target ×4 |
|
||
| redpanda-console | none | document no-observation, no-state-target ×5 |
|
||
| sentry | 1 / 0 / 0 | update unexplained-change, no-state-target ×3 |
|
||
| shlink | none | create unconfirmed, no-state-target ×1 |
|
||
| sonarr | 0 / 1 / 0 | create contradicted, create side-effect, update contradicted |
|
||
| tandoor | none | no-state-target ×4 |
|
||
| umami | 1 / 1 / 1 | — |
|
||
| vault | none | no-state-target ×7 |
|
||
| vikunja | none | no-state-target ×4 |
|
||
| wordpress | none | create side-effect, delete incomplete-entities, document unconfirmed, update unexplained-change |
|
||
|
||
An in-process smoke derived each capture's served state and called every served
|
||
write tool once with a varied body: all 33 applied (status equal to the
|
||
recording), and every created or updated entity read back from the state;
|
||
deleted entities answered the recorded 400 or 404 where one was recorded. Eight
|
||
older published manifests no longer match the tool names the current view
|
||
derives, so their state is unavailable and they serve replay, as designed;
|
||
none of them has a served write tool.
|
||
|
||
## End to end on a published environment
|
||
|
||
Five published environments serve a create, an update and a delete (Linkding,
|
||
Listmonk, Mattermost, NetBox, Umami); `random.SystemRandom().choice` picked
|
||
NetBox (NetBoxEnv-REA, NetBox 4.7.3).
|
||
|
||
- `rea environment-export` (packed from this branch) wrote the bundle; its
|
||
smoke task passed in AgentEnv framework 0.9.1334 with 42 served tools (39
|
||
reads and `post_dcim_sites`, `patch_dcim_sites_by_id`,
|
||
`delete_dcim_sites_by_id`).
|
||
- Four write tasks, each graded by an `env_outcome_verifier` that reads the
|
||
environment's state through `/agentenv` `data/get` (not the agent's reply),
|
||
with the replay-qa agent over the Hugging Face router, one run per task and
|
||
model:
|
||
|
||
| Task | gpt-oss-120b | GLM-5.3-Flash | Tool calls (gpt-oss) |
|
||
| ----------------------------------------------- | ------------ | ------------- | ------------------------------------------------------------ |
|
||
| Create a site in the Copper Valley region | 1.0 | 1.0 | get_dcim_regions, post_dcim_sites |
|
||
| Update a seeded site's facility and description | 1.0 | 1.0 | get_dcim_sites, patch_dcim_sites_by_id |
|
||
| Delete one seeded site and only it | 1.0 | 1.0 | get_dcim_sites, delete_dcim_sites_by_id |
|
||
| Create a site, then set its facility | 1.0 | 1.0 | get_dcim_regions, post_dcim_sites, patch_dcim_sites_by_id ×2 |
|
||
|
||
- The bundle's 12 retrieval tasks (one per seeded object) passed 12/12 for both
|
||
models, as they did on the published replay bundle.
|
||
|
||
## The served model against the live application
|
||
|
||
The same NetBox 4.7.3 image ran on the VM, seeded by the capture's own seed
|
||
script (the same ids), and one scenario ran against both the live API and the
|
||
environment served from the capture. Only `created` and `last_updated` were
|
||
normalised; every other member had to be equal:
|
||
|
||
| Step | Live | Environment | Equal |
|
||
| ----------------------------------------- | ---- | ----------- | ---------------------------- |
|
||
| List sites | 200 | 200 | yes |
|
||
| Create Tide Gauge Annex (id 5) | 201 | 201 | yes |
|
||
| Read it | 200 | 200 | yes |
|
||
| List after the create (sorted by name) | 200 | 200 | yes |
|
||
| Patch its description and facility | 200 | 200 | yes |
|
||
| Read after the patch | 200 | 200 | yes |
|
||
| Create Granite Spur (id 6) | 201 | 201 | yes |
|
||
| Patch a seeded site | 200 | 200 | yes |
|
||
| Delete Tide Gauge Annex | 204 | 204 | yes |
|
||
| Read the deleted site | 404 | 404 | yes |
|
||
| Create Ledger Point (id 7, not reusing 5) | 201 | 201 | yes |
|
||
| Delete a seeded site with devices | 409 | 204 | no |
|
||
| Final list | 200 | 200 | no (follows from the delete) |
|
||
|
||
11 of 13 steps were equal. The difference is NetBox's referential protection:
|
||
it refuses to delete a site that devices reference, while the environment's
|
||
delete removes the entity alone. The capture only deleted a scratch site with
|
||
no dependents, so the gate could not see the rule; it is listed under what is
|
||
not modelled.
|
||
|
||
## Not covered
|
||
|
||
- A run of the hosted `real-agentenv` lane in CI (the AgentEnv runs here used
|
||
the published framework release on a VM).
|
||
- Document writes on a real capture: the four modelled ones are unserved.
|
||
- Collections tied to a create's own collection on a real capture, and UUID
|
||
or token identities through an AgentEnv run (both are covered by tests).
|
||
- Agent token usage and cost.
|
||
|
||
## After merging the base branches and main
|
||
|
||
The branch then merged the serving branch at 5a61853a, the coverage branch at
|
||
b4b7ebc7 (their review fixes) and main at edbd291a, then main at 17f581ff
|
||
(the analysis-view tool description string, which main changes often). The gates above ran again
|
||
and passed: `npm run jscpd` 3.44% over 3579 files; domain and services 7442
|
||
passed with 5 pre-existing skips; the focused boundary set 72 passed. The
|
||
write verdicts over the 50 captures were unchanged plan by plan, and the
|
||
in-process smoke again applied all 33 served write tools. The AgentEnv runs
|
||
and the live comparison above ran on cbb7b613, before these merges.
|
||
After the second main merge: `npm run check:fast`, `npm run format:check`,
|
||
`npm run docs:check` and `npm run mcp-catalog:check` passed; the contracts,
|
||
analysis-view and environment tests 857 passed with 3 pre-existing skips; the
|
||
analysis-view and environment MCP and CLI boundaries 53 passed.
|
||
|
||
## After the review round
|
||
|
||
The branch then merged the serving branch at c00fde41, the coverage branch at
|
||
f6c756aa and main at 1629266e, and took the review fixes (82556cd5, 138ea4b1,
|
||
6d21b62e). The coverage branch quotes field names in response summaries, so the
|
||
replay card pinned by the serve MCP boundary is now 12,818 bytes; removing
|
||
those quotes gives back the previous 12,618-byte digest exactly. The gates ran
|
||
again and passed: `npm run check:fast`, `npm run format:check`, `npx knip`,
|
||
`npm run scan:todos`, `npm run docs:check`, `npm run verify:test-discovery`,
|
||
`npm run verify:module-boundaries` and `npm run mcp-catalog:check`;
|
||
`npm run jscpd` 3.47% over 3601 files; domain and services 7463 passed with 5
|
||
pre-existing skips; the focused boundary set 75 passed; conformance 279
|
||
passed. The gate is stricter, but the write verdicts over the 50 captures were
|
||
unchanged plan by plan, and the in-process smoke again applied all 33 served
|
||
write tools. The AgentEnv runs and the live comparison above ran on cbb7b613,
|
||
before these merges and fixes.
|
||
|
||
A second review round then made the gate and serving agree in five more places
|
||
(reads matched to checked lines as serving matches calls, increments and slugs
|
||
held for every call, insert rules still learned after an item read, the
|
||
document body check in the gate, and decoded parent path values). Each new test
|
||
fails on the previous head and passes here. The same gates passed again
|
||
(domain and services 7468 passed with 5 pre-existing skips, the focused
|
||
boundary set 75, conformance 279, `jscpd` 3.47%), the write verdicts over the 50
|
||
captures were again unchanged plan by plan, and the smoke applied all 33 served
|
||
write tools.
|
||
|
||
A third round kept a checked line's path spelling (replay prefers a verbatim
|
||
one, so two spellings of one parent stay separate lines) and compared a later
|
||
slug rule with the values earlier calls recorded rather than only their member
|
||
names; both new tests fail on the previous head. A fourth round chose a read's
|
||
line as replay does, so a read spelled plainly is checked against the encoded
|
||
line that answers it, and its list against that line's collection; its test
|
||
fails on the previous head.
|
||
|
||
A review of the fidelity branch built on this one found four more places: a
|
||
document write at another path is now unmodelled in the gate, a create picks
|
||
the collection the call spells verbatim, a missing entity is answered with a
|
||
recorded 2xx `null` the gate accepted as an absence, and `data/add` refuses a
|
||
document whose `data/get` form would exceed the intake limit. Each new test
|
||
fails on the previous head. The gates passed again (domain and services 7475
|
||
with 5 pre-existing skips, boundaries 75, conformance, `jscpd` 3.46%), the
|
||
verdicts over the 50 captures were unchanged plan by plan, and the smoke
|
||
applied all 33 served write tools.
|
||
|
||
A sixth round, after merging the serving and coverage branches and main
|
||
335c042d, settled five more findings: a read whose usable body no longer holds
|
||
the entity or list fails its check, an entity lacking a field its view always
|
||
carried confirms nothing, a write a later write replaced is not confirmed by
|
||
reads of the replacement, sequences never reuse a value the state started
|
||
with, and a collection keeps its spelling after it empties. Each new test fails
|
||
on the previous head. Over the 50 captures one capture's verdicts change:
|
||
Linkding's bundle create and patch, confirmed only by a list read after the
|
||
same bundle was deleted, are now unconfirmed, so 31 write tools are served (10
|
||
creates, 11 updates, 10 deletes) and the in-process smoke applies all 31. The
|
||
gates passed again (domain and services, boundaries, conformance and `jscpd`).
|
||
|
||
A seventh round settled two more findings. An update keeps its confirmation
|
||
when a later write overwrites only part of what it changed: a derived change,
|
||
such as an incremented version, counts as part of the update's effect, and a
|
||
later increment adds to the field rather than replacing it. A reset or load
|
||
forgets the collection spellings of the state it replaced. Both new tests fail
|
||
on the previous head. The verdicts over the 50 captures are unchanged plan by
|
||
plan, the in-process smoke applies all 31 served write tools, and the gates
|
||
passed again.
|
||
|
||
An eighth round settled three more. A later write erases the fields of an
|
||
earlier effect it overwrites, so two writes that together overwrite all of an
|
||
update's changes leave it nothing to be confirmed by. A recorded outcome erases
|
||
what earlier writes changed on its entity. A load keeps the Evidence's own
|
||
collection spellings, so a create in a collection the loaded state does not
|
||
hold keeps its recorded spelling. Each new test fails on the previous head. The
|
||
verdicts over the 50 captures are unchanged plan by plan, the smoke applies all
|
||
31 served write tools, and the gates passed again.
|
||
|
||
A ninth round made a create prefer a collection the state holds over an equally
|
||
fitting spelling known only from the Evidence, so a create after a load lands in
|
||
the loaded collection and reads back. Its test fails on the previous head; the
|
||
verdicts and the smoke are unchanged and the gates passed again.
|
||
|
||
A tenth round settled six findings Greptile raised on the list-lines branch
|
||
against this branch's code:
|
||
|
||
- a read with no query on a whole list is checked on the paging line serving
|
||
answers it from, and a line whose query repeats a name is never checked;
|
||
- a read made while a write to its targets was still running checks nothing;
|
||
- a later call that stores a member otherwise than the plan refuses the plan;
|
||
- a create keeps the spelling of a collection the capture listed empty;
|
||
- a wide array no longer overflows the stack while checking path-dependent
|
||
values;
|
||
- the guide counts served write tools among the catalog's.
|
||
|
||
Each new test fails on the previous head. The verdicts over the 50 captures
|
||
are unchanged plan by plan, the smoke applies all 31 served write tools, and
|
||
the gates passed again.
|
||
|
||
An eleventh round settled six more findings, five raised on this branch and one on the fidelity branch:
|
||
|
||
- a passing read confirms only the writes it shows something of;
|
||
- an item read names its entity as serving does, percent-decoded spellings included;
|
||
- a read a write's effect was learned from still contradicts the other writes;
|
||
- a line repeating a query name with one value stays checkable;
|
||
- document writes join the effect tracking, so a replacement overwrites an earlier merge;
|
||
- a call naming a page that the collection has outgrown answers from its recording (`page-exceeded`), and the gate does not check such reads.
|
||
|
||
Each new test fails on the previous head. The verdicts over the 50 captures are unchanged plan by plan, the smoke
|
||
applies all 31 served write tools, and the gates passed again.
|
||
|
||
A twelfth round settled four findings Greptile raised on the list-lines branch against this branch's code:
|
||
|
||
- a create that answers with its id alone is observed only through a read of that id;
|
||
- a later call that is the first to send the path's id in its body adds the identity member to the plan;
|
||
- the copy search walks a request body with an index, so a wide body no longer takes quadratic time (500,000 zeros: 63 s
|
||
before, 74 ms after);
|
||
- the sent values of fixed members a plan keeps are counted against the view's allowance.
|
||
|
||
Each new test fails on the previous head. The verdicts over the 50 captures are unchanged plan by plan, the smoke
|
||
applies all 31 served write tools, and the gates passed again.
|
||
|
||
A thirteenth round settled seven findings Greptile raised on this branch, the fidelity branch and the list-lines branch:
|
||
|
||
- a passing read that shows nothing of a write leaves it for a later check, instead of clearing it;
|
||
- a read begun while a write to an unknown entity still ran neither checks nor shows the entity again;
|
||
- a later create whose sequence fields do not follow the learned sequence refuses the plan;
|
||
- a document replacement below the root overwrites only what it replaced;
|
||
- a built document plan writes later calls at its own object, even one sharing no member with the document;
|
||
- an update or delete finds its entity as an item read does, percent-decoded spellings included;
|
||
- a `_start`–`_end` range bounds a page as `limit` does, in serving and in the gate.
|
||
|
||
Each new test fails on the previous head. The verdicts over the 50 captures are unchanged plan by plan, the smoke
|
||
applies all 31 served write tools, and the gates passed again.
|
||
|
||
A fourteenth round settled two findings Greptile raised on the list-queries branch against this branch's code:
|
||
|
||
- the type's identity field never becomes a constant: an update whose id member the application ignored no longer
|
||
writes the exemplar's id into the entity another call names;
|
||
- a page size of zero bounds nothing, so a whole line recorded with `limit=0` answers from the state.
|
||
|
||
Both tests fail on the previous head. The verdicts over the 50 captures are unchanged plan by plan, the smoke applies
|
||
all 31 served write tools, and the gates passed again.
|
||
|
||
A fifteenth round settled nine findings Greptile raised on this branch, the fidelity branch and the list-lines branch,
|
||
five of them on the previous rounds' own changes:
|
||
|
||
- an entity field whose name looks like a pointer is matched by name; only a document's pointers cover the members
|
||
under them;
|
||
- a failed read contradicts each exercised write no unmodelled write came after, rather than skipping all of them
|
||
when the oldest pending write came before one;
|
||
- an item read, update or delete names its entity as serving does, decoding the id once, so `book%2531` no longer
|
||
reaches `book1`;
|
||
- a plan whose calls use a member both as the identity and as a fixed member, or fix it at different values, is
|
||
refused;
|
||
- a size of zero bounds nothing only on a line whose recording listed items under it;
|
||
- an update whose observation drops a field the entity held is refused;
|
||
- a later create whose clock takes another format, or whose unique value is not a UUID, is refused;
|
||
- a derived state `data/get` would write past 32 MiB is not served.
|
||
|
||
Each new test fails on the previous head. The verdicts over the 50 captures are unchanged plan by plan, the smoke
|
||
applies all 31 served write tools, and the gates passed again.
|
||
|
||
A sixteenth round settled four findings Greptile raised on the fifteenth round's changes, here and on the fidelity,
|
||
list-lines and list-queries branches:
|
||
|
||
- a size of zero is no bound only by itself: a range or another size on the same line still bounds the page;
|
||
- an update is refused for a dropped field only when a later read shows it dropped, since a write's own response may
|
||
leave fields out;
|
||
- interference is weighed against a tool's latest call on the read's entity or collection, not its latest call
|
||
anywhere.
|
||
|
||
Each new test fails on the previous head. The verdicts over the 50 captures are unchanged plan by plan, the smoke
|
||
applies all 31 served write tools, and the gates passed again.
|
||
|
||
A seventeenth round settled two more findings Greptile raised on the list-queries branch against this branch's
|
||
interference rule: an unmodelled write counts against a tool when it followed the tool's call on any target the read
|
||
covers, so a later call on another entity does not hide it; and a failed item read contradicts the writes no
|
||
unmodelled write followed, checking only the others on their members. Both tests fail on the previous head. The
|
||
verdicts over the 50 captures are unchanged plan by plan, the smoke applies all 31 served write tools, and the gates
|
||
passed again. A follow-up weighs interference only on targets where the tool's effect survives, so a call
|
||
whose effect a later write overwrote no longer excuses a later mismatch; its test fails on the previous head.
|
||
|
||
## Collection-history accounting check
|
||
|
||
On 2026-10-22, Node 24.18.0, I compared the source before the write-history
|
||
change (`cfb42ddde`) with the bounded-history implementation. Each operation
|
||
inserted an entity at a new collection path with a 128-character repeated
|
||
suffix and immediately deleted it. Across five fresh Vitest worker processes,
|
||
the median for 1,000 completed cycles was 14.57 ms wall / 23,867 us CPU before
|
||
and 9.98 ms / 16,914 us after. The process RSS samples were 381.1–387.9 MiB
|
||
before and 380.9–387.6 MiB after. This fixture benchmark measures one matched
|
||
completed trace, not a representative user task or process-family peak memory.
|
||
The values include Vitest and V8 and are noisy; they do not establish a memory
|
||
bound. The 1 MiB policy is a deterministic accounting allowance for
|
||
additional identifier history, set to 1/32 of the separate 32 MiB state-intake
|
||
ceiling; it is not derived from RSS samples and does not bound total store or
|
||
process memory.
|