diff --git a/docs/specs/agent-environments/state-writes/tasks.md b/docs/specs/agent-environments/state-writes/tasks.md index 6b913a8edb..c3d238ab67 100644 --- a/docs/specs/agent-environments/state-writes/tasks.md +++ b/docs/specs/agent-environments/state-writes/tasks.md @@ -13,10 +13,10 @@ - [x] Let lines that only page a whole collection back `membership: "all"`, and answer argument-less list calls from them. - [x] Count served write tools in exports and describe them in bundle READMEs. -- [ ] Add domain, services, adapters and boundary coverage. +- [x] Add domain, services, adapters and boundary coverage. - [x] Document the write model and serving. - [x] Run the view over every showcase capture and report per-capture verdicts. -- [ ] Export one randomly chosen environment, run write tasks in the AgentEnv +- [x] Export one randomly chosen environment, run write tasks in the AgentEnv framework and grade them from `data/get`. - [x] Compare the served model with the live application on the same calls. -- [ ] Run static, formatting, duplication, documentation and test gates. +- [x] Run static, formatting, duplication, documentation and test gates. diff --git a/docs/specs/agent-environments/state-writes/verification.md b/docs/specs/agent-environments/state-writes/verification.md index be39503d7a..c52777670b 100644 --- a/docs/specs/agent-environments/state-writes/verification.md +++ b/docs/specs/agent-environments/state-writes/verification.md @@ -1,4 +1,165 @@ # Verification -Pending: this file records the gates, the real-capture results, the AgentEnv -run and the live comparison once the implementation lands. +Verified base: cbb7b613 (this branch before this file). Executed on macOS with +Node v22.23.1 (the repository pins 24.18.0 in `.nvmrc`, which was not +installed); the AgentEnv runs and the live comparison ran on a disposable Linux +VM. Every check below ran as written after the last source change. + +## Gates + +- `npm run build:cached`, `npm run check:fast` (typecheck and lint), + `npm run format:check`, `npx knip`, `npm run scan:todos`, `npm run docs:check` + and `npm run verify:test-discovery`: passed. +- `npm run jscpd`: 3.43% duplicated lines over 3538 files, under the 3.5% gate. +- `npx vitest run --project domain --project services`: 7407 passed, 5 + pre-existing skips. +- `npm run test:prepare`, then `npm run test:focused --` the new write MCP + boundary, the serve MCP, CLI and process boundaries, the export CLI + boundary, the analysis-view CLI and MCP boundaries, the compact input schema + boundary and the contract description tests: 10 files, 72 passed. +- `npm run mcp-catalog:check`: passed. + +New tests: `environmentStateWriteValues` (94), `environmentStateWriteExplain` +(101), `environmentStateWriteApply` (52), `environmentStateWrites` (16, the gate +through the view), `environmentStateWriteAnswer` (14, serving) and +`environmentServeWritesMcp` (1, the built CLI over MCP). The replay card pinned +by the serve MCP boundary changed with the response summaries tool descriptions +carry; its digest now matches the combined branch. + +## Real captures + +The `environment-state` view ran over the 51 showcase captures (44 HAR, 5 +`capture_browser_scenario`, the rest public APIs), each within its budget in +8–110 ms. It served 33 write tools in 16 captures (11 creates, 12 updates, 10 +deletes). Of the other 150 state-changing tools, 102 name no state type or +document (most write to literal single-entity paths the manifest left +untemplated, or to collections whose reads are not state-backed), and 48 are +modelled but not served: + +- 15 unconfirmed (no later read checked them); +- 13 no-observation; +- 8 contradicted, for example a deleted Airbyte source and EspoCRM task still + readable, Immich and BookStack item reads showing fields the create + response lacked, and Sonarr's tag list ordering; +- 7 unexplained-change, such as BookStack's description_html and Paperless's + derived text colour; +- 4 side-effect, for example a Sonarr tag also listed by `/tag/detail`; +- 1 incomplete-entities: WordPress answers a delete with the post in its edit + context, which the reads never showed for seeded posts. + +| Capture | Served (create / update / delete) | Not served | +| ---------------- | --------------------------------- | ----------------------------------------------------------------------------------------------- | +| airbyte | 2 / 2 / 0 | create contradicted, delete contradicted, delete unconfirmed | +| baserow | none | no-state-target ×3 | +| bookstack | 0 / 0 / 1 | create contradicted, create side-effect, update unexplained-change | +| directus | none | no-state-target ×2 | +| espocrm | none | create unconfirmed, delete contradicted, update unexplained-change | +| ghost | none | no-state-target ×2 | +| gitea | 1 / 1 / 0 | no-state-target ×2 | +| grafana | none | create unconfirmed, no-state-target ×5 | +| grist | none | create no-observation, no-state-target ×5 | +| grocy | none | create no-observation, delete no-observation, update no-observation, no-state-target ×2 | +| healthchecks | none | create no-observation, delete no-observation, update no-observation, no-state-target ×2 | +| homebox | none | document no-observation, no-state-target ×3 | +| immich | 0 / 0 / 1 | create contradicted, update contradicted | +| jellyfin | none | no-state-target ×2 | +| keycloak | none | no-state-target ×1 | +| komga | none | create unconfirmed ×2, no-state-target ×4 | +| linkding | 2 / 2 / 2 | create unconfirmed, delete unconfirmed, update unconfirmed, no-state-target ×2 | +| listmonk | 1 / 1 / 1 | — | +| lychee | none | no-state-target ×4 | +| mailpit | none | document no-observation, no-state-target ×5 | +| mattermost | 1 / 1 / 1 | no-state-target ×3 | +| mealie | none | create no-observation, no-state-target ×2 | +| memos | none | no-state-target ×3 | +| metabase | 0 / 0 / 1 | create side-effect, update unexplained-change | +| miniflux | 0 / 1 / 0 | no-state-target ×3 | +| n8n | none | create unconfirmed, no-state-target ×6 | +| navidrome | none | create no-observation, no-state-target ×2 | +| netbox | 1 / 1 / 1 | — | +| nocodb | none | create no-observation, no-state-target ×4 | +| nomad | none | no-state-target ×4 | +| paperless | 0 / 0 / 1 | create unconfirmed, update unexplained-change, no-state-target ×3 | +| photoprism | 1 / 1 / 0 | delete unconfirmed | +| pinry | none | create unconfirmed, delete unconfirmed, update unexplained-change | +| pocketbase | none | no-state-target ×4 | +| redpanda-console | none | document no-observation, no-state-target ×5 | +| sentry | 1 / 0 / 0 | update unexplained-change, no-state-target ×3 | +| shlink | none | create unconfirmed, no-state-target ×1 | +| sonarr | 0 / 1 / 0 | create contradicted, create side-effect, update contradicted | +| tandoor | none | no-state-target ×4 | +| umami | 1 / 1 / 1 | — | +| vault | none | no-state-target ×7 | +| vikunja | none | no-state-target ×4 | +| wordpress | none | create side-effect, delete incomplete-entities, document unconfirmed, update unexplained-change | + +An in-process smoke derived each capture's served state and called every served +write tool once with a varied body: all 33 applied (status equal to the +recording), and every created or updated entity read back from the state; +deleted entities answered the recorded 400 or 404 where one was recorded. Eight +older published manifests no longer match the tool names the current view +derives, so their state is unavailable and they serve replay, as designed; +none of them has a served write tool. + +## End to end on a published environment + +Five published environments serve a create, an update and a delete (Linkding, +Listmonk, Mattermost, NetBox, Umami); `random.SystemRandom().choice` picked +NetBox (NetBoxEnv-REA, NetBox 4.7.3). + +- `rea environment-export` (packed from this branch) wrote the bundle; its + smoke task passed in AgentEnv framework 0.9.1334 with 42 served tools (39 + reads and `post_dcim_sites`, `patch_dcim_sites_by_id`, + `delete_dcim_sites_by_id`). +- Four write tasks, each graded by an `env_outcome_verifier` that reads the + environment's state through `/agentenv` `data/get` (not the agent's reply), + with the replay-qa agent over the Hugging Face router, one run per task and + model: + +| Task | gpt-oss-120b | GLM-5.3-Flash | Tool calls (gpt-oss) | +| ----------------------------------------------- | ------------ | ------------- | ------------------------------------------------------------ | +| Create a site in the Copper Valley region | 1.0 | 1.0 | get_dcim_regions, post_dcim_sites | +| Update a seeded site's facility and description | 1.0 | 1.0 | get_dcim_sites, patch_dcim_sites_by_id | +| Delete one seeded site and only it | 1.0 | 1.0 | get_dcim_sites, delete_dcim_sites_by_id | +| Create a site, then set its facility | 1.0 | 1.0 | get_dcim_regions, post_dcim_sites, patch_dcim_sites_by_id ×2 | + +- The bundle's 12 retrieval tasks (one per seeded object) passed 12/12 for both + models, as they did on the published replay bundle. + +## The served model against the live application + +The same NetBox 4.7.3 image ran on the VM, seeded by the capture's own seed +script (the same ids), and one scenario ran against both the live API and the +environment served from the capture. Only `created` and `last_updated` were +normalised; every other member had to be equal: + +| Step | Live | Environment | Equal | +| ----------------------------------------- | ---- | ----------- | ---------------------------- | +| List sites | 200 | 200 | yes | +| Create Tide Gauge Annex (id 5) | 201 | 201 | yes | +| Read it | 200 | 200 | yes | +| List after the create (sorted by name) | 200 | 200 | yes | +| Patch its description and facility | 200 | 200 | yes | +| Read after the patch | 200 | 200 | yes | +| Create Granite Spur (id 6) | 201 | 201 | yes | +| Patch a seeded site | 200 | 200 | yes | +| Delete Tide Gauge Annex | 204 | 204 | yes | +| Read the deleted site | 404 | 404 | yes | +| Create Ledger Point (id 7, not reusing 5) | 201 | 201 | yes | +| Delete a seeded site with devices | 409 | 204 | no | +| Final list | 200 | 200 | no (follows from the delete) | + +11 of 13 steps were equal. The difference is NetBox's referential protection: +it refuses to delete a site that devices reference, while the environment's +delete removes the entity alone. The capture only deleted a scratch site with +no dependents, so the gate could not see the rule; it is listed under what is +not modelled. + +## Not covered + +- A run of the hosted `real-agentenv` lane in CI (the AgentEnv runs here used + the published framework release on a VM). +- Document writes on a real capture: the four modelled ones are unserved. +- Collections tied to a create's own collection on a real capture, and UUID + or token identities through an AgentEnv run (both are covered by tests). +- Agent token usage and cost.