Files
reverse-engineering-anything/docs/specs/agent-environments/environment-build/verification.md
T
41cf869949 fix(environment): bound create learning and invalidate stale write proofs (#6929)
* feat(environment): report what an environment serves with an environment-coverage view

A new inspect_analysis_view kind over capture Evidence. It projects the
state an environment derives with the given template rules and reports
each tool as answered from state, replayed with its view reason, a served
write, or a withheld write with its reason, the records behind it and a
hint about what to record next. The hints cover every write reason, so a
new reason without one fails to compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(environment): build an environment bundle in one command

rea environment-build <capture> [directory] inspects a HAR (or reads
capture Evidence), projects the environment manifest from the in-memory
Evidence, authenticates the pair, prepares the state and writes the bundle
environment-export writes. Every bundle file is byte-identical to the
three-step path for the same capture and options.

The report adds the capture and manifest identities, the coverage summary,
warnings (a loopback capture named after its file, credential-named body
members by pointer) and the next command. --dry-run writes nothing. The
default template rules are the four the published environments use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(environment): cover environment-build and the coverage view

Domain tests check the hint table against the state view's reasons, the
summary, and the view against the environment-state view it summarizes.
Service tests build the fixture capture and compare every file with the
three-step path, a dry run, loopback naming and the refusals. A CLI test
compares a HAR built in one step with the three CLI steps byte for byte,
and an MCP test projects the view from every source kind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(environment): build the real-agentenv lane's bundle with environment-build

The lane now runs the command users run: it builds the Grafana-shaped HAR
with environment-build and the packed checkout, checks the report, then
runs the bundle with agent-env as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): document environment-build and the coverage view

The agent environments guide opens with the one-step build: the report,
the hint for each withheld write reason, and what a bundle ships. The CLI
guide, the capture guide and the skill reference name the command and the
environment-coverage view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): harden environment-build after review

- The credential scan walks with its own stack, so a deeply nested body
  cannot crash the build, and it also reports credential-named header
  names, query names, form body members and string arrays.
- The build checks SIGINT between stages: cancelled before the bundle
  stage it writes nothing, during the write it names the written bundle.
- A dry run opens the package tarball, so it refuses what a build refuses.
- Repeated template rules are refused, a HAR saved as .json gets a hint,
  file-derived names follow the bundle name rules, `localhost.` counts as
  a loopback origin, and the next command is shell-quoted.
- --capture-timeout-ms passes through to the HAR inspection, and a
  missing directory is refused before inspection.
- The no-observation hint covers updates and deletes of entities the state
  never held; the coverage view states that it does not apply the
  serve-time admission checks; the inspect_analysis_view description
  names the view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): state the build's conditions and record its verification

The guides now state when the one-step bundle equals the three-step one,
which headers inspection excludes, what the credential warning scans, how
cancellation behaves, and that the coverage view does not apply the
serve-time admission checks. The spec folder records the census, the end
to end runs, the gates and the review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): dispatch a pending interrupt before each build stage check

The manifest, state and coverage stages run synchronously, so a SIGINT
that arrived during them was only dispatched once the bundle write had
begun. Each check now yields first, so an interrupt before the bundle
stage cancels the build with nothing written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): scan scenario credentials, own the dry-run tarball and serialize Evidence once

A browser scenario's retained header values and the query parameters of its recorded URLs are scanned for credential
names as a HAR's are, through a review projection of its records. A dry run admits and releases the package tarball
through the export's regular-file scope, so a close it cannot confirm keeps its owner, and checks for cancellation
after that last await. Evidence is written into one exactly sized buffer instead of chunks joined by a copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style(environment): format the build spec

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): scan every form occurrence and MIME-less scenario bodies

The build's credential scan collapsed form bodies with Object.fromEntries, so a
repeated field kept only its last occurrence: `api_key=secret&api_key=` warned
about nothing although the Evidence ships the value, and repeated values were
undercounted. Form fields are now scanned as ordered name/value pairs, as
headers and query strings already are.

The review projection of a capture_browser_scenario dropped a request body
recorded without a media type, so the scan never saw it. Review keeps such a
body's text with an empty media type; the manifest projection still reads only
bodies it can classify.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): scan structured HAR form params and the request side of unattributed scenario transactions

A HAR request recorded with postData.params and no text is scanned by those parameters, each occurrence counting, and
the review projection keeps the request side of a scenario transaction whose response cannot be attributed, so the
credential warning covers what evidence.json ships in both cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): validate build paths and read dry-run tarballs

* fix(environment): report bodies too large for the credential scan

Text bodies over the scan's 4 MiB limit, including a response decoded only as a prefix, are reported as a
credential-scan-incomplete warning by record and side, never by content, so an absent credential-named-members
warning no longer reads as a complete check. The spec and agent-environments guide also state the path validation and
dry-run tarball read that e52ae1cd added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(verification): release cold source transform services

* fix(javascript): retain capture context along covered mutation paths

* fix(environment): admit serialized exports and scan HAR form fallbacks

* fix(environment): admit in-memory build inputs against heap headroom and scan SaveHar null-text forms

The inspected HAR's Evidence and the projected manifest are checked against live heap headroom before they are
serialized, with the same check the file route uses, so a capture too large to parse and seal is refused as
resource_constraint before anything is written. A request whose postData text is SaveHar's null is scanned by its
structured params.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): measure the credential scan limit in UTF-8 bytes

A body counts against the 4 MiB limit by its UTF-8 length, not its UTF-16 code units, so multibyte text over the
limit is reported as unscanned instead of parsed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(javascript): erase inline type imports before binding values

* fix(environment): parse credential form bodies by their media type

* fix(environment): review untyped retained form fields

* fix(environment): bound request comparisons and preserve write proofs

* fix(environment): keep write proofs and bundle review complete

* fix(environment): finalize entity overrides with simulated views

* test(environment): restore 0895c89b's untyped form tests and hint

Integrating main took main's credential scan and coverage view, which
dropped 0895c89b's tests and its body-unusable hint. Main's scan passes
the tests, now reading the scenario by event. The hint names the
response too, since extraction and serving also withhold a write as
body-unusable when its recorded response cannot be read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(environment): scan a scenario content event recorded twice

Main's event scan reads both content events of a transaction whose
content the capture recorded twice; the test fails on 0895c89b, whose
review projection dropped that transaction's response. The spec names
event pointers among the first 20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): retain contradictions and trim retired state

* fix(environment): distinguish mixed array order from changed values

* fix(environment): keep dependent create state pinned

* fix(environment): keep checking retired view observations

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
2026-10-24 23:57:30 +08:00

114 lines
9.1 KiB
Markdown

# Verification
Executed on macOS with Node v24.22.1 (the checkout's `.nvmrc`) on main `3d4f5565`, and on Ubuntu 24.04 VMs for the
end-to-end runs.
## Gates
- `npm run -s check:fast`, `npm run -s format:check`, `npx knip`, `npm run -s scan:todos`, `npm run -s docs:check`,
`npm run -s verify:test-discovery` and `npm run -s mcp-catalog:check`: passed.
- `npm run jscpd`: 3.52% duplicated lines over 4348 files, as on main (3.52% over 4341).
- `npx vitest run --project domain --project services`: 9537 passed, 6 skipped and 151 failed. The same 151 fail on main
`3d4f5565` in this environment (9513 passed there); none is new.
- `npm run -s test:prepare`, then `npm run -s test:focused --` the environment build, export, export-inputs and serve CLI
boundaries, the analysis-view CLI and MCP boundaries and the environment serve MCP boundary: 7 files, 67 passed and 1
failed, "projects portable artifact inventory after the original ZIP is removed", which fails the same way on main.
- `npm run verify:agentenv:environment` (the `real-agentenv` lane) passed on Ubuntu 24.04 with `agent-env 0.9.1334`.
## Equivalence and coverage over the published captures
Each of the 50 published capture-built environments' captures (capture Evidence JSON) and four scoped HAR recordings
(EspoCRM, Ghost, NocoDB, Sonarr) was built twice: once with `rea environment-build <capture> <dir> --json`, and once
with the three CLI steps, `inspect-web-network-capture --format json` (HAR only), `inspect-analysis-view` with an
`evidence-file` source and the four default rules, and `environment-export --name <the build's name>`. Every file of
every bundle was compared by SHA-256.
| Inputs | Bundles identical | State served | Read tools from state / replayed | Write tools served / withheld | Build / three steps |
| ------------------------- | ----------------- | ------------ | -------------------------------- | ----------------------------- | ------------------- |
| 50 capture Evidence files | 50 of 50 | 50 | 1,100 / 290 | 50 / 143 | 32.9 s / 59.2 s |
| 4 scoped HARs | 4 of 4 | 4 | 166 / 21 | 5 / 12 | 8.3 s / 12.4 s |
- The coverage of every capture equals an independent census of the environment-state view with the same rules (read
views by shape, write plans by verdict and reason).
- 43 of the 50 captures have a loopback origin and are named after their file (`grafana.capture.json` becomes `grafana`)
with a `name-from-file` warning. 20 carry a `credential-named-members` warning; the published captures were scrubbed by
their pipeline, whose placeholder strings still count.
- The withheld writes' reasons: no-state-target 86, contradicted 16, unconfirmed 14, no-observation 13,
unexplained-change 8, side-effect 4, body-unusable 1, incomplete-entities 1.
## End to end
Five environments last rebuilt on 10-23 were built again with this branch's packed CLI on disposable Ubuntu VMs, four
from their scoped HAR recordings and Metabase from its capture Evidence, with `rea environment-build <capture> <dir>
--name <app> --package-tarball <tarball>`. Each bundle then ran its smoke task, a retrieval eval generated from the
bundle's own manifest and capture, and the write eval from its recorded write spec, with two models through a LiteLLM
proxy (`fireworks_ai/gpt-oss-120b`, `fireworks_ai/glm-5p3-flash`), one run per task.
| App | Input | Served tools of manifest | Served writes (build report) | Smoke | Retrieval gpt-oss / GLM | Writes gpt-oss / GLM |
| -------- | -------- | ------------------------ | -------------------------------------------------- | ------ | ----------------------- | -------------------- |
| Ghost | HAR | 35 of 38 | 0 (3 no-state-target) | passed | 12/12 / 8/12 | 0/2 / 0/2 |
| Metabase | Evidence | 40 of 42 | 1 (create contradicted, update unexplained-change) | passed | 12/12 / 12/12 | 1/2 / 1/2 |
| EspoCRM | HAR | 85 of 88 | 0 (2 contradicted, 1 unexplained-change) | passed | 12/12 / 10/12 | 0/2 / 0/2 |
| NocoDB | HAR | 23 of 28 | 0 (4 no-state-target, 1 unconfirmed) | passed | 6/6 / 5/6 | 0/3 / 0/3 |
| Sonarr | HAR | 49 of 50 | 5 (1 side-effect) | passed | 12/12 / 12/12 | 3/3 / 3/3 |
- Every write task passed with both models when the build report listed its tool as served (Metabase's card delete and
Sonarr's release-profile create, update and delete), and failed when it listed it as withheld: 8 of 24 write episodes
passed, exactly the served ones. On main these apps' other writes are withheld; the published v0.2.0 versions were
built with the state-target and write-rule changes, which serve more of them.
- The retrieval misses are GLM answers on bundles byte-identical to the three-step path's; gpt-oss passed all 54.
- The `real-agentenv` lane, switched to `environment-build`, passed from a fresh checkout on Ubuntu with
`agent-env 0.9.1334` and the host's Docker: five public cases, the smoke task passed in 26.6 s, no new containers.
## Review
An independent review rebuilt the branch and tried to refute it. It confirmed byte-equality for pretty-printed,
CRLF and newline-free Evidence, Unicode HARs across output part boundaries, inline and file view sources, browser
scenario Evidence and relative or symlinked paths, and that the refactored export answers 16 success and refusal cases
exactly as main does. Its findings, all addressed:
- The credential scan recursed and a 4 KB body nested 5,000 deep crashed the build; it now walks with its own stack, and
also scans header names, query names and form bodies.
- SIGINT after the capture was read was ignored; the build now checks the signal between stages and writes nothing when
cancelled before the bundle stage.
- The coverage view does not apply the serve-time catalog and state admission checks; a limitation and the docs now say
so.
- A dry run accepted a tarball the build refuses; it now opens it for reading.
- Repeated `--template-rule` values were deduplicated; they are refused as the views refuse them. A HAR saved as `.json`
gets a hint, file-derived names follow the bundle name rules, `localhost.` is anonymous, and the next command is
shell-quoted.
- The `no-observation` hint now covers updates and deletes of entities the state never held; the docs no longer claim
the build launches no process, state the conditions of byte-equality, and name the headers inspection excludes; the
MCP tool description names the view.
## Review round on main `584120e8`
The branch merged main `584120e8`: the export's file read now returns permission refusals through `Result`, as main
does, under the build's operation name, and the `environment-coverage` sentence sits in main's `inspect_analysis_view`
description. Four automated review findings were fixed, each with a test that fails on the merge:
- A browser scenario's retained header values and the query parameters of its recorded URLs are scanned for credential
names as a HAR's are; the manifest and replay record projections are unchanged.
- `--dry-run --package-tarball` admits and releases the tarball through the export's own regular-file scope, so a close
it cannot confirm is retried and then reported as incomplete cleanup that keeps its owner.
- A dry run checks for cancellation after reading the tarball, before it reports a plan.
- Evidence is serialized into one exactly sized buffer instead of chunks joined by a copy. Those bytes are still parsed
and sealed before authentication, so authentication reads exactly what ships.
On macOS with Node v22.23.1: the static gates passed (`npm run jscpd`: 3.53%), `npx vitest run --project domain
--project services` passed (9753, 6 skipped), the environment boundary projects passed (102), and the focused build,
export, export-inputs, serve and analysis-view boundaries with tool schema validity passed (9 files, 96). Rebuilt over
the same 50 captures and four HARs, all 54 bundles are byte-identical to the three-step path's, with the same coverage,
served writes and warnings as before the round.
## Review round on main `3d84215b`
The branch merged main through `3d84215b`: #6946, which carries this builder, in `27a4e3986`, then #7008. Integrating
#6946 took main's event-based credential scan, which also answers the review finding that a scenario transaction with
duplicate responses or content events hid their credential names; a test that both content events of a transaction
recorded twice are scanned fails on `0895c89bc`. The integration had dropped `0895c89bc`'s tests for form text recorded
without a media type and its `body-unusable` hint; both are restored, and main's scan passes the tests.
Rebuilt at `05566419e` over the same 50 captures and four HARs, all 54 bundles are byte-identical to the three-step
path's, and their warnings and credential counts equal the previous round's.