From 2fe849da0d06cccd76d1cbdf260a32b91e54a368 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Mon, 26 Oct 2026 10:18:45 -0700 Subject: [PATCH] docs(environment): record verification for blank members and sorted writes Co-Authored-By: Claude Opus 5.5 --- .../write-form-calls/spec.md | 5 +- .../write-form-calls/tasks.md | 2 +- .../write-form-calls/verification.md | 47 ++++++++++++++++--- 3 files changed, 45 insertions(+), 9 deletions(-) diff --git a/docs/specs/agent-environments/write-form-calls/spec.md b/docs/specs/agent-environments/write-form-calls/spec.md index 904fe60fd9..b59e21f266 100644 --- a/docs/specs/agent-environments/write-form-calls/spec.md +++ b/docs/specs/agent-environments/write-form-calls/spec.md @@ -37,8 +37,9 @@ natural-language tasks ask for, or withheld them: or loaded so. An unsent update field that changes only between `null`, `[]` and `{}` is unchanged. - **Unruled fields.** A later create that stored a field no rule writes otherwise than the plan keeps it is `unexplained-change` at the create, so its recorded entity is held and later reads check other writes as before. -- **Skill.** The second scratch create fills only what a person must give, and the skill shows how to read signed-in - pages' accessibility trees from a scenario. +- **Skill.** The two scratch creates differ in what they send, the second only what a person must give, however they + are made; the skill shows how to read signed-in pages' accessibility trees from a scenario, and re-records once with + request steps a UI write whose body the browser did not expose. Plans and views that need none of this are unchanged. diff --git a/docs/specs/agent-environments/write-form-calls/tasks.md b/docs/specs/agent-environments/write-form-calls/tasks.md index 6c32f994f6..a97673c1e6 100644 --- a/docs/specs/agent-environments/write-form-calls/tasks.md +++ b/docs/specs/agent-environments/write-form-calls/tasks.md @@ -7,5 +7,5 @@ - [x] Withhold later creates that store an unruled field otherwise. - [x] Add view, serving and unit tests. - [x] Document the rules and view member, and update the skill. -- [ ] Measure the recorded captures and served write tasks. +- [x] Measure the recorded captures and served write tasks. - [x] Run static, formatting, documentation and test gates. diff --git a/docs/specs/agent-environments/write-form-calls/verification.md b/docs/specs/agent-environments/write-form-calls/verification.md index 2768dc35f4..225734634a 100644 --- a/docs/specs/agent-environments/write-form-calls/verification.md +++ b/docs/specs/agent-environments/write-form-calls/verification.md @@ -1,6 +1,6 @@ # Verification -Executed on macOS with Node v24.22.1 at `54993559` (main `99062fea`), and on Ubuntu 24.04 VMs with snap Chromium for +Executed on macOS with Node v24.22.1 at `5a3c6762` (main `99062fea`), and on Ubuntu 24.04 VMs with snap Chromium for the agent recordings. ## Gates @@ -71,14 +71,49 @@ the required members. | Total | 42 | 18 19 | 24 23 | No recording passes fewer tasks with this branch. Paperless creates pass once served. The saved-values Grafana -recording's rename is served and passes; its create still needs the saved-values branch's composed address. The -Vikunja task create (its project path writes a type its list does not show as a list), EspoCRM's leftover contact and -the main Homebox recording, which wrote no tags, are unchanged. +recording's rename is served and passes; its create still needs the saved-values branch's composed address. +Vikunja's task create, EspoCRM's leftover contact (held as seeded data, so every "nothing else changed" check fails) +and the main Homebox recording, which wrote no tags, are unchanged. -**Fresh recordings.** FRESH_RESULTS +**Fresh recordings.** Each application was recorded afresh with main (skill 49) and with this branch. The branch was +recorded again for the four applications whose first branch recording passed fewer tasks, and a third time after the +skill's sixth step was reworded; the final skill's recordings are the last column (served writes, then tasks): + +| Application | Write tasks | Main | Branch, first two recordings | Branch, final skill | +| ----------- | ----------- | -------------------- | ---------------------------- | -------------------- | +| Paperless | 3 | 2, 1/3 2/3 | 8, 1/3 2/3 | | +| Linkding | 2 | 0, 0/2 0/2 | 5, 2/2 2/2 | | +| Miniflux | 3 | no bundle | 4, 3/3 3/3 | | +| Grafana | 2 | 3, 2/2 2/2 | 1, 0/2 1/2; 3, 1/2 2/2 | 0, 0/2 0/2 | +| Vikunja | 3 + 2 | 3, 1/3 1/3 + 2/2 2/2 | 0, 0/5 0/5; 0, 0/5 0/5 | 1, 1/3 1/3 + 0/2 0/2 | +| EspoCRM | 3 | 2, 1/3 1/3 | 2, 1/3 1/3; 2, 1/3 1/3 | 2, 1/3 1/3 | +| Homebox | 2 | 4, 2/2 2/2 | 1, 0/2 0/2; 1, 0/2 0/2 | 3, 2/2 2/2 | + +Every fresh recording builds the same plans with main and with this branch, except Paperless (main serves 0 of the +branch recording's writes and 2 of 3 of its own), Vikunja's main recording (3 to 4) and the EspoCRM recordings below; +the per-application differences are what each agent recorded: + +- **Paperless:** the branch recording serves creates, renames and deletes of tags, correspondents and document types. + Its two tag creates used a light and a dark color, whose derived `text_color` differs, so the tag create stays + withheld and the tasks, which create a tag, score as on main. +- **Linkding:** main's agent recorded no writes. The branch's second bookmark create sent only `url` and `title`, so + the served create requires only those. +- **Miniflux:** main's agent stopped and asked for an API key. The branch's agent created a token in the UI and sent it + as a literal header value, which REA recorded as written; the instance and its token were destroyed afterwards. +- **EspoCRM:** following the sixth step, the branch's second contact sent only a last name, with `null` for strings the + first stored and for flags it sent `false`. That merge was refused until the empty-spelling fix in this PR; with it, + the create requires only `lastName` and `name` and its task passes, as on main. +- **Vikunja:** the first two branch agents wrote through the UI, whose request bodies Chromium did not expose, so every + write was `body-unusable`. The sixth step then read "fill every field the form offers"; reworded to say how the + creates differ, not where they are made, the third agent wrote with request steps. It recorded no label writes. +- **Homebox:** the first two branch agents recorded item writes only, the third tags as main's did. +- **Grafana:** the branch agents created folders with request steps and their own uids. The second create's address + needs the saved-values branch's composed-address fixes to be explained, so the create is withheld here. ## Not verified -- One or two fresh recordings per application and version; agent runs vary more than the builds do. +- One to three fresh recordings per application and version; agent runs vary more than the builds do. +- Request bodies Chromium does not expose for some UI writes (Vikunja's); the skill now re-records such a write once + with request steps. - Paperless derives `text_color` from `color`, so a create whose two recorded colors differ in brightness stays withheld; EspoCRM's create still requires `name` and `emailAddressData`, which it derives from other members.