fix(environment): retain field provenance and bound state inference work

docs/specs/agent-environments/state-view/spec.md

@@ -35,7 +35,7 @@ later increments.
   and a value that changed between repeats of one request line with no recorded
   write is volatile and keeps the latest; a change across a write under another
   root, including between two tools of the type, leaves every tool of the type
-  recorded. Compare list membership, including empty lists, and field presence before selecting each request line's latest recording; a change across a write also leaves the type recorded. Different query strings remain distinct.
+  recorded. Compare list membership, including empty lists, and field presence before selecting each request line's latest recording; a change across a write also leaves the type recorded. Different query strings remain distinct; omitted fields do not erase their earlier observations on other request lines.
 - A list has `all` membership only when no item-total envelope member (`total`,
   `count` and their compounds, not page counts) differs from the number of
   entities it lists and no `has_more`, `next` or next-page token is set.

docs/specs/agent-environments/state-view/verification.md

@@ -37,3 +37,82 @@ environment manifest view refuses them on unchanged main.
 
 Not covered here: serving tool calls from the state, the AgentEnv data plane
 and write effects, which are later increments.
+
+## Linux follow-up (2026-10-20)
+
+Node 24.18.0 / npm 11.16.0, with the author's `47f8a27a9` main integration
+preserved and main `d72021b0d` integrated. Eight regression cases failed across
+two pre-fix runs: membership additions/removals across a write, field presence,
+sparse field observations, and compact compound totals. The fixes compare
+membership before selecting latest recordings, retain each field's observation
+across sparse reads, and recognize `totalcount`/`itemcount`. API-base grouping
+appends to its owned arrays; identity intersection uses a set and skips the
+first response's self-intersection.
+
+Executed checks:
+
+- Relevant domain, application and contract suites: 39 files, 493 passed,
+  3 existing skips. Includes empty/stable lists, query variants, field additions
+  and removals, sparse reads, identity proof and deferred document paths.
+- Broader compiled CLI/MCP/environment, compact schema and qlog acceptance:
+  7 files, 65 passed, 1 existing skip. After the final state fixes, both selected
+  compiled state CLI/MCP cases passed again (36 other cases deselected).
+- Final serial static/build/artifact gate: 11 tasks passed. Typecheck, lint,
+  format, knip, package metadata and runtime/catalog generation all ran.
+- Final documentation/catalog/format validation: 9 tasks passed; architecture
+  guards passed across 1,602 source files; README consistency: 5 tests passed.
+
+Real Gitea 28.2.0 used an isolated SQLite database. An owned Node fetch HAR
+recorder captured actual API exchanges, omitting transport credentials:
+
+- Three exchanges: empty repository search, creation through `/user/repos`,
+  then the same search. On exactly the same authenticated Evidence, pre-fix
+  `29e745923` published one created repository as initial state; the final
+  source publishes none and keeps the search recorded with `fields-differ`.
+- Eleven exchanges: initial issue list/details and topics, PATCH, PUT and
+  DELETE, then subsequent reads. The state keeps the two original issues
+  before ordinal 4 and the original topics document at ordinal 3. File CLI and
+  default stdio MCP agree; six HTTP SDK replays reproduce original responses.
+- Actual Chromium `capture_browser_scenario`: two transactions, eight events,
+  zero dropped. The state uses record order, extracts one issue, and keeps list
+  and item plans; file CLI and default stdio MCP agree. The earlier 12.6 MB
+  real browser capture also projects, keeping oversized bodies recorded with
+  `body-unusable` rather than parsing them into state. Owned servers and browser
+  processes were stopped.
+
+### Resource and performance measurements
+
+One CPU and one worker. Process-family RSS sums were sampled every 50 ms and
+may count shared pages twice. Final source suites: 39.669 s wall / 39.486 CPU s /
+898.42 MiB peak; broader boundaries: 270.171 s / 267.136 CPU s / 2,130.78 MiB;
+final static/build: 199.683 s / 199.523 CPU s / 4,689.07 MiB; final live HAR
+workflow: 22.836 s / 19.880 CPU s / 1,072.43 MiB; actual browser-state workflow:
+12.734 s / 10.941 CPU s / 1,549.80 MiB. The qlog acceptance lane intentionally
+exhausts an isolated decoder heap and verifies that MCP remains usable; no
+workspace OOM occurred.
+
+Before/after used immutable pre-fix `29e745923` and the final source, the same
+completed Evidence files, Node and CPU affinity. One fresh process per revision,
+with a first call and two warm repeats; filesystem caches were not cleared.
+Complete normalized output digests matched for both synthetic projection tasks.
+
+| Work                                            | Before: first / warm (ms) | After: first / warm (ms) |
+| ----------------------------------------------- | ------------------------- | ------------------------ |
+| Group 8,000 templates (helper only)             | 320.08 / 81.01–81.46      | 11.89 / 3.88–4.33        |
+| File-to-state projection, 2,000 collections     | 1,321.57 / 732.00–939.57  | 1,268.76 / 781.34–811.05 |
+| File-to-state projection, one 15,001-field item | 657.05 / 494.82–501.19    | 377.78 / 347.62–380.87   |
+
+Combined benchmark processes, including imports, three helper calls, six
+projections and the live-capture comparison: before 6.536 s wall / 6.496 CPU s /
+370.82 MiB peak; after 5.077 s / 5.056 CPU s / 386.53 MiB. Grouping and wide
+identity inference improved; the 2,000-collection projection has overlapping
+warm timings, and peak memory did not decrease. These few runs do not establish
+a general speedup. Model token usage was not measured; no model-task scoring
+was run. Initial external probe attempts had incorrect CLI positional arguments
+and assumed coverage counted records rather than grouped tools; those driver
+errors were corrected before the successful complete runs.
+
+No new macOS/Windows runtime, state serving, AgentEnv data-plane or write-effects
+claim is made. A type first observed only after an unrelated-root write may
+still lack the earlier observation needed to detect its effect. Item-shaped
+and delete-only document paths remain recorded, as specified.

docs/web-network-captures.md

@@ -492,18 +492,18 @@ changed between one tool's observations with no recorded write between is named
 in the type's `volatile_fields`, keeping the latest value. A value that changed
 across a write under another root may be that write's effect, so every tool of
 its type is a recorded view with reason `fields-differ`; so is a value one tool
-read before a write and another tool read after it. Repeated reads of one full request line also keep the type recorded when list membership changes across a write, including transitions to or from an empty list, or when entity fields are added or removed. Different query strings remain distinct request lines. Entities are ordered
+read before a write and another tool read after it. Repeated reads of one full request line also keep the type recorded when list membership changes across a write, including transitions to or from an empty list, or when entity fields are added or removed. Different query strings remain distinct request lines. Sparse responses do not erase the last observation of a field supplied by another request line. Entities are ordered
 by type, collection and first sighting; that order is a list's state order.
 Every entity cites its `record_ordinals`.
 
 `views` lists one plan per read tool in tool order:
 
-| Shape      | Plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
-| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `item`     | The entity object at `pointer` (the root or one member), the ordered `fields` the view shows and the `required_fields` every recording had, and `envelope_ordinal`: the recording whose response, with the entity replaced, every item response shares                                                                                                                                                                                                                                           |
-| `list`     | The array at `pointer`, its `fields`, `membership` (`all`: every entity of the collection in state order, when each recording listed exactly that, had no query, no item-total member (`total`, `count`, `totalCount`, `item_count`, but not `total_pages` or `per_page`; a number or a string of digits) differing from the number listed, and no `has_more`, `next` or next-page token member set; `recorded`: the entities each recording listed) and `counters` that equal the number listed |
-| `document` | The path's latest initial value, held in `documents`                                                                                                                                                                                                                                                                                                                                                                                                                                             |
-| `recorded` | A `reason` the tool replays its recordings instead                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+| Shape      | Plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `item`     | The entity object at `pointer` (the root or one member), the ordered `fields` the view shows and the `required_fields` every recording had, and `envelope_ordinal`: the recording whose response, with the entity replaced, every item response shares                                                                                                                                                                                                                                                                      |
+| `list`     | The array at `pointer`, its `fields`, `membership` (`all`: every entity of the collection in state order, when each recording listed exactly that, had no query, no item-total member (`total`, `count`, `totalCount`, `item_count`, `totalcount`, `itemcount`, but not `total_pages` or `per_page`; a number or a string of digits) differing from the number listed, and no `has_more`, `next` or next-page token member set; `recorded`: the entities each recording listed) and `counters` that equal the number listed |
+| `document` | The path's latest initial value, held in `documents`                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| `recorded` | A `reason` the tool replays its recordings instead                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
 
 An item or list plan is kept only when it reproduces the latest initial
 recording of each request line from the state, comparing JSON values without

src/domain/analysisView/EnvironmentStateExtraction.ts

@@ -273,7 +273,11 @@ export class EnvironmentStateExtraction {
         return;
       }
       pointer = located.pointer;
-      keys = (keys ?? located.keys).filter((key) => located.keys.includes(key));
+      if (keys === undefined) keys = located.keys;
+      else {
+        const matching = new Set(located.keys);
+        keys = keys.filter((key) => matching.has(key));
+      }
     }
     const field =
       IDENTITY_NAMES.find((name) => keys?.includes(name) === true) ?? keys?.[0];

src/domain/analysisView/environmentStateGate.ts

@@ -202,11 +202,17 @@ const nameTokens = (name: string): string[] =>
     .split(/[_\-\s]|(?<=[a-z0-9])(?=[A-Z])/u)
     .map((token) => token.toLowerCase());
 
-/** `total`, `count`, `totalCount` or `item_count`, but not `total_pages`, `per_page` or `account_id`. */
+/** Item totals, including compact `totalcount`/`itemcount`; page counts and account IDs are excluded. */
 const isItemTotal = (name: string): boolean => {
   const tokens = nameTokens(name);
   return (
-    tokens.some((token) => token === "total" || token === "count") &&
+    tokens.some(
+      (token) =>
+        token === "total" ||
+        token === "count" ||
+        token === "totalcount" ||
+        token === "itemcount",
+    ) &&
     !tokens.some(
       (token) => token === "page" || token === "pages" || token === "per",
     )

src/domain/analysisView/environmentStateMerge.ts

@@ -73,7 +73,7 @@ export const mergeObservedEntity = (entity: ObservedEntity): StateEntity => {
 /**
  * How observed values changed, per entity in timeline order. A shared field that differs from the
  * previous repeat of the same request line is volatile, unless a recorded write lies between them;
- * a field that differs from the previous observation of another line across a write rewrites the
+ * a field that differs from its previous observation on another line across a write rewrites the
  * type. Either across a write means the reads do not agree on one initial state.
  */
 export const classifyEntityChanges = (
@@ -97,7 +97,12 @@ export const classifyEntityChanges = (
       (left, right) => left.position - right.position,
     );
     const lastOfLine = new Map<string, StateObservation>();
-    timeline.forEach((after, index) => {
+    // Sparse projections do not erase where each field was last observed.
+    const lastOfField = new Map<
+      string,
+      { readonly observation: StateObservation; readonly line: string }
+    >();
+    timeline.forEach((after) => {
       const line = JSON.stringify(lineOf(after));
       const repeat = lastOfLine.get(line);
       if (repeat !== undefined) {
@@ -111,14 +116,18 @@ export const classifyEntityChanges = (
               (volatile.get(entity.type) ?? new Set()).add(name),
             );
       }
-      const previous = timeline[index - 1];
-      if (
-        previous !== undefined &&
-        JSON.stringify(lineOf(previous)) !== line &&
-        changed(previous, after).length > 0 &&
-        writeBetween(previous, after)
-      )
-        rewritten.add(entity.type);
+      const sighting = { observation: after, line };
+      for (const name of Object.keys(after.object)) {
+        const previous = lastOfField.get(name);
+        if (
+          previous !== undefined &&
+          previous.line !== line &&
+          !same(previous.observation.object[name], after.object[name]) &&
+          writeBetween(previous.observation, after)
+        )
+          rewritten.add(entity.type);
+        lastOfField.set(name, sighting);
+      }
       lastOfLine.set(line, after);
     });
   }

src/domain/analysisView/environmentStateView.test.ts

@@ -548,6 +548,62 @@ it("leaves a type recorded when two of its tools read different values across a
   ]);
 });
 
+it.each(["old", "new"])(
+  "retains field observations through sparse reads across a write (%s)",
+  (title) => {
+    const state = stateOf(
+      captureOf([
+        ["GET", "/api/search", [{ id: 1, title: "old" }]],
+        ["POST", "/api/items", { id: 2 }],
+        ["GET", "/api/search?fields=id", [{ id: 1 }]],
+        ["GET", "/api/search/1", { id: 1, title }],
+      ]),
+    );
+    if (title === "new") {
+      expect(state.types).toEqual([]);
+      expect(state.entities).toEqual([]);
+      expect(
+        state.views.every(
+          (view) =>
+            view.shape === "recorded" && view.reason === "fields-differ",
+        ),
+      ).toBe(true);
+    } else {
+      expect(viewOf(state, "get_search_by_id")).toMatchObject({
+        shape: "item",
+      });
+      expect(state.entities[0]?.fields.title).toBe("old");
+    }
+  },
+);
+
+it("proves the identity field against every item response", () => {
+  const state = stateOf(
+    captureOf([
+      ["GET", "/api/widgets/1", { left: 1, right: 1 }],
+      ["GET", "/api/widgets/2", { right: 2, left: "other" }],
+    ]),
+  );
+  expect(state.types).toMatchObject([
+    { id_field: "right", id_basis: "path-parameter" },
+  ]);
+  expect(state.entities.map((entity) => entity.id)).toEqual(["1", "2"]);
+});
+
+it.each(["totalcount", "itemcount"])(
+  "recognizes a compact compound item total (%s)",
+  (name) => {
+    const state = stateOf(
+      captureOf([["GET", "/api/items", { [name]: 50, items: [{ id: 1 }] }]]),
+    );
+    expect(viewOf(state, "get_items")).toMatchObject({
+      shape: "list",
+      membership: "recorded",
+      counters: [],
+    });
+  },
+);
+
 it.each([
   [[], [{ id: 1 }]],
   [[{ id: 1 }], []],