From 466ed8b433af5043a74027acb68e30956ebfa30c Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Wed, 21 Oct 2026 11:53:26 -0700 Subject: [PATCH] fix(environment): name entities and weigh interference per write as serving and the recording show Greptile raised these on this branch, the fidelity branch and the list-lines branch; five follow from the previous round's changes. - An entity field whose name looks like a pointer is matched by name; only a document's pointers cover the members under them. - A failed read contradicts each exercised write that no unmodelled write came after, instead of skipping them all when the oldest pending write preceded one; the shared pending `since` time goes. - An item read, update or delete names its entity as serving's namedEntity does: held collections its path matches, verbatim spellings first, the id decoded once, then as written; `book%2531` no longer reaches `book1`. - A plan whose calls use a member both as the identity and as a fixed member, or fix it at different values, is refused, as serving could not take every recorded call. - A size of zero bounds nothing only on a line whose recording listed items under it (pageBound), in serving and in the gate. - An update whose observation drops a field the entity held is refused, as serving keeps every field it does not write. - A later create whose clock takes another format, or whose unique value is not a UUID, is refused. - A derived state data/get would write past the 32 MiB intake limit is not served (admitServedState); the environment serves replay. The verdicts over the 50 captures are unchanged plan by plan, and the smoke applies all 31 served write tools. Co-Authored-By: Claude Opus 5.5 --- docs/agent-environments.md | 3 +- .../agent-environments/state-writes/spec.md | 33 ++-- .../state-writes/verification.md | 19 ++ .../environment/environmentState.test.ts | 23 +++ .../environment/environmentState.ts | 31 ++- .../environment/environmentStateAnswer.ts | 6 +- .../environment/environmentStateStore.ts | 46 +++-- .../environmentStateWriteAnswer.test.ts | 20 ++ .../EnvironmentStateWriteSimulation.test.ts | 177 ++++++++++++++++++ .../EnvironmentStateWriteSimulation.ts | 51 ++--- .../analysisView/environmentStatePaging.ts | 15 +- .../environmentStateWriteAttribution.ts | 34 ++-- .../environmentStateWriteChecks.ts | 73 +++++--- .../environmentStateWriteCreate.ts | 23 ++- .../environmentStateWriteDocument.ts | 2 +- .../environmentStateWriteEffects.ts | 4 +- .../environmentStateWriteExplain.ts | 15 +- 17 files changed, 458 insertions(+), 117 deletions(-) diff --git a/docs/agent-environments.md b/docs/agent-environments.md index 9efcde5425..04149915d8 100644 --- a/docs/agent-environments.md +++ b/docs/agent-environments.md @@ -404,7 +404,8 @@ A state that is unavailable does not stop the server: it logs a warning naming the error `code` and message before it binds, then serves replay. The state is unavailable when it cannot be derived, such as from a `capture_browser_scenario` capture that selected header values but no response -bodies or over the view's budget, and when its catalog under the served name +bodies or over the view's budget, when `data/get` would write it past the 32 MiB +`data/add` intake limit, and when its catalog under the served name exceeds the [discovery reservation](#what-a-manifest-is). That check runs first, so no state is derived for it, and replay is then admitted exactly as with `--replay-only`. `--replay-only` diff --git a/docs/specs/agent-environments/state-writes/spec.md b/docs/specs/agent-environments/state-writes/spec.md index b9ddfbeace..5dd4ff747c 100644 --- a/docs/specs/agent-environments/state-writes/spec.md +++ b/docs/specs/agent-environments/state-writes/spec.md @@ -105,7 +105,11 @@ the value a recorded call sent it with, and then stores the value that call recorded (its `constant` rule carries it). The type's identity field never takes a rule: sent as the path's id it is the identity member, sent as another value the application ignored it is accepted with that value and stores -nothing, and a stored id that changed is `unexplained-change`. +nothing, and a stored id that changed is `unexplained-change`. A plan whose +calls use a member both as the identity and as a fixed member, or fix it at +different values, is `unexplained-change`, as serving could not take every +recorded call. So is an update whose observation drops a field the entity +held, since serving keeps every field an update does not write. `PUT` requires every member the exemplar sent; `PATCH` and `POST` require at least one. A field the exemplar's update changed without sending it is explained as one of: @@ -159,9 +163,10 @@ the initial state. **Writes.** Only a 2xx write is modelled. A modelled write applies its effect, using the values it recorded for generated fields and the stored values for fields it transformed. A modelled write whose call the plan does not explain brings its entity to the -recorded outcome instead, without attribution. A later create whose sequence -fields hold another value than serving would generate makes the plan -`unexplained-change`; identities are left to the id mapping. An update or +recorded outcome instead, without attribution. A later create whose generated +values take another form than serving gives them (a sequence value out of +order, a clock in another format, a non-UUID unique value) makes the plan +`unexplained-change`; integer identities are left to the id mapping. An update or delete names its entity as an item read does (below). Any other write that may have changed state, including a redirected or @@ -183,8 +188,9 @@ members they sent. from the state: - an item read of a state-backed item view, of the entity serving would name: - the read's collection and id as written, else a held collection whose path - matches once percent-decoded; + in a held collection the read's path matches, each segment verbatim or once + percent-decoded and verbatim spellings first, the id percent-decoded once, + then as written; - a list or document read on one of its view's checked request lines, chosen as replay chooses the line for the read's call: each path parameter the read's value verbatim or once percent-decoded, preferring verbatim @@ -194,7 +200,8 @@ from the state: different values is never chosen, since no call's arguments can name it; one repeating a name with one value is, as a call naming it matches both. A list is compared as served from the collection of that line. A read on a page - (`limit`, `size`, `per_page` and the like, a size of zero bounding nothing, or a `_start`–`_end` range) of a + (`limit`, `size`, `per_page` and the like, or a `_start`–`_end` range; a size + of zero bounds nothing on a line whose recording listed items under it) of a whole list checks nothing once the collection holds more entities than the page, which serving then answers from its recording. - A read made while a modelled write to its entity or collection was still @@ -224,15 +231,17 @@ refuses to answer with. instead), a recorded outcome every field, and a create or delete the whole entity. A document write's effect is the members it merged at its object, or everything under the object it replaces, the whole document at the - root. A write with nothing left on an entity no + root; an entity's fields match by name only, whatever their names look like. A write with nothing left on an entity no longer counts there, nor in a collection where no entity it changed keeps any of its effect. A mismatch contradicts them. A pass confirms those it shows something of: a create's or delete's membership, or a changed field the read's view shows and does not treat as volatile; a read that shows none of an update's changes confirms nothing of it, and leaves it for a later check of that entity or collection. -- A mismatch after an unmodelled write that followed those writes is checked - only on the members they sent: such a write may explain the rest. +- A mismatch contradicts each exercised write that no unmodelled write came + after, whatever earlier write is still pending. When an unmodelled write + followed some of them, an item read checks those only on the members they + sent and a list is not checked again: such a write may explain the rest. - A mismatch on an entity or collection that no write changed contradicts, as `side-effect`, the modelled writes since that previous check whose paths are segment-prefix related to the read's path. @@ -305,7 +314,9 @@ installs and of the type's checked list lines (so a collection the capture listed empty keeps its spelling), never those of a state it replaced. When two spellings fit a call equally, the one holding entities wins. `data/add` refuses a document that `data/get` would write back past the 32 MiB intake limit once it adds the members a document may omit, such as -`types` and the state Evidence id. +`types` and the state Evidence id. A derived state `data/get` would write +past that limit is not served: the environment serves replay and reports the +state `unavailable`. **Clock and generators.** The store's clock starts at `write_clock` and advances one second before each served write; clock fields take its value in diff --git a/docs/specs/agent-environments/state-writes/verification.md b/docs/specs/agent-environments/state-writes/verification.md index 2345a73be8..50a2859ef5 100644 --- a/docs/specs/agent-environments/state-writes/verification.md +++ b/docs/specs/agent-environments/state-writes/verification.md @@ -319,3 +319,22 @@ A fourteenth round settled two findings Greptile raised on the list-queries bran Both tests fail on the previous head. The verdicts over the 50 captures are unchanged plan by plan, the smoke applies all 31 served write tools, and the gates passed again. + +A fifteenth round settled nine findings Greptile raised on this branch, the fidelity branch and the list-lines branch, +five of them on the previous rounds' own changes: + +- an entity field whose name looks like a pointer is matched by name; only a document's pointers cover the members + under them; +- a failed read contradicts each exercised write no unmodelled write came after, rather than skipping all of them + when the oldest pending write came before one; +- an item read, update or delete names its entity as serving does, decoding the id once, so `book%2531` no longer + reaches `book1`; +- a plan whose calls use a member both as the identity and as a fixed member, or fix it at different values, is + refused; +- a size of zero bounds nothing only on a line whose recording listed items under it; +- an update whose observation drops a field the entity held is refused; +- a later create whose clock takes another format, or whose unique value is not a UUID, is refused; +- a derived state `data/get` would write past 32 MiB is not served. + +Each new test fails on the previous head. The verdicts over the 50 captures are unchanged plan by plan, the smoke +applies all 31 served write tools, and the gates passed again. diff --git a/src/application/environment/environmentState.test.ts b/src/application/environment/environmentState.test.ts index bced6224af..d03619dc5b 100644 --- a/src/application/environment/environmentState.test.ts +++ b/src/application/environment/environmentState.test.ts @@ -24,6 +24,7 @@ import { } from "../analysisView/AnalysisViewService.js"; import { loadEnvironmentSource } from "./environmentSource.js"; import { + admitServedState, deriveEnvironmentState, loadEnvironmentState, prepareServedState, @@ -272,6 +273,28 @@ describe("prepareServedState", () => { }); }); +describe("admitServedState", () => { + it("refuses a derived state data/get would write past what data/add loads back", () => { + expect(admitServedState(state, "environment_serve").ok).toBe(true); + // A euro sign is one UTF-16 unit but three bytes as data/get writes it. + const notes = "\u20ac".repeat(12_000_000); + const [first] = state.entities; + const large = { + ...state, + entities: [ + ...state.entities, + { ...first!, id: "large", fields: { ...first!.fields, notes } }, + ], + }; + const admitted = admitServedState(large, "environment_serve"); + if (admitted.ok) throw new Error("Expected the state to be refused"); + expect(projectAnalysisError(admitted.error)).toMatchObject({ + code: "resource_constraint", + message: expect.stringContaining("larger than data/add loads"), + }); + }); +}); + describe("EnvironmentStateStore", () => { it("reports its state as rea-state/v1, accepts that report back and resets to the initial state", () => { const store = new EnvironmentStateStore(state); diff --git a/src/application/environment/environmentState.ts b/src/application/environment/environmentState.ts index 9211969fb8..bd91e0e8e7 100644 --- a/src/application/environment/environmentState.ts +++ b/src/application/environment/environmentState.ts @@ -8,9 +8,9 @@ import { } from "../../domain/analysisView/environmentStateRules.js"; import type { StateWritePlan } from "../../domain/analysisView/environmentStateWriteRules.js"; import type { AnalysisError } from "../../domain/analysisErrorBase.js"; -import type { - AnalysisInputIssue, +import { AnalysisResourceConstraintError, + type AnalysisInputIssue, } from "../../domain/analysisErrorCore.js"; import { projectAnalysisError } from "../../domain/analysisErrorProjection.js"; import { err, ok, type Result } from "../../domain/result.js"; @@ -23,6 +23,8 @@ import { type EnvironmentManifestTool, type EnvironmentSource, } from "./environmentSource.js"; +import { STATE_INTAKE_LIMIT_BYTES } from "./environmentStateDocument.js"; +import { initialStateBytes } from "./environmentStateStore.js"; import { describeWrite } from "./environmentStateWriteDescription.js"; import { environmentToolDefinitions, @@ -238,6 +240,29 @@ export interface PreparedState extends ServedState { >; } +/** The derived state, unless data/get would write it past what data/add loads back, so its own report could not be restored. */ +export const admitServedState = ( + state: EnvironmentState, + operation: string, +): Result => { + const bytes = initialStateBytes(state); + return bytes > STATE_INTAKE_LIMIT_BYTES + ? err( + new AnalysisResourceConstraintError( + operation, + "capacity", + "The derived state is larger than data/add loads, so it cannot be served", + { max_state_bytes: STATE_INTAKE_LIMIT_BYTES, state_bytes: bytes }, + { + retryable: false, + remediationAction: + "Capture fewer or smaller responses, or serve the environment as replay only.", + }, + ), + ) + : ok(state); +}; + const unavailable = (error: AnalysisError): PreparedState => { const { code, message } = projectAnalysisError(error); return { @@ -262,6 +287,8 @@ export const prepareServedState = ( if (!reads.ok) return unavailable(reads.error); const derived = deriveEnvironmentState(source, captureValue, operation); if (!derived.ok) return unavailable(derived.error); + const admitted = admitServedState(derived.value, operation); + if (!admitted.ok) return unavailable(admitted.error); // Served write tools join the catalog, so the whole catalog is admitted again. const catalog = servedCatalog(source, derived.value, name, operation); return catalog.ok diff --git a/src/application/environment/environmentStateAnswer.ts b/src/application/environment/environmentStateAnswer.ts index 7417d6524e..1973b84dd2 100644 --- a/src/application/environment/environmentStateAnswer.ts +++ b/src/application/environment/environmentStateAnswer.ts @@ -16,7 +16,7 @@ import { type StateView, } from "../../domain/analysisView/environmentStateRules.js"; import { - pageSize, + pageBound, pagesOnly, } from "../../domain/analysisView/environmentStatePaging.js"; import { isAbsence } from "../../domain/analysisView/environmentStateWriteChecks.js"; @@ -428,7 +428,9 @@ const answerList = ( const collection = collectionPathOf(base, false); const type = store.type(view.type); // A requested page the collection has outgrown is answered as recorded, never with more items than it asked for. - const size = exact ? pageSize(base.query ?? undefined) : undefined; + const size = exact + ? pageBound(base.query ?? undefined, recorded.length) + : undefined; if ( view.membership === "all" && size !== undefined && diff --git a/src/application/environment/environmentStateStore.ts b/src/application/environment/environmentStateStore.ts index 9807f256ab..f5927cc5f2 100644 --- a/src/application/environment/environmentStateStore.ts +++ b/src/application/environment/environmentStateStore.ts @@ -37,6 +37,27 @@ const itemsBytes = (items: readonly unknown[]): number => items.reduce((sum, item) => sum + jsonBytes(item), 0) + Math.max(0, items.length - 1); +/** What `data/get` reports for some content of a state. */ +const snapshotOf = ( + state: EnvironmentState, + entities: readonly StateEntity[], + documents: readonly HeldDocument[], +): StateSnapshot => ({ + rea_state: { + version: 1, + state_evidence_id: state.evidenceId, + types: state.types, + entities, + documents, + }, +}); + +/** What `data/get` writes for a state as the Evidence holds it, in UTF-8 bytes, summed item by item. */ +export const initialStateBytes = (state: EnvironmentState): number => + jsonBytes(snapshotOf(state, [], [])) + + itemsBytes(state.entities) + + itemsBytes(state.documents); + /** * The live state one served environment answers from, shared by every MCP * session and the data plane. Mutations are synchronous and swap the whole @@ -74,8 +95,8 @@ export class EnvironmentStateStore { ); this.#clockStart = state.writeClock === null ? 0 : Date.parse(state.writeClock); - this.#frameBytes = jsonBytes(this.#snapshotOf([], [])); - this.#initialBytes = this.#contentBytes(state); + this.#frameBytes = jsonBytes(snapshotOf(state, [], [])); + this.#initialBytes = initialStateBytes(state); this.reset(); } @@ -303,7 +324,11 @@ export class EnvironmentStateStore { } snapshot(): StateSnapshot { - return this.#snapshotOf(this.#content.entities, this.#content.documents); + return snapshotOf( + this.state, + this.#content.entities, + this.#content.documents, + ); } /** What `data/get` writes for the current state, in UTF-8 bytes. */ @@ -311,21 +336,6 @@ export class EnvironmentStateStore { return this.#bytes; } - #snapshotOf( - entities: readonly StateEntity[], - documents: readonly HeldDocument[], - ): StateSnapshot { - return { - rea_state: { - version: 1, - state_evidence_id: this.state.evidenceId, - types: this.state.types, - entities, - documents, - }, - }; - } - /** What `data/get` writes for some content, summed item by item so the whole state is never one string. */ #contentBytes(content: StateContent): number { return ( diff --git a/src/application/environment/environmentStateWriteAnswer.test.ts b/src/application/environment/environmentStateWriteAnswer.test.ts index 94851dcc33..12ea6b8974 100644 --- a/src/application/environment/environmentStateWriteAnswer.test.ts +++ b/src/application/environment/environmentStateWriteAnswer.test.ts @@ -709,6 +709,26 @@ describe("paged lines", () => { }); }); + it("answers a zero limit a whole line recorded empty from that recording once a create fills its collection", () => { + const alpha = { id: 1, name: "Alpha", org: "1" }; + const beta = { id: 2, name: "Beta", org: "2" }; + const books = servedFrom([ + ["GET", "/api/orgs/1/books", [alpha]], + ["GET", "/api/orgs/1/books/1", alpha], + ["GET", "/api/orgs/2/books?limit=0", []], + ["POST", "/api/orgs/2/books", beta, 201, { name: "Beta" }], + ["GET", "/api/orgs/2/books/2", beta], + ]); + books.answer("post_orgs_books_by_id", { id: "2", body: { name: "Gamma" } }); + // Only a recording that listed items under it shows that zero lists every entity. + expect( + books.answer("get_orgs_books_by_id", { id: "2", limit: "0" }), + ).toMatchObject({ + body: [], + state: { source: "recording", reason: "page-exceeded" }, + }); + }); + it("answers a range the collection has outgrown from its recording", () => { const books = servedFrom([ ["GET", "/api/books?_start=0&_end=2", [FIRST_BOOK, SECOND_BOOK]], diff --git a/src/domain/analysisView/EnvironmentStateWriteSimulation.test.ts b/src/domain/analysisView/EnvironmentStateWriteSimulation.test.ts index 1f12c2b23e..970551ab2c 100644 --- a/src/domain/analysisView/EnvironmentStateWriteSimulation.test.ts +++ b/src/domain/analysisView/EnvironmentStateWriteSimulation.test.ts @@ -983,3 +983,180 @@ describe("what a later call must follow", () => { }); }); }); + +describe("what a write's effect keeps", () => { + it("keeps an entity member a later write's other member does not cover, though both names look like pointers", () => { + const flat = (id: number, prefs: string, lang: string) => ({ + id, + name: `Book ${String(id)}`, + "/prefs": prefs, + "/prefs/lang": lang, + }); + const rows: TimedCaptureRow[] = [ + ["GET", "/api/books", [flat(1, "x", "en"), flat(2, "x", "en")]], + ["GET", "/api/books/1", flat(1, "x", "en")], + ["GET", "/api/books/2", flat(2, "x", "en")], + [ + "PATCH", + "/api/books/1", + flat(1, "x", "fr"), + 200, + { "/prefs/lang": "fr" }, + ], + ["PUT", "/api/books/1", flat(1, "y", "fr"), 200, { "/prefs": "y" }], + ["GET", "/api/books/1", flat(1, "y", "fr")], + ]; + expect(gated(rows, "patch_books_by_id")).toMatchObject({ + served: true, + checked_record_ordinals: [5], + }); + }); + + it("contradicts a write made after an unmodelled one, though an earlier pending write came before it", () => { + const summary = (name: string) => [ + { id: 1, name }, + { id: 2, name: "B" }, + ]; + const full = (name: string, description: string) => ({ + id: 1, + name, + description, + }); + const rows: TimedCaptureRow[] = [ + ["GET", "/api/books", summary("A")], + ["GET", "/api/books/1", full("A", "x")], + ["GET", "/api/books/2", { id: 2, name: "B", description: "y" }], + ["PATCH", "/api/books/1", full("A", "new"), 200, { description: "new" }], + ["GET", "/api/books", summary("A")], + ["POST", "/api/exports", { queued: true }, 202, { format: "csv" }], + [ + "PUT", + "/api/books/1", + full("Z", "new"), + 200, + { name: "Z", description: "new" }, + ], + ["GET", "/api/books/1", full("Z", "new")], + // The export came before the rename, so it cannot explain the list showing the old name. + ["GET", "/api/books", summary("A")], + ]; + expect(gated(rows, "put_books_by_id")).toMatchObject({ + served: false, + contradicted_by: 8, + }); + }); + + it("refuses an update whose recorded response dropped a field serving would keep", () => { + const rows: TimedCaptureRow[] = [ + [ + "GET", + "/api/books", + [ + { id: 1, name: "A" }, + { id: 2, name: "B" }, + ], + ], + ["GET", "/api/books/1", { id: 1, name: "A", subtitle: "s" }], + ["GET", "/api/books/2", { id: 2, name: "B", subtitle: "t" }], + ["PATCH", "/api/books/1", { id: 1, name: "Z" }, 200, { name: "Z" }], + [ + "GET", + "/api/books", + [ + { id: 1, name: "Z" }, + { id: 2, name: "B" }, + ], + ], + ]; + expect(gated(rows, "patch_books_by_id")).toMatchObject({ + served: false, + reason: "unexplained-change", + }); + }); +}); + +describe("what later calls must keep consistent", () => { + it("refuses an update one call sends an ignored id with and another the path's id", () => { + const renamed = (at: Book, name: string) => ({ ...at, name }); + const rows: TimedCaptureRow[] = [ + ...BOOK_INIT_ROWS, + [ + "PATCH", + "/api/books/1", + renamed(FIRST_BOOK, "B"), + 200, + { id: 999, name: "B" }, + ], + ["GET", "/api/books/1", renamed(FIRST_BOOK, "B")], + [ + "PATCH", + "/api/books/2", + renamed(SECOND_BOOK, "C"), + 200, + { id: 2, name: "C" }, + ], + ["GET", "/api/books/2", renamed(SECOND_BOOK, "C")], + ]; + expect(gated(rows, "patch_books_by_id")).toMatchObject({ + served: false, + reason: "unexplained-change", + }); + }); + + it("refuses a create whose later call records a clock in another format", () => { + const epoch = (iso: string) => Math.floor(Date.parse(iso) / 1000); + const fourth = { + ...book(4, "Fourth Book", bookClock(5)), + created_at: epoch(bookClock(5)), + updated_at: epoch(bookClock(5)), + }; + const rows: TimedCaptureRow[] = [ + ...BOOK_INIT_ROWS, + ["POST", "/api/books", THIRD_BOOK, 201, { name: "Third Book" }], + ["GET", "/api/books/3", THIRD_BOOK], + ["POST", "/api/books", fourth, 201, { name: "Fourth Book" }], + ["GET", "/api/books/4", fourth], + ]; + expect(gated(rows, "post_books")).toMatchObject({ + served: false, + reason: "unexplained-change", + }); + }); + + it("names no entity by a delete whose path id decodes to it only twice, as serving would not", () => { + const uid = (key: string) => ({ uid: key, name: key }); + const rows: TimedCaptureRow[] = [ + ["GET", "/api/books", [uid("book1"), uid("book2")]], + ["GET", "/api/books/uid/book1", uid("book1")], + ["GET", "/api/books/uid/book2", uid("book2")], + ["DELETE", "/api/books/uid/book%2531", "", 204], + ["GET", "/api/books/uid/book1", BOOK_NOT_FOUND, 404], + ]; + const { writes } = environmentStateViewOf(timedCaptureEvidence(rows), [ + "after-uid-literal", + ]); + expect(writes.find((plan) => plan.effect === "delete")).toMatchObject({ + served: false, + checked_record_ordinals: [], + }); + }); + + it("checks nothing by a whole line recorded empty under a zero limit once a create fills its collection", () => { + const alpha = { id: 1, name: "Alpha", org: "1" }; + const beta = { id: 2, name: "Beta", org: "2" }; + const rows: TimedCaptureRow[] = [ + ["GET", "/api/orgs/1/books", [alpha]], + ["GET", "/api/orgs/1/books/1", alpha], + ["GET", "/api/orgs/2/books?limit=0", []], + ["POST", "/api/orgs/2/books", beta, 201, { name: "Beta" }], + ["GET", "/api/orgs/2/books/2", beta], + // An empty recording does not show that zero lists every entity. + ["GET", "/api/orgs/2/books?limit=0", []], + ]; + const { writes } = environmentStateViewOf(timedCaptureEvidence(rows)); + expect(writes.find((plan) => plan.effect === "create")).toMatchObject({ + served: true, + checked_record_ordinals: [4], + }); + }); +}); diff --git a/src/domain/analysisView/EnvironmentStateWriteSimulation.ts b/src/domain/analysisView/EnvironmentStateWriteSimulation.ts index d2deb15dec..058c4ab0a1 100644 --- a/src/domain/analysisView/EnvironmentStateWriteSimulation.ts +++ b/src/domain/analysisView/EnvironmentStateWriteSimulation.ts @@ -88,7 +88,7 @@ export class EnvironmentStateWriteSimulation { readonly #touched: TouchedTargets = new Map(); readonly #effects = new WriteEffects(); readonly #lastCheck = new Map(); - /** The earliest modelled write to each target since its last check, and when the latest of them ended. */ + /** When the latest modelled write to each target since its last check ended. */ readonly #touchedSince: PendingTargets = new Map(); /** Timeline indexes of writes the gate does not model. */ readonly #unmodelledAt: number[] = []; @@ -236,22 +236,18 @@ export class EnvironmentStateWriteSimulation { result.entity.collection, result.entity.id, ); - if (result.kind === "applied") - this.#effects.supersede(this.#touched, write.tool, result); - if (result.kind === "observed") + if (result.kind === "observed") { this.#effects.observed(this.#touched, result.targets); - for (const target of result.targets) { // A recorded outcome replaces the entity's values, not its collection's membership. - if (result.kind === "observed") { - if (target === result.targets[0]) this.#touched.delete(target); - continue; - } + this.#touched.delete(result.targets[0]!); + return; + } + this.#effects.supersede(this.#touched, write.tool, result); + for (const target of result.targets) touchTarget(this.#touched, this.#touchedSince, target, { ...write, fields: result.fields, }); - } - if (result.kind !== "applied") return; this.#log.push(write); if (result.inserted !== undefined) this.#inserted.set(result.inserted, write.tool); @@ -362,7 +358,7 @@ export class EnvironmentStateWriteSimulation { return; const exercised = exercisedBy(this.#touched, keys, borrowers); if (borrowers !== undefined) { - this.#contradictOthers(outcome.pass, keys, exercised, record); + this.#contradictOthers(outcome.pass, exercised, record); return; } const pass = @@ -372,11 +368,17 @@ export class EnvironmentStateWriteSimulation { ? this.#confirm(view, record, keys, exercised) : undefined; if (!pass) { - if (!this.#interfered(keys)) + const interfered = [...exercised].filter((name) => + this.#interferedWith(name), + ); + if (interfered.length === 0) this.#mismatch({ record, index, keys, exercised, view }); - // An unmodelled write since these writes may explain the difference: check their own members only. + // An unmodelled write after some of these writes may explain the difference: those are checked on their own members only. else if (view.shape === "item") this.#checkMembers(view, record); - else for (const key of keys) this.#taint.fail(key); + else { + this.#contradictOthers(false, exercised, record); + for (const key of keys) this.#taint.fail(key); + } if (view.shape === "item") this.#adopt(view, record); } const pending = { @@ -390,16 +392,16 @@ export class EnvironmentStateWriteSimulation { } } - /** A read a write's effect was learned from checks nothing of that write, yet still contradicts the others. */ + /** A failed read contradicts the writes it exercised that no unmodelled write came after, as a read a write's effect was learned from still does the others. */ #contradictOthers( pass: boolean, - keys: readonly string[], exercised: ReadonlySet, record: StateRecordFacts, ): void { - if (pass || this.#interfered(keys)) return; + if (pass) return; for (const name of exercised) - this.#drafts.get(name)!.contradictedBy ??= record.ordinal; + if (!this.#interferedWith(name)) + this.#drafts.get(name)!.contradictedBy ??= record.ordinal; } /** A passing read confirms the writes it exercised that it shows something of, and names, per target, those it shows nothing of. */ @@ -426,12 +428,11 @@ export class EnvironmentStateWriteSimulation { this.#unmodelledAt.push(index); } - /** Whether a write the gate does not model came after the earliest modelled write a failed check exercised. */ - #interfered(keys: readonly string[]): boolean { - const since = Math.min( - ...keys.map((key) => this.#touchedSince.get(key)?.since ?? Infinity), - ); - return this.#unmodelledAt.some((index) => index > since); + /** Whether a write the gate does not model came after a tool's latest modelled call, so it may explain what a read shows of that call. */ + #interferedWith(tool: string): boolean { + const last = + this.#log.findLast((write) => write.tool === tool)?.index ?? -1; + return this.#unmodelledAt.some((index) => index > last); } /** Check the writes to an item read's entity on the members they sent, as a resynchronising read does. */ diff --git a/src/domain/analysisView/environmentStatePaging.ts b/src/domain/analysisView/environmentStatePaging.ts index 237e94f50a..e483c24b17 100644 --- a/src/domain/analysisView/environmentStatePaging.ts +++ b/src/domain/analysisView/environmentStatePaging.ts @@ -42,12 +42,19 @@ export const pageSize = (query: string | undefined): number | undefined => { for (const [name, value] of iterateUrlQueryParameters(query ?? "")) { const key = name.toLowerCase(); if (!/^\d+$/u.test(value)) continue; - // A size of zero bounds nothing: applications read it as no limit, as a whole line naming it shows. - if (PAGE_SIZE_PARAMETERS.has(key)) { - if (Number(value) > 0) size = Math.min(size, Number(value)); - } else if (key === "_start") range.start = Number(value); + if (PAGE_SIZE_PARAMETERS.has(key)) size = Math.min(size, Number(value)); + else if (key === "_start") range.start = Number(value); else if (key === "_end") range.end = Number(value); } const width = Math.min(size, Math.max(range.end - range.start, 0)); return Number.isFinite(width) ? width : undefined; }; + +/** The most items a line's call lists: its page size, but none for a size of zero the line's own recording listed items under, as an application reading zero as no limit does. */ +export const pageBound = ( + query: string | undefined, + recorded: number, +): number | undefined => { + const size = pageSize(query); + return size === 0 && recorded > 0 ? undefined : size; +}; diff --git a/src/domain/analysisView/environmentStateWriteAttribution.ts b/src/domain/analysisView/environmentStateWriteAttribution.ts index 787a0ecaf5..9d586b6a95 100644 --- a/src/domain/analysisView/environmentStateWriteAttribution.ts +++ b/src/domain/analysisView/environmentStateWriteAttribution.ts @@ -13,8 +13,8 @@ export const shownFields = ( /** Modelled writes per target since its last check, with the fields each wrote (null: all). */ export type TouchedTargets = Map>; -/** Per target, the earliest modelled write since its last check and when the latest of them ended. */ -export type PendingTargets = Map; +/** Per target, when the latest modelled write since its last check ended. */ +export type PendingTargets = Map; /** Whether a modelled write to the target was still running when a read began at `start`, so the read may show either state. */ export const runningAt = ( @@ -36,22 +36,19 @@ export const exercisedBy = ( return exercised; }; -/** Record a modelled write on a target: the fields it wrote (undefined: all), when it began and when it ended. */ +/** Record a modelled write on a target: the fields it wrote (undefined: all) and when it ended. */ export const touchTarget = ( touched: TouchedTargets, pending: PendingTargets, target: string, write: { - readonly index: number; readonly tool: string; readonly end: number; readonly fields: readonly string[] | undefined; }, ): void => { - const since = pending.get(target); pending.set(target, { - since: since?.since ?? write.index, - until: Math.max(since?.until ?? -Infinity, write.end), + until: Math.max(pending.get(target)?.until ?? -Infinity, write.end), }); const tools = touched.get(target) ?? new Map(); const before = tools.get(write.tool); @@ -71,14 +68,17 @@ type ChangedFields = readonly string[] | null; interface Overwrite { readonly whole: boolean; readonly fields: ReadonlySet | "all"; + readonly pointers: boolean; } -/** Whether a write's fields cover a field: the same one, or, for a document, a pointer above it. */ -const covers = (fields: ReadonlySet, field: string): boolean => - fields.has(field) || - [...fields].some( - (over) => over.startsWith("/") && field.startsWith(`${over}/`), +/** Whether a write's fields cover a field: the same one, or, for a document's pointers, one above it. */ +const covers = (write: Overwrite, field: string): boolean => { + const { fields } = write; + if (fields === "all" || fields.has(field)) return true; + return ( + write.pointers && [...fields].some((over) => field.startsWith(`${over}/`)) ); +}; /** What an earlier effect keeps after a write: nothing once every field it holds is overwritten; an update leaves a create's or delete's whole effect. */ const remaining = ( @@ -87,9 +87,8 @@ const remaining = ( ): ChangedFields | undefined => { if (write.whole) return undefined; if (claim === null) return null; - const { fields } = write; - if (fields === "all") return undefined; - const left = claim.filter((field) => !covers(fields, field)); + if (write.fields === "all") return undefined; + const left = claim.filter((field) => !covers(write, field)); return left.length > 0 ? left : undefined; }; @@ -160,6 +159,8 @@ export class WriteEffects { readonly fields: readonly string[]; readonly accumulated: readonly string[]; }; + /** Set when the fields are a document's pointers. */ + readonly pointers?: boolean; }, ): void { const [entity, collection] = effect.targets as [string, string?]; @@ -175,6 +176,7 @@ export class WriteEffects { (field) => !effect.changed?.accumulated.includes(field), ), ), + pointers: effect.pointers === true, }, ); this.#changed.set( @@ -196,7 +198,7 @@ export class WriteEffects { this.#overwrite( touched, { entity, collection, writer: undefined }, - { whole: false, fields: "all" }, + { whole: false, fields: "all", pointers: false }, ); } diff --git a/src/domain/analysisView/environmentStateWriteChecks.ts b/src/domain/analysisView/environmentStateWriteChecks.ts index 8efdf0d90f..759ce4bd4f 100644 --- a/src/domain/analysisView/environmentStateWriteChecks.ts +++ b/src/domain/analysisView/environmentStateWriteChecks.ts @@ -6,11 +6,9 @@ import { } from "../unicodeCodePointOrder.js"; import { at, - collectionPathOf, decodedSegment, isObject, isParam, - lastSegmentValue, listItems, missingRequiredFields, renderStateEntity, @@ -28,7 +26,7 @@ import { type StateToolFacts, type StateView, } from "./environmentStateRules.js"; -import { pagesOnly, pageSize } from "./environmentStatePaging.js"; +import { pageBound, pagesOnly } from "./environmentStatePaging.js"; /** A query as replay compares a call's: decoded values by name, in any order; undefined when it repeats a name with another value, which no call's arguments can carry. */ const queryKey = (query: string | undefined): string | undefined => { @@ -253,35 +251,48 @@ interface ReadBodies { } /** An item read: the answer the held entity renders into the served envelope must equal the recording; a recorded 404 or 410 agrees with none held. */ +/** How many of a held collection's segments match a path's verbatim, or undefined when one matches neither verbatim nor once percent-decoded. */ +const segmentMatch = ( + held: readonly string[], + read: readonly string[], +): number | undefined => { + if (held.length !== read.length) return undefined; + let verbatim = 0; + for (const [index, segment] of held.entries()) + if (segment === read[index]) verbatim += 1; + else if (decodedSegment(segment) !== read[index]) return undefined; + return verbatim; +}; + /** - * The entity an item read names, resolved as serving resolves an item call: its collection and id as written, else a - * held collection whose path matches once percent-decoded and an id tried decoded, then as written. + * The entity an item read or write names, as serving names it: in a held collection its path matches, verbatim + * spellings first, the id percent-decoded once, then as written; else the collection and decoded id as written. */ export const itemTarget = ( state: SimulatedState, type: string, record: Pick, ): { readonly collection: string; readonly id: string } => { - const collection = collectionPathOf(record, true); - const id = lastSegmentValue(record.path); - if (state.entity(type, collection, id) !== undefined) - return { collection, id }; - const read = segmentsOf(collection); - const ids = [decodedSegment(id), id]; - for (const candidate of ids) - for (const entity of state.ofType(type)) { - const held = segmentsOf(entity.collection); - if ( - entity.id === candidate && - held.length === read.length && - held.every( - (segment, index) => - segment === read[index] || decodedSegment(segment) === read[index], - ) - ) - return { collection: entity.collection, id: entity.id }; - } - return { collection, id }; + const read = segmentsOf(record.path); + const raw = read.pop() ?? ""; + const ids = [...new Set([decodedSegment(raw), raw])]; + const written = `/${read.join("/")}`; + for (const id of ids) + if (state.entity(type, written, id) !== undefined) + return { collection: written, id }; + const collections = [ + ...new Set(state.ofType(type).map((entity) => entity.collection)), + ] + .flatMap((collection) => { + const verbatim = segmentMatch(segmentsOf(collection), read); + return verbatim === undefined ? [] : [{ collection, verbatim }]; + }) + .sort((left, right) => right.verbatim - left.verbatim); + for (const { collection } of collections) + for (const id of ids) + if (state.entity(type, collection, id) !== undefined) + return { collection, id }; + return { collection: written, id: ids[0]! }; }; export const checkItemRead = ( @@ -403,19 +414,19 @@ export const checkListRead = ( check: ReadBodies & { readonly idField: string; readonly collection: string }, ): CheckOutcome | undefined => { if (!succeeded(record)) return undefined; + const { body, base } = check; + const baseItems = + base === undefined ? undefined : listItems(base, view.pointer); + if (body === undefined || base === undefined || baseItems === undefined) + return undefined; // Serving answers a page too small for the collection from its recording, so such a read checks nothing. - const size = pageSize(record.query); + const size = pageBound(record.query, baseItems.length); if ( view.membership === "all" && size !== undefined && state.members(view.type, check.collection).length > size ) return undefined; - const { body, base } = check; - const baseItems = - base === undefined ? undefined : listItems(base, view.pointer); - if (body === undefined || base === undefined || baseItems === undefined) - return undefined; // A usable body that no longer holds the list is not what serving answers on this line. const items = listItems(body, view.pointer); const bodies = { ...check, body, base, items: items ?? [], baseItems }; diff --git a/src/domain/analysisView/environmentStateWriteCreate.ts b/src/domain/analysisView/environmentStateWriteCreate.ts index 39897b2f5b..7308d4caca 100644 --- a/src/domain/analysisView/environmentStateWriteCreate.ts +++ b/src/domain/analysisView/environmentStateWriteCreate.ts @@ -40,7 +40,7 @@ import { type Observed, } from "./environmentStateWriteObserve.js"; import type { FieldRule, WriteCall } from "./environmentStateWriteRules.js"; -import { integerOf } from "./environmentStateWriteValues.js"; +import { integerOf, isUuid, readClock } from "./environmentStateWriteValues.js"; /** The id a response names alone: a scalar body, or an object's id field or single scalar member. */ const respondedId = ( @@ -186,6 +186,25 @@ const followsSequences = ( return integerOf(created.fields[rule.field]) === next; }); +/** Whether a later call's generated values take the forms serving generates them in: clocks in the learned format, unique values and UUID identities as UUIDs, and sequences in order. */ +const followsGenerated = ( + context: EffectContext, + draft: WriteDraft, + created: Pick, +): boolean => + draft.plan.rules.every((rule) => { + const value = created.fields[rule.field]; + if (rule.rule === "clock") + return ( + JSON.stringify(readClock(value)?.format) === JSON.stringify(rule.format) + ); + return ( + (rule.rule !== "unique" && + (rule.rule !== "identity" || rule.generator !== "uuid")) || + isUuid(value) + ); + }) && followsSequences(context, draft, created); + /** Record the values a create's sequences took, which the next call follows. */ const takeSequences = ( draft: WriteDraft, @@ -275,7 +294,7 @@ export const applyCreate = ( id === undefined || context.state.entity(type, collection, id) !== undefined || (later && - !followsSequences(context, draft, { + !followsGenerated(context, draft, { type, collection, fields: fields.value, diff --git a/src/domain/analysisView/environmentStateWriteDocument.ts b/src/domain/analysisView/environmentStateWriteDocument.ts index caf9da66e8..0a2342bdaa 100644 --- a/src/domain/analysisView/environmentStateWriteDocument.ts +++ b/src/domain/analysisView/environmentStateWriteDocument.ts @@ -134,6 +134,6 @@ export const applyDocument = ( return { kind: "applied", targets: [documentTarget(tool)], - ...(fields === undefined ? {} : { fields }), + ...(fields === undefined ? {} : { fields, pointers: true }), }; }; diff --git a/src/domain/analysisView/environmentStateWriteEffects.ts b/src/domain/analysisView/environmentStateWriteEffects.ts index 4d2ea39fbe..29f853dff3 100644 --- a/src/domain/analysisView/environmentStateWriteEffects.ts +++ b/src/domain/analysisView/environmentStateWriteEffects.ts @@ -122,8 +122,10 @@ export type EffectResult = readonly kind: "applied"; readonly targets: readonly string[]; readonly entity?: WrittenEntity; - /** The fields an update wrote; a create writes the whole entity. */ + /** The fields an update wrote, or the pointers a document write wrote; a create writes the whole entity. */ readonly fields?: readonly string[]; + /** Set when the fields are a document's pointers, each covering the members under it. */ + readonly pointers?: boolean; /** The fields an update sent or changed, derived ones included, and those of them its increments add to. */ readonly changed?: { readonly fields: readonly string[]; diff --git a/src/domain/analysisView/environmentStateWriteExplain.ts b/src/domain/analysisView/environmentStateWriteExplain.ts index 40f4af59db..d5e69470a5 100644 --- a/src/domain/analysisView/environmentStateWriteExplain.ts +++ b/src/domain/analysisView/environmentStateWriteExplain.ts @@ -301,6 +301,9 @@ export const explainUpdate = ( value: evidence.after[member]!, }); } + // Serving keeps every field an update does not write, so one the update removed is a change no rule models. + for (const field of Object.keys(evidence.before)) + if (!Object.hasOwn(evidence.after, field)) unexplained.push(field); for (const field of Object.keys(evidence.after)) { if ( Object.hasOwn(body, field) || @@ -448,13 +451,19 @@ export const conflictsWithPlan = ( explained: ReturnType, ): boolean => { const body = plan.body!; - const fixed = new Set(body.fixed.map((entry) => entry.member)); + const fixed = new Map(body.fixed.map((entry) => [entry.member, entry.sent])); const rules = new Map(plan.rules.map((rule) => [rule.field, rule])); + const { identity } = explained.body; + // Serving checks a call's identity member before its fixed values, so a member both cannot take every recorded call. return ( explained.body.accepted.some((member) => fixed.has(member)) || - explained.body.fixed.some((entry) => - body.accepted.includes(entry.member), + explained.body.fixed.some( + (entry) => + body.accepted.includes(entry.member) || + entry.member === body.identity || + (fixed.has(entry.member) && !same(fixed.get(entry.member), entry.sent)), ) || + (identity !== null && fixed.has(identity)) || explained.rules.some((rule) => { const held = rules.get(rule.field); return held !== undefined && !same(held, rule);