From 27311f88d419c4aa9fc7d0078acbdd6523cdb1d5 Mon Sep 17 00:00:00 2001 From: morluto <76467478+morluto@users.noreply.github.com> Date: Wed, 14 Oct 2026 15:20:08 +0800 Subject: [PATCH] fix(browser): preserve capture uncertainty and frame ancestry Parse required resource rows and retain valid siblings. Missing collections and malformed child subtrees keep unknown omission counts. Compare complete frame forests without capture-local IDs, and never infer aggregate DOM/metadata changes from omissions. Co-authored-by: ShuaixinHuang <1836656514@qq.com> --- docs/browser-observation.md | 23 ++ src/browser/CdpCaptureDocuments.test.ts | 12 +- src/browser/CdpCaptureDocuments.ts | 105 ++++-- src/domain/webCaptureDiff.ts | 85 +++-- src/domain/webCaptureFrameComparison.test.ts | 62 ++++ src/domain/webCaptureFrameComparison.ts | 49 +++ .../cdpDomComparisonCompleteness.test.ts | 266 +++++++++++++++ .../browser/cdpResourceCompleteness.test.ts | 322 ++++++++++++++++++ 8 files changed, 878 insertions(+), 46 deletions(-) create mode 100644 src/domain/webCaptureFrameComparison.test.ts create mode 100644 src/domain/webCaptureFrameComparison.ts create mode 100644 tests/boundary/browser/cdpDomComparisonCompleteness.test.ts create mode 100644 tests/boundary/browser/cdpResourceCompleteness.test.ts diff --git a/docs/browser-observation.md b/docs/browser-observation.md index 9479f4b3c6..9c52ae8779 100644 --- a/docs/browser-observation.md +++ b/docs/browser-observation.md @@ -20,6 +20,17 @@ Inspect native SAZ or HAR 1.2 offline with REA, or connect Everywhere's separate - `observe_web_session` / `rea observe-web-session` arms an observation window of the requested duration for an external user action and records ordered reload, SPA navigation, redirect, failure, lifecycle, and target-termination metadata. - `discover_webmcp_tools` / `rea discover-webmcp-tools` uses the experimental CDP WebMCP domain to return every in-scope registration and its complete observed `input_schema` declaration and canonical `input_schema_sha256` digest inline. Declarations are untrusted JSON data, including declared types, required fields, enums, examples and reference URLs. REA does not resolve schema references. These fields replace `input_schema_shape`. Registrations with the same name and URL in distinct observed frames retain separate identities. `tool_key` includes the observed frame ID; it is not a stable identity across navigation or browser sessions. Same-frame registration updates replace that owner only, and removal leaves other owners intact. Declared `readOnly`, `untrustedContent`, `consequential`, and `autosubmit` annotation hints are reported per registration: `true`/`false` are declared values, `null` means the declaration did not provide the hint in that capture, and a missing `consequential` key means the hint was not recorded, either in a historical capture or because the observed annotation was invalid. Consult discovery completeness and exclusions to distinguish these cases. Malformed annotation values retain the registration with incomplete coverage and an `invalid_protocol_value` exclusion. Other malformed protocol payloads can fail discovery explicitly. REA never exposes `WebMCP.invokeTool`. - `compare_web_captures` / `rea compare-web-captures` compares passive captures or exact-step-aligned browser scenarios. Scenario results commit reproducible literal normalization and expose alignment failures plus artifact-level action, screenshot, DOM, accessibility, URL, history, storage, and event differences. Missing or truncated evidence is never treated as equivalence. + +Passive DOM comparison checks complete node inventories and complete frame +parent/child structure independently. Frame IDs and sibling enumeration order +are capture-local and do not establish a change; duplicate IDs, missing parents, +and cyclic ancestry remain unknown. A complete frame difference can still prove +change when DOM nodes are incomplete, and vice versa. Partial DOM or metadata +collection digests do not prove modification: a missing node or link can result +from a protocol omission. Rejected DOM facts also mark derived URL and agent-hint +metadata incomplete, while valid facts from other nodes and documents remain +available in the capture. + - `capture_web_screenshot` / `rea capture-web-screenshot` returns a content-addressed visible-viewport PNG. - `compare_web_screenshots` / `rea compare-web-screenshots` performs bounded local PNG pixel comparison without OCR or external services. - The CLI comparison commands accept each capture, screenshot artifact or normalization policy as inline JSON or a JSON file path. Use files for screenshot artifacts and scenario captures, whose embedded PNG bytes can exceed the host's command-line length limit. @@ -294,6 +305,18 @@ validation and resource bounds. Omitted script metadata makes its affected section explicitly truncated. Disallowed-origin entries are filtered before retention. +Resource-tree inventories distinguish a valid empty `resources` array from a +missing or malformed collection. Invalid resource entries are omitted with +incomplete coverage, while valid resources from that frame and other frames +remain available. Comparing an incomplete resource inventory cannot prove +addition, removal, or equality; a complete empty inventory can prove removal. +Resource records require a nonempty type and a string MIME type; absent or +malformed required fields are recorded as omissions instead of invented defaults. +An explicitly empty MIME type is valid. The optional content size may be absent; +an invalid supplied size remains unknown with incomplete resource coverage while +the resource's other facts are retained. Zero and nonnegative fractional sizes +are preserved. + CDP discovery, connection, and commands wait for the peer or caller cancellation; a stalled peer can leave an operation waiting. WebSocket messages remain subject to the [64 MiB limit](#request-boundary). REA does not impose a message-count or diff --git a/src/browser/CdpCaptureDocuments.test.ts b/src/browser/CdpCaptureDocuments.test.ts index be8656e8a4..d1e20c2a07 100644 --- a/src/browser/CdpCaptureDocuments.test.ts +++ b/src/browser/CdpCaptureDocuments.test.ts @@ -268,7 +268,9 @@ describe("CDP frame capture", () => { it("walks deeply nested frame trees and returns every frame and resource", () => { let frameTree: Record = { frame: { id: "frame-0", url: `${origin}/0` }, - resources: [{ url: `${origin}/0.js`, type: "Script" }], + resources: [ + { url: `${origin}/0.js`, type: "Script", mimeType: "text/javascript" }, + ], }; for (let index = 1; index < 2_000; index += 1) { frameTree = { @@ -276,7 +278,13 @@ describe("CDP frame capture", () => { id: `frame-${String(index)}`, url: `${origin}/${String(index)}`, }, - resources: [{ url: `${origin}/${String(index)}.js`, type: "Script" }], + resources: [ + { + url: `${origin}/${String(index)}.js`, + type: "Script", + mimeType: "text/javascript", + }, + ], childFrames: [frameTree], }; } diff --git a/src/browser/CdpCaptureDocuments.ts b/src/browser/CdpCaptureDocuments.ts index f9f97d3750..4d3645f1de 100644 --- a/src/browser/CdpCaptureDocuments.ts +++ b/src/browser/CdpCaptureDocuments.ts @@ -14,6 +14,7 @@ import { cdpSnapshotString, cdpSnapshotValueString, } from "./CdpSnapshotStrings.js"; +import type { BrowserExclusionReason } from "../domain/browserCompleteness.js"; import { exclusionReasonForUrl } from "./CdpCaptureEventHelpers.js"; import type { CdpCaptureCompleteness } from "./CdpCaptureCompleteness.js"; import { @@ -43,8 +44,8 @@ export const captureFrames = ( return { items: [] }; } const items: WebPageInspection["frames"] = []; - for (const tree of walkFrameTrees(root, () => - completeness?.exclude("frames", "invalid_protocol_value"), + for (const tree of walkFrameTrees(root, (omittedCount) => + completeness?.exclude("frames", "invalid_protocol_value", omittedCount), )) { const frame = recordValue(tree.frame); const frameId = cdpStringValue(frame?.id); @@ -61,9 +62,16 @@ export const captureFrames = ( continue; } if (maximum !== undefined && items.length >= maximum) continue; + const parentFrameId = cdpStringValue(frame.parentId); + // Only the enclosing tree's root may legitimately omit parent evidence. + if ( + (tree !== root || Object.hasOwn(frame, "parentId")) && + (parentFrameId === undefined || parentFrameId.length === 0) + ) + completeness?.exclude("frames", "invalid_protocol_value"); items.push({ frame_id: frameId, - parent_frame_id: cdpStringValue(frame.parentId) ?? null, + parent_frame_id: parentFrameId ?? null, url: sanitized.url, origin: sanitized.origin, }); @@ -91,10 +99,19 @@ export const captureResources = ( return { items: [] }; } const items: CapturedResource[] = []; - for (const tree of walkFrameTrees(root, () => - completeness?.exclude("resources", "invalid_protocol_value"), + for (const tree of walkFrameTrees(root, (omittedCount) => + completeness?.exclude("resources", "invalid_protocol_value", omittedCount), )) { - for (const resource of recordsValue(tree.resources)) { + if (!Array.isArray(tree.resources)) { + completeness?.exclude("resources", "invalid_protocol_value", null); + continue; + } + for (const entry of tree.resources) { + const resource = recordValue(entry); + if (resource === undefined) { + completeness?.exclude("resources", "invalid_protocol_value"); + continue; + } const url = allowedSanitizedUrl(resource.url, allowedOrigins); if (url === undefined || url.origin === null) { completeness?.exclude( @@ -103,15 +120,26 @@ export const captureResources = ( ); continue; } - const contentSize = numberValue(resource.contentSize); + const type = cdpStringValue(resource.type); + const mimeType = cdpStringValue(resource.mimeType); + if (type === undefined || type.length === 0 || mimeType === undefined) { + completeness?.exclude("resources", "invalid_protocol_value"); + continue; + } + const rawContentSize = numberValue(resource.contentSize); + const contentSize = + rawContentSize !== undefined && rawContentSize >= 0 + ? rawContentSize + : null; + if (resource.contentSize !== undefined && contentSize === null) + completeness?.exclude("resources", "invalid_protocol_value"); items.push({ rawUrl: cdpStringValue(resource.url) ?? "", url: url.url, origin: url.origin, - type: cdpStringValue(resource.type) ?? "Other", - mime_type: cdpStringValue(resource.mimeType) ?? "", - content_size: - contentSize === undefined ? null : Math.max(0, contentSize), + type, + mime_type: mimeType, + content_size: contentSize, }); } } @@ -120,7 +148,7 @@ export const captureResources = ( export const walkFrameTrees = function* ( root: UnknownRecord, - onMalformedChild?: () => void, + onMalformedChild?: (omittedCount: number | null) => void, ): Generator { const pending = [root]; while (pending.length > 0) { @@ -128,11 +156,11 @@ export const walkFrameTrees = function* ( if (tree === undefined) return; const rawChildren = tree.childFrames; if (rawChildren !== undefined && !Array.isArray(rawChildren)) - onMalformedChild?.(); + onMalformedChild?.(null); const children = Array.isArray(rawChildren) ? rawChildren.flatMap((child) => { const parsed = recordValue(child); - if (parsed === undefined) onMalformedChild?.(); + if (parsed === undefined) onMalformedChild?.(null); return parsed === undefined ? [] : [parsed]; }) : []; @@ -187,14 +215,16 @@ export function* captureDom( let excludedUrls = 0; let total = 0; let unresolvedSvgNodes = 0; - for (const document of recordsValue(root.documents)) { + if (!Array.isArray(root.strings)) + excludeDocumentEvidence(completeness, "invalid_protocol_value", null); + for (const document of snapshotDocuments(root.documents, completeness)) { const documentUrl = cdpSnapshotString(strings, document.documentURL); if (documentUrl === undefined) { - completeness?.exclude("dom", "invalid_protocol_value"); + excludeDocumentEvidence(completeness, "invalid_protocol_value"); continue; } if (allowedSanitizedUrl(documentUrl, allowedOrigins) === undefined) { - completeness?.exclude("dom", exclusionReasonForUrl(documentUrl)); + excludeDocumentEvidence(completeness, exclusionReasonForUrl(documentUrl)); continue; } const baseUrlValue = @@ -202,12 +232,15 @@ export function* captureDom( ? documentUrl : cdpSnapshotValueString(strings, document.baseURL); if (baseUrlValue === undefined) { - completeness?.exclude("dom", "invalid_protocol_value"); + excludeDocumentEvidence(completeness, "invalid_protocol_value"); continue; } const baseUrl = baseUrlValue || documentUrl; const documentNodes = recordValue(document.nodes); - if (documentNodes === undefined) continue; + if (documentNodes === undefined) { + excludeDocumentEvidence(completeness, "invalid_protocol_value"); + continue; + } const rawNodeTypes = numberArray(documentNodes.nodeType); const rawNodeNames = numberArray(documentNodes.nodeName); const rawNodeValues = numberArray(documentNodes.nodeValue); @@ -225,12 +258,12 @@ export function* captureDom( rawNodeValues.length !== rawNodeTypes.length || rawParents.length !== rawNodeTypes.length ) { - completeness?.exclude("dom", "invalid_protocol_value"); + excludeDocumentEvidence(completeness, "invalid_protocol_value"); continue; } if (rawAttributes.length > rawNodeTypes.length) - completeness?.exclude( - "dom", + excludeDocumentEvidence( + completeness, "invalid_protocol_value", rawAttributes.length - rawNodeTypes.length, ); @@ -266,7 +299,7 @@ export function* captureDom( attributeIndexes === undefined || !cdpSnapshotAttributesResolve(strings, attributeIndexes) ) { - completeness?.exclude("dom", "invalid_protocol_value"); + excludeDocumentEvidence(completeness, "invalid_protocol_value"); continue; } const attributes = attributeIndexes as readonly number[]; @@ -346,6 +379,32 @@ export function* captureDom( }; } +/** Skipped document facts also make their derived URL and hint inventory incomplete. */ +const excludeDocumentEvidence = ( + completeness: CdpCaptureCompleteness | undefined, + reason: BrowserExclusionReason, + count: number | null = 1, +): void => { + completeness?.exclude("dom", reason, count); + completeness?.exclude("metadata", reason, count); +}; + +const snapshotDocuments = ( + value: unknown, + completeness: CdpCaptureCompleteness | undefined, +): readonly UnknownRecord[] => { + if (!Array.isArray(value) || value.length === 0) { + excludeDocumentEvidence(completeness, "invalid_protocol_value", null); + return []; + } + return value.flatMap((item) => { + const document = recordValue(item); + if (document === undefined) + excludeDocumentEvidence(completeness, "invalid_protocol_value"); + return document === undefined ? [] : [document]; + }); +}; + function* captureDomMetadata( pending: readonly PendingDomMetadata[], completeness: CdpCaptureCompleteness | undefined, diff --git a/src/domain/webCaptureDiff.ts b/src/domain/webCaptureDiff.ts index 39e88e6d6a..e062582a5b 100644 --- a/src/domain/webCaptureDiff.ts +++ b/src/domain/webCaptureDiff.ts @@ -9,6 +9,7 @@ import { type WebCaptureDimension, } from "./webCaptureDiffSchemas.js"; import type { WebMcpDiscovery } from "./webMcpDiscovery.js"; +import { fingerprintWebCaptureFrames } from "./webCaptureFrameComparison.js"; type Dimension = WebCaptureDimension; type Change = WebCaptureChange; @@ -172,13 +173,7 @@ const compareWebCaptureDimensions = ( const before = input.before.inspection; const after = input.after.inspection; return { - dom_structure: compareDimension( - singleton("document", digestCanonicalValue(domProjection(before))), - singleton("document", digestCanonicalValue(domProjection(after))), - sectionsComplete(before, ["frames", "dom"]) && - sectionsComplete(after, ["frames", "dom"]), - "DOM or frame capture was incomplete in at least one observation.", - ), + dom_structure: domDimension(before, after), scripts: compareDimension( keyed( before.scripts.items.map(scriptProjection), @@ -200,9 +195,10 @@ const compareWebCaptureDimensions = ( "Resource inventory was incomplete in at least one observation.", ), network: networkDimension(before, after), - metadata: compareDimension( - singleton("metadata", digestCanonicalValue(metadataProjection(before))), - singleton("metadata", digestCanonicalValue(metadataProjection(after))), + metadata: compareAggregate( + "metadata", + metadataProjection(before), + metadataProjection(after), sectionsComplete(before, ["metadata"]) && sectionsComplete(after, ["metadata"]), "Safe metadata capture was incomplete in at least one observation.", @@ -601,17 +597,64 @@ const incompleteSections = (completeness: { ...completeness.unavailable_sections, ]); -const domProjection = (inspection: WebPageInspection) => ({ - frames: inspection.frames - .map(({ url, origin }) => ({ url, origin })) - .sort( - (left, right) => - compareUnicodeCodePoints(left.url, right.url) || - compareUnicodeCodePoints(left.origin ?? "", right.origin ?? "") || - Number(left.origin !== null) - Number(right.origin !== null), - ), - nodes: inspection.dom.nodes.map(({ index: _index, ...node }) => node), -}); +/** Whole-collection digest differences cannot prove a change across omissions. */ +const compareAggregate = ( + identity: string, + before: unknown, + after: unknown, + complete: boolean, + reason: string, +): Dimension => + compareDimension( + complete ? singleton(identity, digestCanonicalValue(before)) : new Map(), + complete ? singleton(identity, digestCanonicalValue(after)) : new Map(), + complete, + reason, + ); + +const domDimension = ( + before: WebPageInspection, + after: WebPageInspection, +): Dimension => { + const beforeFrames = fingerprintWebCaptureFrames(before.frames); + const afterFrames = fingerprintWebCaptureFrames(after.frames); + const nodesComplete = [before, after].every( + (inspection) => + sectionsComplete(inspection, ["dom"]) && + inspection.dom.total_nodes === inspection.dom.nodes.length, + ); + const framesComplete = + beforeFrames !== null && + afterFrames !== null && + [before, after].every((inspection) => + sectionsComplete(inspection, ["frames"]), + ); + const changes: Change[] = []; + if ( + nodesComplete && + digestCanonicalValue(before.dom.nodes) !== + digestCanonicalValue(after.dom.nodes) + ) + changes.push({ identity: "document", change: "modified" }); + if (framesComplete && beforeFrames !== afterFrames) + changes.push({ identity: "frames", change: "modified" }); + if (changes.length > 0) + return { + status: "changed", + total_changes: changes.length, + changes, + reason: null, + }; + return nodesComplete && framesComplete + ? { status: "unchanged", total_changes: 0, changes: [], reason: null } + : { + status: "unknown", + total_changes: 0, + changes: [], + reason: + "DOM or frame capture was incomplete, or frame ancestry was unresolved in at least one observation.", + }; +}; const scriptProjection = ( script: WebPageInspection["scripts"]["items"][number], diff --git a/src/domain/webCaptureFrameComparison.test.ts b/src/domain/webCaptureFrameComparison.test.ts new file mode 100644 index 0000000000..bd70649a96 --- /dev/null +++ b/src/domain/webCaptureFrameComparison.test.ts @@ -0,0 +1,62 @@ +import { describe, expect, it } from "vitest"; + +import type { WebPageInspection } from "./browserObservationSchemas.js"; +import { fingerprintWebCaptureFrames } from "./webCaptureFrameComparison.js"; + +const frame = ( + id: string, + parent: string | null = null, + url = "https://app.example.test/frame", +): WebPageInspection["frames"][number] => ({ + frame_id: id, + parent_frame_id: parent, + url, + origin: "https://app.example.test", +}); + +describe("capture-local frame graph normalization", () => { + it.each([ + [frame("child", "missing")], + [frame("same"), frame("same")], + [frame("a", "b"), frame("b", "a")], + [frame("self", "self")], + ])("keeps unresolved ancestry unknown (%#)", (...frames) => { + expect(fingerprintWebCaptureFrames(frames)).toBeNull(); + }); + + it("preserves multiplicity while ignoring sibling order and frame IDs", () => { + const left = [frame("root"), frame("a", "root"), frame("b", "root")]; + const renamed = [frame("y", "new"), frame("new"), frame("x", "new")]; + expect(fingerprintWebCaptureFrames(left)).toBe( + fingerprintWebCaptureFrames(renamed), + ); + expect(fingerprintWebCaptureFrames(left)).not.toBe( + fingerprintWebCaptureFrames(left.slice(0, 2)), + ); + expect(fingerprintWebCaptureFrames(left)).not.toBe( + fingerprintWebCaptureFrames([ + frame("root"), + frame("a", "root"), + frame("b", "a"), + ]), + ); + }); + + it("handles deep ancestry without recursive serialization", () => { + const frames = Array.from({ length: 5000 }, (_, index) => + frame(String(index), index === 0 ? null : String(index - 1)), + ); + const renamed = frames + .map((item) => ({ + ...item, + frame_id: "new-" + item.frame_id, + parent_frame_id: + item.parent_frame_id === null ? null : "new-" + item.parent_frame_id, + })) + .reverse(); + expect(fingerprintWebCaptureFrames(frames)).toMatch(/^[a-f0-9]{64}$/u); + expect(fingerprintWebCaptureFrames(frames)).toBe( + fingerprintWebCaptureFrames(renamed), + ); + }); +}); diff --git a/src/domain/webCaptureFrameComparison.ts b/src/domain/webCaptureFrameComparison.ts new file mode 100644 index 0000000000..e85f03e2c1 --- /dev/null +++ b/src/domain/webCaptureFrameComparison.ts @@ -0,0 +1,49 @@ +import type { WebPageInspection } from "./browserObservationSchemas.js"; +import { digestCanonicalValue } from "./canonicalDigest.js"; + +/** Compare complete frame forests without treating capture-local IDs as identity. */ +export const fingerprintWebCaptureFrames = ( + frames: WebPageInspection["frames"], +): string | null => { + const byId = new Map(frames.map((frame) => [frame.frame_id, frame])); + if (byId.size !== frames.length) return null; + const remaining = new Map(frames.map((frame) => [frame.frame_id, 0])); + for (const frame of frames) { + if (frame.parent_frame_id === null) continue; + const count = remaining.get(frame.parent_frame_id); + if (count === undefined) return null; + remaining.set(frame.parent_frame_id, count + 1); + } + const ready = frames.filter((frame) => remaining.get(frame.frame_id) === 0); + const children = new Map(); + const roots: string[] = []; + let examined = 0; + for (let index = 0; index < ready.length; index += 1) { + const frame = ready[index]; + if (frame === undefined) throw new Error("Missing ready frame"); + const fingerprint = digestCanonicalValue({ + url: frame.url, + origin: frame.origin, + children: (children.get(frame.frame_id) ?? []).sort(), + }); + examined += 1; + const parentId = frame.parent_frame_id; + if (parentId === null) { + roots.push(fingerprint); + continue; + } + const siblings = children.get(parentId) ?? []; + siblings.push(fingerprint); + children.set(parentId, siblings); + const count = (remaining.get(parentId) ?? 0) - 1; + remaining.set(parentId, count); + if (count === 0) { + const parent = byId.get(parentId); + if (parent === undefined) + throw new Error("Missing admitted parent frame"); + ready.push(parent); + } + } + // Cyclic ancestry never reaches the leaf queue and cannot prove equality. + return examined === frames.length ? digestCanonicalValue(roots.sort()) : null; +}; diff --git a/tests/boundary/browser/cdpDomComparisonCompleteness.test.ts b/tests/boundary/browser/cdpDomComparisonCompleteness.test.ts new file mode 100644 index 0000000000..d8c5386c32 --- /dev/null +++ b/tests/boundary/browser/cdpDomComparisonCompleteness.test.ts @@ -0,0 +1,266 @@ +import { describe, expect, it } from "vitest"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { CdpBrowserProvider } from "../../../src/browser/CdpBrowserProvider.js"; +import { inspectWebPageInputSchema } from "../../../src/domain/browserObservation.js"; +import type { WebPageInspection } from "../../../src/domain/browserObservationSchemas.js"; +import { compareWebCaptures } from "../../../src/domain/webCaptureDiff.js"; +import { compareWebCapturesInputSchema } from "../../../src/domain/webCaptureDiffSchemas.js"; +import { startFakeCdpBrowser } from "../../fixtures/fakeCdpBrowser.js"; +import { trackBrowser } from "./cdpBrowserProvider.support.js"; +import { runCliJson } from "../../fixtures/cliJsonProcess.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +type SnapshotMode = + | "valid" + | "omitted-node" + | "changed-node" + | "missing-nodes" + | "null-nodes" + | "missing-documents" + | "invalid-documents" + | "empty-documents" + | "invalid-document" + | "mixed-documents" + | "missing-strings"; + +const snapshot = (origin: string, mode: SnapshotMode) => { + const document = { + documentURL: 0, + nodes: { + nodeType: [9, 1], + nodeName: [1, mode === "changed-node" ? 6 : 3], + nodeValue: [2, 2], + parentIndex: [-1, 0], + attributes: [[], mode === "omitted-node" ? null : [4, 5]], + }, + }; + let documents: unknown = [document]; + if (mode === "missing-nodes") documents = [{ documentURL: 0 }]; + if (mode === "null-nodes") documents = [{ documentURL: 0, nodes: null }]; + if (mode === "missing-documents") documents = undefined; + if (mode === "invalid-documents") documents = {}; + if (mode === "empty-documents") documents = []; + if (mode === "invalid-document") documents = [null]; + if (mode === "mixed-documents") documents = [document, null]; + return { + strings: + mode === "missing-strings" + ? undefined + : [origin + "/app", "#document", "", "A", "href", "/target", "SPAN"], + documents, + }; +}; + +const startBrowser = async () => { + const state = { + mode: "valid" as SnapshotMode, + nested: false, + renamed: false, + parentOverride: undefined as { value: unknown } | undefined, + }; + const browser = await startFakeCdpBrowser({ + commandResult(command, origin) { + if (command.method === "DOMSnapshot.captureSnapshot") + return snapshot(origin, state.mode); + if (command.method !== "Page.getFrameTree") return undefined; + const id = (name: string) => (state.renamed ? "new-" + name : name); + const child = { + frame: { + id: id("b"), + parentId: + state.parentOverride === undefined + ? id(state.nested ? "a" : "main") + : state.parentOverride.value, + loaderId: "loader-b", + url: origin + "/b", + }, + }; + const parent = { + frame: { + id: id("a"), + parentId: id("main"), + loaderId: "loader-a", + url: origin + "/a", + }, + childFrames: state.nested ? [child] : [], + }; + return { + frameTree: { + frame: { + id: id("main"), + loaderId: "loader-main", + url: origin + "/app", + }, + childFrames: state.nested ? [parent] : [parent, child], + }, + }; + }, + }); + trackBrowser(browser); + const inspect = async () => { + const result = await new CdpBrowserProvider().inspectPage( + inspectWebPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 0, + }), + ); + if (!result.ok) throw result.error; + return result.value; + }; + return { state, inspect }; +}; + +const compare = (before: WebPageInspection, after: WebPageInspection) => + compareWebCaptures( + compareWebCapturesInputSchema.parse({ + before: { inspection: before }, + after: { inspection: after }, + }), + ); + +describe("DOM protocol omissions at the production capture boundary", () => { + it.each([ + "missing-nodes", + "null-nodes", + "missing-documents", + "invalid-documents", + "empty-documents", + "invalid-document", + "mixed-documents", + "missing-strings", + ] as const)("retains unknown coverage for %s", async (mode) => { + const { state, inspect } = await startBrowser(); + state.mode = mode; + const capture = await inspect(); + for (const section of ["dom", "metadata"]) + expect(capture.completeness.unavailable_sections).toContain(section); + if (mode === "mixed-documents") { + expect(capture.dom.nodes.map(({ node_name }) => node_name)).toEqual([ + "#document", + "A", + ]); + expect(capture.metadata.dom_urls).toHaveLength(1); + } + const comparison = compare(capture, capture); + expect(comparison.dimensions.dom_structure.status).toBe("unknown"); + expect(comparison.dimensions.metadata.status).toBe("unknown"); + }); + + it("does not turn a rejected node and its URL into a page change", async () => { + const { state, inspect } = await startBrowser(); + const valid = await inspect(); + state.mode = "omitted-node"; + const partial = await inspect(); + expect(partial.dom.total_nodes).toBe(2); + expect(partial.dom.nodes).toHaveLength(1); + for (const [before, after] of [ + [valid, partial], + [partial, valid], + ]) { + if (before === undefined || after === undefined) + throw new Error("Missing pair"); + const comparison = compare(before, after); + expect(comparison.dimensions.dom_structure.status).toBe("unknown"); + expect(comparison.dimensions.metadata.status).toBe("unknown"); + expect(comparison.overall_status).toBe("unknown"); + } + }); +}); + +describe("complete frame and DOM comparison", () => { + it.each([undefined, null, 17, ""])( + "keeps missing or invalid child parent evidence unknown (%#)", + async (parent) => { + const { state, inspect } = await startBrowser(); + const before = await inspect(); + state.parentOverride = { value: parent }; + const after = await inspect(); + expect(after.frames).toHaveLength(3); + expect(after.completeness.unavailable_sections).toContain("frames"); + expect(compare(before, after).dimensions.dom_structure.status).toBe( + "unknown", + ); + expect(compare(after, after).dimensions.dom_structure.status).toBe( + "unknown", + ); + }, + ); + it("retains a complete observed node change", async () => { + const { state, inspect } = await startBrowser(); + const before = await inspect(); + expect(compare(before, before).dimensions.dom_structure.status).toBe( + "unchanged", + ); + state.mode = "changed-node"; + expect( + compare(before, await inspect()).dimensions.dom_structure.status, + ).toBe("changed"); + }); + + it("compares frame ancestry independently of volatile IDs", async () => { + const { state, inspect } = await startBrowser(); + const before = await inspect(); + state.renamed = true; + expect( + compare(before, await inspect()).dimensions.dom_structure.status, + ).toBe("unchanged"); + state.nested = true; + const nested = await inspect(); + expect(nested.frames[2]?.parent_frame_id).toBe("new-a"); + expect(compare(before, nested).dimensions.dom_structure.status).toBe( + "changed", + ); + }); + + it("retains proven frame changes when DOM node evidence is incomplete", async () => { + const { state, inspect } = await startBrowser(); + const before = await inspect(); + state.mode = "omitted-node"; + state.nested = true; + const comparison = compare(before, await inspect()); + expect(comparison.dimensions.dom_structure.status).toBe("changed"); + expect(comparison.dimensions.metadata.status).toBe("unknown"); + }); +}); + +describe("DOM evidence through the public CLI", () => { + it("preserves omissions and independently proven frame changes", async () => { + const { state, inspect } = await startBrowser(); + const root = await createTestTempDirectory("rea-dom-comparison-"); + const before = join(root, "before.json"); + const after = join(root, "after.json"); + await writeFile(before, JSON.stringify({ inspection: await inspect() })); + state.mode = "omitted-node"; + for (const nested of [false, true]) { + state.nested = nested; + await writeFile(after, JSON.stringify({ inspection: await inspect() })); + expect( + await runCliJson(["compare-web-captures", before, after, "--json"]), + ).toMatchObject({ + operation: "compare_web_captures", + normalized_result: { + overall_status: nested ? "changed" : "unknown", + dimensions: { + dom_structure: { status: nested ? "changed" : "unknown" }, + metadata: { status: "unknown" }, + }, + }, + }); + } + state.mode = "missing-nodes"; + await writeFile(after, JSON.stringify({ inspection: await inspect() })); + expect( + await runCliJson(["compare-web-captures", after, after, "--json"]), + ).toMatchObject({ + operation: "compare_web_captures", + normalized_result: { + overall_status: "unknown", + dimensions: { dom_structure: { status: "unknown" } }, + }, + }); + }); +}); diff --git a/tests/boundary/browser/cdpResourceCompleteness.test.ts b/tests/boundary/browser/cdpResourceCompleteness.test.ts new file mode 100644 index 0000000000..d40e838cc2 --- /dev/null +++ b/tests/boundary/browser/cdpResourceCompleteness.test.ts @@ -0,0 +1,322 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; + +import { CdpBrowserProvider } from "../../../src/browser/CdpBrowserProvider.js"; +import { inspectWebPageInputSchema } from "../../../src/domain/browserObservation.js"; +import type { WebPageInspection } from "../../../src/domain/browserObservationSchemas.js"; +import { compareWebCaptures } from "../../../src/domain/webCaptureDiff.js"; +import { compareWebCapturesInputSchema } from "../../../src/domain/webCaptureDiffSchemas.js"; +import { startFakeCdpBrowser } from "../../fixtures/fakeCdpBrowser.js"; +import { runCliJson } from "../../fixtures/cliJsonProcess.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { trackBrowser } from "./cdpBrowserProvider.support.js"; + +type ResourceMode = + | "valid" + | "missing" + | "null" + | "wrong-type" + | "invalid-entry" + | "child-missing" + | "child-invalid-entry" + | "child-wrong-type" + | "empty"; + +const startBrowser = async () => { + const state = { + mode: "valid" as ResourceMode, + peerFields: undefined as Record | undefined, + }; + const browser = await startFakeCdpBrowser({ + commandResult(command, origin) { + const malformedChildren = + state.mode === "child-invalid-entry" || + state.mode === "child-wrong-type"; + if (command.method === "Page.getFrameTree" && malformedChildren) + return { + frameTree: { + frame: { id: "frame-main", url: origin + "/app" }, + childFrames: + state.mode === "child-wrong-type" + ? {} + : [ + { + frame: { + id: "child", + parentId: "frame-main", + url: origin + "/child", + }, + }, + null, + ], + }, + }; + if (command.method !== "Page.getResourceTree") return undefined; + const resource = (name: string) => ({ + url: origin + "/" + name, + type: "Script", + mimeType: "text/javascript", + contentSize: 128, + ...(name === "peer.js" ? state.peerFields : undefined), + }); + let resources: unknown = [resource("keep.js"), resource("peer.js")]; + if (state.mode === "missing") resources = undefined; + if (state.mode === "null") resources = null; + if (state.mode === "wrong-type") resources = {}; + if (state.mode === "invalid-entry") + resources = [resource("keep.js"), null]; + if (state.mode === "empty") resources = []; + const malformedChildFrames = + state.mode === "child-wrong-type" + ? {} + : state.mode === "child-invalid-entry" + ? [ + { + frame: { + id: "child", + parentId: "frame-main", + url: origin + "/child", + }, + resources: [resource("child.js")], + }, + null, + ] + : undefined; + return { + frameTree: { + frame: { id: "frame-main", url: origin + "/app" }, + resources, + ...(malformedChildFrames === undefined + ? {} + : { childFrames: malformedChildFrames }), + ...(state.mode === "child-missing" + ? { + childFrames: [ + { + frame: { + id: "child", + parentId: "frame-main", + url: origin + "/child", + }, + }, + ], + } + : {}), + }, + }; + }, + }); + trackBrowser(browser); + const inspect = async () => { + const result = await new CdpBrowserProvider().inspectPage( + inspectWebPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 0, + }), + ); + if (!result.ok) throw result.error; + return result.value; + }; + return { state, inspect, origin: browser.allowedOrigin }; +}; + +const compareResources = ( + before: WebPageInspection, + after: WebPageInspection, +) => + compareWebCaptures( + compareWebCapturesInputSchema.parse({ + before: { inspection: before }, + after: { inspection: after }, + }), + ).dimensions.resources; + +describe("resource inventory omissions through CDP", () => { + it.each([ + { fields: { type: undefined }, retained: false }, + { fields: { type: null }, retained: false }, + { fields: { type: 17 }, retained: false }, + { fields: { type: "" }, retained: false }, + { fields: { mimeType: undefined }, retained: false }, + { fields: { mimeType: null }, retained: false }, + { fields: { mimeType: 17 }, retained: false }, + { fields: { contentSize: -1 }, retained: true }, + { fields: { contentSize: null }, retained: true }, + { fields: { contentSize: "128" }, retained: true }, + ])( + "keeps malformed resource fields uncertain: $fields", + async ({ fields, retained }) => { + const { state, inspect, origin } = await startBrowser(); + const before = await inspect(); + state.peerFields = fields; + const after = await inspect(); + for (const [left, right] of [ + [before, after], + [after, before], + [after, after], + ] as const) + expect(compareResources(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + expect(after.completeness.excluded).toContainEqual({ + section: "resources", + reason: "invalid_protocol_value", + count: 1, + }); + expect(after.resources.map(({ url }) => url).sort()).toEqual( + retained + ? [origin + "/keep.js", origin + "/peer.js"] + : [origin + "/keep.js"], + ); + if (retained) + expect( + after.resources.find(({ url }) => url.endsWith("/peer.js")), + ).toMatchObject({ + type: "Script", + mime_type: "text/javascript", + content_size: null, + }); + }, + ); + + it.each([ + { + fields: { type: "Other", mimeType: "", contentSize: undefined }, + size: null, + }, + { fields: { contentSize: 0 }, size: 0 }, + { fields: { contentSize: 1.5 }, size: 1.5 }, + ])( + "accepts valid optional and empty fields: $fields", + async ({ fields, size }) => { + const { state, inspect } = await startBrowser(); + state.peerFields = fields; + const capture = await inspect(); + expect(capture.resources).toHaveLength(2); + expect( + capture.resources.find(({ url }) => url.endsWith("/peer.js")) + ?.content_size, + ).toBe(size); + expect(capture.completeness.unavailable_sections).not.toContain( + "resources", + ); + expect(compareResources(capture, capture).status).toBe("unchanged"); + }, + ); + + it.each([ + "missing", + "null", + "wrong-type", + "invalid-entry", + "child-missing", + "child-invalid-entry", + "child-wrong-type", + ] as const)( + "does not claim a resource removal or equality for %s", + async (mode) => { + const { state, inspect, origin } = await startBrowser(); + const before = await inspect(); + expect(before.resources).toHaveLength(2); + expect(before.completeness.unavailable_sections).not.toContain( + "resources", + ); + state.mode = mode; + const after = await inspect(); + for (const [left, right] of [ + [before, after], + [after, before], + [after, after], + ] as const) + expect(compareResources(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + expect(after.completeness.unavailable_sections).toContain("resources"); + expect(after.completeness.excluded).toContainEqual({ + section: "resources", + reason: "invalid_protocol_value", + count: mode === "invalid-entry" ? 1 : null, + }); + if (mode === "child-invalid-entry" || mode === "child-wrong-type") { + expect(after.completeness.excluded).toContainEqual({ + section: "frames", + reason: "invalid_protocol_value", + count: null, + }); + expect(after.frames.map(({ frame_id }) => frame_id)).toContain( + "frame-main", + ); + if (mode === "child-invalid-entry") + expect(after.frames.map(({ frame_id }) => frame_id)).toContain( + "child", + ); + } + const expected = + mode === "invalid-entry" + ? [origin + "/keep.js"] + : mode === "child-invalid-entry" + ? [origin + "/child.js", origin + "/keep.js", origin + "/peer.js"] + : mode === "child-missing" + ? [origin + "/keep.js", origin + "/peer.js"] + : mode === "child-wrong-type" + ? [origin + "/keep.js", origin + "/peer.js"] + : []; + expect(after.resources.map(({ url }) => url).sort()).toEqual(expected); + }, + ); + + it("distinguishes a complete empty inventory from an unavailable collection", async () => { + const { state, inspect } = await startBrowser(); + const before = await inspect(); + state.mode = "empty"; + const after = await inspect(); + expect(after.resources).toEqual([]); + expect(after.completeness.unavailable_sections).not.toContain("resources"); + expect(compareResources(before, after)).toMatchObject({ + status: "changed", + total_changes: 2, + changes: [{ change: "removed" }, { change: "removed" }], + }); + expect(compareResources(after, after).status).toBe("unchanged"); + }); +}); + +describe("resource uncertainty in public CLI Evidence", () => { + it("preserves partial membership while complete inventories still prove removal", async () => { + const { state, inspect } = await startBrowser(); + const root = await createTestTempDirectory("rea-resource-comparison-"); + const before = join(root, "before.json"); + const after = join(root, "after.json"); + await writeFile(before, JSON.stringify({ inspection: await inspect() })); + for (const { mode, fields } of [ + { mode: "invalid-entry", fields: undefined }, + { mode: "valid", fields: { type: undefined } }, + { mode: "valid", fields: { contentSize: -1 } }, + { mode: "empty", fields: undefined }, + ] as const) { + state.mode = mode; + state.peerFields = fields; + await writeFile(after, JSON.stringify({ inspection: await inspect() })); + expect( + await runCliJson(["compare-web-captures", before, after, "--json"]), + ).toMatchObject({ + operation: "compare_web_captures", + normalized_result: { + dimensions: { + resources: { + status: mode === "empty" ? "changed" : "unknown", + total_changes: mode === "empty" ? 2 : 0, + }, + }, + }, + }); + } + }); +});