diff --git a/docs/browser-observation.md b/docs/browser-observation.md index 9c52ae8779..0bfbb2789f 100644 --- a/docs/browser-observation.md +++ b/docs/browser-observation.md @@ -281,6 +281,32 @@ REA retains local observation data and removes only URL userinfo credentials, ex - Accessibility structure and roles are retained by default, while names and descriptions require `include_accessibility_text: true`. - Network observations retain method, status, MIME type, size, type, initiator stack location, and complete URLs with userinfo credentials removed. Redirects reported by CDP are attached as ordered `redirects` on the one request ID, with the prior request URL, CDP response URL, method/type, response status/MIME/encoded length, request timestamp, and the next `requestWillBeSent` event timestamp. CDP delivers `redirectResponse` with that next request event, so the latter is an event boundary rather than an exact response timestamp. Redirect response headers and bodies are discarded; hops outside the selected origins are discarded with the request chain. Headers are discarded after an allowlisted projection of length/encoding, structured CSP/Link/policy fields, and untrusted agent hints. Cookies and authorization headers are never retained. - Request/response bodies are not requested or parsed by default. When selected, only allowed-origin JSON media types are read, converted immediately to complete value-free typed paths, types and observation counts, and discarded. Each path segment is either `{ "kind": "property", "name": "raw property name" }` or `{ "kind": "array-element" }`; array-element types include primitives and nested arrays. Literal `*` properties remain distinct from array elements. `Network.getResponseBody` is never sent unless JSON body shape capture is selected. Comparison uses body shapes only when both captures selected them, and compares request/response shapes only where both observations provide those facts. Unavailable or truncated selected coverage makes equivalence unknown; observed differences still report changed. + For normalized passive inputs with complete request inventories on both sides, + request additions and removals remain observable even when selected body shapes, + response metadata, or WebSocket coverage are unavailable. HTTP request and + WebSocket inventory coverage are evaluated separately; complete HTTP coverage + cannot prove a missing WebSocket connection. Limited request inventory still + prevents request-absence claims. This comparison rule does not make live CDP's + attach-limited inventory complete or establish body-shape equality. + An unavailable HTTP resource type also leaves request identity incomplete: + observations with the same method and URL remain possible matches across types, + and identical unknown types cannot establish complete equality. Ambiguous type + groups cannot prove membership or content changes either: a missing type may + redistribute repeated requests between buckets without changing the requests. + Their comparison remains unknown, while independently identified groups retain + proven changes. Different methods/URLs and complete empty peer inventories still + support absence claims. + Relevant dropped-event counts also limit inventory completeness even when no + section is listed as truncated: HTTP uses `network_requests`, while WebSocket + uses `websocket_connections` and `websocket_frames`. Drops in other event kinds + and the aggregate total do not by themselves limit HTTP inventory. Proven + modifications remain reportable despite incomplete inventory. + Repeated identical HTTP content samples establish multiplicity changes only + with complete request inventories; limited inventories retain known content + comparisons without interpreting duplicate sample counts as changes. + WebSocket stream inventories and connection counts likewise require complete + WebSocket coverage. Redirect resource types use only shared known metadata: + an omitted type leaves equality unknown rather than proving a modification. - Console observations with a stack source retain call type, argument types, timestamp, and source location. When console text capture is selected, already-delivered primitive values are retained verbatim; objects, getters, and remote properties are never expanded. - WebSocket observations retain direction, opcode, and payload byte length. When shape capture is selected, text frames are classified as text or complete value-free JSON shape; binary bytes, hashes, prefixes, and raw frames are never retained. - Storage observations always redact values. Key names, IndexedDB names, and cache names require `include_storage_keys`; stable content fingerprints require the additional `include_storage_fingerprints` selection. Cookie fingerprints cover cookies applicable to the current main-frame URL, including its path, rather than every cookie path on that origin. Other storage remains scoped to the origin. Cache bodies above 64 KiB and partial IndexedDB remote objects make fingerprint coverage incomplete, so identical observations remain `unknown`. diff --git a/docs/javascript-application-workflows.md b/docs/javascript-application-workflows.md index 9c5475737f..6d47fc0957 100644 --- a/docs/javascript-application-workflows.md +++ b/docs/javascript-application-workflows.md @@ -83,6 +83,11 @@ callee bodies and built-ins such as `Object.assign`, `Reflect.set`, and values. This uncertainty does not prove that a call actually mutated a value. Interprocedural aliasing beyond the supported call-site references remains outside this mutation pass. +For process configuration, only known environment value slots and canonical +argument indexes are treated as primitive when passed to calls. Unknown or +inherited selections can expose a mutable prototype, so later reads of that +configuration container retain uncertainty; earlier top-level reads and unrelated +configuration containers remain independent. Function fingerprints commit normalized syntax, control-flow shape, relation shape, literal sets, arity, and detected effects without using local names or diff --git a/docs/mcp-contracts.md b/docs/mcp-contracts.md index 8df9845001..02d048014c 100644 --- a/docs/mcp-contracts.md +++ b/docs/mcp-contracts.md @@ -664,10 +664,14 @@ advertised. Every advertised example is a distinct valid canonical input. Root object unions are presented as one object without root `anyOf`, `oneOf`, or `allOf` for model API compatibility. -The projection merges properties, retains shared required fields and -group-specific field dependencies, and describes the accepted input groups. -Canonical runtime validation enforces the exact union, including exclusions -that the advertised projection does not express. Tests validate the schemas +The projection merges properties, retains shared required fields, and +describes the accepted input groups. Which fields belong together is guidance +in that description rather than a `dependentRequired` or `dependencies` +keyword, because Gemini function declarations reject both and one unknown +keyword fails the client's whole tool catalog. Canonical runtime validation +enforces the exact union, including partner fields and exclusions that the +advertised projection does not express, so a client must not rely on the +advertised schema to reject an incomplete group. Tests validate the schemas after SDK conversion and check advertised validation and actual calls. Individual model APIs can impose additional nesting limits; complete producer captures can exceed ten structural levels. @@ -773,9 +777,10 @@ identities remains unknown. `normalization` defaults to `{ "rules": [] }`. The advertised schema includes complete nested capture fields and rejects -incomplete pairs and structurally malformed captures. Mixed comparison families -can pass advertised validation; the SDK rejects them against the canonical -input schema before invoking the handler. Domain validation additionally checks +incomplete pairs and structurally malformed captures. Mixed comparison families, +and `normalization` without both scenario captures, can pass advertised +validation; the SDK rejects them against the canonical input schema before +invoking the handler. Domain validation additionally checks relationships such as event sequence references and retained counts; JSON Schema does not express those cross-field invariants. diff --git a/docs/web-network-captures.md b/docs/web-network-captures.md index 6e29731fd0..9cbec8747a 100644 --- a/docs/web-network-captures.md +++ b/docs/web-network-captures.md @@ -117,7 +117,11 @@ gRPC message bytes. Missing members remain null, distinct from empty bodies. See Telerik's [SAZ format reference](https://www.telerik.com/fiddler/fiddler-everywhere/documentation/knowledge-base/fiddler-archives). The existing ZIP reader verifies read members; expanded archive contents are -limited to 32 MiB in addition to the input/output budgets below. Encrypted +limited to 32 MiB in addition to the input/output budgets below. Exceeding this +expanded-byte budget returns `invalid_request` / `invalid_input` through both +CLI and MCP, with an `out_of_range` issue at +`capture_path` and `maximum_expanded_capture_bytes`. Selecting records does not +bypass the complete-archive admission check or return partial Evidence. Encrypted archives and unrecognized raw session members are unsupported. SAZ requires an enforced decoder memory bound: Node's fixed heap, or the Windows Job Object commit limit. Bun on hosts without that Job Object limit returns capability diff --git a/src/browser/CdpCaptureDocuments.test.ts b/src/browser/CdpCaptureDocuments.test.ts index d1e20c2a07..5993e7e27f 100644 --- a/src/browser/CdpCaptureDocuments.test.ts +++ b/src/browser/CdpCaptureDocuments.test.ts @@ -130,10 +130,12 @@ describe("CDP DOM string and namespace validity", () => { { field: "nodeName", index: -1 }, { field: "nodeName", index: 20 }, { field: "nodeName", index: 2, malformedString: true }, + { field: "nodeValue", index: -2 }, { field: "nodeValue", index: 20 }, { field: "nodeValue", index: 5, malformedString: true }, { field: "attributeName", index: -1 }, { field: "attributeName", index: 20 }, + { field: "attributeValue", index: -2 }, { field: "attributeValue", index: 20 }, ] as const)( // -1 is DOMSnapshot's empty string for values, never for names. diff --git a/src/contracts/inputSchemaPresentation.ts b/src/contracts/inputSchemaPresentation.ts index 3ed83907a1..740f1f3fae 100644 --- a/src/contracts/inputSchemaPresentation.ts +++ b/src/contracts/inputSchemaPresentation.ts @@ -172,8 +172,8 @@ const flattenRootObjectUnion = ( variants.set(name, known); } const required = sharedRequiredProperties(branches); - // A field unique to a group still needs that group's required partners. - // Counting properties alone would let optional fields pad an incomplete pair. + // A coarse property minimum rejects underspecified objects. Partner fields + // and exact group membership remain enforced by the canonical runtime parser. const fewestRequired = Math.min( ...branches.map((branch) => requiredProperties(branch).length), ); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.native_bindings.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.native_bindings.test.ts index 3058533014..6083e2bfe7 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.native_bindings.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.native_bindings.test.ts @@ -225,6 +225,29 @@ describe("JavaScript semantic analysis: configuration value escapes", () => { expect(configurationReads(body)).toEqual([[kind, key]]); }); + it.each([ + "patch(process.argv[key]); return process.argv[3];", + "patch(process.argv.__proto__); return process.argv[3];", + "const args = process.argv; patch(args[key]); return args[3];", + ])( + "retains uncertainty after an object-valued argv selection escapes: %s", + (body) => { + expect(configurationReads(body)).not.toContainEqual(["argv", "3"]); + }, + ); + + it("preserves an earlier top-level read and unrelated configuration after an argv escape", () => { + const captured = analyzeJavaScriptSemantics( + "const saved = process.argv[3]; patch(process.argv[key]); globalThis.saved = saved;", + ); + expect( + captured.configurationOperations.map(({ kind, key }) => [kind, key]), + ).toContainEqual(["argv", "3"]); + expect( + configurationReads("patch(process.argv[key]); return process.env.PORT;"), + ).toContainEqual(["environment", "PORT"]); + }); + it.each([ [ "a direct argv entry", diff --git a/src/domain/javascript/javascriptSemanticMemberMutations.ts b/src/domain/javascript/javascriptSemanticMemberMutations.ts index 34666e65c7..24535f91eb 100644 --- a/src/domain/javascript/javascriptSemanticMemberMutations.ts +++ b/src/domain/javascript/javascriptSemanticMemberMutations.ts @@ -69,21 +69,30 @@ export function* collectSemanticMemberMutationsSteps( node: t.Node | undefined, ): void => { const identity = semanticRequireExportsIdentity(specifier, path); - // Node stores process.env values and process.argv entries as strings, so - // passing one to a call exposes no object that the callee could mutate. - // Replacing an entry records its own write or receiver escape. + let effectPath = identity.path; + // Only known value slots are primitive. Unknown or inherited selections + // can expose the container prototype and affect any subsequently read slot. if ( effect === "escape" && identity.specifier === "process" && - identity.path.length === 2 && - (identity.path[0] === "env" || identity.path[0] === "argv") - ) - return; + effectPath.length === 2 && + (effectPath[0] === "env" || effectPath[0] === "argv") + ) { + const selected = effectPath[1]; + if ( + (typeof selected === "string" || typeof selected === "number") && + (effectPath[0] === "env" + ? !Object.hasOwn(Object.prototype, selected) + : selected === "length" || semanticArrayIndex(selected) !== null) + ) + return; + effectPath = [effectPath[0], null]; + } const key = JSON.stringify([ "required", identity.specifier, effect, - identity.path, + effectPath, node?.start, node?.end, ]); @@ -91,7 +100,7 @@ export function* collectSemanticMemberMutationsSteps( recordedRequiredEffects.add(key); recordedEffectCount += 1; const effects = state.requiredModuleEffects.get(identity.specifier) ?? []; - effects.push({ path: identity.path, node, kind: effect }); + effects.push({ path: effectPath, node, kind: effect }); state.requiredModuleEffects.set(identity.specifier, effects); }; // Ambient indexing owns aliases of global objects. Consume its canonical diff --git a/src/domain/jsonValue.test.ts b/src/domain/jsonValue.test.ts index f27dc058ed..faca71f535 100644 --- a/src/domain/jsonValue.test.ts +++ b/src/domain/jsonValue.test.ts @@ -82,6 +82,18 @@ describe("jsonValueSchema depth bound", () => { }, ); + it("stops enclosing refinements before rejected JSON reaches consumers", () => { + let consumed = 0; + const envelope = z.object({ value: jsonValueSchema }).superRefine(() => { + consumed += 1; + }); + const result = envelope.safeParse({ value: deepObject(10_000) }); + expect(result.success).toBe(false); + if (result.success) return; + expect(result.error.issues[0]?.message).toContain("maximum nesting depth"); + expect(consumed).toBe(0); + }); + it("rejects cyclic values instead of recursing forever", () => { const cyclic: Record = {}; cyclic.self = cyclic; diff --git a/src/domain/webCaptureDiff.network.test.ts b/src/domain/webCaptureDiff.network.test.ts new file mode 100644 index 0000000000..cf1640a1c0 --- /dev/null +++ b/src/domain/webCaptureDiff.network.test.ts @@ -0,0 +1,679 @@ +import { describe, expect, it } from "vitest"; + +import { + passiveNetworkCapture, + withPassiveDroppedEvents, +} from "../../tests/fixtures/passiveNetworkCapture.js"; +import { + browserCaptureComparisonInputSchema, + compareBrowserCaptures, +} from "./browserCaptureComparison.js"; +import type { WebPageInspection } from "./browserObservationSchemas.js"; +import { webCaptureDiffSchema } from "./webCaptureDiffSchemas.js"; + +const compare = (before: WebPageInspection, after: WebPageInspection) => + webCaptureDiffSchema.parse( + compareBrowserCaptures( + browserCaptureComparisonInputSchema.parse({ + before: { inspection: before }, + after: { inspection: after }, + }), + ), + ).dimensions.network; + +const unselectedShapes = (capture: WebPageInspection): WebPageInspection => { + capture.completeness.excluded.push({ + section: "json_body_shapes", + reason: "not_approved", + count: null, + }); + for (const request of capture.network.requests) + request.body_shapes.status = "not_approved"; + return capture; +}; + +const redirectCapture = (resourceType: string | null): WebPageInspection => { + const capture = unselectedShapes(passiveNetworkCapture()); + capture.network.requests[0]!.redirects = [ + { + url: "https://app.example.test/start", + response_url: "https://app.example.test/start", + method: "GET", + resource_type: resourceType, + status: 302, + mime_type: "text/html", + encoded_data_length: 20, + request_timestamp: 1, + redirect_event_timestamp: 2, + }, + ]; + return capture; +}; + +describe("nullable redirect resource types", () => { + it.each([true, false])( + "retains uncertainty when a redirect type is omitted (reversed=%s)", + (reversed) => { + const before = redirectCapture("Document"); + const after = structuredClone(before); + after.network.requests[0]!.redirects![0]!.resource_type = null; + const [left, right] = reversed + ? ([after, before] as const) + : ([before, after] as const); + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + }, + ); + + it.each(["Document", "Fetch"])( + "keeps a known redirect type %s comparable", + (resourceType) => { + const before = redirectCapture("Document"); + const after = structuredClone(before); + after.network.requests[0]!.redirects![0]!.resource_type = resourceType; + expect(compare(before, after)).toMatchObject({ + status: resourceType === "Document" ? "unchanged" : "changed", + total_changes: resourceType === "Document" ? 0 : 1, + }); + }, + ); + + it("retains proven HTTP membership with incomplete redirect metadata", () => { + const before = unselectedShapes(passiveNetworkCapture([])); + const after = redirectCapture(null); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "added" }], + }); + }); +}); + +describe("passive network inventory changes", () => { + it.each(["websocket_connections", "websocket_frames"] as const)( + "does not infer duplicate WebSocket inventory changes from dropped %s", + (kind) => { + const before = unselectedShapes(passiveNetworkCapture()); + before.network.websocket_connections = [ + { + request_id: "ws-one", + url: "wss://app.example.test/live", + events: [], + }, + ]; + const after = structuredClone(before); + after.network.websocket_connections.push({ + request_id: "ws-two", + url: "wss://app.example.test/live", + events: [], + }); + const limited = withPassiveDroppedEvents(before, { [kind]: 1 }); + for (const [left, right] of [ + [limited, after], + [after, limited], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "modified" }], + }); + }, + ); + + it.each(["dropped", "truncated", "unavailable"] as const)( + "does not infer repeated-request count changes from %s HTTP inventory", + (limitation) => { + const after = unselectedShapes(passiveNetworkCapture(["/a", "/a"])); + let before = unselectedShapes(passiveNetworkCapture()); + if (limitation === "dropped") + before = withPassiveDroppedEvents(before, { network_requests: 1 }); + else if (limitation === "truncated") + before.completeness.truncated_sections.push("network_requests"); + else before.completeness.unavailable_sections.push("network_requests"); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + }, + ); + + it.each(["unavailable", "partial", "truncated"] as const)( + "retains complete request additions/removals with %s selected shapes", + (coverage) => { + const before = passiveNetworkCapture(["/a"]); + const after = passiveNetworkCapture(["/a", "/b"]); + for (const capture of [before, after]) { + if (coverage === "truncated") + capture.completeness.truncated_sections.push("json_body_shapes"); + else capture.completeness.unavailable_sections.push("json_body_shapes"); + for (const request of capture.network.requests) { + const shape = { + root_type: "object" as const, + node_count: 1, + max_depth_observed: 0, + properties: [], + }; + if (coverage !== "unavailable") request.body_shapes.request = shape; + if (coverage === "truncated") request.body_shapes.response = shape; + request.body_shapes.status = + coverage === "truncated" ? "included" : coverage; + } + } + for (const [left, right, change] of [ + [before, after, "added"], + [after, before, "removed"], + ] as const) + expect(compare(left, right)).toEqual({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^net_/u), change }], + reason: null, + }); + expect(compare(after, after)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + }, + ); + + it("keeps request absence proof independent of WebSocket inventory and response metadata", () => { + const before = passiveNetworkCapture([]); + const after = passiveNetworkCapture(["/added"]); + for (const capture of [before, after]) { + capture.completeness.attach_limited_sections.push( + "websocket_connections", + "websocket_frames", + ); + for (const request of capture.network.requests) { + request.status = null; + request.mime_type = null; + request.encoded_data_length = null; + } + } + for (const [left, right, change] of [ + [before, after, "added"], + [after, before, "removed"], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change }], + }); + expect(compare(after, after)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + }); +}); + +describe("passive network coverage uncertainty", () => { + it.each([ + "attach_limited_sections", + "unavailable_sections", + "truncated_sections", + "policy_filtered_sections", + ] as const)( + "does not infer request absence when its inventory is %s", + (section) => { + const before = passiveNetworkCapture(); + const after = passiveNetworkCapture(["/a", "/b"]); + after.completeness[section].push("network_requests"); + for (const [left, right] of [ + [before, after], + [after, before], + ]) + expect(compare(left!, right!)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + after.network.requests[0]!.status = 503; + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "modified" }], + }); + }, + ); + + it("does not promote unavailable shapes to a modification and preserves unselected-shape behavior", () => { + const before = passiveNetworkCapture(); + const after = structuredClone(before); + after.network.requests[0]!.body_shapes = { + status: "included", + request: { + root_type: "object", + node_count: 1, + max_depth_observed: 0, + properties: [], + }, + response: { + root_type: "string", + node_count: 1, + max_depth_observed: 0, + properties: [], + }, + }; + expect(compare(before, after)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + before.completeness.excluded.push({ + section: "json_body_shapes", + reason: "not_approved", + count: null, + }); + expect(compare(before, after)).toMatchObject({ + status: "unchanged", + total_changes: 0, + }); + }); +}); + +describe("independent WebSocket inventory evidence", () => { + it("does not use complete HTTP inventory to infer an unobserved WebSocket connection", () => { + const before = passiveNetworkCapture(); + const after = structuredClone(before); + after.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [], + }); + before.completeness.attach_limited_sections.push("websocket_connections"); + expect(compare(before, after)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + expect(compare(after, before)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + after.network.requests.push({ + ...passiveNetworkCapture(["/b"]).network.requests[0]!, + request_id: "request-b", + }); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^net_/u), change: "added" }], + }); + }); + + it("retains complete WebSocket membership changes when HTTP inventory is limited", () => { + const before = passiveNetworkCapture(); + before.completeness.attach_limited_sections.push("network_requests"); + const after = structuredClone(before); + after.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [], + }); + for (const [left, right, change] of [ + [before, after, "added"], + [after, before, "removed"], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^ws_/u), change }], + }); + }); + + it("does not turn missing WebSocket payload facts into membership changes", () => { + const before = passiveNetworkCapture(); + before.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [ + { + direction: "sent", + opcode: 1, + payload_bytes: 4, + payload_shape: { format: "text", json_shape: null }, + }, + ], + }); + const after = structuredClone(before); + const event = after.network.websocket_connections[0]!.events[0]!; + event.payload_bytes = null; + event.payload_shape = null; + expect(compare(before, after)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + after.network.requests.push({ + ...passiveNetworkCapture(["/b"]).network.requests[0]!, + request_id: "request-b", + }); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^net_/u), change: "added" }], + }); + }); +}); + +describe("repeated HTTP request identity uncertainty", () => { + it.each([true, false])( + "keeps repeated-request regrouping unknown when one type is missing (shapes selected=%s)", + (selected) => { + const before = passiveNetworkCapture(["/a", "/a"]); + const after = structuredClone(before); + after.network.requests[1]!.resource_type = null; + if (!selected) + for (const capture of [before, after]) unselectedShapes(capture); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + }, + ); + + it("retains a known repeated-request count change", () => { + const before = unselectedShapes(passiveNetworkCapture(["/a", "/a"])); + const after = unselectedShapes(passiveNetworkCapture()); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "modified" }], + }); + }); + + it("does not assign observed content to an ambiguous type bucket", () => { + const before = unselectedShapes(passiveNetworkCapture(["/a", "/a"])); + before.network.requests[1]!.resource_type = null; + before.network.requests[1]!.status = 503; + const after = structuredClone(before); + after.network.requests[0]!.status = 503; + after.network.requests[1]!.status = 200; + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + }); +}); + +describe("nullable HTTP request identities", () => { + it.each([true, false])( + "keeps known/unknown resource-type matches uncertain (shapes selected=%s)", + (selected) => { + const before = passiveNetworkCapture(); + const after = structuredClone(before); + after.network.requests[0]!.resource_type = null; + if (!selected) + for (const capture of [before, after]) unselectedShapes(capture); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + reason: expect.stringContaining("request identity"), + }); + }, + ); + + it("does not prove equality from two unknown request types", () => { + const capture = unselectedShapes(passiveNetworkCapture()); + capture.network.requests[0]!.resource_type = null; + expect(compare(capture, structuredClone(capture))).toMatchObject({ + status: "unknown", + total_changes: 0, + reason: expect.stringContaining("request identity"), + }); + }); + + it("does not assign an unknown type to one of multiple same-method/URL candidates", () => { + const before = unselectedShapes(passiveNetworkCapture(["/a", "/a"])); + before.network.requests[1]!.resource_type = "Fetch"; + const after = structuredClone(before); + after.network.requests[1]!.resource_type = null; + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + }); + + it.each([ + ["/b", "GET"], + ["/a", "POST"], + ])( + "retains a distinct %s %s request beside ambiguous identities", + (path, method) => { + const before = unselectedShapes(passiveNetworkCapture()); + const after = unselectedShapes(passiveNetworkCapture(["/a", path])); + after.network.requests[0]!.resource_type = null; + after.network.requests[1]!.method = method; + for (const [left, right, change] of [ + [before, after, "added"], + [after, before, "removed"], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^net_/u), change }], + }); + }, + ); + + it("uses a complete empty peer inventory to prove absence despite an unknown type", () => { + const empty = unselectedShapes(passiveNetworkCapture([])); + const observed = unselectedShapes(passiveNetworkCapture()); + observed.network.requests[0]!.resource_type = null; + for (const [left, right, change] of [ + [empty, observed, "added"], + [observed, empty, "removed"], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change }], + }); + expect(compare(empty, empty)).toMatchObject({ + status: "unchanged", + total_changes: 0, + }); + }); + + it("retains definite type changes and independently observed metadata changes", () => { + const before = unselectedShapes(passiveNetworkCapture()); + const changedType = structuredClone(before); + changedType.network.requests[0]!.resource_type = "Fetch"; + expect(compare(before, changedType)).toMatchObject({ + status: "changed", + total_changes: 2, + changes: expect.arrayContaining([ + expect.objectContaining({ change: "added" }), + expect.objectContaining({ change: "removed" }), + ]), + }); + before.network.requests.push({ + ...passiveNetworkCapture(["/b"]).network.requests[0]!, + resource_type: null, + }); + const changedStatus = structuredClone(before); + changedStatus.network.requests[0]!.status = 503; + expect(compare(before, changedStatus)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "modified" }], + }); + }); +}); + +describe("passive dropped-event inventory coverage", () => { + it("does not infer request absence from a capture with independently counted drops", () => { + const before = withPassiveDroppedEvents(passiveNetworkCapture(), { + network_requests: 1, + }); + const after = passiveNetworkCapture(["/a", "/b"]); + expect(before.completeness).toMatchObject({ + status: "truncated", + truncated_sections: [], + unavailable_sections: [], + attach_limited_sections: [], + policy_filtered_sections: [], + dropped_events: { network_requests: 1, total: 1 }, + }); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + const changed = structuredClone(before); + changed.network.requests[0]!.status = 503; + expect(compare(before, changed)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "modified" }], + }); + }); + + it.each([ + "network_requests", + "websocket_connections", + "websocket_frames", + ] as const)( + "cannot prove network equality after dropped %s even when shapes are unselected", + (kind) => { + const before = withPassiveDroppedEvents( + unselectedShapes(passiveNetworkCapture()), + { [kind]: 1 }, + ); + expect(before.completeness.truncated_sections).toEqual([]); + expect(compare(before, structuredClone(before))).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + }, + ); + + it.each(["websocket_connections", "websocket_frames"] as const)( + "keeps %s drops separate from HTTP inventory", + (kind) => { + const before = withPassiveDroppedEvents(passiveNetworkCapture(), { + [kind]: 1, + }); + const after = structuredClone(before); + after.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [], + }); + for (const [left, right] of [ + [before, after], + [after, before], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + after.network.requests.push({ + ...passiveNetworkCapture(["/b"]).network.requests[0]!, + request_id: "request-b", + }); + for (const [left, right, change] of [ + [before, after, "added"], + [after, before, "removed"], + ] as const) + expect(compare(left, right)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^net_/u), change }], + }); + }, + ); + + it("retains complete WebSocket membership despite HTTP drops", () => { + const before = withPassiveDroppedEvents(passiveNetworkCapture(), { + network_requests: 1, + }); + const after = structuredClone(before); + after.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [], + }); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ identity: expect.stringMatching(/^ws_/u), change: "added" }], + }); + expect(compare(after, before)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [ + { identity: expect.stringMatching(/^ws_/u), change: "removed" }, + ], + }); + }); + + it.each([ + "console_events", + "scripts", + "timeline_events", + "webmcp_tools", + ] as const)( + "does not use the total or unrelated %s drops as HTTP absence limits", + (kind) => { + const before = withPassiveDroppedEvents( + unselectedShapes(passiveNetworkCapture()), + { [kind]: 2 }, + ); + expect(before.completeness).toMatchObject({ + status: "truncated", + dropped_events: { total: 2 }, + }); + expect(compare(before, structuredClone(before))).toMatchObject({ + status: "unchanged", + total_changes: 0, + }); + const after = unselectedShapes(passiveNetworkCapture(["/a", "/b"])); + expect(compare(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "added" }], + }); + expect(compare(after, before)).toMatchObject({ + status: "changed", + total_changes: 1, + changes: [{ change: "removed" }], + }); + }, + ); +}); diff --git a/src/domain/webCaptureDiff.ts b/src/domain/webCaptureDiff.ts index da4533690f..de24a0fc3d 100644 --- a/src/domain/webCaptureDiff.ts +++ b/src/domain/webCaptureDiff.ts @@ -74,11 +74,15 @@ const compareDimension = ( after: ReadonlyMap, complete: boolean, reason: string, - inventoryComplete = complete, + inventoryComplete: boolean | ((identity: string) => boolean) = complete, ): Dimension => { // Membership evidence can be complete even when selected content is unknown. const all = compareIdentities(before, after).filter( - ({ change }) => inventoryComplete || change === "modified", + ({ identity, change }) => + change === "modified" || + (typeof inventoryComplete === "boolean" + ? inventoryComplete + : inventoryComplete(identity)), ); if (all.length > 0) return { @@ -246,60 +250,145 @@ const networkIdentity = ( ): string => `net_${digestCanonicalValue({ method: request.method, url: request.url, resource_type: request.resource_type })}`; +/** An unknown request type can match any observed type at the same method/URL. */ +const networkIdentityCoverage = ( + before: WebPageInspection, + after: WebPageInspection, +): { + readonly complete: boolean; + readonly ambiguousIdentities: ReadonlySet; +} => { + const complete = [before, after].every((inspection) => + inspection.network.requests.every( + ({ resource_type }) => resource_type !== null, + ), + ); + const ambiguousIdentities = new Set(); + if (complete) return { complete, ambiguousIdentities }; + type Request = WebPageInspection["network"]["requests"][number]; + const matchKey = (request: Request): string => + JSON.stringify([request.method, request.url]); + const unknownTypes = ( + inspection: WebPageInspection, + ): ReadonlyMap => { + const groups = new Map(); + for (const request of inspection.network.requests) { + const key = matchKey(request); + groups.set( + key, + groups.get(key) === true || request.resource_type === null, + ); + } + return groups; + }; + const retainPotentialMatches = ( + inspection: WebPageInspection, + peer: ReadonlyMap, + ): void => { + for (const request of inspection.network.requests) { + const peerUnknown = peer.get(matchKey(request)); + if ( + peerUnknown === true || + (peerUnknown === false && request.resource_type === null) + ) + ambiguousIdentities.add(networkIdentity(request)); + } + }; + retainPotentialMatches(before, unknownTypes(after)); + retainPotentialMatches(after, unknownTypes(before)); + return { complete, ambiguousIdentities }; +}; + const networkResponseFields = [ "status", "mime_type", "encoded_data_length", ] as const; -type NetworkResponseField = (typeof networkResponseFields)[number]; +const redirectMetadataFields = [ + ...networkResponseFields, + "resource_type", +] as const; +type NetworkMetadataField = (typeof redirectMetadataFields)[number]; -const sharedNetworkResponseFields = ( +/** Own both comparable fields and completeness instead of deriving it from set sizes. */ +const sharedNetworkMetadata = ( before: WebPageInspection, after: WebPageInspection, -): ReadonlyMap> => { - const shared = new Map>(); +): { + readonly fields: ReadonlyMap>; + readonly complete: boolean; +} => { + const shared = new Map>(); + let complete = true; + const retainKnownFields = ( + identity: string, + metadata: Pick< + WebPageInspection["network"]["requests"][number], + NetworkMetadataField + >, + required: readonly NetworkMetadataField[], + ): void => { + const fields = shared.get(identity) ?? new Set(required); + for (const field of required) + if (metadata[field] === null) { + fields.delete(field); + complete = false; + } + shared.set(identity, fields); + }; for (const inspection of [before, after]) for (const request of inspection.network.requests) { const identity = networkIdentity(request); - retainSharedResponseFields(shared, identity, request); + retainKnownFields(identity, request, networkResponseFields); for (const [index, redirect] of (request.redirects ?? []).entries()) - retainSharedResponseFields( - shared, + retainKnownFields( redirectNetworkIdentity(identity, index), redirect, + redirectMetadataFields, ); } - return shared; -}; - -const retainSharedResponseFields = ( - shared: Map>, - identity: string, - response: Pick< - WebPageInspection["network"]["requests"][number], - NetworkResponseField - >, -): void => { - const fields = shared.get(identity) ?? new Set(networkResponseFields); - for (const field of networkResponseFields) - if (response[field] === null) fields.delete(field); - shared.set(identity, fields); + return { fields: shared, complete }; }; const redirectNetworkIdentity = (identity: string, index: number): string => compositeKey([identity, "redirect", index]); +/** Section lists and per-kind dropped counts independently limit absence evidence. */ +const networkInventoryComplete = ( + inspection: WebPageInspection, + sections: readonly ( + | "network_requests" + | "websocket_connections" + | "websocket_frames" + )[], +): boolean => + sectionsComplete(inspection, sections) && + sections.every( + (section) => inspection.completeness.dropped_events[section] === 0, + ); const networkDimension = ( before: WebPageInspection, after: WebPageInspection, ): Dimension => { - const responseFields = sharedNetworkResponseFields(before, after); - const responseFieldsComplete = [...responseFields.values()].every( - (fields) => fields.size === networkResponseFields.length, + const requestsComplete = [before, after].every((inspection) => + networkInventoryComplete(inspection, ["network_requests"]), ); + const webSocketInventoryComplete = [before, after].every((inspection) => + networkInventoryComplete(inspection, [ + "websocket_connections", + "websocket_frames", + ]), + ); + const requestIdentities = networkIdentityCoverage(before, after); + const metadata = sharedNetworkMetadata(before, after); const selected = bodyShapesSelected(before) && bodyShapesSelected(after); const webSocketShapes = webSocketShapesSelected(before) && webSocketShapesSelected(after); + const projection = { + metadataFields: metadata.fields, + ambiguousIdentities: requestIdentities.ambiguousIdentities, + inventoryComplete: requestsComplete, + }; const sources = new Map(); if (selected) { const observedSources = (inspection: WebPageInspection) => { @@ -352,13 +441,6 @@ const networkDimension = ( ); const beforeWebSockets = webSocketGroups(before); const afterWebSockets = webSocketGroups(after); - const inventoryComplete = [before, after].every((inspection) => - sectionsComplete(inspection, [ - "network_requests", - "websocket_connections", - "websocket_frames", - ]), - ); const [beforeWebSocketPayload, afterWebSocketPayload] = webSocketPayloadMaps( beforeWebSockets, afterWebSockets, @@ -366,25 +448,38 @@ const networkDimension = ( ); const compared = compareDimension( new Map([ - ...networkMap(before, sources, responseFields), - ...webSocketMetadataMap(beforeWebSockets), + ...networkMap(before, sources, projection), + ...(webSocketInventoryComplete + ? webSocketMetadataMap(beforeWebSockets) + : []), ...beforeWebSocketPayload, ]), new Map([ - ...networkMap(after, sources, responseFields), - ...webSocketMetadataMap(afterWebSockets), + ...networkMap(after, sources, projection), + ...(webSocketInventoryComplete + ? webSocketMetadataMap(afterWebSockets) + : []), ...afterWebSocketPayload, ]), - inventoryComplete && + requestsComplete && + requestIdentities.complete && + webSocketInventoryComplete && shapesComplete && webSocketPayloadBytesComplete && webSocketShapesComplete, - "Network capture or selected HTTP/WebSocket payload-shape coverage is attach-window limited, unavailable, or incomplete.", - inventoryComplete, + requestIdentities.complete + ? "Network capture or selected HTTP/WebSocket payload-shape coverage is limited, unavailable, incomplete, or includes dropped network events." + : "HTTP request identity is incomplete because a resource type is unavailable; potentially matching type groups cannot establish membership/content changes or complete network equality.", + // HTTP and WebSocket identities have independent inventory coverage. Shape + // selection can prevent equality without erasing proven membership changes. + (identity) => + identity.startsWith("net_") + ? requestsComplete + : identity.startsWith("ws_") && webSocketInventoryComplete, ); - if (responseFieldsComplete) return compared; + if (metadata.complete) return compared; const responseEvidenceReason = - "HTTP response or redirect status, MIME type, or byte-count evidence is incomplete."; + "HTTP response or redirect status, MIME type, byte count, or resource-type evidence is incomplete."; if (compared.status === "changed") return compared; return { status: "unknown", @@ -400,58 +495,71 @@ const networkDimension = ( const networkMap = ( inspection: WebPageInspection, sources: ReadonlyMap, - responseFields: ReadonlyMap>, + projection: { + readonly metadataFields: ReadonlyMap< + string, + ReadonlySet + >; + readonly ambiguousIdentities: ReadonlySet; + readonly inventoryComplete: boolean; + }, ): ReadonlyMap => { - const grouped = new Map(); + const grouped = new Map(); for (const request of inspection.network.requests) { const identity = networkIdentity(request); + // A missing type can move repeated requests between these buckets. Neither + // their counts nor fingerprints establish a change to that typed group. + if (projection.ambiguousIdentities.has(identity)) continue; const shapes = sources.get(identity); const values = grouped.get(identity) ?? []; const response: Record = {}; - for (const field of responseFields.get(identity) ?? []) + for (const field of projection.metadataFields.get(identity) ?? []) response[field] = request[field]; - values.push({ - ...response, - redirects: (request.redirects ?? []).map( - ( - { - request_timestamp: _requestTimestamp, - redirect_event_timestamp: _redirectEventTimestamp, - ...redirect + values.push( + digestCanonicalValue({ + ...response, + redirects: (request.redirects ?? []).map( + ( + { + request_timestamp: _requestTimestamp, + redirect_event_timestamp: _redirectEventTimestamp, + ...redirect + }, + index, + ) => { + const response: Partial< + Record + > = {}; + for (const field of projection.metadataFields.get( + redirectNetworkIdentity(identity, index), + ) ?? []) + if (redirect[field] !== null) response[field] = redirect[field]; + return { + url: redirect.url, + response_url: redirect.response_url, + method: redirect.method, + ...response, + }; }, - index, - ) => { - const response: Partial< - Record - > = {}; - for (const field of responseFields.get( - redirectNetworkIdentity(identity, index), - ) ?? []) - if (redirect[field] !== null) response[field] = redirect[field]; - return { - url: redirect.url, - response_url: redirect.response_url, - method: redirect.method, - resource_type: redirect.resource_type, - ...response, - }; - }, - ), - initiator: request.initiator, - ...(shapes?.request === true - ? { request_shape: request.body_shapes.request } - : {}), - ...(shapes?.response === true - ? { response_shape: request.body_shapes.response } - : {}), - }); + ), + initiator: request.initiator, + ...(shapes?.request === true + ? { request_shape: request.body_shapes.request } + : {}), + ...(shapes?.response === true + ? { response_shape: request.body_shapes.response } + : {}), + }), + ); grouped.set(identity, values); } return new Map( [...grouped].map(([identity, values]) => [ identity, digestCanonicalValue( - values.map((value) => digestCanonicalValue(value)).sort(), + // Repeated samples do not establish multiplicity across omissions; + // keep their known content without interpreting missing rows as change. + (projection.inventoryComplete ? values : [...new Set(values)]).sort(), ), ]), ); diff --git a/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.test.ts b/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.test.ts index 88af2ff6af..6a461031da 100644 --- a/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.test.ts +++ b/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.test.ts @@ -3,6 +3,32 @@ import { expect, it } from "vitest"; import { analyzeJavaScriptStaticSource } from "../../domain/javascript/javascriptStaticAnalysis.js"; import { javaScriptWorkerStaticAnalysisSchema } from "./JavaScriptAnalysisWorkerSchemas.js"; +it("retains candidate endpoint interpretation across the worker boundary", () => { + const facts = analyzeJavaScriptStaticSource(` + client.get("settingKey"); + fetch("/api"); + `); + expect(facts.endpoints).toEqual([ + expect.objectContaining({ + value: "settingKey", + interpretation: "candidate", + }), + expect.objectContaining({ value: "/api" }), + ]); + expect(facts.endpoints[1]).not.toHaveProperty("interpretation"); + const serialized = JSON.parse(JSON.stringify(facts)) as unknown; + expect(javaScriptWorkerStaticAnalysisSchema.parse(serialized)).toEqual(facts); + expect( + javaScriptWorkerStaticAnalysisSchema.safeParse({ + ...facts, + endpoints: facts.endpoints.map((endpoint) => ({ + ...endpoint, + interpretation: "confirmed", + })), + }).success, + ).toBe(false); +}); + it("versions fingerprints through the worker schema and rejects v1", () => { const facts = analyzeJavaScriptStaticSource(`globalThis.webpackChunkApp.push([[1], { diff --git a/tests/acceptance/applications/javascriptCancellationCli.test.ts b/tests/acceptance/applications/javascriptCancellationCli.test.ts index c270e678c2..473ae34362 100644 --- a/tests/acceptance/applications/javascriptCancellationCli.test.ts +++ b/tests/acceptance/applications/javascriptCancellationCli.test.ts @@ -58,13 +58,39 @@ it.skipIf(process.platform === "win32").each([ parsedFiles: 1, }, ])( - "returns typed cancellation and exit 130 after SIGINT through $command during $phase", + "returns typed cancellation and exit 130 through $command during $phase", async ({ command, phase, parsedFiles }) => { const root = await createTestTempDirectory("rea-cli-js-cancellation-"); await writeFile(join(root, "main.js"), "export const observed = 1;\n"); + const nodeArguments: string[] = []; + if (parsedFiles !== null) { + // A tiny graph/hash phase may finish before the parent receives stderr. + // Invoke the real signal handler synchronously in the owned CLI child at + // that progress boundary; the two early-phase cases still use OS SIGINT. + const controls = await createTestTempDirectory("rea-cli-cancel-control-"); + const preload = join(controls, "cancel-at-phase.mjs"); + await writeFile( + preload, + ` + const write = process.stderr.write.bind(process.stderr); + let requested = false; + process.stderr.write = function(chunk, ...args) { + const written = write(chunk, ...args); + let update; + try { update = JSON.parse(String(chunk)); } catch {} + if (!requested && update?.rea_progress?.phase === ${JSON.stringify(phase)}) { + requested = true; + process.emit("SIGINT"); + } + return written; + }; + `, + ); + nodeArguments.push("--import", preload); + } const child = spawn( process.execPath, - ["scripts/rea.mjs", command, root, "--json"], + [...nodeArguments, "scripts/rea.mjs", command, root, "--json"], { stdio: ["ignore", "pipe", "pipe"], }, @@ -95,7 +121,7 @@ it.skipIf(process.platform === "win32").each([ !requested ) { requested = true; - child.kill("SIGINT"); + if (parsedFiles === null) child.kill("SIGINT"); } } }); diff --git a/tests/acceptance/applications/javascriptConfigurationReadsCli.test.ts b/tests/acceptance/applications/javascriptConfigurationReadsCli.test.ts index 8b9ba1abc9..db7af4df5c 100644 --- a/tests/acceptance/applications/javascriptConfigurationReadsCli.test.ts +++ b/tests/acceptance/applications/javascriptConfigurationReadsCli.test.ts @@ -76,3 +76,24 @@ cliTest.for(["CLI", "stdio MCP"] as const)( } }, ); + +cliTest.for(["CLI", "stdio MCP"] as const)( + "does not claim an argv value after an unknown object selection escapes through %s", + async (surface, { cli }) => { + const root = await createTestTempDirectory("rea-config-escape-"); + await writeFixtureFiles(root, { + "entry.mjs": [ + 'throw new Error("static fixture must never execute");', + "export function run(key) { patch(process.argv[key]); return process.argv[3]; }", + "export function independent() { return process.env.PORT; }", + ].join("\n"), + }); + const graph = (await analyzeJavaScriptDirectory(surface, cli, root)) + .semantic_graph; + const reads = graph.nodes + .filter((node) => node.kind === "config-source") + .map((node) => [node.properties.source_kind, node.properties.key]); + expect(reads).not.toContainEqual(["argv", "3"]); + expect(reads).toContainEqual(["environment", "PORT"]); + }, +); diff --git a/tests/acceptance/applications/runtime.test.ts b/tests/acceptance/applications/runtime.test.ts index 80989478ce..d085d27713 100644 --- a/tests/acceptance/applications/runtime.test.ts +++ b/tests/acceptance/applications/runtime.test.ts @@ -13,6 +13,7 @@ import { toolAvailabilityInventory, } from "../../../src/contracts/toolOutputSchemaPrimitives.js"; import { parseEvidence } from "../../../src/domain/evidence.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; const mainPath = fileURLToPath( new URL("../../../dist/main.js", import.meta.url), @@ -69,10 +70,13 @@ const completeStderrRecords = (stderr: string): unknown[] => .filter((line) => line.length > 0) .map((line): unknown => JSON.parse(line)); -const callCurrentDocument = async (target?: { - readonly path: string; - readonly kind?: "database"; -}) => { +const callCurrentDocument = async ( + target?: { + readonly path: string; + readonly kind?: "database"; + }, + inspectSession?: (client: Client) => Promise, +) => { const transport = new StdioClientTransport({ command: process.execPath, args: [mainPath], @@ -105,6 +109,7 @@ const callCurrentDocument = async (target?: { name: "current_document", arguments: {}, }); + await inspectSession?.(client); } finally { await client.close(); await transport.close(); @@ -240,18 +245,62 @@ describe("production stdio runtime", () => { ); expect(fatal).toEqual([]); }); +}); +describe("production stdio saved-database admission", () => { it.runIf(process.platform === "darwin")( - "serves a database-kind initial target without a fatal record on macOS", + "refuses database analysis on macOS while keeping the session usable without a fatal record", async () => { - const { result, records } = await callCurrentDocument({ - path: fileURLToPath(import.meta.url), - kind: "database", + const path = fileURLToPath(import.meta.url); + const { result, records } = await callCurrentDocument( + { path, kind: "database" }, + async (client) => { + const status = await expectAvailableToolInventory( + client, + await client.listTools(), + ); + expect(status.structuredContent).toMatchObject({ + result: { + open: true, + path, + kind: "database", + format: "analysis-database", + analysis_provider_binding: null, + }, + }); + }, + ); + expect(parseMcpToolError(result)).toMatchObject({ + error: { + code: "capability_unavailable", + category: "unsupported_provider", + retryable: false, + details: { + operation: "current_document", + selection_reason: "target_unsupported", + rejections: expect.arrayContaining([ + expect.objectContaining({ + provider_id: "hopper", + code: "target_kind_unsupported", + reason: expect.stringContaining( + "cannot start REA's analysis bridge", + ), + diagnostics: { + target_kind: "database", + target_format: "analysis-database", + architecture: null, + }, + }), + ]), + }, + }, }); - expect(result.isError).not.toBe(true); - evidenceResultOf(z.literal("fixture")).parse(result.structuredContent); expect(records).toContainEqual( - expect.objectContaining({ tool: "current_document", status: "ok" }), + expect.objectContaining({ + tool: "current_document", + status: "error", + level: 40, + }), ); expect( records.some( diff --git a/tests/acceptance/tools/passiveNetworkInventory.test.ts b/tests/acceptance/tools/passiveNetworkInventory.test.ts new file mode 100644 index 0000000000..8e5e7a51ff --- /dev/null +++ b/tests/acceptance/tools/passiveNetworkInventory.test.ts @@ -0,0 +1,215 @@ +import { readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { webCaptureDiffSchema } from "../../../src/domain/webCaptureDiffSchemas.js"; +import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { + passiveNetworkCapture, + withPassiveDroppedEvents, +} from "../../fixtures/passiveNetworkCapture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +const comparisonCases = () => { + const before = passiveNetworkCapture(); + const after = passiveNetworkCapture(["/a", "/b"]); + const incomplete = structuredClone(before); + incomplete.completeness.attach_limited_sections.push("network_requests"); + const wsBefore = structuredClone(before); + const wsAfter = structuredClone(after); + for (const capture of [wsBefore, wsAfter]) + capture.completeness.attach_limited_sections.push( + "websocket_connections", + "websocket_frames", + ); + wsAfter.network.websocket_connections.push({ + request_id: "ws", + url: "wss://app.example.test/live", + events: [], + }); + const repeated = passiveNetworkCapture(["/a", "/a"]); + const repeatedUnknown = structuredClone(repeated); + repeatedUnknown.network.requests[1]!.resource_type = null; + const unknownIdentity = structuredClone(before); + unknownIdentity.network.requests[0]!.resource_type = null; + const unknownIdentityAdded = structuredClone(after); + unknownIdentityAdded.network.requests[0]!.resource_type = null; + const unselectedUnknownIdentity = structuredClone(unknownIdentity); + unselectedUnknownIdentity.completeness.excluded.push({ + section: "json_body_shapes", + reason: "not_approved", + count: null, + }); + unselectedUnknownIdentity.network.requests[0]!.body_shapes.status = + "not_approved"; + const droppedRequest = withPassiveDroppedEvents(before, { + network_requests: 1, + }); + const droppedWebSocket = withPassiveDroppedEvents(before, { + websocket_connections: 1, + websocket_frames: 1, + }); + const unrelatedDrop = withPassiveDroppedEvents(before, { + console_events: 1, + }); + const droppedUnselected = withPassiveDroppedEvents(structuredClone(before), { + network_requests: 1, + }); + droppedUnselected.completeness.excluded.push({ + section: "json_body_shapes", + reason: "not_approved", + count: null, + }); + droppedUnselected.network.requests[0]!.body_shapes.status = "not_approved"; + const droppedRepeated = withPassiveDroppedEvents(before, { + network_requests: 1, + }); + const redirectKnown = structuredClone(before); + redirectKnown.network.requests[0]!.redirects = [ + { + url: "https://app.example.test/start", + response_url: "https://app.example.test/start", + method: "GET", + resource_type: "Document", + status: 302, + mime_type: "text/html", + encoded_data_length: 20, + request_timestamp: 1, + redirect_event_timestamp: 2, + }, + ]; + const redirectUnknown = structuredClone(redirectKnown); + redirectUnknown.network.requests[0]!.redirects![0]!.resource_type = null; + return [ + ["added", before, after, "added"], + ["removed", after, before, "removed"], + ["equal unavailable", before, before, null], + ["incomplete inventory", incomplete, after, null], + ["limited WebSocket inventory", wsBefore, wsAfter, "added"], + ["repeated request type uncertainty", repeated, repeatedUnknown, null], + [ + "repeated request type uncertainty reversed", + repeatedUnknown, + repeated, + null, + ], + ["unknown request identity", before, unknownIdentity, null], + ["unknown request identity reversed", unknownIdentity, before, null], + [ + "unknown identity without selected shapes", + unselectedUnknownIdentity, + unselectedUnknownIdentity, + null, + ], + [ + "different URL beside unknown identity", + before, + unknownIdentityAdded, + "added", + ], + ["dropped request count", droppedRequest, after, null], + ["dropped request count reversed", after, droppedRequest, null], + [ + "dropped request with shapes unselected", + droppedUnselected, + droppedUnselected, + null, + ], + [ + "WebSocket drops do not hide HTTP additions", + droppedWebSocket, + after, + "added", + ], + [ + "unrelated drops do not hide HTTP additions", + unrelatedDrop, + after, + "added", + ], + [ + "duplicate samples with dropped inventory", + droppedRepeated, + repeated, + null, + ], + [ + "duplicate samples with dropped inventory reversed", + repeated, + droppedRepeated, + null, + ], + ["unknown redirect type", redirectKnown, redirectUnknown, null], + ["unknown redirect type reversed", redirectUnknown, redirectKnown, null], + ] as const; +}; + +const cases = comparisonCases().map(([name, left, right, change]) => ({ + name, + left, + right, + change, +})); + +const expectComparison = ( + document: unknown, + name: string, + change: "added" | "removed" | null, +): void => { + const evidence = parseEvidence(document); + expect(evidence.operation).toBe("compare_web_captures"); + const result = webCaptureDiffSchema.parse(evidence.normalized_result); + expect(result.dimensions.network, name).toMatchObject({ + status: change === null ? "unknown" : "changed", + total_changes: change === null ? 0 : 1, + changes: + change === null + ? [] + : [{ identity: expect.stringMatching(/^net_/u), change }], + }); + expect(result.overall_status).toBe(change === null ? "unknown" : "changed"); +}; + +cliTest.for(cases)( + "CLI preserves passive inventory evidence: $name", + async ({ name, left, right, change }, { cli }) => { + const root = await createTestTempDirectory("rea-passive-inventory-"); + const paths = [ + join(root, "before.json"), + join(root, "after.json"), + ] as const; + const serialized = [ + JSON.stringify({ inspection: left }), + JSON.stringify({ inspection: right }), + ]; + await Promise.all( + paths.map((path, index) => writeFile(path, serialized[index]!)), + ); + const response = await cli.run({ + arguments: ["compare-web-captures", ...paths, "--json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(response.exitCode, response.stderr).toBe(0); + expectComparison(response.json, name, change); + expect( + await Promise.all(paths.map((path) => readFile(path, "utf8"))), + ).toEqual(serialized); + }, +); + +cliTest( + "stdio MCP preserves passive inventory evidence on one owned connection", + async () => { + const mcp = await connectLocalToolsMcp(); + for (const { name, left, right, change } of cases) { + const response = await mcp.call("compare_web_captures", { + before: { inspection: left }, + after: { inspection: right }, + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + expectComparison(response.structuredContent, name, change); + } + }, +); diff --git a/tests/boundary/browser/sazWebCapture.test.ts b/tests/boundary/browser/sazWebCapture.test.ts index 3e16fac4c3..9a53ff65a8 100644 --- a/tests/boundary/browser/sazWebCapture.test.ts +++ b/tests/boundary/browser/sazWebCapture.test.ts @@ -15,6 +15,7 @@ import { webNetworkCaptureSchema, } from "../../../src/domain/webNetworkCapture.js"; import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { @@ -529,8 +530,10 @@ it.each([ : "invalid_input", ); // Expanded size is a caller-correctable input bound, not a decoder failure. - if (variant === "expanded-limit") - expect(JSON.parse(result.stdout).details.issues).toEqual([ + if (variant === "expanded-limit") { + const error = JSON.parse(result.stdout); + expect(error.code).toBe("invalid_request"); + expect(error.details.issues).toEqual([ expect.objectContaining({ path: ["capture_path", ""], reason: "out_of_range", @@ -539,6 +542,27 @@ it.each([ }, }), ]); + const { call } = await connectLocalToolsMcp(); + const response = await call("inspect_web_network_capture", { + capture_path: path, + format: "saz", + record_ordinals: [0], + }); + expect(response.isError).toBe(true); + const mcpFailure = parseMcpToolError(response); + expect(mcpFailure).toMatchObject({ + error: { + code: error.code, + category: error.category, + details: error.details, + }, + }); + expect(error).not.toHaveProperty("normalized_result"); + expect(error.details).not.toHaveProperty("partial_observation"); + expect(mcpFailure).not.toHaveProperty("normalized_result"); + expect(mcpFailure.error.details).not.toHaveProperty("partial_observation"); + expect(await readFile(path)).toEqual(bytes); + } expect(JSON.stringify(result)).not.toContain("malformed-private-input"); }); diff --git a/tests/boundary/mcp/browserComparisonInput.test.ts b/tests/boundary/mcp/browserComparisonInput.test.ts index 7e916c9f53..bbd8d08843 100644 --- a/tests/boundary/mcp/browserComparisonInput.test.ts +++ b/tests/boundary/mcp/browserComparisonInput.test.ts @@ -130,7 +130,7 @@ describe("browser comparison input boundary", () => { expect(advertised.inputSchema.description).toContain( "before + after; before_scenario + after_scenario + [normalization]", ); - const mixedGroups = [ + const runtimeOnlyInvalid = [ { ...passive, ...scenario }, { ...passive, normalization: { rules: [] } }, { @@ -153,7 +153,7 @@ describe("browser comparison input boundary", () => { advertisedValid: false, runtimeValid: false, })), - ...mixedGroups.map((input) => ({ + ...runtimeOnlyInvalid.map((input) => ({ input, advertisedValid: true, runtimeValid: false, diff --git a/tests/boundary/mcp/enhancedTools.callPathsAndBatches.test.ts b/tests/boundary/mcp/enhancedTools.callPathsAndBatches.test.ts index 38b7697151..8dee037b9d 100644 --- a/tests/boundary/mcp/enhancedTools.callPathsAndBatches.test.ts +++ b/tests/boundary/mcp/enhancedTools.callPathsAndBatches.test.ts @@ -7,6 +7,7 @@ import { AnalysisCancelledError, AnalysisOutputError, } from "../../../src/domain/analysisErrorCore.js"; +import { parseEvidence } from "../../../src/domain/evidence.js"; import type { RelationshipCoverage } from "../../../src/domain/relationshipCoverage.js"; import { err, ok as resultOk } from "../../../src/domain/result.js"; @@ -36,7 +37,7 @@ const relationshipExecution = ( ), ); -describe("enhanced MCP tools", () => { +describe("enhanced MCP relationship limitations", () => { it("preserves per-node relationship limitations and keeps observed edges", async () => { const limitation = "ARM64 call targets are only partially resolved."; const client = await connect({ @@ -100,6 +101,7 @@ describe("enhanced MCP tools", () => { describe("enhanced MCP graph caveats", () => { it("keeps generic provider caveats visible without truncating complete relationships", async () => { const caveat = "Decompiler output may omit indirect calls."; + const leafCaveat = "Source signatures are unavailable."; const client = await connect({ execute: async (name, arguments_) => { if (name === "procedure_address") @@ -108,7 +110,7 @@ describe("enhanced MCP graph caveats", () => { throw new Error(`Unexpected operation: ${name}`); return relationshipExecution( arguments_.procedure === "0x1" ? ["0x2"] : [], - [caveat], + arguments_.procedure === "0x1" ? [caveat] : [caveat, leafCaveat], { status: "complete" }, ); }, @@ -123,6 +125,7 @@ describe("enhanced MCP graph caveats", () => { "Artifact identity is unavailable for this observation.", "Derived by an REA composed workflow.", caveat, + leafCaveat, ], normalized_result: { graph: { @@ -143,9 +146,17 @@ describe("enhanced MCP graph caveats", () => { }, ], }, - limitations: [caveat], + limitations: [caveat, leafCaveat], }, }); + const evidence = parseEvidence(graph.structuredContent); + const normalized = JSON.stringify(evidence.normalized_result); + for (const item of [caveat, leafCaveat]) { + expect(normalized.split(item)).toHaveLength(2); + expect( + evidence.limitations.filter((limit) => limit === item), + ).toHaveLength(1); + } const trace = await client.callTool({ name: "trace_call_path", @@ -159,6 +170,11 @@ describe("enhanced MCP graph caveats", () => { relationship_coverage: { "0x1": { status: "complete" } }, }, }); + const traceEvidence = parseEvidence(trace.structuredContent); + expect( + traceEvidence.limitations.filter((item) => item === caveat), + ).toHaveLength(1); + expect(JSON.stringify(traceEvidence)).not.toContain(leafCaveat); }); it("keeps unclassified legacy caveats on nodes with unknown coverage", async () => { diff --git a/tests/fixtures/passiveNetworkCapture.ts b/tests/fixtures/passiveNetworkCapture.ts new file mode 100644 index 0000000000..1f563bdafd --- /dev/null +++ b/tests/fixtures/passiveNetworkCapture.ts @@ -0,0 +1,129 @@ +import { webPageInspectionSchema } from "../../src/domain/browserObservationSchemas.js"; +import type { WebPageInspection } from "../../src/domain/browserObservationSchemas.js"; +import { + classifyBrowserCompleteness, + type BrowserDroppedEventKind, +} from "../../src/domain/browserCompleteness.js"; + +/** Owned normalized comparison input, not a claim about live CDP inventory coverage. */ +export const passiveNetworkCapture = (paths: readonly string[] = ["/a"]) => { + const origin = "https://app.example.test"; + const timestamp = "2026-10-14T00:00:00.000Z"; + return webPageInspectionSchema.parse({ + browser: { + product: "normalized-fixture", + protocol_version: "1.3", + revision: "fixture", + user_agent: "fixture", + js_version: "fixture", + }, + target: { + target_id: "page", + type: "page", + title: "Fixture", + url: origin, + origin, + attached: false, + }, + capture_window: { + started_at: timestamp, + ended_at: timestamp, + observation_ms: 0, + }, + completeness: { + status: "complete_within_window", + conditions: ["complete_within_window"], + policy_filtered_sections: [], + attach_limited_sections: [], + truncated_sections: [], + unavailable_sections: [], + excluded: [], + dropped_events: { + scripts: 0, + network_requests: 0, + console_events: 0, + websocket_connections: 0, + websocket_frames: 0, + webmcp_tools: 0, + timeline_events: 0, + total: 0, + }, + }, + network: { + requests: paths.map((path, index) => ({ + request_id: `request-${String(index)}`, + url: `${origin}${path}`, + origin, + method: "GET", + resource_type: "Document", + status: 200, + mime_type: "text/html", + encoded_data_length: 10, + initiator: { type: "parser", url: null, line: null, column: null }, + body_shapes: { status: "unavailable", request: null, response: null }, + })), + websocket_connections: [], + coverage_started_at: timestamp, + prior_activity_available: false, + }, + frames: [], + dom: { total_nodes: 0, nodes: [] }, + accessibility: { + total_nodes: 0, + nodes: [], + text_capture: { + status: "not_approved", + retained_bytes: 0, + excluded_fields: 0, + }, + }, + scripts: { total: 0, items: [] }, + resources: [], + workers: [], + console: { + events: [], + coverage_started_at: timestamp, + prior_activity_available: false, + }, + metadata: { + responses: [], + dom_urls: [], + agent_hints: [], + excluded_dom_urls: 0, + headers_allowlisted: true, + }, + storage: { + origin, + usage_bytes: null, + quota_bytes: null, + local_storage_keys: [], + session_storage_keys: [], + indexed_db_names: [], + cache_names: [], + values_redacted: true, + }, + limitations: [ + "Synthetic normalized inventory for offline comparison tests; not recorded from a live browser.", + ], + }); +}; + +/** Use the production completeness classifier without inventing section-list entries. */ +export const withPassiveDroppedEvents = ( + capture: WebPageInspection, + dropped: Partial>, +): WebPageInspection => { + const { completeness } = capture; + const { total: _total, ...counts } = completeness.dropped_events; + return { + ...capture, + completeness: classifyBrowserCompleteness({ + policyFilteredSections: new Set(completeness.policy_filtered_sections), + attachLimitedSections: new Set(completeness.attach_limited_sections), + truncatedSections: new Set(completeness.truncated_sections), + unavailableSections: new Set(completeness.unavailable_sections), + excluded: completeness.excluded, + droppedEvents: { ...counts, ...dropped }, + }), + }; +};