mirror of
https://github.com/morluto/rea.git
synced 2026-10-27 18:05:45 +00:00
feat(browser): page retained web bundle endpoints and inferences
Add web-bundle analysis view projection for completed analyze_web_bundle Evidence to page endpoint observations and static inferences without re-analyzing captures. - Implement projectWebBundleView in src/domain/analysisView/webBundleView.ts supporting endpoints, inferences, and summary collections. - Support filtering by indices, values, script_keys, mechanisms, and inference kinds (such as endpoint_candidate), with field projection. - Preserve duplicate endpoint string literals across distinct source positions, original producer order, and partial producer completeness without upgrading to complete when page collection is exhausted. - Preserve source_index on endpoint and inference rows to guarantee row identity across boundaries. - Unify summary counts calculation across view shapes and remove unused delivery definitions. - Enforce the 1 MiB transport byte budget with typed resource errors and byte_limited indicators. - Wire web-bundle request and result schemas into inspect_analysis_view and analyze_web_bundle summary delivery contracts. - Add comprehensive domain unit tests and MCP boundary tests with advertised schema discovery and validation. Closes #6198
This commit is contained in:
1 parent
e3066c1705
commit
6b1d5d42c6
7 files changed
+1305
-4
No files matched your search
@@ -17,10 +17,26 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [
|
||||
...toolContractMetadata("inspect_analysis_view"),
|
||||
kind: "application",
|
||||
description:
|
||||
'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of supported completed Evidence without repeating analysis. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. BEAM views use kind beam, collections atoms/exports/imports/lambdas/chunks, exact names/indices/fields and matching-row pages; import modules intersects function names. source_index identifies original parent-and-collection rows, preserving every arity and raw atom index. beam.counts preserves null missing tables versus decoded empty tables; runtime absence remains unknown. PYC views use kind python-pyc, collection code_objects, names/indices/fields and paging; source_index scopes every same-name alternative to the parent, nested_code keeps code-constant references, and pyc.coverage preserves source coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent also supports summary and browser-scenario steps/events/transactions/text pages, preserving step_indices, event_kinds, transaction_ids and selected artifacts. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where viewLine truncated
|
||||
'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Use web-bundle collections endpoints or inferences to page endpoint observations and inferences from retained analyze_web_bundle Evidence; summary returns target URL, origin, and counts. Inspect selected views of supported completed Evidence without repeating analysis. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. BEAM views use kind beam, collections atoms/exports/imports/lambdas/chunks, exact names/indices/fields and matching-row pages; import modules intersects function names. source_index identifies original parent-and-collection rows, preserving every arity and raw atom index. beam.counts preserves null missing tables versus decoded empty tables; runtime absence remains unknown. PYC views use kind python-pyc, collection code_objects, names/indices/fields and paging; source_index scopes every same-name alternative to the parent, nested_code keeps code-constant references, and pyc.coverage preserves source coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent also supports summary and browser-scenario steps/events/transactions/text pages, preserving step_indices, event_kinds, transaction_ids and selected artifacts. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the fLine truncated
|
||||
inputSchema: inspectAnalysisViewInputSchema,
|
||||
outputSchema: evidenceResultOf(analysisViewResultSchema),
|
||||
examples: [
|
||||
{
|
||||
title:
|
||||
"Page endpoint observations from retained web bundle analysis Evidence",
|
||||
input: {
|
||||
source: {
|
||||
kind: "retained-evidence",
|
||||
evidence_id: RETAINED_EXAMPLE_ID,
|
||||
},
|
||||
view: {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
offset: 0,
|
||||
limit: 32,
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
title:
|
||||
"Page selected native inventory fields without repeating the complete list",
|
||||
|
||||
@@ -3,7 +3,11 @@ import {
|
||||
javaScriptRoutesCompleteDeliveryShape,
|
||||
webRoutesSummaryDeliveryShape,
|
||||
} from "../domain/analysisView/javascriptRoutesView.js";
|
||||
import { javaScriptRoutesViewResultSchema } from "../domain/analysisView/analysisView.js";
|
||||
import {
|
||||
javaScriptRoutesViewResultSchema,
|
||||
webBundleSummaryViewResultSchema,
|
||||
webBundlePageViewResultSchema,
|
||||
} from "../domain/analysisView/analysisView.js";
|
||||
import type { ToolContract } from "./toolContractTypes.js";
|
||||
import {
|
||||
inspectWebPageInputSchema,
|
||||
@@ -48,7 +52,12 @@ export const analyzeWebBundleRequestSchema = z
|
||||
])
|
||||
.meta({ "x-rea-root-discriminator": "detail" });
|
||||
const bundleOutputSchema = evidenceResultOf(
|
||||
z.union([webBundleAnalysisSchema, javaScriptRoutesViewResultSchema]),
|
||||
z.union([
|
||||
webBundleAnalysisSchema,
|
||||
javaScriptRoutesViewResultSchema,
|
||||
webBundleSummaryViewResultSchema,
|
||||
webBundlePageViewResultSchema,
|
||||
]),
|
||||
);
|
||||
const observationSessionOutputSchema = evidenceResultOf(
|
||||
webObservationSessionSchema,
|
||||
|
||||
@@ -61,6 +61,14 @@ import {
|
||||
javaScriptRoutesSourceReferenceSchema,
|
||||
projectJavaScriptRoutesView,
|
||||
} from "./javascriptRoutesView.js";
|
||||
import {
|
||||
webBundleViewRequestSchema,
|
||||
webBundleViewItemsSchema,
|
||||
webBundleSummarySchema,
|
||||
projectWebBundleView,
|
||||
} from "./webBundleView.js";
|
||||
import { webBundleAnalysisSchema } from "../webBundleAnalysis.js";
|
||||
import { recordedBrowserOriginSchema } from "../browserObservation.js";
|
||||
import { javaScriptRouteInventorySchema } from "../javascript/javascriptRouteInventory.js";
|
||||
import {
|
||||
sourceToBundleViewRequestSchema,
|
||||
@@ -395,6 +403,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
|
||||
pythonPycViewRequestSchema,
|
||||
nativeExportsViewRequestSchema,
|
||||
javaScriptRoutesViewRequestSchema,
|
||||
webBundleViewRequestSchema,
|
||||
nativeFunctionMatchingViewRequestSchema,
|
||||
sourceToBundleViewRequestSchema,
|
||||
summaryViewSchema,
|
||||
@@ -881,6 +890,30 @@ export const javaScriptRoutesViewResultSchema = z.strictObject({
|
||||
byte_limited: z.boolean(),
|
||||
});
|
||||
|
||||
/** Compact web bundle findings retain endpoint observations and inference candidates. */
|
||||
export const webBundleSummaryViewResultSchema = z.strictObject({
|
||||
...summaryViewResultBase,
|
||||
artifact: artifactSchema.nullable(),
|
||||
parent_operation: z.literal("analyze_web_bundle"),
|
||||
target_url: z.string(),
|
||||
target_origin: recordedBrowserOriginSchema.nullable().optional(),
|
||||
summary: webBundleSummarySchema,
|
||||
});
|
||||
|
||||
export const webBundlePageViewResultSchema = z.strictObject({
|
||||
...viewResultBase,
|
||||
artifact: artifactSchema.nullable(),
|
||||
kind: z.literal("web-bundle"),
|
||||
parent_operation: z.literal("analyze_web_bundle"),
|
||||
target_url: z.string(),
|
||||
target_origin: recordedBrowserOriginSchema.nullable().optional(),
|
||||
view: webBundleViewRequestSchema,
|
||||
completeness: webBundleAnalysisSchema.shape.completeness,
|
||||
items: webBundleViewItemsSchema,
|
||||
summary: webBundleSummarySchema.nullable(),
|
||||
byte_limited: z.boolean(),
|
||||
});
|
||||
|
||||
export const gpuCodeObjectsSummaryViewResultSchema = z.strictObject({
|
||||
...summaryViewResultBase,
|
||||
parent_operation: z.literal("inspect_gpu_code_objects"),
|
||||
@@ -1072,6 +1105,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
|
||||
nativeObservationPageViewResultSchema,
|
||||
nativeContextViewResultSchema,
|
||||
javaScriptRoutesViewResultSchema,
|
||||
webBundlePageViewResultSchema,
|
||||
sourceToBundlePageViewResultSchema,
|
||||
z.strictObject({
|
||||
...viewResultBase,
|
||||
@@ -1159,6 +1193,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
|
||||
nativeFunctionMatchingSummaryViewResultSchema,
|
||||
nativeFunctionSummaryViewResultSchema,
|
||||
sourceToBundleSummaryViewResultSchema,
|
||||
webBundleSummaryViewResultSchema,
|
||||
]),
|
||||
z.strictObject({
|
||||
...viewResultBase,
|
||||
@@ -1372,6 +1407,13 @@ export type InspectAnalysisViewInput = z.output<
|
||||
export type AnalysisViewResult = z.output<typeof analysisViewResultSchema>;
|
||||
export type AnalysisViewCoverage = z.output<typeof coverageSchema>;
|
||||
|
||||
export {
|
||||
webBundleViewRequestSchema,
|
||||
webBundleViewItemsSchema,
|
||||
webBundleSummarySchema,
|
||||
projectWebBundleView,
|
||||
};
|
||||
|
||||
/** Parent record already resolved from inline Evidence or a session reference. */
|
||||
export interface AnalysisViewParent {
|
||||
readonly evidenceId: string;
|
||||
@@ -1490,6 +1532,7 @@ const retainedViewProjectors = new Map<string, typeof projectWasmArtifactView>([
|
||||
["compare_source_to_bundle", projectSourceToBundleView],
|
||||
["inspect_artifact", projectArtifactInventoryView],
|
||||
["inventory_artifact", projectArtifactInventoryView],
|
||||
["analyze_web_bundle", projectWebBundleView],
|
||||
]);
|
||||
|
||||
/** Budgets retained across source admission and the selected grouping projection. */
|
||||
@@ -1593,6 +1636,10 @@ export const projectAnalysisView = (
|
||||
const projected = projectJavaScriptRoutesView(parent, view);
|
||||
return projected.ok ? ok(sealAnalysisView(projected.value)) : projected;
|
||||
}
|
||||
if (view.kind === "web-bundle") {
|
||||
const projected = projectWebBundleView(parent, view);
|
||||
return projected.ok ? ok(sealAnalysisView(projected.value)) : projected;
|
||||
}
|
||||
if (isGroupingView(view)) {
|
||||
const grouped = projectGroupingView(parent, view, accumulators);
|
||||
return grouped.ok ? ok(sealAnalysisView(grouped.value)) : grouped;
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { javascriptSourceRangeSchema } from "../javascript/javascriptSourceCoordinates.js";
|
||||
import { z } from "zod";
|
||||
import { webBundleViewRequestSchema } from "./webBundleView.js";
|
||||
import { digestSchema } from "../digests.js";
|
||||
import { isImmutableJsonSnapshot } from "../immutableJson.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
@@ -122,7 +123,10 @@ export const javaScriptRoutesSummaryDeliveryShape = {
|
||||
};
|
||||
export const webRoutesSummaryDeliveryShape = {
|
||||
detail: z.literal("summary"),
|
||||
view: javaScriptRoutesViewRequestSchema,
|
||||
view: z.union([
|
||||
javaScriptRoutesViewRequestSchema,
|
||||
webBundleViewRequestSchema,
|
||||
]),
|
||||
};
|
||||
export const javaScriptRoutesDeliverySchema = z.discriminatedUnion("detail", [
|
||||
z.strictObject(javaScriptRoutesCompleteDeliveryShape),
|
||||
|
||||
@@ -0,0 +1,488 @@
|
||||
import { expect, it } from "vitest";
|
||||
import type { WebBundleAnalysis } from "../webBundleAnalysis.js";
|
||||
import {
|
||||
analysisViewRequestSchema,
|
||||
projectAnalysisView,
|
||||
type AnalysisViewParent,
|
||||
} from "./analysisView.js";
|
||||
|
||||
const sampleAnalysis = (): WebBundleAnalysis => ({
|
||||
capture: {
|
||||
target_url: "https://example.test/app",
|
||||
target_origin: "https://example.test",
|
||||
scripts_observed: 3,
|
||||
scripts_analyzed: 2,
|
||||
source_artifacts: [],
|
||||
},
|
||||
observations: {
|
||||
chunks: {
|
||||
nodes: [
|
||||
{
|
||||
script_key: "app.js",
|
||||
url: "https://example.test/app.js",
|
||||
artifact_sha256: "a".repeat(64),
|
||||
bytes: 1200,
|
||||
},
|
||||
{
|
||||
script_key: "vendor.js",
|
||||
url: "https://example.test/vendor.js",
|
||||
artifact_sha256: "b".repeat(64),
|
||||
bytes: 4500,
|
||||
},
|
||||
],
|
||||
edges: [],
|
||||
},
|
||||
routes: [],
|
||||
endpoints: [
|
||||
{
|
||||
value: "/api/users",
|
||||
mechanism: "call:fetch",
|
||||
location: { script_key: "app.js", line: 10, column: 2 },
|
||||
},
|
||||
{
|
||||
value: "/api/users",
|
||||
mechanism: "call:fetch",
|
||||
location: { script_key: "vendor.js", line: 42, column: 8 },
|
||||
},
|
||||
{
|
||||
value: "/api/orders",
|
||||
mechanism: "call:sendBeacon",
|
||||
location: { script_key: "app.js", line: 85, column: 4 },
|
||||
},
|
||||
],
|
||||
webmcp_declarations: [],
|
||||
source_maps: {
|
||||
status: "not_requested",
|
||||
requested: 0,
|
||||
processed: 0,
|
||||
items: [],
|
||||
},
|
||||
},
|
||||
inferences: [
|
||||
{
|
||||
kind: "endpoint_candidate",
|
||||
value: "/api/dynamic-user",
|
||||
confidence: "low",
|
||||
basis: [
|
||||
{
|
||||
script_key: "app.js",
|
||||
artifact_sha256: "a".repeat(64),
|
||||
line: 25,
|
||||
column: 4,
|
||||
detector: "call:client.get:unverified-network-receiver",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
kind: "vendor_fingerprint",
|
||||
value: "React",
|
||||
confidence: "high",
|
||||
basis: [
|
||||
{
|
||||
script_key: "vendor.js",
|
||||
artifact_sha256: "b".repeat(64),
|
||||
line: 1,
|
||||
column: 0,
|
||||
detector: "vendor:react",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
unknowns: [
|
||||
{
|
||||
dimension: "network",
|
||||
reason: "Unverified client receiver",
|
||||
affected_script_keys: ["app.js"],
|
||||
},
|
||||
],
|
||||
completeness: {
|
||||
status: "partial",
|
||||
parsed_scripts: 2,
|
||||
parse_failures: 1,
|
||||
visited_ast_nodes: 150,
|
||||
},
|
||||
limitations: ["Script evaluation was omitted"],
|
||||
});
|
||||
|
||||
const makeParent = (
|
||||
overrides?: Partial<WebBundleAnalysis>,
|
||||
): AnalysisViewParent => ({
|
||||
evidenceId: `ev_${"c".repeat(64)}`,
|
||||
operation: "analyze_web_bundle",
|
||||
limitations: ["Parent run was bounded"],
|
||||
normalizedResult: {
|
||||
...sampleAnalysis(),
|
||||
...overrides,
|
||||
},
|
||||
});
|
||||
|
||||
const query = (
|
||||
source: AnalysisViewParent,
|
||||
selection: Record<string, unknown>,
|
||||
) => {
|
||||
const result = projectAnalysisView(
|
||||
source,
|
||||
analysisViewRequestSchema.parse(selection),
|
||||
);
|
||||
if (!result.ok) throw result.error;
|
||||
return result.value;
|
||||
};
|
||||
|
||||
it("pages endpoints and concatenated pages match original records exactly, preserving duplicates", () => {
|
||||
const parent = makeParent();
|
||||
const page1 = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
offset: 0,
|
||||
limit: 2,
|
||||
});
|
||||
const page2 = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
offset: 2,
|
||||
limit: 2,
|
||||
});
|
||||
|
||||
expect(page1.kind).toBe("web-bundle");
|
||||
expect(page2.kind).toBe("web-bundle");
|
||||
if (page1.kind !== "web-bundle" || page2.kind !== "web-bundle") return;
|
||||
|
||||
expect(page1.items).toHaveLength(2);
|
||||
expect(page2.items).toHaveLength(1);
|
||||
expect(page1.coverage).toMatchObject({
|
||||
status: "page",
|
||||
examined: 2,
|
||||
total: 3,
|
||||
next_offset: 2,
|
||||
exhausted: false,
|
||||
});
|
||||
expect(page2.coverage).toMatchObject({
|
||||
status: "page",
|
||||
examined: 1,
|
||||
total: 3,
|
||||
next_offset: null,
|
||||
exhausted: true,
|
||||
});
|
||||
|
||||
const concatenated = [...page1.items, ...page2.items];
|
||||
const original = sampleAnalysis().observations.endpoints;
|
||||
expect(concatenated).toEqual(
|
||||
original.map((row, source_index) => ({ source_index, ...row })),
|
||||
);
|
||||
// Preserves duplicate "/api/users" at different locations
|
||||
expect(concatenated[0]!.value).toBe("/api/users");
|
||||
expect(concatenated[1]!.value).toBe("/api/users");
|
||||
const first = concatenated[0]!;
|
||||
const second = concatenated[1]!;
|
||||
if (!("location" in first) || !("location" in second)) {
|
||||
throw new Error("Expected endpoint rows with location");
|
||||
}
|
||||
expect(first.location).not.toEqual(second.location);
|
||||
});
|
||||
|
||||
it("filters endpoints by values, script_keys, mechanisms, and field projections", () => {
|
||||
const parent = makeParent();
|
||||
const filtered = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
values: ["/api/orders"],
|
||||
fields: ["value"],
|
||||
});
|
||||
|
||||
if (filtered.kind !== "web-bundle")
|
||||
throw new Error("Expected web-bundle view");
|
||||
expect(filtered.items).toEqual([{ source_index: 2, value: "/api/orders" }]);
|
||||
expect(filtered.coverage.total).toBe(1);
|
||||
|
||||
const byScript = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
script_keys: ["vendor.js"],
|
||||
});
|
||||
if (byScript.kind !== "web-bundle")
|
||||
throw new Error("Expected web-bundle view");
|
||||
expect(byScript.items).toHaveLength(1);
|
||||
const item = byScript.items[0]!;
|
||||
if (!("location" in item)) {
|
||||
throw new Error("Expected endpoint row with location");
|
||||
}
|
||||
expect(item.location).toMatchObject({ script_key: "vendor.js" });
|
||||
});
|
||||
|
||||
it("pages inference records and filters by kind (endpoint_candidate)", () => {
|
||||
const parent = makeParent();
|
||||
const candidatesOnly = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "inferences",
|
||||
kinds: ["endpoint_candidate"],
|
||||
});
|
||||
|
||||
if (candidatesOnly.kind !== "web-bundle")
|
||||
throw new Error("Expected web-bundle view");
|
||||
expect(candidatesOnly.items).toHaveLength(1);
|
||||
expect(candidatesOnly.items[0]).toMatchObject({
|
||||
kind: "endpoint_candidate",
|
||||
value: "/api/dynamic-user",
|
||||
confidence: "low",
|
||||
});
|
||||
expect(candidatesOnly.coverage).toMatchObject({
|
||||
total: 1,
|
||||
exhausted: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves partial producer coverage even when view pagination is exhausted", () => {
|
||||
const parent = makeParent({
|
||||
completeness: {
|
||||
status: "partial",
|
||||
parsed_scripts: 1,
|
||||
parse_failures: 1,
|
||||
visited_ast_nodes: 50,
|
||||
},
|
||||
});
|
||||
|
||||
const page = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
offset: 0,
|
||||
limit: 10,
|
||||
});
|
||||
|
||||
if (page.kind !== "web-bundle") throw new Error("Expected web-bundle view");
|
||||
expect(page.coverage.exhausted).toBe(true);
|
||||
// Must remain "partial" and never be upgraded to "complete"
|
||||
expect(page.completeness.status).toBe("partial");
|
||||
expect(page.completeness.parse_failures).toBe(1);
|
||||
});
|
||||
|
||||
it("serves web bundle summary via kind: summary and kind: web-bundle collection: summary", () => {
|
||||
const parent = makeParent();
|
||||
|
||||
const summaryView = query(parent, { kind: "summary" });
|
||||
expect(summaryView.kind).toBe("summary");
|
||||
if (summaryView.kind === "summary") {
|
||||
expect(summaryView.summary).toMatchObject({
|
||||
target_url: "https://example.test/app",
|
||||
counts: {
|
||||
endpoints: 3,
|
||||
inferences: 2,
|
||||
endpoint_candidates: 1,
|
||||
},
|
||||
completeness: { status: "partial" },
|
||||
});
|
||||
}
|
||||
|
||||
const collectionSummary = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "summary",
|
||||
});
|
||||
expect(collectionSummary.kind).toBe("web-bundle");
|
||||
if (collectionSummary.kind === "web-bundle") {
|
||||
expect(collectionSummary.summary).toMatchObject({
|
||||
target_url: "https://example.test/app",
|
||||
counts: {
|
||||
endpoints: 3,
|
||||
inferences: 2,
|
||||
endpoint_candidates: 1,
|
||||
},
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("returns an empty page when offset is beyond total items", () => {
|
||||
const parent = makeParent();
|
||||
const empty = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
offset: 100,
|
||||
limit: 10,
|
||||
});
|
||||
|
||||
if (empty.kind !== "web-bundle") throw new Error("Expected web-bundle view");
|
||||
expect(empty.items).toEqual([]);
|
||||
expect(empty.coverage).toMatchObject({
|
||||
status: "empty",
|
||||
examined: 0,
|
||||
total: 3,
|
||||
next_offset: null,
|
||||
exhausted: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("returns typed AnalysisResourceConstraintError when a single record exceeds 1 MiB", () => {
|
||||
const hugeEndpoint = {
|
||||
value: "/api/" + "x".repeat(1024 * 1024),
|
||||
mechanism: "call:fetch",
|
||||
location: { script_key: "app.js", line: 1, column: 0 },
|
||||
};
|
||||
const parent = makeParent({
|
||||
observations: {
|
||||
...sampleAnalysis().observations,
|
||||
endpoints: [hugeEndpoint],
|
||||
},
|
||||
});
|
||||
|
||||
const result = projectAnalysisView(
|
||||
parent,
|
||||
analysisViewRequestSchema.parse({
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
}),
|
||||
);
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
if (result.ok) throw new Error("Expected resource error");
|
||||
expect(result.error._tag).toBe("AnalysisResourceConstraintError");
|
||||
expect(result.error.message).toContain("byte budget");
|
||||
});
|
||||
|
||||
it("bounds multi-item page before budget overflow and sets byte_limited: true", () => {
|
||||
const largeRecord = (index: number) => ({
|
||||
value: `/api/${index}/` + "x".repeat(300_000),
|
||||
mechanism: "call:fetch",
|
||||
location: { script_key: "app.js", line: index, column: 0 },
|
||||
});
|
||||
const parent = makeParent({
|
||||
observations: {
|
||||
...sampleAnalysis().observations,
|
||||
endpoints: [
|
||||
largeRecord(1),
|
||||
largeRecord(2),
|
||||
largeRecord(3),
|
||||
largeRecord(4),
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
const page = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
limit: 10,
|
||||
});
|
||||
|
||||
if (page.kind !== "web-bundle") throw new Error("Expected web-bundle view");
|
||||
expect(page.byte_limited).toBe(true);
|
||||
expect(page.items.length).toBeGreaterThan(0);
|
||||
expect(page.items.length).toBeLessThan(4);
|
||||
expect(page.coverage.next_offset).toBe(page.items.length);
|
||||
expect(page.coverage.exhausted).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects non-web-bundle parent operation with invalid_value", () => {
|
||||
const invalidParent: AnalysisViewParent = {
|
||||
evidenceId: `ev_${"d".repeat(64)}`,
|
||||
operation: "analyze_javascript_application",
|
||||
limitations: [],
|
||||
normalizedResult: sampleAnalysis(),
|
||||
};
|
||||
|
||||
const result = projectAnalysisView(
|
||||
invalidParent,
|
||||
analysisViewRequestSchema.parse({
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
}),
|
||||
);
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
if (result.ok) throw new Error("Expected failure");
|
||||
expect((result.error as any).issues[0].message).toContain(
|
||||
"Web-bundle views require analyze_web_bundle",
|
||||
);
|
||||
expect((result.error as any).issues[0].reason).toBe("invalid_value");
|
||||
});
|
||||
|
||||
it("preserves parent artifact identity across summary and page views", () => {
|
||||
const artifact = { path: "/app.js", sha256: "e".repeat(64) };
|
||||
const parent: AnalysisViewParent = {
|
||||
...makeParent(),
|
||||
artifact,
|
||||
};
|
||||
|
||||
const summary = query(parent, { kind: "summary" });
|
||||
expect(summary.artifact).toEqual(artifact);
|
||||
|
||||
const page = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
});
|
||||
expect(page.artifact).toEqual(artifact);
|
||||
});
|
||||
|
||||
it("filters endpoints by indices and inferences by script_keys and fields", () => {
|
||||
const parent = makeParent();
|
||||
|
||||
const byIndex = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
indices: [1],
|
||||
});
|
||||
if (byIndex.kind !== "web-bundle")
|
||||
throw new Error("Expected web-bundle view");
|
||||
expect(byIndex.items).toHaveLength(1);
|
||||
expect(byIndex.items[0]).toMatchObject({
|
||||
value: "/api/users",
|
||||
location: { script_key: "vendor.js" },
|
||||
});
|
||||
|
||||
const byScriptKey = query(parent, {
|
||||
kind: "web-bundle",
|
||||
collection: "inferences",
|
||||
script_keys: ["vendor.js"],
|
||||
fields: ["kind", "value"],
|
||||
});
|
||||
if (byScriptKey.kind !== "web-bundle")
|
||||
throw new Error("Expected web-bundle view");
|
||||
expect(byScriptKey.items).toEqual([
|
||||
{ source_index: 1, kind: "vendor_fingerprint", value: "React" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("correctly counts zero findings in summary for an empty web bundle", () => {
|
||||
const emptyParent = makeParent({
|
||||
observations: {
|
||||
chunks: { nodes: [], edges: [] },
|
||||
routes: [],
|
||||
endpoints: [],
|
||||
webmcp_declarations: [],
|
||||
source_maps: {
|
||||
status: "not_requested",
|
||||
requested: 0,
|
||||
processed: 0,
|
||||
items: [],
|
||||
},
|
||||
},
|
||||
inferences: [],
|
||||
});
|
||||
|
||||
const summary = query(emptyParent, { kind: "summary" });
|
||||
if (summary.kind !== "summary") throw new Error("Expected summary");
|
||||
expect(summary.summary).toMatchObject({
|
||||
counts: {
|
||||
endpoints: 0,
|
||||
inferences: 0,
|
||||
endpoint_candidates: 0,
|
||||
chunks: 0,
|
||||
routes: 0,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects malformed parent analysis report", () => {
|
||||
const malformedParent: AnalysisViewParent = {
|
||||
evidenceId: `ev_${"e".repeat(64)}`,
|
||||
operation: "analyze_web_bundle",
|
||||
limitations: [],
|
||||
normalizedResult: { malformed: true },
|
||||
};
|
||||
|
||||
const result = projectAnalysisView(
|
||||
malformedParent,
|
||||
analysisViewRequestSchema.parse({
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
}),
|
||||
);
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
});
|
||||
@@ -0,0 +1,384 @@
|
||||
import { z } from "zod";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { AnalysisResourceConstraintError } from "../analysisErrorCore.js";
|
||||
import { jsonValueByteLength } from "../jsonValueByteLength.js";
|
||||
import type { JsonValue } from "../jsonValue.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import {
|
||||
webBundleAnalysisSchema,
|
||||
type WebBundleAnalysis,
|
||||
} from "../webBundleAnalysis.js";
|
||||
import { recordedBrowserOriginSchema } from "../browserObservation.js";
|
||||
import type {
|
||||
AnalysisViewParent,
|
||||
AnalysisViewRequest,
|
||||
UnsignedAnalysisView,
|
||||
} from "./analysisView.js";
|
||||
import {
|
||||
boundedViewMessages,
|
||||
completeWithinViewCoverage,
|
||||
pageViewCoverage,
|
||||
} from "./analysisViewCoverage.js";
|
||||
import { analysisViewInputError } from "./analysisViewErrors.js";
|
||||
import { createAnalysisViewResultReader } from "./analysisViewParent.js";
|
||||
|
||||
const PAGE_BYTES = 1024 * 1024;
|
||||
const DIGEST_BYTES = jsonValueByteLength({ view_digest: "0".repeat(64) }) - 1;
|
||||
const PROJECTION_BYTES = PAGE_BYTES - DIGEST_BYTES;
|
||||
|
||||
const endpointSchema =
|
||||
webBundleAnalysisSchema.shape.observations.shape.endpoints.element;
|
||||
const inferenceSchema = webBundleAnalysisSchema.shape.inferences.element;
|
||||
const sourceLocationSchema = endpointSchema.shape.location;
|
||||
const basisSchema = inferenceSchema.shape.basis.element;
|
||||
|
||||
const pageShape = {
|
||||
kind: z.literal("web-bundle"),
|
||||
offset: z.number().int().nonnegative().default(0),
|
||||
limit: z.number().int().positive().default(64),
|
||||
};
|
||||
|
||||
export const webBundleViewRequestSchema = z.discriminatedUnion("collection", [
|
||||
z.strictObject({
|
||||
...pageShape,
|
||||
collection: z.literal("endpoints"),
|
||||
indices: z.array(z.number().int().nonnegative()).optional(),
|
||||
values: z.array(z.string()).optional(),
|
||||
script_keys: z.array(z.string()).optional(),
|
||||
mechanisms: z.array(z.string()).optional(),
|
||||
fields: z.array(endpointSchema.keyof()).optional(),
|
||||
}),
|
||||
z.strictObject({
|
||||
...pageShape,
|
||||
collection: z.literal("inferences"),
|
||||
indices: z.array(z.number().int().nonnegative()).optional(),
|
||||
kinds: z.array(inferenceSchema.shape.kind).optional(),
|
||||
values: z.array(z.string()).optional(),
|
||||
script_keys: z.array(z.string()).optional(),
|
||||
fields: z.array(inferenceSchema.keyof()).optional(),
|
||||
}),
|
||||
z.strictObject({
|
||||
kind: z.literal("web-bundle"),
|
||||
collection: z.literal("summary"),
|
||||
}),
|
||||
]);
|
||||
|
||||
export type WebBundleViewRequest = z.output<typeof webBundleViewRequestSchema>;
|
||||
|
||||
export const webBundleSummarySchema = z.strictObject({
|
||||
target_url: z.string(),
|
||||
target_origin: recordedBrowserOriginSchema.nullable().optional(),
|
||||
scripts_observed: z.number().int().nonnegative(),
|
||||
scripts_analyzed: z.number().int().nonnegative(),
|
||||
counts: z.strictObject({
|
||||
endpoints: z.number().int().nonnegative(),
|
||||
inferences: z.number().int().nonnegative(),
|
||||
endpoint_candidates: z.number().int().nonnegative(),
|
||||
chunks: z.number().int().nonnegative(),
|
||||
routes: z.number().int().nonnegative(),
|
||||
}),
|
||||
completeness: webBundleAnalysisSchema.shape.completeness,
|
||||
limitation_count: z.number().int().nonnegative(),
|
||||
});
|
||||
|
||||
export const webBundleEndpointRowSchema = z.strictObject({
|
||||
source_index: z.number().int().nonnegative(),
|
||||
value: endpointSchema.shape.value.optional(),
|
||||
mechanism: endpointSchema.shape.mechanism.optional(),
|
||||
location: sourceLocationSchema.optional(),
|
||||
});
|
||||
|
||||
export const webBundleInferenceRowSchema = z.strictObject({
|
||||
source_index: z.number().int().nonnegative(),
|
||||
kind: inferenceSchema.shape.kind.optional(),
|
||||
value: inferenceSchema.shape.value.optional(),
|
||||
confidence: inferenceSchema.shape.confidence.optional(),
|
||||
basis: z.array(basisSchema).optional(),
|
||||
});
|
||||
|
||||
export const webBundleViewItemsSchema = z.union([
|
||||
z.array(webBundleEndpointRowSchema),
|
||||
z.array(webBundleInferenceRowSchema),
|
||||
]);
|
||||
|
||||
const readReport = createAnalysisViewResultReader(webBundleAnalysisSchema);
|
||||
|
||||
const choose = <T extends Record<string, JsonValue>>(
|
||||
row: T,
|
||||
fields: readonly string[] | undefined,
|
||||
): Record<string, JsonValue> => {
|
||||
if (fields === undefined) return { ...row };
|
||||
const allowed = new Set(fields);
|
||||
const out: Record<string, JsonValue> = {};
|
||||
for (const [k, v] of Object.entries(row)) {
|
||||
if (allowed.has(k)) {
|
||||
out[k] = v;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
};
|
||||
|
||||
/** Historical projection of completed analyze_web_bundle Evidence. */
|
||||
export const projectWebBundleView = (
|
||||
parent: AnalysisViewParent,
|
||||
selected: AnalysisViewRequest,
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
if (selected.kind !== "web-bundle" && selected.kind !== "summary")
|
||||
return err(
|
||||
analysisViewInputError([
|
||||
{
|
||||
path: ["view", "kind"],
|
||||
reason: "invalid_value",
|
||||
message: "Select a web-bundle or summary view.",
|
||||
},
|
||||
]),
|
||||
);
|
||||
|
||||
if (parent.operation !== "analyze_web_bundle")
|
||||
return err(
|
||||
analysisViewInputError([
|
||||
{
|
||||
path: ["source", "operation"],
|
||||
reason: "invalid_value",
|
||||
message: "Web-bundle views require analyze_web_bundle Evidence.",
|
||||
},
|
||||
]),
|
||||
);
|
||||
|
||||
const parsed = readReport(parent);
|
||||
if (!parsed.ok) return parsed;
|
||||
const report: WebBundleAnalysis = parsed.value;
|
||||
|
||||
const shared = {
|
||||
parent_evidence_id: parent.evidenceId,
|
||||
parent_operation: "analyze_web_bundle" as const,
|
||||
parent_digest: parent.evidenceId.slice(3),
|
||||
artifact: parent.artifact ?? null,
|
||||
target_url: report.capture.target_url,
|
||||
target_origin: report.capture.target_origin ?? null,
|
||||
limitations: boundedViewMessages(
|
||||
[...parent.limitations, ...report.limitations],
|
||||
parent.evidenceId,
|
||||
"limitations",
|
||||
),
|
||||
unknowns: boundedViewMessages(
|
||||
[
|
||||
"Static bundle findings are observations or bounded inferences; REA does not execute captured JavaScript.",
|
||||
"String-built routes and endpoints, encrypted configuration, and server-side behavior may remain unknown.",
|
||||
...report.unknowns.map((u) => `${u.dimension}: ${u.reason}`),
|
||||
],
|
||||
parent.evidenceId,
|
||||
"unknowns",
|
||||
),
|
||||
};
|
||||
|
||||
const tooLarge = (collection: string): Result<never, AnalysisError> =>
|
||||
err(
|
||||
new AnalysisResourceConstraintError(
|
||||
"inspect_analysis_view",
|
||||
"transport",
|
||||
"The requested web-bundle page cannot fit the analysis-view byte budget; select fewer fields or a smaller selection. The complete parent is retained.",
|
||||
{
|
||||
parent_evidence_id: parent.evidenceId,
|
||||
limit_bytes: PAGE_BYTES,
|
||||
collection,
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
const buildSummary = (): z.output<typeof webBundleSummarySchema> => ({
|
||||
target_url: report.capture.target_url,
|
||||
target_origin: report.capture.target_origin ?? null,
|
||||
scripts_observed: report.capture.scripts_observed,
|
||||
scripts_analyzed: report.capture.scripts_analyzed,
|
||||
counts: {
|
||||
endpoints: report.observations.endpoints.length,
|
||||
inferences: report.inferences.length,
|
||||
endpoint_candidates: report.inferences.filter(
|
||||
(i) => i.kind === "endpoint_candidate",
|
||||
).length,
|
||||
chunks: report.observations.chunks.nodes.length,
|
||||
routes: report.observations.routes.length,
|
||||
},
|
||||
completeness: report.completeness,
|
||||
limitation_count: report.limitations.length,
|
||||
});
|
||||
|
||||
if (selected.kind === "summary") {
|
||||
const summary = buildSummary();
|
||||
const result: UnsignedAnalysisView = {
|
||||
kind: "summary",
|
||||
view: selected,
|
||||
...shared,
|
||||
summary,
|
||||
coverage: completeWithinViewCoverage(1, 1),
|
||||
};
|
||||
return jsonValueByteLength(
|
||||
result as unknown as JsonValue,
|
||||
PROJECTION_BYTES,
|
||||
) > PROJECTION_BYTES
|
||||
? tooLarge("summary")
|
||||
: ok(result);
|
||||
}
|
||||
|
||||
const view = selected;
|
||||
if (view.collection === "summary") {
|
||||
const summary = buildSummary();
|
||||
const result: UnsignedAnalysisView = {
|
||||
kind: "web-bundle",
|
||||
view,
|
||||
...shared,
|
||||
completeness: report.completeness,
|
||||
items: [],
|
||||
summary,
|
||||
byte_limited: false,
|
||||
coverage: completeWithinViewCoverage(1, 1),
|
||||
};
|
||||
return jsonValueByteLength(
|
||||
result as unknown as JsonValue,
|
||||
PROJECTION_BYTES,
|
||||
) > PROJECTION_BYTES
|
||||
? tooLarge("summary")
|
||||
: ok(result);
|
||||
}
|
||||
|
||||
const deliverRows = <T extends Record<string, JsonValue>>(
|
||||
rows: readonly { readonly row: T; readonly source_index: number }[],
|
||||
project: (row: T, source_index: number) => Record<string, JsonValue>,
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
const baseEnvelope = {
|
||||
...shared,
|
||||
kind: "web-bundle" as const,
|
||||
view,
|
||||
completeness: report.completeness,
|
||||
items: [],
|
||||
summary: null,
|
||||
byte_limited: false,
|
||||
coverage: {
|
||||
status: "complete-within-view" as const,
|
||||
examined: Number.MAX_SAFE_INTEGER,
|
||||
total: Number.MAX_SAFE_INTEGER,
|
||||
next_offset: Number.MAX_SAFE_INTEGER,
|
||||
exhausted: false,
|
||||
},
|
||||
};
|
||||
|
||||
let bytes =
|
||||
DIGEST_BYTES +
|
||||
jsonValueByteLength(
|
||||
baseEnvelope as unknown as JsonValue,
|
||||
PROJECTION_BYTES,
|
||||
);
|
||||
|
||||
if (bytes > PAGE_BYTES) return tooLarge(view.collection);
|
||||
|
||||
const items: Record<string, JsonValue>[] = [];
|
||||
let byte_limited = false;
|
||||
|
||||
for (
|
||||
let index = view.offset;
|
||||
index < rows.length && items.length < view.limit;
|
||||
index++
|
||||
) {
|
||||
const { row, source_index } = rows[index]!;
|
||||
const item = project(row, source_index);
|
||||
const addedBytes =
|
||||
jsonValueByteLength(item, PAGE_BYTES - bytes) +
|
||||
(items.length > 0 ? 1 : 0);
|
||||
if (bytes + addedBytes > PAGE_BYTES) {
|
||||
if (items.length === 0) return tooLarge(view.collection);
|
||||
byte_limited = true;
|
||||
break;
|
||||
}
|
||||
bytes += addedBytes;
|
||||
items.push(item);
|
||||
}
|
||||
|
||||
const coverage = pageViewCoverage(view.offset, items.length, rows.length);
|
||||
const result: UnsignedAnalysisView = {
|
||||
...shared,
|
||||
kind: "web-bundle" as const,
|
||||
view,
|
||||
completeness: report.completeness,
|
||||
items: webBundleViewItemsSchema.parse(items),
|
||||
summary: null,
|
||||
byte_limited,
|
||||
coverage,
|
||||
};
|
||||
|
||||
return jsonValueByteLength(
|
||||
result as unknown as JsonValue,
|
||||
PROJECTION_BYTES,
|
||||
) > PROJECTION_BYTES
|
||||
? tooLarge(view.collection)
|
||||
: ok(result);
|
||||
};
|
||||
|
||||
if (view.collection === "endpoints") {
|
||||
const indices =
|
||||
view.indices !== undefined ? new Set(view.indices) : undefined;
|
||||
const values = view.values !== undefined ? new Set(view.values) : undefined;
|
||||
const scriptKeys =
|
||||
view.script_keys !== undefined ? new Set(view.script_keys) : undefined;
|
||||
const mechanisms =
|
||||
view.mechanisms !== undefined ? new Set(view.mechanisms) : undefined;
|
||||
|
||||
const matched: {
|
||||
readonly row: (typeof report.observations.endpoints)[number];
|
||||
readonly source_index: number;
|
||||
}[] = [];
|
||||
for (const [index, row] of report.observations.endpoints.entries()) {
|
||||
if (indices !== undefined && !indices.has(index)) continue;
|
||||
if (values !== undefined && !values.has(row.value)) continue;
|
||||
if (scriptKeys !== undefined && !scriptKeys.has(row.location.script_key))
|
||||
continue;
|
||||
if (mechanisms !== undefined && !mechanisms.has(row.mechanism)) continue;
|
||||
matched.push({ row, source_index: index });
|
||||
}
|
||||
|
||||
return deliverRows(
|
||||
matched as unknown as readonly {
|
||||
readonly row: Record<string, JsonValue>;
|
||||
readonly source_index: number;
|
||||
}[],
|
||||
(row, source_index) => ({
|
||||
source_index,
|
||||
...choose(row, view.fields),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
const indices =
|
||||
view.indices !== undefined ? new Set(view.indices) : undefined;
|
||||
const kinds = view.kinds !== undefined ? new Set(view.kinds) : undefined;
|
||||
const values = view.values !== undefined ? new Set(view.values) : undefined;
|
||||
const scriptKeys =
|
||||
view.script_keys !== undefined ? new Set(view.script_keys) : undefined;
|
||||
|
||||
const matched: {
|
||||
readonly row: (typeof report.inferences)[number];
|
||||
readonly source_index: number;
|
||||
}[] = [];
|
||||
for (const [index, row] of report.inferences.entries()) {
|
||||
if (indices !== undefined && !indices.has(index)) continue;
|
||||
if (kinds !== undefined && !kinds.has(row.kind)) continue;
|
||||
if (values !== undefined && !values.has(row.value)) continue;
|
||||
if (
|
||||
scriptKeys !== undefined &&
|
||||
!row.basis.some((b) => scriptKeys.has(b.script_key))
|
||||
)
|
||||
continue;
|
||||
matched.push({ row, source_index: index });
|
||||
}
|
||||
|
||||
return deliverRows(
|
||||
matched as unknown as readonly {
|
||||
readonly row: Record<string, JsonValue>;
|
||||
readonly source_index: number;
|
||||
}[],
|
||||
(row, source_index) => ({
|
||||
source_index,
|
||||
...choose(row, view.fields),
|
||||
}),
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,353 @@
|
||||
import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
|
||||
import { writeFile, unlink, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect, it, onTestFinished } from "vitest";
|
||||
import { createServer } from "../../../src/server/createServer.js";
|
||||
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
|
||||
import {
|
||||
webBundlePageViewResultSchema,
|
||||
webBundleSummaryViewResultSchema,
|
||||
} from "../../../src/domain/analysisView/analysisView.js";
|
||||
import type { WebBundleAnalysis } from "../../../src/domain/webBundleAnalysis.js";
|
||||
import { createTestBinarySession } from "../../fixtures/binarySession.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { createMcpSchemaValidation } from "../../fixtures/mcpSchemaValidation.js";
|
||||
import { advertisedMcpToolSchemas } from "../../fixtures/advertisedMcpTool.js";
|
||||
|
||||
const sampleWebBundleAnalysis = (): WebBundleAnalysis => ({
|
||||
capture: {
|
||||
target_url: "https://example.test/app",
|
||||
target_origin: "https://example.test",
|
||||
scripts_observed: 3,
|
||||
scripts_analyzed: 2,
|
||||
source_artifacts: [],
|
||||
},
|
||||
observations: {
|
||||
chunks: {
|
||||
nodes: [
|
||||
{
|
||||
script_key: "app.js",
|
||||
url: "https://example.test/app.js",
|
||||
artifact_sha256: "a".repeat(64),
|
||||
bytes: 1200,
|
||||
},
|
||||
{
|
||||
script_key: "vendor.js",
|
||||
url: "https://example.test/vendor.js",
|
||||
artifact_sha256: "b".repeat(64),
|
||||
bytes: 4500,
|
||||
},
|
||||
],
|
||||
edges: [],
|
||||
},
|
||||
routes: [],
|
||||
endpoints: [
|
||||
{
|
||||
value: "/api/users",
|
||||
mechanism: "call:fetch",
|
||||
location: { script_key: "app.js", line: 10, column: 2 },
|
||||
},
|
||||
{
|
||||
value: "/api/users",
|
||||
mechanism: "call:fetch",
|
||||
location: { script_key: "vendor.js", line: 42, column: 8 },
|
||||
},
|
||||
{
|
||||
value: "/api/orders",
|
||||
mechanism: "call:sendBeacon",
|
||||
location: { script_key: "app.js", line: 85, column: 4 },
|
||||
},
|
||||
],
|
||||
webmcp_declarations: [],
|
||||
source_maps: {
|
||||
status: "not_requested",
|
||||
requested: 0,
|
||||
processed: 0,
|
||||
items: [],
|
||||
},
|
||||
},
|
||||
inferences: [
|
||||
{
|
||||
kind: "endpoint_candidate",
|
||||
value: "/api/dynamic-user",
|
||||
confidence: "low",
|
||||
basis: [
|
||||
{
|
||||
script_key: "app.js",
|
||||
artifact_sha256: "a".repeat(64),
|
||||
line: 25,
|
||||
column: 4,
|
||||
detector: "call:client.get:unverified-network-receiver",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
kind: "vendor_fingerprint",
|
||||
value: "React",
|
||||
confidence: "high",
|
||||
basis: [
|
||||
{
|
||||
script_key: "vendor.js",
|
||||
artifact_sha256: "b".repeat(64),
|
||||
line: 1,
|
||||
column: 0,
|
||||
detector: "vendor:react",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
unknowns: [
|
||||
{
|
||||
dimension: "network",
|
||||
reason: "Unverified client receiver",
|
||||
affected_script_keys: ["app.js"],
|
||||
},
|
||||
],
|
||||
completeness: {
|
||||
status: "partial",
|
||||
parsed_scripts: 2,
|
||||
parse_failures: 1,
|
||||
visited_ast_nodes: 150,
|
||||
},
|
||||
limitations: ["Script evaluation was omitted"],
|
||||
});
|
||||
|
||||
const makeParent = () =>
|
||||
createEvidence(
|
||||
undefined,
|
||||
{ id: "rea-browser", name: "REA Browser", version: "6.8.0" },
|
||||
{
|
||||
operation: "analyze_web_bundle",
|
||||
parameters: { target_url: "https://example.test/app" },
|
||||
result: sampleWebBundleAnalysis(),
|
||||
},
|
||||
);
|
||||
|
||||
const connect = async () => {
|
||||
const schemaValidation = createMcpSchemaValidation();
|
||||
const session = createTestBinarySession(() => {
|
||||
throw new Error("selected views must not start a deep provider");
|
||||
});
|
||||
const server = createServer(
|
||||
{ kind: "session", session },
|
||||
{
|
||||
environment: { ...process.env, REA_MCP_INPUT_SCHEMA_PROFILE: "full" },
|
||||
},
|
||||
);
|
||||
const client = new Client(
|
||||
{ name: "web-bundle-view-mcp", version: "1" },
|
||||
{ jsonSchemaValidator: schemaValidation },
|
||||
);
|
||||
onTestFinished(async () => {
|
||||
await client.close();
|
||||
await server.close();
|
||||
await session.close();
|
||||
});
|
||||
const [clientTransport, serverTransport] =
|
||||
InMemoryTransport.createLinkedPair();
|
||||
await server.connect(serverTransport);
|
||||
await client.connect(clientTransport);
|
||||
const schemas = await advertisedMcpToolSchemas(
|
||||
client,
|
||||
"inspect_analysis_view",
|
||||
schemaValidation.assertions,
|
||||
);
|
||||
return { client, session, schemas };
|
||||
};
|
||||
|
||||
it("projects web bundle summary views over MCP using retained and inline sources", async () => {
|
||||
const { client, session, schemas } = await connect();
|
||||
const parent = makeParent();
|
||||
expect(session.recordEvidence(parent).ok).toBe(true);
|
||||
|
||||
const summaryCall = await client.callTool({
|
||||
...schemas.toolDefinition,
|
||||
arguments: {
|
||||
source: { kind: "retained-evidence", evidence_id: parent.evidence_id },
|
||||
view: { kind: "summary" },
|
||||
},
|
||||
});
|
||||
expect(summaryCall.isError, JSON.stringify(summaryCall)).not.toBe(true);
|
||||
expect(
|
||||
schemas.ajv.validate(schemas.outputSchema, summaryCall.structuredContent),
|
||||
JSON.stringify(schemas.ajv.errors),
|
||||
).toBe(true);
|
||||
const summaryResult = webBundleSummaryViewResultSchema.parse(
|
||||
parseEvidence(summaryCall.structuredContent).normalized_result,
|
||||
);
|
||||
expect(summaryResult.kind).toBe("summary");
|
||||
expect(summaryResult.parent_operation).toBe("analyze_web_bundle");
|
||||
expect(summaryResult.summary).toMatchObject({
|
||||
target_url: "https://example.test/app",
|
||||
counts: { endpoints: 3, inferences: 2, endpoint_candidates: 1 },
|
||||
completeness: { status: "partial" },
|
||||
});
|
||||
|
||||
const collectionSummaryCall = await client.callTool({
|
||||
...schemas.toolDefinition,
|
||||
arguments: {
|
||||
source: { kind: "inline", evidence: parent },
|
||||
view: { kind: "web-bundle", collection: "summary" },
|
||||
},
|
||||
});
|
||||
expect(collectionSummaryCall.isError).not.toBe(true);
|
||||
expect(
|
||||
schemas.ajv.validate(
|
||||
schemas.outputSchema,
|
||||
collectionSummaryCall.structuredContent,
|
||||
),
|
||||
JSON.stringify(schemas.ajv.errors),
|
||||
).toBe(true);
|
||||
const collectionSummaryResult = webBundlePageViewResultSchema.parse(
|
||||
parseEvidence(collectionSummaryCall.structuredContent).normalized_result,
|
||||
);
|
||||
expect(collectionSummaryResult.kind).toBe("web-bundle");
|
||||
expect(collectionSummaryResult.summary).toMatchObject({
|
||||
target_url: "https://example.test/app",
|
||||
counts: { endpoints: 3, inferences: 2, endpoint_candidates: 1 },
|
||||
});
|
||||
});
|
||||
|
||||
it("pages web bundle endpoints and inferences over MCP using evidence-file source", async () => {
|
||||
const { client, schemas } = await connect();
|
||||
const parent = makeParent();
|
||||
const tempDir = await createTestTempDirectory("rea-webbundle-mcp-");
|
||||
const evidenceFile = join(tempDir, "webbundle-evidence.json");
|
||||
await writeFile(evidenceFile, JSON.stringify(parent));
|
||||
|
||||
try {
|
||||
const page1Call = await client.callTool({
|
||||
...schemas.toolDefinition,
|
||||
arguments: {
|
||||
source: { kind: "evidence-file", path: evidenceFile },
|
||||
view: {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
offset: 0,
|
||||
limit: 2,
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(page1Call.isError).not.toBe(true);
|
||||
expect(
|
||||
schemas.ajv.validate(schemas.outputSchema, page1Call.structuredContent),
|
||||
JSON.stringify(schemas.ajv.errors),
|
||||
).toBe(true);
|
||||
const page1 = webBundlePageViewResultSchema.parse(
|
||||
parseEvidence(page1Call.structuredContent).normalized_result,
|
||||
);
|
||||
expect(page1.items).toHaveLength(2);
|
||||
expect(page1.items[0]).toMatchObject({
|
||||
source_index: 0,
|
||||
value: "/api/users",
|
||||
});
|
||||
expect(page1.items[1]).toMatchObject({
|
||||
source_index: 1,
|
||||
value: "/api/users",
|
||||
});
|
||||
expect(page1.coverage).toMatchObject({
|
||||
status: "page",
|
||||
examined: 2,
|
||||
total: 3,
|
||||
next_offset: 2,
|
||||
exhausted: false,
|
||||
});
|
||||
expect(page1.completeness.status).toBe("partial");
|
||||
|
||||
const page2Call = await client.callTool({
|
||||
...schemas.toolDefinition,
|
||||
arguments: {
|
||||
source: { kind: "evidence-file", path: evidenceFile },
|
||||
view: {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
offset: 2,
|
||||
limit: 2,
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(page2Call.isError).not.toBe(true);
|
||||
expect(
|
||||
schemas.ajv.validate(schemas.outputSchema, page2Call.structuredContent),
|
||||
JSON.stringify(schemas.ajv.errors),
|
||||
).toBe(true);
|
||||
const page2 = webBundlePageViewResultSchema.parse(
|
||||
parseEvidence(page2Call.structuredContent).normalized_result,
|
||||
);
|
||||
expect(page2.items).toHaveLength(1);
|
||||
expect(page2.items[0]).toMatchObject({
|
||||
source_index: 2,
|
||||
value: "/api/orders",
|
||||
});
|
||||
expect(page2.coverage).toMatchObject({
|
||||
status: "page",
|
||||
examined: 1,
|
||||
total: 3,
|
||||
next_offset: null,
|
||||
exhausted: true,
|
||||
});
|
||||
|
||||
const candidatesCall = await client.callTool({
|
||||
...schemas.toolDefinition,
|
||||
arguments: {
|
||||
source: { kind: "evidence-file", path: evidenceFile },
|
||||
view: {
|
||||
kind: "web-bundle",
|
||||
collection: "inferences",
|
||||
kinds: ["endpoint_candidate"],
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(candidatesCall.isError).not.toBe(true);
|
||||
expect(
|
||||
schemas.ajv.validate(
|
||||
schemas.outputSchema,
|
||||
candidatesCall.structuredContent,
|
||||
),
|
||||
JSON.stringify(schemas.ajv.errors),
|
||||
).toBe(true);
|
||||
const candidatesPage = webBundlePageViewResultSchema.parse(
|
||||
parseEvidence(candidatesCall.structuredContent).normalized_result,
|
||||
);
|
||||
expect(candidatesPage.items).toEqual([
|
||||
expect.objectContaining({
|
||||
source_index: 0,
|
||||
kind: "endpoint_candidate",
|
||||
value: "/api/dynamic-user",
|
||||
confidence: "low",
|
||||
}),
|
||||
]);
|
||||
|
||||
const emptyCall = await client.callTool({
|
||||
...schemas.toolDefinition,
|
||||
arguments: {
|
||||
source: { kind: "evidence-file", path: evidenceFile },
|
||||
view: {
|
||||
kind: "web-bundle",
|
||||
collection: "endpoints",
|
||||
offset: 100,
|
||||
limit: 10,
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(emptyCall.isError).not.toBe(true);
|
||||
expect(
|
||||
schemas.ajv.validate(schemas.outputSchema, emptyCall.structuredContent),
|
||||
JSON.stringify(schemas.ajv.errors),
|
||||
).toBe(true);
|
||||
const emptyPage = webBundlePageViewResultSchema.parse(
|
||||
parseEvidence(emptyCall.structuredContent).normalized_result,
|
||||
);
|
||||
expect(emptyPage.items).toEqual([]);
|
||||
expect(emptyPage.coverage).toMatchObject({
|
||||
status: "empty",
|
||||
examined: 0,
|
||||
total: 3,
|
||||
next_offset: null,
|
||||
exhausted: true,
|
||||
});
|
||||
} finally {
|
||||
await unlink(evidenceFile).catch(() => {});
|
||||
await rm(tempDir, { recursive: true, force: true }).catch(() => {});
|
||||
}
|
||||
});
|
||||
Reference in new issue
Block a user