Files
reverse-engineering-anything/src/domain/webNetworkCapture.ts
T
morluto 2430b081c9 refactor(providers): bind provider refusals' per-call controls instead of naming CLI flags (#11455)
* refactor(native): bind producer_limits and max_input_bytes as the per-call controls of pwntools and Apple dispatch refusals

producerLimitControl in src/domain/producerResourceLimits.ts names one producer budget as producer_limits.<field>; the shared provider outcome classifier now carries a resource signal's limit. Pwntools file-size, CPU and output refusals and the Apple dispatch input refusal bind their control, and their remediation no longer names CLI flags.

* fix(native): report the selected output allowance as the omitted file_bytes default

* fix(native): bind producer_limits.diagnostic_bytes as the pwntools command output-limit control

* refactor(rust): bind producer_limits as the per-call control of Rust input and report refusals (#11458)

* refactor(rust): bind producer_limits as the per-call control of Rust input and report refusals

* test(rust): cover the complete-report refusal's producer_limits control

* refactor(browser): bind caller-selectable browser deadlines as per-call controls instead of naming CLI flags (#11466)

* refactor(rust): bind producer_limits as the per-call control of Rust input and report refusals

* test(rust): cover the complete-report refusal's producer_limits control

* refactor(browser): bind caller-selectable browser deadlines as per-call controls instead of naming CLI flags

deadlineParameterLimit in src/domain/analysisResourceLimits.ts owns the millisecond deadline control that nativeDeadlineRecovery and the browser recoveries share. CDP command and discovery, web bundle per-script and total, network capture and source-map decode deadlines now carry it; browserDeadlineIncrease no longer takes CLI flag text.

* fix(browser): record a per-script deadline's parameter control in partial web bundle results

* refactor(providers): bind Wabt, SQLite, EVMole, Apktool and Go controls instead of naming CLI commands (#11472)

* refactor(providers): bind Wabt, SQLite, EVMole and Apktool deadline and byte controls instead of naming CLI commands

These remediation strings named rea commands and flags through template text the cli-option guard did not count. Wabt byte refusals and output limits bind max_input_bytes, max_tool_bytes or max_output_bytes while a larger value is selectable; SQLite worker_timeout_ms, EVMole timeout_ms and Apktool decode_timeout_ms bind through deadlineParameterLimit; the shared provider outcome classifier now carries an output limit's control.

* refactor(go): bind Go analysis budgets as per-call controls instead of naming CLI flags

* fix(providers): keep the typed primary when Wabt, EVMole or pwntools command cleanup fails

* ci(guards): count template-built CLI option and command text outside the CLI layer (#11478)

* refactor(providers): bind Wabt, SQLite, EVMole and Apktool deadline and byte controls instead of naming CLI commands

These remediation strings named rea commands and flags through template text the cli-option guard did not count. Wabt byte refusals and output limits bind max_input_bytes, max_tool_bytes or max_output_bytes while a larger value is selectable; SQLite worker_timeout_ms, EVMole timeout_ms and Apktool decode_timeout_ms bind through deadlineParameterLimit; the shared provider outcome classifier now carries an output limit's control.

* refactor(go): bind Go analysis budgets as per-call controls instead of naming CLI flags

* fix(providers): keep the typed primary when Wabt, EVMole or pwntools command cleanup fails

* ci(guards): count template-built CLI option and command text outside the CLI layer

The cli-option-text-outside-cli pattern now also matches "(CLI ...", "on the CLI", "or use rea ..." and backticked `rea <command>` references, so remediation built from a template such as --${name} or rea inspect-... <path> is counted. Doctor.ts is exempt: setup and doctor run only through the CLI. The baseline records the newly counted pre-existing sites (22 -> 45 across 21 files).

* ci(guards): extract the CLI text matcher, exempt host-only commands with written reasons, and test each spelling

* ci(guards): exempt a host-only command and its own arguments in any matched CLI phrase

* fix(native): bind producer_limits.output_bytes for oversized pwndbg debugger replies through one owner

* fix(cli): name an Evidence target source failure's control by the source option that sets it

* feat(cli): render the flags of result-recorded controls in the full-output envelope (#11507)

* feat(cli): render the flags of result-recorded controls in the full-output envelope

* fix(cli): collect result controls only for a full-output envelope, in its requested format, past undefined values

* ci(guards): end a host-only command's arguments before the next CLI command

* refactor(reverse): record the output budget that stopped a one-shot command as a parameter control (#11528)

* fix(cli): read result controls only at positions the output schema declares as recorded controls

* refactor(providers): record the controls that bounded Rust compiler paths and kernel module text instead of naming CLI options (#11532)

* refactor(rust): record the control that bounded compiler-path coverage instead of naming CLI options

* refactor(native): record the controls that omitted kernel module .modinfo text instead of naming CLI options (#11534)

* fix(rust): describe compiler coverage from the retained result after report trimming

* refactor(providers): record managed representation and JavaScript semantic ceiling controls instead of naming CLI options (#11537)

* refactor(managed): record the representation budget on a resource-limit issue instead of naming its CLI option

* refactor(javascript): record the semantic retention ceilings that stopped a graph instead of naming CLI options (#11540)

* fix(cli): name a composed command's source options for the controls its result records

* fix(cli): read result controls from the delivered Evidence when a summary wraps it

* fix(rust): keep the report-byte control recorded when trimming omits every compiler path

When the complete-report allowance left room for no compiler path, rustc_commit became null and its coverage.limit was lost, so --full-output could not name the option that raises the allowance. The report now records rustc_commit_omission { reason, limit } outside the dropped record, and the summary view carries it.

* refactor(flutter): record the string-pool budgets that left a projection partial instead of naming the CLI option (#11541)

* refactor(flutter): record the string-pool budgets that left a projection partial instead of naming the CLI option

* test(flutter): assert the complete set of recorded string-pool budgets

* feat(cli): name the options of controls a failure's partial Evidence records (#11557)

meta.cli_controls was collected only from a successful result. A failure whose partial observation is Evidence of the command's own operation is now read the same way, at the positions the output schema declares as recorded controls; Evidence of another operation or a retained reference is not read. When --evidence_output publishes the partial, the delivery records it before replacing it.

* refactor(javascript): record the evaluator budgets that limited sources instead of naming the CLI option (#11564)

Per-value unknown reasons and the Electron wrapper limitation named --semantic-precision. The static examination now records the semantic_precision controls that limited a source (examination.controls), and analyze_javascript_application's semantic_graph.coverage.controls records every evaluator budget that limited any source (examination and semantic IR coverage), beside the retention ceilings. The CLI renders --semantic-precision from those records in meta.cli_controls; the reason texts name only the contract field.
2026-11-16 07:12:16 +08:00

440 lines
17 KiB
TypeScript

import { deadlineParameterLimit } from "./analysisResourceLimits.js";
import { createHash } from "node:crypto";
import { canonicalBase64Schema } from "./stringPatterns.js";
import { z } from "zod";
import { knownCoverageStatusSchema } from "./coverage.js";
import {
browserDeadlineIncrease,
type BrowserDeadlineRecovery,
browserDeadlineRetry,
optionalBrowserDeadlineMs,
selectedBrowserDeadlineMs,
} from "./browserDeadline.js";
import { jsonValueSchema, MAX_JSON_DEPTH } from "./jsonValue.js";
import { absoluteLocalPathStringSchema } from "./localPath.js";
import {
protobufDecodingSchema,
protobufOptionsSchema,
} from "./protobufWire.js";
import { qlogCaptureFactsSchema } from "./qlogCapture.js";
import { canBorrowParsedJsonSnapshot } from "./immutableJson.js";
/** Default historical evidence budgets; exceeding a budget returns no partial capture. */
export const WEB_NETWORK_CAPTURE_LIMITS = {
inputBytes: 32 * 1024 * 1024,
outputBytes: 96 * 1024 * 1024,
timeoutMs: 30_000,
depth: 64,
} as const;
// Native/SAZ projections start at the reported record root. The complete
// reply (value.records[0].reported) and Evidence envelope have four enclosing
// edges; count them before admitting parser/projection nesting.
const CAPTURE_REPORTED_RECORD_DEPTH = [
"normalized_result",
"records",
0,
"reported",
].length;
const MAX_WEB_NETWORK_CAPTURE_NESTING_DEPTH =
MAX_JSON_DEPTH - CAPTURE_REPORTED_RECORD_DEPTH;
/** One admitted capture's complete-input, complete-reply and parser policy. */
export const webNetworkCaptureBudgetsSchema = z.strictObject({
inputBytes: z.int().min(1),
outputBytes: z.int().min(1),
depth: z.int().min(1).max(MAX_WEB_NETWORK_CAPTURE_NESTING_DEPTH),
});
export type WebNetworkCaptureBudgets = z.infer<
typeof webNetworkCaptureBudgetsSchema
>;
export const selectedWebNetworkCaptureBudgets = (input: {
readonly max_input_bytes?: number | undefined;
readonly max_reply_bytes?: number | undefined;
readonly max_nesting_depth?: number | undefined;
}): WebNetworkCaptureBudgets => ({
inputBytes: input.max_input_bytes ?? WEB_NETWORK_CAPTURE_LIMITS.inputBytes,
outputBytes: input.max_reply_bytes ?? WEB_NETWORK_CAPTURE_LIMITS.outputBytes,
depth: input.max_nesting_depth ?? WEB_NETWORK_CAPTURE_LIMITS.depth,
});
/** Explicit per-call budget facts; these controls do not mutate runtime defaults. */
export const webNetworkCaptureBudgetFacts = (
input: Parameters<typeof selectedWebNetworkCaptureBudgets>[0],
parameter: "max_input_bytes" | "max_reply_bytes" | "max_nesting_depth",
) => {
const budgets = selectedWebNetworkCaptureBudgets(input);
const field =
parameter === "max_input_bytes"
? "inputBytes"
: parameter === "max_reply_bytes"
? "outputBytes"
: "depth";
const effective = budgets[field];
return {
parameter,
effective_value: effective,
default_value: WEB_NETWORK_CAPTURE_LIMITS[field],
source: input[parameter] === undefined ? "default" : "request",
unit: parameter === "max_nesting_depth" ? "levels" : "bytes",
minimum: 1,
maximum:
parameter === "max_nesting_depth"
? MAX_WEB_NETWORK_CAPTURE_NESTING_DEPTH
: Number.MAX_SAFE_INTEGER,
applies_at: "next_operation",
update_mode: "parameter" as const,
pending_restart: false as const,
};
};
/** Recovery follows the owning budget, rather than a different deadline or heap. */
export const webNetworkCaptureBudgetRecovery = (
input: Parameters<typeof selectedWebNetworkCaptureBudgets>[0],
parameter: Parameters<typeof webNetworkCaptureBudgetFacts>[1],
): string => {
const facts = webNetworkCaptureBudgetFacts(input, parameter);
const next =
facts.effective_value < facts.maximum
? `Increase ${parameter} above ${String(facts.effective_value)} on the next inspect_web_network_capture call`
: `${parameter} is at its maximum; select a smaller complete producer export when that preserves the required observation`;
return `${next}. Effective ${parameter}: ${String(facts.effective_value)} ${facts.unit} (source ${facts.source}, default ${String(facts.default_value)}, floor ${String(facts.minimum)}, maximum ${String(facts.maximum)}). No REA restart is required. Other capture budgets, worker limits and client limits remain separate. Selection and delivery do not recreate a completed report from a refused decode.`;
};
/** Capture deadline used when capture_timeout_ms is omitted. */
export const DEFAULT_WEB_NETWORK_CAPTURE_TIMEOUT_MS =
WEB_NETWORK_CAPTURE_LIMITS.timeoutMs;
export const selectedWebNetworkCaptureTimeoutMs = (
selected: number | undefined,
): number =>
selectedBrowserDeadlineMs(selected, DEFAULT_WEB_NETWORK_CAPTURE_TIMEOUT_MS);
/** Caller-visible capture deadline; byte/depth budgets have separate per-call controls. */
export const webNetworkCaptureTimeoutRecovery = (
timeoutMs: number,
): BrowserDeadlineRecovery => ({
userMessage: `inspect_web_network_capture reached its ${String(timeoutMs)} ms capture deadline (deadlineExpired) for capture_timeout_ms. No partial capture is returned. This is not the MCP client deadline.`,
limit: deadlineParameterLimit(
"capture_timeout_ms",
timeoutMs,
DEFAULT_WEB_NETWORK_CAPTURE_TIMEOUT_MS,
),
remediationAction: `${browserDeadlineIncrease("capture_timeout_ms", timeoutMs, DEFAULT_WEB_NETWORK_CAPTURE_TIMEOUT_MS)} ${browserDeadlineRetry} The separate max_input_bytes, max_reply_bytes and max_nesting_depth controls apply to the next call without restarting REA; capture_timeout_ms does not raise them.`,
});
export const WEB_NETWORK_CAPTURE_FORMATS = [
"har",
"mitmproxy",
"saz",
"qlog",
] as const;
/** Inspect an explicit retained capture without contacting its recorded destinations. */
export const inspectWebNetworkCaptureInputSchema = z.strictObject({
capture_path: absoluteLocalPathStringSchema.describe(
"Absolute local HAR, mitmproxy, Fiddler SAZ or aioquic qlog capture path.",
),
format: z.enum(WEB_NETWORK_CAPTURE_FORMATS),
record_ordinals: z
.array(z.int().nonnegative())
.min(1)
.optional()
.describe(
"Zero-based producer record ordinals, in caller-selected order. Omit to inspect every record.",
),
max_input_bytes: webNetworkCaptureBudgetsSchema.shape.inputBytes
.optional()
.describe(
"Complete capture, expanded SAZ member and protobuf descriptor admission in bytes (default 33554432); applies to the next call without restart. Does not raise worker heap, reply or client limits.",
),
max_reply_bytes: webNetworkCaptureBudgetsSchema.shape.outputBytes
.optional()
.describe(
"Complete decoder reply and protobuf working representation allowance in bytes (default 100663296); applies to the next call without restart. Record selection occurs after complete decoding. Does not raise worker heap or client receive limits.",
),
max_nesting_depth: webNetworkCaptureBudgetsSchema.shape.depth
.optional()
.describe(
`Capture/parser nesting allowance (default 64, maximum ${String(MAX_WEB_NETWORK_CAPTURE_NESTING_DEPTH)}, reserving the enclosing reply/Evidence levels within REA JSON representation); applies on the next call without restart. Does not make malformed or unsupported structures valid.`,
),
protobuf: protobufOptionsSchema.optional(),
sensitive_values: z
.array(z.string().min(1))
.default([])
.describe(
"Literal text excluded from capture payloads, byte/numeric projections, producer coordinates, artifact paths/digests and diagnostic text. Declaration strings are never persisted in Evidence parameters. Canonical contract keys, enums and provider identities remain intact.",
),
capture_timeout_ms: optionalBrowserDeadlineMs(
`Capture wall-clock deadline in milliseconds (default ${String(DEFAULT_WEB_NETWORK_CAPTURE_TIMEOUT_MS)}), from decoder start through process exit. Omitted means ${String(DEFAULT_WEB_NETWORK_CAPTURE_TIMEOUT_MS)} and applies on the next capture. deadlineExpired returns no partial capture. Does not raise the separately selected max_input_bytes, max_reply_bytes or max_nesting_depth budgets.`,
),
});
/** Read declarations through their owning schema for service and adapter failures. */
export const captureSensitiveValues = (
rawInput: unknown,
): readonly string[] => {
const parsed =
inspectWebNetworkCaptureInputSchema.shape.sensitive_values.safeParse(
typeof rawInput === "object" &&
rawInput !== null &&
"sensitive_values" in rawInput
? rawInput.sensitive_values
: undefined,
);
return parsed.success ? parsed.data : [];
};
const pointerSchema = z
.string()
.describe("RFC 6901 pointer in the producer's reported object.");
const redactionSchema = z.union([
z.strictObject({
pointer: pointerSchema,
reason: z.enum(["transport-credential", "explicit-sensitive-value"]),
}),
z.strictObject({
pointer: pointerSchema,
reason: z.literal("explicit-sensitive-value"),
scope: z
.literal("property-name")
.describe(
"An omitted property name; pointer identifies its actual parent, not an invented child coordinate.",
),
}),
]);
const binaryBase = {
pointer: pointerSchema,
representation: z.enum(["producer-bytes", "har-base64-content"]),
};
const retainedBinaryDescriptorSchema = z.strictObject({
...binaryBase,
state: z.literal("retained"),
bytes: z.int().nonnegative(),
sha256: z.string().regex(/^[0-9a-f]{64}$/),
});
const redactedBinaryDescriptorSchema = z.strictObject({
...binaryBase,
state: z.literal("redacted"),
bytes: z.null(),
sha256: z.null(),
});
/** Exact byte facts shared by complete records and previews that omit content. */
export const webNetworkCaptureBinaryDescriptorSchema = z.discriminatedUnion(
"state",
[retainedBinaryDescriptorSchema, redactedBinaryDescriptorSchema],
);
const binarySchema = z.discriminatedUnion("state", [
retainedBinaryDescriptorSchema
.extend({ content_base64: canonicalBase64Schema })
.superRefine((field, context) => {
const encoded = field.content_base64;
const decodedBytes = Buffer.byteLength(encoded, "base64");
if (field.bytes !== decodedBytes) {
context.addIssue({
code: "custom",
path: ["bytes"],
message:
"Retained binary byte count differs from its canonical content.",
});
return;
}
// Quartet-aligned windows authenticate every retained byte without
// allocating a second whole body while the capture remains live.
const hash = createHash("sha256");
for (let offset = 0; offset < encoded.length; offset += 65536)
hash.update(
Buffer.from(encoded.slice(offset, offset + 65536), "base64"),
);
if (hash.digest("hex") !== field.sha256)
context.addIssue({
code: "custom",
path: ["sha256"],
message:
"Retained binary SHA-256 differs from its canonical content.",
});
}),
redactedBinaryDescriptorSchema.extend({ content_base64: z.null() }),
]);
const numberSchema = z.strictObject({
pointer: pointerSchema,
producer_type: z.enum(["json-number", "integer", "float"]),
literal: z.string(),
});
/** Preserve producer representations with independent sidecars rather than invented browser IDs. */
export const webNetworkCaptureRecordSchema = z.strictObject({
ordinal: z.int().nonnegative(),
location: z.discriminatedUnion("kind", [
z.strictObject({
kind: z.literal("unknown"),
reason: z.literal("explicit-sensitive-value"),
}),
z.strictObject({ kind: z.literal("json-pointer"), pointer: pointerSchema }),
z.strictObject({
kind: z.literal("archive-members"),
members: z.array(z.string()).min(1),
}),
z.strictObject({
kind: z.literal("byte-range"),
offset: z.int().nonnegative(),
bytes: z.int().positive(),
}),
]),
reported: jsonValueSchema,
binary_fields: z.array(binarySchema).superRefine((fields, context) => {
const pointers = new Set<string>();
for (const [index, field] of fields.entries()) {
if (pointers.has(field.pointer)) {
context.addIssue({
code: "custom",
path: [index, "pointer"],
message:
"Each retained source coordinate can declare binary facts only once.",
});
return;
}
pointers.add(field.pointer);
}
}),
numeric_literals: z.array(numberSchema),
redactions: z.array(redactionSchema),
limitations: z.array(z.string()),
protobuf: z
.array(protobufDecodingSchema)
.exactOptional()
.describe(
"Flat decoding facts for explicitly selected records. Field indices are local to a body; offset_basis states whether ranges refer to retained or base64-decoded bytes. Interpretations without a caller-supplied schema are candidates.",
),
});
/** The selected offline parser profile, independent of authenticated producer identity. */
export const webNetworkCaptureDecoderSchema = z.strictObject({
id: z.string().min(1),
name: z.string().min(1),
version: z.string().min(1),
});
/** Source coverage is shared by every offline decoder. */
export const protobufCaptureCoverageSchema = z
.strictObject({
status: knownCoverageStatusSchema,
selected_records: z.number().int().nonnegative(),
inspected_records: z.number().int().nonnegative(),
})
.superRefine((coverage, context) => {
if (
coverage.inspected_records > coverage.selected_records ||
(coverage.status === "complete" &&
coverage.inspected_records !== coverage.selected_records)
)
context.addIssue({
code: "custom",
path: ["status"],
message: "Protobuf coverage contradicts its inspected selection.",
});
});
const payloadShape = {
protobuf_coverage: protobufCaptureCoverageSchema.exactOptional(),
decoder: webNetworkCaptureDecoderSchema,
container: z.strictObject({
reported: jsonValueSchema,
numeric_literals: z.array(numberSchema),
redactions: z.array(redactionSchema),
records_pointer: pointerSchema.nullable(),
}),
total_records: z.int().nonnegative(),
records: z.array(webNetworkCaptureRecordSchema),
};
const refineCoverage = (
capture: { total_records: number; records: { ordinal: number }[] },
context: z.RefinementCtx,
): void => {
const seen = new Set<number>();
for (const [index, record] of capture.records.entries()) {
if (record.ordinal >= capture.total_records || seen.has(record.ordinal)) {
context.addIssue({
code: "custom",
path: ["records", index, "ordinal"],
message:
"Retained ordinals must be unique and within the declared capture total.",
});
return;
}
seen.add(record.ordinal);
}
};
/** Strict alternatives prevent undefined-valued facets from entering JSON Evidence. */
export const webNetworkCapturePayloadSchema = z
.union([
z.strictObject({ ...payloadShape, qlog: qlogCaptureFactsSchema }),
z.strictObject(payloadShape),
])
.superRefine(refineCoverage);
/** Caller-selected file identity; explicit exclusions remain unknown. */
export const webNetworkCaptureArtifactSchema = z.strictObject({
path: z
.string()
.describe(
"Selected artifact path; empty when explicitly excluded from Evidence.",
),
sha256: z
.string()
.regex(/^[0-9a-f]{64}$/)
.nullable()
.describe("Observed SHA-256; null only when explicitly excluded."),
bytes: z.int().nonnegative(),
});
const artifactShape = {
artifact: webNetworkCaptureArtifactSchema,
runtime_attribution: z.literal("unknown"),
limitations: z.array(z.string()),
};
/** Qlog transport facts occur exactly on the qlog alternative. */
export const webNetworkCaptureSchema = z
.discriminatedUnion("format", [
z.strictObject({
...payloadShape,
...artifactShape,
format: z.enum(["har", "mitmproxy", "saz"]),
}),
z.strictObject({
...payloadShape,
...artifactShape,
format: z.literal("qlog"),
qlog: qlogCaptureFactsSchema,
}),
])
.superRefine(refineCoverage);
export type InspectWebNetworkCaptureInput = z.output<
typeof inspectWebNetworkCaptureInputSchema
>;
export type WebNetworkCapture = z.output<typeof webNetworkCaptureSchema>;
export type WebNetworkCaptureRecord = z.output<
typeof webNetworkCaptureRecordSchema
>;
const validatedImmutableCaptures = new WeakSet<object>();
/** Reuse successful validation of the exact owned snapshot, without a retained copy. */
export const safeParseOwnedWebNetworkCapture = (
input: unknown,
): z.ZodSafeParseResult<WebNetworkCapture> => {
if (
typeof input === "object" &&
input !== null &&
validatedImmutableCaptures.has(input)
)
return { success: true, data: input as WebNetworkCapture };
const parsed = webNetworkCaptureSchema.safeParse(input);
if (parsed.success && canBorrowParsedJsonSnapshot(input, parsed.data)) {
// Borrow only when full validation preserves values, schema normalization
// and alias topology. Merely frozen or mutable objects never gain a proof.
validatedImmutableCaptures.add(input);
return { success: true, data: input };
}
return parsed;
};