Merge remote-tracking branch 'origin/main' into feat/environment-write-quality

This commit is contained in:
Edgar Arakelyan committed 2026-10-26 10:18:52 -07:00
commit 0ceff6fa69
164 files changed
+6292 -1705

No files matched your search

+7 -1
View File
@@ -443,7 +443,13 @@ The selected provider remains bound to the session until an explicit switch
or close. Provider failures are returned with their original reason. For an
`ambiguous` selection error, choose from `details.candidate_ids`; for
`provider_unavailable`, follow `message`, `remediation.action`, and the original
reasons in `details.rejections`. For Hopper, Ghidra, IDA, JADX, and Kuna, run
reasons in `details.rejections`. Each rejection names its `origin`: `availability`
entries describe local readiness failures, including missing configuration or an
invalid existing registration; they do not establish target support. `target`
entries record a refused kind, format, architecture, or role. An availability-only
message describes the recorded rejections without claiming that every provider
is unconfigured or that target support is known.
For Hopper, Ghidra, IDA, JADX, and Kuna, run
`rea doctor --provider ID --json` to diagnose the selected engine. Other adapters,
such as `pwntools-elf`, report their configuration requirements directly and are
not accepted by `doctor --provider`. See
@@ -1,5 +1,9 @@
# Native directory decompilation verification — 2026-10-23
The [2026-10-26 follow-up](https://github.com/morluto/rea/blob/72454004772f826440c5e21002f4dd52b4ad43c0/experiments/analysis-efficiency/decompile-directory.md)
adds a 14,338-function real-provider capacity run, cross-session project reuse,
and a correction preserving historical source Evidence through shared workflows.
The completed analyst task is to export the bound provider's function inventory
as one local pseudocode file per non-external function, with names, body ranges,
provider/profile identity and the limitations of each used observation. The
@@ -0,0 +1,149 @@
# Directory decompilation: capacity, resume and source Evidence
The directory-export feature requested in #6090 already shipped in #6170.
This follow-up verifies the current public workflow and fixes a source-provenance
gap found while combining it with existing Ghidra projects (#1048).
## Completed task and correctness
The analyst asks for all functions as local pseudocode files, without putting
their bodies in agent context. The committed `decompile-directory.mjs` driver
uses the production stdio MCP entrypoint and installed SDK. It discovers the
catalog, exports an initial chunk, closes/reopens the provider, resumes all
remaining work, and resumes again to verify settled files. It authenticates an
explicitly exported Evidence bundle and checks the original target/project.
Every written file is checked against its index SHA-256 and address/profile
header. Existing file hashes and modification times must survive both resumes.
Comparison facts include every address, name, status, body range and pseudocode
digest. An independent fresh-import investigation from #1048 additionally checks
the complete inventory and two nonempty pseudocode observations. This establishes
export preservation of provider observations; it does not establish discovery of
every actual function or equivalence to original source.
| Real artifact | Provider inventory | Written | External | Failed |
| ----------------------- | -----------------: | ------: | -------: | -----: |
| `/usr/bin/ls` | 652 | 541 | 111 | 0 |
| system `libcrypto.so.3` | 14,338 | 14,194 | 143 | 1 |
Target SHA-256 values are respectively
`cb30d69b24245bf2ecdc9e7f53bbad19159999970b6d82c0c00c7d32d9e37aa4` and
`aa01e614af634bc7c8b678c9250b5c4f613c61989bacf0a5d6aed6570e7dcabd`.
These installed ELF x86-64 binaries were analyzed statically, never executed.
The large run explicitly selected a five-second per-function deadline; function
`0x387f00` exceeded it, remained a recorded failure, and was retried on resume.
Later functions were still exported. No disappearance of that failure is claimed.
The resulting large tree occupies about 107 MiB on this filesystem.
## Source-provenance correction
Before this change, `BinarySession` retained provider `sourceEvidence` only inside
its snapshot-eligible branch. Ghidra procedure reads without a document selector
are not snapshot eligible. A composed directory export therefore lost saved
project reports even though the provider supplied them. The workflow also ignored
source-record IDs, and the function-dossier adapter copied observation fields
manually without those links.
The session now authenticates/retains source records independently of replay
eligibility. Plain application helpers also return authenticated immutable source
records with their outcome, including late cancellation. Both adapters explicitly
commit them before delivering links; an adapter without a retention owner refuses
the response. Source collection borrows ledger records without constructing or
hashing an intermediate bundle. The workflow links distinct source IDs without copying their payloads.
The function adapter uses the common observation-to-Evidence constructor,
preserving sources through normal and late-cancelled completion. Queries with no
source records avoid the extra recording path. Complete provider bodies remain
in the exported function files; this does not add per-function payload retention.
A real test project contains historical seed and disabled-analyzer reports saved
by REA's actual Ghidra pre-scripts. The unmodified runtime completed the export
but failed the provenance oracle: its bundle had two workflow records and no
historical source. The fixed runtime retains exactly one historical source and
links the workflow to it. All 652 inventory entries and the sampled pseudocode
still match fresh import; source project content stays unchanged. The report is
historical, with remaining original settings/completeness explicitly unknown.
## Measured costs
Baseline runtime: `19447516a4bdbc9437d112ebd3f12a5fd062db83`.
Initial fixed runtime: `f50763fd21527bdf3bab944633617b9df61dded4`, rebased on
main `f02df45a4`. Both use Node 24.22.1, locked dependencies, actual Ghidra 12.1.4
and JDK 21.0.12.1+1. The fixed driver later delegates transcript publication to
the existing cleanup/receipt owner; the protocol and task are unchanged.
| Completed task | Wall s | Family CPU s | Peak family GiB | Full-wire token estimate |
| --------------------------------- | ------: | -----------: | --------------: | -----------------------: |
| ls, before | 51.411 | 78.984 | 0.943 | 57,082 |
| ls, initial fix | 50.937 | 79.891 | 1.201 | 57,072 |
| ls with historical reports, fixed | 52.827 | 80.236 | 1.097 | 57,207 |
| libcrypto, baseline capacity run | 506.387 | 459.818 | 1.502 | 57,322 |
The plain ls pair has identical complete comparison facts. Runtime is similar;
the higher sampled memory in the fixed run is reported rather than treated as an
improvement. These sequential shared-host samples establish no generic speed or
memory ratio. [PR #7250](https://github.com/morluto/rea/pull/7250) records the final
large-run comparison and CI/review results.
The failed historical baseline is excluded from completed-task comparisons.
The complete MCP replies stayed around 11–12 KiB, including repeated textual and
structured representations, profile and caveats. The normalized result is only
the manifest. For example, the large continuation costs about 3,159 estimated
wire tokens; its complete task is 57,322 because discovery alone costs 47,049.
Both numbers include actual protocol content, rather than only pseudocode bytes.
Estimates use `cl100k_base` on the actual JSON-RPC messages, including discovery,
requests, responses and notifications, excluding only transport framing and the
driver's local transcript wrapper. They are not actual model billing. Host
projection, prompts/reasoning and later agent file reads remain unmeasured.
The historical source increases response token cost slightly while preserving
provenance. No token-reduction claim is made for this correctness fix.
CPU and memory cover the driver, REA and provider process family. Each run uses
CPU affinity 0,1 and an 8-GiB sampled family-RSS guard; Java heaps are 1536M for ls
and 3072M for crypto. RSS is sampled every 100 ms and summed: shared pages may be
counted repeatedly and short peaks may be missed. CPU is waited child usage.
Reported completed runs had no resource stop, OOM or cleanup failure.
Project preparation is excluded and documented separately in
[project reuse](ghidra-project-reuse.md). Provider starts are process-cold, with
host filesystem caches unflushed; verified resume occurs in the second session.
The index and inventory still scale with function count. This 14,338-item run
does not establish a larger capacity guarantee. Real Windows/macOS filesystems,
other decompilers and native DWARF directory layout remain unverified. The current
workflow reports unavailable DWARF layout explicitly instead of inventing it.
Actual CLI `decompile-all` calls additionally selected only `0x10a420`, reported
651 excluded entries and `debug_layout_unavailable` for requested dwarf layout,
then resumed in a new process. The existing file's hash and modification time
stayed unchanged; the returned historical source ID stayed stable. This CLI check
verifies the returned manifest/link and local function file, not offline retrieval
of the separately retained source record after the CLI process exits.
## Reproduce and inspect
Prepare an owned project using the documented
[existing-project route](../../docs/installation.md#reuse-an-existing-ghidra-project).
Use fresh receipt/output paths; the driver refuses existing output directories.
```sh
node experiments/analysis-efficiency/decompile-directory.mjs \
--runtime /absolute/rea --output /absolute/new-receipt \
--target /usr/bin/ls --project /absolute/ls-analysis.gpr --program ls \
--ghidra "$GHIDRA_INSTALL_DIR" --java "$JAVA_HOME"
```
Use `--chunk 512 --heap 3072M --decompile-seconds 5` for the crypto case.
`--control /absolute/previous-receipt` compares all exported facts;
`--golden /absolute/fresh-import-receipt` checks the #1048 driver's inventory and
pseudocode oracle. `--history required` additionally requires a linked historical
source. The driver allows failed functions to succeed on a later retry, verifies only
previously written files remain unchanged, and reloads the final index before
comparison. Complete local receipts include the transcript, resource report, index,
function files, comparison facts and authenticated Evidence export. The resource
report is supplied by the external process-family sampler, not the driver.
The current nine-file suite passed 144 tests. Against unmodified main source,
the three source-retention/constructor suites reproduce seven failures and 70
passes; the fixed full suite passes. This includes invalid source commitments,
live and cursor-dependent reads, shared source deduplication, selected/null/provider
subjects, and source links on late-cancelled completion.
@@ -0,0 +1,328 @@
/** Real-provider directory export, cross-session resume and retained provenance. */
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { execFile } from "node:child_process";
import { createWriteStream } from "node:fs";
import { mkdir, mkdtemp, readFile, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { finished } from "node:stream/promises";
import { pathToFileURL } from "node:url";
import { parseArgs, promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
const { values } = parseArgs({
options: Object.fromEntries(
[
"runtime",
"output",
"target",
"project",
"program",
"ghidra",
"java",
"control",
"heap",
"chunk",
"decompile-seconds",
"history",
"golden",
].map((key) => [key, { type: "string" }]),
),
});
for (const key of [
"runtime",
"output",
"target",
"project",
"ghidra",
"java",
]) {
assert.ok(values[key], `${key} is required`);
values[key] = resolve(values[key]);
}
const chunk = Number(values.chunk ?? 32);
assert.ok(Number.isSafeInteger(chunk) && chunk > 0);
await mkdir(values.output);
const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
const report = {
calls: [],
runtime: values.runtime,
target: values.target,
project: values.project,
};
let client, transport, account, serverLog, serverLogFinished;
// This workflow has a fixed number of public calls; pseudocode stays on disk.
const transcript = [];
let phase = "initialize";
const started = performance.now();
const tree = join(values.output, "tree");
const readIndex = async () =>
JSON.parse(await readFile(join(tree, "decompile-index.json"), "utf8"));
const fileStates = async (index) => {
const states = {};
for (const row of index.functions.filter((row) => row.status === "written")) {
const path = join(tree, row.relative_path);
const metadata = await stat(path);
states[row.address] = {
sha256: digest(await readFile(path)),
mtime_ms: metadata.mtimeMs,
};
assert.equal(states[row.address].sha256, row.sha256);
}
return states;
};
try {
report.revision = (
await promisify(execFile)("git", [
"-C",
values.runtime,
"rev-parse",
"HEAD",
])
).stdout.trim();
report.target_sha256 = digest(await readFile(values.target));
const { resolveGhidraProject } = await import(
pathToFileURL(join(values.runtime, "dist/ghidra/GhidraProjectSnapshot.js"))
.href
);
const { ArtifactResourceScope } = await import(
pathToFileURL(
join(values.runtime, "dist/artifacts/ArtifactResourceScope.js"),
).href
);
const fingerprint = async () => {
const resources = new ArtifactResourceScope();
try {
return await resolveGhidraProject(
{
path: values.project,
...(values.program ? { programName: values.program } : {}),
},
{ path: values.target },
resources,
);
} finally {
await resources.close();
}
};
const original = await fingerprint();
report.project_sha256 = original.sha256;
account = await mkdtemp(join(tmpdir(), "rea-directory-investigation-"));
transport = new StdioClientTransport({
command: process.execPath,
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
cwd: values.runtime,
maxBufferSize: 10 * 1024 * 1024,
env: {
PATH: process.env.PATH,
HOME: account,
USERPROFILE: account,
JAVA_HOME: values.java,
GHIDRA_INSTALL_DIR: values.ghidra,
GHIDRA_HEADLESS_MAXMEM: values.heap ?? "1536M",
JAVA_TOOL_OPTIONS: "-XX:ActiveProcessorCount=2",
NODE_OPTIONS: process.env.NODE_OPTIONS ?? "--max-old-space-size=3072",
REA_GHIDRA_PROJECT_PATH: values.project,
...(values.program ? { REA_GHIDRA_PROGRAM_NAME: values.program } : {}),
REA_GHIDRA_STARTUP_TIMEOUT_MS: "600000",
...(values["decompile-seconds"]
? { REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS: values["decompile-seconds"] }
: {}),
REA_MCP_TOOL_FAMILIES: "direct,enhanced,application,session",
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
REA_MCP_OUTPUT_SCHEMA_PROFILE: "compact",
},
stderr: "pipe",
});
const start = transport.start.bind(transport);
transport.start = async () => {
const receive = transport.onmessage;
transport.onmessage = (message) => {
transcript.push({ phase, direction: "response", message });
receive?.(message);
};
await start();
};
const send = transport.send.bind(transport);
transport.send = async (message) => {
transcript.push({ phase, direction: "request", message });
await send(message);
};
serverLog = createWriteStream(join(values.output, "server.log"));
serverLogFinished = finished(serverLog);
serverLogFinished.catch(() => {});
transport.stderr.pipe(serverLog);
client = new Client({ name: "rea-directory-investigation", version: "1" });
await client.connect(transport);
phase = "discovery";
const catalog = await client.listTools({}, { timeout: 15000 });
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
report.tools = catalog.tools.length;
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
const call = async (name, args, label) => {
phase = label;
const at = performance.now();
const result = await client.callTool(
{ name, arguments: args },
{ timeout: 900000, toolDefinition: definitions.get(name) },
);
report.calls.push({
name,
phase,
elapsed_ms: performance.now() - at,
response_bytes: Buffer.byteLength(JSON.stringify(result)),
is_error: result.isError ?? false,
});
assert.notEqual(
result.isError,
true,
`${name}: ${JSON.stringify(result).slice(0, 800)}`,
);
return result.structuredContent;
};
const open = () =>
call("open_binary", { path: values.target, provider_id: "ghidra" }, "open");
await open();
const first = await call(
"decompile_all",
{ output_directory: tree, limit: chunk },
"first-chunk",
);
report.first = first.normalized_result;
const firstFiles = await fileStates(await readIndex());
await call("close_binary", {}, "close-first-session");
await open();
const remaining = await call(
"decompile_all",
{ output_directory: tree, resume: true },
"cross-session-resume",
);
report.resume = remaining.normalized_result;
report.resume_evidence_links = remaining.evidence_links?.length ?? 0;
let index = await readIndex();
const allFiles = await fileStates(index);
for (const [address, state] of Object.entries(firstFiles))
assert.deepEqual(allFiles[address], state);
report.first_files_unchanged = true;
const settled = await call(
"decompile_all",
{ output_directory: tree, resume: true },
"verified-resume",
);
report.verified_resume = settled.normalized_result;
if (report.resume.counts.failed === 0)
assert.equal(report.verified_resume.counts.written, 0);
assert.equal(report.verified_resume.counts.remaining, 0);
index = await readIndex();
const settledFiles = await fileStates(index);
for (const [address, state] of Object.entries(allFiles))
assert.deepEqual(settledFiles[address], state);
assert.equal(index.functions.length, report.resume.counts.selected);
const facts = [];
for (const row of index.functions) {
const fact = {
address: row.address,
name: row.name,
status: row.status,
body: row.body,
};
if (row.status === "written") {
const text = await readFile(join(tree, row.relative_path), "utf8");
assert.ok(text.includes(`// address: ${row.address}\n`));
assert.ok(
text.includes(
`// analysis_profile: ${index.analysis_profile_digest}\n`,
),
);
fact.pseudocode_sha256 = digest(text.slice(text.indexOf("\n\n") + 2));
}
facts.push(fact);
}
report.function_count = facts.length;
report.written = facts.filter((row) => row.status === "written").length;
report.failed = facts.filter((row) => row.status === "failed").length;
report.external = facts.filter((row) => row.status === "external").length;
report.facts_sha256 = digest(JSON.stringify(facts));
await writeFile(join(values.output, "facts.json"), JSON.stringify(facts));
const evidencePath = join(values.output, "complete-evidence.json");
await call(
"export_evidence_bundle",
{ path: evidencePath },
"complete-evidence-export",
);
await call("close_binary", {}, "close");
await client.close();
await transport.close();
const { parseEvidenceBundle } = await import(
pathToFileURL(join(values.runtime, "dist/domain/evidenceBundle.js")).href
);
const bundle = parseEvidenceBundle(
JSON.parse(await readFile(evidencePath, "utf8")),
);
report.evidence_records = bundle.records.length;
const retained = bundle.records.find(
(record) => record.evidence_id === remaining.evidence_id,
);
assert.ok(retained);
report.retained_links = retained.evidence_links.length;
const byId = new Set(bundle.records.map((record) => record.evidence_id));
for (const id of retained.evidence_links) assert.ok(byId.has(id));
report.all_source_links_resolve = true;
if (values.history === "required") {
const historical = bundle.records.filter(
(record) =>
record.operation === "health" &&
record.parameters.facet === "historical-project-analysis-reports",
);
assert.equal(historical.length, 1);
assert.equal(historical[0].normalized_result.origin, "caller-project");
assert.ok(retained.evidence_links.includes(historical[0].evidence_id));
report.historical_source_retained_and_linked = true;
}
if (values.golden) {
const golden = JSON.parse(
await readFile(join(values.golden, "facts.json"), "utf8"),
);
const labels = (rows) =>
rows
.map((row) => [row.address, row.name ?? row.value])
.sort((a, b) => a[0].localeCompare(b[0], "en"));
assert.deepEqual(labels(facts), labels(golden.procedures));
for (const item of golden.pseudocode) {
const fact = facts.find((row) => row.address === item.procedure);
assert.equal(fact?.pseudocode_sha256, digest(item.result));
}
report.fresh_import_inventory_and_sampled_pseudocode_match = true;
}
if (values.control) {
const control = JSON.parse(
await readFile(join(values.control, "facts.json"), "utf8"),
);
assert.deepEqual(facts, control);
report.control_facts_match = true;
}
assert.deepEqual(await fingerprint(), original);
assert.equal(digest(await readFile(values.target)), report.target_sha256);
report.source_unchanged = true;
report.outcome = "passed";
} catch (cause) {
report.outcome = "failed";
report.failure = String(cause);
process.exitCode = 1;
} finally {
report.elapsed_ms = performance.now() - started;
const published = await finalizeRetainedQueryReceipt({
report,
client,
transport,
account,
serverLog,
serverLogFinished,
output: values.output,
transcript,
});
if (!published) process.exitCode = 1;
}
+9 -6
View File
@@ -108,7 +108,11 @@ export function classifyFailure({
}
export function diagnosticPaths(item) {
return item.kind === "jscpd" ? item.paths : [item.path];
return item.kind === "jscpd"
? item.paths
: item.path === undefined
? []
: [item.path];
}
export function isFailedConclusion(value) {
return value === "failure" || value === "timed_out";
@@ -269,16 +273,15 @@ function normalizeLint(report) {
.filter((item) => item.severity === "error")
.map((item) => {
if (
typeof item.filename !== "string" ||
!repoPath(item.filename) ||
typeof item.code !== "string" ||
(item.filename !== undefined && !repoPath(item.filename)) ||
(item.code !== undefined && typeof item.code !== "string") ||
typeof item.message !== "string"
)
throw new Error("oxlint error diagnostic is malformed");
return {
kind: "lint",
path: item.filename,
code: item.code,
...(item.filename === undefined ? {} : { path: item.filename }),
...(item.code === undefined ? {} : { code: item.code }),
message: item.message,
severity: "error",
};
+6 -2
View File
@@ -15,7 +15,7 @@ const lintReport = z.object({
message: z.string(),
code: z.string().optional(),
severity: z.enum(["error", "warning", "advice"]),
filename: z.string(),
filename: z.string().optional(),
labels: z.array(
z.object({
span: z.object({ line: z.number(), column: z.number() }),
@@ -115,8 +115,12 @@ try {
);
for (const diagnostic of errors) {
const location = diagnostic.labels[0]?.span;
const source =
diagnostic.filename === undefined
? ""
: `${diagnostic.filename}${location === undefined ? "" : `:${location.line}:${location.column}`}: `;
console.error(
`${diagnostic.filename}:${location?.line ?? 1}:${location?.column ?? 1}: ${diagnostic.code ?? "lint"}: ${diagnostic.message}`,
`${source}${diagnostic.code ?? "lint"}: ${diagnostic.message}`,
);
}
const counts = new Map();
+11 -6
View File
@@ -1,3 +1,4 @@
import { requireMcpOutput } from "./mcp-verifier-results.mjs";
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { fileURLToPath } from "node:url";
@@ -116,9 +117,12 @@ export async function verifyBrowserNetworkEvidence(
{ timeout: 60_000 },
);
assert.notEqual(result.isError, true, JSON.stringify(result));
const evidence = parseEvidence(result.structuredContent);
assert.notEqual(result.structuredContent, undefined);
const evidence = parseEvidence(
requireMcpOutput(result, "capture_browser_scenario"),
);
mcpProof = assertBrowserNetworkEvidence(
browserScenarioCaptureSchema.parse(evidence.normalized_result),
evidence.normalized_result,
site.origin,
);
if (attachment !== undefined) {
@@ -131,12 +135,13 @@ export async function verifyBrowserNetworkEvidence(
{ timeout: 60_000 },
);
assert.notEqual(connected.isError, true, JSON.stringify(connected));
const connectedEvidence = parseEvidence(connected.structuredContent);
assert.notEqual(connected.structuredContent, undefined);
const connectedEvidence = parseEvidence(
requireMcpOutput(connected, "capture_browser_scenario"),
);
assert.deepEqual(
assertBrowserNetworkEvidence(
browserScenarioCaptureSchema.parse(
connectedEvidence.normalized_result,
),
connectedEvidence.normalized_result,
site.origin,
"disconnected-external",
),
+5 -4
View File
@@ -1,3 +1,4 @@
import { requireMcpOutput } from "./mcp-verifier-results.mjs";
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { once } from "node:events";
@@ -163,11 +164,11 @@ const verifyTransports = async (executable, origin) => {
{ timeout: 60_000 },
);
assert.notEqual(captured.isError, true, JSON.stringify(captured));
const evidence = parseEvidence(captured.structuredContent);
assertNetwork(
browserScenarioCaptureSchema.parse(evidence.normalized_result),
origin,
assert.notEqual(captured.structuredContent, undefined);
const evidence = parseEvidence(
requireMcpOutput(captured, "capture_browser_scenario"),
);
assertNetwork(evidence.normalized_result, origin);
} finally {
try {
await client.close();
+8 -4
View File
@@ -1,4 +1,7 @@
import { requireMcpToolError } from "./mcp-verifier-results.mjs";
import {
requireMcpOutput,
requireMcpToolError,
} from "./mcp-verifier-results.mjs";
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { mkdtemp, readFile, rm } from "node:fs/promises";
@@ -95,10 +98,11 @@ export async function verifyLargeScreenshotE2e(endpoint, origin) {
{ timeout: 60000 },
);
assert.notEqual(captured.isError, true, JSON.stringify(captured));
const mcpEvidence = parseEvidence(captured.structuredContent);
const mcpScreenshot = webScreenshotSchema.parse(
mcpEvidence.normalized_result,
assert.notEqual(captured.structuredContent, undefined);
const mcpEvidence = parseEvidence(
requireMcpOutput(captured, "capture_web_screenshot"),
);
const mcpScreenshot = mcpEvidence.normalized_result;
assert.deepEqual(mcpScreenshot.artifact, screenshot.artifact);
const compared = comparePngScreenshots({
before: screenshot.artifact,
+10 -10
View File
@@ -38,7 +38,16 @@ export const requireMcpOutput = (
throw new Error(`${operation} failed: ${mcpTextValue(result)}`);
if (contract.name !== operation)
throw new Error(`${operation} received the ${contract.name} contract`);
return contract.outputSchema.parse(jsonValue(result));
const text = jsonValue(result);
const output = contract.outputSchema.parse(text);
if (result.structuredContent !== undefined) {
// The producer emits both representations from the same JSON value.
// Compare before schema projection can discard disagreeing source fields.
if (!isDeepStrictEqual(text, result.structuredContent))
throw new Error(`${operation} text and structured results disagree`);
}
// Downstream parseEvidence must authenticate source facts, not a schema projection.
return Object.hasOwn(output, "normalized_result") ? text : output;
};
const operationResultKey = (contract) => {
@@ -124,15 +133,6 @@ export const requireMcpEvidenceBundle = (result) => {
throw new Error(
"get_evidence_bundle returned summary metadata instead of complete Evidence",
);
if (result.structuredContent !== undefined) {
const structured = contract.outputSchema.parse(
result.structuredContent,
).result;
if (!isDeepStrictEqual(bundle, structured))
throw new Error(
"get_evidence_bundle text and structured results disagree",
);
}
return authenticateParsedEvidenceBundle(bundle);
};
+27 -16
View File
@@ -14,6 +14,7 @@ import { nativeValueTraceSchema } from "../../dist/domain/native/nativeValueTrac
import {
requireMcpEvidenceBundle,
requireMcpLifecycleResult,
requireMcpOutput,
} from "./mcp-verifier-results.mjs";
import { verifyGhidraBoundaries } from "./real-ghidra-boundaries.mjs";
@@ -198,12 +199,15 @@ async function verifyNativeValueAdapters(
{ timeout: 180000 },
);
assert.notEqual(result.isError, true, JSON.stringify(result));
const mcpEvidence = parseEvidence(result.structuredContent);
assert.notEqual(result.structuredContent, undefined);
const mcpEvidence = parseEvidence(
requireMcpOutput(result, "trace_native_values"),
);
assert.equal(mcpEvidence.operation, "trace_native_values");
assert.equal(mcpEvidence.provider.id, "rea-workflow");
assert.deepEqual(mcpEvidence.analysis_profile, workflowProfile);
const mcpGraph = verifyGraph(
result.structuredContent?.normalized_result,
mcpEvidence.normalized_result,
target,
procedure,
globalAddress,
@@ -218,7 +222,10 @@ async function verifyNativeValueAdapters(
{ timeout: 180000 },
);
assert.notEqual(analyzed.isError, true, JSON.stringify(analyzed));
const functionEvidence = parseEvidence(analyzed.structuredContent);
assert.notEqual(analyzed.structuredContent, undefined);
const functionEvidence = parseEvidence(
requireMcpOutput(analyzed, "analyze_function"),
);
assert.equal(functionEvidence.operation, "analyze_function");
assert.equal(functionEvidence.provider.id, "ghidra");
assert.deepEqual(
@@ -246,10 +253,6 @@ async function verifyNativeValueAdapters(
functionEvidence.analysis_profile,
);
assert.deepEqual(functionProfile, upstreamProfile);
assert.deepEqual(
functionEvidence.normalized_result,
analyzed.structuredContent?.normalized_result,
);
const comparison = await client.callTool(
{
name: "compare_functions",
@@ -261,16 +264,15 @@ async function verifyNativeValueAdapters(
{ timeout: 180000 },
);
assert.notEqual(comparison.isError, true, JSON.stringify(comparison));
const comparisonEvidence = parseEvidence(comparison.structuredContent);
assert.notEqual(comparison.structuredContent, undefined);
const comparisonEvidence = parseEvidence(
requireMcpOutput(comparison, "compare_functions"),
);
assert.equal(
comparisonEvidence.normalized_result.status,
"unchanged",
JSON.stringify(comparisonEvidence.normalized_result.dimensions),
);
assert.deepEqual(
comparisonEvidence.normalized_result,
comparison.structuredContent?.normalized_result,
);
assert.ok(
comparisonEvidence.evidence_links.includes(functionEvidence.evidence_id),
);
@@ -346,8 +348,9 @@ async function verifyUnresolvedCallEvidence(client) {
arguments: {},
});
assert.notEqual(inventory.isError, true, JSON.stringify(inventory));
assert.notEqual(inventory.structuredContent, undefined);
const procedures = parseEvidence(
inventory.structuredContent,
requireMcpOutput(inventory, "list_procedures"),
).normalized_result;
const indirect = procedures.find((item) =>
item.value.endsWith("rea_ghidra_inventory_indirect"),
@@ -367,7 +370,10 @@ async function verifyUnresolvedCallEvidence(client) {
{ timeout: 180000 },
);
assert.notEqual(analyzed.isError, true, JSON.stringify(analyzed));
const evidence = parseEvidence(analyzed.structuredContent);
assert.notEqual(analyzed.structuredContent, undefined);
const evidence = parseEvidence(
requireMcpOutput(analyzed, "analyze_function"),
);
assert.equal(evidence.relationship_coverage?.status, "partial");
assert.ok(evidence.normalized_result.unresolved_calls.length > 0);
const traced = await client.callTool(
@@ -382,7 +388,10 @@ async function verifyUnresolvedCallEvidence(client) {
{ timeout: 180000 },
);
assert.notEqual(traced.isError, true, JSON.stringify(traced));
const trace = parseEvidence(traced.structuredContent).normalized_result;
assert.notEqual(traced.structuredContent, undefined);
const trace = parseEvidence(
requireMcpOutput(traced, "trace_call_path"),
).normalized_result;
assert.equal(trace.goal_status, "unknown");
assert.equal(trace.truncated, true);
assert.ok(trace.residual_unknowns.length > 0);
@@ -397,8 +406,10 @@ async function verifyUnresolvedCallEvidence(client) {
{ timeout: 180000 },
);
assert.notEqual(compared.isError, true, JSON.stringify(compared));
assert.notEqual(compared.structuredContent, undefined);
assert.equal(
parseEvidence(compared.structuredContent).normalized_result.status,
parseEvidence(requireMcpOutput(compared, "compare_functions"))
.normalized_result.status,
"unknown",
);
}
+5 -2
View File
@@ -26,7 +26,10 @@ import {
locateApkSigningBlock,
parseZipTail,
} from "../dist/domain/apkSignature/apkSigningBlock.js";
import { mcpTextValue } from "./lib/mcp-verifier-results.mjs";
import {
mcpTextValue,
requireMcpOutput,
} from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
const execute = promisify(execFile);
@@ -261,7 +264,7 @@ try {
ajv.validate(advertised.outputSchema, reply.structuredContent),
JSON.stringify(ajv.errors),
);
const mcp = parseEvidence(reply.structuredContent);
const mcp = parseEvidence(requireMcpOutput(reply, "inspect_apk_signature"));
assert.deepEqual(cli, mcp, "CLI and MCP evidence differ");
cases += 2;
return mcp.normalized_result;
+2 -5
View File
@@ -26,7 +26,7 @@ import {
} from "./lib/http3-verifier-lifecycle.mjs";
import {
requireMcpEvidenceBundle,
mcpTextValue,
requireMcpOutput,
} from "./lib/mcp-verifier-results.mjs";
const root = process.argv[2];
@@ -137,14 +137,11 @@ async function mcp(input, name) {
join(directory, `${name}.mcp.reply.json`),
JSON.stringify(reply, null, 2),
);
const value = JSON.parse(mcpTextValue(reply));
assert.notEqual(reply.isError, true, mcpTextValue(reply));
const evidence = requireMcpOutput(reply, contract.name, contract);
assert.ok(
validateOutput(reply.structuredContent),
JSON.stringify(validateOutput.errors),
);
const evidence = contract.outputSchema.parse(value);
assert.deepEqual(reply.structuredContent, value);
retainedEvidence.push(evidence);
return evidence;
}
+13 -22
View File
@@ -1,6 +1,7 @@
import {
requireMcpEvidenceBundle,
requireMcpLifecycleResult,
requireMcpOutput,
requireMcpToolError,
} from "./lib/mcp-verifier-results.mjs";
import assert from "node:assert/strict";
@@ -172,31 +173,23 @@ const verifyMcpInlineArtifactEvidence = async (
{ name: "inspect_artifact", arguments: {} },
mcpOptions,
);
assert.notEqual(inspected.isError, true, JSON.stringify(inspected));
const source = parseEvidence(inspected.structuredContent);
assert.deepEqual(
source.normalized_result,
inspected.structuredContent?.normalized_result,
);
assert.deepEqual(
JSON.parse(prompts.mcpText(inspected)),
inspected.structuredContent,
assert.notEqual(inspected.structuredContent, undefined);
const source = parseEvidence(
requireMcpOutput(inspected, "inspect_artifact"),
);
const compared = await client.callTool(
{
name: "compare_artifacts",
arguments: {
left: inspected.structuredContent,
right: inspected.structuredContent,
left: source,
right: source,
},
},
mcpOptions,
);
assert.notEqual(compared.isError, true, JSON.stringify(compared));
const comparison = parseEvidence(compared.structuredContent);
assert.deepEqual(
comparison.normalized_result,
compared.structuredContent?.normalized_result,
assert.notEqual(compared.structuredContent, undefined);
const comparison = parseEvidence(
requireMcpOutput(compared, "compare_artifacts"),
);
assert.ok(comparison.evidence_links.includes(source.evidence_id));
const bundleReply = await client.callTool(
@@ -220,13 +213,11 @@ const verifyMcpInlineArtifactEvidence = async (
{ name: "extract_artifact", arguments: {} },
mcpOptions,
);
assert.notEqual(extracted.isError, true, JSON.stringify(extracted));
const extraction = parseEvidence(extracted.structuredContent);
assert.deepEqual(
extraction.normalized_result,
extracted.structuredContent?.normalized_result,
assert.notEqual(extracted.structuredContent, undefined);
const extraction = parseEvidence(
requireMcpOutput(extracted, "extract_artifact"),
);
const materialized = extracted.structuredContent?.normalized_result;
const materialized = extraction.normalized_result;
assert.equal(typeof materialized?.output_root, "string");
outputRoot = materialized.output_root;
assert.equal(materialized.containment_verified, true);
+22 -19
View File
@@ -1,6 +1,10 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { requireMcpEvidenceBundle } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpEvidenceBundle,
requireMcpOutput,
requireMcpToolError,
} from "./lib/mcp-verifier-results.mjs";
import { estimateTokenCount } from "tokenx";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
@@ -124,12 +128,13 @@ await withVerifierCleanup(async () => {
name: tool.name,
arguments: input,
});
assert.notEqual(response.isError, true, JSON.stringify(response));
assert.ok(
ajv.validate(tool.outputSchema, response.structuredContent),
JSON.stringify(ajv.errors),
);
const evidence = parseEvidence(response.structuredContent);
const evidence = parseEvidence(
requireMcpOutput(response, "inspect_packet_capture"),
);
const report = evidence.normalized_result;
assert.equal(
report.artifact.sha256,
@@ -305,12 +310,13 @@ await withVerifierCleanup(async () => {
name: pcapTool.name,
arguments: { path: capturePath, detail: "summary" },
});
assert.notEqual(response.isError, true, JSON.stringify(response));
assert.ok(
pcapAjv.validate(pcapTool.outputSchema, response.structuredContent),
pcapAjv.errorsText(),
);
const summary = parseEvidence(response.structuredContent).normalized_result;
const summary = parseEvidence(
requireMcpOutput(response, "inspect_pcap_capture"),
).normalized_result;
assert.equal(summary.summary.capture_start, expectedCaptureStart);
assert.equal(summary.summary.capture_end, expectedCaptureEnd);
assert.equal(
@@ -343,10 +349,9 @@ await withVerifierCleanup(async () => {
view: { kind: "pcap", facet: "protocols", limit: 10 },
},
});
assert.notEqual(page.isError, true, JSON.stringify(page));
assert.equal(
parseEvidence(page.structuredContent).normalized_result
.parent_evidence_id,
parseEvidence(requireMcpOutput(page, "inspect_analysis_view"))
.normalized_result.parent_evidence_id,
source.evidence_id,
);
cases++;
@@ -368,19 +373,16 @@ await withVerifierCleanup(async () => {
name: tool.name,
arguments: input,
});
assert.equal(response.isError, true);
requireMcpToolError(response);
await client.ping();
cases++;
}
await writeFile(path, "not a capture");
assert.equal(
(
await client.callTool({
name: tool.name,
arguments: { path, fields: ["frame.number"] },
})
).isError,
true,
requireMcpToolError(
await client.callTool({
name: tool.name,
arguments: { path, fields: ["frame.number"] },
}),
);
cases++;
const discovery = (await client.listTools()).tools.find(
@@ -414,12 +416,13 @@ await withVerifierCleanup(async () => {
name: discovery.name,
arguments: input,
});
assert.notEqual(response.isError, true, JSON.stringify(response));
assert.ok(
ajv.validate(discovery.outputSchema, response.structuredContent),
JSON.stringify(ajv.errors),
);
const evidence = parseEvidence(response.structuredContent);
const evidence = parseEvidence(
requireMcpOutput(response, "inspect_packet_fields"),
);
const report = evidence.normalized_result;
const kind = input.kind ?? "protocols";
const query = (input.query ?? "").toLowerCase();
+13 -9
View File
@@ -10,6 +10,11 @@ import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { Ajv2020 } from "ajv/dist/2020.js";
import { parseEvidence } from "../dist/domain/evidence.js";
import { toolContract } from "../dist/contracts/toolContracts.js";
import {
requireMcpOutput,
requireMcpToolError,
} from "./lib/mcp-verifier-results.mjs";
// Caller-supplied producer evidence stays local; no fixture capture or secrets are bundled.
const [controlPath, lossPath, reportPath] = process.argv.slice(2);
@@ -40,6 +45,7 @@ transport.stderr?.on("data", (bytes) => {
});
const ajv = new Ajv2020({ strict: false, validateFormats: false });
const measurements = [];
const contract = toolContract("inspect_web_network_capture");
try {
try {
await client.connect(transport);
@@ -68,12 +74,13 @@ try {
name: tool.name,
arguments: input,
});
assert.notEqual(response.isError, true);
const mcp = parseEvidence(
requireMcpOutput(response, contract.name, contract),
);
assert.ok(
validateOutput(response.structuredContent),
JSON.stringify(validateOutput.errors),
);
const mcp = parseEvidence(response.structuredContent);
const completed = await promisify(execFile)(
process.execPath,
[entrypoint, "inspect-web-network-capture", path, "qlog", "--json"],
@@ -153,17 +160,14 @@ try {
view: { kind: "api-surface" },
},
});
assert.equal(retained.isError, true);
assert.equal(
JSON.parse(retained.content.find((item) => item.type === "text").text)
.error.code,
"unsupported_target",
);
assert.equal(requireMcpToolError(retained).code, "unsupported_target");
const selected = await client.callTool({
name: tool.name,
arguments: { ...input, record_ordinals: [capture.total_records - 1, 0] },
});
const subset = parseEvidence(selected.structuredContent).normalized_result;
const subset = parseEvidence(
requireMcpOutput(selected, contract.name, contract),
).normalized_result;
assert.deepEqual(
subset.records.map((record) => record.ordinal),
[capture.total_records - 1, 0],
+5 -3
View File
@@ -18,7 +18,10 @@ import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { parseEvidence } from "../dist/domain/evidence.js";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpOperationResult,
requireMcpOutput,
} from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
if (process.argv.length !== 2)
@@ -464,10 +467,9 @@ async function call(name, args = {}) {
{ name, arguments: args },
{ timeout: 240000 },
);
assert.notEqual(result.isError, true, JSON.stringify(result));
if (["open_binary", "binary_session", "close_binary"].includes(name))
return requireMcpOperationResult(result, name);
return result.structuredContent;
return requireMcpOutput(result, name);
}
function evidence(value, sha256) {
@@ -9,7 +9,10 @@ import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { parseEvidence } from "../dist/domain/evidence.js";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpOperationResult,
requireMcpOutput,
} from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
if (process.argv.length !== 2 || process.platform !== "linux")
@@ -320,8 +323,7 @@ async function call(name, args = {}) {
{ name, arguments: args },
{ timeout: 240000 },
);
assert.notEqual(result.isError, true, JSON.stringify(result));
return ["open_binary", "binary_session", "close_binary"].includes(name)
? requireMcpOperationResult(result, name)
: result.structuredContent;
: requireMcpOutput(result, name);
}
+8 -10
View File
@@ -1,6 +1,9 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpLifecycleResult,
requireMcpOutput,
} from "./lib/mcp-verifier-results.mjs";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
@@ -339,16 +342,11 @@ async function call(name, args = {}) {
{ name, arguments: args },
{ timeout: 240000 },
);
assert.notEqual(value.isError, true, JSON.stringify(value));
const projected = requireMcpOperationResult(value, name);
if (name === "open_binary" || name === "close_binary") return projected;
const evidence = parseEvidence(value.structuredContent);
if (["open_binary", "close_binary"].includes(name))
return requireMcpLifecycleResult(value, name);
const evidence = parseEvidence(requireMcpOutput(value, name));
assert.equal(evidence.subject.digest.sha256, sha256);
assert.deepEqual(
evidence.normalized_result,
value.structuredContent.normalized_result,
);
return projected;
return evidence.normalized_result;
}
async function cli(command, address, options = []) {
const { stdout } = await promisify(execFile)(
+5 -4
View File
@@ -1,6 +1,9 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpOperationResult,
requireMcpOutput,
} from "./lib/mcp-verifier-results.mjs";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import {
@@ -555,7 +558,6 @@ async function call(name, arguments_ = {}) {
{ name, arguments: arguments_ },
{ timeout: 240000 },
);
assert.notEqual(result.isError, true, JSON.stringify(result));
assert.ok(result.structuredContent, `${name} omitted structured result`);
return requireMcpOperationResult(result, name);
}
@@ -564,8 +566,7 @@ async function evidenceCall(name, arguments_) {
{ name, arguments: arguments_ },
{ timeout: 240000 },
);
assert.notEqual(result.isError, true, JSON.stringify(result));
const evidence = parseEvidence(result.structuredContent);
const evidence = parseEvidence(requireMcpOutput(result, name));
assert.equal(evidence.subject.digest.sha256, sha256);
assert.equal(evidence.provider.id, "ghidra");
assertProfile(evidence.analysis_profile);
+5 -3
View File
@@ -16,7 +16,10 @@ import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { parseEvidence } from "../dist/domain/evidence.js";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpOperationResult,
requireMcpOutput,
} from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
if (process.argv.length !== 2)
@@ -281,10 +284,9 @@ async function call(name, args = {}) {
{ name, arguments: args },
{ timeout: 240000 },
);
assert.notEqual(result.isError, true, JSON.stringify(result));
if (["open_binary", "binary_session", "close_binary"].includes(name))
return requireMcpOperationResult(result, name);
return result.structuredContent;
return requireMcpOutput(result, name);
}
function evidence(value) {
+7 -10
View File
@@ -1,6 +1,9 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpLifecycleResult,
requireMcpOutput,
} from "./lib/mcp-verifier-results.mjs";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import {
@@ -394,14 +397,8 @@ async function call(name, args = {}) {
{ name, arguments: args },
{ timeout: 360000 },
);
assert.notEqual(response.isError, true, JSON.stringify(response));
const projected = requireMcpOperationResult(response, name);
if (name === "close_binary") return projected;
const evidence = parseEvidence(response.structuredContent);
if (name === "close_binary") return requireMcpLifecycleResult(response, name);
const evidence = parseEvidence(requireMcpOutput(response, name));
assert.equal(evidence.subject.digest.sha256, sha256);
assert.deepEqual(
evidence.normalized_result,
response.structuredContent.normalized_result,
);
return projected;
return evidence.normalized_result;
}
+7 -10
View File
@@ -12,7 +12,10 @@ import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { parseEvidence } from "../dist/domain/evidence.js";
import { inspectVerifierGhidraInstallation } from "./lib/ghidra-installation-inspection.mjs";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpLifecycleResult,
requireMcpOutput,
} from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
if (process.argv.length !== 2)
@@ -351,15 +354,9 @@ async function call(name, args = {}, targetSha256 = sha256) {
{ name, arguments: args },
{ timeout: 240000 },
);
assert.notEqual(result.isError, true, JSON.stringify(result));
const projected = requireMcpOperationResult(result, name);
if (
name === "open_binary" ||
name === "close_binary" ||
name === "binary_session"
)
return projected;
const evidence = parseEvidence(result.structuredContent);
if (["open_binary", "close_binary", "binary_session"].includes(name))
return requireMcpLifecycleResult(result, name);
const evidence = parseEvidence(requireMcpOutput(result, name));
assert.equal(evidence.subject.digest.sha256, targetSha256);
assert.equal(evidence.subject.format, "raw-binary");
return evidence;
+13 -6
View File
@@ -26,7 +26,8 @@ import { inspectVerifierGhidraInstallation } from "./lib/ghidra-installation-ins
import { buildRiscvFixture } from "../tests/conformance/ghidra/riscv-fixture.mjs";
import {
mcpTextValue,
requireMcpOperationResult,
requireMcpLifecycleResult,
requireMcpOutput,
requireMcpToolError,
} from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
@@ -141,18 +142,24 @@ try {
{ name, arguments: args },
{ timeout: 240000 },
);
const projected = requireMcpOperationResult(reply, name);
const lifecycle = [
"open_binary",
"close_binary",
"binary_session",
].includes(name);
const admitted = lifecycle
? requireMcpLifecycleResult(reply, name)
: parseEvidence(requireMcpOutput(reply, name));
const textual = JSON.parse(mcpTextValue(reply));
assert.deepEqual(reply.structuredContent, textual);
assert.ok(reply.structuredContent, `${name} omitted structured result`);
const validate = validators.get(name);
assert.ok(validate, `${name} missing from advertised catalog`);
assert.ok(
validate(textual),
`${name}: ${JSON.stringify(validate.errors)}`,
);
if (!["open_binary", "close_binary", "binary_session"].includes(name))
assertEvidence(parseEvidence(textual));
return projected;
if (!lifecycle) assertEvidence(admitted);
return lifecycle ? admitted : admitted.normalized_result;
};
const opened = await call("open_binary", {
path: fixture.path,
+1 -2
View File
@@ -21,7 +21,6 @@ import {
ProviderProcessSupervisor,
spawnOwnedProviderProcess,
} from "../dist/process/ProviderProcess.js";
import { toolContract } from "../dist/contracts/toolContracts.js";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
@@ -215,7 +214,7 @@ async function call(name, arguments_) {
{ timeout: 180_000 },
);
if (name === "analyze_swift_types")
toolContract(name).outputSchema.parse(response.structuredContent);
assert.ok(response.structuredContent, `${name} omitted structured result`);
return requireMcpOperationResult(response, name);
}
+5 -3
View File
@@ -1,6 +1,9 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpOperationResult,
requireMcpOutput,
} from "./lib/mcp-verifier-results.mjs";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import {
@@ -326,9 +329,8 @@ async function toolResult(client, name, arguments_ = {}) {
{ name, arguments: arguments_ },
{ timeout: 240000 },
);
assert.notEqual(result.isError, true, JSON.stringify(result));
assert.ok(result.structuredContent, `${name} omitted structured result`);
return result.structuredContent;
return requireMcpOutput(result, name);
}
async function call(client, name, arguments_ = {}) {
const result = await client.callTool(
+6 -9
View File
@@ -8,7 +8,7 @@ import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { tsImport } from "tsx/esm/api";
import { toolContract } from "../dist/contracts/toolContracts.js";
import { requireMcpOutput } from "./lib/mcp-verifier-results.mjs";
import { parseEvidence } from "../dist/domain/evidence.js";
import { createEvidenceBundle } from "../dist/domain/evidenceBundle.js";
import { withVerifierCleanup } from "./lib/verifier-cleanup.mjs";
@@ -80,10 +80,8 @@ await withVerifierCleanup(async () => {
{ name: "binary_session", arguments: {} },
{ toolDefinition: sessionTool },
);
assert.notEqual(sessionReply.isError, true);
const sessionStatus = toolContract("binary_session").outputSchema.parse(
sessionReply.structuredContent,
).result;
assert.notEqual(sessionReply.structuredContent, undefined);
const sessionStatus = requireMcpOutput(sessionReply, "binary_session").result;
const javascriptMcp = await analyzeMcp(client, javascriptPath);
const electronMcp = await analyzeMcp(client, electronPath);
assert.equal(
@@ -194,11 +192,10 @@ async function analyzeMcp(client_, inputPath) {
name: "analyze_javascript_application",
arguments: { input_path: inputPath },
});
assert.notEqual(response.isError, true);
toolContract("analyze_javascript_application").outputSchema.parse(
response.structuredContent,
assert.notEqual(response.structuredContent, undefined);
return parseEvidence(
requireMcpOutput(response, "analyze_javascript_application"),
);
return parseEvidence(response.structuredContent);
}
function verifySyntheticEvaluator() {
+17 -12
View File
@@ -1,6 +1,10 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { requireMcpEvidenceBundle } from "./lib/mcp-verifier-results.mjs";
import {
requireMcpEvidenceBundle,
requireMcpOutput,
requireMcpToolError,
} from "./lib/mcp-verifier-results.mjs";
import { estimateTokenCount } from "tokenx";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
@@ -47,9 +51,7 @@ const assertRejectedSelector = (
field,
expectedExitCode = 1,
) => {
assert.equal(response.isError, true);
const text = response.content.find((item) => item.type === "text")?.text;
const { error } = JSON.parse(text ?? "{}");
const error = requireMcpToolError(response);
assert.equal(error.code, "invalid_request");
assert.equal(error.details.operation, operation);
assert.deepEqual(
@@ -123,12 +125,13 @@ await withVerifierCleanup(async () => {
name: advertised.name,
arguments: input,
});
assert.notEqual(response.isError, true, JSON.stringify(response));
assert.ok(
ajv.validate(advertised.outputSchema, response.structuredContent),
JSON.stringify(ajv.errors),
);
const evidence = parseEvidence(response.structuredContent);
const evidence = parseEvidence(
requireMcpOutput(response, "decode_signal_recording"),
);
const report = evidence.normalized_result;
for (const byte of ["52", "45", "41"])
assert.match(
@@ -238,12 +241,13 @@ await withVerifierCleanup(async () => {
name: advertised.name,
arguments: input,
});
assert.notEqual(response.isError, true, JSON.stringify(response));
assert.ok(
ajv.validate(advertised.outputSchema, response.structuredContent),
JSON.stringify(ajv.errors),
);
const evidence = parseEvidence(response.structuredContent);
const evidence = parseEvidence(
requireMcpOutput(response, "decode_signal_recording"),
);
const report = evidence.normalized_result;
assert.deepEqual(report.output_window, {
total_lines: 120,
@@ -298,9 +302,9 @@ await withVerifierCleanup(async () => {
name: advertised.name,
arguments: input,
});
assert.equal(response.isError, true);
if (rejectedField !== undefined)
assertRejectedSelector(response, advertised.name, rejectedField);
else requireMcpToolError(response);
await client.ping();
cases++;
}
@@ -309,7 +313,7 @@ await withVerifierCleanup(async () => {
name: advertised.name,
arguments: { path, decoders: "uart" },
});
assert.equal(invalid.isError, true);
requireMcpToolError(invalid);
cases++;
const discoveryTool = (await client.listTools()).tools.find(
({ name }) => name === "inspect_signal_decoders",
@@ -367,12 +371,13 @@ await withVerifierCleanup(async () => {
name: discoveryTool.name,
arguments: input,
});
assert.notEqual(response.isError, true, JSON.stringify(response));
assert.ok(
ajv.validate(discoveryTool.outputSchema, response.structuredContent),
JSON.stringify(ajv.errors),
);
const evidence = parseEvidence(response.structuredContent);
const evidence = parseEvidence(
requireMcpOutput(response, "inspect_signal_decoders"),
);
const report = evidence.normalized_result;
discoveryMeasurements.push({
input,
+3 -2
View File
@@ -21,6 +21,7 @@ import { PrivateRuntimeRoot } from "../dist/process/PrivateRuntimeRoot.js";
import { parseEvidence } from "../dist/domain/evidence.js";
import {
requireMcpEvidenceBundle,
requireMcpOutput,
mcpTextValue,
requireMcpToolError,
} from "./lib/mcp-verifier-results.mjs";
@@ -278,7 +279,7 @@ try {
ajv.validate(advertised.outputSchema, reply.structuredContent),
JSON.stringify(ajv.errors),
);
const mcp = parseEvidence(reply.structuredContent);
const mcp = parseEvidence(requireMcpOutput(reply, "inspect_wasm_artifact"));
assert.equal(cli.evidence_id, mcp.evidence_id);
assert.deepEqual(cli, mcp);
const report = mcp.normalized_result;
@@ -516,7 +517,7 @@ try {
ajv.validate(advertised.outputSchema, reply.structuredContent),
JSON.stringify(ajv.errors),
);
const mcp = parseEvidence(reply.structuredContent);
const mcp = parseEvidence(requireMcpOutput(reply, "inspect_wasm_artifact"));
assert.deepEqual(cli, mcp);
const report = mcp.normalized_result;
assert.equal(report.validation, "valid");
+5 -2
View File
@@ -7,6 +7,7 @@ import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { parseEvidence } from "../dist/domain/evidence.js";
import { requireMcpOutput } from "./lib/mcp-verifier-results.mjs";
const execute = promisify(execFile);
const bin = process.env.REA_WABT_BIN_DIRECTORY;
@@ -53,7 +54,8 @@ const inspect = async (path) => {
arguments: { path },
});
assert.notEqual(result.isError, true, JSON.stringify(result));
return parseEvidence(result.structuredContent);
assert.notEqual(result.structuredContent, undefined);
return parseEvidence(requireMcpOutput(result, "inspect_wasm_artifact"));
};
const compare = async (left, right, scope = "both") => {
const input = {
@@ -66,7 +68,8 @@ const compare = async (left, right, scope = "both") => {
arguments: input,
});
assert.notEqual(reply.isError, true, JSON.stringify(reply));
const mcp = parseEvidence(reply.structuredContent);
assert.notEqual(reply.structuredContent, undefined);
const mcp = parseEvidence(requireMcpOutput(reply, "compare_wasm_interfaces"));
const file = join(root, "comparison.json");
await writeFile(file, JSON.stringify({ left, right, scope }));
const local = parseEvidence(
+15 -13
View File
@@ -18,7 +18,11 @@ import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { PrivateRuntimeRoot } from "../dist/process/PrivateRuntimeRoot.js";
import { runOwnedCommand } from "../dist/process/OwnedCommand.js";
import { WEB_NETWORK_CAPTURE_LIMITS } from "../dist/domain/webNetworkCapture.js";
import { mcpTextValue } from "./lib/mcp-verifier-results.mjs";
import {
mcpTextValue,
requireMcpOutput,
requireMcpToolError,
} from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
const executable = process.env.REA_MITMDUMP_COMMAND;
@@ -903,15 +907,15 @@ async function inspect(mode, input, errorCategory) {
name: "inspect_web_network_capture",
arguments: input,
});
const value = JSON.parse(mcpTextValue(result));
if (errorCategory !== undefined) {
assert.equal(result.isError, true);
assert.equal(value.error.category, errorCategory);
const error = requireMcpToolError(result);
assert.equal(error.category, errorCategory);
const rawError = JSON.parse(mcpTextValue(result)).error;
for (const literal of input.sensitive_values ?? [])
assert.ok(!JSON.stringify(value.error).includes(literal));
return value.error;
assert.ok(!JSON.stringify(rawError).includes(literal));
return error;
}
assert.notEqual(result.isError, true, mcpTextValue(result));
const value = requireMcpOutput(result, "inspect_web_network_capture");
assertSensitiveLimitations(value, input.sensitive_values ?? []);
if (input.format === "har") {
const surface = await client.callTool({
@@ -923,13 +927,12 @@ async function inspect(mode, input, errorCategory) {
});
if ((input.sensitive_values ?? []).length > 0) {
// Derived view text cannot be checked against undisclosed declarations.
assert.equal(surface.isError, true);
requireMcpToolError(surface);
const refused = JSON.parse(mcpTextValue(surface));
for (const literal of input.sensitive_values)
assert.ok(!JSON.stringify(refused).includes(literal));
} else {
assert.notEqual(surface.isError, true, mcpTextValue(surface));
const projected = JSON.parse(mcpTextValue(surface));
const projected = requireMcpOutput(surface, "inspect_analysis_view");
assert.equal(projected.normalized_result.kind, "api-surface");
assert.equal(
projected.normalized_result.records.examined,
@@ -944,13 +947,12 @@ async function inspect(mode, input, errorCategory) {
},
});
if ((input.sensitive_values ?? []).length > 0) {
assert.equal(manifest.isError, true);
requireMcpToolError(manifest);
const refusedManifest = JSON.parse(mcpTextValue(manifest));
for (const literal of input.sensitive_values)
assert.ok(!JSON.stringify(refusedManifest).includes(literal));
} else {
assert.notEqual(manifest.isError, true, mcpTextValue(manifest));
const projected = JSON.parse(mcpTextValue(manifest));
const projected = requireMcpOutput(manifest, "inspect_analysis_view");
assert.equal(projected.normalized_result.kind, "environment-manifest");
assert.equal(projected.normalized_result.unknowns.length, 3);
for (const record of projected.normalized_result.records)
+6 -8
View File
@@ -1,3 +1,4 @@
import { requireMcpOutput } from "../../lib/mcp-verifier-results.mjs";
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { access, writeFile } from "node:fs/promises";
@@ -147,17 +148,14 @@ try {
{ name: task.name, arguments: task.input },
{ timeout: 150_000 },
);
assert.notEqual(response.isError, true, JSON.stringify(response));
const structured = response.structuredContent;
assert.ok(structured !== undefined);
const evidence = parseEvidence(structured);
assert.notEqual(response.structuredContent, undefined);
const evidence = parseEvidence(requireMcpOutput(response, task.name));
assert.deepEqual(
evidence.normalized_result,
cliResults.get(task.name).normalized_result,
);
// Raw observations retain different workspace paths and load durations.
// Validate each evidence ID independently; compare portable results/intent.
assert.equal(structured.evidence_id, evidence.evidence_id);
assert.deepEqual(evidence.parameters, cliResults.get(task.name).parameters);
assert.deepEqual(evidence.provider, cliResults.get(task.name).provider);
assert.deepEqual(
@@ -169,7 +167,6 @@ try {
cliResults.get(task.name).limitations,
);
assert.equal(evidence.confidence, cliResults.get(task.name).confidence);
assert.deepEqual(structured.normalized_result, evidence.normalized_result);
console.log(`PASS MCP/CLI parity ${task.name}`);
}
const overloadedInput = {
@@ -210,9 +207,10 @@ try {
{ name: "trace_android_references", arguments: overloadedInput },
{ timeout: 150_000 },
);
assert.notEqual(overloadedMcp.isError, true, JSON.stringify(overloadedMcp));
assert.notEqual(overloadedMcp.structuredContent, undefined);
assert.deepEqual(
parseEvidence(overloadedMcp.structuredContent).normalized_result,
parseEvidence(requireMcpOutput(overloadedMcp, "trace_android_references"))
.normalized_result,
overloadedEvidence.normalized_result,
);
console.log("PASS real overloaded method references through CLI/MCP");
+5 -5
View File
@@ -1,6 +1,6 @@
import assert from "node:assert/strict";
import { parseEvidence } from "../../../dist/domain/evidence.js";
import { requireMcpOperationResult } from "../../lib/mcp-verifier-results.mjs";
import { requireMcpOutput } from "../../lib/mcp-verifier-results.mjs";
/** Compare native selection with an explicit full-inventory oracle on one live session. */
export const verifyAndroidClassSearch = async ({ client, path, sha256 }) => {
@@ -9,11 +9,11 @@ export const verifyAndroidClassSearch = async ({ client, path, sha256 }) => {
{ name: "search_android_classes", arguments: { path, query } },
{ timeout: 150_000 },
);
const result = requireMcpOperationResult(
response,
"search_android_classes",
assert.notEqual(response.structuredContent, undefined);
const evidence = parseEvidence(
requireMcpOutput(response, "search_android_classes"),
);
const evidence = parseEvidence(response.structuredContent);
const result = evidence.normalized_result;
assert.equal(evidence.subject.digest.sha256, sha256);
assert.equal(evidence.parameters.query, query);
assert.equal(result.query, query);
+9 -5
View File
@@ -1,3 +1,4 @@
import { requireMcpOutput } from "../../lib/mcp-verifier-results.mjs";
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
@@ -172,8 +173,10 @@ try {
{ name: "inspect_artifact", arguments: {} },
{ timeout: 150_000 },
);
assert.notEqual(inspected.isError, true, JSON.stringify(inspected));
const mcpInspection = parseEvidence(inspected.structuredContent);
assert.notEqual(inspected.structuredContent, undefined);
const mcpInspection = parseEvidence(
requireMcpOutput(inspected, "inspect_artifact"),
);
assert.deepEqual(
mcpInspection.normalized_result,
inspection.normalized_result,
@@ -186,9 +189,10 @@ try {
},
{ timeout: 150_000 },
);
assert.notEqual(response.isError, true, JSON.stringify(response));
assert.ok(response.structuredContent !== undefined);
const mcpEvidence = parseEvidence(response.structuredContent);
assert.notEqual(response.structuredContent, undefined);
const mcpEvidence = parseEvidence(
requireMcpOutput(response, "project_harmony_application_graph"),
);
assert.deepEqual(
mcpEvidence.normalized_result,
projectionEvidence.normalized_result,
+18
View File
@@ -1505,6 +1505,24 @@ exit 1`,
}
});
it("preserves process names printed by Toybox without treating their whitespace as column padding", async () => {
// Toybox's final NAME column is left justified without padding; only the
// single separator after S belongs to the table (toys/posix/ps.c show_ps).
const name = " native tool ";
const { binary } = await writeAdbStub(
shellStub({
ps: `USER PID PPID VSZ RSS WCHAN ADDR S NAME\nroot 1234 567 123456 7890 0 0 S ${name}\r\n`,
}),
);
const result = await evidenceResult(providerOver(binary), {
operation: "list_adb_processes",
input: { serial: "emulator-5554" },
});
expect(result.processes).toEqual([
{ user: "root", pid: 1234, ppid: 567, rss: 7890, state: "S", name },
]);
});
it("lists processes, features, and services through fixed commands", async () => {
const { binary } = await writeAdbStub(
shellStub({
+8 -7
View File
@@ -27,8 +27,10 @@ export interface AdbProcessEntry {
readonly name: string;
}
// Toybox emits one separator before the unpadded final NAME column.
// Remaining whitespace belongs to argv[0], not the table layout.
const PROCESS_LINE =
/^(\S+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(.+)$/u;
/^(\S+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\S+)\s+(\S+)\s+(\S+) (.+)$/u;
const safeInteger = (text: string): number | null => {
const value = Number(text);
@@ -46,22 +48,21 @@ export const parseProcessListOutput = (
const processes: AdbProcessEntry[] = [];
const unparsed = collectUnparsedLines();
let columns: string | null = null;
for (const rawLine of stdout.split(/\r?\n/u)) {
const line = rawLine.trimEnd();
if (line === "") continue;
for (const line of stdout.split(/\r?\n/u)) {
if (line.trim() === "") continue;
if (/^USER\s+PID\s+PPID\b/u.test(line)) {
columns = line.trim();
continue;
}
const match = PROCESS_LINE.exec(line);
if (match === null) {
unparsed.add(rawLine);
unparsed.add(line);
continue;
}
const pid = safeInteger(match[2]!);
const ppid = safeInteger(match[3]!);
const rss = safeInteger(match[5]!);
if (pid === null || ppid === null || rss === null) unparsed.add(rawLine);
if (pid === null || ppid === null || rss === null) unparsed.add(line);
if (pid === null) continue;
processes.push({
user: match[1]!,
@@ -69,7 +70,7 @@ export const parseProcessListOutput = (
ppid,
rss,
state: match[8]!,
name: match[9]!.trim(),
name: match[9]!,
});
}
return { columns, processes, unparsedLines: unparsed.facts() };
+13 -10
View File
@@ -1,3 +1,4 @@
import { snapshotEnvironment } from "../process/snapshotEnvironment.js";
import type { AnalysisSettingsReader } from "../domain/analysisSettings.js";
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { AnalysisCleanupLease } from "../artifacts/AnalysisCleanupLease.js";
@@ -295,7 +296,8 @@ export class JadxProvider implements AndroidAnalysisPort {
/** Report actual local prerequisites without starting JADX or opening a target. */
inspectAvailability(signal?: AbortSignal) {
const pending = inspectJadxAvailability(this.environment, {
const environment = snapshotEnvironment(this.environment);
const pending = inspectJadxAvailability(environment, {
operationTimeoutMs:
this.settings?.().overrides["jadx.operation_timeout_ms"],
resourceOwner: this.#prerequisiteOwner,
@@ -442,10 +444,11 @@ export class JadxProvider implements AndroidAnalysisPort {
request: AndroidRequest,
options?: ExecutionOptions,
): Promise<Outcome> {
const environment = snapshotEnvironment(this.environment);
// Resolve the accepted operation's override before acquiring any resource.
// An unusable launch value is refused without starting the engine.
const timeoutMs = resolveJadxOperationTimeout(
this.environment,
environment,
request.operation,
this.settings?.().overrides["jadx.operation_timeout_ms"],
);
@@ -471,7 +474,7 @@ export class JadxProvider implements AndroidAnalysisPort {
`Target ${target.path} is ${target.format}; provide one standalone APK.`,
);
const configuration = await resolveJadxConfigurationInputs(
this.environment,
environment,
request.operation,
signal,
);
@@ -513,11 +516,11 @@ export class JadxProvider implements AndroidAnalysisPort {
java: configuration.java,
arguments: configuration.jvmArguments,
options: [
this.environment.JAVA_TOOL_OPTIONS,
this.environment._JAVA_OPTIONS,
this.environment.JDK_JAVA_OPTIONS,
this.environment.PATH,
this.environment.JAVA_HOME,
environment.JAVA_TOOL_OPTIONS,
environment._JAVA_OPTIONS,
environment.JDK_JAVA_OPTIONS,
environment.PATH,
environment.JAVA_HOME,
],
});
if (this.#retained !== undefined && this.#retained.key !== key) {
@@ -528,7 +531,7 @@ export class JadxProvider implements AndroidAnalysisPort {
if (retained === undefined) {
await validateJadxConfiguration(
configuration,
this.environment,
environment,
request.operation,
{
resourceOwner: this.#prerequisiteOwner,
@@ -589,7 +592,7 @@ export class JadxProvider implements AndroidAnalysisPort {
bridge,
],
cwd: root.path,
hostEnvironment: this.environment,
hostEnvironment: environment,
signal: workspaceSignal,
},
this.launcher,
+13 -2
View File
@@ -13,6 +13,7 @@ import { silentLogger } from "../logger.js";
import type { Logger } from "pino";
import { withEvidenceCallerInputPath } from "../domain/evidence.js";
import { createImmutableEvidenceBundle } from "../domain/evidenceBundle.js";
import { retainAnalysisResultSources } from "./investigation/recordEvidenceSources.js";
import type { Evidence } from "../domain/evidence.js";
import type { NativeToolName } from "../contracts/native/nativeToolContracts.js";
import {
@@ -570,12 +571,17 @@ const executeAnalysisTool = async (input: {
if (tool === "analyze_function") {
const { executeFunctionAnalysisEvidence } =
await import("./FunctionAnalysisEvidence.js");
const result = await executeFunctionAnalysisEvidence(
let result = await executeFunctionAnalysisEvidence(
session,
input.arguments,
input.openedTarget,
{ signal },
);
result = retainAnalysisResultSources(
tool,
(source) => session.recordEvidence(source),
result,
);
return result.ok ? ok({ evidence: result.value }) : result;
}
if (
@@ -588,12 +594,17 @@ const executeAnalysisTool = async (input: {
tool === "decompile_all"
) {
const { EnhancedTools } = await import("./EnhancedTools.js");
const result = await executeComposedWorkflow(
let result = await executeComposedWorkflow(
session,
(observed) =>
new EnhancedTools(observed).execute(tool, input.arguments, signal),
composedLimitationBudget(tool),
);
result = retainAnalysisResultSources(
tool,
(source) => session.recordEvidence(source),
result,
);
if (!result.ok) return result;
const evidence = createWorkflowEvidence({
target: input.openedTarget,
@@ -2,7 +2,7 @@ import { expect, it } from "vitest";
import { createAnalysisExecution } from "./AnalysisProvider.js";
import { executeFunctionAnalysisEvidence } from "./FunctionAnalysisEvidence.js";
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
import { parseEvidence } from "../domain/evidence.js";
import { createEvidence, parseEvidence } from "../domain/evidence.js";
import { createAnalysisProfile } from "../domain/analysisProfile.js";
import { ghidraFunctionDossier } from "../domain/ghidraValues.fixture.js";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
@@ -62,6 +62,16 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
path: "/observed/provider.elf",
sha256: "b".repeat(64),
};
const source = createEvidence(
undefined,
provider,
{
operation: "health",
parameters: { facet: "historical-report" },
result: { original: "unknown" },
},
{ immutable: true },
);
for (const aborted of [false, true]) {
const controller = new AbortController();
const execution = createAnalysisExecution(dossier, provider, {
@@ -73,6 +83,7 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
rawResult: { provider_text: "preserved" },
limitations: ["Synthetic provider observation"],
locations: [{ kind: "address", address: "0x401000" }],
sourceEvidence: [source],
});
const result = await executeFunctionAnalysisEvidence(
{
@@ -85,6 +96,7 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
target,
{ signal: controller.signal },
);
expect(result.sourceEvidence).toEqual([source]);
if (aborted) {
if (result.ok) throw new Error("Expected late cancellation");
expect(result.error).toMatchObject({ _tag: "AnalysisCancelledError" });
@@ -100,6 +112,7 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
expect(evidence.provider).toEqual(provider);
expect(evidence.analysis_profile).toEqual(analysisProfile);
expect(evidence.relationship_coverage).toEqual(relationshipCoverage);
expect(evidence.evidence_links).toEqual([source.evidence_id]);
expect(evidence.limitations).toContain("Synthetic provider observation");
expect(evidence.locations).toContainEqual({
kind: "address",
+34 -24
View File
@@ -5,10 +5,17 @@ import type {
} from "./AnalysisProvider.js";
import { nativeProcedureInputSchema } from "../domain/native/nativeProcedureInput.js";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
import { createEvidence } from "../domain/evidence.js";
import { createAnalysisExecutionEvidence } from "../domain/analysisObservation.js";
import { parseFunctionDossier } from "../domain/hopperValues.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { completeProviderEvidence } from "./ProviderOperationExecution.js";
import type { Evidence } from "../domain/evidence.js";
import { EvidenceLedger } from "./investigation/EvidenceLedger.js";
import {
recordEvidenceSources,
withEvidenceSources,
type AnalysisResultWithSources,
} from "./investigation/recordEvidenceSources.js";
/**
* Preserve the provider's observations when presenting a validated function
@@ -20,7 +27,7 @@ export const executeFunctionAnalysisEvidence = async (
parameters: Readonly<Record<string, JsonValue>>,
target: BinaryTarget | undefined,
options?: ExecutionOptions,
) => {
): Promise<AnalysisResultWithSources<Evidence>> => {
const input = parseAnalysisInput(
"analyze_function",
parameters,
@@ -35,30 +42,33 @@ export const executeFunctionAnalysisEvidence = async (
options,
);
if (!execution.ok) return execution;
const sourceLedger = new EvidenceLedger();
const recorded = recordEvidenceSources(
(source) => sourceLedger.record(source),
execution.value.sourceEvidence ?? [],
);
const sources = sourceLedger.recordsForAnalysis();
if (!recorded.ok) return withEvidenceSources(recorded, sources);
const dossier = parseFunctionDossier(execution.value.result);
if (!dossier.ok) return dossier;
if (!dossier.ok) return withEvidenceSources(dossier, sources);
const observation = execution.value;
return completeProviderEvidence(
createEvidence(
observation.subject === undefined
? target
: (observation.subject ?? undefined),
observation.provider,
{
operation: "analyze_function",
parameters: arguments_,
result: dossier.value,
rawResult: observation.rawResult,
limitations: observation.limitations,
locations: observation.locations,
...(observation.relationshipCoverage === undefined
? {}
: { relationshipCoverage: observation.relationshipCoverage }),
...(observation.analysisProfile === undefined
? {}
: { analysisProfile: observation.analysisProfile }),
},
return withEvidenceSources(
completeProviderEvidence(
createAnalysisExecutionEvidence(
"analyze_function",
arguments_,
{
...observation,
sourceEvidence: sources,
result: dossier.value,
...(observation.subject === undefined && target !== undefined
? { subject: target }
: {}),
},
"observed",
),
options?.signal,
),
options?.signal,
sources,
);
};
+42
View File
@@ -19,6 +19,48 @@ import { projectNativeApiInspection } from "./native/NativeApiInspection.js";
import { functionDossierSchema } from "../domain/hopperValues.js";
import { ghidraFunctionDossier } from "../domain/ghidraValues.fixture.js";
import { ANALYSIS_SNAPSHOT_PROFILE } from "../domain/analysisSnapshot.fixture.js";
import { createEvidence } from "../domain/evidence.js";
it("links shared provider source Evidence once without copying its payload into the workflow", async () => {
const source = createEvidence(
undefined,
{ id: "fixture", name: "Fixture", version: "1" },
{
operation: "health",
parameters: { facet: "saved-analysis-report" },
result: { report: "historical content ".repeat(1_000) },
},
{ immutable: true },
);
const result = await executeComposedWorkflow(
{
execute: async () =>
ok(
createAnalysisExecution(
"observed",
{ id: "fixture", name: "Fixture", version: "1" },
{ sourceEvidence: [source] },
),
),
},
async (observed) => {
await observed.execute("list_procedures", {});
await observed.execute("procedure_pseudo_code", { procedure: "0x1000" });
return ok({ count: 1 });
},
);
expect(result).toEqual({
...ok({
result: { count: 1 },
limitations: [],
evidenceLinks: [source.evidence_id],
}),
sourceEvidence: [source],
});
expect(result.ok && JSON.stringify(result.value)).not.toContain(
"historical content",
);
});
it.each([false, true])(
"propagates transient=%s through composed snapshot decisions while retaining Evidence",
+41 -25
View File
@@ -23,6 +23,12 @@ import {
import type { JsonValue } from "../domain/jsonValue.js";
import type { UnknownRegistryPort } from "./investigation/InvestigationRecordPort.js";
import { UnknownRegistryError } from "../domain/unknownRegistryError.js";
import { EvidenceLedger } from "./investigation/EvidenceLedger.js";
import {
recordEvidenceSources,
withEvidenceSources,
type AnalysisResultWithSources,
} from "./investigation/recordEvidenceSources.js";
import { workflowUnknownInput } from "./investigation/WorkflowUnknown.js";
import {
renderSourceLimitations,
@@ -71,18 +77,16 @@ export const executeComposedWorkflow = async (
) => Promise<Result<JsonValue, AnalysisError>>,
budget?: ComposedLimitationBudget,
): Promise<
Result<
{
readonly result: JsonValue;
readonly limitations: readonly string[];
readonly cachePolicy?: "live";
readonly evidenceLinks?: readonly string[];
},
AnalysisError
>
AnalysisResultWithSources<{
readonly result: JsonValue;
readonly limitations: readonly string[];
readonly cachePolicy?: "live";
readonly evidenceLinks?: readonly string[];
}>
> => {
const limitations = new Map<string, SourceLimitation>();
const evidenceLinks = new Set<string>();
const sourceLedger = new EvidenceLedger();
const perItemGroups = new Set<string>();
let perItemBytes = 0;
let omittedObservations = 0;
@@ -93,6 +97,13 @@ export const executeComposedWorkflow = async (
const execution = await analysis.execute(operation, parameters, options);
if (execution.ok && execution.value.evidenceId !== undefined)
evidenceLinks.add(execution.value.evidenceId);
if (execution.ok && execution.value.sourceEvidence !== undefined) {
const recorded = recordEvidenceSources(
(source) => sourceLedger.record(source),
execution.value.sourceEvidence,
);
if (!recorded.ok) return recorded;
}
if (execution.ok && execution.value.cachePolicy === "live")
cachePolicy = "live";
if (execution.ok)
@@ -129,22 +140,27 @@ export const executeComposedWorkflow = async (
return execution;
},
});
if (!result.ok) return result;
return ok({
result: result.value,
limitations: [
...renderSourceLimitations([...limitations.values()]),
...(omittedObservations === 0
? []
: [
`${omittedObservations} per-item provider limitation observation${omittedObservations === 1 ? " was" : "s were"} omitted from this answer; ${budget?.retainedIn}.`,
]),
],
...(cachePolicy === undefined ? {} : { cachePolicy }),
...(evidenceLinks.size === 0
? {}
: { evidenceLinks: [...evidenceLinks].sort(compareUnicodeCodePoints) }),
});
const sources = sourceLedger.recordsForAnalysis();
if (!result.ok) return withEvidenceSources(result, sources);
for (const source of sources) evidenceLinks.add(source.evidence_id);
return withEvidenceSources(
ok({
result: result.value,
limitations: [
...renderSourceLimitations([...limitations.values()]),
...(omittedObservations === 0
? []
: [
`${omittedObservations} per-item provider limitation observation${omittedObservations === 1 ? " was" : "s were"} omitted from this answer; ${budget?.retainedIn}.`,
]),
],
...(cachePolicy === undefined ? {} : { cachePolicy }),
...(evidenceLinks.size === 0
? {}
: { evidenceLinks: [...evidenceLinks].sort(compareUnicodeCodePoints) }),
}),
sources,
);
};
/** Build the shared Evidence record for a successful composed binary workflow. */
@@ -433,6 +433,131 @@ describe("analysis provider registry: selection failures", () => {
});
});
describe("analysis provider registry: rejection origins", () => {
it("distinguishes recorded readiness failures from an unobserved target refusal", async () => {
const registry = new AnalysisProviderRegistry([
candidate("alpha", { available: false, unconfigured: true }).provider,
candidate("gamma", { available: false, unconfigured: true }).provider,
]);
const selection = await registry.select(DATABASE_TARGET);
expect(selection.ok).toBe(true);
if (!selection.ok) return;
const projected = projectAnalysisError(
registry.unboundOperationError("binary_overview", selection.value),
);
expect(projected.message).toContain(
"Provider selection failed local readiness checks. No target-support refusal was reported.",
);
});
it("reports a configured provider's readiness failure without hiding a ready alternative", async () => {
const invalid = candidate("alpha", { available: false });
const ready = candidate("beta");
const registry = new AnalysisProviderRegistry([
{
...invalid.provider,
inspectAvailability: () => ({
status: "unavailable",
code: "open_options_invalid",
reason: "Configured registration is invalid",
diagnostics: { registration_path: "/configured/alpha.json" },
}),
},
ready.provider,
]);
const selected = await registry.select(DATABASE_TARGET, "alpha");
if (selected.ok)
throw new Error("Expected the selected provider to refuse");
const projected = projectAnalysisError(selected.error);
expect(projected.message).toContain("local readiness");
expect(projected.message).not.toContain(
"No analysis provider is configured",
);
expect(projected.details).toMatchObject({
requested_provider_id: "alpha",
candidate_ids: ["alpha", "beta"],
rejections: [
{
provider_id: "alpha",
origin: "availability",
code: "open_options_invalid",
reason: "Configured registration is invalid",
diagnostics: { registration_path: "/configured/alpha.json" },
},
],
});
expect((await registry.select(DATABASE_TARGET, "beta")).ok).toBe(true);
expect(invalid.created).toEqual([]);
});
it("names whether a rejection came from provider availability or target support", async () => {
const availabilityOnly = new AnalysisProviderRegistry([
candidate("alpha", { available: false, unconfigured: true }).provider,
candidate("gamma", { available: false, unconfigured: true }).provider,
]);
const availabilitySelection =
await availabilityOnly.select(DATABASE_TARGET);
expect(availabilitySelection.ok).toBe(true);
if (availabilitySelection.ok) {
const { details } = projectAnalysisError(
availabilityOnly.unboundOperationError(
"binary_overview",
availabilitySelection.value,
),
);
const rejections = details?.rejections as unknown as readonly {
origin: string;
}[];
expect(rejections.map(({ origin }) => origin)).toEqual([
"availability",
"availability",
]);
}
const unconfiguredButAdmitting: AnalysisProviderCandidate = {
...candidate("alpha", { available: false, unconfigured: true }).provider,
inspectTargetSupport: () => ({
status: "supported",
code: null,
reason: null,
diagnostics: { target_kind: "archive" },
}),
};
const refusingTarget: AnalysisProviderCandidate = {
...candidate("beta").provider,
inspectTargetSupport: () => ({
status: "unsupported",
code: "target_format_unsupported",
reason: "beta refuses archives",
diagnostics: { target_kind: "archive" },
}),
};
const mixed = new AnalysisProviderRegistry([
unconfiguredButAdmitting,
refusingTarget,
]);
const mixedSelection = await mixed.select(ARTIFACT_TARGET);
expect(mixedSelection.ok).toBe(true);
if (mixedSelection.ok) {
const projected = projectAnalysisError(
mixed.unboundOperationError("binary_overview", mixedSelection.value),
);
expect(projected.message).toContain(
"The selected analysis provider is unavailable",
);
const { details } = projected;
const rejections = details?.rejections as unknown as readonly {
origin: string;
code: string;
}[];
expect(rejections).toMatchObject([
{ origin: "availability", code: "not_configured" },
{ origin: "target", code: "target_format_unsupported" },
]);
}
});
});
describe("analysis provider registry: profile and cleanup failures", () => {
it("does not bind a profile with unresolved version or adapter failure", async () => {
const unresolved = candidate("unresolved", { profile: "missing" });
@@ -21,6 +21,7 @@ import {
import { AnalysisError } from "../../domain/analysisErrorBase.js";
import {
ProviderSelectionError,
type ProviderRejectionOrigin,
type ProviderSelectionErrorOptions,
type ProviderSelectionFailureReason,
type ProviderSelectionRejection,
@@ -880,21 +881,43 @@ const selectionError = (
const candidateRejections = (
candidates: readonly AnalysisProviderCandidateStatus[],
): readonly ProviderSelectionRejection[] =>
candidates.flatMap((candidate) =>
[
...(candidate.availability.status === "unavailable"
? [candidate.availability]
: []),
...(candidate.targetSupport.status === "unsupported"
? [candidate.targetSupport]
: []),
].map((rejection) => ({
providerId: candidate.provider.id,
code: rejection.code,
reason: rejection.reason,
diagnostics: structuredClone(rejection.diagnostics),
})),
);
candidates.flatMap((candidate) => {
const rejections: ProviderSelectionRejection[] = [];
if (candidate.availability.status === "unavailable")
rejections.push(
selectedRejection(candidate, "availability", candidate.availability),
);
if (candidate.targetSupport.status === "unsupported")
rejections.push(
selectedRejection(candidate, "target", candidate.targetSupport),
);
return rejections;
});
/** One availability or target-support observation that rejected a candidate. */
type RejectionObservation =
| Extract<
AnalysisProviderCandidateStatus["availability"],
{ status: "unavailable" }
>
| Extract<
AnalysisProviderCandidateStatus["targetSupport"],
{ status: "unsupported" }
>;
/** Name the observation a rejection came from so availability cannot read as a refusal. */
const selectedRejection = (
candidate: AnalysisProviderCandidateStatus,
origin: ProviderRejectionOrigin,
rejection: RejectionObservation,
): ProviderSelectionRejection => ({
providerId: candidate.provider.id,
origin,
code: rejection.code,
reason: rejection.reason,
// ProviderSelectionError owns the single defensive diagnostic copy.
diagnostics: rejection.diagnostics,
});
const cloneAvailability = (
availability: AnalysisProviderCandidateStatus["availability"],
+9 -5
View File
@@ -1254,6 +1254,15 @@ export class BinarySession
)
this.#active.nativeSourceAdmitted = true;
this.#observeRuntimeAvailability(name, execution);
// Source facts belong to the observation, even when this query cannot be
// replayed (for example a Ghidra procedure read without a document selector).
if (execution.ok && execution.value.sourceEvidence !== undefined) {
const sourcesRecorded = recordEvidenceSources(
(source) => this.recordEvidence(source),
execution.value.sourceEvidence,
);
if (!sourcesRecorded.ok) return sourcesRecorded;
}
if (
execution.ok &&
cacheable &&
@@ -1276,11 +1285,6 @@ export class BinarySession
},
"observed",
);
const sourcesRecorded = recordEvidenceSources(
(source) => this.recordEvidence(source),
execution.value.sourceEvidence ?? [],
);
if (!sourcesRecorded.ok) return sourcesRecorded;
const recorded = this.recordEvidence(evidence);
if (!recorded.ok) return recorded;
const retained = { ...execution.value, evidenceId: evidence.evidence_id };
@@ -1,14 +1,16 @@
import { basename } from "node:path";
import { isIP } from "node:net";
import { isJsonLike } from "../../domain/analysisView/EnvironmentManifestAccumulator.js";
import {
isJsonLike,
mediaTypeEssence,
} from "../../domain/analysisView/recordedMediaType.js";
import {
decodeRetainedBase64,
readRecordedResponseBody,
type RecordedResponseBody,
} from "../../domain/analysisView/recordedResponseBody.js";
import { scenarioContentMediaType } from "../../domain/analysisView/scenarioCaptureRecords.js";
import { mediaTypeEssence } from "../../domain/analysisView/webNetworkRouteTemplates.js";
import type { BrowserScenarioEvent } from "../../domain/browserScenarioCaptureValues.js";
import type { EnvironmentBuildReport } from "../../domain/environmentBundleExport.js";
import {
@@ -10,6 +10,7 @@ import {
environmentStateServeFixture,
} from "../../../tests/fixtures/environmentServe.js";
import { compareEnvironment } from "./environmentCompare.js";
import { requestBody } from "./environmentCompareProbe.js";
import { REPLAY_BODY_LIMIT_BYTES } from "./environmentSource.js";
const responseCapture = (
@@ -144,3 +145,24 @@ describe("environment comparison response representation", () => {
expect(compare(json, text).probes[0]?.results[0]?.equal).toBe(false);
});
});
it.each([
{ mimeType: " TEXT/PLAIN ; charset=utf-8", expected: { value: 1 } },
{ mimeType: "application/problem+json", expected: { value: 1 } },
{ mimeType: "text/problem+json", expected: '{"value":1}' },
])(
"reads probe request arguments independently of body-size and parameter admission: $mimeType",
({ mimeType, expected }) => {
const capture = environmentManifestCaptureFixture([
{
request: {
method: "POST",
url: "http://app.example/api/value",
bodySize: "not-a-size",
postData: { mimeType, text: '{"value":1}', params: 7 },
},
},
]);
expect(requestBody(capture.records[0]!)).toEqual(expected);
},
);
@@ -1,11 +1,15 @@
import { z } from "zod";
import { isJsonLike } from "../../domain/analysisView/EnvironmentManifestAccumulator.js";
import { readRecordedRequestBody } from "../../domain/analysisView/recordedExchangeBody.js";
import {
isJsonLike,
mediaTypeEssence,
} from "../../domain/analysisView/recordedMediaType.js";
import {
primaryOrigin,
type OriginStats,
} from "../../domain/analysisView/environmentManifestOrigin.js";
import {
mediaTypeEssence,
readRecordedRequestLine,
type RecordedRequestLine,
type RequestLineExclusion,
@@ -44,12 +48,6 @@ export interface ProbeStep {
const BODYLESS_STATUSES: ReadonlySet<number> = new Set([204, 205, 304]);
const startedSchema = z.object({ startedDateTime: z.string() });
const postDataSchema = z.object({
request: z.object({
postData: z.object({ mimeType: z.string(), text: z.string() }),
}),
});
const methodSchema = z.object({ request: z.object({ method: z.string() }) });
const startedInstant = (
@@ -90,9 +88,11 @@ export const sendsBodies = (
export const requestBody = (
record: WebNetworkCaptureRecord,
): JsonValue | undefined => {
const parsed = postDataSchema.safeParse(record.reported);
if (!parsed.success) return undefined;
const { mimeType, text } = parsed.data.request.postData;
const facts = readRecordedRequestBody(record.reported);
if (facts.kind !== "request" || facts.postData.kind !== "object")
return undefined;
const { mimeType, text } = facts.postData;
if (mimeType === undefined || text === undefined) return undefined;
return isJsonLike(mediaTypeEssence(mimeType))
? (parseExactJson(text) ?? text)
: text;
@@ -1,5 +1,10 @@
import { z } from "zod";
import {
isJsonLike,
mediaTypeEssence,
} from "../../domain/analysisView/recordedMediaType.js";
import {
analysisViewResultSchema,
type AnalysisViewResult,
@@ -9,9 +14,7 @@ import {
readRecordedResponseBody,
type RecordedResponseBody,
} from "../../domain/analysisView/recordedResponseBody.js";
import { isJsonLike } from "../../domain/analysisView/EnvironmentManifestAccumulator.js";
import {
mediaTypeEssence,
readRecordedRequestLine,
type RecordedRequestLine,
} from "../../domain/analysisView/webNetworkRouteTemplates.js";
@@ -76,6 +76,7 @@ const expectReloadsUnchanged = (books: Served): void => {
describe("confirmed write response bodies", () => {
it.each<{
name: string;
status?: number;
content: Record<string, JsonValue>;
shape: "recorded" | "empty";
bodyRecorded: boolean;
@@ -96,6 +97,23 @@ describe("confirmed write response bodies", () => {
shape: "recorded",
bodyRecorded: false,
},
{
name: "205 without retained text",
status: 205,
content: { mimeType: "text/html" },
shape: "recorded",
bodyRecorded: false,
},
...[undefined, null].map((text) => ({
name: `zero-size HTML with ${text === null ? "null" : "omitted"} text`,
content: {
mimeType: "text/html",
size: 0,
...(text === null ? { text } : {}),
},
shape: "recorded" as const,
bodyRecorded: false,
})),
{
name: "known empty text",
content: { mimeType: "application/json", size: 0, text: "" },
@@ -104,7 +122,7 @@ describe("confirmed write response bodies", () => {
},
])(
"preserves $name when serving a confirmed document write",
({ content, shape, bodyRecorded }) => {
({ content, shape, bodyRecorded, status = 200 }) => {
const entries: Record<string, JsonValue>[] = [
{
request: { method: "GET", url: "http://app.example/api/settings" },
@@ -125,7 +143,7 @@ describe("confirmed write response bodies", () => {
text: '{"theme":"dark"}',
},
},
response: { status: 200, content },
response: { status, content },
},
{
request: { method: "GET", url: "http://app.example/api/settings" },
@@ -169,7 +187,7 @@ describe("confirmed write response bodies", () => {
const write = respond(patch, { body: { theme: "blue" } });
if (!write.ok) throw write.error;
expect(write.value).toMatchObject({
status: 200,
status,
body: null,
replay: {
record_ordinal: 1,
@@ -26,7 +26,13 @@ import { webNetworkCaptureSchema } from "../../domain/webNetworkCapture.js";
const archived = (value: boolean) => ({ ...FIRST_BOOK, archived: value });
describe("bodyless action evidence", () => {
it.each([
it.each<{
name: string;
postData: unknown;
bodySize?: unknown;
served?: boolean;
shape?: boolean;
}>([
{
name: "form",
postData: {
@@ -46,9 +52,45 @@ describe("bodyless action evidence", () => {
},
},
{ name: "unretained body", postData: undefined },
{
name: "omitted body with zero size",
postData: undefined,
bodySize: 0,
served: true,
},
{ name: "explicit null with zero size", postData: null, bodySize: 0 },
{
name: "params-only with zero size",
postData: {
mimeType: "application/json",
params: [{ name: "archived", value: "true" }],
},
bodySize: 0,
},
{
name: "empty text with nonempty params",
postData: {
mimeType: "application/json",
text: "",
params: [{ name: "archived", value: "true" }],
},
bodySize: 0,
},
{
name: "explicit empty body with zero size",
postData: { mimeType: "application/json", text: "", params: [] },
bodySize: 0,
served: true,
},
{
name: "invalid size with retained JSON",
postData: { mimeType: "application/json", text: '{"archived":true}' },
bodySize: "unknown",
shape: true,
},
])(
"does not use unreadable or unretained request bodies as a repeated bodyless action: $name",
({ postData }) => {
"preserves literal request-body availability for repeated actions: $name",
({ postData, bodySize = 12, served = false, shape = false }) => {
const capture = timedCaptureEvidence([
["GET", "/api/books", [archived(false), SECOND_BOOK]],
["GET", "/api/books/1", archived(false)],
@@ -60,14 +102,18 @@ describe("bodyless action evidence", () => {
]);
const result = capture.normalized_result as {
records: {
reported: { request: { postData?: unknown; bodySize?: number } };
reported: { request: { postData?: unknown; bodySize?: unknown } };
}[];
};
const request = result.records[5]!.reported.request;
request.bodySize = 12;
request.bodySize = bodySize;
if (postData === undefined) delete request.postData;
else request.postData = postData;
const { state } = environmentStateServeInputs(
if (shape) {
result.records[3]!.reported.request.bodySize = bodySize;
result.records[3]!.reported.request.postData = postData;
}
const { source, state } = environmentStateServeInputs(
environmentStateServeFixture(
analysisViewCaptureEvidence(
webNetworkCaptureSchema.parse(capture.normalized_result),
@@ -75,8 +121,17 @@ describe("bodyless action evidence", () => {
),
);
expect(state.writes.get("post_books_archive_by_id")).toMatchObject({
served: false,
served,
});
const tool = source.manifest.tools.find(
({ name }) => name === "post_books_archive_by_id",
);
expect(tool).toBeDefined();
if (shape)
expect(tool?.body_shape).toMatchObject({
properties: { archived: "boolean" },
});
else if (bodySize === 0) expect(tool?.body_shape).toBeNull();
},
);
});
@@ -198,6 +198,11 @@ export class EvidenceLedger {
);
}
/** Borrow immutable source records without constructing or hashing a bundle. */
recordsForAnalysis(): readonly Evidence[] {
return [...this.#records.values()];
}
/** Check whether one immutable Evidence ID is present in this ledger. */
has(evidenceId: string): boolean {
return this.#records.has(evidenceId);
@@ -0,0 +1,32 @@
import { expect, it } from "vitest";
import { createEvidence } from "../../domain/evidence.js";
import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js";
import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
import { err, ok } from "../../domain/result.js";
import {
retainAnalysisResultSources,
withEvidenceSources,
} from "./recordEvidenceSources.js";
it.each([false, true])(
"reports source recording failure without replacing earlier cancellation (cancelled=%s)",
(cancelled) => {
const source = createEvidence(
undefined,
{ id: "fixture", name: "Fixture", version: "1" },
{ operation: "health", parameters: {}, result: "historical" },
);
const primary = new AnalysisCancelledError("analyze_function", {
cause: "caller",
});
const failure = new EvidenceIntegrityError("Recorder failed");
const result = retainAnalysisResultSources(
"analyze_function",
() => err(failure),
withEvidenceSources(cancelled ? err(primary) : ok("complete"), [source]),
);
expect(result.ok).toBe(false);
if (result.ok) throw new Error("Expected recording failure");
expect(result.error).toBe(cancelled ? primary : failure);
},
);
@@ -1,7 +1,49 @@
import type { EvidenceWriter } from "./InvestigationRecordPort.js";
import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
import type { Evidence } from "../../domain/evidence.js";
import { ok, type Result } from "../../domain/result.js";
import { err, ok, type Result } from "../../domain/result.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import { AnalysisCapabilityUnavailableError } from "../../domain/analysisErrorCore.js";
/** Source facts remain available even when a completed observation is cancelled. */
export type AnalysisResultWithSources<Value> = Result<Value, AnalysisError> & {
readonly sourceEvidence?: readonly Evidence[];
};
export const withEvidenceSources = <
Outcome extends Result<unknown, AnalysisError>,
>(
result: Outcome,
sources: readonly Evidence[],
): Outcome & { readonly sourceEvidence?: readonly Evidence[] } =>
sources.length === 0 ? result : { ...result, sourceEvidence: sources };
/** Never deliver links to records a plain-port adapter cannot retain. */
export const retainAnalysisResultSources = <Value>(
operation: string,
recordEvidence: EvidenceWriter["recordEvidence"] | undefined,
result: AnalysisResultWithSources<Value>,
): Result<Value, AnalysisError> => {
const sources = result.sourceEvidence ?? [];
const retained =
sources.length > 0 && recordEvidence === undefined
? err(
new AnalysisCapabilityUnavailableError(
"rea",
operation,
"Provider source Evidence requires an explicit retention owner.",
{
userMessage:
"Use an analysis session that retains provider source Evidence, or supply an Evidence recorder when registering this adapter.",
},
),
)
: recordEvidenceSources(recordEvidence, sources);
if (retained.ok) return result;
if (result.ok) return retained;
result.error.retainSecondaryFailure(retained.error);
return result;
};
/** Record source evidence in order, stopping at the first session rejection. */
export const recordEvidenceSources = (
+1 -1
View File
@@ -21,7 +21,7 @@ import { readAssemblyReferenceName } from "../dotnet/ManagedMetadataInventoryRow
import { ManagedReaderFailure } from "../dotnet/ManagedReaderFailure.js";
import { ArtifactReaderFailure } from "./ArtifactReader.js";
import { abortIfNeeded } from "./ArtifactHash.js";
import { hasMachoMagic } from "./apple/MachoLoadCommandReader.js";
import { hasMachoMagic } from "../domain/apple/machoContainer.js";
export interface DesktopReadBudget {
bytes: number;
@@ -728,3 +728,48 @@ it("retains SEA admission for a complete segment with a nonaligned command size"
await handle.close();
}
});
it.each([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe])(
"classifies four-byte thin magic %i without admitting a truncated SEA carrier",
async (magic) => {
const bytes = Buffer.alloc(4);
bytes.writeUInt32BE(magic);
const path = await writeCarrier("thin-prefix", bytes);
const inventory = await scanCanonicalArtifactInventory(path);
expect(inventory.manifest.root_format).toBe("mach-o");
const handle = await open(path, "r");
try {
expect(await locateNodeSea(handle, bytes.length)).toBeUndefined();
} finally {
await handle.close();
}
},
);
it("confirms SEA naming before rejecting an unrelated cyclic PE resource branch", async () => {
const fixture = peResourceFixture(true, [
{ type: 3, name: 1, language: 1033, payload: Buffer.from("other") },
{
type: 10,
name: "NODE_SEA_BLOB",
language: 1033,
payload: buildNodeSeaBlob(populated),
},
]);
const other = fixture.entryOffsets.find(
(at) => fixture.bytes.readUInt32LE(at) === 3,
);
if (other === undefined) throw new Error("Expected unrelated resource type");
fixture.bytes.writeUInt32LE(0x80000000, other + 4);
const handle = await open(
await writeCarrier("sea-with-unrelated-cycle", fixture.bytes),
"r",
);
try {
await expect(
locateNodeSea(handle, fixture.bytes.length),
).rejects.toMatchObject({ reason: "format" });
} finally {
await handle.close();
}
});
+66
View File
@@ -0,0 +1,66 @@
import { err, ok, type Result } from "../../domain/result.js";
/**
* Preserve REA's source integer interpretation, including uint64
* wrapping for unusual widths. Callers own byte-range and safe-number admission.
*/
export const readBinaryPlistUnsigned64 = (
bytes: Uint8Array,
offset: number,
width: number,
): bigint => {
let value = 0n;
for (let index = 0; index < width; index += 1)
value = BigInt.asUintN(
64,
(value << 8n) | BigInt(bytes[offset + index] ?? 0),
);
return value;
};
/** Raw declarations tied to one source or decoder-copy byte representation. */
export interface BinaryPlistLayout {
readonly trailerOffset: number;
readonly offsetWidth: number;
readonly referenceWidth: number;
readonly objectCount: bigint;
readonly topObject: bigint;
readonly offsetTable: bigint;
readonly objectOffset: (object: number) => bigint;
readonly objectOffsets: () => Generator<bigint>;
}
/** Observe declarations only; signature, bounds, capacities and reachability remain caller policy. */
export const readBinaryPlistLayout = (
bytes: Uint8Array,
): Result<
BinaryPlistLayout,
{ readonly _tag: "InvalidBinaryPlistLayout"; readonly reason: "truncated" }
> => {
if (bytes.byteLength < 32)
return err({ _tag: "InvalidBinaryPlistLayout", reason: "truncated" });
const trailerOffset = bytes.byteLength - 32;
const offsetWidth = bytes[trailerOffset + 6]!;
const referenceWidth = bytes[trailerOffset + 7]!;
const objectCount = readBinaryPlistUnsigned64(bytes, trailerOffset + 8, 8);
const offsetTable = readBinaryPlistUnsigned64(bytes, trailerOffset + 24, 8);
const objectOffset = (object: number): bigint =>
readBinaryPlistUnsigned64(
bytes,
Number(offsetTable) + object * offsetWidth,
offsetWidth,
);
return ok({
trailerOffset,
offsetWidth,
referenceWidth,
objectCount,
topObject: readBinaryPlistUnsigned64(bytes, trailerOffset + 16, 8),
offsetTable,
objectOffset,
*objectOffsets() {
for (let object = 0; object < Number(objectCount); object += 1)
yield objectOffset(object);
},
});
};
@@ -1,5 +1,8 @@
import { expect, it } from "vitest";
import { binaryPlistObjectTable } from "../../../tests/fixtures/binaryPlistObjectTable.js";
import { createPlistNumberProjection } from "../../domain/apple/plistNumbers.js";
import { projectPlistValue } from "../../domain/apple/plistValue.js";
import { scanBinaryPlistScalars } from "./PlistScalars.js";
import { plistUidValue } from "../../domain/apple/plistUid.js";
import { decodeBinaryPlistValue } from "./BinaryPlistMaterializer.js";
import { estimateBinaryPlistExpansion } from "./PropertyListDecodeBudget.js";
@@ -80,3 +83,67 @@ it("keeps a source UID distinct from an ordinary UID-shaped dictionary", () => {
expect(plistUidValue(value[0])).toBe(7);
expect(plistUidValue(value[1])).toBeUndefined();
});
it("materializes normalized three-byte tables from a nonzero root without losing source numbers", () => {
const tableOffset = 36;
const bytes = Buffer.alloc(tableOffset + 9 + 32);
bytes.write("bplist00");
bytes[8] = 0x14;
bytes.writeBigUInt64BE(1n, 9);
bytes.writeBigUInt64BE(2n, 17);
bytes[25] = 0xa2;
bytes.writeUIntBE(0, 26, 3);
bytes.writeUIntBE(2, 29, 3);
bytes[32] = 0x82;
bytes.writeUIntBE(0xabcdef, 33, 3);
for (const [object, offset] of [8, 25, 32].entries())
bytes.writeUIntBE(offset, tableOffset + object * 3, 3);
const trailer = bytes.length - 32;
bytes[trailer + 6] = 3;
bytes[trailer + 7] = 3;
bytes.writeBigUInt64BE(3n, trailer + 8);
bytes.writeBigUInt64BE(1n, trailer + 16);
bytes.writeBigUInt64BE(BigInt(tableOffset), trailer + 24);
const original = Buffer.from(bytes);
const plan = estimateBinaryPlistExpansion(bytes, 1024 * 1024, {
numberProjection: true,
uidProjection: true,
});
const metadata = scanBinaryPlistScalars(bytes, plan);
expect(metadata.wideIntegers).toEqual([
{ offset: 8, value: 18446744073709551618n },
]);
expect(metadata.literals.incomplete).toBe(false);
const decoded = decodeBinaryPlistValue(bytes, plan, metadata) as unknown[];
expect(
createPlistNumberProjection(metadata.literals).project(
projectPlistValue(decoded[0]).value,
),
).toEqual({
$plist_type: "integer",
decimal: "18446744073709551618",
});
expect(plistUidValue(decoded[1])).toBe(0xabcdef);
expect(decoded).toHaveLength(2);
expect(bytes.equals(original)).toBe(true);
});
it("keeps uint64 wrapping for nine-byte offset and reference declarations", () => {
const bytes = Buffer.alloc(70);
bytes.write("bplist00");
bytes.set([0x10, 7, 0xa1], 8);
bytes[11] = 0xff;
bytes[20] = 0xff;
bytes.writeBigUInt64BE(8n, 21);
bytes[29] = 0xff;
bytes.writeBigUInt64BE(10n, 30);
const trailer = 38;
bytes[trailer + 6] = 9;
bytes[trailer + 7] = 9;
bytes.writeBigUInt64BE(2n, trailer + 8);
bytes.writeBigUInt64BE(1n, trailer + 16);
bytes.writeBigUInt64BE(20n, trailer + 24);
const original = Buffer.from(bytes);
expect(decode(bytes)).toEqual([7]);
expect(bytes.equals(original)).toBe(true);
});
+10 -5
View File
@@ -1,7 +1,7 @@
import { readBinaryPlistLayout } from "./BinaryPlistLayout.js";
import { parseBinary } from "plist";
import { markPlistDictionary } from "../../domain/apple/plistUid.js";
import {
binaryPlistObjectOffsets,
decodeBinaryPlistScalars,
scanBinaryPlistScalars,
type BinaryPlistScalarMetadata,
@@ -13,9 +13,14 @@ const scalarArrayInput = (
bytes: Buffer,
objects: readonly number[],
): Buffer => {
const trailer = bytes.length - 32;
const referenceWidth = bytes[trailer + 7]!;
const objectCount = Number(bytes.readBigUInt64BE(trailer + 8));
// Normalization may relocate objects and change both widths. These facts
// belong to the compatible decoder copy, not the original source plan.
const compatibleLayoutResult = readBinaryPlistLayout(bytes);
if (!compatibleLayoutResult.ok)
throw new RangeError("binary plist trailer is truncated");
const compatibleLayout = compatibleLayoutResult.value;
const referenceWidth = compatibleLayout.referenceWidth;
const objectCount = Number(compatibleLayout.objectCount);
const rootOffset = bytes.length;
const rootLength = 10 + objects.length * referenceWidth;
const tableOffset = rootOffset + rootLength;
@@ -33,7 +38,7 @@ const scalarArrayInput = (
}
}
let object = 0;
for (const offset of binaryPlistObjectOffsets(bytes)) {
for (const offset of compatibleLayout.objectOffsets()) {
output.writeBigUInt64BE(offset, tableOffset + object * 8);
object += 1;
}
+2 -1
View File
@@ -20,7 +20,8 @@ import {
} from "../ArtifactFileOperation.js";
import type { ArtifactResourceScope } from "../ArtifactResourceScope.js";
import { readArtifactFileChunks } from "../readArtifactFileChunks.js";
import { hasMachoMagic, readMachoImage } from "./MachoLoadCommandReader.js";
import { hasMachoMagic } from "../../domain/apple/machoContainer.js";
import { readMachoImage } from "./MachoLoadCommandReader.js";
/** One owner for dyld descriptor admission, physical identity and finalization. */
export class DylibFileAccess {
@@ -27,9 +27,9 @@ import {
MAX_LAZY_INFO_DEPENDENCIES,
MAX_LAZY_INFO_SYMBOL_REFERENCES,
MAX_LOAD_COMMAND_BYTES,
hasMachoMagic,
readMachoImage,
} from "./MachoLoadCommandReader.js";
import { hasMachoMagic } from "../../domain/apple/machoContainer.js";
import { traceDylibResolution } from "./DylibResolutionReader.js";
import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
+13 -34
View File
@@ -22,8 +22,11 @@ import {
} from "../../domain/apple/machoPlatform.js";
import {
classifyFatContainerIdentity,
hasMachoMagic,
readFatContainerHeader,
readFatHeaderPrefix,
readFatSliceDeclarations,
readMachoHeaderPrefixFacts,
validateFatSlice,
validateFatSliceIdentity,
type FatSliceDeclaration,
@@ -46,14 +49,6 @@ export const MAX_LAZY_INFO_DEPENDENCIES = 1024;
* offset, so stop before that map and those decodes grow.
*/
export const MAX_LAZY_INFO_SYMBOL_REFERENCES = 64 * 1024;
const MH_MAGIC = 0xfeedface;
const MH_MAGIC_64 = 0xfeedfacf;
const MH_CIGAM = 0xcefaedfe;
const MH_CIGAM_64 = 0xcffaedfe;
const FAT_MAGIC = 0xcafebabe;
const FAT_MAGIC_64 = 0xcafebabf;
const FAT_CIGAM = 0xbebafeca;
const FAT_CIGAM_64 = 0xbfbafeca;
const LC_SEGMENT = 0x1;
const LC_SEGMENT_64 = 0x19;
@@ -114,26 +109,6 @@ class MachoFormatIssue extends Error {
}
}
/** Report whether the first bytes carry a Mach-O or FAT magic number. */
export const hasMachoMagic = (head: Uint8Array): boolean => {
if (head.byteLength < 4) return false;
const magic = new DataView(
head.buffer,
head.byteOffset,
head.byteLength,
).getUint32(0, false);
return [
MH_MAGIC,
MH_MAGIC_64,
MH_CIGAM,
MH_CIGAM_64,
FAT_MAGIC,
FAT_MAGIC_64,
FAT_CIGAM,
FAT_CIGAM_64,
].includes(magic);
};
/** Decode every slice's dylib-loading load commands without reading code. */
export const readMachoImage = async (
readAt: ReadAt,
@@ -146,10 +121,13 @@ export const readMachoImage = async (
return hasMachoMagic(head)
? { status: "malformed", reason: "Mach-O header is truncated" }
: { status: "not-mach-o" };
const view = viewOf(head);
const magic = view.getUint32(0, false);
const fatPrefix = readFatHeaderPrefix(head);
const mayBeJvmClass =
fatPrefix.state === "parsed" &&
!fatPrefix.wide &&
!fatPrefix.littleEndian;
let header: ReturnType<typeof readFatContainerHeader>;
if (magic === FAT_MAGIC && size <= MAX_JVM_CLASS_IDENTITY_BYTES) {
if (mayBeJvmClass && size <= MAX_JVM_CLASS_IDENTITY_BYTES) {
const complete = await readAt(0, size);
header =
complete.byteLength === size
@@ -187,7 +165,7 @@ export const readMachoImage = async (
// A large CAFEBABE input whose bounded prefix resembles an invalid
// known-CPU FAT table remains unknown until its JVM identity can be
// checked; do not turn the resource limit into a malformed claim.
if (magic === FAT_MAGIC && size > MAX_JVM_CLASS_IDENTITY_BYTES)
if (mayBeJvmClass && size > MAX_JVM_CLASS_IDENTITY_BYTES)
throw new MachoFormatIssue(
"unsupported",
JVM_CLASS_IDENTITY_LIMIT_REASON,
@@ -203,9 +181,10 @@ export const readMachoImage = async (
...(hiddenSlice === null ? {} : { hiddenSlices: [hiddenSlice] }),
};
}
if (magic === MH_CIGAM || magic === MH_CIGAM_64)
const thin = readMachoHeaderPrefixFacts(head).format;
if (thin?.littleEndian === true)
return { status: "parsed", slices: [await readSlice(readAt, 0, size)] };
if (magic === MH_MAGIC || magic === MH_MAGIC_64)
if (thin !== null)
throw new MachoFormatIssue(
"unsupported",
"big-endian Mach-O images are not supported",
+26 -1
View File
@@ -488,7 +488,15 @@ describe("binary plist shared object-table entries", () => {
const table = Number(bytes.readBigUInt64BE(bytes.length - 8));
bytes[table + 2] = 7;
expect(bytes.subarray(0, 8).toString("ascii")).toBe("bplist00");
expect(scanBinaryPlistScalars(bytes).literals.incomplete).toBe(true);
const metadata = scanBinaryPlistScalars(bytes);
expect(metadata.literals.incomplete).toBe(true);
expect(metadata.literals.integers).toEqual(
new Map([
[9007199254740992, new Set(["9007199254740993"])],
[Number(1152921504606846979n), new Set(["1152921504606846979"])],
]),
);
expect(metadata.literals.unsafeIntegerLiterals).toBe(2);
});
});
@@ -586,3 +594,20 @@ describe("binary plist scalar clone budgeting", () => {
expect(bytes.readDoubleBE(14)).toBe(0.1005);
});
});
it.each([4n, 0xffffffffffffffffn])(
"scans source scalars independently of invalid top object %s",
(topObject) => {
const bytes = sharedScalarEntries(int64(9007199254740993n));
const expected = scanBinaryPlistScalars(bytes);
bytes.writeBigUInt64BE(topObject, bytes.length - 16);
expect(scanBinaryPlistScalars(bytes)).toEqual(expected);
expect(expected.literals.incomplete).toBe(false);
expect(expected.literals.unsafeIntegerLiterals).toBe(2);
expect(() =>
estimateBinaryPlistExpansion(bytes, 1024 * 1024, {
numberProjection: true,
}),
).toThrow("binary plist trailer or object table is invalid");
},
);
+33 -48
View File
@@ -1,3 +1,8 @@
import {
readBinaryPlistLayout,
readBinaryPlistUnsigned64,
type BinaryPlistLayout,
} from "./BinaryPlistLayout.js";
import {
createPlistNumberCollector,
type PlistNumberLiterals,
@@ -196,10 +201,7 @@ export const binaryPlistObjectEnd = (
const width = 1 << (extendedMarker & 0x0f);
if (width > 8 || cursor + 1 + width > objectEnd) return undefined;
cursor += 1;
let extendedLength = 0n;
for (let index = 0; index < width; index += 1)
extendedLength =
(extendedLength << 8n) | BigInt(bytes[cursor + index] ?? 0);
const extendedLength = readBinaryPlistUnsigned64(bytes, cursor, width);
if (extendedLength > BigInt(Number.MAX_SAFE_INTEGER)) return undefined;
length = Number(extendedLength);
cursor += width;
@@ -247,11 +249,17 @@ export const scanBinaryPlistScalars = (
});
if (bytes.length < 40 || !hasBinaryPlistSignature(bytes)) return result(true);
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
const trailer = bytes.length - 32;
const offsetWidth = bytes[trailer + 6] ?? 0;
const sourceLayoutResult = readBinaryPlistLayout(bytes);
if (!sourceLayoutResult.ok) return result(true);
const sourceLayout = sourceLayoutResult.value;
const {
trailerOffset: trailer,
offsetWidth,
referenceWidth,
objectCount: countRaw,
offsetTable: tableRaw,
} = sourceLayout;
if (offsetWidth < 1) return result(true);
const countRaw = view.getBigUint64(trailer + 8);
const tableRaw = view.getBigUint64(trailer + 24);
const safeMaximum = BigInt(Number.MAX_SAFE_INTEGER);
if (countRaw === 0n || countRaw > safeMaximum || tableRaw > safeMaximum)
return result(true);
@@ -260,23 +268,12 @@ export const scanBinaryPlistScalars = (
const capacity = Math.floor((trailer - offsetTable) / offsetWidth);
if (countRaw > BigInt(capacity)) return result(true);
const count = Number(countRaw);
const referenceWidth = bytes[trailer + 7] ?? 0;
if (referenceWidth < 1) return result(true);
const unsigned = (offset: number): number | undefined => {
if (offset < offsetTable || offset + offsetWidth > trailer)
return undefined;
let value = 0n;
for (let index = 0; index < offsetWidth; index += 1)
value = BigInt.asUintN(
64,
(value << 8n) | BigInt(bytes[offset + index] ?? 0),
);
return value > safeMaximum ? undefined : Number(value);
};
const objectOffsets: number[] = [];
let incomplete = false;
for (let index = 0; index < count; index += 1) {
const offset = unsigned(offsetTable + index * offsetWidth);
const rawOffset = sourceLayout.objectOffset(index);
const offset = rawOffset > safeMaximum ? undefined : Number(rawOffset);
if (offset === undefined || offset < 8 || offset >= offsetTable) {
incomplete = true;
continue;
@@ -405,8 +402,11 @@ export const decodeBinaryPlistScalars = (
replacement.writeDoubleBE(Number(value), 1);
replacements.set(offset, replacement);
}
const trailerOffset = bytes.length - 32;
const sourceReferenceSize = bytes[trailerOffset + 7] ?? 0;
const sourceLayoutResult = readBinaryPlistLayout(bytes);
if (!sourceLayoutResult.ok)
throw new RangeError("binary plist trailer is truncated");
const sourceLayout = sourceLayoutResult.value;
const sourceReferenceSize = sourceLayout.referenceWidth;
const normalizedReferenceSize =
scalarPlan.normalizedReferenceSize ?? sourceReferenceSize;
for (const container of scalarPlan.containersToNormalize ?? [])
@@ -415,8 +415,8 @@ export const decodeBinaryPlistScalars = (
binaryPlistContainerRecord(bytes, container, normalizedReferenceSize),
);
const needsTableNormalization =
(scalarPlan.normalizedOffsetSize ?? bytes[trailerOffset + 6] ?? 0) !==
(bytes[trailerOffset + 6] ?? 0) ||
(scalarPlan.normalizedOffsetSize ?? sourceLayout.offsetWidth) !==
sourceLayout.offsetWidth ||
normalizedReferenceSize !== sourceReferenceSize;
if (
reachableWideIntegers.length === 0 &&
@@ -430,8 +430,9 @@ export const decodeBinaryPlistScalars = (
? Buffer.from(bytes)
: cloneBinaryPlistObjects(
bytes,
sourceLayout,
replacements,
scalarPlan.normalizedOffsetSize ?? bytes[trailerOffset + 6] ?? 0,
scalarPlan.normalizedOffsetSize ?? sourceLayout.offsetWidth,
normalizedReferenceSize,
);
for (const { offset, value } of reachableWideIntegers) {
@@ -450,34 +451,18 @@ export const decodeBinaryPlistScalars = (
return compatible;
};
/** Iterate source offset declarations using the same uint64 interpretation as normalization. */
export function* binaryPlistObjectOffsets(bytes: Buffer): Generator<bigint> {
const trailer = bytes.length - 32;
const width = bytes[trailer + 6] ?? 0;
const count = Number(bytes.readBigUInt64BE(trailer + 8));
const table = Number(bytes.readBigUInt64BE(trailer + 24));
for (let object = 0; object < count; object += 1) {
let offset = 0n;
for (let byte = 0; byte < width; byte += 1)
offset = BigInt.asUintN(
64,
(offset << 8n) | BigInt(bytes[table + object * width + byte] ?? 0),
);
yield offset;
}
}
const cloneBinaryPlistObjects = (
bytes: Buffer,
sourceLayout: BinaryPlistLayout,
replacements: ReadonlyMap<number, Buffer>,
normalizedOffsetSize?: number,
normalizedReferenceSize?: number,
): Buffer => {
const trailerOffset = bytes.length - 32;
const offsetSize = bytes[trailerOffset + 6] ?? 0;
const objectCount = Number(bytes.readBigUInt64BE(trailerOffset + 8));
const oldTableOffset = Number(bytes.readBigUInt64BE(trailerOffset + 24));
const offsets = [...binaryPlistObjectOffsets(bytes)];
const trailerOffset = sourceLayout.trailerOffset;
const offsetSize = sourceLayout.offsetWidth;
const objectCount = Number(sourceLayout.objectCount);
const oldTableOffset = Number(sourceLayout.offsetTable);
const offsets = [...sourceLayout.objectOffsets()];
const cloneEntries = [...replacements.entries()];
const replacementLengths = cloneEntries.map(([, record]) => record.length);
const newOffsetSize =
+21 -18
View File
@@ -1,3 +1,7 @@
import {
readBinaryPlistLayout,
readBinaryPlistUnsigned64,
} from "./BinaryPlistLayout.js";
import { decodeXmlPlistText } from "../../domain/propertyListXmlText.js";
import { hasBinaryPlistSignature } from "../../domain/apple/appleArchiveFormat.js";
import {
@@ -265,24 +269,26 @@ export const estimateBinaryPlistExpansion = (
): PlistDecodePreflight => {
if (bytes.length < 40)
throw new TypeError("binary plist trailer is truncated");
const trailer = bytes.length - 32;
const offsetSize = bytes[trailer + 6] ?? 0;
const referenceSize = bytes[trailer + 7] ?? 0;
const sourceLayoutResult = readBinaryPlistLayout(bytes);
if (!sourceLayoutResult.ok)
throw new TypeError("binary plist trailer is truncated");
const sourceLayout = sourceLayoutResult.value;
const {
trailerOffset: trailer,
offsetWidth: offsetSize,
referenceWidth: referenceSize,
} = sourceLayout;
if (offsetSize < 1 || referenceSize < 1)
throw new TypeError("binary plist trailer integer sizes are invalid");
const safeDeclaration = (value: bigint): number =>
value <= BigInt(Number.MAX_SAFE_INTEGER) ? Number(value) : -1;
const readInteger = (offset: number, size: number): number => {
if (offset < 0 || size < 1 || offset + size > bytes.length) return -1;
let value = 0n;
for (let index = 0; index < size; index += 1)
value = BigInt.asUintN(
64,
(value << 8n) | BigInt(bytes[offset + index] ?? 0),
);
return value <= BigInt(Number.MAX_SAFE_INTEGER) ? Number(value) : -1;
return safeDeclaration(readBinaryPlistUnsigned64(bytes, offset, size));
};
const objectCount = readInteger(trailer + 8, 8);
const topObject = readInteger(trailer + 16, 8);
const offsetTable = readInteger(trailer + 24, 8);
const objectCount = safeDeclaration(sourceLayout.objectCount);
const topObject = safeDeclaration(sourceLayout.topObject);
const offsetTable = safeDeclaration(sourceLayout.offsetTable);
const tableEnd = bounds === "strict" ? trailer : bytes.length;
const objectsEnd = bounds === "strict" ? offsetTable : bytes.length;
if (
@@ -341,7 +347,7 @@ export const estimateBinaryPlistExpansion = (
const readDictionaryKey = (object: number): string => {
const prior = dictionaryKeys.get(object);
if (prior !== undefined) return prior;
const offset = readInteger(offsetTable + object * offsetSize, offsetSize);
const offset = safeDeclaration(sourceLayout.objectOffset(object));
if (offset < 8 || offset >= objectsEnd)
throw new TypeError("binary plist object offset is invalid");
const marker = bytes[offset] ?? 0;
@@ -403,10 +409,7 @@ export const estimateBinaryPlistExpansion = (
"expansion",
`binary plist reference expansion exceeds ${budget}`,
);
const offset = readInteger(
offsetTable + entry.object * offsetSize,
offsetSize,
);
const offset = safeDeclaration(sourceLayout.objectOffset(entry.object));
if (offset < 8 || offset >= objectsEnd)
throw new TypeError("binary plist object offset is invalid");
if (numberProjection) reachableObjectOffsets.add(offset);
+7 -1
View File
@@ -129,7 +129,9 @@ it("preserves explicit unknown slice evidence instead of falling back to retaine
const header = readFatContainerHeader(bytes, bytes.length);
if (header.state !== "parsed") throw new Error("Expected FAT header");
const offset = header.slices[0]!.offset;
const unknown = new Map([[offset, { magic: null, identity: null }]]);
const unknown = new Map([
[offset, readMachoHeaderPrefixFacts(Buffer.alloc(0))],
]);
expect(classifyFatContainerEvidence(header, bytes, new Map())).toEqual({
state: "mach-o",
});
@@ -258,6 +260,7 @@ describe("FAT Mach-O identity", () => {
bigEndian.writeUInt32BE(CPU.x86_64.subtype, 8);
expect(readMachoHeaderPrefixFacts(littleEndian.subarray(0, 12))).toEqual({
magic: 0xcffaedfe,
format: { wide: true, littleEndian: true, headerSize: 32 },
identity: {
cpuType: CPU.arm64.type,
cpuSubtype: CPU.arm64.subtype,
@@ -265,6 +268,7 @@ describe("FAT Mach-O identity", () => {
});
expect(readMachoHeaderPrefixFacts(bigEndian)).toEqual({
magic: 0xfeedfacf,
format: { wide: true, littleEndian: false, headerSize: 32 },
identity: {
cpuType: CPU.x86_64.type,
cpuSubtype: CPU.x86_64.subtype,
@@ -272,10 +276,12 @@ describe("FAT Mach-O identity", () => {
});
expect(readMachoHeaderPrefixFacts(littleEndian.subarray(0, 11))).toEqual({
magic: 0xcffaedfe,
format: { wide: true, littleEndian: true, headerSize: 32 },
identity: null,
});
expect(readMachoHeaderPrefixFacts(littleEndian.subarray(0, 3))).toEqual({
magic: null,
format: null,
identity: null,
});
});
@@ -32,6 +32,7 @@ import { parseMzHeaderEvidence } from "../../domain/dosMz.js";
import {
classifyFatContainerEvidence,
readFatContainerHeader,
readMachoHeaderPrefixFacts,
} from "../../domain/apple/machoContainer.js";
/** Mutable internal occurrence used until root-bound IDs are known. */
@@ -581,9 +582,7 @@ export const classifyArtifactBytes = (
const fatFormat = classifyFatArtifact(evidence, fileSize);
if (fatFormat !== null) return fatFormat;
if (prefix.length >= 4) {
const magic = prefix.readUInt32BE(0);
if ([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(magic))
return "mach-o";
if (readMachoHeaderPrefixFacts(prefix).format !== null) return "mach-o";
if (prefix.subarray(0, 4).equals(Buffer.from([0x7f, 0x45, 0x4c, 0x46])))
return "elf";
}
+16 -11
View File
@@ -17,7 +17,11 @@ import type { FileHandle } from "node:fs/promises";
import { abortIfNeeded } from "../ArtifactHash.js";
import { ArtifactReaderFailure } from "../ArtifactReader.js";
import { readFatContainerHeader } from "../../domain/apple/machoContainer.js";
import {
readFatContainerHeader,
readFatHeaderPrefix,
readMachoHeaderPrefixFacts,
} from "../../domain/apple/machoContainer.js";
import { readElfFixedHeader } from "../../domain/elfFixedHeader.js";
import { readMachoLoadCommandTableHeader } from "../../domain/apple/machoLoadCommands.js";
import {
@@ -52,9 +56,6 @@ const FUSE_LENGTH = "NODE_SEA_FUSE_".length + 32 + 2;
// PE header structures must fit this prefix for the bounded name probe.
const PE_HEADER_PROBE_BYTES = 64 * 1024;
const THIN_MAGICS = new Set([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe]);
const FAT_MAGICS = new Set([0xcafebabe, 0xcafebabf, 0xbebafeca, 0xbfbafeca]);
/** One injected blob inside a native carrier. Offsets are carrier file offsets. */
export interface NodeSeaLocation {
readonly carrier: "mach-o" | "elf" | "pe";
@@ -96,11 +97,13 @@ export const locateNodeSea = async (
const carrier: Carrier = { handle, fileSize, signal };
const prefix = await readExact(carrier, 0, Math.min(fileSize, 64));
if (prefix === undefined || prefix.length < 4) return undefined;
const magic = prefix.readUInt32BE(0);
if (THIN_MAGICS.has(magic) || FAT_MAGICS.has(magic)) {
const ranges = THIN_MAGICS.has(magic)
? await locateThinMachO(carrier, 0, fileSize)
: await locateFatMachO(carrier, prefix);
const thin = readMachoHeaderPrefixFacts(prefix).format;
const fat = readFatHeaderPrefix(prefix);
if (thin !== null || fat.state === "incomplete" || fat.state === "parsed") {
const ranges =
thin !== null
? await locateThinMachO(carrier, 0, fileSize)
: await locateFatMachO(carrier, prefix);
return collapseRanges(carrier, "mach-o", ranges);
}
if (prefix.subarray(0, 4).equals(Buffer.from([0x7f, 0x45, 0x4c, 0x46])))
@@ -216,7 +219,7 @@ const locateFatMachO = async (
header = readFatContainerHeader(bytes, carrier.fileSize);
}
if (header.state !== "parsed") {
if (bytes.length >= 4 && THIN_MAGICS.has(bytes.readUInt32BE(0)))
if (readMachoHeaderPrefixFacts(bytes).format !== null)
return locateThinMachO(carrier, 0, carrier.fileSize);
return [];
}
@@ -588,8 +591,9 @@ const locatePeResources = async (carrier: Carrier): Promise<BlobRange[]> => {
if (bytes === undefined) throw formatFailure(NODE_SEA_TRUNCATED);
// A native-looking file need not be an admitted PE image. Once the layout
// is admitted, resource errors remain failures rather than inferred absence.
let layout: ReturnType<typeof readPeResourceLayout>;
try {
readPeResourceLayout(bytes);
layout = readPeResourceLayout(bytes);
} catch (cause) {
if (cause instanceof ArtifactReaderFailure && cause.reason === "format")
return [];
@@ -608,6 +612,7 @@ const locatePeResources = async (carrier: Carrier): Promise<BlobRange[]> => {
max_entries: MAX_INSPECT_PE_DIRECTORY_ENTRIES,
},
carrier.signal,
layout,
);
return resources.resources.flatMap((resource) => {
if (
@@ -4,9 +4,8 @@ import type {
} from "../../domain/artifactFormatMetadata.js";
import type { CArchiveEntry, PyInstallerCookie } from "./CArchiveLayout.js";
import type { PyzArchive, PyzEntry } from "./PyzArchive.js";
import { pythonVersionForMagic } from "./PythonBytecode.js";
const pythonVersion = (value: number) =>
const cookiePythonVersion = (value: number) =>
value >= 100
? { major: Math.floor(value / 100), minor: value % 100 }
: value >= 20
@@ -18,7 +17,7 @@ export const cookieMetadata = (
entries: readonly CArchiveEntry[],
encryptionMarker: boolean,
): PyInstallerMetadata => {
const version = pythonVersion(cookie.pythonVersion);
const version = cookiePythonVersion(cookie.pythonVersion);
return {
kind: "pyinstaller",
layout_version: cookie.layout,
@@ -45,13 +44,13 @@ export const scriptPythonMagic = (
cookie: PyInstallerCookie,
archives: Iterable<PyzArchive>,
): Buffer | undefined => {
const version = pythonVersion(cookie.pythonVersion);
const version = cookiePythonVersion(cookie.pythonVersion);
if (version === undefined) return undefined;
let selected: Buffer | undefined;
for (const archive of archives) {
const magic = archive.pythonMagic;
if (
pythonVersionForMagic(magic) !==
archive.pythonVersion !==
`${String(version.major)}.${String(version.minor)}` ||
(selected !== undefined && !selected.equals(magic))
)
@@ -123,10 +122,7 @@ export const cArchiveEntryMetadata = (
directory_record_size: entry.tocRecordLength,
pyz_sha256: archive?.sha256 ?? null,
pyz_python_magic: archive?.pythonMagic.toString("hex") ?? null,
pyz_python_version:
archive === undefined
? null
: (pythonVersionForMagic(archive.pythonMagic) ?? null),
pyz_python_version: archive?.pythonVersion ?? null,
pyz_entry_count: archive?.kind === "entries" ? archive.entries.length : null,
encryption_source:
archive !== undefined && encryptionMarker ? "carchive-marker" : null,
+69 -22
View File
@@ -1,3 +1,4 @@
import { createHash } from "node:crypto";
import { open, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
@@ -31,6 +32,7 @@ it("lists modules from a supported PYZ and leaves an unknown magic opaque", asyn
if (!opaque.ok || opaque.value.kind !== "opaque") return;
expect(Buffer.compare(opaque.value.pythonMagic, Buffer.alloc(4))).toBe(0);
expect(opaque.value.limitation).toContain("left opaque");
expect(opaque.value.pythonVersion).toBeUndefined();
});
it("leaves a PYZ opaque when the table uses an unsupported marshal type", async () => {
@@ -44,6 +46,10 @@ it("leaves a PYZ opaque when the table uses an unsupported marshal type", async
if (!parsed.ok || parsed.value.kind !== "opaque") return;
expect(parsed.value.limitation).toContain("left opaque");
expect(parsed.value.pythonMagic).toEqual(PYTHON_39_MAGIC);
expect(parsed.value.pythonVersion).toBe("3.9");
expect(parsed.value.sha256).toBe(
createHash("sha256").update(image).digest("hex"),
);
});
it("reports an entry that extends outside the PYZ as an integrity error", async () => {
@@ -58,26 +64,29 @@ it("reports an entry that extends outside the PYZ as an integrity error", async
expect(parsed.error.reason).toBe("integrity");
});
it("rejects an oversized sparse PYZ table before allocating its bytes", async () => {
const root = await createTestTempDirectory("rea-pyz-table-budget-");
const path = join(root, "image.pyz");
const header = Buffer.alloc(12);
header.write("PYZ\0");
PYTHON_39_MAGIC.copy(header, 4);
header.writeUInt32BE(12, 8);
await writeFile(path, header);
const handle = await open(path, "r+");
try {
const size = 12 + MAX_PYZ_TABLE_BYTES + 1;
await handle.truncate(size);
expect(await readPyzArchive(handle, 0, size)).toMatchObject({
ok: false,
error: { reason: "limit" },
});
} finally {
await handle.close();
}
});
it.each(["PYZ\0", "BAD\0"])(
"rejects an oversized sparse table before deferred signature rejection (%s)",
async (signature) => {
const root = await createTestTempDirectory("rea-pyz-table-budget-");
const path = join(root, "image.pyz");
const header = Buffer.alloc(12);
header.write(signature);
PYTHON_39_MAGIC.copy(header, 4);
header.writeUInt32BE(12, 8);
await writeFile(path, header);
const handle = await open(path, "r+");
try {
const size = 12 + MAX_PYZ_TABLE_BYTES + 1;
await handle.truncate(size);
expect(await readPyzArchive(handle, 0, size)).toMatchObject({
ok: false,
error: { reason: "limit" },
});
} finally {
await handle.close();
}
},
);
it("rejects a PYZ table whose back-references repeat one long row", async () => {
// One namespace row with a 16 KiB name, flagged for reference, then 999
@@ -124,14 +133,52 @@ const int32 = (value: number): Buffer => {
return bytes;
};
const readImage = async (bytes: Buffer) => {
const readImage = async (bytes: Buffer, declaredSize = bytes.length) => {
const root = await createTestTempDirectory("rea-pyz-file-");
const path = join(root, "image.pyz");
await writeFile(path, bytes);
const handle = await open(path, "r");
try {
return await readPyzArchive(handle, 0, bytes.length);
return await readPyzArchive(handle, 0, declaredSize);
} finally {
await handle.close();
}
};
it.each([0, 0xffffffff])(
"retains an unknown Python magic and full-image digest with invalid TOC offset %s",
async (offset) => {
const image = pyzArchive({ magic: Buffer.from([1, 2, 3, 4]), modules: [] });
image.writeUInt32BE(offset, 8);
expect(await readImage(image)).toMatchObject({
ok: true,
value: {
kind: "opaque",
pythonMagic: Buffer.from([1, 2, 3, 4]),
pythonVersion: undefined,
sha256: createHash("sha256").update(image).digest("hex"),
limitation: expect.stringContaining("not in the supported set"),
},
});
},
);
it("reports a truncated table before deferred signature rejection", async () => {
const header = Buffer.alloc(12);
header.write("BAD\0");
PYTHON_39_MAGIC.copy(header, 4);
header.writeUInt32BE(12, 8);
expect(await readImage(header, 16)).toMatchObject({
ok: false,
error: { reason: "format", message: "PYZ table is truncated" },
});
});
it("reports the signature after completing a readable table and image hash", async () => {
const image = pyzArchive({ magic: PYTHON_39_MAGIC, modules: [] });
image.write("BAD\0");
expect(await readImage(image)).toMatchObject({
ok: false,
error: { reason: "format", message: "PYZ archive magic does not match" },
});
});
+38 -18
View File
@@ -38,6 +38,7 @@ export type PyzArchive =
| {
readonly kind: "opaque";
readonly pythonMagic: Buffer;
readonly pythonVersion: string | undefined;
readonly sha256: string;
readonly limitation: string;
};
@@ -45,6 +46,31 @@ export type PyzArchive =
/** Bound the transient marshal table allocation independently of image expansion. */
export const MAX_PYZ_TABLE_BYTES = 16 * 1024 * 1024;
interface PyzHeaderFacts {
readonly signatureMatches: boolean;
readonly pythonMagic: Buffer;
readonly pythonVersion: string | undefined;
readonly tocOffset: number;
/** Null means the declaration lies outside this image, not an empty table. */
readonly tableBytes: number | null;
}
/** Called only after the exact fixed-header read has succeeded. */
const readPyzHeaderFacts = (header: Buffer, size: number): PyzHeaderFacts => {
const pythonMagic = Buffer.from(header.subarray(4, 8));
const tocOffset = header.readUInt32BE(8);
return {
signatureMatches: header.subarray(0, 4).equals(PYZ_MAGIC),
pythonMagic,
pythonVersion: pythonVersionForMagic(pythonMagic),
tocOffset,
tableBytes:
tocOffset >= PYZ_HEADER_BYTES && tocOffset <= size
? size - tocOffset
: null,
};
};
/** Parse only the header and table in memory; hash the complete file range in chunks. */
export const readPyzArchive = async (
handle: FileHandle,
@@ -63,12 +89,9 @@ export const readPyzArchive = async (
signal,
});
if (!headerRead.ok) return headerRead;
const header = headerRead.value;
const tocOffset =
header.length === PYZ_HEADER_BYTES ? header.readUInt32BE(8) : size;
const validOffset = tocOffset >= PYZ_HEADER_BYTES && tocOffset <= size;
const knownMagic = pythonVersionForMagic(header.subarray(4, 8)) !== undefined;
const tableSize = validOffset && knownMagic ? size - tocOffset : 0;
const header = readPyzHeaderFacts(headerRead.value, size);
const tableSize =
header.pythonVersion === undefined ? 0 : (header.tableBytes ?? 0);
if (tableSize > MAX_PYZ_TABLE_BYTES)
return err(
pyInstallerFailure(
@@ -77,7 +100,7 @@ export const readPyzArchive = async (
),
);
const read = await readPyInstallerBytes(handle, {
position: start + tocOffset,
position: start + header.tocOffset,
length: tableSize,
label: "PYZ table",
signal,
@@ -96,33 +119,28 @@ export const readPyzArchive = async (
return err(
pyInstallerFailure("format", "PYZ image extends outside the file"),
);
return parsePyzParts(header, table, size, digest.sha256);
return parsePyzParts(header, table, digest.sha256);
};
const parsePyzParts = (
header: Buffer,
header: PyzHeaderFacts,
table: Buffer,
size: number,
sha256: string,
): Result<PyzArchive, PyInstallerFailure> => {
if (
header.length < PYZ_HEADER_BYTES ||
!header.subarray(0, 4).equals(PYZ_MAGIC)
)
if (!header.signatureMatches)
return err(
pyInstallerFailure("format", "PYZ archive magic does not match"),
);
const pythonMagic = Buffer.from(header.subarray(4, 8));
const pythonVersion = pythonVersionForMagic(pythonMagic);
const { pythonMagic, pythonVersion, tocOffset } = header;
if (pythonVersion === undefined)
return ok({
kind: "opaque",
pythonMagic,
pythonVersion,
sha256,
limitation: `PYZ Python magic ${pythonMagic.toString("hex")} is not in the supported set; the member was left opaque.`,
});
const tocOffset = header.readUInt32BE(8);
if (tocOffset < PYZ_HEADER_BYTES || tocOffset > size)
if (header.tableBytes === null)
return err(
pyInstallerFailure(
"format",
@@ -135,6 +153,7 @@ const parsePyzParts = (
return ok({
kind: "opaque",
pythonMagic,
pythonVersion,
sha256,
limitation: `PYZ table of contents could not be parsed (${marshalled.error.message}); the member was left opaque.`,
});
@@ -145,6 +164,7 @@ const parsePyzParts = (
? ok({
kind: "opaque",
pythonMagic,
pythonVersion,
sha256,
limitation: `${entries.error.message}; the member was left opaque.`,
})
+16 -33
View File
@@ -5,13 +5,16 @@ import type {
} from "../domain/javascript/electronObservation.js";
import type { CdpConnection } from "./CdpConnection.js";
import { CdpCaptureCompleteness } from "./CdpCaptureCompleteness.js";
import { recordValue, cdpStringValue } from "./CdpCaptureValues.js";
import { recordValue } from "./CdpCaptureValues.js";
import type { CdpEndpointTarget } from "./CdpEndpoint.js";
import { optionalCdpCommand } from "./CdpOptionalCommand.js";
import { authorizedElectronFile } from "./ElectronFileScope.js";
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
import { AnalysisCancelledError } from "../domain/analysisErrorCore.js";
import { workerTargetOwnership } from "./CdpWorkerTargetOwnership.js";
import {
parseCdpWorkerTarget,
workerTargetOwnership,
} from "./CdpWorkerTarget.js";
/** Inventory local worker targets without attaching to them. */
export const captureElectronWorkers = async (input: {
@@ -47,33 +50,27 @@ export const captureElectronWorkers = async (input: {
}
for (const entry of targetInfos) {
throwIfElectronWorkerCaptureCancelled(input.signal);
const target = recordValue(entry);
if (target === undefined) {
const parsed = parseCdpWorkerTarget(entry);
if (!parsed.ok) {
input.completeness.exclude("workers", "invalid_protocol_value");
continue;
}
const type = cdpStringValue(target.type) ?? "";
if (!type.includes("worker")) continue;
const target = parsed.value;
if (target.kind === "non_worker") continue;
const ownership = workerTargetOwnership(
target,
input.target.id,
input.frameIds,
);
if (!ownership.ok) {
input.completeness.exclude("workers", "invalid_protocol_value");
continue;
}
if (ownership.value.scope !== "selected") {
if (ownership !== "selected") {
input.completeness.exclude(
"workers",
ownership.value.scope === "unknown"
? "unattributed_target"
: "out_of_target_scope",
ownership === "unknown" ? "unattributed_target" : "out_of_target_scope",
);
continue;
}
const path = await authorizedElectronFile(
cdpStringValue(target.url) ?? "",
target.url,
input.signal,
"inspect_electron_page",
input.asarHeaderAdmission,
@@ -82,22 +79,13 @@ export const captureElectronWorkers = async (input: {
input.completeness.exclude("workers", "out_of_target_scope");
continue;
}
const targetId = boundedTargetField(target.targetId);
if (targetId === null) {
input.completeness.exclude("workers", "invalid_protocol_value");
continue;
}
if (typeof target.attached !== "boolean") {
input.completeness.exclude("workers", "invalid_protocol_value");
continue;
}
workers.push({
target_id: targetId,
type,
target_id: target.targetId,
type: target.type,
file_path: path,
attached: target.attached,
opener_target_id: ownership.value.openerTargetId,
parent_frame_id: ownership.value.parentFrameId,
opener_target_id: target.openerTargetId,
parent_frame_id: target.parentFrameId,
});
}
return workers.sort((left, right) =>
@@ -111,8 +99,3 @@ const throwIfElectronWorkerCaptureCancelled = (signal?: AbortSignal): void => {
cause: signal.reason,
});
};
const boundedTargetField = (value: unknown): string | null => {
const field = cdpStringValue(value);
return field === undefined || field === "" ? null : field;
};
+73 -2
View File
@@ -9,7 +9,7 @@ import type { CaptureContext } from "./CdpPageCapture.js";
const origin = "http://127.0.0.1:43127";
const capture = async (inventory: unknown) => {
const capture = async (inventory: unknown, frameIds = new Set<string>()) => {
const input = inspectWebPageInputSchema.parse({
cdp_endpoint: origin,
allowed_origins: [origin],
@@ -38,7 +38,7 @@ const capture = async (inventory: unknown) => {
limitations: [],
events,
},
new Set(),
frameIds,
);
return { workers, completeness: events.completeness.snapshot() };
};
@@ -199,3 +199,74 @@ it.each([{ openerId: "" }, { parentFrameId: null }])(
});
},
);
it.each([
{ type: undefined },
{ type: 7 },
{ type: "" },
{ targetId: "" },
{ url: undefined },
{ attached: null },
{ openerId: null },
{ parentFrameId: "" },
])(
"reports malformed worker facts before ownership and URL policy: %j",
async (override) => {
const result = await capture({
targetInfos: [
{
targetId: "worker",
type: "worker",
url: "https://outside.example/worker.js",
attached: false,
openerId: "foreign-page",
...override,
},
],
});
expect(result.workers).toEqual([]);
expect(result.completeness.excluded).toEqual([
{ section: "workers", reason: "invalid_protocol_value", count: 1 },
]);
},
);
it("uses parsed frame relationships without treating valid non-workers as malformed", async () => {
const result = await capture(
{
targetInfos: [
{ type: "page" },
{
targetId: "worker",
type: "shared_worker",
url: `${origin}/worker.js`,
attached: false,
openerId: "foreign-page",
parentFrameId: "selected-frame",
},
{
targetId: "foreign",
type: "worker",
url: `${origin}/foreign.js`,
attached: true,
parentFrameId: "foreign-frame",
},
],
},
new Set(["selected-frame"]),
);
expect(result.workers).toEqual([
{
target_id: "worker",
type: "shared_worker",
url: `${origin}/worker.js`,
origin,
attached: false,
opener_target_id: "foreign-page",
parent_frame_id: "selected-frame",
},
]);
expect(result.completeness.excluded).toEqual([
{ section: "workers", reason: "out_of_target_scope", count: 1 },
]);
});
+15 -34
View File
@@ -4,11 +4,13 @@ import {
admitRecords,
allowedSanitizedUrl,
recordValue,
cdpStringValue,
} from "./CdpCaptureValues.js";
import type { CaptureContext } from "./CdpPageCapture.js";
import { optionalCdpCommand } from "./CdpOptionalCommand.js";
import { workerTargetOwnership } from "./CdpWorkerTargetOwnership.js";
import {
parseCdpWorkerTarget,
workerTargetOwnership,
} from "./CdpWorkerTarget.js";
import { exclusionReasonForUrl } from "./CdpCaptureEventHelpers.js";
interface CaptureWorkersInput {
@@ -42,59 +44,38 @@ export const captureWorkers = async (
}
let malformedEntries = 0;
for (const entry of targets ?? []) {
const target = recordValue(entry);
if (target === undefined) {
malformedEntries += 1;
continue;
}
const type = cdpStringValue(target.type);
if (type === undefined || type.length === 0) {
malformedEntries += 1;
continue;
}
if (!type.includes("worker")) continue;
const targetId = cdpStringValue(target.targetId);
if (
targetId === undefined ||
targetId.length === 0 ||
typeof target.attached !== "boolean" ||
typeof target.url !== "string"
) {
const parsed = parseCdpWorkerTarget(entry);
if (!parsed.ok) {
malformedEntries += 1;
continue;
}
const target = parsed.value;
if (target.kind === "non_worker") continue;
const ownership = workerTargetOwnership(
target,
context.target.id,
frameIds,
);
if (!ownership.ok) {
malformedEntries += 1;
continue;
}
if (ownership.value.scope !== "selected") {
if (ownership !== "selected") {
completeness.exclude(
"workers",
ownership.value.scope === "unknown"
? "unattributed_target"
: "out_of_target_scope",
ownership === "unknown" ? "unattributed_target" : "out_of_target_scope",
);
continue;
}
const url = allowedSanitizedUrl(target.url, allowedOrigins);
if (url === undefined) {
const rawUrl = cdpStringValue(target.url);
completeness.exclude("workers", exclusionReasonForUrl(rawUrl));
completeness.exclude("workers", exclusionReasonForUrl(target.url));
continue;
}
items.push({
target_id: targetId,
type,
target_id: target.targetId,
type: target.type,
url: url.url,
origin: url.origin,
attached: target.attached,
opener_target_id: ownership.value.openerTargetId,
parent_frame_id: ownership.value.parentFrameId,
opener_target_id: target.openerTargetId,
parent_frame_id: target.parentFrameId,
});
}
if (malformedEntries > 0)
+70
View File
@@ -0,0 +1,70 @@
import { err, ok, type Result } from "../domain/result.js";
import { cdpStringValue, recordValue } from "./CdpCaptureValues.js";
export interface CdpWorkerRelationships {
readonly openerTargetId: string | null;
readonly parentFrameId: string | null;
}
export interface CdpWorkerTarget extends CdpWorkerRelationships {
readonly kind: "worker";
readonly targetId: string;
readonly type: string;
readonly url: string;
readonly attached: boolean;
}
/** Admit protocol facts before either consumer applies target or URL policy. */
export const parseCdpWorkerTarget = (
input: unknown,
): Result<
CdpWorkerTarget | { readonly kind: "non_worker" },
{ readonly _tag: "InvalidWorkerTarget" }
> => {
const target = recordValue(input);
const type = cdpStringValue(target?.type);
if (target === undefined || type === undefined || type === "")
return err({ _tag: "InvalidWorkerTarget" });
if (!type.includes("worker")) return ok({ kind: "non_worker" });
const targetId = cdpStringValue(target.targetId);
const url = cdpStringValue(target.url);
const openerTargetId = cdpStringValue(target.openerId);
const parentFrameId = cdpStringValue(target.parentFrameId);
if (
targetId === undefined ||
targetId === "" ||
url === undefined ||
typeof target.attached !== "boolean" ||
(target.openerId !== undefined &&
(openerTargetId === undefined || openerTargetId === "")) ||
(target.parentFrameId !== undefined &&
(parentFrameId === undefined || parentFrameId === ""))
)
return err({ _tag: "InvalidWorkerTarget" });
return ok({
kind: "worker",
targetId,
type,
url,
attached: target.attached,
openerTargetId: openerTargetId ?? null,
parentFrameId: parentFrameId ?? null,
});
};
/** Optional CDP relationships establish ownership only when supplied. */
export const workerTargetOwnership = (
relationships: CdpWorkerRelationships,
selectedTargetId: string,
frameIds: ReadonlySet<string>,
): "selected" | "foreign" | "unknown" => {
const { openerTargetId, parentFrameId } = relationships;
if (
openerTargetId === selectedTargetId ||
(parentFrameId !== null && frameIds.has(parentFrameId))
)
return "selected";
return openerTargetId === null && parentFrameId === null
? "unknown"
: "foreign";
};
-38
View File
@@ -1,38 +0,0 @@
import { err, ok, type Result } from "../domain/result.js";
import { cdpStringValue } from "./CdpCaptureValues.js";
/** Optional CDP relationships establish ownership only when supplied. */
export const workerTargetOwnership = (
target: Readonly<Record<string, unknown>>,
selectedTargetId: string,
frameIds: ReadonlySet<string>,
): Result<
{
readonly scope: "selected" | "foreign" | "unknown";
readonly openerTargetId: string | null;
readonly parentFrameId: string | null;
},
{ readonly _tag: "InvalidWorkerRelationship" }
> => {
const openerTargetId = cdpStringValue(target.openerId);
const parentFrameId = cdpStringValue(target.parentFrameId);
if (
(target.openerId !== undefined &&
(openerTargetId === undefined || openerTargetId === "")) ||
(target.parentFrameId !== undefined &&
(parentFrameId === undefined || parentFrameId === ""))
)
return err({ _tag: "InvalidWorkerRelationship" });
const selected =
openerTargetId === selectedTargetId ||
(parentFrameId !== undefined && frameIds.has(parentFrameId));
return ok({
scope: selected
? "selected"
: openerTargetId === undefined && parentFrameId === undefined
? "unknown"
: "foreign",
openerTargetId: openerTargetId ?? null,
parentFrameId: parentFrameId ?? null,
});
};
+110 -1
View File
@@ -78,6 +78,14 @@ type TestMode =
readonly oomMarker?: string;
readonly stderrMarker?: string;
}
| {
readonly kind: "result";
readonly encoding:
| "invalid-utf8"
| "replacement"
| "bom"
| "malformed-json";
}
| { readonly kind: "delayed-ready" }
| { readonly kind: "exit-before-ready" }
| {
@@ -166,6 +174,33 @@ const protocolWorkerProgram = (
});
`;
const resultWorkerProgram = (encoding: string): string => `
import { readFile, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";
import { createInterface } from "node:readline";
import { getHeapStatistics } from "node:v8";
const { analyzeCapturedWebScript } = await import(${JSON.stringify(new URL("../domain/webBundleAnalyzer.ts", import.meta.url).href)});
const send = (value) => process.stdout.write(JSON.stringify(value) + "\\n");
send({ kind: "ready", heap_limit_bytes: getHeapStatistics().heap_size_limit });
createInterface({ input: process.stdin, crlfDelay: Infinity }).once("line", async (line) => {
const request = JSON.parse(line);
const script = JSON.parse(await readFile(request.id + ".source", "utf8"));
const facts = analyzeCapturedWebScript(script);
let bytes = Buffer.from(JSON.stringify(facts));
const encoding = ${JSON.stringify(encoding)};
if (encoding === "invalid-utf8") {
const index = bytes.indexOf(Buffer.from("result-marker"));
if (index < 0) throw new Error("Expected endpoint marker in schema-valid facts");
bytes[index] = 0xff;
} else if (encoding === "bom") bytes = Buffer.concat([Buffer.from([0xef, 0xbb, 0xbf]), bytes]);
else if (encoding === "malformed-json") bytes = bytes.subarray(0, bytes.length - 1);
await writeFile(request.id + ".result", bytes);
send({ kind: "complete", id: request.id, result: {
bytes: bytes.length, sha256: createHash("sha256").update(bytes).digest("hex"),
} });
});
`;
const delayedReadyProgram = `
import { createInterface } from "node:readline";
import { getHeapStatistics } from "node:v8";
@@ -203,7 +238,9 @@ const launcherFor = (
? delayedReadyProgram
: mode.kind === "exit-before-ready"
? exitBeforeReadyProgram
: protocolWorkerProgram(mode.failure);
: mode.kind === "result"
? resultWorkerProgram(mode.encoding)
: protocolWorkerProgram(mode.failure);
const spawned = await spawnOwnedProviderProcess({
...request,
arguments: [
@@ -926,3 +963,75 @@ it.each([
}
},
);
it.each(["invalid-utf8", "bom", "malformed-json"] as const)(
"rejects %s worker result bytes with a matching descriptor",
async (encoding) => {
let root: string | undefined;
const worker = new WebBundleAnalysisWorker(
WEB_BUNDLE_WORKER_LIMITS.heapMb,
launcherFor(
{ kind: "result", encoding },
{
onSpawn: (spawned) => {
root = spawned.root;
},
},
),
);
try {
const result = await worker.analyze(
makeScript("fetch('/result-marker');"),
10_000,
);
const error = failureOf(result);
expect(error).toBeInstanceOf(AnalysisOutputError);
expect(error.cause).toBeInstanceOf(
encoding === "invalid-utf8" ? TypeError : SyntaxError,
);
expect(error.cleanupIncomplete).toBe(false);
} finally {
await worker.close();
}
expect(root).toBeDefined();
await expect(access(root!)).rejects.toMatchObject({
code: "ENOENT",
path: root,
});
},
);
it("preserves literal replacement characters in valid UTF-8 worker facts", async () => {
let root: string | undefined;
const worker = new WebBundleAnalysisWorker(
WEB_BUNDLE_WORKER_LIMITS.heapMb,
launcherFor(
{ kind: "result", encoding: "replacement" },
{
onSpawn: (spawned) => {
root = spawned.root;
},
},
),
);
try {
const result = await worker.analyze(
makeScript("fetch('/literal-�');"),
10_000,
);
expect(result.ok).toBe(true);
if (!result.ok) throw new Error(result.error.error.message);
if (result.value.status !== "analyzed")
throw new Error("Expected analyzed worker facts");
expect(result.value.observations.endpoints).toContainEqual(
expect.objectContaining({ value: "/literal-�" }),
);
} finally {
await worker.close();
}
expect(root).toBeDefined();
await expect(access(root!)).rejects.toMatchObject({
code: "ENOENT",
path: root,
});
});
+4 -3
View File
@@ -31,6 +31,7 @@ import {
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import { jsonParts, bufferedJsonParts } from "../domain/jsonSerialization.js";
import { safeParseUtf8Json } from "../domain/safeJson.js";
import { javaScriptRuntime } from "../domain/runtimeVersion.js";
import { err, ok, type Result } from "../domain/result.js";
import type { IncludedScript } from "../domain/webBundleAnalyzerState.js";
@@ -787,9 +788,9 @@ export class WebBundleAnalysisWorker {
OPERATION,
"Web analyzer result differs from its descriptor",
);
const facts = webBundleScriptAnalysisSchema.parse(
JSON.parse(stored.bytes.toString("utf8")),
);
const parsed = safeParseUtf8Json(stored.bytes);
if (!parsed.ok) throw parsed.cause;
const facts = webBundleScriptAnalysisSchema.parse(parsed.value);
if (facts.status !== "analyzed" || facts.script_key !== script.script_key)
throw new AnalysisOutputError(
OPERATION,
+2 -1
View File
@@ -1,6 +1,7 @@
import { z } from "zod";
import { address, document } from "./toolContractHelpers.js";
import {
MAX_BYTE_SEARCH_MATCHES,
bytePatternFields,
validateBytePattern,
} from "../domain/native/bytePattern.js";
@@ -17,7 +18,7 @@ export const HOPPER_MEMORY_TOOL_DEFINITIONS = [
document,
start: address.optional(),
end: address.optional(),
limit: z.int().min(1).max(1000).default(100),
limit: z.int().min(1).max(MAX_BYTE_SEARCH_MATCHES).default(100),
cursor: z.string().min(1).optional(),
})
.superRefine(validateBytePattern),
+1
View File
@@ -448,6 +448,7 @@ const providerErrorDetails = (
candidate_ids: [...error.candidateIds],
rejections: error.rejections.map((rejection) => ({
provider_id: rejection.providerId,
origin: rejection.origin,
code: rejection.code,
reason: rejection.reason,
diagnostics: rejection.diagnostics,
@@ -1,4 +1,3 @@
import { z } from "zod";
import type { AnalysisError } from "../analysisErrorBase.js";
import { jsonValueType, parseJsonObject } from "../jsonShape.js";
import { ok, type Result } from "../result.js";
@@ -11,7 +10,12 @@ import {
type WebNetworkCaptureRecord,
} from "../webNetworkCapture.js";
import { completeWithinViewCoverage } from "./analysisViewCoverage.js";
import { recordedResponseBodyIsEmpty } from "./recordedResponseBody.js";
import {
recordedResponseBodyIsEmpty,
readRecordedResponseDeclaration,
} from "./recordedResponseBody.js";
import { readRecordedRequestBody } from "./recordedExchangeBody.js";
import { isJson, isJsonLike, mediaTypeEssence } from "./recordedMediaType.js";
import type { UnsignedAnalysisView } from "./analysisView.js";
import type { ScenarioResponseBodyRetention } from "./scenarioCaptureRecords.js";
import { WebNetworkProjectionAccumulator } from "./WebNetworkProjectionAccumulator.js";
@@ -41,7 +45,6 @@ import {
GROUP_BYTES,
RECORD_SCRATCH_BYTES,
SEGMENT_BYTES,
mediaTypeEssence,
readQueryParameterNames,
readRecordedRequestLine,
routePartCount,
@@ -97,24 +100,15 @@ interface GroupState extends ToolGroup {
bodiless: boolean;
}
const responseTextSchema = z.object({
response: z.object({
status: z.unknown().optional(),
content: z
.object({ text: z.unknown().optional(), size: z.unknown().optional() })
.nullable()
.optional(),
}),
});
/** Whether a record's response had no body: a 204 or 205, an empty text, or a recorded size of zero; a missing text alone may be a body not kept. */
const emptyResponse = (record: WebNetworkCaptureRecord): boolean => {
const parsed = responseTextSchema.safeParse(record.reported);
if (!parsed.success) return false;
const { status, content } = parsed.data.response;
/** Manifest eligibility also accepts declared bodyless statuses and a recorded zero size when no text was retained. */
const manifestResponseIsEmpty = (record: WebNetworkCaptureRecord): boolean => {
const response = readRecordedResponseDeclaration(record);
if (response === undefined) return false;
const { status, content } = response;
return (
status === 204 ||
status === 205 ||
recordedResponseBodyIsEmpty(record) ||
recordedResponseBodyIsEmpty(record, response) ||
((content?.text === undefined || content.text === null) &&
content?.size === 0)
);
@@ -123,18 +117,6 @@ const emptyResponse = (record: WebNetworkCaptureRecord): boolean => {
const answeredEmpty = (group: GroupState): boolean =>
group.bodiless &&
[...group.statuses].some((status) => status >= 200 && status < 300);
const postDataSchema = z.object({
request: z.object({
postData: z
.object({
mimeType: z.unknown().optional(),
text: z.unknown().optional(),
})
.nullable()
.optional(),
}),
});
const JSON_LIKE_TEXT = new Set(["text/plain", "text/markdown"]);
const ASSET_ESSENCES = new Set([
"text/html",
"application/xhtml+xml",
@@ -170,17 +152,6 @@ const UNKNOWNS: Manifest["unknowns"] = [
},
];
/** `application/json` or `application/*+json`; a `+json` suffix on another type is not JSON here. */
const isJson = (essence: string | null): boolean =>
essence === "application/json" ||
(essence !== null &&
essence.startsWith("application/") &&
essence.endsWith("+json"));
/** JSON, `+json`, `text/plain` or `text/markdown`: the content types a replay parses as JSON. */
export const isJsonLike = (essence: string | null): boolean =>
isJson(essence) || (essence !== null && JSON_LIKE_TEXT.has(essence));
const isApiPath = (path: string): boolean =>
path.startsWith("/api/") || path.startsWith("/apis/");
@@ -260,9 +231,11 @@ export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumula
this.scratch,
);
if (line === null) return this.overBudget();
const releaseGrouping = this.temporaryCheckpoint();
if (line.kind === "excluded") {
this.#unrepresented += 1;
return ok(undefined);
releaseGrouping();
return this.onRecordedRequestLine(record, line);
}
// Coverage uses the same admitted request line as grouping, including origin and raw path.
if (scenarioResponseBody !== undefined) {
@@ -300,28 +273,39 @@ export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumula
? this.#bodyShape(record.reported)
: ok(null);
if (!bodyShape.ok) return bodyShape;
return this.#append(
const grouped = this.#append(
record.ordinal,
line,
route,
queryNames,
{ key, group },
{ shape: bodyShape.value, empty: emptyResponse(record) },
{ shape: bodyShape.value, empty: manifestResponseIsEmpty(record) },
);
if (!grouped.ok) return grouped;
// Keep the parsed line's admission while releasing grouping-only scratch.
releaseGrouping();
return this.onRecordedRequestLine(record, line);
} finally {
this.releaseTemporary();
}
}
/** Consume the owner's parsed request line after successful grouping, including exclusions. */
protected onRecordedRequestLine(
_record: WebNetworkCaptureRecord,
_line: RecordedRequestLine,
): Result<void, AnalysisError> {
return ok(undefined);
}
/** Read the top-level key types of the group's first recorded JSON-typed request body; nested values are never walked. */
#bodyShape(reported: unknown): Result<BodyShape | null, AnalysisError> {
const parsed = postDataSchema.safeParse(reported);
const body = parsed.success ? parsed.data.request.postData : undefined;
const request = readRecordedRequestBody(reported);
const body = request.kind === "request" ? request.postData : undefined;
if (
body === undefined ||
body === null ||
typeof body.text !== "string" ||
typeof body.mimeType !== "string"
body?.kind !== "object" ||
body.text === undefined ||
body.mimeType === undefined
)
return ok(null);
// The media type is scanned and lowercased before it is known to be JSON; charge it first.
@@ -1,5 +1,4 @@
import { z } from "zod";
import type { ScenarioResponseBodyRetention } from "./scenarioCaptureRecords.js";
import type { AnalysisError } from "../analysisErrorBase.js";
import { parseExactJson } from "../exactJsonNumber.js";
import { MAX_JSON_DEPTH, type JsonValue } from "../jsonValue.js";
@@ -14,10 +13,9 @@ import {
type WebNetworkCaptureRecord,
} from "../webNetworkCapture.js";
import type { UnsignedAnalysisView } from "./analysisView.js";
import {
EnvironmentManifestAccumulator,
isJsonLike,
} from "./EnvironmentManifestAccumulator.js";
import { EnvironmentManifestAccumulator } from "./EnvironmentManifestAccumulator.js";
import { isJsonLike, mediaTypeEssence } from "./recordedMediaType.js";
import { readRecordedRequestBody } from "./recordedExchangeBody.js";
import { EnvironmentStateExtraction } from "./EnvironmentStateExtraction.js";
import type { TemplateRule } from "./environmentManifestRules.js";
import {
@@ -32,14 +30,14 @@ import {
import {
base64PrefixChars,
readRecordedResponseBody,
readRecordedResponseDeclaration,
recordedResponseBodyIsEmpty,
type RecordedResponseBody,
} from "./recordedResponseBody.js";
import {
ENTRY_BYTES,
RECORD_SCRATCH_BYTES,
mediaTypeEssence,
readRecordedRequestLine,
type RecordedRequestLine,
textBytes,
} from "./webNetworkRouteTemplates.js";
@@ -96,22 +94,6 @@ const redactedWriteSchema = z.object({
request: z.object({ method: z.string() }),
response: z.object({ status: z.number() }).optional(),
});
const contentSchema = z.object({
response: z.object({ content: z.object({ text: z.string() }) }),
});
const requestSchema = z.object({
request: z.object({
postData: z.unknown().optional(),
bodySize: z.number().optional(),
}),
});
const requestContentSchema = z.object({
mimeType: z.string(),
text: z.string(),
params: z.array(z.unknown()).optional(),
});
interface RecordEntry {
readonly facts: StateRecordFacts;
/** Null for a write whose URL was redacted: its target is unknown, so it bounds every type. */
@@ -157,91 +139,83 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
return false;
}
override add(
protected override onRecordedRequestLine(
record: WebNetworkCaptureRecord,
scenarioResponseBody?: ScenarioResponseBodyRetention,
line: RecordedRequestLine,
): Result<void, AnalysisError> {
const grouped = super.add(record, scenarioResponseBody);
if (!grouped.ok) return grouped;
try {
const line = readRecordedRequestLine(record.reported, true, this.scratch);
if (line === null) return this.overBudget();
const timing = timingSchema.safeParse(record.reported);
// Admit normalization and fraction parsing before materializing their strings.
if (
timing.success &&
!this.scratch(
RECORD_SCRATCH_BYTES +
textBytes(timing.data.startedDateTime.length + 6) * 3,
)
const timing = timingSchema.safeParse(record.reported);
// Admit normalization and fraction parsing before materializing their strings.
if (
timing.success &&
!this.scratch(
RECORD_SCRATCH_BYTES +
textBytes(timing.data.startedDateTime.length + 6) * 3,
)
return this.overBudget();
const startInstant = timing.success
? readRecordedDateTime(timing.data.startedDateTime)?.instant
: undefined;
// Keep the established millisecond interval policy independently of sorting.
const start =
startInstant === undefined
? record.ordinal
: startInstant.epochSecond * 1000 +
Number(startInstant.fraction.slice(0, 3).padEnd(3, "0"));
// A fraction substring may retain its normalized timestamp's backing store,
// including trailing zeroes removed from the exact comparison value.
const instantBytes =
timing.success && startInstant !== undefined
? ENTRY_BYTES + textBytes(timing.data.startedDateTime.length + 6)
: 0;
if (line.kind === "excluded")
return line.reason === "url-redacted"
? this.#redactedWrite(
record,
timing.success && startInstant !== undefined
? {
start,
end: start + timing.data.time,
startInstant,
instantBytes,
}
: undefined,
)
: ok(undefined);
// The entry keeps the line's strings and a reference to the already admitted record.
if (
!this.retain(
ENTRY_BYTES * 8 +
instantBytes +
textBytes(
line.method.length +
line.origin.length +
line.path.length +
(line.query?.length ?? 0) +
line.rawMime.length,
),
)
)
return this.overBudget();
this.#entries.set(record.ordinal, {
facts: {
ordinal: record.ordinal,
method: line.method,
path: line.path,
query: line.query === "" ? undefined : line.query,
status: line.status,
start,
end:
)
return this.overBudget();
const startInstant = timing.success
? readRecordedDateTime(timing.data.startedDateTime)?.instant
: undefined;
// Keep the established millisecond interval policy independently of sorting.
const start =
startInstant === undefined
? record.ordinal
: startInstant.epochSecond * 1000 +
Number(startInstant.fraction.slice(0, 3).padEnd(3, "0"));
// A fraction substring may retain its normalized timestamp's backing store,
// including trailing zeroes removed from the exact comparison value.
const instantBytes =
timing.success && startInstant !== undefined
? ENTRY_BYTES + textBytes(timing.data.startedDateTime.length + 6)
: 0;
if (line.kind === "excluded")
return line.reason === "url-redacted"
? this.#redactedWrite(
record,
timing.success && startInstant !== undefined
? start + timing.data.time
: record.ordinal,
},
origin: line.origin,
contentType: line.rawMime,
startInstant,
record,
});
return ok(undefined);
} finally {
this.releaseTemporary();
}
? {
start,
end: start + timing.data.time,
startInstant,
instantBytes,
}
: undefined,
)
: ok(undefined);
// The entry keeps the line's strings and a reference to the already admitted record.
if (
!this.retain(
ENTRY_BYTES * 8 +
instantBytes +
textBytes(
line.method.length +
line.origin.length +
line.path.length +
(line.query?.length ?? 0) +
line.rawMime.length,
),
)
)
return this.overBudget();
this.#entries.set(record.ordinal, {
facts: {
ordinal: record.ordinal,
method: line.method,
path: line.path,
query: line.query === "" ? undefined : line.query,
status: line.status,
start,
end:
timing.success && startInstant !== undefined
? start + timing.data.time
: record.ordinal,
},
origin: line.origin,
contentType: line.rawMime,
startInstant,
record,
});
return ok(undefined);
}
/** Keep a successful write whose URL was redacted, with an empty path whose root relates to every type. */
@@ -294,17 +268,22 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
// A body that is not JSON-like is refused before its decoding is charged.
if (entry === undefined || !isJsonLike(mediaTypeEssence(entry.contentType)))
return { kind: "unusable" };
const content = contentSchema.safeParse(entry.record.reported);
if (!content.success) return { kind: "unusable" };
const declaration = readRecordedResponseDeclaration(entry.record);
const content = declaration?.content;
if (typeof content?.text !== "string") return { kind: "unusable" };
// Larger bodies are never parsed; within the limit, decoding copies at most the retained text and the tree is linear in it.
const length = content.data.response.content.text.length;
const length = content.text.length;
if (length > base64PrefixChars(STATE_BODY_LIMIT_BYTES))
return { kind: "unusable" };
if (!this.temporary(RECORD_SCRATCH_BYTES + textBytes(length) * 4))
return null;
const value = stateBodyJson(
entry.contentType,
readRecordedResponseBody(entry.record, STATE_BODY_LIMIT_BYTES),
readRecordedResponseBody(
entry.record,
STATE_BODY_LIMIT_BYTES,
declaration,
),
);
return value === undefined ? { kind: "unusable" } : { kind: "json", value };
}
@@ -315,16 +294,30 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
use: (body: ParsedRequestBody) => T,
): T | null {
const entry = this.#entries.get(ordinal);
const request = requestSchema.safeParse(entry?.record.reported);
if (entry === undefined || !request.success)
const request = readRecordedRequestBody(entry?.record.reported);
if (
entry === undefined ||
request.kind !== "request" ||
request.bodySize.kind === "invalid"
)
return use({ kind: "unusable" });
const { postData, bodySize } = request.data.request;
if (postData === undefined)
return use({ kind: bodySize === 0 ? "absent" : "unusable" });
const parsed = requestContentSchema.safeParse(postData);
if (!parsed.success) return use({ kind: "unusable" });
const { mimeType, text, params } = parsed.data;
if (bodySize === 0 && text === "" && (params?.length ?? 0) === 0)
const { postData, bodySize } = request;
const zeroSize = bodySize.kind === "number" && bodySize.value === 0;
if (postData.kind === "missing")
return use({ kind: zeroSize ? "absent" : "unusable" });
if (
postData.kind !== "object" ||
postData.text === undefined ||
postData.mimeType === undefined ||
postData.params.kind === "invalid"
)
return use({ kind: "unusable" });
const { mimeType, text, params } = postData;
if (
zeroSize &&
text === "" &&
(params.kind === "array" ? params.count : 0) === 0
)
return use({ kind: "absent" });
if (
!isJsonLike(mediaTypeEssence(mimeType)) ||
@@ -344,7 +337,7 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
}
}
/** Whether a record's response carried no body: a 204, or known decoded empty content. */
/** State inference accepts a 204 or known decoded empty content, independently of manifest eligibility. */
#emptyResponse(ordinal: number): boolean {
const entry = this.#entries.get(ordinal);
if (entry === undefined) return false;
@@ -1,3 +1,4 @@
import { mediaTypeEssence } from "./recordedMediaType.js";
import type { AnalysisError } from "../analysisErrorBase.js";
import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js";
import { ok, type Result } from "../result.js";
@@ -14,7 +15,6 @@ import {
GROUP_BYTES,
RECORD_SCRATCH_BYTES,
SEGMENT_BYTES,
mediaTypeEssence,
readQueryParameterNames,
readRecordedRequestLine,
scanRouteTemplate,
@@ -0,0 +1,72 @@
/** Parsed declarations borrow source strings; they do not decode bodies or normalize media types. */
export type RecordedRequestBody =
| { readonly kind: "request-unavailable" }
| {
readonly kind: "request";
readonly bodySize:
| { readonly kind: "missing" | "invalid" }
| { readonly kind: "number"; readonly value: number };
readonly postData:
| { readonly kind: "missing" | "null" | "malformed" }
| {
readonly kind: "object";
readonly text: string | undefined;
readonly mimeType: string | undefined;
readonly params:
| { readonly kind: "missing" | "invalid" }
| { readonly kind: "array"; readonly count: number };
};
};
const object = (value: unknown): Record<string, unknown> | undefined =>
typeof value === "object" && value !== null && !Array.isArray(value)
? (value as Record<string, unknown>)
: undefined;
/** Read HAR request body declarations once without turning omission or null into absence. */
export const readRecordedRequestBody = (
reported: unknown,
): RecordedRequestBody => {
const request = object(object(reported)?.request);
if (request === undefined) return { kind: "request-unavailable" };
const size = request.bodySize;
const bodySize: Extract<
RecordedRequestBody,
{ kind: "request" }
>["bodySize"] =
size === undefined
? { kind: "missing" }
: typeof size === "number" && Number.isFinite(size)
? { kind: "number", value: size }
: { kind: "invalid" };
const source = request.postData;
const postData = object(source);
return {
kind: "request",
bodySize,
postData:
postData === undefined
? {
kind:
source === undefined
? "missing"
: source === null
? "null"
: "malformed",
}
: {
kind: "object",
text: typeof postData.text === "string" ? postData.text : undefined,
mimeType:
typeof postData.mimeType === "string"
? postData.mimeType
: undefined,
params:
postData.params === undefined
? { kind: "missing" }
: Array.isArray(postData.params)
? { kind: "array", count: postData.params.length }
: { kind: "invalid" },
},
};
};
@@ -0,0 +1,34 @@
const MEDIA_TYPE_ESSENCE =
/^[A-Za-z0-9!#$%&'*+.^_`|~-]+\/[A-Za-z0-9!#$%&'*+.^_`|~-]+$/u;
const JSON_LIKE_TEXT = new Set(["text/plain", "text/markdown"]);
/** Lowercase type/subtype without parameters; malformed essences are unknown. */
export const mediaTypeEssence = (reported: string): string | null => {
const separator = reported.indexOf(";");
let start = 0;
let end = separator < 0 ? reported.length : separator;
// HTTP OWS is SP/HTAB. Scan from both ends so malformed internal runs of
// whitespace cannot cause repeated trailing-regex searches.
while (start < end && (reported[start] === " " || reported[start] === "\t"))
start += 1;
while (
end > start &&
(reported[end - 1] === " " || reported[end - 1] === "\t")
)
end -= 1;
const essence = reported.slice(start, end);
// Validate ASCII tokens before case folding can turn Unicode into ASCII.
return MEDIA_TYPE_ESSENCE.test(essence) ? essence.toLowerCase() : null;
};
/** `application/json` or `application/*+json`; a `+json` suffix on another type is not JSON here. */
export const isJson = (essence: string | null): boolean =>
essence === "application/json" ||
(essence !== null &&
essence.startsWith("application/") &&
essence.endsWith("+json"));
/** JSON, `+json`, `text/plain` or `text/markdown`: the content types a replay parses as JSON. */
export const isJsonLike = (essence: string | null): boolean =>
isJson(essence) || (essence !== null && JSON_LIKE_TEXT.has(essence));
@@ -11,15 +11,32 @@ export interface RecordedResponseBody {
readonly bodyCut?: true;
}
const contentTextSchema = z.object({
const responseDeclarationSchema = z.object({
response: z.object({
content: z.object({
text: z.unknown().optional(),
encoding: z.unknown().optional(),
}),
status: z.unknown().optional(),
content: z
.object({
text: z.unknown().optional(),
encoding: z.unknown().optional(),
size: z.unknown().optional(),
})
.nullable()
.optional(),
}),
});
/** Retained response declarations shared by decoding and consumer-specific empty-body policies. */
export const readRecordedResponseDeclaration = (
record: WebNetworkCaptureRecord,
) => {
const parsed = responseDeclarationSchema.safeParse(record.reported);
return parsed.success ? parsed.data.response : undefined;
};
export type RecordedResponseDeclaration = ReturnType<
typeof readRecordedResponseDeclaration
>;
/** Strict, and keeping a leading byte-order mark, so callers see the recorded text unchanged. */
const strictUtf8 = (): TextDecoder =>
new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
@@ -53,10 +70,13 @@ export const decodeRetainedBase64 = (
export const readRecordedResponseBody = (
record: WebNetworkCaptureRecord,
limitBytes: number,
declaration: RecordedResponseDeclaration = readRecordedResponseDeclaration(
record,
),
): RecordedResponseBody => {
const content = contentTextSchema.safeParse(record.reported);
if (!content.success) return { bodyText: null };
const { text, encoding } = content.data.response.content;
const content = declaration?.content;
if (content == null) return { bodyText: null };
const { text, encoding } = content;
if (typeof text !== "string") return { bodyText: null };
if (encoding === undefined) {
// A hidden content property could have been the encoding declaration;
@@ -80,8 +100,11 @@ export const readRecordedResponseBody = (
/** Prove decoded emptiness with a bounded prefix, preserving encoding and cut uncertainty. */
export const recordedResponseBodyIsEmpty = (
record: WebNetworkCaptureRecord,
declaration: RecordedResponseDeclaration = readRecordedResponseDeclaration(
record,
),
): boolean => {
const body = readRecordedResponseBody(record, 0);
const body = readRecordedResponseBody(record, 0, declaration);
return (
body.bodyText === "" &&
body.bodyEncoding === undefined &&
@@ -52,9 +52,6 @@ const responseSchema = z.object({
}),
});
const contentSchema = z.object({ mimeType: z.unknown().optional() });
const MEDIA_TYPE_ESSENCE =
/^[A-Za-z0-9!#$%&'*+.^_`|~-]+\/[A-Za-z0-9!#$%&'*+.^_`|~-]+$/u;
// Conservative representation allowances, not an enforced V8 heap limit.
// Fixed charges include the accumulator, bucket/set entries, array growth and
// page/schema projection. Text includes UTF-16 storage and two worst-case JSON
@@ -67,25 +64,6 @@ export const CAPTURE_CONTAINER_BYTES = 512;
const SEGMENT_SCRATCH_BYTES = 512;
export const textBytes = (length: number): number => 32 + length * 14;
/** Lowercase type/subtype without parameters; malformed essences are unknown. */
export const mediaTypeEssence = (reported: string): string | null => {
const separator = reported.indexOf(";");
let start = 0;
let end = separator < 0 ? reported.length : separator;
// HTTP OWS is SP/HTAB. Scan from both ends so malformed internal runs of
// whitespace cannot cause repeated trailing-regex searches.
while (start < end && (reported[start] === " " || reported[start] === "\t"))
start += 1;
while (
end > start &&
(reported[end - 1] === " " || reported[end - 1] === "\t")
)
end -= 1;
const essence = reported.slice(start, end);
// Validate ASCII tokens before case folding can turn Unicode into ASCII.
return MEDIA_TYPE_ESSENCE.test(essence) ? essence.toLowerCase() : null;
};
/** Route parts of a template: one more than its separators. */
export const routePartCount = (template: string): number => {
let count = 1;
+49 -19
View File
@@ -46,6 +46,12 @@ export interface FatSliceHeaderEvidence {
export interface MachoHeaderPrefixFacts {
/** Raw format evidence; null means fewer than four bytes were observed. */
readonly magic: number | null;
/** Present for a recognized four-byte thin magic, before header admission. */
readonly format: {
readonly wide: boolean;
readonly littleEndian: boolean;
readonly headerSize: 28 | 32;
} | null;
/** Present only for a recognized Mach-O magic and a complete 12-byte prefix. */
readonly identity: {
readonly cpuType: number;
@@ -57,18 +63,22 @@ export interface MachoHeaderPrefixFacts {
export const readMachoHeaderPrefixFacts = (
bytes: Uint8Array,
): MachoHeaderPrefixFacts => {
if (bytes.byteLength < 4) return { magic: null, identity: null };
if (bytes.byteLength < 4)
return { magic: null, format: null, identity: null };
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
const magic = view.getUint32(0, false);
if (bytes.byteLength < 12 || !MACHO_HEADER_MAGICS.includes(magic))
return { magic, identity: null };
const littleEndian = magic === 0xcefaedfe || magic === 0xcffaedfe;
const format = thinMachoFormat(magic);
if (format === null) return { magic, format: null, identity: null };
return {
magic,
identity: {
cpuType: view.getUint32(4, littleEndian),
cpuSubtype: view.getUint32(8, littleEndian),
},
format,
identity:
bytes.byteLength < 12
? null
: {
cpuType: view.getUint32(4, format.littleEndian),
cpuSubtype: view.getUint32(8, format.littleEndian),
},
};
};
@@ -108,9 +118,23 @@ const MAX_FAT_ARCHITECTURES = 128;
const KNOWN_MACHO_CPU_TYPES: readonly number[] = [
7, 0x01000007, 12, 0x0100000c, 0x0200000c, 18, 0x01000012,
];
const MACHO_HEADER_MAGICS: readonly number[] = [
0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe,
];
const thinMachoFormat = (magic: number): MachoHeaderPrefixFacts["format"] => {
if (![0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(magic))
return null;
const wide = magic === 0xfeedfacf || magic === 0xcffaedfe;
return {
wide,
littleEndian: magic === 0xcefaedfe || magic === 0xcffaedfe,
headerSize: wide ? 32 : 28,
};
};
/** Raw thin/FAT signature recognition; FAT magic alone does not resolve JVM ambiguity. */
export const hasMachoMagic = (bytes: Uint8Array): boolean => {
if (readMachoHeaderPrefixFacts(bytes).format !== null) return true;
const fat = readFatHeaderPrefix(bytes);
return fat.state !== "unknown" && fat.state !== "not-fat";
};
interface FatHeaderFormat {
readonly magic: number;
@@ -131,18 +155,24 @@ export type FatHeaderPrefix =
}
));
/** Interpret a raw FAT magic without asserting container identity or header readiness. */
export const readFatMagicFormat = (magic: number): FatHeaderFormat | null => {
if (![FAT_MAGIC, FAT_MAGIC_64, FAT_CIGAM, FAT_CIGAM_64].includes(magic))
return null;
return {
magic,
wide: magic === FAT_MAGIC_64 || magic === FAT_CIGAM_64,
littleEndian: magic === FAT_CIGAM || magic === FAT_CIGAM_64,
};
};
/** Observe the fixed FAT header without table, count-limit or container-identity admission. */
export const readFatHeaderPrefix = (bytes: Uint8Array): FatHeaderPrefix => {
if (bytes.byteLength < 4) return { state: "unknown", requiredBytes: 4 };
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
const magic = view.getUint32(0, false);
if (![FAT_MAGIC, FAT_MAGIC_64, FAT_CIGAM, FAT_CIGAM_64].includes(magic))
return { state: "not-fat", magic };
const format = {
magic,
wide: magic === FAT_MAGIC_64 || magic === FAT_CIGAM_64,
littleEndian: magic === FAT_CIGAM || magic === FAT_CIGAM_64,
};
const format = readFatMagicFormat(magic);
if (format === null) return { state: "not-fat", magic };
if (bytes.byteLength < 8)
return { state: "incomplete", ...format, requiredBytes: 8 };
const count = view.getUint32(4, format.littleEndian);
@@ -255,7 +285,7 @@ export const classifyFatContainerIdentity = (
const magic = sliceHeaderAt(slice.offset);
if (magic === undefined || magic === null)
missingHeaders.push(slice.offset);
else if (MACHO_HEADER_MAGICS.includes(magic)) observedMachO = true;
else if (thinMachoFormat(magic) !== null) observedMachO = true;
else observedNonMachO = true;
}
if (observedNonMachO) return { state: "not-mach-o" };
@@ -1,4 +1,9 @@
import { expect, it } from "vitest";
import { hasMachoMagic, readMachoHeaderPrefixFacts } from "./machoContainer.js";
import {
claimsExecutableHeader,
parseExecutableHeader,
} from "../binaryTarget.js";
import {
readMachoHeaderDeclarations,
readMachoLoadCommandDeclarations,
@@ -31,6 +36,29 @@ it.each([
fileType: 0xffffffff,
identity: { cpuType: 0xfedcba98, cpuSubtype: 0x89abcdef },
};
const format = { wide, littleEndian, headerSize: wide ? 32 : 28 };
for (const length of [0, 3, 4, 11, 12]) {
const prefix = bytes.subarray(0, length);
expect(hasMachoMagic(prefix)).toBe(length >= 4);
expect(readMachoHeaderPrefixFacts(prefix)).toEqual({
magic: length < 4 ? null : magic,
format: length < 4 ? null : format,
identity: length < 12 ? null : expected.identity,
});
expect(readMachoHeaderDeclarations(prefix, 32)).toEqual(
length < 4
? { ok: true, value: null }
: {
ok: false,
error: `truncated Mach-O header: a ${wide ? 64 : 32}-bit header needs ${format.headerSize} bytes; the file has 32`,
},
);
}
expect(claimsExecutableHeader(bytes.subarray(0, 4))).toBe(true);
expect(parseExecutableHeader(bytes.subarray(0, 4), "arm64")).toEqual({
ok: false,
error: "truncated or unsupported binary header",
});
expect(readMachoHeaderDeclarations(bytes, 32)).toEqual({
ok: true,
value: expected,
@@ -57,3 +85,25 @@ it.each([
).toMatchObject({ ok: false });
},
);
it.each([
0xcafebabe, 0xcafebabf, 0xbebafeca, 0xbfbafeca, 0x7f454c46, 0xffffffff,
])(
"retains raw non-thin magic %s without thin format or CPU claims",
(magic) => {
const bytes = Buffer.alloc(12, 0xff);
bytes.writeUInt32BE(magic, 0);
expect(hasMachoMagic(bytes.subarray(0, 4))).toBe(
magic !== 0x7f454c46 && magic !== 0xffffffff,
);
expect(readMachoHeaderPrefixFacts(bytes)).toEqual({
magic,
format: null,
identity: null,
});
expect(readMachoHeaderDeclarations(bytes, bytes.length)).toEqual({
ok: true,
value: null,
});
},
);
+2 -9
View File
@@ -24,19 +24,12 @@ export const readMachoHeaderDeclarations = (
fileSize: number,
): Result<MachoHeaderDeclarations | null, string> => {
const prefix = readMachoHeaderPrefixFacts(bytes);
const magic = prefix.magic;
if (
magic === null ||
![0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(magic)
)
return ok(null);
const wide = magic === 0xfeedfacf || magic === 0xcffaedfe;
const headerSize = wide ? 32 : 28;
if (prefix.format === null) return ok(null);
const { wide, headerSize, littleEndian } = prefix.format;
if (bytes.byteLength < 28 || prefix.identity === null)
return err(
`truncated Mach-O header: a ${wide ? 64 : 32}-bit header needs ${headerSize} bytes; the file has ${fileSize}`,
);
const littleEndian = magic === 0xcefaedfe || magic === 0xcffaedfe;
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
const commandBytes = view.getUint32(20, littleEndian);
return ok({
+2 -5
View File
@@ -103,9 +103,7 @@ export const claimsExecutableHeader = (
): boolean =>
(bytes.length >= 4 &&
(bytes.subarray(0, 4).equals(Buffer.from([0x7f, 0x45, 0x4c, 0x46])) ||
[0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(
bytes.readUInt32BE(0),
))) ||
readMachoHeaderPrefixFacts(bytes).format !== null)) ||
(bytes.length >= 2 && bytes[0] === 0x4d && bytes[1] === 0x5a) ||
readFatContainerHeader(bytes, fileSize).state !== "not-fat";
@@ -132,8 +130,7 @@ export const parseExecutableHeader = (
parseMzHeaderEvidence(bytes, fileSize, mzWindowsHeaderSignature),
);
if (bytes.length < 8) return err("truncated or unsupported binary header");
const magic = bytes.readUInt32BE(0);
if ([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(magic))
if (readMachoHeaderPrefixFacts(bytes).format !== null)
return parseThinMachO(bytes, fileSize);
const fatHeader =
fatEvidence?.header ?? readFatContainerHeader(bytes, fileSize);
+64 -1
View File
@@ -1,5 +1,5 @@
import { expect, it } from "vitest";
import { readElfFixedHeader } from "./elfFixedHeader.js";
import { readElfFixedHeader, readElfHeaderFacts } from "./elfFixedHeader.js";
import { elf } from "./binaryTarget.fixture.js";
it.each([
@@ -92,3 +92,66 @@ it("accepts NVIDIA's vendor ELF version only for the CUDA machine and OS ABI", (
bytes.writeUInt16LE(62, 18);
expect(readElfFixedHeader(bytes).ok).toBe(false);
});
it.each([
{ cls: 1, data: 1 },
{ cls: 1, data: 2 },
{ cls: 2, data: 1 },
{ cls: 2, data: 2 },
] as const)(
"preserves raw version, ABI and declared size facts ($cls/$data)",
({ cls, data }) => {
const original = elf(cls, data, 0xfffe);
const bytes = Buffer.concat([Buffer.alloc(7), original]).subarray(7);
bytes[8] = 0xff;
if (data === 1) {
bytes.writeUInt32LE(0x89abcdef, 20);
bytes.writeUInt16LE(0, cls === 1 ? 40 : 52);
} else {
bytes.writeUInt32BE(0x89abcdef, 20);
bytes.writeUInt16BE(0, cls === 1 ? 40 : 52);
}
expect(readElfHeaderFacts(bytes)).toMatchObject({
ok: true,
value: {
elfClass: cls === 1 ? 32 : 64,
byteOrder: data === 1 ? "little" : "big",
machine: 0xfffe,
version: 0x89abcdef,
abiVersion: 0xff,
headerSize: 0,
},
});
expect(readElfFixedHeader(bytes)).toEqual({
ok: false,
error: "unsupported ELF header version",
});
},
);
it("keeps framing failures ahead of version and declared-size admission", () => {
const bytes = elf(2, 1, 62);
bytes[6] = 0;
bytes.writeUInt32LE(0, 20);
bytes.writeUInt16LE(0, 52);
expect(readElfFixedHeader(bytes.subarray(0, 16), 16)).toEqual({
ok: false,
error: "unsupported ELF identification version",
});
bytes[6] = 1;
const truncated = readElfFixedHeader(bytes, 63);
expect(truncated).toEqual(readElfHeaderFacts(bytes, 63));
expect(truncated).toMatchObject({
ok: false,
error: expect.stringContaining("truncated ELF header"),
});
expect(readElfFixedHeader(bytes)).toEqual({
ok: false,
error: "unsupported ELF header version",
});
bytes.writeUInt32LE(1, 20);
expect(readElfFixedHeader(bytes)).toEqual({
ok: false,
error: "invalid ELF header size",
});
});
+44 -13
View File
@@ -10,7 +10,8 @@ type ElfArchitecture = Exclude<
RawOnlyArchitecture | "arm64_32"
>;
export interface ElfFixedHeader {
/** Fixed declarations after common ELF framing, before consumer admission. */
export interface ElfHeaderFacts {
readonly elfClass: 32 | 64;
readonly byteOrder: "little" | "big";
readonly headerSize: number;
@@ -18,20 +19,28 @@ export interface ElfFixedHeader {
readonly type: number;
readonly flags: number;
readonly osAbi: number;
readonly version: number;
readonly abiVersion: number;
}
export interface ElfFixedHeader extends Omit<
ElfHeaderFacts,
"version" | "abiVersion"
> {
readonly architecture: ElfArchitecture | null;
}
/**
* Validate the common, fixed ELF header commitments.
* Decode common ELF framing and fixed fields; version and declared-size admission belong to consumers.
*
* `bytes` may be a bounded prefix. `totalBytes` is the containing file size,
* so the declared header extent is checked against the file without requiring
* the whole declared extent to be present in this prefix.
* `bytes` may be a bounded prefix. `totalBytes` is the containing file size;
* common framing requires a complete fixed header in both. The declared
* headerSize is retained without requiring or admitting that entire extent.
*/
export const readElfFixedHeader = (
export const readElfHeaderFacts = (
bytes: Buffer,
totalBytes = bytes.length,
): Result<ElfFixedHeader, string> => {
): Result<ElfHeaderFacts, string> => {
if (bytes.length >= 4 && bytes.toString("hex", 0, 4) !== "7f454c46")
return err("not an ELF image");
if (bytes.length < 16) return err("ELF identification is truncated");
@@ -59,12 +68,7 @@ export const readElfFixedHeader = (
const readU32 = (offset: number): number =>
littleEndian ? bytes.readUInt32LE(offset) : bytes.readUInt32BE(offset);
const machine = readU16(18);
// NVIDIA CUBIN uses ELFOSABI_CUDA and a vendor ABI version in e_version.
if (readU32(20) !== 1 && !(machine === 190 && bytes[7] === 0x33))
return err("unsupported ELF header version");
const headerSize = readU16(elfClass === 32 ? 40 : 52);
if (headerSize < minimumSize || headerSize > totalBytes)
return err("invalid ELF header size");
return ok({
elfClass,
byteOrder: littleEndian ? "little" : "big",
@@ -73,7 +77,34 @@ export const readElfFixedHeader = (
type: readU16(16),
flags: readU32(elfClass === 32 ? 36 : 48),
osAbi: bytes[7]!,
architecture: elfArchitecture(machine) ?? null,
version: readU32(20),
abiVersion: bytes[8]!,
});
};
/** Admit the ELF version and declared header extent for general analysis. */
export const readElfFixedHeader = (
bytes: Buffer,
totalBytes = bytes.length,
): Result<ElfFixedHeader, string> => {
const parsed = readElfHeaderFacts(bytes, totalBytes);
if (!parsed.ok) return parsed;
const facts = parsed.value;
// NVIDIA CUBIN uses ELFOSABI_CUDA and a vendor ABI version in e_version.
if (facts.version !== 1 && !(facts.machine === 190 && facts.osAbi === 0x33))
return err("unsupported ELF header version");
const minimumSize = facts.elfClass === 32 ? 52 : 64;
if (facts.headerSize < minimumSize || facts.headerSize > totalBytes)
return err("invalid ELF header size");
return ok({
elfClass: facts.elfClass,
byteOrder: facts.byteOrder,
headerSize: facts.headerSize,
machine: facts.machine,
type: facts.type,
flags: facts.flags,
osAbi: facts.osAbi,
architecture: elfArchitecture(facts.machine) ?? null,
});
};
+50
View File
@@ -0,0 +1,50 @@
/** Bounded source observation; consumers own scan limits, charging and caching. */
export type ElfSectionNameObservation =
| {
readonly state: "outside";
readonly bytes: null;
readonly examinedBytes: 0;
readonly contentBytes: 0;
}
| {
readonly state: "terminated";
readonly bytes: Buffer;
readonly examinedBytes: number;
readonly contentBytes: number;
}
| {
readonly state: "truncated";
readonly bytes: null;
readonly examinedBytes: number;
readonly contentBytes: number;
};
/** Observe an ELF string-table name without treating a bounded missing NUL as absence. */
export const observeElfSectionName = (
table: Buffer,
offset: number,
maximumBytes: number,
): ElfSectionNameObservation => {
if (offset < 0 || offset >= table.length)
return { state: "outside", bytes: null, examinedBytes: 0, contentBytes: 0 };
const window = table.subarray(offset, offset + maximumBytes);
const end = window.indexOf(0);
if (end < 0)
return {
state: "truncated",
bytes: null,
examinedBytes: window.length,
contentBytes: window.length,
};
return {
state: "terminated",
bytes: window.subarray(0, end),
examinedBytes: end + 1,
contentBytes: end,
};
};
/** Project admitted source bytes without inferring an original text encoding. */
export const elfSectionNameText = (
observed: ElfSectionNameObservation,
): string | null => observed.bytes?.toString("latin1") ?? null;
+107 -102
View File
@@ -88,87 +88,88 @@ export const analysisBookmarkSchema = z.strictObject({
name: z.string().nullable(),
});
/** Complete observed function-body ranges; every range endpoint is inclusive. */
export const functionBodySchema = z.discriminatedUnion("available", [
unavailableAnalysisFactSchema,
z
.strictObject({
available: z.literal(true),
provenance: z.string().min(1),
ranges: z.array(
z.strictObject({
start: z.string(),
end: z
.string()
.describe("Inclusive last address of this observed body range."),
}),
),
total_bytes: z.int().min(0).max(Number.MAX_SAFE_INTEGER),
span_bytes: z.int().min(0).max(Number.MAX_SAFE_INTEGER).nullable(),
non_contiguous: z.boolean(),
contains_entry: z.boolean(),
})
.superRefine((body, context) => {
let total = 0n;
const bounds = new Map<string | null, { start: bigint; end: bigint }>();
const closedSpaces = new Set<string | null>();
let previousSpace: string | null | undefined;
for (const range of body.ranges) {
const start = addressCoordinate(range.start);
const end = addressCoordinate(range.end);
if (
start === null ||
end === null ||
start.space !== end.space ||
end.offset < start.offset
) {
context.addIssue({
code: "custom",
message: "Function body range has invalid inclusive endpoints",
});
return;
}
if (previousSpace !== undefined && previousSpace !== start.space)
closedSpaces.add(previousSpace);
const previous = bounds.get(start.space);
if (
closedSpaces.has(start.space) ||
(previous !== undefined && start.offset <= previous.end + 1n)
) {
context.addIssue({
code: "custom",
message:
"Function body ranges must be ordered, disjoint and maximal within each address space",
});
return;
}
bounds.set(start.space, {
start: previous?.start ?? start.offset,
end: end.offset,
});
previousSpace = start.space;
total += end.offset - start.offset + 1n;
}
const onlyBounds =
bounds.size === 1 ? [...bounds.values()][0] : undefined;
const span =
bounds.size === 0
? 0n
: onlyBounds === undefined
? null
: onlyBounds.end - onlyBounds.start + 1n;
const reportedSpan =
body.span_bytes === null ? null : BigInt(body.span_bytes);
export const availableFunctionBodySchema = z
.strictObject({
available: z.literal(true),
provenance: z.string().min(1),
ranges: z.array(
z.strictObject({
start: z.string(),
end: z
.string()
.describe("Inclusive last address of this observed body range."),
}),
),
total_bytes: z.int().min(0).max(Number.MAX_SAFE_INTEGER),
span_bytes: z.int().min(0).max(Number.MAX_SAFE_INTEGER).nullable(),
non_contiguous: z.boolean(),
contains_entry: z.boolean(),
})
.superRefine((body, context) => {
let total = 0n;
const bounds = new Map<string | null, { start: bigint; end: bigint }>();
const closedSpaces = new Set<string | null>();
let previousSpace: string | null | undefined;
for (const range of body.ranges) {
const start = addressCoordinate(range.start);
const end = addressCoordinate(range.end);
if (
total !== BigInt(body.total_bytes) ||
span !== reportedSpan ||
body.non_contiguous !== body.ranges.length > 1
)
start === null ||
end === null ||
start.space !== end.space ||
end.offset < start.offset
) {
context.addIssue({
code: "custom",
message: "Function body range has invalid inclusive endpoints",
});
return;
}
if (previousSpace !== undefined && previousSpace !== start.space)
closedSpaces.add(previousSpace);
const previous = bounds.get(start.space);
if (
closedSpaces.has(start.space) ||
(previous !== undefined && start.offset <= previous.end + 1n)
) {
context.addIssue({
code: "custom",
message:
"Function body counts, enclosing span or contiguity disagree with inclusive ranges",
"Function body ranges must be ordered, disjoint and maximal within each address space",
});
}),
return;
}
bounds.set(start.space, {
start: previous?.start ?? start.offset,
end: end.offset,
});
previousSpace = start.space;
total += end.offset - start.offset + 1n;
}
const onlyBounds = bounds.size === 1 ? [...bounds.values()][0] : undefined;
const span =
bounds.size === 0
? 0n
: onlyBounds === undefined
? null
: onlyBounds.end - onlyBounds.start + 1n;
const reportedSpan =
body.span_bytes === null ? null : BigInt(body.span_bytes);
if (
total !== BigInt(body.total_bytes) ||
span !== reportedSpan ||
body.non_contiguous !== body.ranges.length > 1
)
context.addIssue({
code: "custom",
message:
"Function body counts, enclosing span or contiguity disagree with inclusive ranges",
});
});
export const functionBodySchema = z.discriminatedUnion("available", [
unavailableAnalysisFactSchema,
availableFunctionBodySchema,
]);
/** Check the declared entry membership against complete, inclusive body ranges. */
@@ -325,23 +326,25 @@ const swiftStringLiteralFields = {
};
/** A Swift String literal decoded from the two words of its ABI value, not a data reference. */
export const swiftSmallStringLiteralSchema = z.strictObject({
...swiftStringLiteralFields,
representation: z
.literal("small")
.describe("The UTF-8 bytes are inside the two words."),
address: z.null().describe("A small literal has no storage of its own."),
});
export const swiftLargeStringLiteralSchema = z.strictObject({
...swiftStringLiteralFields,
representation: z
.literal("large")
.describe(
"An immortal literal whose words hold its count and a pointer to its bytes.",
),
address: z.string().describe("Address of a large literal's bytes."),
});
export const swiftStringLiteralSchema = z.discriminatedUnion("representation", [
z.strictObject({
...swiftStringLiteralFields,
representation: z
.literal("small")
.describe("The UTF-8 bytes are inside the two words."),
address: z.null().describe("A small literal has no storage of its own."),
}),
z.strictObject({
...swiftStringLiteralFields,
representation: z
.literal("large")
.describe(
"An immortal literal whose words hold its count and a pointer to its bytes.",
),
address: z.string().describe("Address of a large literal's bytes."),
}),
swiftSmallStringLiteralSchema,
swiftLargeStringLiteralSchema,
]);
const swiftStringLiteralOmissionFields = {
@@ -357,19 +360,21 @@ const swiftStringLiteralOmissionFields = {
};
/** A Swift literal candidate omitted because decoding capacity or control flow left it unknown. */
export const swiftSmallStringLiteralOmissionSchema = z.strictObject({
...swiftStringLiteralOmissionFields,
representation: z.literal("small"),
address: z.null(),
});
export const swiftLargeStringLiteralOmissionSchema = z.strictObject({
...swiftStringLiteralOmissionFields,
representation: z.literal("large"),
address: z.string().describe("Address of the literal's bytes."),
});
export const swiftStringLiteralOmissionSchema = z.discriminatedUnion(
"representation",
[
z.strictObject({
...swiftStringLiteralOmissionFields,
representation: z.literal("small"),
address: z.null(),
}),
z.strictObject({
...swiftStringLiteralOmissionFields,
representation: z.literal("large"),
address: z.string().describe("Address of the literal's bytes."),
}),
swiftSmallStringLiteralOmissionSchema,
swiftLargeStringLiteralOmissionSchema,
],
);
@@ -416,14 +416,11 @@ const stdioMode = (
method: JavaScriptSemanticChildProcessSpawn["method"],
context: ChildSpawnContext,
): string => {
// Async exec/execFile do not forward options.stdio to their underlying spawn.
if (method === "exec" || method === "execFile") return "default";
if (options === undefined) return "default";
const expression = unwrapJavaScriptExpression(options).node;
if (
context.isNullish(expression) ||
((method === "exec" || method === "execFile") &&
context.isCallback(expression))
)
return "default";
if (context.isNullish(expression)) return "default";
const lookup = staticObjectProperty(expression, "stdio", context.isNullish);
if (lookup.status === "missing")
return unsetStdioMode(expression, method, context);
@@ -13,9 +13,7 @@ import {
} from "./javascriptSemanticTraversal.js";
it.each([
["function f(){read(cb);function cb(){}}", ["callback"]],
["function f(){const cb=()=>{};const alias=cb;read(alias)}", ["callback"]],
["function f(cb){cb=()=>{};arguments[0]={};read(cb)}", ["unknown"]],
["function f(a){a=[];arguments[0]={};read(a)}", ["unknown"]],
["declare let erased:any;function f(){const a=[];read(a)}", ["unknown"]],
["namespace X{};function f(){const a=[];read(a)}", ["unknown"]],
["var a=[];(()=>{}).constructor('a={}')();read(a);", ["unknown"]],
@@ -60,11 +58,9 @@ it.each([
shapes.push(
resolver.isArray(arg)
? "array"
: resolver.isCallback(arg)
? "callback"
: resolver.isNullish(arg)
? "nullish"
: "unknown",
: resolver.isNullish(arg)
? "nullish"
: "unknown",
);
},
});
@@ -72,10 +68,10 @@ it.each([
},
);
it("bounds long callback aliases without invoking the shared callable resolver", () => {
const source = `export function run(){const cb=()=>{};
${Array.from({ length: 4096 }, (_, i) => `const a${i}=${i === 0 ? "cb" : `a${i - 1}`};`).join("\n")}
read(a4095);read(a10);read(cb);}`;
it("bounds long array aliases without invoking the shared callable resolver", () => {
const source = `export function run(){const argv=[];
${Array.from({ length: 4096 }, (_, i) => `const a${i}=${i === 0 ? "argv" : `a${i - 1}`};`).join("\n")}
read(a4095);read(a10);read(argv);}`;
const file = parseJavaScriptSource(source, "deep.mjs");
if (file === null) throw new Error("Expected valid deep source");
const state = completeSemanticSteps(
@@ -89,7 +85,7 @@ it("bounds long callback aliases without invoking the shared callable resolver",
t.isCallExpression(node) &&
t.isIdentifier(node.callee, { name: "read" })
)
results.push(resolver.isCallback(node.arguments[0]!));
results.push(resolver.isArray(node.arguments[0]!));
},
});
expect(results).toEqual([false, true, true]);
@@ -97,55 +93,30 @@ it("bounds long callback aliases without invoking the shared callable resolver",
});
it.each([
["fixture.cjs", "const argv=[];const done=()=>{};", ["inherit", "default"]],
["fixture.cts", "const argv=[];const done=()=>{};", ["inherit", "default"]],
["fixture.cjs", "const argv=[];", ["inherit"]],
["fixture.cts", "const argv=[];", ["inherit"]],
["fixture.cjs", "var argv=[];{function argv(){}}", ["dynamic"]],
["fixture.mjs", "var argv=[];{function argv(){}}", ["inherit"]],
["fixture.js", "const argv=[];", ["dynamic"]],
["fixture.cjs", "let argv=[];argv=input;", ["dynamic"]],
["fixture.cts", "const argv=[];eval(input);", ["inherit"]],
["fixture.cts", "namespace N{};const argv=[];", ["dynamic"]],
[
"fixture.cjs",
"var argv=[];{function argv(){}}const done=()=>{};",
["dynamic", "default"],
],
[
"fixture.mjs",
"var argv=[];{function argv(){}}const done=()=>{};",
["inherit", "default"],
],
["fixture.js", "const argv=[];const done=()=>{};", ["dynamic", "dynamic"]],
[
"fixture.cjs",
"let argv=[];argv=input;const done=()=>{};",
["dynamic", "default"],
],
[
"fixture.cts",
"const argv=[];const done=()=>{};eval(input);",
["inherit", "default"],
],
[
"fixture.cts",
"namespace N{};const argv=[];const done=()=>{};",
["dynamic", "dynamic"],
],
[
"fixture.cjs",
"var module=[];const argv=module;const done=()=>{};const args=arguments;args[2]={};",
["dynamic", "default"],
],
[
"fixture.cts",
"const argv=[];function exports(){};const args=arguments;args[0]={};const done=exports;",
["inherit", "dynamic"],
"var module=[];const argv=module;const args=arguments;args[2]={};",
["dynamic"],
],
] as const)(
"distinguishes wrapper-local child identities and uncertain bindings: %s (%s)",
(path, declarations, expected) => {
const loader = path.endsWith(".mjs")
? 'import {spawn,exec} from "node:child_process";'
: 'const {spawn,exec}=require("node:child_process");';
? 'import {spawn} from "node:child_process";'
: 'const {spawn}=require("node:child_process");';
const file = parseJavaScriptSource(
`
${loader}
${declarations}
spawn("argv",argv,{stdio:"inherit"}); exec("callback",done);
spawn("argv",argv,{stdio:"inherit"});
`,
path,
);
@@ -23,7 +23,7 @@ import {
import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js";
import { javaScriptRuntimeContext } from "./javascriptSourceParser.js";
type Shape = "array" | "callback" | "unknown";
type Shape = "array" | "unknown";
/** Bounded argument-shape queries shared by call-signature owners. */
export type JavaScriptExpressionShapeResolver = ReturnType<
@@ -41,15 +41,6 @@ const commonJsWrapperParameterNames = new Set([
export const createSemanticExpressionShapeResolver = (
state: JavaScriptSemanticAnalysisState,
) => {
const hoistedFunction = (node: t.Node): boolean => {
const parent = state.parentsByNode.get(node);
return (
t.isFunctionDeclaration(node) &&
(t.isProgram(parent) ||
(t.isBlockStatement(parent) &&
t.isFunction(state.parentsByNode.get(parent))))
);
};
// Most modules have no named child arguments. Collect source-wide blockers
// only when a query actually needs a named binding or ambient undefined.
let blocked: boolean | undefined;
@@ -80,8 +71,6 @@ export const createSemanticExpressionShapeResolver = (
if (seen.has(node)) return "unknown";
seen.add(node);
if (t.isArrayExpression(node)) return "array";
if (t.isFunction(node) && !("declare" in node && node.declare))
return "callback";
if (!t.isIdentifier(node)) return "unknown";
const binding = resolveSemanticBindingState(state, node, node.name);
if (binding === undefined || binding.referenceInitializers.length > 0)
@@ -106,11 +95,7 @@ export const createSemanticExpressionShapeResolver = (
if (initializer === undefined || initializer.projection.length > 0)
return "unknown";
const read = semanticBindingAtRead(binding, node, state.parentsByNode);
// Hoisted function declarations are the only accepted unordered source.
if (
!hoistedFunction(initializer.node) &&
(read.status !== "ordered" || !read.hasDefiniteInitializer)
)
if (read.status !== "ordered" || !read.hasDefiniteInitializer)
return "unknown";
node = initializer.node;
depth++;
@@ -118,7 +103,6 @@ export const createSemanticExpressionShapeResolver = (
};
return {
isArray: (node: t.Node) => shape(node) === "array",
isCallback: (node: t.Node) => shape(node) === "callback",
isNullish: (input: t.Node) => {
const { node } = unwrapJavaScriptExpression(input);
return (
+4 -1
View File
@@ -3,6 +3,9 @@ import { addressCoordinate } from "./addressCoordinate.js";
import { AnalysisInputError } from "../analysisErrorCore.js";
import { err, ok, type Result } from "../result.js";
/** Maximum matches delivered by one byte-search page. */
export const MAX_BYTE_SEARCH_MATCHES = 1000;
/** Exact grammar belongs to the object refinement, not a repeating advertised regex. */
export const bytePatternFields = {
pattern: z
@@ -152,7 +155,7 @@ export const byteSearchResultSchema = z
.nullable(),
}),
)
.max(1000),
.max(MAX_BYTE_SEARCH_MATCHES),
searched: z.strictObject({
blocks: z.array(coverageBlock),
bytes: z.int().nonnegative().max(Number.MAX_SAFE_INTEGER),
+2 -2
View File
@@ -87,7 +87,7 @@ const inferredJumpTableSchema = z
})
.strict();
const availableNativeApiBoundarySchema = z
export const availableNativeApiBoundarySchema = z
.object({
available: z.literal(true),
provenance: z.string().min(1),
@@ -115,7 +115,7 @@ const availableNativeApiBoundarySchema = z
})
.strict();
const unavailableNativeApiBoundarySchema = z
export const unavailableNativeApiBoundarySchema = z
.object({
available: z.literal(false),
reason: z.string().min(1),
Loaded 100 of 164 files, more files were not shown because too many files have changed in this diff. Show more