mirror of
https://github.com/morluto/rea.git
synced 2026-11-05 14:26:07 +00:00
Merge remote-tracking branch 'origin/main' into feat/environment-write-quality
This commit is contained in:
commit
0ceff6fa69
164 files changed
+6292
-1705
No files matched your search
+7
-1
@@ -443,7 +443,13 @@ The selected provider remains bound to the session until an explicit switch
|
||||
or close. Provider failures are returned with their original reason. For an
|
||||
`ambiguous` selection error, choose from `details.candidate_ids`; for
|
||||
`provider_unavailable`, follow `message`, `remediation.action`, and the original
|
||||
reasons in `details.rejections`. For Hopper, Ghidra, IDA, JADX, and Kuna, run
|
||||
reasons in `details.rejections`. Each rejection names its `origin`: `availability`
|
||||
entries describe local readiness failures, including missing configuration or an
|
||||
invalid existing registration; they do not establish target support. `target`
|
||||
entries record a refused kind, format, architecture, or role. An availability-only
|
||||
message describes the recorded rejections without claiming that every provider
|
||||
is unconfigured or that target support is known.
|
||||
For Hopper, Ghidra, IDA, JADX, and Kuna, run
|
||||
`rea doctor --provider ID --json` to diagnose the selected engine. Other adapters,
|
||||
such as `pwntools-elf`, report their configuration requirements directly and are
|
||||
not accepted by `doctor --provider`. See
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Native directory decompilation verification — 2026-10-23
|
||||
|
||||
The [2026-10-26 follow-up](https://github.com/morluto/rea/blob/72454004772f826440c5e21002f4dd52b4ad43c0/experiments/analysis-efficiency/decompile-directory.md)
|
||||
adds a 14,338-function real-provider capacity run, cross-session project reuse,
|
||||
and a correction preserving historical source Evidence through shared workflows.
|
||||
|
||||
The completed analyst task is to export the bound provider's function inventory
|
||||
as one local pseudocode file per non-external function, with names, body ranges,
|
||||
provider/profile identity and the limitations of each used observation. The
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
# Directory decompilation: capacity, resume and source Evidence
|
||||
|
||||
The directory-export feature requested in #6090 already shipped in #6170.
|
||||
This follow-up verifies the current public workflow and fixes a source-provenance
|
||||
gap found while combining it with existing Ghidra projects (#1048).
|
||||
|
||||
## Completed task and correctness
|
||||
|
||||
The analyst asks for all functions as local pseudocode files, without putting
|
||||
their bodies in agent context. The committed `decompile-directory.mjs` driver
|
||||
uses the production stdio MCP entrypoint and installed SDK. It discovers the
|
||||
catalog, exports an initial chunk, closes/reopens the provider, resumes all
|
||||
remaining work, and resumes again to verify settled files. It authenticates an
|
||||
explicitly exported Evidence bundle and checks the original target/project.
|
||||
|
||||
Every written file is checked against its index SHA-256 and address/profile
|
||||
header. Existing file hashes and modification times must survive both resumes.
|
||||
Comparison facts include every address, name, status, body range and pseudocode
|
||||
digest. An independent fresh-import investigation from #1048 additionally checks
|
||||
the complete inventory and two nonempty pseudocode observations. This establishes
|
||||
export preservation of provider observations; it does not establish discovery of
|
||||
every actual function or equivalence to original source.
|
||||
|
||||
| Real artifact | Provider inventory | Written | External | Failed |
|
||||
| ----------------------- | -----------------: | ------: | -------: | -----: |
|
||||
| `/usr/bin/ls` | 652 | 541 | 111 | 0 |
|
||||
| system `libcrypto.so.3` | 14,338 | 14,194 | 143 | 1 |
|
||||
|
||||
Target SHA-256 values are respectively
|
||||
`cb30d69b24245bf2ecdc9e7f53bbad19159999970b6d82c0c00c7d32d9e37aa4` and
|
||||
`aa01e614af634bc7c8b678c9250b5c4f613c61989bacf0a5d6aed6570e7dcabd`.
|
||||
These installed ELF x86-64 binaries were analyzed statically, never executed.
|
||||
The large run explicitly selected a five-second per-function deadline; function
|
||||
`0x387f00` exceeded it, remained a recorded failure, and was retried on resume.
|
||||
Later functions were still exported. No disappearance of that failure is claimed.
|
||||
The resulting large tree occupies about 107 MiB on this filesystem.
|
||||
|
||||
## Source-provenance correction
|
||||
|
||||
Before this change, `BinarySession` retained provider `sourceEvidence` only inside
|
||||
its snapshot-eligible branch. Ghidra procedure reads without a document selector
|
||||
are not snapshot eligible. A composed directory export therefore lost saved
|
||||
project reports even though the provider supplied them. The workflow also ignored
|
||||
source-record IDs, and the function-dossier adapter copied observation fields
|
||||
manually without those links.
|
||||
|
||||
The session now authenticates/retains source records independently of replay
|
||||
eligibility. Plain application helpers also return authenticated immutable source
|
||||
records with their outcome, including late cancellation. Both adapters explicitly
|
||||
commit them before delivering links; an adapter without a retention owner refuses
|
||||
the response. Source collection borrows ledger records without constructing or
|
||||
hashing an intermediate bundle. The workflow links distinct source IDs without copying their payloads.
|
||||
The function adapter uses the common observation-to-Evidence constructor,
|
||||
preserving sources through normal and late-cancelled completion. Queries with no
|
||||
source records avoid the extra recording path. Complete provider bodies remain
|
||||
in the exported function files; this does not add per-function payload retention.
|
||||
|
||||
A real test project contains historical seed and disabled-analyzer reports saved
|
||||
by REA's actual Ghidra pre-scripts. The unmodified runtime completed the export
|
||||
but failed the provenance oracle: its bundle had two workflow records and no
|
||||
historical source. The fixed runtime retains exactly one historical source and
|
||||
links the workflow to it. All 652 inventory entries and the sampled pseudocode
|
||||
still match fresh import; source project content stays unchanged. The report is
|
||||
historical, with remaining original settings/completeness explicitly unknown.
|
||||
|
||||
## Measured costs
|
||||
|
||||
Baseline runtime: `19447516a4bdbc9437d112ebd3f12a5fd062db83`.
|
||||
Initial fixed runtime: `f50763fd21527bdf3bab944633617b9df61dded4`, rebased on
|
||||
main `f02df45a4`. Both use Node 24.22.1, locked dependencies, actual Ghidra 12.1.4
|
||||
and JDK 21.0.12.1+1. The fixed driver later delegates transcript publication to
|
||||
the existing cleanup/receipt owner; the protocol and task are unchanged.
|
||||
|
||||
| Completed task | Wall s | Family CPU s | Peak family GiB | Full-wire token estimate |
|
||||
| --------------------------------- | ------: | -----------: | --------------: | -----------------------: |
|
||||
| ls, before | 51.411 | 78.984 | 0.943 | 57,082 |
|
||||
| ls, initial fix | 50.937 | 79.891 | 1.201 | 57,072 |
|
||||
| ls with historical reports, fixed | 52.827 | 80.236 | 1.097 | 57,207 |
|
||||
| libcrypto, baseline capacity run | 506.387 | 459.818 | 1.502 | 57,322 |
|
||||
|
||||
The plain ls pair has identical complete comparison facts. Runtime is similar;
|
||||
the higher sampled memory in the fixed run is reported rather than treated as an
|
||||
improvement. These sequential shared-host samples establish no generic speed or
|
||||
memory ratio. [PR #7250](https://github.com/morluto/rea/pull/7250) records the final
|
||||
large-run comparison and CI/review results.
|
||||
The failed historical baseline is excluded from completed-task comparisons.
|
||||
|
||||
The complete MCP replies stayed around 11–12 KiB, including repeated textual and
|
||||
structured representations, profile and caveats. The normalized result is only
|
||||
the manifest. For example, the large continuation costs about 3,159 estimated
|
||||
wire tokens; its complete task is 57,322 because discovery alone costs 47,049.
|
||||
Both numbers include actual protocol content, rather than only pseudocode bytes.
|
||||
|
||||
Estimates use `cl100k_base` on the actual JSON-RPC messages, including discovery,
|
||||
requests, responses and notifications, excluding only transport framing and the
|
||||
driver's local transcript wrapper. They are not actual model billing. Host
|
||||
projection, prompts/reasoning and later agent file reads remain unmeasured.
|
||||
The historical source increases response token cost slightly while preserving
|
||||
provenance. No token-reduction claim is made for this correctness fix.
|
||||
|
||||
CPU and memory cover the driver, REA and provider process family. Each run uses
|
||||
CPU affinity 0,1 and an 8-GiB sampled family-RSS guard; Java heaps are 1536M for ls
|
||||
and 3072M for crypto. RSS is sampled every 100 ms and summed: shared pages may be
|
||||
counted repeatedly and short peaks may be missed. CPU is waited child usage.
|
||||
Reported completed runs had no resource stop, OOM or cleanup failure.
|
||||
Project preparation is excluded and documented separately in
|
||||
[project reuse](ghidra-project-reuse.md). Provider starts are process-cold, with
|
||||
host filesystem caches unflushed; verified resume occurs in the second session.
|
||||
|
||||
The index and inventory still scale with function count. This 14,338-item run
|
||||
does not establish a larger capacity guarantee. Real Windows/macOS filesystems,
|
||||
other decompilers and native DWARF directory layout remain unverified. The current
|
||||
workflow reports unavailable DWARF layout explicitly instead of inventing it.
|
||||
|
||||
Actual CLI `decompile-all` calls additionally selected only `0x10a420`, reported
|
||||
651 excluded entries and `debug_layout_unavailable` for requested dwarf layout,
|
||||
then resumed in a new process. The existing file's hash and modification time
|
||||
stayed unchanged; the returned historical source ID stayed stable. This CLI check
|
||||
verifies the returned manifest/link and local function file, not offline retrieval
|
||||
of the separately retained source record after the CLI process exits.
|
||||
|
||||
## Reproduce and inspect
|
||||
|
||||
Prepare an owned project using the documented
|
||||
[existing-project route](../../docs/installation.md#reuse-an-existing-ghidra-project).
|
||||
Use fresh receipt/output paths; the driver refuses existing output directories.
|
||||
|
||||
```sh
|
||||
node experiments/analysis-efficiency/decompile-directory.mjs \
|
||||
--runtime /absolute/rea --output /absolute/new-receipt \
|
||||
--target /usr/bin/ls --project /absolute/ls-analysis.gpr --program ls \
|
||||
--ghidra "$GHIDRA_INSTALL_DIR" --java "$JAVA_HOME"
|
||||
```
|
||||
|
||||
Use `--chunk 512 --heap 3072M --decompile-seconds 5` for the crypto case.
|
||||
`--control /absolute/previous-receipt` compares all exported facts;
|
||||
`--golden /absolute/fresh-import-receipt` checks the #1048 driver's inventory and
|
||||
pseudocode oracle. `--history required` additionally requires a linked historical
|
||||
source. The driver allows failed functions to succeed on a later retry, verifies only
|
||||
previously written files remain unchanged, and reloads the final index before
|
||||
comparison. Complete local receipts include the transcript, resource report, index,
|
||||
function files, comparison facts and authenticated Evidence export. The resource
|
||||
report is supplied by the external process-family sampler, not the driver.
|
||||
|
||||
The current nine-file suite passed 144 tests. Against unmodified main source,
|
||||
the three source-retention/constructor suites reproduce seven failures and 70
|
||||
passes; the fixed full suite passes. This includes invalid source commitments,
|
||||
live and cursor-dependent reads, shared source deduplication, selected/null/provider
|
||||
subjects, and source links on late-cancelled completion.
|
||||
@@ -0,0 +1,328 @@
|
||||
/** Real-provider directory export, cross-session resume and retained provenance. */
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createWriteStream } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, stat, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { finished } from "node:stream/promises";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { parseArgs, promisify } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: Object.fromEntries(
|
||||
[
|
||||
"runtime",
|
||||
"output",
|
||||
"target",
|
||||
"project",
|
||||
"program",
|
||||
"ghidra",
|
||||
"java",
|
||||
"control",
|
||||
"heap",
|
||||
"chunk",
|
||||
"decompile-seconds",
|
||||
"history",
|
||||
"golden",
|
||||
].map((key) => [key, { type: "string" }]),
|
||||
),
|
||||
});
|
||||
for (const key of [
|
||||
"runtime",
|
||||
"output",
|
||||
"target",
|
||||
"project",
|
||||
"ghidra",
|
||||
"java",
|
||||
]) {
|
||||
assert.ok(values[key], `${key} is required`);
|
||||
values[key] = resolve(values[key]);
|
||||
}
|
||||
const chunk = Number(values.chunk ?? 32);
|
||||
assert.ok(Number.isSafeInteger(chunk) && chunk > 0);
|
||||
await mkdir(values.output);
|
||||
const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
|
||||
const report = {
|
||||
calls: [],
|
||||
runtime: values.runtime,
|
||||
target: values.target,
|
||||
project: values.project,
|
||||
};
|
||||
let client, transport, account, serverLog, serverLogFinished;
|
||||
// This workflow has a fixed number of public calls; pseudocode stays on disk.
|
||||
const transcript = [];
|
||||
let phase = "initialize";
|
||||
const started = performance.now();
|
||||
const tree = join(values.output, "tree");
|
||||
const readIndex = async () =>
|
||||
JSON.parse(await readFile(join(tree, "decompile-index.json"), "utf8"));
|
||||
const fileStates = async (index) => {
|
||||
const states = {};
|
||||
for (const row of index.functions.filter((row) => row.status === "written")) {
|
||||
const path = join(tree, row.relative_path);
|
||||
const metadata = await stat(path);
|
||||
states[row.address] = {
|
||||
sha256: digest(await readFile(path)),
|
||||
mtime_ms: metadata.mtimeMs,
|
||||
};
|
||||
assert.equal(states[row.address].sha256, row.sha256);
|
||||
}
|
||||
return states;
|
||||
};
|
||||
try {
|
||||
report.revision = (
|
||||
await promisify(execFile)("git", [
|
||||
"-C",
|
||||
values.runtime,
|
||||
"rev-parse",
|
||||
"HEAD",
|
||||
])
|
||||
).stdout.trim();
|
||||
report.target_sha256 = digest(await readFile(values.target));
|
||||
const { resolveGhidraProject } = await import(
|
||||
pathToFileURL(join(values.runtime, "dist/ghidra/GhidraProjectSnapshot.js"))
|
||||
.href
|
||||
);
|
||||
const { ArtifactResourceScope } = await import(
|
||||
pathToFileURL(
|
||||
join(values.runtime, "dist/artifacts/ArtifactResourceScope.js"),
|
||||
).href
|
||||
);
|
||||
const fingerprint = async () => {
|
||||
const resources = new ArtifactResourceScope();
|
||||
try {
|
||||
return await resolveGhidraProject(
|
||||
{
|
||||
path: values.project,
|
||||
...(values.program ? { programName: values.program } : {}),
|
||||
},
|
||||
{ path: values.target },
|
||||
resources,
|
||||
);
|
||||
} finally {
|
||||
await resources.close();
|
||||
}
|
||||
};
|
||||
const original = await fingerprint();
|
||||
report.project_sha256 = original.sha256;
|
||||
account = await mkdtemp(join(tmpdir(), "rea-directory-investigation-"));
|
||||
transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
|
||||
cwd: values.runtime,
|
||||
maxBufferSize: 10 * 1024 * 1024,
|
||||
env: {
|
||||
PATH: process.env.PATH,
|
||||
HOME: account,
|
||||
USERPROFILE: account,
|
||||
JAVA_HOME: values.java,
|
||||
GHIDRA_INSTALL_DIR: values.ghidra,
|
||||
GHIDRA_HEADLESS_MAXMEM: values.heap ?? "1536M",
|
||||
JAVA_TOOL_OPTIONS: "-XX:ActiveProcessorCount=2",
|
||||
NODE_OPTIONS: process.env.NODE_OPTIONS ?? "--max-old-space-size=3072",
|
||||
REA_GHIDRA_PROJECT_PATH: values.project,
|
||||
...(values.program ? { REA_GHIDRA_PROGRAM_NAME: values.program } : {}),
|
||||
REA_GHIDRA_STARTUP_TIMEOUT_MS: "600000",
|
||||
...(values["decompile-seconds"]
|
||||
? { REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS: values["decompile-seconds"] }
|
||||
: {}),
|
||||
REA_MCP_TOOL_FAMILIES: "direct,enhanced,application,session",
|
||||
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
|
||||
REA_MCP_OUTPUT_SCHEMA_PROFILE: "compact",
|
||||
},
|
||||
stderr: "pipe",
|
||||
});
|
||||
const start = transport.start.bind(transport);
|
||||
transport.start = async () => {
|
||||
const receive = transport.onmessage;
|
||||
transport.onmessage = (message) => {
|
||||
transcript.push({ phase, direction: "response", message });
|
||||
receive?.(message);
|
||||
};
|
||||
await start();
|
||||
};
|
||||
const send = transport.send.bind(transport);
|
||||
transport.send = async (message) => {
|
||||
transcript.push({ phase, direction: "request", message });
|
||||
await send(message);
|
||||
};
|
||||
serverLog = createWriteStream(join(values.output, "server.log"));
|
||||
serverLogFinished = finished(serverLog);
|
||||
serverLogFinished.catch(() => {});
|
||||
transport.stderr.pipe(serverLog);
|
||||
client = new Client({ name: "rea-directory-investigation", version: "1" });
|
||||
await client.connect(transport);
|
||||
phase = "discovery";
|
||||
const catalog = await client.listTools({}, { timeout: 15000 });
|
||||
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
|
||||
report.tools = catalog.tools.length;
|
||||
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
|
||||
const call = async (name, args, label) => {
|
||||
phase = label;
|
||||
const at = performance.now();
|
||||
const result = await client.callTool(
|
||||
{ name, arguments: args },
|
||||
{ timeout: 900000, toolDefinition: definitions.get(name) },
|
||||
);
|
||||
report.calls.push({
|
||||
name,
|
||||
phase,
|
||||
elapsed_ms: performance.now() - at,
|
||||
response_bytes: Buffer.byteLength(JSON.stringify(result)),
|
||||
is_error: result.isError ?? false,
|
||||
});
|
||||
assert.notEqual(
|
||||
result.isError,
|
||||
true,
|
||||
`${name}: ${JSON.stringify(result).slice(0, 800)}`,
|
||||
);
|
||||
return result.structuredContent;
|
||||
};
|
||||
const open = () =>
|
||||
call("open_binary", { path: values.target, provider_id: "ghidra" }, "open");
|
||||
await open();
|
||||
const first = await call(
|
||||
"decompile_all",
|
||||
{ output_directory: tree, limit: chunk },
|
||||
"first-chunk",
|
||||
);
|
||||
report.first = first.normalized_result;
|
||||
const firstFiles = await fileStates(await readIndex());
|
||||
await call("close_binary", {}, "close-first-session");
|
||||
await open();
|
||||
const remaining = await call(
|
||||
"decompile_all",
|
||||
{ output_directory: tree, resume: true },
|
||||
"cross-session-resume",
|
||||
);
|
||||
report.resume = remaining.normalized_result;
|
||||
report.resume_evidence_links = remaining.evidence_links?.length ?? 0;
|
||||
let index = await readIndex();
|
||||
const allFiles = await fileStates(index);
|
||||
for (const [address, state] of Object.entries(firstFiles))
|
||||
assert.deepEqual(allFiles[address], state);
|
||||
report.first_files_unchanged = true;
|
||||
const settled = await call(
|
||||
"decompile_all",
|
||||
{ output_directory: tree, resume: true },
|
||||
"verified-resume",
|
||||
);
|
||||
report.verified_resume = settled.normalized_result;
|
||||
if (report.resume.counts.failed === 0)
|
||||
assert.equal(report.verified_resume.counts.written, 0);
|
||||
assert.equal(report.verified_resume.counts.remaining, 0);
|
||||
index = await readIndex();
|
||||
const settledFiles = await fileStates(index);
|
||||
for (const [address, state] of Object.entries(allFiles))
|
||||
assert.deepEqual(settledFiles[address], state);
|
||||
assert.equal(index.functions.length, report.resume.counts.selected);
|
||||
const facts = [];
|
||||
for (const row of index.functions) {
|
||||
const fact = {
|
||||
address: row.address,
|
||||
name: row.name,
|
||||
status: row.status,
|
||||
body: row.body,
|
||||
};
|
||||
if (row.status === "written") {
|
||||
const text = await readFile(join(tree, row.relative_path), "utf8");
|
||||
assert.ok(text.includes(`// address: ${row.address}\n`));
|
||||
assert.ok(
|
||||
text.includes(
|
||||
`// analysis_profile: ${index.analysis_profile_digest}\n`,
|
||||
),
|
||||
);
|
||||
fact.pseudocode_sha256 = digest(text.slice(text.indexOf("\n\n") + 2));
|
||||
}
|
||||
facts.push(fact);
|
||||
}
|
||||
report.function_count = facts.length;
|
||||
report.written = facts.filter((row) => row.status === "written").length;
|
||||
report.failed = facts.filter((row) => row.status === "failed").length;
|
||||
report.external = facts.filter((row) => row.status === "external").length;
|
||||
report.facts_sha256 = digest(JSON.stringify(facts));
|
||||
await writeFile(join(values.output, "facts.json"), JSON.stringify(facts));
|
||||
const evidencePath = join(values.output, "complete-evidence.json");
|
||||
await call(
|
||||
"export_evidence_bundle",
|
||||
{ path: evidencePath },
|
||||
"complete-evidence-export",
|
||||
);
|
||||
await call("close_binary", {}, "close");
|
||||
await client.close();
|
||||
await transport.close();
|
||||
const { parseEvidenceBundle } = await import(
|
||||
pathToFileURL(join(values.runtime, "dist/domain/evidenceBundle.js")).href
|
||||
);
|
||||
const bundle = parseEvidenceBundle(
|
||||
JSON.parse(await readFile(evidencePath, "utf8")),
|
||||
);
|
||||
report.evidence_records = bundle.records.length;
|
||||
const retained = bundle.records.find(
|
||||
(record) => record.evidence_id === remaining.evidence_id,
|
||||
);
|
||||
assert.ok(retained);
|
||||
report.retained_links = retained.evidence_links.length;
|
||||
const byId = new Set(bundle.records.map((record) => record.evidence_id));
|
||||
for (const id of retained.evidence_links) assert.ok(byId.has(id));
|
||||
report.all_source_links_resolve = true;
|
||||
if (values.history === "required") {
|
||||
const historical = bundle.records.filter(
|
||||
(record) =>
|
||||
record.operation === "health" &&
|
||||
record.parameters.facet === "historical-project-analysis-reports",
|
||||
);
|
||||
assert.equal(historical.length, 1);
|
||||
assert.equal(historical[0].normalized_result.origin, "caller-project");
|
||||
assert.ok(retained.evidence_links.includes(historical[0].evidence_id));
|
||||
report.historical_source_retained_and_linked = true;
|
||||
}
|
||||
if (values.golden) {
|
||||
const golden = JSON.parse(
|
||||
await readFile(join(values.golden, "facts.json"), "utf8"),
|
||||
);
|
||||
const labels = (rows) =>
|
||||
rows
|
||||
.map((row) => [row.address, row.name ?? row.value])
|
||||
.sort((a, b) => a[0].localeCompare(b[0], "en"));
|
||||
assert.deepEqual(labels(facts), labels(golden.procedures));
|
||||
for (const item of golden.pseudocode) {
|
||||
const fact = facts.find((row) => row.address === item.procedure);
|
||||
assert.equal(fact?.pseudocode_sha256, digest(item.result));
|
||||
}
|
||||
report.fresh_import_inventory_and_sampled_pseudocode_match = true;
|
||||
}
|
||||
if (values.control) {
|
||||
const control = JSON.parse(
|
||||
await readFile(join(values.control, "facts.json"), "utf8"),
|
||||
);
|
||||
assert.deepEqual(facts, control);
|
||||
report.control_facts_match = true;
|
||||
}
|
||||
assert.deepEqual(await fingerprint(), original);
|
||||
assert.equal(digest(await readFile(values.target)), report.target_sha256);
|
||||
report.source_unchanged = true;
|
||||
report.outcome = "passed";
|
||||
} catch (cause) {
|
||||
report.outcome = "failed";
|
||||
report.failure = String(cause);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
report.elapsed_ms = performance.now() - started;
|
||||
const published = await finalizeRetainedQueryReceipt({
|
||||
report,
|
||||
client,
|
||||
transport,
|
||||
account,
|
||||
serverLog,
|
||||
serverLogFinished,
|
||||
output: values.output,
|
||||
transcript,
|
||||
});
|
||||
if (!published) process.exitCode = 1;
|
||||
}
|
||||
@@ -108,7 +108,11 @@ export function classifyFailure({
|
||||
}
|
||||
|
||||
export function diagnosticPaths(item) {
|
||||
return item.kind === "jscpd" ? item.paths : [item.path];
|
||||
return item.kind === "jscpd"
|
||||
? item.paths
|
||||
: item.path === undefined
|
||||
? []
|
||||
: [item.path];
|
||||
}
|
||||
export function isFailedConclusion(value) {
|
||||
return value === "failure" || value === "timed_out";
|
||||
@@ -269,16 +273,15 @@ function normalizeLint(report) {
|
||||
.filter((item) => item.severity === "error")
|
||||
.map((item) => {
|
||||
if (
|
||||
typeof item.filename !== "string" ||
|
||||
!repoPath(item.filename) ||
|
||||
typeof item.code !== "string" ||
|
||||
(item.filename !== undefined && !repoPath(item.filename)) ||
|
||||
(item.code !== undefined && typeof item.code !== "string") ||
|
||||
typeof item.message !== "string"
|
||||
)
|
||||
throw new Error("oxlint error diagnostic is malformed");
|
||||
return {
|
||||
kind: "lint",
|
||||
path: item.filename,
|
||||
code: item.code,
|
||||
...(item.filename === undefined ? {} : { path: item.filename }),
|
||||
...(item.code === undefined ? {} : { code: item.code }),
|
||||
message: item.message,
|
||||
severity: "error",
|
||||
};
|
||||
|
||||
@@ -15,7 +15,7 @@ const lintReport = z.object({
|
||||
message: z.string(),
|
||||
code: z.string().optional(),
|
||||
severity: z.enum(["error", "warning", "advice"]),
|
||||
filename: z.string(),
|
||||
filename: z.string().optional(),
|
||||
labels: z.array(
|
||||
z.object({
|
||||
span: z.object({ line: z.number(), column: z.number() }),
|
||||
@@ -115,8 +115,12 @@ try {
|
||||
);
|
||||
for (const diagnostic of errors) {
|
||||
const location = diagnostic.labels[0]?.span;
|
||||
const source =
|
||||
diagnostic.filename === undefined
|
||||
? ""
|
||||
: `${diagnostic.filename}${location === undefined ? "" : `:${location.line}:${location.column}`}: `;
|
||||
console.error(
|
||||
`${diagnostic.filename}:${location?.line ?? 1}:${location?.column ?? 1}: ${diagnostic.code ?? "lint"}: ${diagnostic.message}`,
|
||||
`${source}${diagnostic.code ?? "lint"}: ${diagnostic.message}`,
|
||||
);
|
||||
}
|
||||
const counts = new Map();
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { requireMcpOutput } from "./mcp-verifier-results.mjs";
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { fileURLToPath } from "node:url";
|
||||
@@ -116,9 +117,12 @@ export async function verifyBrowserNetworkEvidence(
|
||||
{ timeout: 60_000 },
|
||||
);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
const evidence = parseEvidence(result.structuredContent);
|
||||
assert.notEqual(result.structuredContent, undefined);
|
||||
const evidence = parseEvidence(
|
||||
requireMcpOutput(result, "capture_browser_scenario"),
|
||||
);
|
||||
mcpProof = assertBrowserNetworkEvidence(
|
||||
browserScenarioCaptureSchema.parse(evidence.normalized_result),
|
||||
evidence.normalized_result,
|
||||
site.origin,
|
||||
);
|
||||
if (attachment !== undefined) {
|
||||
@@ -131,12 +135,13 @@ export async function verifyBrowserNetworkEvidence(
|
||||
{ timeout: 60_000 },
|
||||
);
|
||||
assert.notEqual(connected.isError, true, JSON.stringify(connected));
|
||||
const connectedEvidence = parseEvidence(connected.structuredContent);
|
||||
assert.notEqual(connected.structuredContent, undefined);
|
||||
const connectedEvidence = parseEvidence(
|
||||
requireMcpOutput(connected, "capture_browser_scenario"),
|
||||
);
|
||||
assert.deepEqual(
|
||||
assertBrowserNetworkEvidence(
|
||||
browserScenarioCaptureSchema.parse(
|
||||
connectedEvidence.normalized_result,
|
||||
),
|
||||
connectedEvidence.normalized_result,
|
||||
site.origin,
|
||||
"disconnected-external",
|
||||
),
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { requireMcpOutput } from "./mcp-verifier-results.mjs";
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { once } from "node:events";
|
||||
@@ -163,11 +164,11 @@ const verifyTransports = async (executable, origin) => {
|
||||
{ timeout: 60_000 },
|
||||
);
|
||||
assert.notEqual(captured.isError, true, JSON.stringify(captured));
|
||||
const evidence = parseEvidence(captured.structuredContent);
|
||||
assertNetwork(
|
||||
browserScenarioCaptureSchema.parse(evidence.normalized_result),
|
||||
origin,
|
||||
assert.notEqual(captured.structuredContent, undefined);
|
||||
const evidence = parseEvidence(
|
||||
requireMcpOutput(captured, "capture_browser_scenario"),
|
||||
);
|
||||
assertNetwork(evidence.normalized_result, origin);
|
||||
} finally {
|
||||
try {
|
||||
await client.close();
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
import { requireMcpToolError } from "./mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpOutput,
|
||||
requireMcpToolError,
|
||||
} from "./mcp-verifier-results.mjs";
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
@@ -95,10 +98,11 @@ export async function verifyLargeScreenshotE2e(endpoint, origin) {
|
||||
{ timeout: 60000 },
|
||||
);
|
||||
assert.notEqual(captured.isError, true, JSON.stringify(captured));
|
||||
const mcpEvidence = parseEvidence(captured.structuredContent);
|
||||
const mcpScreenshot = webScreenshotSchema.parse(
|
||||
mcpEvidence.normalized_result,
|
||||
assert.notEqual(captured.structuredContent, undefined);
|
||||
const mcpEvidence = parseEvidence(
|
||||
requireMcpOutput(captured, "capture_web_screenshot"),
|
||||
);
|
||||
const mcpScreenshot = mcpEvidence.normalized_result;
|
||||
assert.deepEqual(mcpScreenshot.artifact, screenshot.artifact);
|
||||
const compared = comparePngScreenshots({
|
||||
before: screenshot.artifact,
|
||||
|
||||
@@ -38,7 +38,16 @@ export const requireMcpOutput = (
|
||||
throw new Error(`${operation} failed: ${mcpTextValue(result)}`);
|
||||
if (contract.name !== operation)
|
||||
throw new Error(`${operation} received the ${contract.name} contract`);
|
||||
return contract.outputSchema.parse(jsonValue(result));
|
||||
const text = jsonValue(result);
|
||||
const output = contract.outputSchema.parse(text);
|
||||
if (result.structuredContent !== undefined) {
|
||||
// The producer emits both representations from the same JSON value.
|
||||
// Compare before schema projection can discard disagreeing source fields.
|
||||
if (!isDeepStrictEqual(text, result.structuredContent))
|
||||
throw new Error(`${operation} text and structured results disagree`);
|
||||
}
|
||||
// Downstream parseEvidence must authenticate source facts, not a schema projection.
|
||||
return Object.hasOwn(output, "normalized_result") ? text : output;
|
||||
};
|
||||
|
||||
const operationResultKey = (contract) => {
|
||||
@@ -124,15 +133,6 @@ export const requireMcpEvidenceBundle = (result) => {
|
||||
throw new Error(
|
||||
"get_evidence_bundle returned summary metadata instead of complete Evidence",
|
||||
);
|
||||
if (result.structuredContent !== undefined) {
|
||||
const structured = contract.outputSchema.parse(
|
||||
result.structuredContent,
|
||||
).result;
|
||||
if (!isDeepStrictEqual(bundle, structured))
|
||||
throw new Error(
|
||||
"get_evidence_bundle text and structured results disagree",
|
||||
);
|
||||
}
|
||||
return authenticateParsedEvidenceBundle(bundle);
|
||||
};
|
||||
|
||||
|
||||
@@ -14,6 +14,7 @@ import { nativeValueTraceSchema } from "../../dist/domain/native/nativeValueTrac
|
||||
import {
|
||||
requireMcpEvidenceBundle,
|
||||
requireMcpLifecycleResult,
|
||||
requireMcpOutput,
|
||||
} from "./mcp-verifier-results.mjs";
|
||||
import { verifyGhidraBoundaries } from "./real-ghidra-boundaries.mjs";
|
||||
|
||||
@@ -198,12 +199,15 @@ async function verifyNativeValueAdapters(
|
||||
{ timeout: 180000 },
|
||||
);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
const mcpEvidence = parseEvidence(result.structuredContent);
|
||||
assert.notEqual(result.structuredContent, undefined);
|
||||
const mcpEvidence = parseEvidence(
|
||||
requireMcpOutput(result, "trace_native_values"),
|
||||
);
|
||||
assert.equal(mcpEvidence.operation, "trace_native_values");
|
||||
assert.equal(mcpEvidence.provider.id, "rea-workflow");
|
||||
assert.deepEqual(mcpEvidence.analysis_profile, workflowProfile);
|
||||
const mcpGraph = verifyGraph(
|
||||
result.structuredContent?.normalized_result,
|
||||
mcpEvidence.normalized_result,
|
||||
target,
|
||||
procedure,
|
||||
globalAddress,
|
||||
@@ -218,7 +222,10 @@ async function verifyNativeValueAdapters(
|
||||
{ timeout: 180000 },
|
||||
);
|
||||
assert.notEqual(analyzed.isError, true, JSON.stringify(analyzed));
|
||||
const functionEvidence = parseEvidence(analyzed.structuredContent);
|
||||
assert.notEqual(analyzed.structuredContent, undefined);
|
||||
const functionEvidence = parseEvidence(
|
||||
requireMcpOutput(analyzed, "analyze_function"),
|
||||
);
|
||||
assert.equal(functionEvidence.operation, "analyze_function");
|
||||
assert.equal(functionEvidence.provider.id, "ghidra");
|
||||
assert.deepEqual(
|
||||
@@ -246,10 +253,6 @@ async function verifyNativeValueAdapters(
|
||||
functionEvidence.analysis_profile,
|
||||
);
|
||||
assert.deepEqual(functionProfile, upstreamProfile);
|
||||
assert.deepEqual(
|
||||
functionEvidence.normalized_result,
|
||||
analyzed.structuredContent?.normalized_result,
|
||||
);
|
||||
const comparison = await client.callTool(
|
||||
{
|
||||
name: "compare_functions",
|
||||
@@ -261,16 +264,15 @@ async function verifyNativeValueAdapters(
|
||||
{ timeout: 180000 },
|
||||
);
|
||||
assert.notEqual(comparison.isError, true, JSON.stringify(comparison));
|
||||
const comparisonEvidence = parseEvidence(comparison.structuredContent);
|
||||
assert.notEqual(comparison.structuredContent, undefined);
|
||||
const comparisonEvidence = parseEvidence(
|
||||
requireMcpOutput(comparison, "compare_functions"),
|
||||
);
|
||||
assert.equal(
|
||||
comparisonEvidence.normalized_result.status,
|
||||
"unchanged",
|
||||
JSON.stringify(comparisonEvidence.normalized_result.dimensions),
|
||||
);
|
||||
assert.deepEqual(
|
||||
comparisonEvidence.normalized_result,
|
||||
comparison.structuredContent?.normalized_result,
|
||||
);
|
||||
assert.ok(
|
||||
comparisonEvidence.evidence_links.includes(functionEvidence.evidence_id),
|
||||
);
|
||||
@@ -346,8 +348,9 @@ async function verifyUnresolvedCallEvidence(client) {
|
||||
arguments: {},
|
||||
});
|
||||
assert.notEqual(inventory.isError, true, JSON.stringify(inventory));
|
||||
assert.notEqual(inventory.structuredContent, undefined);
|
||||
const procedures = parseEvidence(
|
||||
inventory.structuredContent,
|
||||
requireMcpOutput(inventory, "list_procedures"),
|
||||
).normalized_result;
|
||||
const indirect = procedures.find((item) =>
|
||||
item.value.endsWith("rea_ghidra_inventory_indirect"),
|
||||
@@ -367,7 +370,10 @@ async function verifyUnresolvedCallEvidence(client) {
|
||||
{ timeout: 180000 },
|
||||
);
|
||||
assert.notEqual(analyzed.isError, true, JSON.stringify(analyzed));
|
||||
const evidence = parseEvidence(analyzed.structuredContent);
|
||||
assert.notEqual(analyzed.structuredContent, undefined);
|
||||
const evidence = parseEvidence(
|
||||
requireMcpOutput(analyzed, "analyze_function"),
|
||||
);
|
||||
assert.equal(evidence.relationship_coverage?.status, "partial");
|
||||
assert.ok(evidence.normalized_result.unresolved_calls.length > 0);
|
||||
const traced = await client.callTool(
|
||||
@@ -382,7 +388,10 @@ async function verifyUnresolvedCallEvidence(client) {
|
||||
{ timeout: 180000 },
|
||||
);
|
||||
assert.notEqual(traced.isError, true, JSON.stringify(traced));
|
||||
const trace = parseEvidence(traced.structuredContent).normalized_result;
|
||||
assert.notEqual(traced.structuredContent, undefined);
|
||||
const trace = parseEvidence(
|
||||
requireMcpOutput(traced, "trace_call_path"),
|
||||
).normalized_result;
|
||||
assert.equal(trace.goal_status, "unknown");
|
||||
assert.equal(trace.truncated, true);
|
||||
assert.ok(trace.residual_unknowns.length > 0);
|
||||
@@ -397,8 +406,10 @@ async function verifyUnresolvedCallEvidence(client) {
|
||||
{ timeout: 180000 },
|
||||
);
|
||||
assert.notEqual(compared.isError, true, JSON.stringify(compared));
|
||||
assert.notEqual(compared.structuredContent, undefined);
|
||||
assert.equal(
|
||||
parseEvidence(compared.structuredContent).normalized_result.status,
|
||||
parseEvidence(requireMcpOutput(compared, "compare_functions"))
|
||||
.normalized_result.status,
|
||||
"unknown",
|
||||
);
|
||||
}
|
||||
@@ -26,7 +26,10 @@ import {
|
||||
locateApkSigningBlock,
|
||||
parseZipTail,
|
||||
} from "../dist/domain/apkSignature/apkSigningBlock.js";
|
||||
import { mcpTextValue } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
mcpTextValue,
|
||||
requireMcpOutput,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
|
||||
|
||||
const execute = promisify(execFile);
|
||||
@@ -261,7 +264,7 @@ try {
|
||||
ajv.validate(advertised.outputSchema, reply.structuredContent),
|
||||
JSON.stringify(ajv.errors),
|
||||
);
|
||||
const mcp = parseEvidence(reply.structuredContent);
|
||||
const mcp = parseEvidence(requireMcpOutput(reply, "inspect_apk_signature"));
|
||||
assert.deepEqual(cli, mcp, "CLI and MCP evidence differ");
|
||||
cases += 2;
|
||||
return mcp.normalized_result;
|
||||
|
||||
@@ -26,7 +26,7 @@ import {
|
||||
} from "./lib/http3-verifier-lifecycle.mjs";
|
||||
import {
|
||||
requireMcpEvidenceBundle,
|
||||
mcpTextValue,
|
||||
requireMcpOutput,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
|
||||
const root = process.argv[2];
|
||||
@@ -137,14 +137,11 @@ async function mcp(input, name) {
|
||||
join(directory, `${name}.mcp.reply.json`),
|
||||
JSON.stringify(reply, null, 2),
|
||||
);
|
||||
const value = JSON.parse(mcpTextValue(reply));
|
||||
assert.notEqual(reply.isError, true, mcpTextValue(reply));
|
||||
const evidence = requireMcpOutput(reply, contract.name, contract);
|
||||
assert.ok(
|
||||
validateOutput(reply.structuredContent),
|
||||
JSON.stringify(validateOutput.errors),
|
||||
);
|
||||
const evidence = contract.outputSchema.parse(value);
|
||||
assert.deepEqual(reply.structuredContent, value);
|
||||
retainedEvidence.push(evidence);
|
||||
return evidence;
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import {
|
||||
requireMcpEvidenceBundle,
|
||||
requireMcpLifecycleResult,
|
||||
requireMcpOutput,
|
||||
requireMcpToolError,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import assert from "node:assert/strict";
|
||||
@@ -172,31 +173,23 @@ const verifyMcpInlineArtifactEvidence = async (
|
||||
{ name: "inspect_artifact", arguments: {} },
|
||||
mcpOptions,
|
||||
);
|
||||
assert.notEqual(inspected.isError, true, JSON.stringify(inspected));
|
||||
const source = parseEvidence(inspected.structuredContent);
|
||||
assert.deepEqual(
|
||||
source.normalized_result,
|
||||
inspected.structuredContent?.normalized_result,
|
||||
);
|
||||
assert.deepEqual(
|
||||
JSON.parse(prompts.mcpText(inspected)),
|
||||
inspected.structuredContent,
|
||||
assert.notEqual(inspected.structuredContent, undefined);
|
||||
const source = parseEvidence(
|
||||
requireMcpOutput(inspected, "inspect_artifact"),
|
||||
);
|
||||
const compared = await client.callTool(
|
||||
{
|
||||
name: "compare_artifacts",
|
||||
arguments: {
|
||||
left: inspected.structuredContent,
|
||||
right: inspected.structuredContent,
|
||||
left: source,
|
||||
right: source,
|
||||
},
|
||||
},
|
||||
mcpOptions,
|
||||
);
|
||||
assert.notEqual(compared.isError, true, JSON.stringify(compared));
|
||||
const comparison = parseEvidence(compared.structuredContent);
|
||||
assert.deepEqual(
|
||||
comparison.normalized_result,
|
||||
compared.structuredContent?.normalized_result,
|
||||
assert.notEqual(compared.structuredContent, undefined);
|
||||
const comparison = parseEvidence(
|
||||
requireMcpOutput(compared, "compare_artifacts"),
|
||||
);
|
||||
assert.ok(comparison.evidence_links.includes(source.evidence_id));
|
||||
const bundleReply = await client.callTool(
|
||||
@@ -220,13 +213,11 @@ const verifyMcpInlineArtifactEvidence = async (
|
||||
{ name: "extract_artifact", arguments: {} },
|
||||
mcpOptions,
|
||||
);
|
||||
assert.notEqual(extracted.isError, true, JSON.stringify(extracted));
|
||||
const extraction = parseEvidence(extracted.structuredContent);
|
||||
assert.deepEqual(
|
||||
extraction.normalized_result,
|
||||
extracted.structuredContent?.normalized_result,
|
||||
assert.notEqual(extracted.structuredContent, undefined);
|
||||
const extraction = parseEvidence(
|
||||
requireMcpOutput(extracted, "extract_artifact"),
|
||||
);
|
||||
const materialized = extracted.structuredContent?.normalized_result;
|
||||
const materialized = extraction.normalized_result;
|
||||
assert.equal(typeof materialized?.output_root, "string");
|
||||
outputRoot = materialized.output_root;
|
||||
assert.equal(materialized.containment_verified, true);
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from "node:assert/strict";
|
||||
import { requireMcpEvidenceBundle } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpEvidenceBundle,
|
||||
requireMcpOutput,
|
||||
requireMcpToolError,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { estimateTokenCount } from "tokenx";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
@@ -124,12 +128,13 @@ await withVerifierCleanup(async () => {
|
||||
name: tool.name,
|
||||
arguments: input,
|
||||
});
|
||||
assert.notEqual(response.isError, true, JSON.stringify(response));
|
||||
assert.ok(
|
||||
ajv.validate(tool.outputSchema, response.structuredContent),
|
||||
JSON.stringify(ajv.errors),
|
||||
);
|
||||
const evidence = parseEvidence(response.structuredContent);
|
||||
const evidence = parseEvidence(
|
||||
requireMcpOutput(response, "inspect_packet_capture"),
|
||||
);
|
||||
const report = evidence.normalized_result;
|
||||
assert.equal(
|
||||
report.artifact.sha256,
|
||||
@@ -305,12 +310,13 @@ await withVerifierCleanup(async () => {
|
||||
name: pcapTool.name,
|
||||
arguments: { path: capturePath, detail: "summary" },
|
||||
});
|
||||
assert.notEqual(response.isError, true, JSON.stringify(response));
|
||||
assert.ok(
|
||||
pcapAjv.validate(pcapTool.outputSchema, response.structuredContent),
|
||||
pcapAjv.errorsText(),
|
||||
);
|
||||
const summary = parseEvidence(response.structuredContent).normalized_result;
|
||||
const summary = parseEvidence(
|
||||
requireMcpOutput(response, "inspect_pcap_capture"),
|
||||
).normalized_result;
|
||||
assert.equal(summary.summary.capture_start, expectedCaptureStart);
|
||||
assert.equal(summary.summary.capture_end, expectedCaptureEnd);
|
||||
assert.equal(
|
||||
@@ -343,10 +349,9 @@ await withVerifierCleanup(async () => {
|
||||
view: { kind: "pcap", facet: "protocols", limit: 10 },
|
||||
},
|
||||
});
|
||||
assert.notEqual(page.isError, true, JSON.stringify(page));
|
||||
assert.equal(
|
||||
parseEvidence(page.structuredContent).normalized_result
|
||||
.parent_evidence_id,
|
||||
parseEvidence(requireMcpOutput(page, "inspect_analysis_view"))
|
||||
.normalized_result.parent_evidence_id,
|
||||
source.evidence_id,
|
||||
);
|
||||
cases++;
|
||||
@@ -368,19 +373,16 @@ await withVerifierCleanup(async () => {
|
||||
name: tool.name,
|
||||
arguments: input,
|
||||
});
|
||||
assert.equal(response.isError, true);
|
||||
requireMcpToolError(response);
|
||||
await client.ping();
|
||||
cases++;
|
||||
}
|
||||
await writeFile(path, "not a capture");
|
||||
assert.equal(
|
||||
(
|
||||
await client.callTool({
|
||||
name: tool.name,
|
||||
arguments: { path, fields: ["frame.number"] },
|
||||
})
|
||||
).isError,
|
||||
true,
|
||||
requireMcpToolError(
|
||||
await client.callTool({
|
||||
name: tool.name,
|
||||
arguments: { path, fields: ["frame.number"] },
|
||||
}),
|
||||
);
|
||||
cases++;
|
||||
const discovery = (await client.listTools()).tools.find(
|
||||
@@ -414,12 +416,13 @@ await withVerifierCleanup(async () => {
|
||||
name: discovery.name,
|
||||
arguments: input,
|
||||
});
|
||||
assert.notEqual(response.isError, true, JSON.stringify(response));
|
||||
assert.ok(
|
||||
ajv.validate(discovery.outputSchema, response.structuredContent),
|
||||
JSON.stringify(ajv.errors),
|
||||
);
|
||||
const evidence = parseEvidence(response.structuredContent);
|
||||
const evidence = parseEvidence(
|
||||
requireMcpOutput(response, "inspect_packet_fields"),
|
||||
);
|
||||
const report = evidence.normalized_result;
|
||||
const kind = input.kind ?? "protocols";
|
||||
const query = (input.query ?? "").toLowerCase();
|
||||
|
||||
@@ -10,6 +10,11 @@ import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { Ajv2020 } from "ajv/dist/2020.js";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { toolContract } from "../dist/contracts/toolContracts.js";
|
||||
import {
|
||||
requireMcpOutput,
|
||||
requireMcpToolError,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
|
||||
// Caller-supplied producer evidence stays local; no fixture capture or secrets are bundled.
|
||||
const [controlPath, lossPath, reportPath] = process.argv.slice(2);
|
||||
@@ -40,6 +45,7 @@ transport.stderr?.on("data", (bytes) => {
|
||||
});
|
||||
const ajv = new Ajv2020({ strict: false, validateFormats: false });
|
||||
const measurements = [];
|
||||
const contract = toolContract("inspect_web_network_capture");
|
||||
try {
|
||||
try {
|
||||
await client.connect(transport);
|
||||
@@ -68,12 +74,13 @@ try {
|
||||
name: tool.name,
|
||||
arguments: input,
|
||||
});
|
||||
assert.notEqual(response.isError, true);
|
||||
const mcp = parseEvidence(
|
||||
requireMcpOutput(response, contract.name, contract),
|
||||
);
|
||||
assert.ok(
|
||||
validateOutput(response.structuredContent),
|
||||
JSON.stringify(validateOutput.errors),
|
||||
);
|
||||
const mcp = parseEvidence(response.structuredContent);
|
||||
const completed = await promisify(execFile)(
|
||||
process.execPath,
|
||||
[entrypoint, "inspect-web-network-capture", path, "qlog", "--json"],
|
||||
@@ -153,17 +160,14 @@ try {
|
||||
view: { kind: "api-surface" },
|
||||
},
|
||||
});
|
||||
assert.equal(retained.isError, true);
|
||||
assert.equal(
|
||||
JSON.parse(retained.content.find((item) => item.type === "text").text)
|
||||
.error.code,
|
||||
"unsupported_target",
|
||||
);
|
||||
assert.equal(requireMcpToolError(retained).code, "unsupported_target");
|
||||
const selected = await client.callTool({
|
||||
name: tool.name,
|
||||
arguments: { ...input, record_ordinals: [capture.total_records - 1, 0] },
|
||||
});
|
||||
const subset = parseEvidence(selected.structuredContent).normalized_result;
|
||||
const subset = parseEvidence(
|
||||
requireMcpOutput(selected, contract.name, contract),
|
||||
).normalized_result;
|
||||
assert.deepEqual(
|
||||
subset.records.map((record) => record.ordinal),
|
||||
[capture.total_records - 1, 0],
|
||||
|
||||
@@ -18,7 +18,10 @@ import { promisify } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpOperationResult,
|
||||
requireMcpOutput,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
|
||||
|
||||
if (process.argv.length !== 2)
|
||||
@@ -464,10 +467,9 @@ async function call(name, args = {}) {
|
||||
{ name, arguments: args },
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
if (["open_binary", "binary_session", "close_binary"].includes(name))
|
||||
return requireMcpOperationResult(result, name);
|
||||
return result.structuredContent;
|
||||
return requireMcpOutput(result, name);
|
||||
}
|
||||
|
||||
function evidence(value, sha256) {
|
||||
|
||||
@@ -9,7 +9,10 @@ import { promisify } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpOperationResult,
|
||||
requireMcpOutput,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
|
||||
|
||||
if (process.argv.length !== 2 || process.platform !== "linux")
|
||||
@@ -320,8 +323,7 @@ async function call(name, args = {}) {
|
||||
{ name, arguments: args },
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
return ["open_binary", "binary_session", "close_binary"].includes(name)
|
||||
? requireMcpOperationResult(result, name)
|
||||
: result.structuredContent;
|
||||
: requireMcpOutput(result, name);
|
||||
}
|
||||
@@ -1,6 +1,9 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from "node:assert/strict";
|
||||
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpLifecycleResult,
|
||||
requireMcpOutput,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
@@ -339,16 +342,11 @@ async function call(name, args = {}) {
|
||||
{ name, arguments: args },
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
assert.notEqual(value.isError, true, JSON.stringify(value));
|
||||
const projected = requireMcpOperationResult(value, name);
|
||||
if (name === "open_binary" || name === "close_binary") return projected;
|
||||
const evidence = parseEvidence(value.structuredContent);
|
||||
if (["open_binary", "close_binary"].includes(name))
|
||||
return requireMcpLifecycleResult(value, name);
|
||||
const evidence = parseEvidence(requireMcpOutput(value, name));
|
||||
assert.equal(evidence.subject.digest.sha256, sha256);
|
||||
assert.deepEqual(
|
||||
evidence.normalized_result,
|
||||
value.structuredContent.normalized_result,
|
||||
);
|
||||
return projected;
|
||||
return evidence.normalized_result;
|
||||
}
|
||||
async function cli(command, address, options = []) {
|
||||
const { stdout } = await promisify(execFile)(
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from "node:assert/strict";
|
||||
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpOperationResult,
|
||||
requireMcpOutput,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
@@ -555,7 +558,6 @@ async function call(name, arguments_ = {}) {
|
||||
{ name, arguments: arguments_ },
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
assert.ok(result.structuredContent, `${name} omitted structured result`);
|
||||
return requireMcpOperationResult(result, name);
|
||||
}
|
||||
@@ -564,8 +566,7 @@ async function evidenceCall(name, arguments_) {
|
||||
{ name, arguments: arguments_ },
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
const evidence = parseEvidence(result.structuredContent);
|
||||
const evidence = parseEvidence(requireMcpOutput(result, name));
|
||||
assert.equal(evidence.subject.digest.sha256, sha256);
|
||||
assert.equal(evidence.provider.id, "ghidra");
|
||||
assertProfile(evidence.analysis_profile);
|
||||
|
||||
@@ -16,7 +16,10 @@ import { promisify } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpOperationResult,
|
||||
requireMcpOutput,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
|
||||
|
||||
if (process.argv.length !== 2)
|
||||
@@ -281,10 +284,9 @@ async function call(name, args = {}) {
|
||||
{ name, arguments: args },
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
if (["open_binary", "binary_session", "close_binary"].includes(name))
|
||||
return requireMcpOperationResult(result, name);
|
||||
return result.structuredContent;
|
||||
return requireMcpOutput(result, name);
|
||||
}
|
||||
|
||||
function evidence(value) {
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from "node:assert/strict";
|
||||
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpLifecycleResult,
|
||||
requireMcpOutput,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
@@ -394,14 +397,8 @@ async function call(name, args = {}) {
|
||||
{ name, arguments: args },
|
||||
{ timeout: 360000 },
|
||||
);
|
||||
assert.notEqual(response.isError, true, JSON.stringify(response));
|
||||
const projected = requireMcpOperationResult(response, name);
|
||||
if (name === "close_binary") return projected;
|
||||
const evidence = parseEvidence(response.structuredContent);
|
||||
if (name === "close_binary") return requireMcpLifecycleResult(response, name);
|
||||
const evidence = parseEvidence(requireMcpOutput(response, name));
|
||||
assert.equal(evidence.subject.digest.sha256, sha256);
|
||||
assert.deepEqual(
|
||||
evidence.normalized_result,
|
||||
response.structuredContent.normalized_result,
|
||||
);
|
||||
return projected;
|
||||
return evidence.normalized_result;
|
||||
}
|
||||
@@ -12,7 +12,10 @@ import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { inspectVerifierGhidraInstallation } from "./lib/ghidra-installation-inspection.mjs";
|
||||
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpLifecycleResult,
|
||||
requireMcpOutput,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
|
||||
|
||||
if (process.argv.length !== 2)
|
||||
@@ -351,15 +354,9 @@ async function call(name, args = {}, targetSha256 = sha256) {
|
||||
{ name, arguments: args },
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
const projected = requireMcpOperationResult(result, name);
|
||||
if (
|
||||
name === "open_binary" ||
|
||||
name === "close_binary" ||
|
||||
name === "binary_session"
|
||||
)
|
||||
return projected;
|
||||
const evidence = parseEvidence(result.structuredContent);
|
||||
if (["open_binary", "close_binary", "binary_session"].includes(name))
|
||||
return requireMcpLifecycleResult(result, name);
|
||||
const evidence = parseEvidence(requireMcpOutput(result, name));
|
||||
assert.equal(evidence.subject.digest.sha256, targetSha256);
|
||||
assert.equal(evidence.subject.format, "raw-binary");
|
||||
return evidence;
|
||||
|
||||
@@ -26,7 +26,8 @@ import { inspectVerifierGhidraInstallation } from "./lib/ghidra-installation-ins
|
||||
import { buildRiscvFixture } from "../tests/conformance/ghidra/riscv-fixture.mjs";
|
||||
import {
|
||||
mcpTextValue,
|
||||
requireMcpOperationResult,
|
||||
requireMcpLifecycleResult,
|
||||
requireMcpOutput,
|
||||
requireMcpToolError,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
|
||||
@@ -141,18 +142,24 @@ try {
|
||||
{ name, arguments: args },
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
const projected = requireMcpOperationResult(reply, name);
|
||||
const lifecycle = [
|
||||
"open_binary",
|
||||
"close_binary",
|
||||
"binary_session",
|
||||
].includes(name);
|
||||
const admitted = lifecycle
|
||||
? requireMcpLifecycleResult(reply, name)
|
||||
: parseEvidence(requireMcpOutput(reply, name));
|
||||
const textual = JSON.parse(mcpTextValue(reply));
|
||||
assert.deepEqual(reply.structuredContent, textual);
|
||||
assert.ok(reply.structuredContent, `${name} omitted structured result`);
|
||||
const validate = validators.get(name);
|
||||
assert.ok(validate, `${name} missing from advertised catalog`);
|
||||
assert.ok(
|
||||
validate(textual),
|
||||
`${name}: ${JSON.stringify(validate.errors)}`,
|
||||
);
|
||||
if (!["open_binary", "close_binary", "binary_session"].includes(name))
|
||||
assertEvidence(parseEvidence(textual));
|
||||
return projected;
|
||||
if (!lifecycle) assertEvidence(admitted);
|
||||
return lifecycle ? admitted : admitted.normalized_result;
|
||||
};
|
||||
const opened = await call("open_binary", {
|
||||
path: fixture.path,
|
||||
|
||||
@@ -21,7 +21,6 @@ import {
|
||||
ProviderProcessSupervisor,
|
||||
spawnOwnedProviderProcess,
|
||||
} from "../dist/process/ProviderProcess.js";
|
||||
import { toolContract } from "../dist/contracts/toolContracts.js";
|
||||
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
|
||||
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
|
||||
|
||||
@@ -215,7 +214,7 @@ async function call(name, arguments_) {
|
||||
{ timeout: 180_000 },
|
||||
);
|
||||
if (name === "analyze_swift_types")
|
||||
toolContract(name).outputSchema.parse(response.structuredContent);
|
||||
assert.ok(response.structuredContent, `${name} omitted structured result`);
|
||||
return requireMcpOperationResult(response, name);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from "node:assert/strict";
|
||||
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpOperationResult,
|
||||
requireMcpOutput,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
@@ -326,9 +329,8 @@ async function toolResult(client, name, arguments_ = {}) {
|
||||
{ name, arguments: arguments_ },
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
assert.ok(result.structuredContent, `${name} omitted structured result`);
|
||||
return result.structuredContent;
|
||||
return requireMcpOutput(result, name);
|
||||
}
|
||||
async function call(client, name, arguments_ = {}) {
|
||||
const result = await client.callTool(
|
||||
|
||||
@@ -8,7 +8,7 @@ import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { tsImport } from "tsx/esm/api";
|
||||
|
||||
import { toolContract } from "../dist/contracts/toolContracts.js";
|
||||
import { requireMcpOutput } from "./lib/mcp-verifier-results.mjs";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { createEvidenceBundle } from "../dist/domain/evidenceBundle.js";
|
||||
import { withVerifierCleanup } from "./lib/verifier-cleanup.mjs";
|
||||
@@ -80,10 +80,8 @@ await withVerifierCleanup(async () => {
|
||||
{ name: "binary_session", arguments: {} },
|
||||
{ toolDefinition: sessionTool },
|
||||
);
|
||||
assert.notEqual(sessionReply.isError, true);
|
||||
const sessionStatus = toolContract("binary_session").outputSchema.parse(
|
||||
sessionReply.structuredContent,
|
||||
).result;
|
||||
assert.notEqual(sessionReply.structuredContent, undefined);
|
||||
const sessionStatus = requireMcpOutput(sessionReply, "binary_session").result;
|
||||
const javascriptMcp = await analyzeMcp(client, javascriptPath);
|
||||
const electronMcp = await analyzeMcp(client, electronPath);
|
||||
assert.equal(
|
||||
@@ -194,11 +192,10 @@ async function analyzeMcp(client_, inputPath) {
|
||||
name: "analyze_javascript_application",
|
||||
arguments: { input_path: inputPath },
|
||||
});
|
||||
assert.notEqual(response.isError, true);
|
||||
toolContract("analyze_javascript_application").outputSchema.parse(
|
||||
response.structuredContent,
|
||||
assert.notEqual(response.structuredContent, undefined);
|
||||
return parseEvidence(
|
||||
requireMcpOutput(response, "analyze_javascript_application"),
|
||||
);
|
||||
return parseEvidence(response.structuredContent);
|
||||
}
|
||||
|
||||
function verifySyntheticEvaluator() {
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from "node:assert/strict";
|
||||
import { requireMcpEvidenceBundle } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
requireMcpEvidenceBundle,
|
||||
requireMcpOutput,
|
||||
requireMcpToolError,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { estimateTokenCount } from "tokenx";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
@@ -47,9 +51,7 @@ const assertRejectedSelector = (
|
||||
field,
|
||||
expectedExitCode = 1,
|
||||
) => {
|
||||
assert.equal(response.isError, true);
|
||||
const text = response.content.find((item) => item.type === "text")?.text;
|
||||
const { error } = JSON.parse(text ?? "{}");
|
||||
const error = requireMcpToolError(response);
|
||||
assert.equal(error.code, "invalid_request");
|
||||
assert.equal(error.details.operation, operation);
|
||||
assert.deepEqual(
|
||||
@@ -123,12 +125,13 @@ await withVerifierCleanup(async () => {
|
||||
name: advertised.name,
|
||||
arguments: input,
|
||||
});
|
||||
assert.notEqual(response.isError, true, JSON.stringify(response));
|
||||
assert.ok(
|
||||
ajv.validate(advertised.outputSchema, response.structuredContent),
|
||||
JSON.stringify(ajv.errors),
|
||||
);
|
||||
const evidence = parseEvidence(response.structuredContent);
|
||||
const evidence = parseEvidence(
|
||||
requireMcpOutput(response, "decode_signal_recording"),
|
||||
);
|
||||
const report = evidence.normalized_result;
|
||||
for (const byte of ["52", "45", "41"])
|
||||
assert.match(
|
||||
@@ -238,12 +241,13 @@ await withVerifierCleanup(async () => {
|
||||
name: advertised.name,
|
||||
arguments: input,
|
||||
});
|
||||
assert.notEqual(response.isError, true, JSON.stringify(response));
|
||||
assert.ok(
|
||||
ajv.validate(advertised.outputSchema, response.structuredContent),
|
||||
JSON.stringify(ajv.errors),
|
||||
);
|
||||
const evidence = parseEvidence(response.structuredContent);
|
||||
const evidence = parseEvidence(
|
||||
requireMcpOutput(response, "decode_signal_recording"),
|
||||
);
|
||||
const report = evidence.normalized_result;
|
||||
assert.deepEqual(report.output_window, {
|
||||
total_lines: 120,
|
||||
@@ -298,9 +302,9 @@ await withVerifierCleanup(async () => {
|
||||
name: advertised.name,
|
||||
arguments: input,
|
||||
});
|
||||
assert.equal(response.isError, true);
|
||||
if (rejectedField !== undefined)
|
||||
assertRejectedSelector(response, advertised.name, rejectedField);
|
||||
else requireMcpToolError(response);
|
||||
await client.ping();
|
||||
cases++;
|
||||
}
|
||||
@@ -309,7 +313,7 @@ await withVerifierCleanup(async () => {
|
||||
name: advertised.name,
|
||||
arguments: { path, decoders: "uart" },
|
||||
});
|
||||
assert.equal(invalid.isError, true);
|
||||
requireMcpToolError(invalid);
|
||||
cases++;
|
||||
const discoveryTool = (await client.listTools()).tools.find(
|
||||
({ name }) => name === "inspect_signal_decoders",
|
||||
@@ -367,12 +371,13 @@ await withVerifierCleanup(async () => {
|
||||
name: discoveryTool.name,
|
||||
arguments: input,
|
||||
});
|
||||
assert.notEqual(response.isError, true, JSON.stringify(response));
|
||||
assert.ok(
|
||||
ajv.validate(discoveryTool.outputSchema, response.structuredContent),
|
||||
JSON.stringify(ajv.errors),
|
||||
);
|
||||
const evidence = parseEvidence(response.structuredContent);
|
||||
const evidence = parseEvidence(
|
||||
requireMcpOutput(response, "inspect_signal_decoders"),
|
||||
);
|
||||
const report = evidence.normalized_result;
|
||||
discoveryMeasurements.push({
|
||||
input,
|
||||
|
||||
@@ -21,6 +21,7 @@ import { PrivateRuntimeRoot } from "../dist/process/PrivateRuntimeRoot.js";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import {
|
||||
requireMcpEvidenceBundle,
|
||||
requireMcpOutput,
|
||||
mcpTextValue,
|
||||
requireMcpToolError,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
@@ -278,7 +279,7 @@ try {
|
||||
ajv.validate(advertised.outputSchema, reply.structuredContent),
|
||||
JSON.stringify(ajv.errors),
|
||||
);
|
||||
const mcp = parseEvidence(reply.structuredContent);
|
||||
const mcp = parseEvidence(requireMcpOutput(reply, "inspect_wasm_artifact"));
|
||||
assert.equal(cli.evidence_id, mcp.evidence_id);
|
||||
assert.deepEqual(cli, mcp);
|
||||
const report = mcp.normalized_result;
|
||||
@@ -516,7 +517,7 @@ try {
|
||||
ajv.validate(advertised.outputSchema, reply.structuredContent),
|
||||
JSON.stringify(ajv.errors),
|
||||
);
|
||||
const mcp = parseEvidence(reply.structuredContent);
|
||||
const mcp = parseEvidence(requireMcpOutput(reply, "inspect_wasm_artifact"));
|
||||
assert.deepEqual(cli, mcp);
|
||||
const report = mcp.normalized_result;
|
||||
assert.equal(report.validation, "valid");
|
||||
|
||||
@@ -7,6 +7,7 @@ import { promisify } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { requireMcpOutput } from "./lib/mcp-verifier-results.mjs";
|
||||
|
||||
const execute = promisify(execFile);
|
||||
const bin = process.env.REA_WABT_BIN_DIRECTORY;
|
||||
@@ -53,7 +54,8 @@ const inspect = async (path) => {
|
||||
arguments: { path },
|
||||
});
|
||||
assert.notEqual(result.isError, true, JSON.stringify(result));
|
||||
return parseEvidence(result.structuredContent);
|
||||
assert.notEqual(result.structuredContent, undefined);
|
||||
return parseEvidence(requireMcpOutput(result, "inspect_wasm_artifact"));
|
||||
};
|
||||
const compare = async (left, right, scope = "both") => {
|
||||
const input = {
|
||||
@@ -66,7 +68,8 @@ const compare = async (left, right, scope = "both") => {
|
||||
arguments: input,
|
||||
});
|
||||
assert.notEqual(reply.isError, true, JSON.stringify(reply));
|
||||
const mcp = parseEvidence(reply.structuredContent);
|
||||
assert.notEqual(reply.structuredContent, undefined);
|
||||
const mcp = parseEvidence(requireMcpOutput(reply, "compare_wasm_interfaces"));
|
||||
const file = join(root, "comparison.json");
|
||||
await writeFile(file, JSON.stringify({ left, right, scope }));
|
||||
const local = parseEvidence(
|
||||
|
||||
@@ -18,7 +18,11 @@ import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { PrivateRuntimeRoot } from "../dist/process/PrivateRuntimeRoot.js";
|
||||
import { runOwnedCommand } from "../dist/process/OwnedCommand.js";
|
||||
import { WEB_NETWORK_CAPTURE_LIMITS } from "../dist/domain/webNetworkCapture.js";
|
||||
import { mcpTextValue } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
mcpTextValue,
|
||||
requireMcpOutput,
|
||||
requireMcpToolError,
|
||||
} from "./lib/mcp-verifier-results.mjs";
|
||||
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
|
||||
|
||||
const executable = process.env.REA_MITMDUMP_COMMAND;
|
||||
@@ -903,15 +907,15 @@ async function inspect(mode, input, errorCategory) {
|
||||
name: "inspect_web_network_capture",
|
||||
arguments: input,
|
||||
});
|
||||
const value = JSON.parse(mcpTextValue(result));
|
||||
if (errorCategory !== undefined) {
|
||||
assert.equal(result.isError, true);
|
||||
assert.equal(value.error.category, errorCategory);
|
||||
const error = requireMcpToolError(result);
|
||||
assert.equal(error.category, errorCategory);
|
||||
const rawError = JSON.parse(mcpTextValue(result)).error;
|
||||
for (const literal of input.sensitive_values ?? [])
|
||||
assert.ok(!JSON.stringify(value.error).includes(literal));
|
||||
return value.error;
|
||||
assert.ok(!JSON.stringify(rawError).includes(literal));
|
||||
return error;
|
||||
}
|
||||
assert.notEqual(result.isError, true, mcpTextValue(result));
|
||||
const value = requireMcpOutput(result, "inspect_web_network_capture");
|
||||
assertSensitiveLimitations(value, input.sensitive_values ?? []);
|
||||
if (input.format === "har") {
|
||||
const surface = await client.callTool({
|
||||
@@ -923,13 +927,12 @@ async function inspect(mode, input, errorCategory) {
|
||||
});
|
||||
if ((input.sensitive_values ?? []).length > 0) {
|
||||
// Derived view text cannot be checked against undisclosed declarations.
|
||||
assert.equal(surface.isError, true);
|
||||
requireMcpToolError(surface);
|
||||
const refused = JSON.parse(mcpTextValue(surface));
|
||||
for (const literal of input.sensitive_values)
|
||||
assert.ok(!JSON.stringify(refused).includes(literal));
|
||||
} else {
|
||||
assert.notEqual(surface.isError, true, mcpTextValue(surface));
|
||||
const projected = JSON.parse(mcpTextValue(surface));
|
||||
const projected = requireMcpOutput(surface, "inspect_analysis_view");
|
||||
assert.equal(projected.normalized_result.kind, "api-surface");
|
||||
assert.equal(
|
||||
projected.normalized_result.records.examined,
|
||||
@@ -944,13 +947,12 @@ async function inspect(mode, input, errorCategory) {
|
||||
},
|
||||
});
|
||||
if ((input.sensitive_values ?? []).length > 0) {
|
||||
assert.equal(manifest.isError, true);
|
||||
requireMcpToolError(manifest);
|
||||
const refusedManifest = JSON.parse(mcpTextValue(manifest));
|
||||
for (const literal of input.sensitive_values)
|
||||
assert.ok(!JSON.stringify(refusedManifest).includes(literal));
|
||||
} else {
|
||||
assert.notEqual(manifest.isError, true, mcpTextValue(manifest));
|
||||
const projected = JSON.parse(mcpTextValue(manifest));
|
||||
const projected = requireMcpOutput(manifest, "inspect_analysis_view");
|
||||
assert.equal(projected.normalized_result.kind, "environment-manifest");
|
||||
assert.equal(projected.normalized_result.unknowns.length, 3);
|
||||
for (const record of projected.normalized_result.records)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { requireMcpOutput } from "../../lib/mcp-verifier-results.mjs";
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { access, writeFile } from "node:fs/promises";
|
||||
@@ -147,17 +148,14 @@ try {
|
||||
{ name: task.name, arguments: task.input },
|
||||
{ timeout: 150_000 },
|
||||
);
|
||||
assert.notEqual(response.isError, true, JSON.stringify(response));
|
||||
const structured = response.structuredContent;
|
||||
assert.ok(structured !== undefined);
|
||||
const evidence = parseEvidence(structured);
|
||||
assert.notEqual(response.structuredContent, undefined);
|
||||
const evidence = parseEvidence(requireMcpOutput(response, task.name));
|
||||
assert.deepEqual(
|
||||
evidence.normalized_result,
|
||||
cliResults.get(task.name).normalized_result,
|
||||
);
|
||||
// Raw observations retain different workspace paths and load durations.
|
||||
// Validate each evidence ID independently; compare portable results/intent.
|
||||
assert.equal(structured.evidence_id, evidence.evidence_id);
|
||||
assert.deepEqual(evidence.parameters, cliResults.get(task.name).parameters);
|
||||
assert.deepEqual(evidence.provider, cliResults.get(task.name).provider);
|
||||
assert.deepEqual(
|
||||
@@ -169,7 +167,6 @@ try {
|
||||
cliResults.get(task.name).limitations,
|
||||
);
|
||||
assert.equal(evidence.confidence, cliResults.get(task.name).confidence);
|
||||
assert.deepEqual(structured.normalized_result, evidence.normalized_result);
|
||||
console.log(`PASS MCP/CLI parity ${task.name}`);
|
||||
}
|
||||
const overloadedInput = {
|
||||
@@ -210,9 +207,10 @@ try {
|
||||
{ name: "trace_android_references", arguments: overloadedInput },
|
||||
{ timeout: 150_000 },
|
||||
);
|
||||
assert.notEqual(overloadedMcp.isError, true, JSON.stringify(overloadedMcp));
|
||||
assert.notEqual(overloadedMcp.structuredContent, undefined);
|
||||
assert.deepEqual(
|
||||
parseEvidence(overloadedMcp.structuredContent).normalized_result,
|
||||
parseEvidence(requireMcpOutput(overloadedMcp, "trace_android_references"))
|
||||
.normalized_result,
|
||||
overloadedEvidence.normalized_result,
|
||||
);
|
||||
console.log("PASS real overloaded method references through CLI/MCP");
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { parseEvidence } from "../../../dist/domain/evidence.js";
|
||||
import { requireMcpOperationResult } from "../../lib/mcp-verifier-results.mjs";
|
||||
import { requireMcpOutput } from "../../lib/mcp-verifier-results.mjs";
|
||||
|
||||
/** Compare native selection with an explicit full-inventory oracle on one live session. */
|
||||
export const verifyAndroidClassSearch = async ({ client, path, sha256 }) => {
|
||||
@@ -9,11 +9,11 @@ export const verifyAndroidClassSearch = async ({ client, path, sha256 }) => {
|
||||
{ name: "search_android_classes", arguments: { path, query } },
|
||||
{ timeout: 150_000 },
|
||||
);
|
||||
const result = requireMcpOperationResult(
|
||||
response,
|
||||
"search_android_classes",
|
||||
assert.notEqual(response.structuredContent, undefined);
|
||||
const evidence = parseEvidence(
|
||||
requireMcpOutput(response, "search_android_classes"),
|
||||
);
|
||||
const evidence = parseEvidence(response.structuredContent);
|
||||
const result = evidence.normalized_result;
|
||||
assert.equal(evidence.subject.digest.sha256, sha256);
|
||||
assert.equal(evidence.parameters.query, query);
|
||||
assert.equal(result.query, query);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { requireMcpOutput } from "../../lib/mcp-verifier-results.mjs";
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
@@ -172,8 +173,10 @@ try {
|
||||
{ name: "inspect_artifact", arguments: {} },
|
||||
{ timeout: 150_000 },
|
||||
);
|
||||
assert.notEqual(inspected.isError, true, JSON.stringify(inspected));
|
||||
const mcpInspection = parseEvidence(inspected.structuredContent);
|
||||
assert.notEqual(inspected.structuredContent, undefined);
|
||||
const mcpInspection = parseEvidence(
|
||||
requireMcpOutput(inspected, "inspect_artifact"),
|
||||
);
|
||||
assert.deepEqual(
|
||||
mcpInspection.normalized_result,
|
||||
inspection.normalized_result,
|
||||
@@ -186,9 +189,10 @@ try {
|
||||
},
|
||||
{ timeout: 150_000 },
|
||||
);
|
||||
assert.notEqual(response.isError, true, JSON.stringify(response));
|
||||
assert.ok(response.structuredContent !== undefined);
|
||||
const mcpEvidence = parseEvidence(response.structuredContent);
|
||||
assert.notEqual(response.structuredContent, undefined);
|
||||
const mcpEvidence = parseEvidence(
|
||||
requireMcpOutput(response, "project_harmony_application_graph"),
|
||||
);
|
||||
assert.deepEqual(
|
||||
mcpEvidence.normalized_result,
|
||||
projectionEvidence.normalized_result,
|
||||
|
||||
@@ -1505,6 +1505,24 @@ exit 1`,
|
||||
}
|
||||
});
|
||||
|
||||
it("preserves process names printed by Toybox without treating their whitespace as column padding", async () => {
|
||||
// Toybox's final NAME column is left justified without padding; only the
|
||||
// single separator after S belongs to the table (toys/posix/ps.c show_ps).
|
||||
const name = " native tool ";
|
||||
const { binary } = await writeAdbStub(
|
||||
shellStub({
|
||||
ps: `USER PID PPID VSZ RSS WCHAN ADDR S NAME\nroot 1234 567 123456 7890 0 0 S ${name}\r\n`,
|
||||
}),
|
||||
);
|
||||
const result = await evidenceResult(providerOver(binary), {
|
||||
operation: "list_adb_processes",
|
||||
input: { serial: "emulator-5554" },
|
||||
});
|
||||
expect(result.processes).toEqual([
|
||||
{ user: "root", pid: 1234, ppid: 567, rss: 7890, state: "S", name },
|
||||
]);
|
||||
});
|
||||
|
||||
it("lists processes, features, and services through fixed commands", async () => {
|
||||
const { binary } = await writeAdbStub(
|
||||
shellStub({
|
||||
|
||||
@@ -27,8 +27,10 @@ export interface AdbProcessEntry {
|
||||
readonly name: string;
|
||||
}
|
||||
|
||||
// Toybox emits one separator before the unpadded final NAME column.
|
||||
// Remaining whitespace belongs to argv[0], not the table layout.
|
||||
const PROCESS_LINE =
|
||||
/^(\S+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(.+)$/u;
|
||||
/^(\S+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\S+)\s+(\S+)\s+(\S+) (.+)$/u;
|
||||
|
||||
const safeInteger = (text: string): number | null => {
|
||||
const value = Number(text);
|
||||
@@ -46,22 +48,21 @@ export const parseProcessListOutput = (
|
||||
const processes: AdbProcessEntry[] = [];
|
||||
const unparsed = collectUnparsedLines();
|
||||
let columns: string | null = null;
|
||||
for (const rawLine of stdout.split(/\r?\n/u)) {
|
||||
const line = rawLine.trimEnd();
|
||||
if (line === "") continue;
|
||||
for (const line of stdout.split(/\r?\n/u)) {
|
||||
if (line.trim() === "") continue;
|
||||
if (/^USER\s+PID\s+PPID\b/u.test(line)) {
|
||||
columns = line.trim();
|
||||
continue;
|
||||
}
|
||||
const match = PROCESS_LINE.exec(line);
|
||||
if (match === null) {
|
||||
unparsed.add(rawLine);
|
||||
unparsed.add(line);
|
||||
continue;
|
||||
}
|
||||
const pid = safeInteger(match[2]!);
|
||||
const ppid = safeInteger(match[3]!);
|
||||
const rss = safeInteger(match[5]!);
|
||||
if (pid === null || ppid === null || rss === null) unparsed.add(rawLine);
|
||||
if (pid === null || ppid === null || rss === null) unparsed.add(line);
|
||||
if (pid === null) continue;
|
||||
processes.push({
|
||||
user: match[1]!,
|
||||
@@ -69,7 +70,7 @@ export const parseProcessListOutput = (
|
||||
ppid,
|
||||
rss,
|
||||
state: match[8]!,
|
||||
name: match[9]!.trim(),
|
||||
name: match[9]!,
|
||||
});
|
||||
}
|
||||
return { columns, processes, unparsedLines: unparsed.facts() };
|
||||
|
||||
+13
-10
@@ -1,3 +1,4 @@
|
||||
import { snapshotEnvironment } from "../process/snapshotEnvironment.js";
|
||||
import type { AnalysisSettingsReader } from "../domain/analysisSettings.js";
|
||||
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
|
||||
import { AnalysisCleanupLease } from "../artifacts/AnalysisCleanupLease.js";
|
||||
@@ -295,7 +296,8 @@ export class JadxProvider implements AndroidAnalysisPort {
|
||||
|
||||
/** Report actual local prerequisites without starting JADX or opening a target. */
|
||||
inspectAvailability(signal?: AbortSignal) {
|
||||
const pending = inspectJadxAvailability(this.environment, {
|
||||
const environment = snapshotEnvironment(this.environment);
|
||||
const pending = inspectJadxAvailability(environment, {
|
||||
operationTimeoutMs:
|
||||
this.settings?.().overrides["jadx.operation_timeout_ms"],
|
||||
resourceOwner: this.#prerequisiteOwner,
|
||||
@@ -442,10 +444,11 @@ export class JadxProvider implements AndroidAnalysisPort {
|
||||
request: AndroidRequest,
|
||||
options?: ExecutionOptions,
|
||||
): Promise<Outcome> {
|
||||
const environment = snapshotEnvironment(this.environment);
|
||||
// Resolve the accepted operation's override before acquiring any resource.
|
||||
// An unusable launch value is refused without starting the engine.
|
||||
const timeoutMs = resolveJadxOperationTimeout(
|
||||
this.environment,
|
||||
environment,
|
||||
request.operation,
|
||||
this.settings?.().overrides["jadx.operation_timeout_ms"],
|
||||
);
|
||||
@@ -471,7 +474,7 @@ export class JadxProvider implements AndroidAnalysisPort {
|
||||
`Target ${target.path} is ${target.format}; provide one standalone APK.`,
|
||||
);
|
||||
const configuration = await resolveJadxConfigurationInputs(
|
||||
this.environment,
|
||||
environment,
|
||||
request.operation,
|
||||
signal,
|
||||
);
|
||||
@@ -513,11 +516,11 @@ export class JadxProvider implements AndroidAnalysisPort {
|
||||
java: configuration.java,
|
||||
arguments: configuration.jvmArguments,
|
||||
options: [
|
||||
this.environment.JAVA_TOOL_OPTIONS,
|
||||
this.environment._JAVA_OPTIONS,
|
||||
this.environment.JDK_JAVA_OPTIONS,
|
||||
this.environment.PATH,
|
||||
this.environment.JAVA_HOME,
|
||||
environment.JAVA_TOOL_OPTIONS,
|
||||
environment._JAVA_OPTIONS,
|
||||
environment.JDK_JAVA_OPTIONS,
|
||||
environment.PATH,
|
||||
environment.JAVA_HOME,
|
||||
],
|
||||
});
|
||||
if (this.#retained !== undefined && this.#retained.key !== key) {
|
||||
@@ -528,7 +531,7 @@ export class JadxProvider implements AndroidAnalysisPort {
|
||||
if (retained === undefined) {
|
||||
await validateJadxConfiguration(
|
||||
configuration,
|
||||
this.environment,
|
||||
environment,
|
||||
request.operation,
|
||||
{
|
||||
resourceOwner: this.#prerequisiteOwner,
|
||||
@@ -589,7 +592,7 @@ export class JadxProvider implements AndroidAnalysisPort {
|
||||
bridge,
|
||||
],
|
||||
cwd: root.path,
|
||||
hostEnvironment: this.environment,
|
||||
hostEnvironment: environment,
|
||||
signal: workspaceSignal,
|
||||
},
|
||||
this.launcher,
|
||||
|
||||
@@ -13,6 +13,7 @@ import { silentLogger } from "../logger.js";
|
||||
import type { Logger } from "pino";
|
||||
import { withEvidenceCallerInputPath } from "../domain/evidence.js";
|
||||
import { createImmutableEvidenceBundle } from "../domain/evidenceBundle.js";
|
||||
import { retainAnalysisResultSources } from "./investigation/recordEvidenceSources.js";
|
||||
import type { Evidence } from "../domain/evidence.js";
|
||||
import type { NativeToolName } from "../contracts/native/nativeToolContracts.js";
|
||||
import {
|
||||
@@ -570,12 +571,17 @@ const executeAnalysisTool = async (input: {
|
||||
if (tool === "analyze_function") {
|
||||
const { executeFunctionAnalysisEvidence } =
|
||||
await import("./FunctionAnalysisEvidence.js");
|
||||
const result = await executeFunctionAnalysisEvidence(
|
||||
let result = await executeFunctionAnalysisEvidence(
|
||||
session,
|
||||
input.arguments,
|
||||
input.openedTarget,
|
||||
{ signal },
|
||||
);
|
||||
result = retainAnalysisResultSources(
|
||||
tool,
|
||||
(source) => session.recordEvidence(source),
|
||||
result,
|
||||
);
|
||||
return result.ok ? ok({ evidence: result.value }) : result;
|
||||
}
|
||||
if (
|
||||
@@ -588,12 +594,17 @@ const executeAnalysisTool = async (input: {
|
||||
tool === "decompile_all"
|
||||
) {
|
||||
const { EnhancedTools } = await import("./EnhancedTools.js");
|
||||
const result = await executeComposedWorkflow(
|
||||
let result = await executeComposedWorkflow(
|
||||
session,
|
||||
(observed) =>
|
||||
new EnhancedTools(observed).execute(tool, input.arguments, signal),
|
||||
composedLimitationBudget(tool),
|
||||
);
|
||||
result = retainAnalysisResultSources(
|
||||
tool,
|
||||
(source) => session.recordEvidence(source),
|
||||
result,
|
||||
);
|
||||
if (!result.ok) return result;
|
||||
const evidence = createWorkflowEvidence({
|
||||
target: input.openedTarget,
|
||||
|
||||
@@ -2,7 +2,7 @@ import { expect, it } from "vitest";
|
||||
import { createAnalysisExecution } from "./AnalysisProvider.js";
|
||||
import { executeFunctionAnalysisEvidence } from "./FunctionAnalysisEvidence.js";
|
||||
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
|
||||
import { parseEvidence } from "../domain/evidence.js";
|
||||
import { createEvidence, parseEvidence } from "../domain/evidence.js";
|
||||
import { createAnalysisProfile } from "../domain/analysisProfile.js";
|
||||
import { ghidraFunctionDossier } from "../domain/ghidraValues.fixture.js";
|
||||
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
|
||||
@@ -62,6 +62,16 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
|
||||
path: "/observed/provider.elf",
|
||||
sha256: "b".repeat(64),
|
||||
};
|
||||
const source = createEvidence(
|
||||
undefined,
|
||||
provider,
|
||||
{
|
||||
operation: "health",
|
||||
parameters: { facet: "historical-report" },
|
||||
result: { original: "unknown" },
|
||||
},
|
||||
{ immutable: true },
|
||||
);
|
||||
for (const aborted of [false, true]) {
|
||||
const controller = new AbortController();
|
||||
const execution = createAnalysisExecution(dossier, provider, {
|
||||
@@ -73,6 +83,7 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
|
||||
rawResult: { provider_text: "preserved" },
|
||||
limitations: ["Synthetic provider observation"],
|
||||
locations: [{ kind: "address", address: "0x401000" }],
|
||||
sourceEvidence: [source],
|
||||
});
|
||||
const result = await executeFunctionAnalysisEvidence(
|
||||
{
|
||||
@@ -85,6 +96,7 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
|
||||
target,
|
||||
{ signal: controller.signal },
|
||||
);
|
||||
expect(result.sourceEvidence).toEqual([source]);
|
||||
if (aborted) {
|
||||
if (result.ok) throw new Error("Expected late cancellation");
|
||||
expect(result.error).toMatchObject({ _tag: "AnalysisCancelledError" });
|
||||
@@ -100,6 +112,7 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
|
||||
expect(evidence.provider).toEqual(provider);
|
||||
expect(evidence.analysis_profile).toEqual(analysisProfile);
|
||||
expect(evidence.relationship_coverage).toEqual(relationshipCoverage);
|
||||
expect(evidence.evidence_links).toEqual([source.evidence_id]);
|
||||
expect(evidence.limitations).toContain("Synthetic provider observation");
|
||||
expect(evidence.locations).toContainEqual({
|
||||
kind: "address",
|
||||
|
||||
@@ -5,10 +5,17 @@ import type {
|
||||
} from "./AnalysisProvider.js";
|
||||
import { nativeProcedureInputSchema } from "../domain/native/nativeProcedureInput.js";
|
||||
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
|
||||
import { createEvidence } from "../domain/evidence.js";
|
||||
import { createAnalysisExecutionEvidence } from "../domain/analysisObservation.js";
|
||||
import { parseFunctionDossier } from "../domain/hopperValues.js";
|
||||
import type { JsonValue } from "../domain/jsonValue.js";
|
||||
import { completeProviderEvidence } from "./ProviderOperationExecution.js";
|
||||
import type { Evidence } from "../domain/evidence.js";
|
||||
import { EvidenceLedger } from "./investigation/EvidenceLedger.js";
|
||||
import {
|
||||
recordEvidenceSources,
|
||||
withEvidenceSources,
|
||||
type AnalysisResultWithSources,
|
||||
} from "./investigation/recordEvidenceSources.js";
|
||||
|
||||
/**
|
||||
* Preserve the provider's observations when presenting a validated function
|
||||
@@ -20,7 +27,7 @@ export const executeFunctionAnalysisEvidence = async (
|
||||
parameters: Readonly<Record<string, JsonValue>>,
|
||||
target: BinaryTarget | undefined,
|
||||
options?: ExecutionOptions,
|
||||
) => {
|
||||
): Promise<AnalysisResultWithSources<Evidence>> => {
|
||||
const input = parseAnalysisInput(
|
||||
"analyze_function",
|
||||
parameters,
|
||||
@@ -35,30 +42,33 @@ export const executeFunctionAnalysisEvidence = async (
|
||||
options,
|
||||
);
|
||||
if (!execution.ok) return execution;
|
||||
const sourceLedger = new EvidenceLedger();
|
||||
const recorded = recordEvidenceSources(
|
||||
(source) => sourceLedger.record(source),
|
||||
execution.value.sourceEvidence ?? [],
|
||||
);
|
||||
const sources = sourceLedger.recordsForAnalysis();
|
||||
if (!recorded.ok) return withEvidenceSources(recorded, sources);
|
||||
const dossier = parseFunctionDossier(execution.value.result);
|
||||
if (!dossier.ok) return dossier;
|
||||
if (!dossier.ok) return withEvidenceSources(dossier, sources);
|
||||
const observation = execution.value;
|
||||
return completeProviderEvidence(
|
||||
createEvidence(
|
||||
observation.subject === undefined
|
||||
? target
|
||||
: (observation.subject ?? undefined),
|
||||
observation.provider,
|
||||
{
|
||||
operation: "analyze_function",
|
||||
parameters: arguments_,
|
||||
result: dossier.value,
|
||||
rawResult: observation.rawResult,
|
||||
limitations: observation.limitations,
|
||||
locations: observation.locations,
|
||||
...(observation.relationshipCoverage === undefined
|
||||
? {}
|
||||
: { relationshipCoverage: observation.relationshipCoverage }),
|
||||
...(observation.analysisProfile === undefined
|
||||
? {}
|
||||
: { analysisProfile: observation.analysisProfile }),
|
||||
},
|
||||
return withEvidenceSources(
|
||||
completeProviderEvidence(
|
||||
createAnalysisExecutionEvidence(
|
||||
"analyze_function",
|
||||
arguments_,
|
||||
{
|
||||
...observation,
|
||||
sourceEvidence: sources,
|
||||
result: dossier.value,
|
||||
...(observation.subject === undefined && target !== undefined
|
||||
? { subject: target }
|
||||
: {}),
|
||||
},
|
||||
"observed",
|
||||
),
|
||||
options?.signal,
|
||||
),
|
||||
options?.signal,
|
||||
sources,
|
||||
);
|
||||
};
|
||||
@@ -19,6 +19,48 @@ import { projectNativeApiInspection } from "./native/NativeApiInspection.js";
|
||||
import { functionDossierSchema } from "../domain/hopperValues.js";
|
||||
import { ghidraFunctionDossier } from "../domain/ghidraValues.fixture.js";
|
||||
import { ANALYSIS_SNAPSHOT_PROFILE } from "../domain/analysisSnapshot.fixture.js";
|
||||
import { createEvidence } from "../domain/evidence.js";
|
||||
|
||||
it("links shared provider source Evidence once without copying its payload into the workflow", async () => {
|
||||
const source = createEvidence(
|
||||
undefined,
|
||||
{ id: "fixture", name: "Fixture", version: "1" },
|
||||
{
|
||||
operation: "health",
|
||||
parameters: { facet: "saved-analysis-report" },
|
||||
result: { report: "historical content ".repeat(1_000) },
|
||||
},
|
||||
{ immutable: true },
|
||||
);
|
||||
const result = await executeComposedWorkflow(
|
||||
{
|
||||
execute: async () =>
|
||||
ok(
|
||||
createAnalysisExecution(
|
||||
"observed",
|
||||
{ id: "fixture", name: "Fixture", version: "1" },
|
||||
{ sourceEvidence: [source] },
|
||||
),
|
||||
),
|
||||
},
|
||||
async (observed) => {
|
||||
await observed.execute("list_procedures", {});
|
||||
await observed.execute("procedure_pseudo_code", { procedure: "0x1000" });
|
||||
return ok({ count: 1 });
|
||||
},
|
||||
);
|
||||
expect(result).toEqual({
|
||||
...ok({
|
||||
result: { count: 1 },
|
||||
limitations: [],
|
||||
evidenceLinks: [source.evidence_id],
|
||||
}),
|
||||
sourceEvidence: [source],
|
||||
});
|
||||
expect(result.ok && JSON.stringify(result.value)).not.toContain(
|
||||
"historical content",
|
||||
);
|
||||
});
|
||||
|
||||
it.each([false, true])(
|
||||
"propagates transient=%s through composed snapshot decisions while retaining Evidence",
|
||||
|
||||
@@ -23,6 +23,12 @@ import {
|
||||
import type { JsonValue } from "../domain/jsonValue.js";
|
||||
import type { UnknownRegistryPort } from "./investigation/InvestigationRecordPort.js";
|
||||
import { UnknownRegistryError } from "../domain/unknownRegistryError.js";
|
||||
import { EvidenceLedger } from "./investigation/EvidenceLedger.js";
|
||||
import {
|
||||
recordEvidenceSources,
|
||||
withEvidenceSources,
|
||||
type AnalysisResultWithSources,
|
||||
} from "./investigation/recordEvidenceSources.js";
|
||||
import { workflowUnknownInput } from "./investigation/WorkflowUnknown.js";
|
||||
import {
|
||||
renderSourceLimitations,
|
||||
@@ -71,18 +77,16 @@ export const executeComposedWorkflow = async (
|
||||
) => Promise<Result<JsonValue, AnalysisError>>,
|
||||
budget?: ComposedLimitationBudget,
|
||||
): Promise<
|
||||
Result<
|
||||
{
|
||||
readonly result: JsonValue;
|
||||
readonly limitations: readonly string[];
|
||||
readonly cachePolicy?: "live";
|
||||
readonly evidenceLinks?: readonly string[];
|
||||
},
|
||||
AnalysisError
|
||||
>
|
||||
AnalysisResultWithSources<{
|
||||
readonly result: JsonValue;
|
||||
readonly limitations: readonly string[];
|
||||
readonly cachePolicy?: "live";
|
||||
readonly evidenceLinks?: readonly string[];
|
||||
}>
|
||||
> => {
|
||||
const limitations = new Map<string, SourceLimitation>();
|
||||
const evidenceLinks = new Set<string>();
|
||||
const sourceLedger = new EvidenceLedger();
|
||||
const perItemGroups = new Set<string>();
|
||||
let perItemBytes = 0;
|
||||
let omittedObservations = 0;
|
||||
@@ -93,6 +97,13 @@ export const executeComposedWorkflow = async (
|
||||
const execution = await analysis.execute(operation, parameters, options);
|
||||
if (execution.ok && execution.value.evidenceId !== undefined)
|
||||
evidenceLinks.add(execution.value.evidenceId);
|
||||
if (execution.ok && execution.value.sourceEvidence !== undefined) {
|
||||
const recorded = recordEvidenceSources(
|
||||
(source) => sourceLedger.record(source),
|
||||
execution.value.sourceEvidence,
|
||||
);
|
||||
if (!recorded.ok) return recorded;
|
||||
}
|
||||
if (execution.ok && execution.value.cachePolicy === "live")
|
||||
cachePolicy = "live";
|
||||
if (execution.ok)
|
||||
@@ -129,22 +140,27 @@ export const executeComposedWorkflow = async (
|
||||
return execution;
|
||||
},
|
||||
});
|
||||
if (!result.ok) return result;
|
||||
return ok({
|
||||
result: result.value,
|
||||
limitations: [
|
||||
...renderSourceLimitations([...limitations.values()]),
|
||||
...(omittedObservations === 0
|
||||
? []
|
||||
: [
|
||||
`${omittedObservations} per-item provider limitation observation${omittedObservations === 1 ? " was" : "s were"} omitted from this answer; ${budget?.retainedIn}.`,
|
||||
]),
|
||||
],
|
||||
...(cachePolicy === undefined ? {} : { cachePolicy }),
|
||||
...(evidenceLinks.size === 0
|
||||
? {}
|
||||
: { evidenceLinks: [...evidenceLinks].sort(compareUnicodeCodePoints) }),
|
||||
});
|
||||
const sources = sourceLedger.recordsForAnalysis();
|
||||
if (!result.ok) return withEvidenceSources(result, sources);
|
||||
for (const source of sources) evidenceLinks.add(source.evidence_id);
|
||||
return withEvidenceSources(
|
||||
ok({
|
||||
result: result.value,
|
||||
limitations: [
|
||||
...renderSourceLimitations([...limitations.values()]),
|
||||
...(omittedObservations === 0
|
||||
? []
|
||||
: [
|
||||
`${omittedObservations} per-item provider limitation observation${omittedObservations === 1 ? " was" : "s were"} omitted from this answer; ${budget?.retainedIn}.`,
|
||||
]),
|
||||
],
|
||||
...(cachePolicy === undefined ? {} : { cachePolicy }),
|
||||
...(evidenceLinks.size === 0
|
||||
? {}
|
||||
: { evidenceLinks: [...evidenceLinks].sort(compareUnicodeCodePoints) }),
|
||||
}),
|
||||
sources,
|
||||
);
|
||||
};
|
||||
|
||||
/** Build the shared Evidence record for a successful composed binary workflow. */
|
||||
|
||||
@@ -433,6 +433,131 @@ describe("analysis provider registry: selection failures", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("analysis provider registry: rejection origins", () => {
|
||||
it("distinguishes recorded readiness failures from an unobserved target refusal", async () => {
|
||||
const registry = new AnalysisProviderRegistry([
|
||||
candidate("alpha", { available: false, unconfigured: true }).provider,
|
||||
candidate("gamma", { available: false, unconfigured: true }).provider,
|
||||
]);
|
||||
const selection = await registry.select(DATABASE_TARGET);
|
||||
expect(selection.ok).toBe(true);
|
||||
if (!selection.ok) return;
|
||||
const projected = projectAnalysisError(
|
||||
registry.unboundOperationError("binary_overview", selection.value),
|
||||
);
|
||||
expect(projected.message).toContain(
|
||||
"Provider selection failed local readiness checks. No target-support refusal was reported.",
|
||||
);
|
||||
});
|
||||
|
||||
it("reports a configured provider's readiness failure without hiding a ready alternative", async () => {
|
||||
const invalid = candidate("alpha", { available: false });
|
||||
const ready = candidate("beta");
|
||||
const registry = new AnalysisProviderRegistry([
|
||||
{
|
||||
...invalid.provider,
|
||||
inspectAvailability: () => ({
|
||||
status: "unavailable",
|
||||
code: "open_options_invalid",
|
||||
reason: "Configured registration is invalid",
|
||||
diagnostics: { registration_path: "/configured/alpha.json" },
|
||||
}),
|
||||
},
|
||||
ready.provider,
|
||||
]);
|
||||
const selected = await registry.select(DATABASE_TARGET, "alpha");
|
||||
if (selected.ok)
|
||||
throw new Error("Expected the selected provider to refuse");
|
||||
const projected = projectAnalysisError(selected.error);
|
||||
expect(projected.message).toContain("local readiness");
|
||||
expect(projected.message).not.toContain(
|
||||
"No analysis provider is configured",
|
||||
);
|
||||
expect(projected.details).toMatchObject({
|
||||
requested_provider_id: "alpha",
|
||||
candidate_ids: ["alpha", "beta"],
|
||||
rejections: [
|
||||
{
|
||||
provider_id: "alpha",
|
||||
origin: "availability",
|
||||
code: "open_options_invalid",
|
||||
reason: "Configured registration is invalid",
|
||||
diagnostics: { registration_path: "/configured/alpha.json" },
|
||||
},
|
||||
],
|
||||
});
|
||||
expect((await registry.select(DATABASE_TARGET, "beta")).ok).toBe(true);
|
||||
expect(invalid.created).toEqual([]);
|
||||
});
|
||||
|
||||
it("names whether a rejection came from provider availability or target support", async () => {
|
||||
const availabilityOnly = new AnalysisProviderRegistry([
|
||||
candidate("alpha", { available: false, unconfigured: true }).provider,
|
||||
candidate("gamma", { available: false, unconfigured: true }).provider,
|
||||
]);
|
||||
const availabilitySelection =
|
||||
await availabilityOnly.select(DATABASE_TARGET);
|
||||
expect(availabilitySelection.ok).toBe(true);
|
||||
if (availabilitySelection.ok) {
|
||||
const { details } = projectAnalysisError(
|
||||
availabilityOnly.unboundOperationError(
|
||||
"binary_overview",
|
||||
availabilitySelection.value,
|
||||
),
|
||||
);
|
||||
const rejections = details?.rejections as unknown as readonly {
|
||||
origin: string;
|
||||
}[];
|
||||
expect(rejections.map(({ origin }) => origin)).toEqual([
|
||||
"availability",
|
||||
"availability",
|
||||
]);
|
||||
}
|
||||
|
||||
const unconfiguredButAdmitting: AnalysisProviderCandidate = {
|
||||
...candidate("alpha", { available: false, unconfigured: true }).provider,
|
||||
inspectTargetSupport: () => ({
|
||||
status: "supported",
|
||||
code: null,
|
||||
reason: null,
|
||||
diagnostics: { target_kind: "archive" },
|
||||
}),
|
||||
};
|
||||
const refusingTarget: AnalysisProviderCandidate = {
|
||||
...candidate("beta").provider,
|
||||
inspectTargetSupport: () => ({
|
||||
status: "unsupported",
|
||||
code: "target_format_unsupported",
|
||||
reason: "beta refuses archives",
|
||||
diagnostics: { target_kind: "archive" },
|
||||
}),
|
||||
};
|
||||
const mixed = new AnalysisProviderRegistry([
|
||||
unconfiguredButAdmitting,
|
||||
refusingTarget,
|
||||
]);
|
||||
const mixedSelection = await mixed.select(ARTIFACT_TARGET);
|
||||
expect(mixedSelection.ok).toBe(true);
|
||||
if (mixedSelection.ok) {
|
||||
const projected = projectAnalysisError(
|
||||
mixed.unboundOperationError("binary_overview", mixedSelection.value),
|
||||
);
|
||||
expect(projected.message).toContain(
|
||||
"The selected analysis provider is unavailable",
|
||||
);
|
||||
const { details } = projected;
|
||||
const rejections = details?.rejections as unknown as readonly {
|
||||
origin: string;
|
||||
code: string;
|
||||
}[];
|
||||
expect(rejections).toMatchObject([
|
||||
{ origin: "availability", code: "not_configured" },
|
||||
{ origin: "target", code: "target_format_unsupported" },
|
||||
]);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("analysis provider registry: profile and cleanup failures", () => {
|
||||
it("does not bind a profile with unresolved version or adapter failure", async () => {
|
||||
const unresolved = candidate("unresolved", { profile: "missing" });
|
||||
|
||||
@@ -21,6 +21,7 @@ import {
|
||||
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import {
|
||||
ProviderSelectionError,
|
||||
type ProviderRejectionOrigin,
|
||||
type ProviderSelectionErrorOptions,
|
||||
type ProviderSelectionFailureReason,
|
||||
type ProviderSelectionRejection,
|
||||
@@ -880,21 +881,43 @@ const selectionError = (
|
||||
const candidateRejections = (
|
||||
candidates: readonly AnalysisProviderCandidateStatus[],
|
||||
): readonly ProviderSelectionRejection[] =>
|
||||
candidates.flatMap((candidate) =>
|
||||
[
|
||||
...(candidate.availability.status === "unavailable"
|
||||
? [candidate.availability]
|
||||
: []),
|
||||
...(candidate.targetSupport.status === "unsupported"
|
||||
? [candidate.targetSupport]
|
||||
: []),
|
||||
].map((rejection) => ({
|
||||
providerId: candidate.provider.id,
|
||||
code: rejection.code,
|
||||
reason: rejection.reason,
|
||||
diagnostics: structuredClone(rejection.diagnostics),
|
||||
})),
|
||||
);
|
||||
candidates.flatMap((candidate) => {
|
||||
const rejections: ProviderSelectionRejection[] = [];
|
||||
if (candidate.availability.status === "unavailable")
|
||||
rejections.push(
|
||||
selectedRejection(candidate, "availability", candidate.availability),
|
||||
);
|
||||
if (candidate.targetSupport.status === "unsupported")
|
||||
rejections.push(
|
||||
selectedRejection(candidate, "target", candidate.targetSupport),
|
||||
);
|
||||
return rejections;
|
||||
});
|
||||
|
||||
/** One availability or target-support observation that rejected a candidate. */
|
||||
type RejectionObservation =
|
||||
| Extract<
|
||||
AnalysisProviderCandidateStatus["availability"],
|
||||
{ status: "unavailable" }
|
||||
>
|
||||
| Extract<
|
||||
AnalysisProviderCandidateStatus["targetSupport"],
|
||||
{ status: "unsupported" }
|
||||
>;
|
||||
|
||||
/** Name the observation a rejection came from so availability cannot read as a refusal. */
|
||||
const selectedRejection = (
|
||||
candidate: AnalysisProviderCandidateStatus,
|
||||
origin: ProviderRejectionOrigin,
|
||||
rejection: RejectionObservation,
|
||||
): ProviderSelectionRejection => ({
|
||||
providerId: candidate.provider.id,
|
||||
origin,
|
||||
code: rejection.code,
|
||||
reason: rejection.reason,
|
||||
// ProviderSelectionError owns the single defensive diagnostic copy.
|
||||
diagnostics: rejection.diagnostics,
|
||||
});
|
||||
|
||||
const cloneAvailability = (
|
||||
availability: AnalysisProviderCandidateStatus["availability"],
|
||||
|
||||
@@ -1254,6 +1254,15 @@ export class BinarySession
|
||||
)
|
||||
this.#active.nativeSourceAdmitted = true;
|
||||
this.#observeRuntimeAvailability(name, execution);
|
||||
// Source facts belong to the observation, even when this query cannot be
|
||||
// replayed (for example a Ghidra procedure read without a document selector).
|
||||
if (execution.ok && execution.value.sourceEvidence !== undefined) {
|
||||
const sourcesRecorded = recordEvidenceSources(
|
||||
(source) => this.recordEvidence(source),
|
||||
execution.value.sourceEvidence,
|
||||
);
|
||||
if (!sourcesRecorded.ok) return sourcesRecorded;
|
||||
}
|
||||
if (
|
||||
execution.ok &&
|
||||
cacheable &&
|
||||
@@ -1276,11 +1285,6 @@ export class BinarySession
|
||||
},
|
||||
"observed",
|
||||
);
|
||||
const sourcesRecorded = recordEvidenceSources(
|
||||
(source) => this.recordEvidence(source),
|
||||
execution.value.sourceEvidence ?? [],
|
||||
);
|
||||
if (!sourcesRecorded.ok) return sourcesRecorded;
|
||||
const recorded = this.recordEvidence(evidence);
|
||||
if (!recorded.ok) return recorded;
|
||||
const retained = { ...execution.value, evidenceId: evidence.evidence_id };
|
||||
|
||||
@@ -1,14 +1,16 @@
|
||||
import { basename } from "node:path";
|
||||
import { isIP } from "node:net";
|
||||
|
||||
import { isJsonLike } from "../../domain/analysisView/EnvironmentManifestAccumulator.js";
|
||||
import {
|
||||
isJsonLike,
|
||||
mediaTypeEssence,
|
||||
} from "../../domain/analysisView/recordedMediaType.js";
|
||||
import {
|
||||
decodeRetainedBase64,
|
||||
readRecordedResponseBody,
|
||||
type RecordedResponseBody,
|
||||
} from "../../domain/analysisView/recordedResponseBody.js";
|
||||
import { scenarioContentMediaType } from "../../domain/analysisView/scenarioCaptureRecords.js";
|
||||
import { mediaTypeEssence } from "../../domain/analysisView/webNetworkRouteTemplates.js";
|
||||
import type { BrowserScenarioEvent } from "../../domain/browserScenarioCaptureValues.js";
|
||||
import type { EnvironmentBuildReport } from "../../domain/environmentBundleExport.js";
|
||||
import {
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
environmentStateServeFixture,
|
||||
} from "../../../tests/fixtures/environmentServe.js";
|
||||
import { compareEnvironment } from "./environmentCompare.js";
|
||||
import { requestBody } from "./environmentCompareProbe.js";
|
||||
import { REPLAY_BODY_LIMIT_BYTES } from "./environmentSource.js";
|
||||
|
||||
const responseCapture = (
|
||||
@@ -144,3 +145,24 @@ describe("environment comparison response representation", () => {
|
||||
expect(compare(json, text).probes[0]?.results[0]?.equal).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ mimeType: " TEXT/PLAIN ; charset=utf-8", expected: { value: 1 } },
|
||||
{ mimeType: "application/problem+json", expected: { value: 1 } },
|
||||
{ mimeType: "text/problem+json", expected: '{"value":1}' },
|
||||
])(
|
||||
"reads probe request arguments independently of body-size and parameter admission: $mimeType",
|
||||
({ mimeType, expected }) => {
|
||||
const capture = environmentManifestCaptureFixture([
|
||||
{
|
||||
request: {
|
||||
method: "POST",
|
||||
url: "http://app.example/api/value",
|
||||
bodySize: "not-a-size",
|
||||
postData: { mimeType, text: '{"value":1}', params: 7 },
|
||||
},
|
||||
},
|
||||
]);
|
||||
expect(requestBody(capture.records[0]!)).toEqual(expected);
|
||||
},
|
||||
);
|
||||
@@ -1,11 +1,15 @@
|
||||
import { z } from "zod";
|
||||
import { isJsonLike } from "../../domain/analysisView/EnvironmentManifestAccumulator.js";
|
||||
|
||||
import { readRecordedRequestBody } from "../../domain/analysisView/recordedExchangeBody.js";
|
||||
import {
|
||||
isJsonLike,
|
||||
mediaTypeEssence,
|
||||
} from "../../domain/analysisView/recordedMediaType.js";
|
||||
import {
|
||||
primaryOrigin,
|
||||
type OriginStats,
|
||||
} from "../../domain/analysisView/environmentManifestOrigin.js";
|
||||
import {
|
||||
mediaTypeEssence,
|
||||
readRecordedRequestLine,
|
||||
type RecordedRequestLine,
|
||||
type RequestLineExclusion,
|
||||
@@ -44,12 +48,6 @@ export interface ProbeStep {
|
||||
const BODYLESS_STATUSES: ReadonlySet<number> = new Set([204, 205, 304]);
|
||||
|
||||
const startedSchema = z.object({ startedDateTime: z.string() });
|
||||
const postDataSchema = z.object({
|
||||
request: z.object({
|
||||
postData: z.object({ mimeType: z.string(), text: z.string() }),
|
||||
}),
|
||||
});
|
||||
|
||||
const methodSchema = z.object({ request: z.object({ method: z.string() }) });
|
||||
|
||||
const startedInstant = (
|
||||
@@ -90,9 +88,11 @@ export const sendsBodies = (
|
||||
export const requestBody = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
): JsonValue | undefined => {
|
||||
const parsed = postDataSchema.safeParse(record.reported);
|
||||
if (!parsed.success) return undefined;
|
||||
const { mimeType, text } = parsed.data.request.postData;
|
||||
const facts = readRecordedRequestBody(record.reported);
|
||||
if (facts.kind !== "request" || facts.postData.kind !== "object")
|
||||
return undefined;
|
||||
const { mimeType, text } = facts.postData;
|
||||
if (mimeType === undefined || text === undefined) return undefined;
|
||||
return isJsonLike(mediaTypeEssence(mimeType))
|
||||
? (parseExactJson(text) ?? text)
|
||||
: text;
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
import { z } from "zod";
|
||||
|
||||
import {
|
||||
isJsonLike,
|
||||
mediaTypeEssence,
|
||||
} from "../../domain/analysisView/recordedMediaType.js";
|
||||
|
||||
import {
|
||||
analysisViewResultSchema,
|
||||
type AnalysisViewResult,
|
||||
@@ -9,9 +14,7 @@ import {
|
||||
readRecordedResponseBody,
|
||||
type RecordedResponseBody,
|
||||
} from "../../domain/analysisView/recordedResponseBody.js";
|
||||
import { isJsonLike } from "../../domain/analysisView/EnvironmentManifestAccumulator.js";
|
||||
import {
|
||||
mediaTypeEssence,
|
||||
readRecordedRequestLine,
|
||||
type RecordedRequestLine,
|
||||
} from "../../domain/analysisView/webNetworkRouteTemplates.js";
|
||||
|
||||
@@ -76,6 +76,7 @@ const expectReloadsUnchanged = (books: Served): void => {
|
||||
describe("confirmed write response bodies", () => {
|
||||
it.each<{
|
||||
name: string;
|
||||
status?: number;
|
||||
content: Record<string, JsonValue>;
|
||||
shape: "recorded" | "empty";
|
||||
bodyRecorded: boolean;
|
||||
@@ -96,6 +97,23 @@ describe("confirmed write response bodies", () => {
|
||||
shape: "recorded",
|
||||
bodyRecorded: false,
|
||||
},
|
||||
{
|
||||
name: "205 without retained text",
|
||||
status: 205,
|
||||
content: { mimeType: "text/html" },
|
||||
shape: "recorded",
|
||||
bodyRecorded: false,
|
||||
},
|
||||
...[undefined, null].map((text) => ({
|
||||
name: `zero-size HTML with ${text === null ? "null" : "omitted"} text`,
|
||||
content: {
|
||||
mimeType: "text/html",
|
||||
size: 0,
|
||||
...(text === null ? { text } : {}),
|
||||
},
|
||||
shape: "recorded" as const,
|
||||
bodyRecorded: false,
|
||||
})),
|
||||
{
|
||||
name: "known empty text",
|
||||
content: { mimeType: "application/json", size: 0, text: "" },
|
||||
@@ -104,7 +122,7 @@ describe("confirmed write response bodies", () => {
|
||||
},
|
||||
])(
|
||||
"preserves $name when serving a confirmed document write",
|
||||
({ content, shape, bodyRecorded }) => {
|
||||
({ content, shape, bodyRecorded, status = 200 }) => {
|
||||
const entries: Record<string, JsonValue>[] = [
|
||||
{
|
||||
request: { method: "GET", url: "http://app.example/api/settings" },
|
||||
@@ -125,7 +143,7 @@ describe("confirmed write response bodies", () => {
|
||||
text: '{"theme":"dark"}',
|
||||
},
|
||||
},
|
||||
response: { status: 200, content },
|
||||
response: { status, content },
|
||||
},
|
||||
{
|
||||
request: { method: "GET", url: "http://app.example/api/settings" },
|
||||
@@ -169,7 +187,7 @@ describe("confirmed write response bodies", () => {
|
||||
const write = respond(patch, { body: { theme: "blue" } });
|
||||
if (!write.ok) throw write.error;
|
||||
expect(write.value).toMatchObject({
|
||||
status: 200,
|
||||
status,
|
||||
body: null,
|
||||
replay: {
|
||||
record_ordinal: 1,
|
||||
|
||||
@@ -26,7 +26,13 @@ import { webNetworkCaptureSchema } from "../../domain/webNetworkCapture.js";
|
||||
const archived = (value: boolean) => ({ ...FIRST_BOOK, archived: value });
|
||||
|
||||
describe("bodyless action evidence", () => {
|
||||
it.each([
|
||||
it.each<{
|
||||
name: string;
|
||||
postData: unknown;
|
||||
bodySize?: unknown;
|
||||
served?: boolean;
|
||||
shape?: boolean;
|
||||
}>([
|
||||
{
|
||||
name: "form",
|
||||
postData: {
|
||||
@@ -46,9 +52,45 @@ describe("bodyless action evidence", () => {
|
||||
},
|
||||
},
|
||||
{ name: "unretained body", postData: undefined },
|
||||
{
|
||||
name: "omitted body with zero size",
|
||||
postData: undefined,
|
||||
bodySize: 0,
|
||||
served: true,
|
||||
},
|
||||
{ name: "explicit null with zero size", postData: null, bodySize: 0 },
|
||||
{
|
||||
name: "params-only with zero size",
|
||||
postData: {
|
||||
mimeType: "application/json",
|
||||
params: [{ name: "archived", value: "true" }],
|
||||
},
|
||||
bodySize: 0,
|
||||
},
|
||||
{
|
||||
name: "empty text with nonempty params",
|
||||
postData: {
|
||||
mimeType: "application/json",
|
||||
text: "",
|
||||
params: [{ name: "archived", value: "true" }],
|
||||
},
|
||||
bodySize: 0,
|
||||
},
|
||||
{
|
||||
name: "explicit empty body with zero size",
|
||||
postData: { mimeType: "application/json", text: "", params: [] },
|
||||
bodySize: 0,
|
||||
served: true,
|
||||
},
|
||||
{
|
||||
name: "invalid size with retained JSON",
|
||||
postData: { mimeType: "application/json", text: '{"archived":true}' },
|
||||
bodySize: "unknown",
|
||||
shape: true,
|
||||
},
|
||||
])(
|
||||
"does not use unreadable or unretained request bodies as a repeated bodyless action: $name",
|
||||
({ postData }) => {
|
||||
"preserves literal request-body availability for repeated actions: $name",
|
||||
({ postData, bodySize = 12, served = false, shape = false }) => {
|
||||
const capture = timedCaptureEvidence([
|
||||
["GET", "/api/books", [archived(false), SECOND_BOOK]],
|
||||
["GET", "/api/books/1", archived(false)],
|
||||
@@ -60,14 +102,18 @@ describe("bodyless action evidence", () => {
|
||||
]);
|
||||
const result = capture.normalized_result as {
|
||||
records: {
|
||||
reported: { request: { postData?: unknown; bodySize?: number } };
|
||||
reported: { request: { postData?: unknown; bodySize?: unknown } };
|
||||
}[];
|
||||
};
|
||||
const request = result.records[5]!.reported.request;
|
||||
request.bodySize = 12;
|
||||
request.bodySize = bodySize;
|
||||
if (postData === undefined) delete request.postData;
|
||||
else request.postData = postData;
|
||||
const { state } = environmentStateServeInputs(
|
||||
if (shape) {
|
||||
result.records[3]!.reported.request.bodySize = bodySize;
|
||||
result.records[3]!.reported.request.postData = postData;
|
||||
}
|
||||
const { source, state } = environmentStateServeInputs(
|
||||
environmentStateServeFixture(
|
||||
analysisViewCaptureEvidence(
|
||||
webNetworkCaptureSchema.parse(capture.normalized_result),
|
||||
@@ -75,8 +121,17 @@ describe("bodyless action evidence", () => {
|
||||
),
|
||||
);
|
||||
expect(state.writes.get("post_books_archive_by_id")).toMatchObject({
|
||||
served: false,
|
||||
served,
|
||||
});
|
||||
const tool = source.manifest.tools.find(
|
||||
({ name }) => name === "post_books_archive_by_id",
|
||||
);
|
||||
expect(tool).toBeDefined();
|
||||
if (shape)
|
||||
expect(tool?.body_shape).toMatchObject({
|
||||
properties: { archived: "boolean" },
|
||||
});
|
||||
else if (bodySize === 0) expect(tool?.body_shape).toBeNull();
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
@@ -198,6 +198,11 @@ export class EvidenceLedger {
|
||||
);
|
||||
}
|
||||
|
||||
/** Borrow immutable source records without constructing or hashing a bundle. */
|
||||
recordsForAnalysis(): readonly Evidence[] {
|
||||
return [...this.#records.values()];
|
||||
}
|
||||
|
||||
/** Check whether one immutable Evidence ID is present in this ledger. */
|
||||
has(evidenceId: string): boolean {
|
||||
return this.#records.has(evidenceId);
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { createEvidence } from "../../domain/evidence.js";
|
||||
import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js";
|
||||
import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
|
||||
import { err, ok } from "../../domain/result.js";
|
||||
import {
|
||||
retainAnalysisResultSources,
|
||||
withEvidenceSources,
|
||||
} from "./recordEvidenceSources.js";
|
||||
|
||||
it.each([false, true])(
|
||||
"reports source recording failure without replacing earlier cancellation (cancelled=%s)",
|
||||
(cancelled) => {
|
||||
const source = createEvidence(
|
||||
undefined,
|
||||
{ id: "fixture", name: "Fixture", version: "1" },
|
||||
{ operation: "health", parameters: {}, result: "historical" },
|
||||
);
|
||||
const primary = new AnalysisCancelledError("analyze_function", {
|
||||
cause: "caller",
|
||||
});
|
||||
const failure = new EvidenceIntegrityError("Recorder failed");
|
||||
const result = retainAnalysisResultSources(
|
||||
"analyze_function",
|
||||
() => err(failure),
|
||||
withEvidenceSources(cancelled ? err(primary) : ok("complete"), [source]),
|
||||
);
|
||||
expect(result.ok).toBe(false);
|
||||
if (result.ok) throw new Error("Expected recording failure");
|
||||
expect(result.error).toBe(cancelled ? primary : failure);
|
||||
},
|
||||
);
|
||||
@@ -1,7 +1,49 @@
|
||||
import type { EvidenceWriter } from "./InvestigationRecordPort.js";
|
||||
import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
|
||||
import type { Evidence } from "../../domain/evidence.js";
|
||||
import { ok, type Result } from "../../domain/result.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { AnalysisCapabilityUnavailableError } from "../../domain/analysisErrorCore.js";
|
||||
|
||||
/** Source facts remain available even when a completed observation is cancelled. */
|
||||
export type AnalysisResultWithSources<Value> = Result<Value, AnalysisError> & {
|
||||
readonly sourceEvidence?: readonly Evidence[];
|
||||
};
|
||||
|
||||
export const withEvidenceSources = <
|
||||
Outcome extends Result<unknown, AnalysisError>,
|
||||
>(
|
||||
result: Outcome,
|
||||
sources: readonly Evidence[],
|
||||
): Outcome & { readonly sourceEvidence?: readonly Evidence[] } =>
|
||||
sources.length === 0 ? result : { ...result, sourceEvidence: sources };
|
||||
|
||||
/** Never deliver links to records a plain-port adapter cannot retain. */
|
||||
export const retainAnalysisResultSources = <Value>(
|
||||
operation: string,
|
||||
recordEvidence: EvidenceWriter["recordEvidence"] | undefined,
|
||||
result: AnalysisResultWithSources<Value>,
|
||||
): Result<Value, AnalysisError> => {
|
||||
const sources = result.sourceEvidence ?? [];
|
||||
const retained =
|
||||
sources.length > 0 && recordEvidence === undefined
|
||||
? err(
|
||||
new AnalysisCapabilityUnavailableError(
|
||||
"rea",
|
||||
operation,
|
||||
"Provider source Evidence requires an explicit retention owner.",
|
||||
{
|
||||
userMessage:
|
||||
"Use an analysis session that retains provider source Evidence, or supply an Evidence recorder when registering this adapter.",
|
||||
},
|
||||
),
|
||||
)
|
||||
: recordEvidenceSources(recordEvidence, sources);
|
||||
if (retained.ok) return result;
|
||||
if (result.ok) return retained;
|
||||
result.error.retainSecondaryFailure(retained.error);
|
||||
return result;
|
||||
};
|
||||
|
||||
/** Record source evidence in order, stopping at the first session rejection. */
|
||||
export const recordEvidenceSources = (
|
||||
|
||||
@@ -21,7 +21,7 @@ import { readAssemblyReferenceName } from "../dotnet/ManagedMetadataInventoryRow
|
||||
import { ManagedReaderFailure } from "../dotnet/ManagedReaderFailure.js";
|
||||
import { ArtifactReaderFailure } from "./ArtifactReader.js";
|
||||
import { abortIfNeeded } from "./ArtifactHash.js";
|
||||
import { hasMachoMagic } from "./apple/MachoLoadCommandReader.js";
|
||||
import { hasMachoMagic } from "../domain/apple/machoContainer.js";
|
||||
|
||||
export interface DesktopReadBudget {
|
||||
bytes: number;
|
||||
|
||||
@@ -728,3 +728,48 @@ it("retains SEA admission for a complete segment with a nonaligned command size"
|
||||
await handle.close();
|
||||
}
|
||||
});
|
||||
|
||||
it.each([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe])(
|
||||
"classifies four-byte thin magic %i without admitting a truncated SEA carrier",
|
||||
async (magic) => {
|
||||
const bytes = Buffer.alloc(4);
|
||||
bytes.writeUInt32BE(magic);
|
||||
const path = await writeCarrier("thin-prefix", bytes);
|
||||
const inventory = await scanCanonicalArtifactInventory(path);
|
||||
expect(inventory.manifest.root_format).toBe("mach-o");
|
||||
const handle = await open(path, "r");
|
||||
try {
|
||||
expect(await locateNodeSea(handle, bytes.length)).toBeUndefined();
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("confirms SEA naming before rejecting an unrelated cyclic PE resource branch", async () => {
|
||||
const fixture = peResourceFixture(true, [
|
||||
{ type: 3, name: 1, language: 1033, payload: Buffer.from("other") },
|
||||
{
|
||||
type: 10,
|
||||
name: "NODE_SEA_BLOB",
|
||||
language: 1033,
|
||||
payload: buildNodeSeaBlob(populated),
|
||||
},
|
||||
]);
|
||||
const other = fixture.entryOffsets.find(
|
||||
(at) => fixture.bytes.readUInt32LE(at) === 3,
|
||||
);
|
||||
if (other === undefined) throw new Error("Expected unrelated resource type");
|
||||
fixture.bytes.writeUInt32LE(0x80000000, other + 4);
|
||||
const handle = await open(
|
||||
await writeCarrier("sea-with-unrelated-cycle", fixture.bytes),
|
||||
"r",
|
||||
);
|
||||
try {
|
||||
await expect(
|
||||
locateNodeSea(handle, fixture.bytes.length),
|
||||
).rejects.toMatchObject({ reason: "format" });
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,66 @@
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
|
||||
/**
|
||||
* Preserve REA's source integer interpretation, including uint64
|
||||
* wrapping for unusual widths. Callers own byte-range and safe-number admission.
|
||||
*/
|
||||
export const readBinaryPlistUnsigned64 = (
|
||||
bytes: Uint8Array,
|
||||
offset: number,
|
||||
width: number,
|
||||
): bigint => {
|
||||
let value = 0n;
|
||||
for (let index = 0; index < width; index += 1)
|
||||
value = BigInt.asUintN(
|
||||
64,
|
||||
(value << 8n) | BigInt(bytes[offset + index] ?? 0),
|
||||
);
|
||||
return value;
|
||||
};
|
||||
|
||||
/** Raw declarations tied to one source or decoder-copy byte representation. */
|
||||
export interface BinaryPlistLayout {
|
||||
readonly trailerOffset: number;
|
||||
readonly offsetWidth: number;
|
||||
readonly referenceWidth: number;
|
||||
readonly objectCount: bigint;
|
||||
readonly topObject: bigint;
|
||||
readonly offsetTable: bigint;
|
||||
readonly objectOffset: (object: number) => bigint;
|
||||
readonly objectOffsets: () => Generator<bigint>;
|
||||
}
|
||||
|
||||
/** Observe declarations only; signature, bounds, capacities and reachability remain caller policy. */
|
||||
export const readBinaryPlistLayout = (
|
||||
bytes: Uint8Array,
|
||||
): Result<
|
||||
BinaryPlistLayout,
|
||||
{ readonly _tag: "InvalidBinaryPlistLayout"; readonly reason: "truncated" }
|
||||
> => {
|
||||
if (bytes.byteLength < 32)
|
||||
return err({ _tag: "InvalidBinaryPlistLayout", reason: "truncated" });
|
||||
const trailerOffset = bytes.byteLength - 32;
|
||||
const offsetWidth = bytes[trailerOffset + 6]!;
|
||||
const referenceWidth = bytes[trailerOffset + 7]!;
|
||||
const objectCount = readBinaryPlistUnsigned64(bytes, trailerOffset + 8, 8);
|
||||
const offsetTable = readBinaryPlistUnsigned64(bytes, trailerOffset + 24, 8);
|
||||
const objectOffset = (object: number): bigint =>
|
||||
readBinaryPlistUnsigned64(
|
||||
bytes,
|
||||
Number(offsetTable) + object * offsetWidth,
|
||||
offsetWidth,
|
||||
);
|
||||
return ok({
|
||||
trailerOffset,
|
||||
offsetWidth,
|
||||
referenceWidth,
|
||||
objectCount,
|
||||
topObject: readBinaryPlistUnsigned64(bytes, trailerOffset + 16, 8),
|
||||
offsetTable,
|
||||
objectOffset,
|
||||
*objectOffsets() {
|
||||
for (let object = 0; object < Number(objectCount); object += 1)
|
||||
yield objectOffset(object);
|
||||
},
|
||||
});
|
||||
};
|
||||
@@ -1,5 +1,8 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { binaryPlistObjectTable } from "../../../tests/fixtures/binaryPlistObjectTable.js";
|
||||
import { createPlistNumberProjection } from "../../domain/apple/plistNumbers.js";
|
||||
import { projectPlistValue } from "../../domain/apple/plistValue.js";
|
||||
import { scanBinaryPlistScalars } from "./PlistScalars.js";
|
||||
import { plistUidValue } from "../../domain/apple/plistUid.js";
|
||||
import { decodeBinaryPlistValue } from "./BinaryPlistMaterializer.js";
|
||||
import { estimateBinaryPlistExpansion } from "./PropertyListDecodeBudget.js";
|
||||
@@ -80,3 +83,67 @@ it("keeps a source UID distinct from an ordinary UID-shaped dictionary", () => {
|
||||
expect(plistUidValue(value[0])).toBe(7);
|
||||
expect(plistUidValue(value[1])).toBeUndefined();
|
||||
});
|
||||
|
||||
it("materializes normalized three-byte tables from a nonzero root without losing source numbers", () => {
|
||||
const tableOffset = 36;
|
||||
const bytes = Buffer.alloc(tableOffset + 9 + 32);
|
||||
bytes.write("bplist00");
|
||||
bytes[8] = 0x14;
|
||||
bytes.writeBigUInt64BE(1n, 9);
|
||||
bytes.writeBigUInt64BE(2n, 17);
|
||||
bytes[25] = 0xa2;
|
||||
bytes.writeUIntBE(0, 26, 3);
|
||||
bytes.writeUIntBE(2, 29, 3);
|
||||
bytes[32] = 0x82;
|
||||
bytes.writeUIntBE(0xabcdef, 33, 3);
|
||||
for (const [object, offset] of [8, 25, 32].entries())
|
||||
bytes.writeUIntBE(offset, tableOffset + object * 3, 3);
|
||||
const trailer = bytes.length - 32;
|
||||
bytes[trailer + 6] = 3;
|
||||
bytes[trailer + 7] = 3;
|
||||
bytes.writeBigUInt64BE(3n, trailer + 8);
|
||||
bytes.writeBigUInt64BE(1n, trailer + 16);
|
||||
bytes.writeBigUInt64BE(BigInt(tableOffset), trailer + 24);
|
||||
const original = Buffer.from(bytes);
|
||||
const plan = estimateBinaryPlistExpansion(bytes, 1024 * 1024, {
|
||||
numberProjection: true,
|
||||
uidProjection: true,
|
||||
});
|
||||
const metadata = scanBinaryPlistScalars(bytes, plan);
|
||||
expect(metadata.wideIntegers).toEqual([
|
||||
{ offset: 8, value: 18446744073709551618n },
|
||||
]);
|
||||
expect(metadata.literals.incomplete).toBe(false);
|
||||
const decoded = decodeBinaryPlistValue(bytes, plan, metadata) as unknown[];
|
||||
expect(
|
||||
createPlistNumberProjection(metadata.literals).project(
|
||||
projectPlistValue(decoded[0]).value,
|
||||
),
|
||||
).toEqual({
|
||||
$plist_type: "integer",
|
||||
decimal: "18446744073709551618",
|
||||
});
|
||||
expect(plistUidValue(decoded[1])).toBe(0xabcdef);
|
||||
expect(decoded).toHaveLength(2);
|
||||
expect(bytes.equals(original)).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps uint64 wrapping for nine-byte offset and reference declarations", () => {
|
||||
const bytes = Buffer.alloc(70);
|
||||
bytes.write("bplist00");
|
||||
bytes.set([0x10, 7, 0xa1], 8);
|
||||
bytes[11] = 0xff;
|
||||
bytes[20] = 0xff;
|
||||
bytes.writeBigUInt64BE(8n, 21);
|
||||
bytes[29] = 0xff;
|
||||
bytes.writeBigUInt64BE(10n, 30);
|
||||
const trailer = 38;
|
||||
bytes[trailer + 6] = 9;
|
||||
bytes[trailer + 7] = 9;
|
||||
bytes.writeBigUInt64BE(2n, trailer + 8);
|
||||
bytes.writeBigUInt64BE(1n, trailer + 16);
|
||||
bytes.writeBigUInt64BE(20n, trailer + 24);
|
||||
const original = Buffer.from(bytes);
|
||||
expect(decode(bytes)).toEqual([7]);
|
||||
expect(bytes.equals(original)).toBe(true);
|
||||
});
|
||||
@@ -1,7 +1,7 @@
|
||||
import { readBinaryPlistLayout } from "./BinaryPlistLayout.js";
|
||||
import { parseBinary } from "plist";
|
||||
import { markPlistDictionary } from "../../domain/apple/plistUid.js";
|
||||
import {
|
||||
binaryPlistObjectOffsets,
|
||||
decodeBinaryPlistScalars,
|
||||
scanBinaryPlistScalars,
|
||||
type BinaryPlistScalarMetadata,
|
||||
@@ -13,9 +13,14 @@ const scalarArrayInput = (
|
||||
bytes: Buffer,
|
||||
objects: readonly number[],
|
||||
): Buffer => {
|
||||
const trailer = bytes.length - 32;
|
||||
const referenceWidth = bytes[trailer + 7]!;
|
||||
const objectCount = Number(bytes.readBigUInt64BE(trailer + 8));
|
||||
// Normalization may relocate objects and change both widths. These facts
|
||||
// belong to the compatible decoder copy, not the original source plan.
|
||||
const compatibleLayoutResult = readBinaryPlistLayout(bytes);
|
||||
if (!compatibleLayoutResult.ok)
|
||||
throw new RangeError("binary plist trailer is truncated");
|
||||
const compatibleLayout = compatibleLayoutResult.value;
|
||||
const referenceWidth = compatibleLayout.referenceWidth;
|
||||
const objectCount = Number(compatibleLayout.objectCount);
|
||||
const rootOffset = bytes.length;
|
||||
const rootLength = 10 + objects.length * referenceWidth;
|
||||
const tableOffset = rootOffset + rootLength;
|
||||
@@ -33,7 +38,7 @@ const scalarArrayInput = (
|
||||
}
|
||||
}
|
||||
let object = 0;
|
||||
for (const offset of binaryPlistObjectOffsets(bytes)) {
|
||||
for (const offset of compatibleLayout.objectOffsets()) {
|
||||
output.writeBigUInt64BE(offset, tableOffset + object * 8);
|
||||
object += 1;
|
||||
}
|
||||
|
||||
@@ -20,7 +20,8 @@ import {
|
||||
} from "../ArtifactFileOperation.js";
|
||||
import type { ArtifactResourceScope } from "../ArtifactResourceScope.js";
|
||||
import { readArtifactFileChunks } from "../readArtifactFileChunks.js";
|
||||
import { hasMachoMagic, readMachoImage } from "./MachoLoadCommandReader.js";
|
||||
import { hasMachoMagic } from "../../domain/apple/machoContainer.js";
|
||||
import { readMachoImage } from "./MachoLoadCommandReader.js";
|
||||
|
||||
/** One owner for dyld descriptor admission, physical identity and finalization. */
|
||||
export class DylibFileAccess {
|
||||
|
||||
@@ -27,9 +27,9 @@ import {
|
||||
MAX_LAZY_INFO_DEPENDENCIES,
|
||||
MAX_LAZY_INFO_SYMBOL_REFERENCES,
|
||||
MAX_LOAD_COMMAND_BYTES,
|
||||
hasMachoMagic,
|
||||
readMachoImage,
|
||||
} from "./MachoLoadCommandReader.js";
|
||||
import { hasMachoMagic } from "../../domain/apple/machoContainer.js";
|
||||
import { traceDylibResolution } from "./DylibResolutionReader.js";
|
||||
import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js";
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
|
||||
@@ -22,8 +22,11 @@ import {
|
||||
} from "../../domain/apple/machoPlatform.js";
|
||||
import {
|
||||
classifyFatContainerIdentity,
|
||||
hasMachoMagic,
|
||||
readFatContainerHeader,
|
||||
readFatHeaderPrefix,
|
||||
readFatSliceDeclarations,
|
||||
readMachoHeaderPrefixFacts,
|
||||
validateFatSlice,
|
||||
validateFatSliceIdentity,
|
||||
type FatSliceDeclaration,
|
||||
@@ -46,14 +49,6 @@ export const MAX_LAZY_INFO_DEPENDENCIES = 1024;
|
||||
* offset, so stop before that map and those decodes grow.
|
||||
*/
|
||||
export const MAX_LAZY_INFO_SYMBOL_REFERENCES = 64 * 1024;
|
||||
const MH_MAGIC = 0xfeedface;
|
||||
const MH_MAGIC_64 = 0xfeedfacf;
|
||||
const MH_CIGAM = 0xcefaedfe;
|
||||
const MH_CIGAM_64 = 0xcffaedfe;
|
||||
const FAT_MAGIC = 0xcafebabe;
|
||||
const FAT_MAGIC_64 = 0xcafebabf;
|
||||
const FAT_CIGAM = 0xbebafeca;
|
||||
const FAT_CIGAM_64 = 0xbfbafeca;
|
||||
|
||||
const LC_SEGMENT = 0x1;
|
||||
const LC_SEGMENT_64 = 0x19;
|
||||
@@ -114,26 +109,6 @@ class MachoFormatIssue extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
/** Report whether the first bytes carry a Mach-O or FAT magic number. */
|
||||
export const hasMachoMagic = (head: Uint8Array): boolean => {
|
||||
if (head.byteLength < 4) return false;
|
||||
const magic = new DataView(
|
||||
head.buffer,
|
||||
head.byteOffset,
|
||||
head.byteLength,
|
||||
).getUint32(0, false);
|
||||
return [
|
||||
MH_MAGIC,
|
||||
MH_MAGIC_64,
|
||||
MH_CIGAM,
|
||||
MH_CIGAM_64,
|
||||
FAT_MAGIC,
|
||||
FAT_MAGIC_64,
|
||||
FAT_CIGAM,
|
||||
FAT_CIGAM_64,
|
||||
].includes(magic);
|
||||
};
|
||||
|
||||
/** Decode every slice's dylib-loading load commands without reading code. */
|
||||
export const readMachoImage = async (
|
||||
readAt: ReadAt,
|
||||
@@ -146,10 +121,13 @@ export const readMachoImage = async (
|
||||
return hasMachoMagic(head)
|
||||
? { status: "malformed", reason: "Mach-O header is truncated" }
|
||||
: { status: "not-mach-o" };
|
||||
const view = viewOf(head);
|
||||
const magic = view.getUint32(0, false);
|
||||
const fatPrefix = readFatHeaderPrefix(head);
|
||||
const mayBeJvmClass =
|
||||
fatPrefix.state === "parsed" &&
|
||||
!fatPrefix.wide &&
|
||||
!fatPrefix.littleEndian;
|
||||
let header: ReturnType<typeof readFatContainerHeader>;
|
||||
if (magic === FAT_MAGIC && size <= MAX_JVM_CLASS_IDENTITY_BYTES) {
|
||||
if (mayBeJvmClass && size <= MAX_JVM_CLASS_IDENTITY_BYTES) {
|
||||
const complete = await readAt(0, size);
|
||||
header =
|
||||
complete.byteLength === size
|
||||
@@ -187,7 +165,7 @@ export const readMachoImage = async (
|
||||
// A large CAFEBABE input whose bounded prefix resembles an invalid
|
||||
// known-CPU FAT table remains unknown until its JVM identity can be
|
||||
// checked; do not turn the resource limit into a malformed claim.
|
||||
if (magic === FAT_MAGIC && size > MAX_JVM_CLASS_IDENTITY_BYTES)
|
||||
if (mayBeJvmClass && size > MAX_JVM_CLASS_IDENTITY_BYTES)
|
||||
throw new MachoFormatIssue(
|
||||
"unsupported",
|
||||
JVM_CLASS_IDENTITY_LIMIT_REASON,
|
||||
@@ -203,9 +181,10 @@ export const readMachoImage = async (
|
||||
...(hiddenSlice === null ? {} : { hiddenSlices: [hiddenSlice] }),
|
||||
};
|
||||
}
|
||||
if (magic === MH_CIGAM || magic === MH_CIGAM_64)
|
||||
const thin = readMachoHeaderPrefixFacts(head).format;
|
||||
if (thin?.littleEndian === true)
|
||||
return { status: "parsed", slices: [await readSlice(readAt, 0, size)] };
|
||||
if (magic === MH_MAGIC || magic === MH_MAGIC_64)
|
||||
if (thin !== null)
|
||||
throw new MachoFormatIssue(
|
||||
"unsupported",
|
||||
"big-endian Mach-O images are not supported",
|
||||
|
||||
@@ -488,7 +488,15 @@ describe("binary plist shared object-table entries", () => {
|
||||
const table = Number(bytes.readBigUInt64BE(bytes.length - 8));
|
||||
bytes[table + 2] = 7;
|
||||
expect(bytes.subarray(0, 8).toString("ascii")).toBe("bplist00");
|
||||
expect(scanBinaryPlistScalars(bytes).literals.incomplete).toBe(true);
|
||||
const metadata = scanBinaryPlistScalars(bytes);
|
||||
expect(metadata.literals.incomplete).toBe(true);
|
||||
expect(metadata.literals.integers).toEqual(
|
||||
new Map([
|
||||
[9007199254740992, new Set(["9007199254740993"])],
|
||||
[Number(1152921504606846979n), new Set(["1152921504606846979"])],
|
||||
]),
|
||||
);
|
||||
expect(metadata.literals.unsafeIntegerLiterals).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -586,3 +594,20 @@ describe("binary plist scalar clone budgeting", () => {
|
||||
expect(bytes.readDoubleBE(14)).toBe(0.1005);
|
||||
});
|
||||
});
|
||||
|
||||
it.each([4n, 0xffffffffffffffffn])(
|
||||
"scans source scalars independently of invalid top object %s",
|
||||
(topObject) => {
|
||||
const bytes = sharedScalarEntries(int64(9007199254740993n));
|
||||
const expected = scanBinaryPlistScalars(bytes);
|
||||
bytes.writeBigUInt64BE(topObject, bytes.length - 16);
|
||||
expect(scanBinaryPlistScalars(bytes)).toEqual(expected);
|
||||
expect(expected.literals.incomplete).toBe(false);
|
||||
expect(expected.literals.unsafeIntegerLiterals).toBe(2);
|
||||
expect(() =>
|
||||
estimateBinaryPlistExpansion(bytes, 1024 * 1024, {
|
||||
numberProjection: true,
|
||||
}),
|
||||
).toThrow("binary plist trailer or object table is invalid");
|
||||
},
|
||||
);
|
||||
@@ -1,3 +1,8 @@
|
||||
import {
|
||||
readBinaryPlistLayout,
|
||||
readBinaryPlistUnsigned64,
|
||||
type BinaryPlistLayout,
|
||||
} from "./BinaryPlistLayout.js";
|
||||
import {
|
||||
createPlistNumberCollector,
|
||||
type PlistNumberLiterals,
|
||||
@@ -196,10 +201,7 @@ export const binaryPlistObjectEnd = (
|
||||
const width = 1 << (extendedMarker & 0x0f);
|
||||
if (width > 8 || cursor + 1 + width > objectEnd) return undefined;
|
||||
cursor += 1;
|
||||
let extendedLength = 0n;
|
||||
for (let index = 0; index < width; index += 1)
|
||||
extendedLength =
|
||||
(extendedLength << 8n) | BigInt(bytes[cursor + index] ?? 0);
|
||||
const extendedLength = readBinaryPlistUnsigned64(bytes, cursor, width);
|
||||
if (extendedLength > BigInt(Number.MAX_SAFE_INTEGER)) return undefined;
|
||||
length = Number(extendedLength);
|
||||
cursor += width;
|
||||
@@ -247,11 +249,17 @@ export const scanBinaryPlistScalars = (
|
||||
});
|
||||
if (bytes.length < 40 || !hasBinaryPlistSignature(bytes)) return result(true);
|
||||
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
||||
const trailer = bytes.length - 32;
|
||||
const offsetWidth = bytes[trailer + 6] ?? 0;
|
||||
const sourceLayoutResult = readBinaryPlistLayout(bytes);
|
||||
if (!sourceLayoutResult.ok) return result(true);
|
||||
const sourceLayout = sourceLayoutResult.value;
|
||||
const {
|
||||
trailerOffset: trailer,
|
||||
offsetWidth,
|
||||
referenceWidth,
|
||||
objectCount: countRaw,
|
||||
offsetTable: tableRaw,
|
||||
} = sourceLayout;
|
||||
if (offsetWidth < 1) return result(true);
|
||||
const countRaw = view.getBigUint64(trailer + 8);
|
||||
const tableRaw = view.getBigUint64(trailer + 24);
|
||||
const safeMaximum = BigInt(Number.MAX_SAFE_INTEGER);
|
||||
if (countRaw === 0n || countRaw > safeMaximum || tableRaw > safeMaximum)
|
||||
return result(true);
|
||||
@@ -260,23 +268,12 @@ export const scanBinaryPlistScalars = (
|
||||
const capacity = Math.floor((trailer - offsetTable) / offsetWidth);
|
||||
if (countRaw > BigInt(capacity)) return result(true);
|
||||
const count = Number(countRaw);
|
||||
const referenceWidth = bytes[trailer + 7] ?? 0;
|
||||
if (referenceWidth < 1) return result(true);
|
||||
const unsigned = (offset: number): number | undefined => {
|
||||
if (offset < offsetTable || offset + offsetWidth > trailer)
|
||||
return undefined;
|
||||
let value = 0n;
|
||||
for (let index = 0; index < offsetWidth; index += 1)
|
||||
value = BigInt.asUintN(
|
||||
64,
|
||||
(value << 8n) | BigInt(bytes[offset + index] ?? 0),
|
||||
);
|
||||
return value > safeMaximum ? undefined : Number(value);
|
||||
};
|
||||
const objectOffsets: number[] = [];
|
||||
let incomplete = false;
|
||||
for (let index = 0; index < count; index += 1) {
|
||||
const offset = unsigned(offsetTable + index * offsetWidth);
|
||||
const rawOffset = sourceLayout.objectOffset(index);
|
||||
const offset = rawOffset > safeMaximum ? undefined : Number(rawOffset);
|
||||
if (offset === undefined || offset < 8 || offset >= offsetTable) {
|
||||
incomplete = true;
|
||||
continue;
|
||||
@@ -405,8 +402,11 @@ export const decodeBinaryPlistScalars = (
|
||||
replacement.writeDoubleBE(Number(value), 1);
|
||||
replacements.set(offset, replacement);
|
||||
}
|
||||
const trailerOffset = bytes.length - 32;
|
||||
const sourceReferenceSize = bytes[trailerOffset + 7] ?? 0;
|
||||
const sourceLayoutResult = readBinaryPlistLayout(bytes);
|
||||
if (!sourceLayoutResult.ok)
|
||||
throw new RangeError("binary plist trailer is truncated");
|
||||
const sourceLayout = sourceLayoutResult.value;
|
||||
const sourceReferenceSize = sourceLayout.referenceWidth;
|
||||
const normalizedReferenceSize =
|
||||
scalarPlan.normalizedReferenceSize ?? sourceReferenceSize;
|
||||
for (const container of scalarPlan.containersToNormalize ?? [])
|
||||
@@ -415,8 +415,8 @@ export const decodeBinaryPlistScalars = (
|
||||
binaryPlistContainerRecord(bytes, container, normalizedReferenceSize),
|
||||
);
|
||||
const needsTableNormalization =
|
||||
(scalarPlan.normalizedOffsetSize ?? bytes[trailerOffset + 6] ?? 0) !==
|
||||
(bytes[trailerOffset + 6] ?? 0) ||
|
||||
(scalarPlan.normalizedOffsetSize ?? sourceLayout.offsetWidth) !==
|
||||
sourceLayout.offsetWidth ||
|
||||
normalizedReferenceSize !== sourceReferenceSize;
|
||||
if (
|
||||
reachableWideIntegers.length === 0 &&
|
||||
@@ -430,8 +430,9 @@ export const decodeBinaryPlistScalars = (
|
||||
? Buffer.from(bytes)
|
||||
: cloneBinaryPlistObjects(
|
||||
bytes,
|
||||
sourceLayout,
|
||||
replacements,
|
||||
scalarPlan.normalizedOffsetSize ?? bytes[trailerOffset + 6] ?? 0,
|
||||
scalarPlan.normalizedOffsetSize ?? sourceLayout.offsetWidth,
|
||||
normalizedReferenceSize,
|
||||
);
|
||||
for (const { offset, value } of reachableWideIntegers) {
|
||||
@@ -450,34 +451,18 @@ export const decodeBinaryPlistScalars = (
|
||||
return compatible;
|
||||
};
|
||||
|
||||
/** Iterate source offset declarations using the same uint64 interpretation as normalization. */
|
||||
export function* binaryPlistObjectOffsets(bytes: Buffer): Generator<bigint> {
|
||||
const trailer = bytes.length - 32;
|
||||
const width = bytes[trailer + 6] ?? 0;
|
||||
const count = Number(bytes.readBigUInt64BE(trailer + 8));
|
||||
const table = Number(bytes.readBigUInt64BE(trailer + 24));
|
||||
for (let object = 0; object < count; object += 1) {
|
||||
let offset = 0n;
|
||||
for (let byte = 0; byte < width; byte += 1)
|
||||
offset = BigInt.asUintN(
|
||||
64,
|
||||
(offset << 8n) | BigInt(bytes[table + object * width + byte] ?? 0),
|
||||
);
|
||||
yield offset;
|
||||
}
|
||||
}
|
||||
|
||||
const cloneBinaryPlistObjects = (
|
||||
bytes: Buffer,
|
||||
sourceLayout: BinaryPlistLayout,
|
||||
replacements: ReadonlyMap<number, Buffer>,
|
||||
normalizedOffsetSize?: number,
|
||||
normalizedReferenceSize?: number,
|
||||
): Buffer => {
|
||||
const trailerOffset = bytes.length - 32;
|
||||
const offsetSize = bytes[trailerOffset + 6] ?? 0;
|
||||
const objectCount = Number(bytes.readBigUInt64BE(trailerOffset + 8));
|
||||
const oldTableOffset = Number(bytes.readBigUInt64BE(trailerOffset + 24));
|
||||
const offsets = [...binaryPlistObjectOffsets(bytes)];
|
||||
const trailerOffset = sourceLayout.trailerOffset;
|
||||
const offsetSize = sourceLayout.offsetWidth;
|
||||
const objectCount = Number(sourceLayout.objectCount);
|
||||
const oldTableOffset = Number(sourceLayout.offsetTable);
|
||||
const offsets = [...sourceLayout.objectOffsets()];
|
||||
const cloneEntries = [...replacements.entries()];
|
||||
const replacementLengths = cloneEntries.map(([, record]) => record.length);
|
||||
const newOffsetSize =
|
||||
|
||||
@@ -1,3 +1,7 @@
|
||||
import {
|
||||
readBinaryPlistLayout,
|
||||
readBinaryPlistUnsigned64,
|
||||
} from "./BinaryPlistLayout.js";
|
||||
import { decodeXmlPlistText } from "../../domain/propertyListXmlText.js";
|
||||
import { hasBinaryPlistSignature } from "../../domain/apple/appleArchiveFormat.js";
|
||||
import {
|
||||
@@ -265,24 +269,26 @@ export const estimateBinaryPlistExpansion = (
|
||||
): PlistDecodePreflight => {
|
||||
if (bytes.length < 40)
|
||||
throw new TypeError("binary plist trailer is truncated");
|
||||
const trailer = bytes.length - 32;
|
||||
const offsetSize = bytes[trailer + 6] ?? 0;
|
||||
const referenceSize = bytes[trailer + 7] ?? 0;
|
||||
const sourceLayoutResult = readBinaryPlistLayout(bytes);
|
||||
if (!sourceLayoutResult.ok)
|
||||
throw new TypeError("binary plist trailer is truncated");
|
||||
const sourceLayout = sourceLayoutResult.value;
|
||||
const {
|
||||
trailerOffset: trailer,
|
||||
offsetWidth: offsetSize,
|
||||
referenceWidth: referenceSize,
|
||||
} = sourceLayout;
|
||||
if (offsetSize < 1 || referenceSize < 1)
|
||||
throw new TypeError("binary plist trailer integer sizes are invalid");
|
||||
const safeDeclaration = (value: bigint): number =>
|
||||
value <= BigInt(Number.MAX_SAFE_INTEGER) ? Number(value) : -1;
|
||||
const readInteger = (offset: number, size: number): number => {
|
||||
if (offset < 0 || size < 1 || offset + size > bytes.length) return -1;
|
||||
let value = 0n;
|
||||
for (let index = 0; index < size; index += 1)
|
||||
value = BigInt.asUintN(
|
||||
64,
|
||||
(value << 8n) | BigInt(bytes[offset + index] ?? 0),
|
||||
);
|
||||
return value <= BigInt(Number.MAX_SAFE_INTEGER) ? Number(value) : -1;
|
||||
return safeDeclaration(readBinaryPlistUnsigned64(bytes, offset, size));
|
||||
};
|
||||
const objectCount = readInteger(trailer + 8, 8);
|
||||
const topObject = readInteger(trailer + 16, 8);
|
||||
const offsetTable = readInteger(trailer + 24, 8);
|
||||
const objectCount = safeDeclaration(sourceLayout.objectCount);
|
||||
const topObject = safeDeclaration(sourceLayout.topObject);
|
||||
const offsetTable = safeDeclaration(sourceLayout.offsetTable);
|
||||
const tableEnd = bounds === "strict" ? trailer : bytes.length;
|
||||
const objectsEnd = bounds === "strict" ? offsetTable : bytes.length;
|
||||
if (
|
||||
@@ -341,7 +347,7 @@ export const estimateBinaryPlistExpansion = (
|
||||
const readDictionaryKey = (object: number): string => {
|
||||
const prior = dictionaryKeys.get(object);
|
||||
if (prior !== undefined) return prior;
|
||||
const offset = readInteger(offsetTable + object * offsetSize, offsetSize);
|
||||
const offset = safeDeclaration(sourceLayout.objectOffset(object));
|
||||
if (offset < 8 || offset >= objectsEnd)
|
||||
throw new TypeError("binary plist object offset is invalid");
|
||||
const marker = bytes[offset] ?? 0;
|
||||
@@ -403,10 +409,7 @@ export const estimateBinaryPlistExpansion = (
|
||||
"expansion",
|
||||
`binary plist reference expansion exceeds ${budget}`,
|
||||
);
|
||||
const offset = readInteger(
|
||||
offsetTable + entry.object * offsetSize,
|
||||
offsetSize,
|
||||
);
|
||||
const offset = safeDeclaration(sourceLayout.objectOffset(entry.object));
|
||||
if (offset < 8 || offset >= objectsEnd)
|
||||
throw new TypeError("binary plist object offset is invalid");
|
||||
if (numberProjection) reachableObjectOffsets.add(offset);
|
||||
|
||||
@@ -129,7 +129,9 @@ it("preserves explicit unknown slice evidence instead of falling back to retaine
|
||||
const header = readFatContainerHeader(bytes, bytes.length);
|
||||
if (header.state !== "parsed") throw new Error("Expected FAT header");
|
||||
const offset = header.slices[0]!.offset;
|
||||
const unknown = new Map([[offset, { magic: null, identity: null }]]);
|
||||
const unknown = new Map([
|
||||
[offset, readMachoHeaderPrefixFacts(Buffer.alloc(0))],
|
||||
]);
|
||||
expect(classifyFatContainerEvidence(header, bytes, new Map())).toEqual({
|
||||
state: "mach-o",
|
||||
});
|
||||
@@ -258,6 +260,7 @@ describe("FAT Mach-O identity", () => {
|
||||
bigEndian.writeUInt32BE(CPU.x86_64.subtype, 8);
|
||||
expect(readMachoHeaderPrefixFacts(littleEndian.subarray(0, 12))).toEqual({
|
||||
magic: 0xcffaedfe,
|
||||
format: { wide: true, littleEndian: true, headerSize: 32 },
|
||||
identity: {
|
||||
cpuType: CPU.arm64.type,
|
||||
cpuSubtype: CPU.arm64.subtype,
|
||||
@@ -265,6 +268,7 @@ describe("FAT Mach-O identity", () => {
|
||||
});
|
||||
expect(readMachoHeaderPrefixFacts(bigEndian)).toEqual({
|
||||
magic: 0xfeedfacf,
|
||||
format: { wide: true, littleEndian: false, headerSize: 32 },
|
||||
identity: {
|
||||
cpuType: CPU.x86_64.type,
|
||||
cpuSubtype: CPU.x86_64.subtype,
|
||||
@@ -272,10 +276,12 @@ describe("FAT Mach-O identity", () => {
|
||||
});
|
||||
expect(readMachoHeaderPrefixFacts(littleEndian.subarray(0, 11))).toEqual({
|
||||
magic: 0xcffaedfe,
|
||||
format: { wide: true, littleEndian: true, headerSize: 32 },
|
||||
identity: null,
|
||||
});
|
||||
expect(readMachoHeaderPrefixFacts(littleEndian.subarray(0, 3))).toEqual({
|
||||
magic: null,
|
||||
format: null,
|
||||
identity: null,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -32,6 +32,7 @@ import { parseMzHeaderEvidence } from "../../domain/dosMz.js";
|
||||
import {
|
||||
classifyFatContainerEvidence,
|
||||
readFatContainerHeader,
|
||||
readMachoHeaderPrefixFacts,
|
||||
} from "../../domain/apple/machoContainer.js";
|
||||
|
||||
/** Mutable internal occurrence used until root-bound IDs are known. */
|
||||
@@ -581,9 +582,7 @@ export const classifyArtifactBytes = (
|
||||
const fatFormat = classifyFatArtifact(evidence, fileSize);
|
||||
if (fatFormat !== null) return fatFormat;
|
||||
if (prefix.length >= 4) {
|
||||
const magic = prefix.readUInt32BE(0);
|
||||
if ([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(magic))
|
||||
return "mach-o";
|
||||
if (readMachoHeaderPrefixFacts(prefix).format !== null) return "mach-o";
|
||||
if (prefix.subarray(0, 4).equals(Buffer.from([0x7f, 0x45, 0x4c, 0x46])))
|
||||
return "elf";
|
||||
}
|
||||
|
||||
@@ -17,7 +17,11 @@ import type { FileHandle } from "node:fs/promises";
|
||||
|
||||
import { abortIfNeeded } from "../ArtifactHash.js";
|
||||
import { ArtifactReaderFailure } from "../ArtifactReader.js";
|
||||
import { readFatContainerHeader } from "../../domain/apple/machoContainer.js";
|
||||
import {
|
||||
readFatContainerHeader,
|
||||
readFatHeaderPrefix,
|
||||
readMachoHeaderPrefixFacts,
|
||||
} from "../../domain/apple/machoContainer.js";
|
||||
import { readElfFixedHeader } from "../../domain/elfFixedHeader.js";
|
||||
import { readMachoLoadCommandTableHeader } from "../../domain/apple/machoLoadCommands.js";
|
||||
import {
|
||||
@@ -52,9 +56,6 @@ const FUSE_LENGTH = "NODE_SEA_FUSE_".length + 32 + 2;
|
||||
// PE header structures must fit this prefix for the bounded name probe.
|
||||
const PE_HEADER_PROBE_BYTES = 64 * 1024;
|
||||
|
||||
const THIN_MAGICS = new Set([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe]);
|
||||
const FAT_MAGICS = new Set([0xcafebabe, 0xcafebabf, 0xbebafeca, 0xbfbafeca]);
|
||||
|
||||
/** One injected blob inside a native carrier. Offsets are carrier file offsets. */
|
||||
export interface NodeSeaLocation {
|
||||
readonly carrier: "mach-o" | "elf" | "pe";
|
||||
@@ -96,11 +97,13 @@ export const locateNodeSea = async (
|
||||
const carrier: Carrier = { handle, fileSize, signal };
|
||||
const prefix = await readExact(carrier, 0, Math.min(fileSize, 64));
|
||||
if (prefix === undefined || prefix.length < 4) return undefined;
|
||||
const magic = prefix.readUInt32BE(0);
|
||||
if (THIN_MAGICS.has(magic) || FAT_MAGICS.has(magic)) {
|
||||
const ranges = THIN_MAGICS.has(magic)
|
||||
? await locateThinMachO(carrier, 0, fileSize)
|
||||
: await locateFatMachO(carrier, prefix);
|
||||
const thin = readMachoHeaderPrefixFacts(prefix).format;
|
||||
const fat = readFatHeaderPrefix(prefix);
|
||||
if (thin !== null || fat.state === "incomplete" || fat.state === "parsed") {
|
||||
const ranges =
|
||||
thin !== null
|
||||
? await locateThinMachO(carrier, 0, fileSize)
|
||||
: await locateFatMachO(carrier, prefix);
|
||||
return collapseRanges(carrier, "mach-o", ranges);
|
||||
}
|
||||
if (prefix.subarray(0, 4).equals(Buffer.from([0x7f, 0x45, 0x4c, 0x46])))
|
||||
@@ -216,7 +219,7 @@ const locateFatMachO = async (
|
||||
header = readFatContainerHeader(bytes, carrier.fileSize);
|
||||
}
|
||||
if (header.state !== "parsed") {
|
||||
if (bytes.length >= 4 && THIN_MAGICS.has(bytes.readUInt32BE(0)))
|
||||
if (readMachoHeaderPrefixFacts(bytes).format !== null)
|
||||
return locateThinMachO(carrier, 0, carrier.fileSize);
|
||||
return [];
|
||||
}
|
||||
@@ -588,8 +591,9 @@ const locatePeResources = async (carrier: Carrier): Promise<BlobRange[]> => {
|
||||
if (bytes === undefined) throw formatFailure(NODE_SEA_TRUNCATED);
|
||||
// A native-looking file need not be an admitted PE image. Once the layout
|
||||
// is admitted, resource errors remain failures rather than inferred absence.
|
||||
let layout: ReturnType<typeof readPeResourceLayout>;
|
||||
try {
|
||||
readPeResourceLayout(bytes);
|
||||
layout = readPeResourceLayout(bytes);
|
||||
} catch (cause) {
|
||||
if (cause instanceof ArtifactReaderFailure && cause.reason === "format")
|
||||
return [];
|
||||
@@ -608,6 +612,7 @@ const locatePeResources = async (carrier: Carrier): Promise<BlobRange[]> => {
|
||||
max_entries: MAX_INSPECT_PE_DIRECTORY_ENTRIES,
|
||||
},
|
||||
carrier.signal,
|
||||
layout,
|
||||
);
|
||||
return resources.resources.flatMap((resource) => {
|
||||
if (
|
||||
|
||||
@@ -4,9 +4,8 @@ import type {
|
||||
} from "../../domain/artifactFormatMetadata.js";
|
||||
import type { CArchiveEntry, PyInstallerCookie } from "./CArchiveLayout.js";
|
||||
import type { PyzArchive, PyzEntry } from "./PyzArchive.js";
|
||||
import { pythonVersionForMagic } from "./PythonBytecode.js";
|
||||
|
||||
const pythonVersion = (value: number) =>
|
||||
const cookiePythonVersion = (value: number) =>
|
||||
value >= 100
|
||||
? { major: Math.floor(value / 100), minor: value % 100 }
|
||||
: value >= 20
|
||||
@@ -18,7 +17,7 @@ export const cookieMetadata = (
|
||||
entries: readonly CArchiveEntry[],
|
||||
encryptionMarker: boolean,
|
||||
): PyInstallerMetadata => {
|
||||
const version = pythonVersion(cookie.pythonVersion);
|
||||
const version = cookiePythonVersion(cookie.pythonVersion);
|
||||
return {
|
||||
kind: "pyinstaller",
|
||||
layout_version: cookie.layout,
|
||||
@@ -45,13 +44,13 @@ export const scriptPythonMagic = (
|
||||
cookie: PyInstallerCookie,
|
||||
archives: Iterable<PyzArchive>,
|
||||
): Buffer | undefined => {
|
||||
const version = pythonVersion(cookie.pythonVersion);
|
||||
const version = cookiePythonVersion(cookie.pythonVersion);
|
||||
if (version === undefined) return undefined;
|
||||
let selected: Buffer | undefined;
|
||||
for (const archive of archives) {
|
||||
const magic = archive.pythonMagic;
|
||||
if (
|
||||
pythonVersionForMagic(magic) !==
|
||||
archive.pythonVersion !==
|
||||
`${String(version.major)}.${String(version.minor)}` ||
|
||||
(selected !== undefined && !selected.equals(magic))
|
||||
)
|
||||
@@ -123,10 +122,7 @@ export const cArchiveEntryMetadata = (
|
||||
directory_record_size: entry.tocRecordLength,
|
||||
pyz_sha256: archive?.sha256 ?? null,
|
||||
pyz_python_magic: archive?.pythonMagic.toString("hex") ?? null,
|
||||
pyz_python_version:
|
||||
archive === undefined
|
||||
? null
|
||||
: (pythonVersionForMagic(archive.pythonMagic) ?? null),
|
||||
pyz_python_version: archive?.pythonVersion ?? null,
|
||||
pyz_entry_count: archive?.kind === "entries" ? archive.entries.length : null,
|
||||
encryption_source:
|
||||
archive !== undefined && encryptionMarker ? "carchive-marker" : null,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { open, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
@@ -31,6 +32,7 @@ it("lists modules from a supported PYZ and leaves an unknown magic opaque", asyn
|
||||
if (!opaque.ok || opaque.value.kind !== "opaque") return;
|
||||
expect(Buffer.compare(opaque.value.pythonMagic, Buffer.alloc(4))).toBe(0);
|
||||
expect(opaque.value.limitation).toContain("left opaque");
|
||||
expect(opaque.value.pythonVersion).toBeUndefined();
|
||||
});
|
||||
|
||||
it("leaves a PYZ opaque when the table uses an unsupported marshal type", async () => {
|
||||
@@ -44,6 +46,10 @@ it("leaves a PYZ opaque when the table uses an unsupported marshal type", async
|
||||
if (!parsed.ok || parsed.value.kind !== "opaque") return;
|
||||
expect(parsed.value.limitation).toContain("left opaque");
|
||||
expect(parsed.value.pythonMagic).toEqual(PYTHON_39_MAGIC);
|
||||
expect(parsed.value.pythonVersion).toBe("3.9");
|
||||
expect(parsed.value.sha256).toBe(
|
||||
createHash("sha256").update(image).digest("hex"),
|
||||
);
|
||||
});
|
||||
|
||||
it("reports an entry that extends outside the PYZ as an integrity error", async () => {
|
||||
@@ -58,26 +64,29 @@ it("reports an entry that extends outside the PYZ as an integrity error", async
|
||||
expect(parsed.error.reason).toBe("integrity");
|
||||
});
|
||||
|
||||
it("rejects an oversized sparse PYZ table before allocating its bytes", async () => {
|
||||
const root = await createTestTempDirectory("rea-pyz-table-budget-");
|
||||
const path = join(root, "image.pyz");
|
||||
const header = Buffer.alloc(12);
|
||||
header.write("PYZ\0");
|
||||
PYTHON_39_MAGIC.copy(header, 4);
|
||||
header.writeUInt32BE(12, 8);
|
||||
await writeFile(path, header);
|
||||
const handle = await open(path, "r+");
|
||||
try {
|
||||
const size = 12 + MAX_PYZ_TABLE_BYTES + 1;
|
||||
await handle.truncate(size);
|
||||
expect(await readPyzArchive(handle, 0, size)).toMatchObject({
|
||||
ok: false,
|
||||
error: { reason: "limit" },
|
||||
});
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
});
|
||||
it.each(["PYZ\0", "BAD\0"])(
|
||||
"rejects an oversized sparse table before deferred signature rejection (%s)",
|
||||
async (signature) => {
|
||||
const root = await createTestTempDirectory("rea-pyz-table-budget-");
|
||||
const path = join(root, "image.pyz");
|
||||
const header = Buffer.alloc(12);
|
||||
header.write(signature);
|
||||
PYTHON_39_MAGIC.copy(header, 4);
|
||||
header.writeUInt32BE(12, 8);
|
||||
await writeFile(path, header);
|
||||
const handle = await open(path, "r+");
|
||||
try {
|
||||
const size = 12 + MAX_PYZ_TABLE_BYTES + 1;
|
||||
await handle.truncate(size);
|
||||
expect(await readPyzArchive(handle, 0, size)).toMatchObject({
|
||||
ok: false,
|
||||
error: { reason: "limit" },
|
||||
});
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("rejects a PYZ table whose back-references repeat one long row", async () => {
|
||||
// One namespace row with a 16 KiB name, flagged for reference, then 999
|
||||
@@ -124,14 +133,52 @@ const int32 = (value: number): Buffer => {
|
||||
return bytes;
|
||||
};
|
||||
|
||||
const readImage = async (bytes: Buffer) => {
|
||||
const readImage = async (bytes: Buffer, declaredSize = bytes.length) => {
|
||||
const root = await createTestTempDirectory("rea-pyz-file-");
|
||||
const path = join(root, "image.pyz");
|
||||
await writeFile(path, bytes);
|
||||
const handle = await open(path, "r");
|
||||
try {
|
||||
return await readPyzArchive(handle, 0, bytes.length);
|
||||
return await readPyzArchive(handle, 0, declaredSize);
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
};
|
||||
|
||||
it.each([0, 0xffffffff])(
|
||||
"retains an unknown Python magic and full-image digest with invalid TOC offset %s",
|
||||
async (offset) => {
|
||||
const image = pyzArchive({ magic: Buffer.from([1, 2, 3, 4]), modules: [] });
|
||||
image.writeUInt32BE(offset, 8);
|
||||
expect(await readImage(image)).toMatchObject({
|
||||
ok: true,
|
||||
value: {
|
||||
kind: "opaque",
|
||||
pythonMagic: Buffer.from([1, 2, 3, 4]),
|
||||
pythonVersion: undefined,
|
||||
sha256: createHash("sha256").update(image).digest("hex"),
|
||||
limitation: expect.stringContaining("not in the supported set"),
|
||||
},
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("reports a truncated table before deferred signature rejection", async () => {
|
||||
const header = Buffer.alloc(12);
|
||||
header.write("BAD\0");
|
||||
PYTHON_39_MAGIC.copy(header, 4);
|
||||
header.writeUInt32BE(12, 8);
|
||||
expect(await readImage(header, 16)).toMatchObject({
|
||||
ok: false,
|
||||
error: { reason: "format", message: "PYZ table is truncated" },
|
||||
});
|
||||
});
|
||||
|
||||
it("reports the signature after completing a readable table and image hash", async () => {
|
||||
const image = pyzArchive({ magic: PYTHON_39_MAGIC, modules: [] });
|
||||
image.write("BAD\0");
|
||||
expect(await readImage(image)).toMatchObject({
|
||||
ok: false,
|
||||
error: { reason: "format", message: "PYZ archive magic does not match" },
|
||||
});
|
||||
});
|
||||
@@ -38,6 +38,7 @@ export type PyzArchive =
|
||||
| {
|
||||
readonly kind: "opaque";
|
||||
readonly pythonMagic: Buffer;
|
||||
readonly pythonVersion: string | undefined;
|
||||
readonly sha256: string;
|
||||
readonly limitation: string;
|
||||
};
|
||||
@@ -45,6 +46,31 @@ export type PyzArchive =
|
||||
/** Bound the transient marshal table allocation independently of image expansion. */
|
||||
export const MAX_PYZ_TABLE_BYTES = 16 * 1024 * 1024;
|
||||
|
||||
interface PyzHeaderFacts {
|
||||
readonly signatureMatches: boolean;
|
||||
readonly pythonMagic: Buffer;
|
||||
readonly pythonVersion: string | undefined;
|
||||
readonly tocOffset: number;
|
||||
/** Null means the declaration lies outside this image, not an empty table. */
|
||||
readonly tableBytes: number | null;
|
||||
}
|
||||
|
||||
/** Called only after the exact fixed-header read has succeeded. */
|
||||
const readPyzHeaderFacts = (header: Buffer, size: number): PyzHeaderFacts => {
|
||||
const pythonMagic = Buffer.from(header.subarray(4, 8));
|
||||
const tocOffset = header.readUInt32BE(8);
|
||||
return {
|
||||
signatureMatches: header.subarray(0, 4).equals(PYZ_MAGIC),
|
||||
pythonMagic,
|
||||
pythonVersion: pythonVersionForMagic(pythonMagic),
|
||||
tocOffset,
|
||||
tableBytes:
|
||||
tocOffset >= PYZ_HEADER_BYTES && tocOffset <= size
|
||||
? size - tocOffset
|
||||
: null,
|
||||
};
|
||||
};
|
||||
|
||||
/** Parse only the header and table in memory; hash the complete file range in chunks. */
|
||||
export const readPyzArchive = async (
|
||||
handle: FileHandle,
|
||||
@@ -63,12 +89,9 @@ export const readPyzArchive = async (
|
||||
signal,
|
||||
});
|
||||
if (!headerRead.ok) return headerRead;
|
||||
const header = headerRead.value;
|
||||
const tocOffset =
|
||||
header.length === PYZ_HEADER_BYTES ? header.readUInt32BE(8) : size;
|
||||
const validOffset = tocOffset >= PYZ_HEADER_BYTES && tocOffset <= size;
|
||||
const knownMagic = pythonVersionForMagic(header.subarray(4, 8)) !== undefined;
|
||||
const tableSize = validOffset && knownMagic ? size - tocOffset : 0;
|
||||
const header = readPyzHeaderFacts(headerRead.value, size);
|
||||
const tableSize =
|
||||
header.pythonVersion === undefined ? 0 : (header.tableBytes ?? 0);
|
||||
if (tableSize > MAX_PYZ_TABLE_BYTES)
|
||||
return err(
|
||||
pyInstallerFailure(
|
||||
@@ -77,7 +100,7 @@ export const readPyzArchive = async (
|
||||
),
|
||||
);
|
||||
const read = await readPyInstallerBytes(handle, {
|
||||
position: start + tocOffset,
|
||||
position: start + header.tocOffset,
|
||||
length: tableSize,
|
||||
label: "PYZ table",
|
||||
signal,
|
||||
@@ -96,33 +119,28 @@ export const readPyzArchive = async (
|
||||
return err(
|
||||
pyInstallerFailure("format", "PYZ image extends outside the file"),
|
||||
);
|
||||
return parsePyzParts(header, table, size, digest.sha256);
|
||||
return parsePyzParts(header, table, digest.sha256);
|
||||
};
|
||||
|
||||
const parsePyzParts = (
|
||||
header: Buffer,
|
||||
header: PyzHeaderFacts,
|
||||
table: Buffer,
|
||||
size: number,
|
||||
sha256: string,
|
||||
): Result<PyzArchive, PyInstallerFailure> => {
|
||||
if (
|
||||
header.length < PYZ_HEADER_BYTES ||
|
||||
!header.subarray(0, 4).equals(PYZ_MAGIC)
|
||||
)
|
||||
if (!header.signatureMatches)
|
||||
return err(
|
||||
pyInstallerFailure("format", "PYZ archive magic does not match"),
|
||||
);
|
||||
const pythonMagic = Buffer.from(header.subarray(4, 8));
|
||||
const pythonVersion = pythonVersionForMagic(pythonMagic);
|
||||
const { pythonMagic, pythonVersion, tocOffset } = header;
|
||||
if (pythonVersion === undefined)
|
||||
return ok({
|
||||
kind: "opaque",
|
||||
pythonMagic,
|
||||
pythonVersion,
|
||||
sha256,
|
||||
limitation: `PYZ Python magic ${pythonMagic.toString("hex")} is not in the supported set; the member was left opaque.`,
|
||||
});
|
||||
const tocOffset = header.readUInt32BE(8);
|
||||
if (tocOffset < PYZ_HEADER_BYTES || tocOffset > size)
|
||||
if (header.tableBytes === null)
|
||||
return err(
|
||||
pyInstallerFailure(
|
||||
"format",
|
||||
@@ -135,6 +153,7 @@ const parsePyzParts = (
|
||||
return ok({
|
||||
kind: "opaque",
|
||||
pythonMagic,
|
||||
pythonVersion,
|
||||
sha256,
|
||||
limitation: `PYZ table of contents could not be parsed (${marshalled.error.message}); the member was left opaque.`,
|
||||
});
|
||||
@@ -145,6 +164,7 @@ const parsePyzParts = (
|
||||
? ok({
|
||||
kind: "opaque",
|
||||
pythonMagic,
|
||||
pythonVersion,
|
||||
sha256,
|
||||
limitation: `${entries.error.message}; the member was left opaque.`,
|
||||
})
|
||||
|
||||
@@ -5,13 +5,16 @@ import type {
|
||||
} from "../domain/javascript/electronObservation.js";
|
||||
import type { CdpConnection } from "./CdpConnection.js";
|
||||
import { CdpCaptureCompleteness } from "./CdpCaptureCompleteness.js";
|
||||
import { recordValue, cdpStringValue } from "./CdpCaptureValues.js";
|
||||
import { recordValue } from "./CdpCaptureValues.js";
|
||||
import type { CdpEndpointTarget } from "./CdpEndpoint.js";
|
||||
import { optionalCdpCommand } from "./CdpOptionalCommand.js";
|
||||
import { authorizedElectronFile } from "./ElectronFileScope.js";
|
||||
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
|
||||
import { AnalysisCancelledError } from "../domain/analysisErrorCore.js";
|
||||
import { workerTargetOwnership } from "./CdpWorkerTargetOwnership.js";
|
||||
import {
|
||||
parseCdpWorkerTarget,
|
||||
workerTargetOwnership,
|
||||
} from "./CdpWorkerTarget.js";
|
||||
|
||||
/** Inventory local worker targets without attaching to them. */
|
||||
export const captureElectronWorkers = async (input: {
|
||||
@@ -47,33 +50,27 @@ export const captureElectronWorkers = async (input: {
|
||||
}
|
||||
for (const entry of targetInfos) {
|
||||
throwIfElectronWorkerCaptureCancelled(input.signal);
|
||||
const target = recordValue(entry);
|
||||
if (target === undefined) {
|
||||
const parsed = parseCdpWorkerTarget(entry);
|
||||
if (!parsed.ok) {
|
||||
input.completeness.exclude("workers", "invalid_protocol_value");
|
||||
continue;
|
||||
}
|
||||
const type = cdpStringValue(target.type) ?? "";
|
||||
if (!type.includes("worker")) continue;
|
||||
const target = parsed.value;
|
||||
if (target.kind === "non_worker") continue;
|
||||
const ownership = workerTargetOwnership(
|
||||
target,
|
||||
input.target.id,
|
||||
input.frameIds,
|
||||
);
|
||||
if (!ownership.ok) {
|
||||
input.completeness.exclude("workers", "invalid_protocol_value");
|
||||
continue;
|
||||
}
|
||||
if (ownership.value.scope !== "selected") {
|
||||
if (ownership !== "selected") {
|
||||
input.completeness.exclude(
|
||||
"workers",
|
||||
ownership.value.scope === "unknown"
|
||||
? "unattributed_target"
|
||||
: "out_of_target_scope",
|
||||
ownership === "unknown" ? "unattributed_target" : "out_of_target_scope",
|
||||
);
|
||||
continue;
|
||||
}
|
||||
const path = await authorizedElectronFile(
|
||||
cdpStringValue(target.url) ?? "",
|
||||
target.url,
|
||||
input.signal,
|
||||
"inspect_electron_page",
|
||||
input.asarHeaderAdmission,
|
||||
@@ -82,22 +79,13 @@ export const captureElectronWorkers = async (input: {
|
||||
input.completeness.exclude("workers", "out_of_target_scope");
|
||||
continue;
|
||||
}
|
||||
const targetId = boundedTargetField(target.targetId);
|
||||
if (targetId === null) {
|
||||
input.completeness.exclude("workers", "invalid_protocol_value");
|
||||
continue;
|
||||
}
|
||||
if (typeof target.attached !== "boolean") {
|
||||
input.completeness.exclude("workers", "invalid_protocol_value");
|
||||
continue;
|
||||
}
|
||||
workers.push({
|
||||
target_id: targetId,
|
||||
type,
|
||||
target_id: target.targetId,
|
||||
type: target.type,
|
||||
file_path: path,
|
||||
attached: target.attached,
|
||||
opener_target_id: ownership.value.openerTargetId,
|
||||
parent_frame_id: ownership.value.parentFrameId,
|
||||
opener_target_id: target.openerTargetId,
|
||||
parent_frame_id: target.parentFrameId,
|
||||
});
|
||||
}
|
||||
return workers.sort((left, right) =>
|
||||
@@ -111,8 +99,3 @@ const throwIfElectronWorkerCaptureCancelled = (signal?: AbortSignal): void => {
|
||||
cause: signal.reason,
|
||||
});
|
||||
};
|
||||
|
||||
const boundedTargetField = (value: unknown): string | null => {
|
||||
const field = cdpStringValue(value);
|
||||
return field === undefined || field === "" ? null : field;
|
||||
};
|
||||
@@ -9,7 +9,7 @@ import type { CaptureContext } from "./CdpPageCapture.js";
|
||||
|
||||
const origin = "http://127.0.0.1:43127";
|
||||
|
||||
const capture = async (inventory: unknown) => {
|
||||
const capture = async (inventory: unknown, frameIds = new Set<string>()) => {
|
||||
const input = inspectWebPageInputSchema.parse({
|
||||
cdp_endpoint: origin,
|
||||
allowed_origins: [origin],
|
||||
@@ -38,7 +38,7 @@ const capture = async (inventory: unknown) => {
|
||||
limitations: [],
|
||||
events,
|
||||
},
|
||||
new Set(),
|
||||
frameIds,
|
||||
);
|
||||
return { workers, completeness: events.completeness.snapshot() };
|
||||
};
|
||||
@@ -199,3 +199,74 @@ it.each([{ openerId: "" }, { parentFrameId: null }])(
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
{ type: undefined },
|
||||
{ type: 7 },
|
||||
{ type: "" },
|
||||
{ targetId: "" },
|
||||
{ url: undefined },
|
||||
{ attached: null },
|
||||
{ openerId: null },
|
||||
{ parentFrameId: "" },
|
||||
])(
|
||||
"reports malformed worker facts before ownership and URL policy: %j",
|
||||
async (override) => {
|
||||
const result = await capture({
|
||||
targetInfos: [
|
||||
{
|
||||
targetId: "worker",
|
||||
type: "worker",
|
||||
url: "https://outside.example/worker.js",
|
||||
attached: false,
|
||||
openerId: "foreign-page",
|
||||
...override,
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(result.workers).toEqual([]);
|
||||
expect(result.completeness.excluded).toEqual([
|
||||
{ section: "workers", reason: "invalid_protocol_value", count: 1 },
|
||||
]);
|
||||
},
|
||||
);
|
||||
|
||||
it("uses parsed frame relationships without treating valid non-workers as malformed", async () => {
|
||||
const result = await capture(
|
||||
{
|
||||
targetInfos: [
|
||||
{ type: "page" },
|
||||
{
|
||||
targetId: "worker",
|
||||
type: "shared_worker",
|
||||
url: `${origin}/worker.js`,
|
||||
attached: false,
|
||||
openerId: "foreign-page",
|
||||
parentFrameId: "selected-frame",
|
||||
},
|
||||
{
|
||||
targetId: "foreign",
|
||||
type: "worker",
|
||||
url: `${origin}/foreign.js`,
|
||||
attached: true,
|
||||
parentFrameId: "foreign-frame",
|
||||
},
|
||||
],
|
||||
},
|
||||
new Set(["selected-frame"]),
|
||||
);
|
||||
expect(result.workers).toEqual([
|
||||
{
|
||||
target_id: "worker",
|
||||
type: "shared_worker",
|
||||
url: `${origin}/worker.js`,
|
||||
origin,
|
||||
attached: false,
|
||||
opener_target_id: "foreign-page",
|
||||
parent_frame_id: "selected-frame",
|
||||
},
|
||||
]);
|
||||
expect(result.completeness.excluded).toEqual([
|
||||
{ section: "workers", reason: "out_of_target_scope", count: 1 },
|
||||
]);
|
||||
});
|
||||
@@ -4,11 +4,13 @@ import {
|
||||
admitRecords,
|
||||
allowedSanitizedUrl,
|
||||
recordValue,
|
||||
cdpStringValue,
|
||||
} from "./CdpCaptureValues.js";
|
||||
import type { CaptureContext } from "./CdpPageCapture.js";
|
||||
import { optionalCdpCommand } from "./CdpOptionalCommand.js";
|
||||
import { workerTargetOwnership } from "./CdpWorkerTargetOwnership.js";
|
||||
import {
|
||||
parseCdpWorkerTarget,
|
||||
workerTargetOwnership,
|
||||
} from "./CdpWorkerTarget.js";
|
||||
import { exclusionReasonForUrl } from "./CdpCaptureEventHelpers.js";
|
||||
|
||||
interface CaptureWorkersInput {
|
||||
@@ -42,59 +44,38 @@ export const captureWorkers = async (
|
||||
}
|
||||
let malformedEntries = 0;
|
||||
for (const entry of targets ?? []) {
|
||||
const target = recordValue(entry);
|
||||
if (target === undefined) {
|
||||
malformedEntries += 1;
|
||||
continue;
|
||||
}
|
||||
const type = cdpStringValue(target.type);
|
||||
if (type === undefined || type.length === 0) {
|
||||
malformedEntries += 1;
|
||||
continue;
|
||||
}
|
||||
if (!type.includes("worker")) continue;
|
||||
const targetId = cdpStringValue(target.targetId);
|
||||
if (
|
||||
targetId === undefined ||
|
||||
targetId.length === 0 ||
|
||||
typeof target.attached !== "boolean" ||
|
||||
typeof target.url !== "string"
|
||||
) {
|
||||
const parsed = parseCdpWorkerTarget(entry);
|
||||
if (!parsed.ok) {
|
||||
malformedEntries += 1;
|
||||
continue;
|
||||
}
|
||||
const target = parsed.value;
|
||||
if (target.kind === "non_worker") continue;
|
||||
const ownership = workerTargetOwnership(
|
||||
target,
|
||||
context.target.id,
|
||||
frameIds,
|
||||
);
|
||||
if (!ownership.ok) {
|
||||
malformedEntries += 1;
|
||||
continue;
|
||||
}
|
||||
if (ownership.value.scope !== "selected") {
|
||||
if (ownership !== "selected") {
|
||||
completeness.exclude(
|
||||
"workers",
|
||||
ownership.value.scope === "unknown"
|
||||
? "unattributed_target"
|
||||
: "out_of_target_scope",
|
||||
ownership === "unknown" ? "unattributed_target" : "out_of_target_scope",
|
||||
);
|
||||
continue;
|
||||
}
|
||||
const url = allowedSanitizedUrl(target.url, allowedOrigins);
|
||||
if (url === undefined) {
|
||||
const rawUrl = cdpStringValue(target.url);
|
||||
completeness.exclude("workers", exclusionReasonForUrl(rawUrl));
|
||||
completeness.exclude("workers", exclusionReasonForUrl(target.url));
|
||||
continue;
|
||||
}
|
||||
items.push({
|
||||
target_id: targetId,
|
||||
type,
|
||||
target_id: target.targetId,
|
||||
type: target.type,
|
||||
url: url.url,
|
||||
origin: url.origin,
|
||||
attached: target.attached,
|
||||
opener_target_id: ownership.value.openerTargetId,
|
||||
parent_frame_id: ownership.value.parentFrameId,
|
||||
opener_target_id: target.openerTargetId,
|
||||
parent_frame_id: target.parentFrameId,
|
||||
});
|
||||
}
|
||||
if (malformedEntries > 0)
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { cdpStringValue, recordValue } from "./CdpCaptureValues.js";
|
||||
|
||||
export interface CdpWorkerRelationships {
|
||||
readonly openerTargetId: string | null;
|
||||
readonly parentFrameId: string | null;
|
||||
}
|
||||
|
||||
export interface CdpWorkerTarget extends CdpWorkerRelationships {
|
||||
readonly kind: "worker";
|
||||
readonly targetId: string;
|
||||
readonly type: string;
|
||||
readonly url: string;
|
||||
readonly attached: boolean;
|
||||
}
|
||||
|
||||
/** Admit protocol facts before either consumer applies target or URL policy. */
|
||||
export const parseCdpWorkerTarget = (
|
||||
input: unknown,
|
||||
): Result<
|
||||
CdpWorkerTarget | { readonly kind: "non_worker" },
|
||||
{ readonly _tag: "InvalidWorkerTarget" }
|
||||
> => {
|
||||
const target = recordValue(input);
|
||||
const type = cdpStringValue(target?.type);
|
||||
if (target === undefined || type === undefined || type === "")
|
||||
return err({ _tag: "InvalidWorkerTarget" });
|
||||
if (!type.includes("worker")) return ok({ kind: "non_worker" });
|
||||
const targetId = cdpStringValue(target.targetId);
|
||||
const url = cdpStringValue(target.url);
|
||||
const openerTargetId = cdpStringValue(target.openerId);
|
||||
const parentFrameId = cdpStringValue(target.parentFrameId);
|
||||
if (
|
||||
targetId === undefined ||
|
||||
targetId === "" ||
|
||||
url === undefined ||
|
||||
typeof target.attached !== "boolean" ||
|
||||
(target.openerId !== undefined &&
|
||||
(openerTargetId === undefined || openerTargetId === "")) ||
|
||||
(target.parentFrameId !== undefined &&
|
||||
(parentFrameId === undefined || parentFrameId === ""))
|
||||
)
|
||||
return err({ _tag: "InvalidWorkerTarget" });
|
||||
return ok({
|
||||
kind: "worker",
|
||||
targetId,
|
||||
type,
|
||||
url,
|
||||
attached: target.attached,
|
||||
openerTargetId: openerTargetId ?? null,
|
||||
parentFrameId: parentFrameId ?? null,
|
||||
});
|
||||
};
|
||||
|
||||
/** Optional CDP relationships establish ownership only when supplied. */
|
||||
export const workerTargetOwnership = (
|
||||
relationships: CdpWorkerRelationships,
|
||||
selectedTargetId: string,
|
||||
frameIds: ReadonlySet<string>,
|
||||
): "selected" | "foreign" | "unknown" => {
|
||||
const { openerTargetId, parentFrameId } = relationships;
|
||||
if (
|
||||
openerTargetId === selectedTargetId ||
|
||||
(parentFrameId !== null && frameIds.has(parentFrameId))
|
||||
)
|
||||
return "selected";
|
||||
return openerTargetId === null && parentFrameId === null
|
||||
? "unknown"
|
||||
: "foreign";
|
||||
};
|
||||
@@ -1,38 +0,0 @@
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { cdpStringValue } from "./CdpCaptureValues.js";
|
||||
|
||||
/** Optional CDP relationships establish ownership only when supplied. */
|
||||
export const workerTargetOwnership = (
|
||||
target: Readonly<Record<string, unknown>>,
|
||||
selectedTargetId: string,
|
||||
frameIds: ReadonlySet<string>,
|
||||
): Result<
|
||||
{
|
||||
readonly scope: "selected" | "foreign" | "unknown";
|
||||
readonly openerTargetId: string | null;
|
||||
readonly parentFrameId: string | null;
|
||||
},
|
||||
{ readonly _tag: "InvalidWorkerRelationship" }
|
||||
> => {
|
||||
const openerTargetId = cdpStringValue(target.openerId);
|
||||
const parentFrameId = cdpStringValue(target.parentFrameId);
|
||||
if (
|
||||
(target.openerId !== undefined &&
|
||||
(openerTargetId === undefined || openerTargetId === "")) ||
|
||||
(target.parentFrameId !== undefined &&
|
||||
(parentFrameId === undefined || parentFrameId === ""))
|
||||
)
|
||||
return err({ _tag: "InvalidWorkerRelationship" });
|
||||
const selected =
|
||||
openerTargetId === selectedTargetId ||
|
||||
(parentFrameId !== undefined && frameIds.has(parentFrameId));
|
||||
return ok({
|
||||
scope: selected
|
||||
? "selected"
|
||||
: openerTargetId === undefined && parentFrameId === undefined
|
||||
? "unknown"
|
||||
: "foreign",
|
||||
openerTargetId: openerTargetId ?? null,
|
||||
parentFrameId: parentFrameId ?? null,
|
||||
});
|
||||
};
|
||||
@@ -78,6 +78,14 @@ type TestMode =
|
||||
readonly oomMarker?: string;
|
||||
readonly stderrMarker?: string;
|
||||
}
|
||||
| {
|
||||
readonly kind: "result";
|
||||
readonly encoding:
|
||||
| "invalid-utf8"
|
||||
| "replacement"
|
||||
| "bom"
|
||||
| "malformed-json";
|
||||
}
|
||||
| { readonly kind: "delayed-ready" }
|
||||
| { readonly kind: "exit-before-ready" }
|
||||
| {
|
||||
@@ -166,6 +174,33 @@ const protocolWorkerProgram = (
|
||||
});
|
||||
`;
|
||||
|
||||
const resultWorkerProgram = (encoding: string): string => `
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createInterface } from "node:readline";
|
||||
import { getHeapStatistics } from "node:v8";
|
||||
const { analyzeCapturedWebScript } = await import(${JSON.stringify(new URL("../domain/webBundleAnalyzer.ts", import.meta.url).href)});
|
||||
const send = (value) => process.stdout.write(JSON.stringify(value) + "\\n");
|
||||
send({ kind: "ready", heap_limit_bytes: getHeapStatistics().heap_size_limit });
|
||||
createInterface({ input: process.stdin, crlfDelay: Infinity }).once("line", async (line) => {
|
||||
const request = JSON.parse(line);
|
||||
const script = JSON.parse(await readFile(request.id + ".source", "utf8"));
|
||||
const facts = analyzeCapturedWebScript(script);
|
||||
let bytes = Buffer.from(JSON.stringify(facts));
|
||||
const encoding = ${JSON.stringify(encoding)};
|
||||
if (encoding === "invalid-utf8") {
|
||||
const index = bytes.indexOf(Buffer.from("result-marker"));
|
||||
if (index < 0) throw new Error("Expected endpoint marker in schema-valid facts");
|
||||
bytes[index] = 0xff;
|
||||
} else if (encoding === "bom") bytes = Buffer.concat([Buffer.from([0xef, 0xbb, 0xbf]), bytes]);
|
||||
else if (encoding === "malformed-json") bytes = bytes.subarray(0, bytes.length - 1);
|
||||
await writeFile(request.id + ".result", bytes);
|
||||
send({ kind: "complete", id: request.id, result: {
|
||||
bytes: bytes.length, sha256: createHash("sha256").update(bytes).digest("hex"),
|
||||
} });
|
||||
});
|
||||
`;
|
||||
|
||||
const delayedReadyProgram = `
|
||||
import { createInterface } from "node:readline";
|
||||
import { getHeapStatistics } from "node:v8";
|
||||
@@ -203,7 +238,9 @@ const launcherFor = (
|
||||
? delayedReadyProgram
|
||||
: mode.kind === "exit-before-ready"
|
||||
? exitBeforeReadyProgram
|
||||
: protocolWorkerProgram(mode.failure);
|
||||
: mode.kind === "result"
|
||||
? resultWorkerProgram(mode.encoding)
|
||||
: protocolWorkerProgram(mode.failure);
|
||||
const spawned = await spawnOwnedProviderProcess({
|
||||
...request,
|
||||
arguments: [
|
||||
@@ -926,3 +963,75 @@ it.each([
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["invalid-utf8", "bom", "malformed-json"] as const)(
|
||||
"rejects %s worker result bytes with a matching descriptor",
|
||||
async (encoding) => {
|
||||
let root: string | undefined;
|
||||
const worker = new WebBundleAnalysisWorker(
|
||||
WEB_BUNDLE_WORKER_LIMITS.heapMb,
|
||||
launcherFor(
|
||||
{ kind: "result", encoding },
|
||||
{
|
||||
onSpawn: (spawned) => {
|
||||
root = spawned.root;
|
||||
},
|
||||
},
|
||||
),
|
||||
);
|
||||
try {
|
||||
const result = await worker.analyze(
|
||||
makeScript("fetch('/result-marker');"),
|
||||
10_000,
|
||||
);
|
||||
const error = failureOf(result);
|
||||
expect(error).toBeInstanceOf(AnalysisOutputError);
|
||||
expect(error.cause).toBeInstanceOf(
|
||||
encoding === "invalid-utf8" ? TypeError : SyntaxError,
|
||||
);
|
||||
expect(error.cleanupIncomplete).toBe(false);
|
||||
} finally {
|
||||
await worker.close();
|
||||
}
|
||||
expect(root).toBeDefined();
|
||||
await expect(access(root!)).rejects.toMatchObject({
|
||||
code: "ENOENT",
|
||||
path: root,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("preserves literal replacement characters in valid UTF-8 worker facts", async () => {
|
||||
let root: string | undefined;
|
||||
const worker = new WebBundleAnalysisWorker(
|
||||
WEB_BUNDLE_WORKER_LIMITS.heapMb,
|
||||
launcherFor(
|
||||
{ kind: "result", encoding: "replacement" },
|
||||
{
|
||||
onSpawn: (spawned) => {
|
||||
root = spawned.root;
|
||||
},
|
||||
},
|
||||
),
|
||||
);
|
||||
try {
|
||||
const result = await worker.analyze(
|
||||
makeScript("fetch('/literal-�');"),
|
||||
10_000,
|
||||
);
|
||||
expect(result.ok).toBe(true);
|
||||
if (!result.ok) throw new Error(result.error.error.message);
|
||||
if (result.value.status !== "analyzed")
|
||||
throw new Error("Expected analyzed worker facts");
|
||||
expect(result.value.observations.endpoints).toContainEqual(
|
||||
expect.objectContaining({ value: "/literal-�" }),
|
||||
);
|
||||
} finally {
|
||||
await worker.close();
|
||||
}
|
||||
expect(root).toBeDefined();
|
||||
await expect(access(root!)).rejects.toMatchObject({
|
||||
code: "ENOENT",
|
||||
path: root,
|
||||
});
|
||||
});
|
||||
@@ -31,6 +31,7 @@ import {
|
||||
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
|
||||
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
|
||||
import { jsonParts, bufferedJsonParts } from "../domain/jsonSerialization.js";
|
||||
import { safeParseUtf8Json } from "../domain/safeJson.js";
|
||||
import { javaScriptRuntime } from "../domain/runtimeVersion.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import type { IncludedScript } from "../domain/webBundleAnalyzerState.js";
|
||||
@@ -787,9 +788,9 @@ export class WebBundleAnalysisWorker {
|
||||
OPERATION,
|
||||
"Web analyzer result differs from its descriptor",
|
||||
);
|
||||
const facts = webBundleScriptAnalysisSchema.parse(
|
||||
JSON.parse(stored.bytes.toString("utf8")),
|
||||
);
|
||||
const parsed = safeParseUtf8Json(stored.bytes);
|
||||
if (!parsed.ok) throw parsed.cause;
|
||||
const facts = webBundleScriptAnalysisSchema.parse(parsed.value);
|
||||
if (facts.status !== "analyzed" || facts.script_key !== script.script_key)
|
||||
throw new AnalysisOutputError(
|
||||
OPERATION,
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { z } from "zod";
|
||||
import { address, document } from "./toolContractHelpers.js";
|
||||
import {
|
||||
MAX_BYTE_SEARCH_MATCHES,
|
||||
bytePatternFields,
|
||||
validateBytePattern,
|
||||
} from "../domain/native/bytePattern.js";
|
||||
@@ -17,7 +18,7 @@ export const HOPPER_MEMORY_TOOL_DEFINITIONS = [
|
||||
document,
|
||||
start: address.optional(),
|
||||
end: address.optional(),
|
||||
limit: z.int().min(1).max(1000).default(100),
|
||||
limit: z.int().min(1).max(MAX_BYTE_SEARCH_MATCHES).default(100),
|
||||
cursor: z.string().min(1).optional(),
|
||||
})
|
||||
.superRefine(validateBytePattern),
|
||||
|
||||
@@ -448,6 +448,7 @@ const providerErrorDetails = (
|
||||
candidate_ids: [...error.candidateIds],
|
||||
rejections: error.rejections.map((rejection) => ({
|
||||
provider_id: rejection.providerId,
|
||||
origin: rejection.origin,
|
||||
code: rejection.code,
|
||||
reason: rejection.reason,
|
||||
diagnostics: rejection.diagnostics,
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
import { z } from "zod";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { jsonValueType, parseJsonObject } from "../jsonShape.js";
|
||||
import { ok, type Result } from "../result.js";
|
||||
@@ -11,7 +10,12 @@ import {
|
||||
type WebNetworkCaptureRecord,
|
||||
} from "../webNetworkCapture.js";
|
||||
import { completeWithinViewCoverage } from "./analysisViewCoverage.js";
|
||||
import { recordedResponseBodyIsEmpty } from "./recordedResponseBody.js";
|
||||
import {
|
||||
recordedResponseBodyIsEmpty,
|
||||
readRecordedResponseDeclaration,
|
||||
} from "./recordedResponseBody.js";
|
||||
import { readRecordedRequestBody } from "./recordedExchangeBody.js";
|
||||
import { isJson, isJsonLike, mediaTypeEssence } from "./recordedMediaType.js";
|
||||
import type { UnsignedAnalysisView } from "./analysisView.js";
|
||||
import type { ScenarioResponseBodyRetention } from "./scenarioCaptureRecords.js";
|
||||
import { WebNetworkProjectionAccumulator } from "./WebNetworkProjectionAccumulator.js";
|
||||
@@ -41,7 +45,6 @@ import {
|
||||
GROUP_BYTES,
|
||||
RECORD_SCRATCH_BYTES,
|
||||
SEGMENT_BYTES,
|
||||
mediaTypeEssence,
|
||||
readQueryParameterNames,
|
||||
readRecordedRequestLine,
|
||||
routePartCount,
|
||||
@@ -97,24 +100,15 @@ interface GroupState extends ToolGroup {
|
||||
bodiless: boolean;
|
||||
}
|
||||
|
||||
const responseTextSchema = z.object({
|
||||
response: z.object({
|
||||
status: z.unknown().optional(),
|
||||
content: z
|
||||
.object({ text: z.unknown().optional(), size: z.unknown().optional() })
|
||||
.nullable()
|
||||
.optional(),
|
||||
}),
|
||||
});
|
||||
/** Whether a record's response had no body: a 204 or 205, an empty text, or a recorded size of zero; a missing text alone may be a body not kept. */
|
||||
const emptyResponse = (record: WebNetworkCaptureRecord): boolean => {
|
||||
const parsed = responseTextSchema.safeParse(record.reported);
|
||||
if (!parsed.success) return false;
|
||||
const { status, content } = parsed.data.response;
|
||||
/** Manifest eligibility also accepts declared bodyless statuses and a recorded zero size when no text was retained. */
|
||||
const manifestResponseIsEmpty = (record: WebNetworkCaptureRecord): boolean => {
|
||||
const response = readRecordedResponseDeclaration(record);
|
||||
if (response === undefined) return false;
|
||||
const { status, content } = response;
|
||||
return (
|
||||
status === 204 ||
|
||||
status === 205 ||
|
||||
recordedResponseBodyIsEmpty(record) ||
|
||||
recordedResponseBodyIsEmpty(record, response) ||
|
||||
((content?.text === undefined || content.text === null) &&
|
||||
content?.size === 0)
|
||||
);
|
||||
@@ -123,18 +117,6 @@ const emptyResponse = (record: WebNetworkCaptureRecord): boolean => {
|
||||
const answeredEmpty = (group: GroupState): boolean =>
|
||||
group.bodiless &&
|
||||
[...group.statuses].some((status) => status >= 200 && status < 300);
|
||||
const postDataSchema = z.object({
|
||||
request: z.object({
|
||||
postData: z
|
||||
.object({
|
||||
mimeType: z.unknown().optional(),
|
||||
text: z.unknown().optional(),
|
||||
})
|
||||
.nullable()
|
||||
.optional(),
|
||||
}),
|
||||
});
|
||||
const JSON_LIKE_TEXT = new Set(["text/plain", "text/markdown"]);
|
||||
const ASSET_ESSENCES = new Set([
|
||||
"text/html",
|
||||
"application/xhtml+xml",
|
||||
@@ -170,17 +152,6 @@ const UNKNOWNS: Manifest["unknowns"] = [
|
||||
},
|
||||
];
|
||||
|
||||
/** `application/json` or `application/*+json`; a `+json` suffix on another type is not JSON here. */
|
||||
const isJson = (essence: string | null): boolean =>
|
||||
essence === "application/json" ||
|
||||
(essence !== null &&
|
||||
essence.startsWith("application/") &&
|
||||
essence.endsWith("+json"));
|
||||
|
||||
/** JSON, `+json`, `text/plain` or `text/markdown`: the content types a replay parses as JSON. */
|
||||
export const isJsonLike = (essence: string | null): boolean =>
|
||||
isJson(essence) || (essence !== null && JSON_LIKE_TEXT.has(essence));
|
||||
|
||||
const isApiPath = (path: string): boolean =>
|
||||
path.startsWith("/api/") || path.startsWith("/apis/");
|
||||
|
||||
@@ -260,9 +231,11 @@ export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumula
|
||||
this.scratch,
|
||||
);
|
||||
if (line === null) return this.overBudget();
|
||||
const releaseGrouping = this.temporaryCheckpoint();
|
||||
if (line.kind === "excluded") {
|
||||
this.#unrepresented += 1;
|
||||
return ok(undefined);
|
||||
releaseGrouping();
|
||||
return this.onRecordedRequestLine(record, line);
|
||||
}
|
||||
// Coverage uses the same admitted request line as grouping, including origin and raw path.
|
||||
if (scenarioResponseBody !== undefined) {
|
||||
@@ -300,28 +273,39 @@ export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumula
|
||||
? this.#bodyShape(record.reported)
|
||||
: ok(null);
|
||||
if (!bodyShape.ok) return bodyShape;
|
||||
return this.#append(
|
||||
const grouped = this.#append(
|
||||
record.ordinal,
|
||||
line,
|
||||
route,
|
||||
queryNames,
|
||||
{ key, group },
|
||||
{ shape: bodyShape.value, empty: emptyResponse(record) },
|
||||
{ shape: bodyShape.value, empty: manifestResponseIsEmpty(record) },
|
||||
);
|
||||
if (!grouped.ok) return grouped;
|
||||
// Keep the parsed line's admission while releasing grouping-only scratch.
|
||||
releaseGrouping();
|
||||
return this.onRecordedRequestLine(record, line);
|
||||
} finally {
|
||||
this.releaseTemporary();
|
||||
}
|
||||
}
|
||||
|
||||
/** Consume the owner's parsed request line after successful grouping, including exclusions. */
|
||||
protected onRecordedRequestLine(
|
||||
_record: WebNetworkCaptureRecord,
|
||||
_line: RecordedRequestLine,
|
||||
): Result<void, AnalysisError> {
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
/** Read the top-level key types of the group's first recorded JSON-typed request body; nested values are never walked. */
|
||||
#bodyShape(reported: unknown): Result<BodyShape | null, AnalysisError> {
|
||||
const parsed = postDataSchema.safeParse(reported);
|
||||
const body = parsed.success ? parsed.data.request.postData : undefined;
|
||||
const request = readRecordedRequestBody(reported);
|
||||
const body = request.kind === "request" ? request.postData : undefined;
|
||||
if (
|
||||
body === undefined ||
|
||||
body === null ||
|
||||
typeof body.text !== "string" ||
|
||||
typeof body.mimeType !== "string"
|
||||
body?.kind !== "object" ||
|
||||
body.text === undefined ||
|
||||
body.mimeType === undefined
|
||||
)
|
||||
return ok(null);
|
||||
// The media type is scanned and lowercased before it is known to be JSON; charge it first.
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
import { z } from "zod";
|
||||
import type { ScenarioResponseBodyRetention } from "./scenarioCaptureRecords.js";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { parseExactJson } from "../exactJsonNumber.js";
|
||||
import { MAX_JSON_DEPTH, type JsonValue } from "../jsonValue.js";
|
||||
@@ -14,10 +13,9 @@ import {
|
||||
type WebNetworkCaptureRecord,
|
||||
} from "../webNetworkCapture.js";
|
||||
import type { UnsignedAnalysisView } from "./analysisView.js";
|
||||
import {
|
||||
EnvironmentManifestAccumulator,
|
||||
isJsonLike,
|
||||
} from "./EnvironmentManifestAccumulator.js";
|
||||
import { EnvironmentManifestAccumulator } from "./EnvironmentManifestAccumulator.js";
|
||||
import { isJsonLike, mediaTypeEssence } from "./recordedMediaType.js";
|
||||
import { readRecordedRequestBody } from "./recordedExchangeBody.js";
|
||||
import { EnvironmentStateExtraction } from "./EnvironmentStateExtraction.js";
|
||||
import type { TemplateRule } from "./environmentManifestRules.js";
|
||||
import {
|
||||
@@ -32,14 +30,14 @@ import {
|
||||
import {
|
||||
base64PrefixChars,
|
||||
readRecordedResponseBody,
|
||||
readRecordedResponseDeclaration,
|
||||
recordedResponseBodyIsEmpty,
|
||||
type RecordedResponseBody,
|
||||
} from "./recordedResponseBody.js";
|
||||
import {
|
||||
ENTRY_BYTES,
|
||||
RECORD_SCRATCH_BYTES,
|
||||
mediaTypeEssence,
|
||||
readRecordedRequestLine,
|
||||
type RecordedRequestLine,
|
||||
textBytes,
|
||||
} from "./webNetworkRouteTemplates.js";
|
||||
|
||||
@@ -96,22 +94,6 @@ const redactedWriteSchema = z.object({
|
||||
request: z.object({ method: z.string() }),
|
||||
response: z.object({ status: z.number() }).optional(),
|
||||
});
|
||||
const contentSchema = z.object({
|
||||
response: z.object({ content: z.object({ text: z.string() }) }),
|
||||
});
|
||||
const requestSchema = z.object({
|
||||
request: z.object({
|
||||
postData: z.unknown().optional(),
|
||||
bodySize: z.number().optional(),
|
||||
}),
|
||||
});
|
||||
|
||||
const requestContentSchema = z.object({
|
||||
mimeType: z.string(),
|
||||
text: z.string(),
|
||||
params: z.array(z.unknown()).optional(),
|
||||
});
|
||||
|
||||
interface RecordEntry {
|
||||
readonly facts: StateRecordFacts;
|
||||
/** Null for a write whose URL was redacted: its target is unknown, so it bounds every type. */
|
||||
@@ -157,91 +139,83 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
|
||||
return false;
|
||||
}
|
||||
|
||||
override add(
|
||||
protected override onRecordedRequestLine(
|
||||
record: WebNetworkCaptureRecord,
|
||||
scenarioResponseBody?: ScenarioResponseBodyRetention,
|
||||
line: RecordedRequestLine,
|
||||
): Result<void, AnalysisError> {
|
||||
const grouped = super.add(record, scenarioResponseBody);
|
||||
if (!grouped.ok) return grouped;
|
||||
try {
|
||||
const line = readRecordedRequestLine(record.reported, true, this.scratch);
|
||||
if (line === null) return this.overBudget();
|
||||
const timing = timingSchema.safeParse(record.reported);
|
||||
// Admit normalization and fraction parsing before materializing their strings.
|
||||
if (
|
||||
timing.success &&
|
||||
!this.scratch(
|
||||
RECORD_SCRATCH_BYTES +
|
||||
textBytes(timing.data.startedDateTime.length + 6) * 3,
|
||||
)
|
||||
const timing = timingSchema.safeParse(record.reported);
|
||||
// Admit normalization and fraction parsing before materializing their strings.
|
||||
if (
|
||||
timing.success &&
|
||||
!this.scratch(
|
||||
RECORD_SCRATCH_BYTES +
|
||||
textBytes(timing.data.startedDateTime.length + 6) * 3,
|
||||
)
|
||||
return this.overBudget();
|
||||
const startInstant = timing.success
|
||||
? readRecordedDateTime(timing.data.startedDateTime)?.instant
|
||||
: undefined;
|
||||
// Keep the established millisecond interval policy independently of sorting.
|
||||
const start =
|
||||
startInstant === undefined
|
||||
? record.ordinal
|
||||
: startInstant.epochSecond * 1000 +
|
||||
Number(startInstant.fraction.slice(0, 3).padEnd(3, "0"));
|
||||
// A fraction substring may retain its normalized timestamp's backing store,
|
||||
// including trailing zeroes removed from the exact comparison value.
|
||||
const instantBytes =
|
||||
timing.success && startInstant !== undefined
|
||||
? ENTRY_BYTES + textBytes(timing.data.startedDateTime.length + 6)
|
||||
: 0;
|
||||
if (line.kind === "excluded")
|
||||
return line.reason === "url-redacted"
|
||||
? this.#redactedWrite(
|
||||
record,
|
||||
timing.success && startInstant !== undefined
|
||||
? {
|
||||
start,
|
||||
end: start + timing.data.time,
|
||||
startInstant,
|
||||
instantBytes,
|
||||
}
|
||||
: undefined,
|
||||
)
|
||||
: ok(undefined);
|
||||
// The entry keeps the line's strings and a reference to the already admitted record.
|
||||
if (
|
||||
!this.retain(
|
||||
ENTRY_BYTES * 8 +
|
||||
instantBytes +
|
||||
textBytes(
|
||||
line.method.length +
|
||||
line.origin.length +
|
||||
line.path.length +
|
||||
(line.query?.length ?? 0) +
|
||||
line.rawMime.length,
|
||||
),
|
||||
)
|
||||
)
|
||||
return this.overBudget();
|
||||
this.#entries.set(record.ordinal, {
|
||||
facts: {
|
||||
ordinal: record.ordinal,
|
||||
method: line.method,
|
||||
path: line.path,
|
||||
query: line.query === "" ? undefined : line.query,
|
||||
status: line.status,
|
||||
start,
|
||||
end:
|
||||
)
|
||||
return this.overBudget();
|
||||
const startInstant = timing.success
|
||||
? readRecordedDateTime(timing.data.startedDateTime)?.instant
|
||||
: undefined;
|
||||
// Keep the established millisecond interval policy independently of sorting.
|
||||
const start =
|
||||
startInstant === undefined
|
||||
? record.ordinal
|
||||
: startInstant.epochSecond * 1000 +
|
||||
Number(startInstant.fraction.slice(0, 3).padEnd(3, "0"));
|
||||
// A fraction substring may retain its normalized timestamp's backing store,
|
||||
// including trailing zeroes removed from the exact comparison value.
|
||||
const instantBytes =
|
||||
timing.success && startInstant !== undefined
|
||||
? ENTRY_BYTES + textBytes(timing.data.startedDateTime.length + 6)
|
||||
: 0;
|
||||
if (line.kind === "excluded")
|
||||
return line.reason === "url-redacted"
|
||||
? this.#redactedWrite(
|
||||
record,
|
||||
timing.success && startInstant !== undefined
|
||||
? start + timing.data.time
|
||||
: record.ordinal,
|
||||
},
|
||||
origin: line.origin,
|
||||
contentType: line.rawMime,
|
||||
startInstant,
|
||||
record,
|
||||
});
|
||||
return ok(undefined);
|
||||
} finally {
|
||||
this.releaseTemporary();
|
||||
}
|
||||
? {
|
||||
start,
|
||||
end: start + timing.data.time,
|
||||
startInstant,
|
||||
instantBytes,
|
||||
}
|
||||
: undefined,
|
||||
)
|
||||
: ok(undefined);
|
||||
// The entry keeps the line's strings and a reference to the already admitted record.
|
||||
if (
|
||||
!this.retain(
|
||||
ENTRY_BYTES * 8 +
|
||||
instantBytes +
|
||||
textBytes(
|
||||
line.method.length +
|
||||
line.origin.length +
|
||||
line.path.length +
|
||||
(line.query?.length ?? 0) +
|
||||
line.rawMime.length,
|
||||
),
|
||||
)
|
||||
)
|
||||
return this.overBudget();
|
||||
this.#entries.set(record.ordinal, {
|
||||
facts: {
|
||||
ordinal: record.ordinal,
|
||||
method: line.method,
|
||||
path: line.path,
|
||||
query: line.query === "" ? undefined : line.query,
|
||||
status: line.status,
|
||||
start,
|
||||
end:
|
||||
timing.success && startInstant !== undefined
|
||||
? start + timing.data.time
|
||||
: record.ordinal,
|
||||
},
|
||||
origin: line.origin,
|
||||
contentType: line.rawMime,
|
||||
startInstant,
|
||||
record,
|
||||
});
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
/** Keep a successful write whose URL was redacted, with an empty path whose root relates to every type. */
|
||||
@@ -294,17 +268,22 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
|
||||
// A body that is not JSON-like is refused before its decoding is charged.
|
||||
if (entry === undefined || !isJsonLike(mediaTypeEssence(entry.contentType)))
|
||||
return { kind: "unusable" };
|
||||
const content = contentSchema.safeParse(entry.record.reported);
|
||||
if (!content.success) return { kind: "unusable" };
|
||||
const declaration = readRecordedResponseDeclaration(entry.record);
|
||||
const content = declaration?.content;
|
||||
if (typeof content?.text !== "string") return { kind: "unusable" };
|
||||
// Larger bodies are never parsed; within the limit, decoding copies at most the retained text and the tree is linear in it.
|
||||
const length = content.data.response.content.text.length;
|
||||
const length = content.text.length;
|
||||
if (length > base64PrefixChars(STATE_BODY_LIMIT_BYTES))
|
||||
return { kind: "unusable" };
|
||||
if (!this.temporary(RECORD_SCRATCH_BYTES + textBytes(length) * 4))
|
||||
return null;
|
||||
const value = stateBodyJson(
|
||||
entry.contentType,
|
||||
readRecordedResponseBody(entry.record, STATE_BODY_LIMIT_BYTES),
|
||||
readRecordedResponseBody(
|
||||
entry.record,
|
||||
STATE_BODY_LIMIT_BYTES,
|
||||
declaration,
|
||||
),
|
||||
);
|
||||
return value === undefined ? { kind: "unusable" } : { kind: "json", value };
|
||||
}
|
||||
@@ -315,16 +294,30 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
|
||||
use: (body: ParsedRequestBody) => T,
|
||||
): T | null {
|
||||
const entry = this.#entries.get(ordinal);
|
||||
const request = requestSchema.safeParse(entry?.record.reported);
|
||||
if (entry === undefined || !request.success)
|
||||
const request = readRecordedRequestBody(entry?.record.reported);
|
||||
if (
|
||||
entry === undefined ||
|
||||
request.kind !== "request" ||
|
||||
request.bodySize.kind === "invalid"
|
||||
)
|
||||
return use({ kind: "unusable" });
|
||||
const { postData, bodySize } = request.data.request;
|
||||
if (postData === undefined)
|
||||
return use({ kind: bodySize === 0 ? "absent" : "unusable" });
|
||||
const parsed = requestContentSchema.safeParse(postData);
|
||||
if (!parsed.success) return use({ kind: "unusable" });
|
||||
const { mimeType, text, params } = parsed.data;
|
||||
if (bodySize === 0 && text === "" && (params?.length ?? 0) === 0)
|
||||
const { postData, bodySize } = request;
|
||||
const zeroSize = bodySize.kind === "number" && bodySize.value === 0;
|
||||
if (postData.kind === "missing")
|
||||
return use({ kind: zeroSize ? "absent" : "unusable" });
|
||||
if (
|
||||
postData.kind !== "object" ||
|
||||
postData.text === undefined ||
|
||||
postData.mimeType === undefined ||
|
||||
postData.params.kind === "invalid"
|
||||
)
|
||||
return use({ kind: "unusable" });
|
||||
const { mimeType, text, params } = postData;
|
||||
if (
|
||||
zeroSize &&
|
||||
text === "" &&
|
||||
(params.kind === "array" ? params.count : 0) === 0
|
||||
)
|
||||
return use({ kind: "absent" });
|
||||
if (
|
||||
!isJsonLike(mediaTypeEssence(mimeType)) ||
|
||||
@@ -344,7 +337,7 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether a record's response carried no body: a 204, or known decoded empty content. */
|
||||
/** State inference accepts a 204 or known decoded empty content, independently of manifest eligibility. */
|
||||
#emptyResponse(ordinal: number): boolean {
|
||||
const entry = this.#entries.get(ordinal);
|
||||
if (entry === undefined) return false;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { mediaTypeEssence } from "./recordedMediaType.js";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js";
|
||||
import { ok, type Result } from "../result.js";
|
||||
@@ -14,7 +15,6 @@ import {
|
||||
GROUP_BYTES,
|
||||
RECORD_SCRATCH_BYTES,
|
||||
SEGMENT_BYTES,
|
||||
mediaTypeEssence,
|
||||
readQueryParameterNames,
|
||||
readRecordedRequestLine,
|
||||
scanRouteTemplate,
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
/** Parsed declarations borrow source strings; they do not decode bodies or normalize media types. */
|
||||
export type RecordedRequestBody =
|
||||
| { readonly kind: "request-unavailable" }
|
||||
| {
|
||||
readonly kind: "request";
|
||||
readonly bodySize:
|
||||
| { readonly kind: "missing" | "invalid" }
|
||||
| { readonly kind: "number"; readonly value: number };
|
||||
readonly postData:
|
||||
| { readonly kind: "missing" | "null" | "malformed" }
|
||||
| {
|
||||
readonly kind: "object";
|
||||
readonly text: string | undefined;
|
||||
readonly mimeType: string | undefined;
|
||||
readonly params:
|
||||
| { readonly kind: "missing" | "invalid" }
|
||||
| { readonly kind: "array"; readonly count: number };
|
||||
};
|
||||
};
|
||||
|
||||
const object = (value: unknown): Record<string, unknown> | undefined =>
|
||||
typeof value === "object" && value !== null && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: undefined;
|
||||
|
||||
/** Read HAR request body declarations once without turning omission or null into absence. */
|
||||
export const readRecordedRequestBody = (
|
||||
reported: unknown,
|
||||
): RecordedRequestBody => {
|
||||
const request = object(object(reported)?.request);
|
||||
if (request === undefined) return { kind: "request-unavailable" };
|
||||
const size = request.bodySize;
|
||||
const bodySize: Extract<
|
||||
RecordedRequestBody,
|
||||
{ kind: "request" }
|
||||
>["bodySize"] =
|
||||
size === undefined
|
||||
? { kind: "missing" }
|
||||
: typeof size === "number" && Number.isFinite(size)
|
||||
? { kind: "number", value: size }
|
||||
: { kind: "invalid" };
|
||||
const source = request.postData;
|
||||
const postData = object(source);
|
||||
return {
|
||||
kind: "request",
|
||||
bodySize,
|
||||
postData:
|
||||
postData === undefined
|
||||
? {
|
||||
kind:
|
||||
source === undefined
|
||||
? "missing"
|
||||
: source === null
|
||||
? "null"
|
||||
: "malformed",
|
||||
}
|
||||
: {
|
||||
kind: "object",
|
||||
text: typeof postData.text === "string" ? postData.text : undefined,
|
||||
mimeType:
|
||||
typeof postData.mimeType === "string"
|
||||
? postData.mimeType
|
||||
: undefined,
|
||||
params:
|
||||
postData.params === undefined
|
||||
? { kind: "missing" }
|
||||
: Array.isArray(postData.params)
|
||||
? { kind: "array", count: postData.params.length }
|
||||
: { kind: "invalid" },
|
||||
},
|
||||
};
|
||||
};
|
||||
@@ -0,0 +1,34 @@
|
||||
const MEDIA_TYPE_ESSENCE =
|
||||
/^[A-Za-z0-9!#$%&'*+.^_`|~-]+\/[A-Za-z0-9!#$%&'*+.^_`|~-]+$/u;
|
||||
|
||||
const JSON_LIKE_TEXT = new Set(["text/plain", "text/markdown"]);
|
||||
|
||||
/** Lowercase type/subtype without parameters; malformed essences are unknown. */
|
||||
export const mediaTypeEssence = (reported: string): string | null => {
|
||||
const separator = reported.indexOf(";");
|
||||
let start = 0;
|
||||
let end = separator < 0 ? reported.length : separator;
|
||||
// HTTP OWS is SP/HTAB. Scan from both ends so malformed internal runs of
|
||||
// whitespace cannot cause repeated trailing-regex searches.
|
||||
while (start < end && (reported[start] === " " || reported[start] === "\t"))
|
||||
start += 1;
|
||||
while (
|
||||
end > start &&
|
||||
(reported[end - 1] === " " || reported[end - 1] === "\t")
|
||||
)
|
||||
end -= 1;
|
||||
const essence = reported.slice(start, end);
|
||||
// Validate ASCII tokens before case folding can turn Unicode into ASCII.
|
||||
return MEDIA_TYPE_ESSENCE.test(essence) ? essence.toLowerCase() : null;
|
||||
};
|
||||
|
||||
/** `application/json` or `application/*+json`; a `+json` suffix on another type is not JSON here. */
|
||||
export const isJson = (essence: string | null): boolean =>
|
||||
essence === "application/json" ||
|
||||
(essence !== null &&
|
||||
essence.startsWith("application/") &&
|
||||
essence.endsWith("+json"));
|
||||
|
||||
/** JSON, `+json`, `text/plain` or `text/markdown`: the content types a replay parses as JSON. */
|
||||
export const isJsonLike = (essence: string | null): boolean =>
|
||||
isJson(essence) || (essence !== null && JSON_LIKE_TEXT.has(essence));
|
||||
@@ -11,15 +11,32 @@ export interface RecordedResponseBody {
|
||||
readonly bodyCut?: true;
|
||||
}
|
||||
|
||||
const contentTextSchema = z.object({
|
||||
const responseDeclarationSchema = z.object({
|
||||
response: z.object({
|
||||
content: z.object({
|
||||
text: z.unknown().optional(),
|
||||
encoding: z.unknown().optional(),
|
||||
}),
|
||||
status: z.unknown().optional(),
|
||||
content: z
|
||||
.object({
|
||||
text: z.unknown().optional(),
|
||||
encoding: z.unknown().optional(),
|
||||
size: z.unknown().optional(),
|
||||
})
|
||||
.nullable()
|
||||
.optional(),
|
||||
}),
|
||||
});
|
||||
|
||||
/** Retained response declarations shared by decoding and consumer-specific empty-body policies. */
|
||||
export const readRecordedResponseDeclaration = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
) => {
|
||||
const parsed = responseDeclarationSchema.safeParse(record.reported);
|
||||
return parsed.success ? parsed.data.response : undefined;
|
||||
};
|
||||
|
||||
export type RecordedResponseDeclaration = ReturnType<
|
||||
typeof readRecordedResponseDeclaration
|
||||
>;
|
||||
|
||||
/** Strict, and keeping a leading byte-order mark, so callers see the recorded text unchanged. */
|
||||
const strictUtf8 = (): TextDecoder =>
|
||||
new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
|
||||
@@ -53,10 +70,13 @@ export const decodeRetainedBase64 = (
|
||||
export const readRecordedResponseBody = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
limitBytes: number,
|
||||
declaration: RecordedResponseDeclaration = readRecordedResponseDeclaration(
|
||||
record,
|
||||
),
|
||||
): RecordedResponseBody => {
|
||||
const content = contentTextSchema.safeParse(record.reported);
|
||||
if (!content.success) return { bodyText: null };
|
||||
const { text, encoding } = content.data.response.content;
|
||||
const content = declaration?.content;
|
||||
if (content == null) return { bodyText: null };
|
||||
const { text, encoding } = content;
|
||||
if (typeof text !== "string") return { bodyText: null };
|
||||
if (encoding === undefined) {
|
||||
// A hidden content property could have been the encoding declaration;
|
||||
@@ -80,8 +100,11 @@ export const readRecordedResponseBody = (
|
||||
/** Prove decoded emptiness with a bounded prefix, preserving encoding and cut uncertainty. */
|
||||
export const recordedResponseBodyIsEmpty = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
declaration: RecordedResponseDeclaration = readRecordedResponseDeclaration(
|
||||
record,
|
||||
),
|
||||
): boolean => {
|
||||
const body = readRecordedResponseBody(record, 0);
|
||||
const body = readRecordedResponseBody(record, 0, declaration);
|
||||
return (
|
||||
body.bodyText === "" &&
|
||||
body.bodyEncoding === undefined &&
|
||||
|
||||
@@ -52,9 +52,6 @@ const responseSchema = z.object({
|
||||
}),
|
||||
});
|
||||
const contentSchema = z.object({ mimeType: z.unknown().optional() });
|
||||
const MEDIA_TYPE_ESSENCE =
|
||||
/^[A-Za-z0-9!#$%&'*+.^_`|~-]+\/[A-Za-z0-9!#$%&'*+.^_`|~-]+$/u;
|
||||
|
||||
// Conservative representation allowances, not an enforced V8 heap limit.
|
||||
// Fixed charges include the accumulator, bucket/set entries, array growth and
|
||||
// page/schema projection. Text includes UTF-16 storage and two worst-case JSON
|
||||
@@ -67,25 +64,6 @@ export const CAPTURE_CONTAINER_BYTES = 512;
|
||||
const SEGMENT_SCRATCH_BYTES = 512;
|
||||
export const textBytes = (length: number): number => 32 + length * 14;
|
||||
|
||||
/** Lowercase type/subtype without parameters; malformed essences are unknown. */
|
||||
export const mediaTypeEssence = (reported: string): string | null => {
|
||||
const separator = reported.indexOf(";");
|
||||
let start = 0;
|
||||
let end = separator < 0 ? reported.length : separator;
|
||||
// HTTP OWS is SP/HTAB. Scan from both ends so malformed internal runs of
|
||||
// whitespace cannot cause repeated trailing-regex searches.
|
||||
while (start < end && (reported[start] === " " || reported[start] === "\t"))
|
||||
start += 1;
|
||||
while (
|
||||
end > start &&
|
||||
(reported[end - 1] === " " || reported[end - 1] === "\t")
|
||||
)
|
||||
end -= 1;
|
||||
const essence = reported.slice(start, end);
|
||||
// Validate ASCII tokens before case folding can turn Unicode into ASCII.
|
||||
return MEDIA_TYPE_ESSENCE.test(essence) ? essence.toLowerCase() : null;
|
||||
};
|
||||
|
||||
/** Route parts of a template: one more than its separators. */
|
||||
export const routePartCount = (template: string): number => {
|
||||
let count = 1;
|
||||
|
||||
@@ -46,6 +46,12 @@ export interface FatSliceHeaderEvidence {
|
||||
export interface MachoHeaderPrefixFacts {
|
||||
/** Raw format evidence; null means fewer than four bytes were observed. */
|
||||
readonly magic: number | null;
|
||||
/** Present for a recognized four-byte thin magic, before header admission. */
|
||||
readonly format: {
|
||||
readonly wide: boolean;
|
||||
readonly littleEndian: boolean;
|
||||
readonly headerSize: 28 | 32;
|
||||
} | null;
|
||||
/** Present only for a recognized Mach-O magic and a complete 12-byte prefix. */
|
||||
readonly identity: {
|
||||
readonly cpuType: number;
|
||||
@@ -57,18 +63,22 @@ export interface MachoHeaderPrefixFacts {
|
||||
export const readMachoHeaderPrefixFacts = (
|
||||
bytes: Uint8Array,
|
||||
): MachoHeaderPrefixFacts => {
|
||||
if (bytes.byteLength < 4) return { magic: null, identity: null };
|
||||
if (bytes.byteLength < 4)
|
||||
return { magic: null, format: null, identity: null };
|
||||
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
||||
const magic = view.getUint32(0, false);
|
||||
if (bytes.byteLength < 12 || !MACHO_HEADER_MAGICS.includes(magic))
|
||||
return { magic, identity: null };
|
||||
const littleEndian = magic === 0xcefaedfe || magic === 0xcffaedfe;
|
||||
const format = thinMachoFormat(magic);
|
||||
if (format === null) return { magic, format: null, identity: null };
|
||||
return {
|
||||
magic,
|
||||
identity: {
|
||||
cpuType: view.getUint32(4, littleEndian),
|
||||
cpuSubtype: view.getUint32(8, littleEndian),
|
||||
},
|
||||
format,
|
||||
identity:
|
||||
bytes.byteLength < 12
|
||||
? null
|
||||
: {
|
||||
cpuType: view.getUint32(4, format.littleEndian),
|
||||
cpuSubtype: view.getUint32(8, format.littleEndian),
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
@@ -108,9 +118,23 @@ const MAX_FAT_ARCHITECTURES = 128;
|
||||
const KNOWN_MACHO_CPU_TYPES: readonly number[] = [
|
||||
7, 0x01000007, 12, 0x0100000c, 0x0200000c, 18, 0x01000012,
|
||||
];
|
||||
const MACHO_HEADER_MAGICS: readonly number[] = [
|
||||
0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe,
|
||||
];
|
||||
const thinMachoFormat = (magic: number): MachoHeaderPrefixFacts["format"] => {
|
||||
if (![0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(magic))
|
||||
return null;
|
||||
const wide = magic === 0xfeedfacf || magic === 0xcffaedfe;
|
||||
return {
|
||||
wide,
|
||||
littleEndian: magic === 0xcefaedfe || magic === 0xcffaedfe,
|
||||
headerSize: wide ? 32 : 28,
|
||||
};
|
||||
};
|
||||
|
||||
/** Raw thin/FAT signature recognition; FAT magic alone does not resolve JVM ambiguity. */
|
||||
export const hasMachoMagic = (bytes: Uint8Array): boolean => {
|
||||
if (readMachoHeaderPrefixFacts(bytes).format !== null) return true;
|
||||
const fat = readFatHeaderPrefix(bytes);
|
||||
return fat.state !== "unknown" && fat.state !== "not-fat";
|
||||
};
|
||||
|
||||
interface FatHeaderFormat {
|
||||
readonly magic: number;
|
||||
@@ -131,18 +155,24 @@ export type FatHeaderPrefix =
|
||||
}
|
||||
));
|
||||
|
||||
/** Interpret a raw FAT magic without asserting container identity or header readiness. */
|
||||
export const readFatMagicFormat = (magic: number): FatHeaderFormat | null => {
|
||||
if (![FAT_MAGIC, FAT_MAGIC_64, FAT_CIGAM, FAT_CIGAM_64].includes(magic))
|
||||
return null;
|
||||
return {
|
||||
magic,
|
||||
wide: magic === FAT_MAGIC_64 || magic === FAT_CIGAM_64,
|
||||
littleEndian: magic === FAT_CIGAM || magic === FAT_CIGAM_64,
|
||||
};
|
||||
};
|
||||
|
||||
/** Observe the fixed FAT header without table, count-limit or container-identity admission. */
|
||||
export const readFatHeaderPrefix = (bytes: Uint8Array): FatHeaderPrefix => {
|
||||
if (bytes.byteLength < 4) return { state: "unknown", requiredBytes: 4 };
|
||||
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
||||
const magic = view.getUint32(0, false);
|
||||
if (![FAT_MAGIC, FAT_MAGIC_64, FAT_CIGAM, FAT_CIGAM_64].includes(magic))
|
||||
return { state: "not-fat", magic };
|
||||
const format = {
|
||||
magic,
|
||||
wide: magic === FAT_MAGIC_64 || magic === FAT_CIGAM_64,
|
||||
littleEndian: magic === FAT_CIGAM || magic === FAT_CIGAM_64,
|
||||
};
|
||||
const format = readFatMagicFormat(magic);
|
||||
if (format === null) return { state: "not-fat", magic };
|
||||
if (bytes.byteLength < 8)
|
||||
return { state: "incomplete", ...format, requiredBytes: 8 };
|
||||
const count = view.getUint32(4, format.littleEndian);
|
||||
@@ -255,7 +285,7 @@ export const classifyFatContainerIdentity = (
|
||||
const magic = sliceHeaderAt(slice.offset);
|
||||
if (magic === undefined || magic === null)
|
||||
missingHeaders.push(slice.offset);
|
||||
else if (MACHO_HEADER_MAGICS.includes(magic)) observedMachO = true;
|
||||
else if (thinMachoFormat(magic) !== null) observedMachO = true;
|
||||
else observedNonMachO = true;
|
||||
}
|
||||
if (observedNonMachO) return { state: "not-mach-o" };
|
||||
|
||||
@@ -1,4 +1,9 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { hasMachoMagic, readMachoHeaderPrefixFacts } from "./machoContainer.js";
|
||||
import {
|
||||
claimsExecutableHeader,
|
||||
parseExecutableHeader,
|
||||
} from "../binaryTarget.js";
|
||||
import {
|
||||
readMachoHeaderDeclarations,
|
||||
readMachoLoadCommandDeclarations,
|
||||
@@ -31,6 +36,29 @@ it.each([
|
||||
fileType: 0xffffffff,
|
||||
identity: { cpuType: 0xfedcba98, cpuSubtype: 0x89abcdef },
|
||||
};
|
||||
const format = { wide, littleEndian, headerSize: wide ? 32 : 28 };
|
||||
for (const length of [0, 3, 4, 11, 12]) {
|
||||
const prefix = bytes.subarray(0, length);
|
||||
expect(hasMachoMagic(prefix)).toBe(length >= 4);
|
||||
expect(readMachoHeaderPrefixFacts(prefix)).toEqual({
|
||||
magic: length < 4 ? null : magic,
|
||||
format: length < 4 ? null : format,
|
||||
identity: length < 12 ? null : expected.identity,
|
||||
});
|
||||
expect(readMachoHeaderDeclarations(prefix, 32)).toEqual(
|
||||
length < 4
|
||||
? { ok: true, value: null }
|
||||
: {
|
||||
ok: false,
|
||||
error: `truncated Mach-O header: a ${wide ? 64 : 32}-bit header needs ${format.headerSize} bytes; the file has 32`,
|
||||
},
|
||||
);
|
||||
}
|
||||
expect(claimsExecutableHeader(bytes.subarray(0, 4))).toBe(true);
|
||||
expect(parseExecutableHeader(bytes.subarray(0, 4), "arm64")).toEqual({
|
||||
ok: false,
|
||||
error: "truncated or unsupported binary header",
|
||||
});
|
||||
expect(readMachoHeaderDeclarations(bytes, 32)).toEqual({
|
||||
ok: true,
|
||||
value: expected,
|
||||
@@ -57,3 +85,25 @@ it.each([
|
||||
).toMatchObject({ ok: false });
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
0xcafebabe, 0xcafebabf, 0xbebafeca, 0xbfbafeca, 0x7f454c46, 0xffffffff,
|
||||
])(
|
||||
"retains raw non-thin magic %s without thin format or CPU claims",
|
||||
(magic) => {
|
||||
const bytes = Buffer.alloc(12, 0xff);
|
||||
bytes.writeUInt32BE(magic, 0);
|
||||
expect(hasMachoMagic(bytes.subarray(0, 4))).toBe(
|
||||
magic !== 0x7f454c46 && magic !== 0xffffffff,
|
||||
);
|
||||
expect(readMachoHeaderPrefixFacts(bytes)).toEqual({
|
||||
magic,
|
||||
format: null,
|
||||
identity: null,
|
||||
});
|
||||
expect(readMachoHeaderDeclarations(bytes, bytes.length)).toEqual({
|
||||
ok: true,
|
||||
value: null,
|
||||
});
|
||||
},
|
||||
);
|
||||
@@ -24,19 +24,12 @@ export const readMachoHeaderDeclarations = (
|
||||
fileSize: number,
|
||||
): Result<MachoHeaderDeclarations | null, string> => {
|
||||
const prefix = readMachoHeaderPrefixFacts(bytes);
|
||||
const magic = prefix.magic;
|
||||
if (
|
||||
magic === null ||
|
||||
![0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(magic)
|
||||
)
|
||||
return ok(null);
|
||||
const wide = magic === 0xfeedfacf || magic === 0xcffaedfe;
|
||||
const headerSize = wide ? 32 : 28;
|
||||
if (prefix.format === null) return ok(null);
|
||||
const { wide, headerSize, littleEndian } = prefix.format;
|
||||
if (bytes.byteLength < 28 || prefix.identity === null)
|
||||
return err(
|
||||
`truncated Mach-O header: a ${wide ? 64 : 32}-bit header needs ${headerSize} bytes; the file has ${fileSize}`,
|
||||
);
|
||||
const littleEndian = magic === 0xcefaedfe || magic === 0xcffaedfe;
|
||||
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
||||
const commandBytes = view.getUint32(20, littleEndian);
|
||||
return ok({
|
||||
|
||||
@@ -103,9 +103,7 @@ export const claimsExecutableHeader = (
|
||||
): boolean =>
|
||||
(bytes.length >= 4 &&
|
||||
(bytes.subarray(0, 4).equals(Buffer.from([0x7f, 0x45, 0x4c, 0x46])) ||
|
||||
[0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(
|
||||
bytes.readUInt32BE(0),
|
||||
))) ||
|
||||
readMachoHeaderPrefixFacts(bytes).format !== null)) ||
|
||||
(bytes.length >= 2 && bytes[0] === 0x4d && bytes[1] === 0x5a) ||
|
||||
readFatContainerHeader(bytes, fileSize).state !== "not-fat";
|
||||
|
||||
@@ -132,8 +130,7 @@ export const parseExecutableHeader = (
|
||||
parseMzHeaderEvidence(bytes, fileSize, mzWindowsHeaderSignature),
|
||||
);
|
||||
if (bytes.length < 8) return err("truncated or unsupported binary header");
|
||||
const magic = bytes.readUInt32BE(0);
|
||||
if ([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(magic))
|
||||
if (readMachoHeaderPrefixFacts(bytes).format !== null)
|
||||
return parseThinMachO(bytes, fileSize);
|
||||
const fatHeader =
|
||||
fatEvidence?.header ?? readFatContainerHeader(bytes, fileSize);
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { readElfFixedHeader } from "./elfFixedHeader.js";
|
||||
import { readElfFixedHeader, readElfHeaderFacts } from "./elfFixedHeader.js";
|
||||
import { elf } from "./binaryTarget.fixture.js";
|
||||
|
||||
it.each([
|
||||
@@ -92,3 +92,66 @@ it("accepts NVIDIA's vendor ELF version only for the CUDA machine and OS ABI", (
|
||||
bytes.writeUInt16LE(62, 18);
|
||||
expect(readElfFixedHeader(bytes).ok).toBe(false);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ cls: 1, data: 1 },
|
||||
{ cls: 1, data: 2 },
|
||||
{ cls: 2, data: 1 },
|
||||
{ cls: 2, data: 2 },
|
||||
] as const)(
|
||||
"preserves raw version, ABI and declared size facts ($cls/$data)",
|
||||
({ cls, data }) => {
|
||||
const original = elf(cls, data, 0xfffe);
|
||||
const bytes = Buffer.concat([Buffer.alloc(7), original]).subarray(7);
|
||||
bytes[8] = 0xff;
|
||||
if (data === 1) {
|
||||
bytes.writeUInt32LE(0x89abcdef, 20);
|
||||
bytes.writeUInt16LE(0, cls === 1 ? 40 : 52);
|
||||
} else {
|
||||
bytes.writeUInt32BE(0x89abcdef, 20);
|
||||
bytes.writeUInt16BE(0, cls === 1 ? 40 : 52);
|
||||
}
|
||||
expect(readElfHeaderFacts(bytes)).toMatchObject({
|
||||
ok: true,
|
||||
value: {
|
||||
elfClass: cls === 1 ? 32 : 64,
|
||||
byteOrder: data === 1 ? "little" : "big",
|
||||
machine: 0xfffe,
|
||||
version: 0x89abcdef,
|
||||
abiVersion: 0xff,
|
||||
headerSize: 0,
|
||||
},
|
||||
});
|
||||
expect(readElfFixedHeader(bytes)).toEqual({
|
||||
ok: false,
|
||||
error: "unsupported ELF header version",
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps framing failures ahead of version and declared-size admission", () => {
|
||||
const bytes = elf(2, 1, 62);
|
||||
bytes[6] = 0;
|
||||
bytes.writeUInt32LE(0, 20);
|
||||
bytes.writeUInt16LE(0, 52);
|
||||
expect(readElfFixedHeader(bytes.subarray(0, 16), 16)).toEqual({
|
||||
ok: false,
|
||||
error: "unsupported ELF identification version",
|
||||
});
|
||||
bytes[6] = 1;
|
||||
const truncated = readElfFixedHeader(bytes, 63);
|
||||
expect(truncated).toEqual(readElfHeaderFacts(bytes, 63));
|
||||
expect(truncated).toMatchObject({
|
||||
ok: false,
|
||||
error: expect.stringContaining("truncated ELF header"),
|
||||
});
|
||||
expect(readElfFixedHeader(bytes)).toEqual({
|
||||
ok: false,
|
||||
error: "unsupported ELF header version",
|
||||
});
|
||||
bytes.writeUInt32LE(1, 20);
|
||||
expect(readElfFixedHeader(bytes)).toEqual({
|
||||
ok: false,
|
||||
error: "invalid ELF header size",
|
||||
});
|
||||
});
|
||||
@@ -10,7 +10,8 @@ type ElfArchitecture = Exclude<
|
||||
RawOnlyArchitecture | "arm64_32"
|
||||
>;
|
||||
|
||||
export interface ElfFixedHeader {
|
||||
/** Fixed declarations after common ELF framing, before consumer admission. */
|
||||
export interface ElfHeaderFacts {
|
||||
readonly elfClass: 32 | 64;
|
||||
readonly byteOrder: "little" | "big";
|
||||
readonly headerSize: number;
|
||||
@@ -18,20 +19,28 @@ export interface ElfFixedHeader {
|
||||
readonly type: number;
|
||||
readonly flags: number;
|
||||
readonly osAbi: number;
|
||||
readonly version: number;
|
||||
readonly abiVersion: number;
|
||||
}
|
||||
|
||||
export interface ElfFixedHeader extends Omit<
|
||||
ElfHeaderFacts,
|
||||
"version" | "abiVersion"
|
||||
> {
|
||||
readonly architecture: ElfArchitecture | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate the common, fixed ELF header commitments.
|
||||
* Decode common ELF framing and fixed fields; version and declared-size admission belong to consumers.
|
||||
*
|
||||
* `bytes` may be a bounded prefix. `totalBytes` is the containing file size,
|
||||
* so the declared header extent is checked against the file without requiring
|
||||
* the whole declared extent to be present in this prefix.
|
||||
* `bytes` may be a bounded prefix. `totalBytes` is the containing file size;
|
||||
* common framing requires a complete fixed header in both. The declared
|
||||
* headerSize is retained without requiring or admitting that entire extent.
|
||||
*/
|
||||
export const readElfFixedHeader = (
|
||||
export const readElfHeaderFacts = (
|
||||
bytes: Buffer,
|
||||
totalBytes = bytes.length,
|
||||
): Result<ElfFixedHeader, string> => {
|
||||
): Result<ElfHeaderFacts, string> => {
|
||||
if (bytes.length >= 4 && bytes.toString("hex", 0, 4) !== "7f454c46")
|
||||
return err("not an ELF image");
|
||||
if (bytes.length < 16) return err("ELF identification is truncated");
|
||||
@@ -59,12 +68,7 @@ export const readElfFixedHeader = (
|
||||
const readU32 = (offset: number): number =>
|
||||
littleEndian ? bytes.readUInt32LE(offset) : bytes.readUInt32BE(offset);
|
||||
const machine = readU16(18);
|
||||
// NVIDIA CUBIN uses ELFOSABI_CUDA and a vendor ABI version in e_version.
|
||||
if (readU32(20) !== 1 && !(machine === 190 && bytes[7] === 0x33))
|
||||
return err("unsupported ELF header version");
|
||||
const headerSize = readU16(elfClass === 32 ? 40 : 52);
|
||||
if (headerSize < minimumSize || headerSize > totalBytes)
|
||||
return err("invalid ELF header size");
|
||||
return ok({
|
||||
elfClass,
|
||||
byteOrder: littleEndian ? "little" : "big",
|
||||
@@ -73,7 +77,34 @@ export const readElfFixedHeader = (
|
||||
type: readU16(16),
|
||||
flags: readU32(elfClass === 32 ? 36 : 48),
|
||||
osAbi: bytes[7]!,
|
||||
architecture: elfArchitecture(machine) ?? null,
|
||||
version: readU32(20),
|
||||
abiVersion: bytes[8]!,
|
||||
});
|
||||
};
|
||||
|
||||
/** Admit the ELF version and declared header extent for general analysis. */
|
||||
export const readElfFixedHeader = (
|
||||
bytes: Buffer,
|
||||
totalBytes = bytes.length,
|
||||
): Result<ElfFixedHeader, string> => {
|
||||
const parsed = readElfHeaderFacts(bytes, totalBytes);
|
||||
if (!parsed.ok) return parsed;
|
||||
const facts = parsed.value;
|
||||
// NVIDIA CUBIN uses ELFOSABI_CUDA and a vendor ABI version in e_version.
|
||||
if (facts.version !== 1 && !(facts.machine === 190 && facts.osAbi === 0x33))
|
||||
return err("unsupported ELF header version");
|
||||
const minimumSize = facts.elfClass === 32 ? 52 : 64;
|
||||
if (facts.headerSize < minimumSize || facts.headerSize > totalBytes)
|
||||
return err("invalid ELF header size");
|
||||
return ok({
|
||||
elfClass: facts.elfClass,
|
||||
byteOrder: facts.byteOrder,
|
||||
headerSize: facts.headerSize,
|
||||
machine: facts.machine,
|
||||
type: facts.type,
|
||||
flags: facts.flags,
|
||||
osAbi: facts.osAbi,
|
||||
architecture: elfArchitecture(facts.machine) ?? null,
|
||||
});
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
/** Bounded source observation; consumers own scan limits, charging and caching. */
|
||||
export type ElfSectionNameObservation =
|
||||
| {
|
||||
readonly state: "outside";
|
||||
readonly bytes: null;
|
||||
readonly examinedBytes: 0;
|
||||
readonly contentBytes: 0;
|
||||
}
|
||||
| {
|
||||
readonly state: "terminated";
|
||||
readonly bytes: Buffer;
|
||||
readonly examinedBytes: number;
|
||||
readonly contentBytes: number;
|
||||
}
|
||||
| {
|
||||
readonly state: "truncated";
|
||||
readonly bytes: null;
|
||||
readonly examinedBytes: number;
|
||||
readonly contentBytes: number;
|
||||
};
|
||||
|
||||
/** Observe an ELF string-table name without treating a bounded missing NUL as absence. */
|
||||
export const observeElfSectionName = (
|
||||
table: Buffer,
|
||||
offset: number,
|
||||
maximumBytes: number,
|
||||
): ElfSectionNameObservation => {
|
||||
if (offset < 0 || offset >= table.length)
|
||||
return { state: "outside", bytes: null, examinedBytes: 0, contentBytes: 0 };
|
||||
const window = table.subarray(offset, offset + maximumBytes);
|
||||
const end = window.indexOf(0);
|
||||
if (end < 0)
|
||||
return {
|
||||
state: "truncated",
|
||||
bytes: null,
|
||||
examinedBytes: window.length,
|
||||
contentBytes: window.length,
|
||||
};
|
||||
return {
|
||||
state: "terminated",
|
||||
bytes: window.subarray(0, end),
|
||||
examinedBytes: end + 1,
|
||||
contentBytes: end,
|
||||
};
|
||||
};
|
||||
|
||||
/** Project admitted source bytes without inferring an original text encoding. */
|
||||
export const elfSectionNameText = (
|
||||
observed: ElfSectionNameObservation,
|
||||
): string | null => observed.bytes?.toString("latin1") ?? null;
|
||||
+107
-102
@@ -88,87 +88,88 @@ export const analysisBookmarkSchema = z.strictObject({
|
||||
name: z.string().nullable(),
|
||||
});
|
||||
/** Complete observed function-body ranges; every range endpoint is inclusive. */
|
||||
export const functionBodySchema = z.discriminatedUnion("available", [
|
||||
unavailableAnalysisFactSchema,
|
||||
z
|
||||
.strictObject({
|
||||
available: z.literal(true),
|
||||
provenance: z.string().min(1),
|
||||
ranges: z.array(
|
||||
z.strictObject({
|
||||
start: z.string(),
|
||||
end: z
|
||||
.string()
|
||||
.describe("Inclusive last address of this observed body range."),
|
||||
}),
|
||||
),
|
||||
total_bytes: z.int().min(0).max(Number.MAX_SAFE_INTEGER),
|
||||
span_bytes: z.int().min(0).max(Number.MAX_SAFE_INTEGER).nullable(),
|
||||
non_contiguous: z.boolean(),
|
||||
contains_entry: z.boolean(),
|
||||
})
|
||||
.superRefine((body, context) => {
|
||||
let total = 0n;
|
||||
const bounds = new Map<string | null, { start: bigint; end: bigint }>();
|
||||
const closedSpaces = new Set<string | null>();
|
||||
let previousSpace: string | null | undefined;
|
||||
for (const range of body.ranges) {
|
||||
const start = addressCoordinate(range.start);
|
||||
const end = addressCoordinate(range.end);
|
||||
if (
|
||||
start === null ||
|
||||
end === null ||
|
||||
start.space !== end.space ||
|
||||
end.offset < start.offset
|
||||
) {
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message: "Function body range has invalid inclusive endpoints",
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (previousSpace !== undefined && previousSpace !== start.space)
|
||||
closedSpaces.add(previousSpace);
|
||||
const previous = bounds.get(start.space);
|
||||
if (
|
||||
closedSpaces.has(start.space) ||
|
||||
(previous !== undefined && start.offset <= previous.end + 1n)
|
||||
) {
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message:
|
||||
"Function body ranges must be ordered, disjoint and maximal within each address space",
|
||||
});
|
||||
return;
|
||||
}
|
||||
bounds.set(start.space, {
|
||||
start: previous?.start ?? start.offset,
|
||||
end: end.offset,
|
||||
});
|
||||
previousSpace = start.space;
|
||||
total += end.offset - start.offset + 1n;
|
||||
}
|
||||
const onlyBounds =
|
||||
bounds.size === 1 ? [...bounds.values()][0] : undefined;
|
||||
const span =
|
||||
bounds.size === 0
|
||||
? 0n
|
||||
: onlyBounds === undefined
|
||||
? null
|
||||
: onlyBounds.end - onlyBounds.start + 1n;
|
||||
const reportedSpan =
|
||||
body.span_bytes === null ? null : BigInt(body.span_bytes);
|
||||
export const availableFunctionBodySchema = z
|
||||
.strictObject({
|
||||
available: z.literal(true),
|
||||
provenance: z.string().min(1),
|
||||
ranges: z.array(
|
||||
z.strictObject({
|
||||
start: z.string(),
|
||||
end: z
|
||||
.string()
|
||||
.describe("Inclusive last address of this observed body range."),
|
||||
}),
|
||||
),
|
||||
total_bytes: z.int().min(0).max(Number.MAX_SAFE_INTEGER),
|
||||
span_bytes: z.int().min(0).max(Number.MAX_SAFE_INTEGER).nullable(),
|
||||
non_contiguous: z.boolean(),
|
||||
contains_entry: z.boolean(),
|
||||
})
|
||||
.superRefine((body, context) => {
|
||||
let total = 0n;
|
||||
const bounds = new Map<string | null, { start: bigint; end: bigint }>();
|
||||
const closedSpaces = new Set<string | null>();
|
||||
let previousSpace: string | null | undefined;
|
||||
for (const range of body.ranges) {
|
||||
const start = addressCoordinate(range.start);
|
||||
const end = addressCoordinate(range.end);
|
||||
if (
|
||||
total !== BigInt(body.total_bytes) ||
|
||||
span !== reportedSpan ||
|
||||
body.non_contiguous !== body.ranges.length > 1
|
||||
)
|
||||
start === null ||
|
||||
end === null ||
|
||||
start.space !== end.space ||
|
||||
end.offset < start.offset
|
||||
) {
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message: "Function body range has invalid inclusive endpoints",
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (previousSpace !== undefined && previousSpace !== start.space)
|
||||
closedSpaces.add(previousSpace);
|
||||
const previous = bounds.get(start.space);
|
||||
if (
|
||||
closedSpaces.has(start.space) ||
|
||||
(previous !== undefined && start.offset <= previous.end + 1n)
|
||||
) {
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message:
|
||||
"Function body counts, enclosing span or contiguity disagree with inclusive ranges",
|
||||
"Function body ranges must be ordered, disjoint and maximal within each address space",
|
||||
});
|
||||
}),
|
||||
return;
|
||||
}
|
||||
bounds.set(start.space, {
|
||||
start: previous?.start ?? start.offset,
|
||||
end: end.offset,
|
||||
});
|
||||
previousSpace = start.space;
|
||||
total += end.offset - start.offset + 1n;
|
||||
}
|
||||
const onlyBounds = bounds.size === 1 ? [...bounds.values()][0] : undefined;
|
||||
const span =
|
||||
bounds.size === 0
|
||||
? 0n
|
||||
: onlyBounds === undefined
|
||||
? null
|
||||
: onlyBounds.end - onlyBounds.start + 1n;
|
||||
const reportedSpan =
|
||||
body.span_bytes === null ? null : BigInt(body.span_bytes);
|
||||
if (
|
||||
total !== BigInt(body.total_bytes) ||
|
||||
span !== reportedSpan ||
|
||||
body.non_contiguous !== body.ranges.length > 1
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
message:
|
||||
"Function body counts, enclosing span or contiguity disagree with inclusive ranges",
|
||||
});
|
||||
});
|
||||
|
||||
export const functionBodySchema = z.discriminatedUnion("available", [
|
||||
unavailableAnalysisFactSchema,
|
||||
availableFunctionBodySchema,
|
||||
]);
|
||||
|
||||
/** Check the declared entry membership against complete, inclusive body ranges. */
|
||||
@@ -325,23 +326,25 @@ const swiftStringLiteralFields = {
|
||||
};
|
||||
|
||||
/** A Swift String literal decoded from the two words of its ABI value, not a data reference. */
|
||||
export const swiftSmallStringLiteralSchema = z.strictObject({
|
||||
...swiftStringLiteralFields,
|
||||
representation: z
|
||||
.literal("small")
|
||||
.describe("The UTF-8 bytes are inside the two words."),
|
||||
address: z.null().describe("A small literal has no storage of its own."),
|
||||
});
|
||||
export const swiftLargeStringLiteralSchema = z.strictObject({
|
||||
...swiftStringLiteralFields,
|
||||
representation: z
|
||||
.literal("large")
|
||||
.describe(
|
||||
"An immortal literal whose words hold its count and a pointer to its bytes.",
|
||||
),
|
||||
address: z.string().describe("Address of a large literal's bytes."),
|
||||
});
|
||||
export const swiftStringLiteralSchema = z.discriminatedUnion("representation", [
|
||||
z.strictObject({
|
||||
...swiftStringLiteralFields,
|
||||
representation: z
|
||||
.literal("small")
|
||||
.describe("The UTF-8 bytes are inside the two words."),
|
||||
address: z.null().describe("A small literal has no storage of its own."),
|
||||
}),
|
||||
z.strictObject({
|
||||
...swiftStringLiteralFields,
|
||||
representation: z
|
||||
.literal("large")
|
||||
.describe(
|
||||
"An immortal literal whose words hold its count and a pointer to its bytes.",
|
||||
),
|
||||
address: z.string().describe("Address of a large literal's bytes."),
|
||||
}),
|
||||
swiftSmallStringLiteralSchema,
|
||||
swiftLargeStringLiteralSchema,
|
||||
]);
|
||||
|
||||
const swiftStringLiteralOmissionFields = {
|
||||
@@ -357,19 +360,21 @@ const swiftStringLiteralOmissionFields = {
|
||||
};
|
||||
|
||||
/** A Swift literal candidate omitted because decoding capacity or control flow left it unknown. */
|
||||
export const swiftSmallStringLiteralOmissionSchema = z.strictObject({
|
||||
...swiftStringLiteralOmissionFields,
|
||||
representation: z.literal("small"),
|
||||
address: z.null(),
|
||||
});
|
||||
export const swiftLargeStringLiteralOmissionSchema = z.strictObject({
|
||||
...swiftStringLiteralOmissionFields,
|
||||
representation: z.literal("large"),
|
||||
address: z.string().describe("Address of the literal's bytes."),
|
||||
});
|
||||
export const swiftStringLiteralOmissionSchema = z.discriminatedUnion(
|
||||
"representation",
|
||||
[
|
||||
z.strictObject({
|
||||
...swiftStringLiteralOmissionFields,
|
||||
representation: z.literal("small"),
|
||||
address: z.null(),
|
||||
}),
|
||||
z.strictObject({
|
||||
...swiftStringLiteralOmissionFields,
|
||||
representation: z.literal("large"),
|
||||
address: z.string().describe("Address of the literal's bytes."),
|
||||
}),
|
||||
swiftSmallStringLiteralOmissionSchema,
|
||||
swiftLargeStringLiteralOmissionSchema,
|
||||
],
|
||||
);
|
||||
|
||||
|
||||
@@ -416,14 +416,11 @@ const stdioMode = (
|
||||
method: JavaScriptSemanticChildProcessSpawn["method"],
|
||||
context: ChildSpawnContext,
|
||||
): string => {
|
||||
// Async exec/execFile do not forward options.stdio to their underlying spawn.
|
||||
if (method === "exec" || method === "execFile") return "default";
|
||||
if (options === undefined) return "default";
|
||||
const expression = unwrapJavaScriptExpression(options).node;
|
||||
if (
|
||||
context.isNullish(expression) ||
|
||||
((method === "exec" || method === "execFile") &&
|
||||
context.isCallback(expression))
|
||||
)
|
||||
return "default";
|
||||
if (context.isNullish(expression)) return "default";
|
||||
const lookup = staticObjectProperty(expression, "stdio", context.isNullish);
|
||||
if (lookup.status === "missing")
|
||||
return unsetStdioMode(expression, method, context);
|
||||
|
||||
@@ -13,9 +13,7 @@ import {
|
||||
} from "./javascriptSemanticTraversal.js";
|
||||
|
||||
it.each([
|
||||
["function f(){read(cb);function cb(){}}", ["callback"]],
|
||||
["function f(){const cb=()=>{};const alias=cb;read(alias)}", ["callback"]],
|
||||
["function f(cb){cb=()=>{};arguments[0]={};read(cb)}", ["unknown"]],
|
||||
["function f(a){a=[];arguments[0]={};read(a)}", ["unknown"]],
|
||||
["declare let erased:any;function f(){const a=[];read(a)}", ["unknown"]],
|
||||
["namespace X{};function f(){const a=[];read(a)}", ["unknown"]],
|
||||
["var a=[];(()=>{}).constructor('a={}')();read(a);", ["unknown"]],
|
||||
@@ -60,11 +58,9 @@ it.each([
|
||||
shapes.push(
|
||||
resolver.isArray(arg)
|
||||
? "array"
|
||||
: resolver.isCallback(arg)
|
||||
? "callback"
|
||||
: resolver.isNullish(arg)
|
||||
? "nullish"
|
||||
: "unknown",
|
||||
: resolver.isNullish(arg)
|
||||
? "nullish"
|
||||
: "unknown",
|
||||
);
|
||||
},
|
||||
});
|
||||
@@ -72,10 +68,10 @@ it.each([
|
||||
},
|
||||
);
|
||||
|
||||
it("bounds long callback aliases without invoking the shared callable resolver", () => {
|
||||
const source = `export function run(){const cb=()=>{};
|
||||
${Array.from({ length: 4096 }, (_, i) => `const a${i}=${i === 0 ? "cb" : `a${i - 1}`};`).join("\n")}
|
||||
read(a4095);read(a10);read(cb);}`;
|
||||
it("bounds long array aliases without invoking the shared callable resolver", () => {
|
||||
const source = `export function run(){const argv=[];
|
||||
${Array.from({ length: 4096 }, (_, i) => `const a${i}=${i === 0 ? "argv" : `a${i - 1}`};`).join("\n")}
|
||||
read(a4095);read(a10);read(argv);}`;
|
||||
const file = parseJavaScriptSource(source, "deep.mjs");
|
||||
if (file === null) throw new Error("Expected valid deep source");
|
||||
const state = completeSemanticSteps(
|
||||
@@ -89,7 +85,7 @@ it("bounds long callback aliases without invoking the shared callable resolver",
|
||||
t.isCallExpression(node) &&
|
||||
t.isIdentifier(node.callee, { name: "read" })
|
||||
)
|
||||
results.push(resolver.isCallback(node.arguments[0]!));
|
||||
results.push(resolver.isArray(node.arguments[0]!));
|
||||
},
|
||||
});
|
||||
expect(results).toEqual([false, true, true]);
|
||||
@@ -97,55 +93,30 @@ it("bounds long callback aliases without invoking the shared callable resolver",
|
||||
});
|
||||
|
||||
it.each([
|
||||
["fixture.cjs", "const argv=[];const done=()=>{};", ["inherit", "default"]],
|
||||
["fixture.cts", "const argv=[];const done=()=>{};", ["inherit", "default"]],
|
||||
["fixture.cjs", "const argv=[];", ["inherit"]],
|
||||
["fixture.cts", "const argv=[];", ["inherit"]],
|
||||
["fixture.cjs", "var argv=[];{function argv(){}}", ["dynamic"]],
|
||||
["fixture.mjs", "var argv=[];{function argv(){}}", ["inherit"]],
|
||||
["fixture.js", "const argv=[];", ["dynamic"]],
|
||||
["fixture.cjs", "let argv=[];argv=input;", ["dynamic"]],
|
||||
["fixture.cts", "const argv=[];eval(input);", ["inherit"]],
|
||||
["fixture.cts", "namespace N{};const argv=[];", ["dynamic"]],
|
||||
[
|
||||
"fixture.cjs",
|
||||
"var argv=[];{function argv(){}}const done=()=>{};",
|
||||
["dynamic", "default"],
|
||||
],
|
||||
[
|
||||
"fixture.mjs",
|
||||
"var argv=[];{function argv(){}}const done=()=>{};",
|
||||
["inherit", "default"],
|
||||
],
|
||||
["fixture.js", "const argv=[];const done=()=>{};", ["dynamic", "dynamic"]],
|
||||
[
|
||||
"fixture.cjs",
|
||||
"let argv=[];argv=input;const done=()=>{};",
|
||||
["dynamic", "default"],
|
||||
],
|
||||
[
|
||||
"fixture.cts",
|
||||
"const argv=[];const done=()=>{};eval(input);",
|
||||
["inherit", "default"],
|
||||
],
|
||||
[
|
||||
"fixture.cts",
|
||||
"namespace N{};const argv=[];const done=()=>{};",
|
||||
["dynamic", "dynamic"],
|
||||
],
|
||||
[
|
||||
"fixture.cjs",
|
||||
"var module=[];const argv=module;const done=()=>{};const args=arguments;args[2]={};",
|
||||
["dynamic", "default"],
|
||||
],
|
||||
[
|
||||
"fixture.cts",
|
||||
"const argv=[];function exports(){};const args=arguments;args[0]={};const done=exports;",
|
||||
["inherit", "dynamic"],
|
||||
"var module=[];const argv=module;const args=arguments;args[2]={};",
|
||||
["dynamic"],
|
||||
],
|
||||
] as const)(
|
||||
"distinguishes wrapper-local child identities and uncertain bindings: %s (%s)",
|
||||
(path, declarations, expected) => {
|
||||
const loader = path.endsWith(".mjs")
|
||||
? 'import {spawn,exec} from "node:child_process";'
|
||||
: 'const {spawn,exec}=require("node:child_process");';
|
||||
? 'import {spawn} from "node:child_process";'
|
||||
: 'const {spawn}=require("node:child_process");';
|
||||
const file = parseJavaScriptSource(
|
||||
`
|
||||
${loader}
|
||||
${declarations}
|
||||
spawn("argv",argv,{stdio:"inherit"}); exec("callback",done);
|
||||
spawn("argv",argv,{stdio:"inherit"});
|
||||
`,
|
||||
path,
|
||||
);
|
||||
|
||||
@@ -23,7 +23,7 @@ import {
|
||||
import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js";
|
||||
import { javaScriptRuntimeContext } from "./javascriptSourceParser.js";
|
||||
|
||||
type Shape = "array" | "callback" | "unknown";
|
||||
type Shape = "array" | "unknown";
|
||||
|
||||
/** Bounded argument-shape queries shared by call-signature owners. */
|
||||
export type JavaScriptExpressionShapeResolver = ReturnType<
|
||||
@@ -41,15 +41,6 @@ const commonJsWrapperParameterNames = new Set([
|
||||
export const createSemanticExpressionShapeResolver = (
|
||||
state: JavaScriptSemanticAnalysisState,
|
||||
) => {
|
||||
const hoistedFunction = (node: t.Node): boolean => {
|
||||
const parent = state.parentsByNode.get(node);
|
||||
return (
|
||||
t.isFunctionDeclaration(node) &&
|
||||
(t.isProgram(parent) ||
|
||||
(t.isBlockStatement(parent) &&
|
||||
t.isFunction(state.parentsByNode.get(parent))))
|
||||
);
|
||||
};
|
||||
// Most modules have no named child arguments. Collect source-wide blockers
|
||||
// only when a query actually needs a named binding or ambient undefined.
|
||||
let blocked: boolean | undefined;
|
||||
@@ -80,8 +71,6 @@ export const createSemanticExpressionShapeResolver = (
|
||||
if (seen.has(node)) return "unknown";
|
||||
seen.add(node);
|
||||
if (t.isArrayExpression(node)) return "array";
|
||||
if (t.isFunction(node) && !("declare" in node && node.declare))
|
||||
return "callback";
|
||||
if (!t.isIdentifier(node)) return "unknown";
|
||||
const binding = resolveSemanticBindingState(state, node, node.name);
|
||||
if (binding === undefined || binding.referenceInitializers.length > 0)
|
||||
@@ -106,11 +95,7 @@ export const createSemanticExpressionShapeResolver = (
|
||||
if (initializer === undefined || initializer.projection.length > 0)
|
||||
return "unknown";
|
||||
const read = semanticBindingAtRead(binding, node, state.parentsByNode);
|
||||
// Hoisted function declarations are the only accepted unordered source.
|
||||
if (
|
||||
!hoistedFunction(initializer.node) &&
|
||||
(read.status !== "ordered" || !read.hasDefiniteInitializer)
|
||||
)
|
||||
if (read.status !== "ordered" || !read.hasDefiniteInitializer)
|
||||
return "unknown";
|
||||
node = initializer.node;
|
||||
depth++;
|
||||
@@ -118,7 +103,6 @@ export const createSemanticExpressionShapeResolver = (
|
||||
};
|
||||
return {
|
||||
isArray: (node: t.Node) => shape(node) === "array",
|
||||
isCallback: (node: t.Node) => shape(node) === "callback",
|
||||
isNullish: (input: t.Node) => {
|
||||
const { node } = unwrapJavaScriptExpression(input);
|
||||
return (
|
||||
|
||||
@@ -3,6 +3,9 @@ import { addressCoordinate } from "./addressCoordinate.js";
|
||||
import { AnalysisInputError } from "../analysisErrorCore.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
|
||||
/** Maximum matches delivered by one byte-search page. */
|
||||
export const MAX_BYTE_SEARCH_MATCHES = 1000;
|
||||
|
||||
/** Exact grammar belongs to the object refinement, not a repeating advertised regex. */
|
||||
export const bytePatternFields = {
|
||||
pattern: z
|
||||
@@ -152,7 +155,7 @@ export const byteSearchResultSchema = z
|
||||
.nullable(),
|
||||
}),
|
||||
)
|
||||
.max(1000),
|
||||
.max(MAX_BYTE_SEARCH_MATCHES),
|
||||
searched: z.strictObject({
|
||||
blocks: z.array(coverageBlock),
|
||||
bytes: z.int().nonnegative().max(Number.MAX_SAFE_INTEGER),
|
||||
|
||||
@@ -87,7 +87,7 @@ const inferredJumpTableSchema = z
|
||||
})
|
||||
.strict();
|
||||
|
||||
const availableNativeApiBoundarySchema = z
|
||||
export const availableNativeApiBoundarySchema = z
|
||||
.object({
|
||||
available: z.literal(true),
|
||||
provenance: z.string().min(1),
|
||||
@@ -115,7 +115,7 @@ const availableNativeApiBoundarySchema = z
|
||||
})
|
||||
.strict();
|
||||
|
||||
const unavailableNativeApiBoundarySchema = z
|
||||
export const unavailableNativeApiBoundarySchema = z
|
||||
.object({
|
||||
available: z.literal(false),
|
||||
reason: z.string().min(1),
|
||||
|
||||
Loaded 100 of 164 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user