diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md index 5c1a6bf9ea..12a47c3896 100644 --- a/.agents/skills/reverse-engineer-anything/SKILL.md +++ b/.agents/skills/reverse-engineer-anything/SKILL.md @@ -354,7 +354,13 @@ Start with the default result and use its inline Evidence and graph context. For a JavaScript application or ELF layout whose size is unknown or large, call `analyze_javascript_application` or `inspect_binary_layout` with `"detail": "summary"`, then inspect the retained `parent_evidence_id` with -`inspect_analysis_view`. +`inspect_analysis_view`. Over MCP, `analyze_function` is summary-first without +a flag: its default reply is the summary, and the complete dossier stays +retained under `parent_evidence_id` for facet paging. Pass that ID as a +`retained-evidence` reference to `compare_functions`, or inside +`match_native_functions` `left_observations`/`right_observations`; request +`"detail": "complete"` only when the dossier must travel inline, as +`build_call_path` requires. Page module identities and select the returned `node_id` before requesting an item; a single file path can represent several graph roles. Do not repeat an identical analysis or rejected call without a changed input or resolved blocker. diff --git a/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md b/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md index fe542fa90d..a59d2b2acd 100644 --- a/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md +++ b/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md @@ -48,7 +48,9 @@ or absence: - `compare_artifacts` compares inventory pages by occurrence path, content, metadata, and graph relations. - `compare_functions` compares explicit function Evidence, not fuzzy - whole-binary matches. + whole-binary matches. It rejects an `analyze_function` summary reply; pass + the summary's `parent_evidence_id` as a `retained-evidence` reference or + the complete dossier. - `compare_bundles` compares canonical bundle membership and unknown history. - `find_changed_behavior` combines existing comparisons; static differences remain candidates, not causal proof. diff --git a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md index ba4c391e45..fdf876fa20 100644 --- a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md +++ b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md @@ -4,8 +4,10 @@ For release comparisons, export `list_procedures` Evidence from each target befo closing it. Close clears session records: import both bundles after collecting the two targets, or supply portable inline Evidence. `match_native_functions` pairs unique exact names and, when supplied, complete observed body bytes. Reuse optional same-context `list_names`, -`procedure_info`, `read_bytes` and existing `analyze_function` Evidence; avoid -decompiling every function merely to match it. Its summary retains a parent for +`procedure_info`, `read_bytes` and existing `analyze_function` dossiers as +`left_observations`/`right_observations`; an MCP `analyze_function` summary +reply contributes its `parent_evidence_id` as a `retained-evidence` reference, +not the summary itself. Avoid decompiling every function merely to match it. Its summary retains a parent for `inspect_analysis_view` `function-matches` collections `matches` and `functions`. Select columns, batch IDs and page individual facts; aliases remain unresolved. Unpaired observations do not prove additions/removals. Existing matched dossier @@ -231,6 +233,9 @@ unknown rather than false. For a completed `analyze_function` dossier, use `inspect_analysis_view` with `view: {kind: "native", facet: "value_flow_operations", offset: 0, limit: 32}` and the exact retained Evidence ID to read selected facts without reanalysis. +The default MCP reply is already a summary of that dossier: its +`normalized_result.parent_evidence_id` is the retained ID to page, and its +null counts mark facets the provider did not record or declared unavailable. Native facets also include procedure, pseudocode, assembly, basic blocks, comments, callers/callees, references, unresolved calls, referenced strings and names, the native API record, and value-flow summaries. Pseudocode offsets and limits count UTF-16 diff --git a/docs/cli.md b/docs/cli.md index 1e1b58ba86..53ee0bd3ed 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -339,6 +339,16 @@ rea trace /absolute/path/to/program "search" --provider ghidra --json a function dossier; `decompile` returns pseudocode. For assembly instructions alone, use `rea instructions`. A macOS `.app` bundle can be supplied directly. +A complete dossier can reach several megabytes of value-flow and references. +`rea function ... --detail summary --evidence_output /absolute/path/dossier.json` returns the +compact summary and saves the complete dossier as an Evidence bundle, never +overwriting an existing file. Page that bundle without reanalysis by naming it +as an `inspect-analysis-view` source, selecting the summary's +`normalized_result.parent_evidence_id` when the bundle holds several records: +`{"source": {"kind": "evidence-file", "path": "/absolute/path/dossier.json", "evidence_id": "ev_…"}, "view": {"kind": "native", "facet": "pseudocode", "offset": 0, "limit": 4096}}`. +MCP `analyze_function` defaults to the same summary and retains the complete +dossier in the session for `inspect_analysis_view`. + Run `rea --help` or a command's `--help` for arguments and output options. The [generated catalog](mcp-contracts.md#generated-catalog) lists all CLI commands and MCP tools. Provider-specific target support is described in the diff --git a/docs/ida-provider.md b/docs/ida-provider.md index 04221abe8a..bdc036930b 100644 --- a/docs/ida-provider.md +++ b/docs/ida-provider.md @@ -140,7 +140,7 @@ Add the registration path to the **REA** MCP entry's environment, keeping the up } ``` -Use a package version that actually includes this adapter; repository main can be ahead of the npm release. After reconnecting, call: +Use a package version that actually includes this adapter; repository main can be ahead of the npm release. After reconnecting, call the sequence below; the default `analyze_function` reply is a summary that retains the dossier, so add `"detail":"complete"` when the dossier itself must come back inline: ```json {"name":"open_binary","arguments":{"path":"/absolute/path/to/program","provider_id":"ida"}} diff --git a/docs/kuna-provider.md b/docs/kuna-provider.md index 6fee2d9ad8..35098cbe1e 100644 --- a/docs/kuna-provider.md +++ b/docs/kuna-provider.md @@ -70,7 +70,7 @@ The assembly label index is scanned once per export reader. Cancelling a request Regex procedure and string search uses ECMAScript Unicode matching in a cancellable worker with a matching deadline and a worker old-generation heap. Literal matching remains the default. Cancellation or a matching deadline leaves the session available. The deadline is `REA_REGEX_MATCHING_DEADLINE_MS`, a positive decimal integer from 1 to 2147483647 ms. Unset uses 5000 ms. A larger value is refused because Node schedules a delay past that signed 32-bit bound as 1 ms. Invalid values are configuration errors and do not fall back to the default. The setting is read from the environment that starts REA, so a change needs a restart. A deadline failure names the effective milliseconds, whether they came from the default or the setting, the default, and the maximum. The worker old-generation heap does not raise it. The worker heap request is `REA_REGEX_WORKER_HEAP_REQUEST_MIB`, a decimal integer from 7 to 16384 MiB. Unset uses 128 MiB. A request below 7 MiB aborts the worker during startup, and a larger value is refused. Invalid values are configuration errors and do not fall back to 128. The setting is read from the environment that starts REA, so a change needs a restart. A heap failure names the effective request, whether it came from the default or the setting, the default, the floor, and the maximum. A positive Node.js `--max-old-space-size` on the process that starts REA, including `NODE_OPTIONS`, overrides that request and sets the effective heap. `--max-old-space-size-percentage` takes precedence over that flag in either order. REA does not turn the percentage into a MiB, so the effective heap stays unestablished instead of the worker heap request or the absolute flag. The reported `NODE_OPTIONS` value is the process initial environment when that block can be read. A later edit to the environment is not the value Node applied, and an unreadable initial value leaves the effective heap unestablished. The launch Node options are the initial command line on Linux when that buffer still has a program entry, not a later edit to `process.execArgv`. A preload can clear that array after V8 has applied the flags. Setting `process.title` or `--title` overwrites the command line, so a buffer without a program entry leaves the effective heap unestablished. `--env-file` or `--env-file-if-exists` can supply `NODE_OPTIONS` when the initial environment has none. That file is not part of the initial environment block, so the effective heap stays unestablished instead of the worker heap request. An initial `NODE_OPTIONS` is still used, because Node does not let the file replace it. A positive `--max-heap-size`, including the underscore form and launch `NODE_OPTIONS` on Node versions that allow it, sets the total heap rather than the old generation and can be lower than the worker heap request. The effective old generation stays unestablished while that value is in effect. A zero or empty value clears it. `--experimental-config-file` or `--experimental-default-config-file` can set `nodeOptions`, including a heap percentage, from a file that is not read back. The effective heap stays unestablished when either flag is present, including when an absolute `--max-old-space-size` is also on the command line or in the recovered `NODE_OPTIONS`. Node accepts the config path as `=` or the next argument. -Single-function subprocess output and each cached source facet share a 16 MiB buffered-text capacity. Cached spans are checked before allocation; larger text remains in the local export and the error identifies the file and range. `analyze_function` initially returns the complete dossier, which can still be large; later retained views do not remove that first transfer. Prefer the individual text tools for a focused question. +Single-function subprocess output and each cached source facet share a 16 MiB buffered-text capacity. Cached spans are checked before allocation; larger text remains in the local export and the error identifies the file and range. `analyze_function` with `detail: "complete"` (the CLI default) returns the complete dossier inline, which can still be large; later retained views do not remove that first transfer. The MCP default retains the dossier and returns its summary, so page the facets you need through `inspect_analysis_view`, or prefer the individual text tools for a focused question. ## Strings diff --git a/docs/mcp-contracts.md b/docs/mcp-contracts.md index b0246cf697..c37f8153c7 100644 --- a/docs/mcp-contracts.md +++ b/docs/mcp-contracts.md @@ -21,11 +21,24 @@ as an upstream tool's original report, and is `null` when there is none. The sam record is retained in the session bundle. `analysis_profile` is always present: a concrete profile object or `null`; either value participates in semantic identity. Records omitting this field are rejected. Pass the returned Evidence -directly to a compatible comparison tool: `analyze_function` Evidence can be passed -directly to `compare_functions`, and `inspect_artifact` Evidence to -`compare_artifacts`. Use `get_evidence_bundle` when the task needs broader +directly to a compatible comparison tool: complete `analyze_function` Evidence +can be passed directly to `compare_functions`, and `inspect_artifact` Evidence +to `compare_artifacts`. Use `get_evidence_bundle` when the task needs broader retained session history or an explicit bundle for transfer. +`analyze_function` does not deliver its complete record by default. Without +`detail`, it retains the complete dossier in the session and returns +`inspect_analysis_view` summary Evidence whose +`normalized_result.parent_evidence_id` names that dossier; see +[retained analysis Evidence](#retained-application-evidence-inputs). Page the +dossier with `inspect_analysis_view`. Tools that consume function dossiers +reject the summary record. Give `compare_functions` the retained dossier as +`{"kind": "retained-evidence", "evidence_id": ""}`; give +`match_native_functions` the same reference inside `left_observations` or +`right_observations`, since its `left` and `right` take `list_procedures` +Evidence. Request `detail: "complete"` for `build_call_path`, which accepts +only inline complete dossiers. + Saved records from the pre-6.6 identity projection that omitted `relationship_coverage: null` from its digest remain incompatible, even when the serialized field is present. REA recognizes that exact previous commitment @@ -346,11 +359,13 @@ after target content was read. ## Retained application Evidence inputs `analyze_javascript_application`, `inspect_binary_layout`, `inspect_artifact`, -`inspect_wasm_artifact`, `inspect_macho`, and `decode_interface_builder` -accept `detail`: -`complete` (default) returns the complete analysis Evidence; `summary` records +`inspect_wasm_artifact`, `inspect_macho`, `decode_interface_builder`, and +`analyze_function` accept `detail`: +`complete` returns the complete analysis Evidence; `summary` records that complete Evidence in the session and returns the derived -`inspect_analysis_view` summary Evidence instead. Its +`inspect_analysis_view` summary Evidence instead. `complete` is the default +except for `analyze_function`, whose dossier can reach several megabytes of +value flow and references, so it defaults to `summary`. Its `normalized_result.parent_evidence_id` and `evidence_links` name the retained analysis, so later views and application workflows read it without repeating analysis or moving the complete record through the response. A server without diff --git a/docs/native-function-matching.md b/docs/native-function-matching.md index 8d98fc59c2..5e38b48464 100644 --- a/docs/native-function-matching.md +++ b/docs/native-function-matching.md @@ -34,8 +34,10 @@ index. Re-select compatible observations; authentication or contradictory produc facts retain their separate integrity failures. For unmatched candidates, optionally supply `left_observations` and -`right_observations`. `procedure_info` or existing `analyze_function` Evidence -gives exact inclusive body ranges; `read_bytes` supplies memory bytes. Observations +`right_observations`. `procedure_info` or existing `analyze_function` dossiers +give exact inclusive body ranges; `read_bytes` supplies memory bytes. An MCP +`analyze_function` summary reply is not a dossier: pass its +`parent_evidence_id` as a `retained-evidence` reference instead. Observations must belong to the side's exact target/provider/profile and document context. Adjacent, consistent overlapping and partially returned reads compose by their observed extent; conflicting intervals remain gaps. A byte match requires every @@ -86,9 +88,9 @@ inventory Evidence and the owning inventory’s `source_index`, preserving same- One validation, lazy member index and bounded recent filter are retained per immutable parent; no live target state is cached. -When both functions already have unambiguous `analyze_function` dossiers, a matched -relation includes `compare_functions_input` with directly usable retained -references. Otherwise analyze just the selected canonical addresses in their +When both functions already have unambiguous `analyze_function` dossiers in the +observations, a matched relation includes `compare_functions_input` with +directly usable retained references. Otherwise analyze just the selected canonical addresses in their respective targets, then compare those dossiers. Matching never claims behavioral equivalence. Optional informative CFG/call/string fingerprints from supplied dossiers connect unresolved candidate groups only. They require observed CFG, diff --git a/docs/verification/native-function-summary-2026-10-23.md b/docs/verification/native-function-summary-2026-10-23.md new file mode 100644 index 0000000000..43ebfcc1b5 --- /dev/null +++ b/docs/verification/native-function-summary-2026-10-23.md @@ -0,0 +1,149 @@ +# Native function summary verification — 2026-10-23 + +## Behavior and source + +MCP `analyze_function` defaults to a compact derived summary. The complete +function Evidence stays in the session and can be selected by the returned +`parent_evidence_id` through `inspect_analysis_view`. `detail: "complete"` +keeps the historical full reply. Delivery detail stays outside provider inputs +and parent Evidence identity. This MCP default is a breaking change. + +CLI `rea function` still defaults to complete output. Opting into summary +requires an exclusive `--evidence_output` bundle destination. The authenticated +bundle can be paged by an absolute `evidence-file` source and explicit record +ID after the provider closes. Plain records also honor an explicit ID. + +The implementation reuses the existing JSON authentication/headroom owner and +one bundle-authentication function for import and file views. It reads one file +once per request. Ambiguous-bundle guidance retains complete IDs and bounds +JSON-encoded reference previews to 2 KiB, omitting oversized operation labels. +Native summaries inherit function locations and relationship coverage; omitted +or unavailable facts stay unknown rather than becoming zero. + +Measured source: + +- Before: `fb6c88d518ab9b84ed396cf82dcb1c28a608c501`. +- After: `4e3d930ada93014efa43610265b58307de49a687`. +- Harness SHA-256: `d67b23db21024361b34f796add1ff91bf3fbcda1991ea9e369ef8bcf64c71ec4`. +- Linux x86-64; pinned Node 24.22.1 / npm 11.16.0; bring-your-own Ghidra 12.1.4 + and JDK 21.0.12.1+1. No target execution. +- Target SHA-256: + `9c49d2eef9c2083f6f66f38de4ed0ee1bac17ca5b73dc8f3a90d9ba62a8d1d5d`. + Its 3,000-branch C source is the timeout verifier fixture, SHA-256 + `f99074876faae8e18e45265dab007b6fcc396fe909d57efab72adc8925a960c7`. + Known entries `0x1010f9` and `0x128ed6` apply only to this exact artifact. + +## Completed public MCP comparison + +One fresh production stdio MCP client discovers all 221 tools, opens the target, +then obtains two answers: the small function's identity, availability/counts and +complete pseudocode; and the large function's identity, availability, assembly/ +block counts and first 20 assembly rows. It repeats both reads in the same +session, performs one matched complete-small compatibility read and closes. + +Before, each default function reply carries the full dossier. After, each +answer uses the default summary and one retained-Evidence page. Both lanes use +a one-second native decompile limit, an explicit 600-second SDK tool deadline +and matching 128 MiB server/client receive budgets. Selected facts match +between implementations and rounds. The large function has 45,007 assembly +rows and 9,001 blocks with pseudocode unavailable; the small pseudocode is +64 bytes, SHA-256 +`c947ff22cab1ffef9b962b12d82e12d0e5f5e2a6c5634a2ed5ea3c353e1d5807`. +Each run retained the same observed JVM start identity through its tasks and +completed cleanup with an unchanged target digest. + +| Whole benchmark | Before | After | +| --------------------------------- | ---------: | --------: | +| Estimated exchange tokens | 44,217,047 | 1,033,322 | +| Elapsed seconds | 237.715 | 128.600 | +| Waited process-family CPU seconds | 211.656 | 123.083 | +| Sampled family RSS MiB | 3951.7 | 2827.8 | +| Catalog/tool exchanges | 8 | 12 | + +Estimated context falls 97.66% for this completed task, including the larger catalog and +four extra page calls. The catalog alone grows from 982,457 to 985,662 estimated +tokens. The complete-small compatibility read remains included in both totals. + +| MCP phase elapsed seconds | Before | After | +| ---------------------------------------------------- | ------: | ------: | +| First small function, including deferred JVM startup | 80.3747 | 69.0745 | +| First large function, with the JVM already warm | 57.2394 | 19.8205 | +| Repeated small function | 0.0534 | 0.0601 | +| Repeated large function | 56.4148 | 19.7744 | + +The repeated small task adds a page call and takes about 6.7 ms longer in this +pair. Large partial dossiers remain live and retry native analysis; healthy +small-function snapshot reuse remains available. This change reduces agent +output and its serialization/transport work while still producing and retaining +the complete parent dossier. + +Tokens are `tokenx` 1.3.0 estimates over the actual catalog, arguments and whole +SDK results, including both text and structured projections. Protocol framing, +handshake, prompts, client rendering/replay and actual model usage are +unmeasured. The whole benchmark includes token estimation, hashing, resource +sampling and exchange-file writes. Summed measured MCP phases exclude the +subsequent counting/writing overhead: 217.327 → 127.538 seconds. +The whole-benchmark timings must not be presented as pure provider analysis time. + +Node heaps were 2 GiB, Ghidra's heap 1 GiB, affinity CPUs 0/1 and Java active +processor count two. The observed-family early-stop guard allowed 6 GiB RSS, +600 seconds CPU and 900 seconds elapsed; neither completed run hit a guard. +RSS was sampled every 50 ms through recursive `/proc` observations. Shared +pages can be double-counted and brief/unobserved child transitions can be +missed; this is not a kernel cgroup peak or hard family memory limit. Waited +CPU includes reaped descendants. Baseline ran first, OS caches were not flushed +and the host was shared with other work. This is one paired investigation, +with cold/result-cache distinctions above, rather than a universal speed claim. + +An earlier baseline attempt used a 240-second SDK deadline and began with the +large function. It timed out, then closed successfully; the whole attempt took +280.628 seconds and 219.562 seconds waited CPU, with no resource-guard stop. +Its failed receipt/exchanges remain separate. It is excluded from completed-task +comparisons. The final pair uses the same longer deadline and task order on +both implementations. + +## Public CLI and retained files + +The real default CLI command returned a complete small-function dossier with +the expected pseudocode (70.627 s elapsed, 70.034 s CPU). The first verification +harness assumed bare Evidence and failed to unwrap the successful public +`ok`/`data`/`meta` envelope. After correcting the harness, its completed receipt +and output were verified and reused without another provider analysis. + +The summary CLI command analyzed the large function and wrote a 29,141,012-byte +bundle. It returned an 8,499-byte summary envelope in 78.163 seconds, with +94.196 seconds CPU and 2,381.2 MiB sampled family RSS. Two fresh CLI processes, +with the Ghidra installation path unavailable, authenticated that bundle and +returned its first and final 20 assembly rows. Coverage reports 45,007 total +rows, correct continuation and exhaustion. The target digest remained unchanged. + +Those independent file-page commands took 17.218 / 18.175 seconds, 15.150 / +17.015 seconds CPU and about 666 / 669 MiB RSS. Each fresh command reads and +authenticates the complete bundle; this is not sublinear disk access or a +cross-process cache. Complete evidence remains available at the caller's file. + +## Validation and limits + +- Full `npm run check` passed after integration with main + `a062529bb2aa064be37e82df39c59c4fa798fe50` at source `b7c3f0cd79c4a02b135da56fff06de08dd456ff7`. + `docs:check` also passed. +- The latest focused contract/progress/snapshot/local-path run passed 139 tests; + the measured integration's selected run passed 136 and an earlier broader + CLI-selector/contract run passed 169. Overlaps are not summed. The earlier domain/Evidence run had 87 passes and one legacy + progress-fixture failure. The explicit-complete fixture and new summary's + increasing notifications subsequently passed in a separate eight-case MCP + run. +- Actual SDK/schema checks passed all 15 cases after regenerating ignored test + catalog metadata. The initial run had 14 passes and one stale-catalog + equality failure; its receipt is retained. +- The production CLI/MCP observations above use real Linux Ghidra. Hopper, + IDA, Kuna, macOS and Windows behavior is covered only to the extent of the + selected source/boundary tests, not new real-provider verification. +- Earlier source integration fixes and later main changes were not included in + every historical test/performance pin. The measured pair remains tied to the + explicit source commits above. + +Raw exchanges, protocol observations, control failures, scripts and resource +receipts are retained under `/tmp/rea-maintenance-20261023/5660/`. Provider +projects, generated binaries, build outputs and local planning files are not +committed. diff --git a/scripts/lib/ghidra-reference-snapshot-e2e.mjs b/scripts/lib/ghidra-reference-snapshot-e2e.mjs index 7a99099b63..7214a35917 100644 --- a/scripts/lib/ghidra-reference-snapshot-e2e.mjs +++ b/scripts/lib/ghidra-reference-snapshot-e2e.mjs @@ -142,7 +142,7 @@ export async function verifyLegacyGhidraReferenceSnapshot( const retained = requireMcpEvidenceResult( await client.callTool({ name: "analyze_function", - arguments: { procedure }, + arguments: { procedure, detail: "complete" }, }), "analyze_function", ); diff --git a/scripts/lib/native-value-e2e.mjs b/scripts/lib/native-value-e2e.mjs index 916533a87f..ab88bb8606 100644 --- a/scripts/lib/native-value-e2e.mjs +++ b/scripts/lib/native-value-e2e.mjs @@ -208,7 +208,10 @@ async function verifyNativeValueAdapters( assert.deepEqual(mcpEvidence.normalized_result, mcpGraph); assert.deepEqual(mcpGraph, cliGraph); const analyzed = await client.callTool( - { name: "analyze_function", arguments: { procedure } }, + { + name: "analyze_function", + arguments: { procedure, detail: "complete" }, + }, { timeout: 180000 }, ); assert.notEqual(analyzed.isError, true, JSON.stringify(analyzed)); @@ -350,7 +353,10 @@ async function verifyUnresolvedCallEvidence(client) { "Source fixture lacks the unresolved callback and its possible target", ); const analyzed = await client.callTool( - { name: "analyze_function", arguments: { procedure: indirect.address } }, + { + name: "analyze_function", + arguments: { procedure: indirect.address, detail: "complete" }, + }, { timeout: 180000 }, ); assert.notEqual(analyzed.isError, true, JSON.stringify(analyzed)); diff --git a/scripts/lib/real-ghidra-address-boundaries.mjs b/scripts/lib/real-ghidra-address-boundaries.mjs index a2211a1bad..e896b7d148 100644 --- a/scripts/lib/real-ghidra-address-boundaries.mjs +++ b/scripts/lib/real-ghidra-address-boundaries.mjs @@ -11,7 +11,10 @@ export async function verifyGhidraAddressBoundaries(client, entry, cli) { assert.notEqual(reply.isError, true, JSON.stringify(reply)); return reply.structuredContent.normalized_result; }; - const before = await query("analyze_function", { procedure: entry }); + const before = await query("analyze_function", { + procedure: entry, + detail: "complete", + }); // The segmented parser used to discard the high 32 bits and alias this entry. const overflow = (0x100000000n + BigInt(entry)).toString(16); const representations = [ @@ -52,7 +55,7 @@ export async function verifyGhidraAddressBoundaries(client, entry, cli) { } } assert.deepEqual( - await query("analyze_function", { procedure: entry }), + await query("analyze_function", { procedure: entry, detail: "complete" }), before, "Rejected oversized annotations changed the aliased function", ); diff --git a/scripts/lib/real-ghidra-boundaries.mjs b/scripts/lib/real-ghidra-boundaries.mjs index 54376c389c..d2c7607613 100644 --- a/scripts/lib/real-ghidra-boundaries.mjs +++ b/scripts/lib/real-ghidra-boundaries.mjs @@ -106,7 +106,10 @@ export async function verifyGhidraBoundaries( ); return JSON.parse(stdout).normalized_result; }; - const original = await call("analyze_function", { procedure }); + const original = await call("analyze_function", { + procedure, + detail: "complete", + }); const roots = (await readdir(env.TMPDIR)).filter((name) => name.startsWith("rea-ghidra-"), ); @@ -130,6 +133,7 @@ export async function verifyGhidraBoundaries( assert.ok(external, "Source fixture lacks an external function"); const externalDossier = await call("analyze_function", { procedure: external.address, + detail: "complete", }); assert.equal(externalDossier.pseudocode, ""); assert.deepEqual(externalDossier.assembly, []); @@ -227,6 +231,7 @@ export async function verifyGhidraBoundaries( assert.equal(interiorEdge.kind.data, true); const leafDossier = await call("analyze_function", { procedure: leaf.address, + detail: "complete", }); assert.ok( leafDossier.incoming_references.some( @@ -545,7 +550,7 @@ export async function verifyGhidraBoundaries( assert.equal(compileError.remediation.action, regexError.remediation.action); rejectedCalls++; assert.deepEqual( - await call("analyze_function", { procedure: address }), + await call("analyze_function", { procedure: address, detail: "complete" }), updated.dossier, "Regex stack exhaustion must preserve the live annotation database", ); @@ -600,8 +605,12 @@ export async function verifyGhidraBoundaries( address, ); assert.equal( - (await call("analyze_function", { procedure: changes.name })).procedure - .name, + ( + await call("analyze_function", { + procedure: changes.name, + detail: "complete", + }) + ).procedure.name, changes.name, ); assert.equal(await call("address_name", { address }), changes.name); @@ -735,7 +744,7 @@ export async function verifyGhidraBoundaries( const snapshotLifecycle = await verifyGhidraSnapshotLifecycle( client, target, - await call("analyze_function", { procedure: address }), + await call("analyze_function", { procedure: address, detail: "complete" }), cli, ); successfulCalls += snapshotLifecycle.successfulCalls; @@ -777,7 +786,8 @@ export async function verifyGhidraBoundaries( indirectProcedure.address, ); assert.deepEqual( - (await call("analyze_function", { procedure: address })).comments, + (await call("analyze_function", { procedure: address, detail: "complete" })) + .comments, original.comments, ); assert.equal( diff --git a/scripts/lib/real-ghidra-entry-aliases.mjs b/scripts/lib/real-ghidra-entry-aliases.mjs index d628c3172c..9fd3cefdfc 100644 --- a/scripts/lib/real-ghidra-entry-aliases.mjs +++ b/scripts/lib/real-ghidra-entry-aliases.mjs @@ -73,7 +73,8 @@ export async function verifyGhidraEntryAliases({ alias.address, ); assert.equal( - (await call("analyze_function", { procedure })).procedure.address, + (await call("analyze_function", { procedure, detail: "complete" })) + .procedure.address, alias.address, ); assert.equal( diff --git a/scripts/lib/real-ghidra-large-results.mjs b/scripts/lib/real-ghidra-large-results.mjs index 6890bd4a28..f721ff503d 100644 --- a/scripts/lib/real-ghidra-large-results.mjs +++ b/scripts/lib/real-ghidra-large-results.mjs @@ -110,8 +110,11 @@ export async function verifyGhidraLargeResults({ entry.address, "A delivery constraint must preserve the successfully committed edit", ); + // The default summary fits the receive budget; only the complete dossier + // exercises the oversized-delivery path with a retained reference. const dossierError = await reject("analyze_function", { procedure: "rea_frame_probe", + detail: "complete", }); const dossierId = retainedReference(dossierError, "analyze_function"); const after = await call("binary_session"); diff --git a/scripts/lib/real-ghidra-namespace-annotations.mjs b/scripts/lib/real-ghidra-namespace-annotations.mjs index 6966e77ad1..bd5c35940a 100644 --- a/scripts/lib/real-ghidra-namespace-annotations.mjs +++ b/scripts/lib/real-ghidra-namespace-annotations.mjs @@ -165,8 +165,12 @@ export async function verifyGhidraNamespaceAnnotations({ "alpha::cli_renamed", ); assert.equal( - (await call("analyze_function", { procedure: entries[0].address })) - .procedure.name, + ( + await call("analyze_function", { + procedure: entries[0].address, + detail: "complete", + }) + ).procedure.name, "alpha::qualified", "The independent CLI session must not alter the active MCP database", ); diff --git a/scripts/lib/real-ghidra-snapshot-lifecycle.mjs b/scripts/lib/real-ghidra-snapshot-lifecycle.mjs index e359f22ac8..7b4221bc92 100644 --- a/scripts/lib/real-ghidra-snapshot-lifecycle.mjs +++ b/scripts/lib/real-ghidra-snapshot-lifecycle.mjs @@ -68,7 +68,10 @@ export async function verifyGhidraSnapshotLifecycle( await assert.rejects(access(destination), { code: "ENOENT" }); assert.equal((await call("binary_session")).analysis_run.run_id, runId); assert.deepEqual( - await call("analyze_function", { procedure: entry }), + await call("analyze_function", { + procedure: entry, + detail: "complete", + }), edited, ); } @@ -80,7 +83,7 @@ export async function verifyGhidraSnapshotLifecycle( assert.deepEqual(await readFile(snapshotPath), snapshotBytes); assert.equal((await call("binary_session")).analysis_run.run_id, runId); assert.deepEqual( - await call("analyze_function", { procedure: entry }), + await call("analyze_function", { procedure: entry, detail: "complete" }), edited, ); @@ -109,7 +112,10 @@ export async function verifyGhidraSnapshotLifecycle( provider_id: "ghidra", snapshot_path: snapshotPath, }); - const restored = await call("analyze_function", { procedure: entry }); + const restored = await call("analyze_function", { + procedure: entry, + detail: "complete", + }); assert.deepEqual(restored.procedure, original.procedure); assert.deepEqual(restored.comments, original.comments); assert.notEqual((await call("binary_session")).analysis_run.run_id, runId); @@ -165,14 +171,15 @@ async function verifyConcurrentSnapshotClose({ await assert.rejects(access(destination), { code: "ENOENT" }); assert.equal((await call("binary_session")).open, true); assert.deepEqual( - await call("analyze_function", { procedure: entry }), + await call("analyze_function", { procedure: entry, detail: "complete" }), updated.dossier, ); // Discard the ephemeral edit and leave the caller with a pristine database. await call("close_binary"); await call("open_binary", { path: target.path, provider_id: "ghidra" }); assert.deepEqual( - (await call("analyze_function", { procedure: entry })).procedure, + (await call("analyze_function", { procedure: entry, detail: "complete" })) + .procedure, original.procedure, ); } diff --git a/scripts/lib/real-ghidra-target-admission.mjs b/scripts/lib/real-ghidra-target-admission.mjs index 3bbe8dbbf1..d2c2fb7688 100644 --- a/scripts/lib/real-ghidra-target-admission.mjs +++ b/scripts/lib/real-ghidra-target-admission.mjs @@ -88,7 +88,10 @@ export async function verifyGhidraTargetAdmission({ (await call("binary_session")).analysis_run.run_id, before.analysis_run.run_id, ); - const dossier = await call("analyze_function", { procedure: entry }); + const dossier = await call("analyze_function", { + procedure: entry, + detail: "complete", + }); const instruction = await call("inspect_native_instruction", { address: entry, }); @@ -114,7 +117,7 @@ export async function verifyGhidraTargetAdmission({ // Imported facts belong to the captured bytes even if the caller's path disappears. await rm(source); assert.deepEqual( - await call("analyze_function", { procedure: entry }), + await call("analyze_function", { procedure: entry, detail: "complete" }), dossier, ); assert.deepEqual( diff --git a/scripts/lib/real-hopper-boundaries.mjs b/scripts/lib/real-hopper-boundaries.mjs index 5f11bade79..837a81a97a 100644 --- a/scripts/lib/real-hopper-boundaries.mjs +++ b/scripts/lib/real-hopper-boundaries.mjs @@ -234,7 +234,12 @@ export async function verifyHopperBoundaryContracts( ), ); assert.equal( - (await call("analyze_function", { procedure: address })).procedure.name, + ( + await call("analyze_function", { + procedure: address, + detail: "complete", + }) + ).procedure.name, renamed, ); assert.equal( @@ -426,6 +431,7 @@ export async function verifyHopperLifecycleAndCli(client, options, targets) { assert.ok(entry, "primary fixture omitted its entry procedure"); const expectedFunction = await call("analyze_function", { procedure: entry.value, + detail: "complete", }); const firstDocument = await call("current_document"); const linuxApplicationIsolation = @@ -518,6 +524,7 @@ export async function verifyHopperLifecycleAndCli(client, options, targets) { assert.ok(delegate, "Objective-C fixture omitted its tail-call method"); const expected = await call("analyze_function", { procedure: delegate.address, + detail: "complete", }); await call("close_binary"); const analyzed = await runCli([ diff --git a/scripts/lib/real-hopper-cli-selectors.mjs b/scripts/lib/real-hopper-cli-selectors.mjs index 7bfa4d1270..182773d90e 100644 --- a/scripts/lib/real-hopper-cli-selectors.mjs +++ b/scripts/lib/real-hopper-cli-selectors.mjs @@ -12,7 +12,8 @@ export async function verifyHopperCliSelectors(call, runCli, dispatcher, path) { command: "function", option: "procedure", operation: "analyze_function", - parameters: { procedure: delegate.value }, + // MCP defaults to a summary; match the CLI's complete dossier. + parameters: { procedure: delegate.value, detail: "complete" }, }, { command: "instructions", diff --git a/scripts/lib/real-hopper-function-basics.mjs b/scripts/lib/real-hopper-function-basics.mjs index 3969c72251..04ad28f122 100644 --- a/scripts/lib/real-hopper-function-basics.mjs +++ b/scripts/lib/real-hopper-function-basics.mjs @@ -90,7 +90,10 @@ export async function verifyHopperTerminalInstructions(call) { throw new Error( `Native fixture procedure ${address} omitted its terminal call`, ); - const dossier = await call("analyze_function", { procedure: address }); + const dossier = await call("analyze_function", { + procedure: address, + detail: "complete", + }); const instructions = await call("read_function_instructions", { procedure: address, }); @@ -160,6 +163,7 @@ async function verifyHopperTerminalComment(call, address, terminal) { }); const annotated = await call("analyze_function", { procedure: address, + detail: "complete", }); if ( !annotated.comments.some( diff --git a/scripts/lib/real-hopper-navigation.mjs b/scripts/lib/real-hopper-navigation.mjs index 95353f795b..15f9fddb7b 100644 --- a/scripts/lib/real-hopper-navigation.mjs +++ b/scripts/lib/real-hopper-navigation.mjs @@ -72,7 +72,10 @@ async function verifyNavigation( assert.ok(local.description.includes(local.stack_displacement)); assert.doesNotMatch(local.description, /object at 0x/u); } - const dossier = await call("analyze_function", { procedure: address }); + const dossier = await call("analyze_function", { + procedure: address, + detail: "complete", + }); assert.deepEqual(dossier.procedure.locals, info.locals); assert.equal(dossier.procedure.classification, null); assert.equal(dossier.procedure.body.available, false); diff --git a/scripts/lib/real-hopper-search.mjs b/scripts/lib/real-hopper-search.mjs index 0efd2bfb7c..6decda4ad5 100644 --- a/scripts/lib/real-hopper-search.mjs +++ b/scripts/lib/real-hopper-search.mjs @@ -126,6 +126,7 @@ async function verifySearchModes(call, address) { assert.ok(literalProcedure); const dossier = await call("analyze_function", { procedure: literalProcedure.address, + detail: "complete", }); const referenced = dossier.referenced_strings.find( (item) => item.address === longItem.address, diff --git a/scripts/lib/real-hopper-semantic.mjs b/scripts/lib/real-hopper-semantic.mjs index 7739db6bef..1f520a690f 100644 --- a/scripts/lib/real-hopper-semantic.mjs +++ b/scripts/lib/real-hopper-semantic.mjs @@ -331,7 +331,7 @@ export const analyzeFixtureProcedure = async ( const reply = await client.callTool( { name: "analyze_function", - arguments: { procedure: procedure.address }, + arguments: { procedure: procedure.address, detail: "complete" }, }, options, ); diff --git a/scripts/verify-real-ghidra-com.mjs b/scripts/verify-real-ghidra-com.mjs index 8a82331d8e..c63b3b0222 100644 --- a/scripts/verify-real-ghidra-com.mjs +++ b/scripts/verify-real-ghidra-com.mjs @@ -141,7 +141,10 @@ try { assert.ok(procedures.some((p) => p.address === fixture.entry)); assert.ok(procedures.some((p) => p.address === fixture.near)); const dossier = functionDossierSchema.parse( - await call("analyze_function", { procedure: fixture.entry }), + await call("analyze_function", { + procedure: fixture.entry, + detail: "complete", + }), ); assert.ok(dossier.pseudocode.trim().length > 0); assert.equal(dossier.procedure.body.available, true); @@ -203,8 +206,12 @@ try { ); assert.ok((await call("list_names")).some((p) => p.value === changes.name)); assert.equal( - (await call("analyze_function", { procedure: fixture.entry })).procedure - .name, + ( + await call("analyze_function", { + procedure: fixture.entry, + detail: "complete", + }) + ).procedure.name, changes.name, ); const rejected = await client.callTool({ diff --git a/scripts/verify-real-ghidra-dos.mjs b/scripts/verify-real-ghidra-dos.mjs index 7fd094af05..ec49ca8499 100644 --- a/scripts/verify-real-ghidra-dos.mjs +++ b/scripts/verify-real-ghidra-dos.mjs @@ -213,7 +213,10 @@ try { const resolved = await call("procedure_address", { procedure }); assert.equal(resolved, procedure, "Flat address round-trip drifted"); const info = await call("procedure_info", { procedure }); - const dossierResult = await evidenceCall("analyze_function", { procedure }); + const dossierResult = await evidenceCall("analyze_function", { + procedure, + detail: "complete", + }); const dossier = functionDossierSchema.parse( dossierResult.normalized_result, ); diff --git a/scripts/verify-real-ghidra-mips-relocatable.mjs b/scripts/verify-real-ghidra-mips-relocatable.mjs index 5ce1704e22..49de4c1bfe 100755 --- a/scripts/verify-real-ghidra-mips-relocatable.mjs +++ b/scripts/verify-real-ghidra-mips-relocatable.mjs @@ -194,7 +194,7 @@ async function verifyFixture(byteOrder, kernelModule) { "Missing call to the external function", ); const dossier = functionDossierSchema.parse( - await call("analyze_function", { procedure: entry }), + await call("analyze_function", { procedure: entry, detail: "complete" }), ); for (const name of [ "rea_mips_rel_leaf", @@ -207,6 +207,7 @@ async function verifyFixture(byteOrder, kernelModule) { const scaled = functionDossierSchema.parse( await call("analyze_function", { procedure: named("rea_mips_rel_scaled"), + detail: "complete", }), ); assert.match(scaled.pseudocode, /\bpowf\(1\.5,2\.5\)/u); diff --git a/scripts/verify-real-ghidra-mips-shared.mjs b/scripts/verify-real-ghidra-mips-shared.mjs index da52495ce4..133306de32 100644 --- a/scripts/verify-real-ghidra-mips-shared.mjs +++ b/scripts/verify-real-ghidra-mips-shared.mjs @@ -243,11 +243,13 @@ try { ); const dossier = await call("analyze_function", { procedure: compressed, + detail: "complete", }); assert.match(dossier.pseudocode, /\^\s*0x5a/u); assert.match(dossier.pseudocode, /\+\s*(?:7|0x7)\b/u); const standardDossier = await call("analyze_function", { procedure: standard, + detail: "complete", }); assert.match(standardDossier.pseudocode, /\+\s*(?:3|0x3)\b/u); const snapshotPath = join( diff --git a/scripts/verify-real-ghidra-mips.mjs b/scripts/verify-real-ghidra-mips.mjs index 2edecf4f56..7d7c2c47de 100644 --- a/scripts/verify-real-ghidra-mips.mjs +++ b/scripts/verify-real-ghidra-mips.mjs @@ -100,6 +100,7 @@ async function verifySoftFloat(call, symbols, byteOrder) { const caller = functionDossierSchema.parse( await call("analyze_function", { procedure: symbols.rea_mips_soft_caller, + detail: "complete", }), ); // Includes the struct call and the indirect call through a typed pointer. @@ -211,7 +212,10 @@ try { legacy.symbols.rea_mips_probe, ); const dossier = functionDossierSchema.parse( - await call("analyze_function", { procedure: entry.address }), + await call("analyze_function", { + procedure: entry.address, + detail: "complete", + }), ); assert.ok(dossier.pseudocode.trim().length > 0); const saved = await call("close_binary", { @@ -461,7 +465,10 @@ try { }); assert.ok(assembly.trim().length > 0); const dossier = functionDossierSchema.parse( - await call("analyze_function", { procedure: entry.address }), + await call("analyze_function", { + procedure: entry.address, + detail: "complete", + }), ); assert.ok(dossier.pseudocode.trim().length > 0); const mapping = await call("address_to_file_offset", { diff --git a/scripts/verify-real-ghidra-nativeaot.mjs b/scripts/verify-real-ghidra-nativeaot.mjs index edac6495ba..e6280f601a 100644 --- a/scripts/verify-real-ghidra-nativeaot.mjs +++ b/scripts/verify-real-ghidra-nativeaot.mjs @@ -290,6 +290,7 @@ try { ); const compute = await call("analyze_function", { procedure: addressOf(fixtureSymbols.compute), + detail: "complete", }); assert.ok(compute.pseudocode.trim()); assert.doesNotMatch( @@ -300,6 +301,7 @@ try { assert.match(compute.pseudocode, /(?:0xb|\b11\b|<<)/u); const dispatch = await call("analyze_function", { procedure: addressOf(fixtureSymbols.dispatch), + detail: "complete", }); assert.ok(dispatch.pseudocode.trim()); for (const literal of oracle.expected_strings) diff --git a/scripts/verify-real-ghidra-ppc.mjs b/scripts/verify-real-ghidra-ppc.mjs index 84313625d0..32d56f83e5 100644 --- a/scripts/verify-real-ghidra-ppc.mjs +++ b/scripts/verify-real-ghidra-ppc.mjs @@ -161,7 +161,10 @@ try { }); assert.ok(assembly.trim().length > 0); const dossier = functionDossierSchema.parse( - await call("analyze_function", { procedure: entry.address }), + await call("analyze_function", { + procedure: entry.address, + detail: "complete", + }), ); assert.ok(dossier.pseudocode.trim().length > 0); const mapping = await call("address_to_file_offset", { diff --git a/scripts/verify-real-ghidra-riscv.mjs b/scripts/verify-real-ghidra-riscv.mjs index b0c036b980..766b064434 100644 --- a/scripts/verify-real-ghidra-riscv.mjs +++ b/scripts/verify-real-ghidra-riscv.mjs @@ -213,6 +213,7 @@ try { const dossier = functionDossierSchema.parse( await call("analyze_function", { procedure: fixture.symbols.rea_riscv_entry, + detail: "complete", }), ); assert.ok(dossier.pseudocode.trim().length > 0); diff --git a/scripts/verify-real-ghidra-swift.mjs b/scripts/verify-real-ghidra-swift.mjs index ecb8239508..09d92c4fb1 100644 --- a/scripts/verify-real-ghidra-swift.mjs +++ b/scripts/verify-real-ghidra-swift.mjs @@ -243,6 +243,7 @@ async function verifyStringLiterals() { assert(procedure, `${name} is missing from the procedure inventory`); const dossier = await call("analyze_function", { procedure: procedure.address, + detail: "complete", }); const observed = dossier.swift_string_literals; assert(Array.isArray(observed), `${name} recorded no Swift literals`); @@ -270,6 +271,7 @@ async function verifyStringLiterals() { ); const unresolvedDossier = await call("analyze_function", { procedure: unresolvedJump.address, + detail: "complete", }); assert.deepEqual(unresolvedDossier.swift_string_literals, []); assert.deepEqual( diff --git a/scripts/verify-real-hopper.mjs b/scripts/verify-real-hopper.mjs index ad6be40474..17ddde21c8 100644 --- a/scripts/verify-real-hopper.mjs +++ b/scripts/verify-real-hopper.mjs @@ -203,7 +203,7 @@ const verifyCurrentTarget = async (client, options) => { await client.callTool( { name: "analyze_function", - arguments: { procedure: firstAddress }, + arguments: { procedure: firstAddress, detail: "complete" }, }, options, ), diff --git a/scripts/verify-real-ida.mjs b/scripts/verify-real-ida.mjs index 1eb20d32b4..47a5f63f9d 100644 --- a/scripts/verify-real-ida.mjs +++ b/scripts/verify-real-ida.mjs @@ -187,6 +187,7 @@ const mcp = async (cliDossier) => { opened = true; const analyzed = await call("analyze_function", { procedure: values.procedure, + detail: "complete", }); const dossier = functionDossierSchema.parse(observe(analyzed)); assert.deepEqual(dossier.procedure, cliDossier.procedure); diff --git a/scripts/verify-real-kuna.mjs b/scripts/verify-real-kuna.mjs index d801cc067c..926e085c94 100644 --- a/scripts/verify-real-kuna.mjs +++ b/scripts/verify-real-kuna.mjs @@ -369,6 +369,7 @@ const verifyKuna = async ( ); const dossier = await cold.call("analyze_function", { procedure: address, + detail: "complete", }); assert.deepEqual( dossier.structured.relationship_coverage, diff --git a/scripts/verify/firmware/analysis.mjs b/scripts/verify/firmware/analysis.mjs index cbbd242014..a3945c02a5 100644 --- a/scripts/verify/firmware/analysis.mjs +++ b/scripts/verify/firmware/analysis.mjs @@ -351,6 +351,7 @@ async function stageGhidra(mcpFiles) { const dossierRaw = await callTool("analyze_function", { procedure: "firmware_probe", + detail: "complete", }); const dossier = parseEvidence(dossierRaw); assert.equal( diff --git a/src/application/EvidenceBundleFiles.test.ts b/src/application/EvidenceBundleFiles.test.ts new file mode 100644 index 0000000000..26e7ae1d2c --- /dev/null +++ b/src/application/EvidenceBundleFiles.test.ts @@ -0,0 +1,68 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { afterEach, describe, expect, it, vi } from "vitest"; + +import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; +import { createEvidence } from "../domain/evidence.js"; +import { createEvidenceBundle } from "../domain/evidenceBundle.js"; + +const HEADROOM_MODULE = "./JsonAuthenticationHeadroom.js"; + +afterEach(() => { + vi.doUnmock(HEADROOM_MODULE); + vi.resetModules(); +}); + +/** The reader bound to the real budget, measured against a heap that holds 1024 bytes. */ +const readerUnderStarvedHeap = async () => { + vi.resetModules(); + vi.doMock(HEADROOM_MODULE, async (importOriginal) => { + const actual = + await importOriginal(); + return { + ...actual, + requireAuthenticationHeadroom: ( + value: unknown, + options: Parameters[1], + ) => + actual.requireAuthenticationHeadroom(value, { + ...options, + availableBytes: 1024, + }), + }; + }); + return import("./EvidenceBundleFiles.js"); +}; + +describe("evidence bundle import headroom", () => { + it("refuses to authenticate a bundle the live heap cannot hold", async () => { + const path = join( + await createTestTempDirectory("rea-evidence-bundle-headroom-"), + "bundle.json", + ); + const record = createEvidence( + undefined, + { id: "fixture", name: "Fixture", version: "1" }, + { + operation: "observe", + parameters: {}, + result: { text: "y".repeat(10_000) }, + }, + ); + await writeFile(path, JSON.stringify(createEvidenceBundle([record]))); + const { readEvidenceBundle } = await readerUnderStarvedHeap(); + expect(await readEvidenceBundle(path)).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisResourceConstraintError", + operation: "import_evidence_bundle", + resource: "memory", + reportedLimits: { + input_path: path, + available_heap_bytes: 1024, + }, + }, + }); + }); +}); diff --git a/src/application/EvidenceBundleFiles.ts b/src/application/EvidenceBundleFiles.ts index 1cf55ebc7b..5b290cb0a9 100644 --- a/src/application/EvidenceBundleFiles.ts +++ b/src/application/EvidenceBundleFiles.ts @@ -6,7 +6,6 @@ import { safeParseImmutableEvidenceBundle, safeParseEvidenceBundle, type EvidenceBundle, - type EvidenceBundleFailure, } from "../domain/evidenceBundle.js"; import { EvidenceFileError, @@ -42,7 +41,7 @@ export const readEvidenceBundle = ( resources?: ArtifactResourceScope, operation = "import_evidence_bundle", ): Promise> => - readBundle(path, safeParseEvidenceBundle, signal, resources, operation); + readBundle(path, false, signal, resources, operation); /** Read an authenticated immutable snapshot for a subsequent session import. */ export const readImmutableEvidenceBundle = ( @@ -51,19 +50,11 @@ export const readImmutableEvidenceBundle = ( resources?: ArtifactResourceScope, operation = "import_evidence_bundle", ): Promise> => - readBundle( - path, - safeParseImmutableEvidenceBundle, - signal, - resources, - operation, - ); + readBundle(path, true, signal, resources, operation); const readBundle = async ( path: string, - parseBundle: ( - input: unknown, - ) => Result, + immutable: boolean, signal?: AbortSignal, resources?: ArtifactResourceScope, operation = "import_evidence_bundle", @@ -82,14 +73,30 @@ const readBundle = async ( cause: signal.reason, }), ); - const headroom = requireAuthenticationHeadroom(loaded.value, { + return authenticateEvidenceBundleValue(loaded.value, path, { + operation, + immutable, + }); +}; + +/** Authenticate a loaded bundle once, preserving file-reader failure and process-capture semantics. */ +export const authenticateEvidenceBundleValue = ( + value: unknown, + path: string, + options: { readonly operation: string; readonly immutable?: boolean }, +): Result => { + const { operation } = options; + const headroom = requireAuthenticationHeadroom(value, { operation, path: absoluteCallerPath(path), description: "Insufficient heap headroom to authenticate the Evidence bundle", }); if (!headroom.ok) return headroom; - const parsed = parseBundle(loaded.value); + const parsed = + options.immutable === true + ? safeParseImmutableEvidenceBundle(value) + : safeParseEvidenceBundle(value); if (!parsed.ok) { const failure = parsed.error; if (failure.kind === "single-record") diff --git a/src/application/FunctionAnalysisEvidence.ts b/src/application/FunctionAnalysisEvidence.ts index eed135ca34..2502adde81 100644 --- a/src/application/FunctionAnalysisEvidence.ts +++ b/src/application/FunctionAnalysisEvidence.ts @@ -12,7 +12,11 @@ import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js" import { AnalysisCancelledError } from "../domain/analysisErrorCore.js"; import { completeProviderEvidence } from "./ProviderOperationExecution.js"; -/** Preserve the provider's observations when presenting a validated function dossier. */ +/** + * Preserve the provider's observations when presenting a validated function + * dossier. Parameters are the provider input only: adapters separate delivery + * options such as MCP detail before calling. + */ export const executeFunctionAnalysisEvidence = async ( analysis: AnalysisOperationPort, parameters: Readonly>, diff --git a/src/application/analysisView/AnalysisViewService.test.ts b/src/application/analysisView/AnalysisViewService.test.ts index cb3e96ec8a..9b8476673a 100644 --- a/src/application/analysisView/AnalysisViewService.test.ts +++ b/src/application/analysisView/AnalysisViewService.test.ts @@ -183,6 +183,17 @@ it("projects a small native view from an authenticated >10 MiB retained dossier" { kind: "artifact-path", path: "/fixtures/large.exe" }, { kind: "address", address: "0x401000" }, ]); + // The summary describes the same function, so its envelope keeps the address. + const summary = await inspectAnalysisView( + { source: input.source, view: { kind: "summary" } }, + (id) => (id === parent.evidence_id ? parent : undefined), + ); + if (!summary.ok) throw summary.error; + expect(summary.value.normalized_result).toMatchObject({ + kind: "summary", + procedure_address: "0x401000", + }); + expect(summary.value.locations).toEqual(view.value.locations); expect(JSON.stringify(view.value).length).toBeLessThan(12000); expect(JSON.stringify(parent).length).toBeGreaterThan(10 * 1024 * 1024); expect(await inspectAnalysisView(input, () => undefined)).toMatchObject({ diff --git a/src/application/analysisView/AnalysisViewService.ts b/src/application/analysisView/AnalysisViewService.ts index 8ac2f6f9f4..5b386e14a5 100644 --- a/src/application/analysisView/AnalysisViewService.ts +++ b/src/application/analysisView/AnalysisViewService.ts @@ -92,7 +92,11 @@ const inspectFileView = async ( lookup?: EvidenceLookup, options: EvidenceFileReadOptions = {}, ): Promise> => { - const loaded = await readEvidenceFileSource(input.source.path, options); + const loaded = await readEvidenceFileSource( + input.source.path, + options, + input.source.evidence_id, + ); // Authentication and projection are synchronous; yield so a pending abort is dispatched before each check. const cancelled = async (): Promise => { await setImmediate(); @@ -363,6 +367,12 @@ const createAnalysisViewEvidence = ( ? {} : { architecture: parent.subject.architecture }), }; + const nativeFunctionView = + result.kind === "native" || + (result.kind === "summary" && + result.parent_operation === "analyze_function") + ? result + : undefined; return createEvidence( target, ANALYSIS_VIEW_PROVIDER, @@ -386,17 +396,23 @@ const createAnalysisViewEvidence = ( : "derived", authority: parent.authority, limitations: result.limitations, - ...(result.kind === "native" && parent.relationship_coverage !== null + ...(nativeFunctionView !== undefined && + parent.relationship_coverage !== null ? { relationshipCoverage: parent.relationship_coverage } : {}), locations: [ ...(result.artifact != null && result.artifact.path.length > 0 ? [{ kind: "artifact-path" as const, path: result.artifact.path }] : []), - ...(result.kind === "native" && - result.procedure_address !== null && - result.procedure_address.length > 0 - ? [{ kind: "address" as const, address: result.procedure_address }] + ...(nativeFunctionView !== undefined && + nativeFunctionView.procedure_address !== null && + nativeFunctionView.procedure_address.length > 0 + ? [ + { + kind: "address" as const, + address: nativeFunctionView.procedure_address, + }, + ] : []), ], evidenceLinks: [parent.evidence_id], diff --git a/src/application/analysisView/evidenceFileSource.test.ts b/src/application/analysisView/evidenceFileSource.test.ts index b9a71de605..c44b3d4137 100644 --- a/src/application/analysisView/evidenceFileSource.test.ts +++ b/src/application/analysisView/evidenceFileSource.test.ts @@ -3,16 +3,24 @@ import { join } from "node:path"; import { describe, expect, it, vi } from "vitest"; -import { analysisViewCaptureEvidence } from "../../../tests/fixtures/analysisView.js"; +import { + analysisViewCaptureEvidence, + analysisViewNativeEvidence, +} from "../../../tests/fixtures/analysisView.js"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; import { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; -import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js"; +import { + AnalysisCancelledError, + AnalysisResourceConstraintError, +} from "../../domain/analysisErrorCore.js"; import { JsonInputPhysicalReadFailure } from "../JsonInputFile.js"; import { failFileReadAfterAbort } from "../../../tests/support/failFileReadAfterAbort.js"; import { holdFirstFileRead } from "../../../tests/support/holdFirstFileRead.js"; +import { createEvidence } from "../../domain/evidence.js"; +import { createEvidenceBundle } from "../../domain/evidenceBundle.js"; import { readEvidenceFileSource } from "./evidenceFileSource.js"; -describe("evidence-file source", () => { +describe("evidence-file source reads", () => { it("refuses a cancelled request before reading and reads within the caller's resource scope", async () => { const controller = new AbortController(); const reason = new JsonInputPhysicalReadFailure( @@ -44,6 +52,63 @@ describe("evidence-file source", () => { if (!read.ok) throw read.error; expect(read.value).toEqual(JSON.parse(JSON.stringify(parent))); }); + + it("projects the selected or only record of an Evidence bundle file", async () => { + const root = await createTestTempDirectory("rea-evidence-bundle-source-"); + const native = analysisViewNativeEvidence(); + const capture = analysisViewCaptureEvidence(); + const single = join(root, "single.json"); + await writeFile(single, JSON.stringify(createEvidenceBundle([native]))); + const only = await readEvidenceFileSource(single); + if (!only.ok) throw only.error; + expect(only.value.evidence_id).toBe(native.evidence_id); + + const several = join(root, "several.json"); + await writeFile( + several, + JSON.stringify(createEvidenceBundle([native, capture])), + ); + const selected = await readEvidenceFileSource( + several, + {}, + capture.evidence_id, + ); + if (!selected.ok) throw selected.error; + expect(selected.value.evidence_id).toBe(capture.evidence_id); + + // Ambiguity and a missing selection are input errors that name the records held. + const ambiguous = await readEvidenceFileSource(several); + expect(ambiguous).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisInputError", + issues: [{ path: ["source", "evidence_id"], reason: "invalid_value" }], + }, + }); + if (ambiguous.ok) throw new Error("Expected an ambiguous bundle"); + expect(ambiguous.error.userMessage).toContain(native.evidence_id); + expect(ambiguous.error.userMessage).toContain("analyze_function"); + const absent = "ev_" + "f".repeat(64); + const missing = await readEvidenceFileSource(several, {}, absent); + expect(missing).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisInputError", + userMessage: expect.stringContaining(absent), + }, + }); + + // Bundle authentication uses the same integrity classification as bundle import. + const invalid = join(root, "invalid.json"); + await writeFile(invalid, JSON.stringify({ records: [{ hello: 1 }] })); + expect(await readEvidenceFileSource(invalid)).toMatchObject({ + ok: false, + error: { + _tag: "EvidenceIntegrityError", + userMessage: expect.stringContaining("Evidence bundle"), + }, + }); + }); }); describe("evidence-file read failure provenance", () => { @@ -162,3 +227,52 @@ describe("evidence-file read failure provenance", () => { } }); }); + +it("honors an explicit identity for a single-record Evidence file", async () => { + const root = await createTestTempDirectory("rea-evidence-file-selection-"); + const path = join(root, "record.json"); + const record = analysisViewNativeEvidence(); + const other = analysisViewCaptureEvidence(); + await writeFile(path, JSON.stringify(record)); + expect( + await readEvidenceFileSource(path, {}, record.evidence_id), + ).toMatchObject({ ok: true }); + expect( + await readEvidenceFileSource(path, {}, other.evidence_id), + ).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisInputError", + issues: [{ path: ["source", "evidence_id"] }], + }, + }); +}); + +it("keeps bundle-selection errors useful when operation labels exceed the reply budget", async () => { + const root = await createTestTempDirectory("rea-evidence-file-labels-"); + const path = join(root, "records.json"); + const operation = '\u0000"'.repeat(50_000); + const records = [1, 2].map((ordinal) => + createEvidence( + undefined, + { id: "fixture", name: "Fixture", version: "1" }, + { + operation, + parameters: {}, + result: { ordinal }, + }, + ), + ); + await writeFile(path, JSON.stringify(createEvidenceBundle(records))); + const read = await readEvidenceFileSource(path); + expect(read).toMatchObject({ + ok: false, + error: { _tag: "AnalysisInputError" }, + }); + if (read.ok) throw new Error("Expected a record-selection error"); + const encoded = JSON.stringify(read); + expect(Buffer.byteLength(encoded, "utf8")).toBeLessThan(8192); + for (const record of records) expect(encoded).toContain(record.evidence_id); + expect(encoded).toContain("operation omitted"); + expect(encoded).not.toContain(operation); +}); diff --git a/src/application/analysisView/evidenceFileSource.ts b/src/application/analysisView/evidenceFileSource.ts index a1ca150ce4..a8e34e9012 100644 --- a/src/application/analysisView/evidenceFileSource.ts +++ b/src/application/analysisView/evidenceFileSource.ts @@ -1,63 +1,149 @@ import { requireAuthenticationHeadroom } from "../JsonAuthenticationHeadroom.js"; - +import { authenticateEvidenceBundleValue } from "../EvidenceBundleFiles.js"; import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; import type { AnalysisError } from "../../domain/analysisErrorBase.js"; -import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js"; +import { + AnalysisCancelledError, + AnalysisInputError, +} from "../../domain/analysisErrorCore.js"; import { INSPECT_ANALYSIS_VIEW_OPERATION } from "../../domain/analysisView/analysisViewErrors.js"; -import { err, type Result } from "../../domain/result.js"; +import { err, ok, type Result } from "../../domain/result.js"; import type { Evidence } from "../../domain/evidence.js"; import { safeParseEvidence } from "../../domain/evidenceValidation.js"; import { EvidenceFileError } from "../../domain/evidenceErrors.js"; import { projectInputIssues } from "../../domain/inputIssueProjection.js"; import { readJsonFile } from "../JsonFiles.js"; -/** The request's cancellation and the scope that owns descriptors whose close failed. */ +/** The request cancellation and the scope retaining descriptors whose close failed. */ export interface EvidenceFileReadOptions { readonly signal?: AbortSignal; readonly resources?: ArtifactResourceScope; } -/** Read and authenticate one Evidence file under the request's cancellation and resource scope. */ +const selectionError = (message: string): AnalysisInputError => + new AnalysisInputError( + INSPECT_ANALYSIS_VIEW_OPERATION, + { userMessage: message }, + [ + { + path: ["source", "evidence_id"], + reason: "invalid_value", + message, + }, + ], + ); + +/** Listed identities stay bounded so a large bundle cannot grow the error itself. */ +const RECORD_REFERENCE_PREVIEW_BYTES = 2048; + +/** Select the one record a view projects from an authenticated bundle. */ +const selectBundleRecord = ( + records: readonly Evidence[], + path: string, + evidenceId: string | undefined, +): Result => { + if (evidenceId !== undefined) { + const selected = records.find( + (record) => record.evidence_id === evidenceId, + ); + return selected === undefined + ? err( + selectionError( + `Evidence bundle ${path} holds no record ${evidenceId}. It holds ${listRecordIds(records)}.`, + ), + ) + : ok(selected); + } + const only = records[0]; + if (only === undefined) + return err(selectionError(`Evidence bundle ${path} holds no records.`)); + if (records.length > 1) + return err( + selectionError( + `Evidence bundle ${path} holds ${String(records.length)} records; select one with source.evidence_id. It holds ${listRecordIds(records)}.`, + ), + ); + return ok(only); +}; + +const listRecordIds = (records: readonly Evidence[]): string => { + const labels: string[] = []; + let bytes = 0; + for (const record of records) { + const separator = labels.length === 0 ? "" : ", "; + const remaining = RECORD_REFERENCE_PREVIEW_BYTES - bytes; + let label = `${record.evidence_id} (operation omitted)`; + // UTF-8/JSON cannot use fewer bytes than the source UTF-16 code units. + // Reject a label before copying it when even that lower bound cannot fit. + if (record.operation.length <= remaining) { + const complete = `${record.evidence_id} (${record.operation})`; + if ( + Buffer.byteLength(JSON.stringify(separator + complete), "utf8") <= + remaining + ) + label = complete; + } + const encodedBytes = Buffer.byteLength( + JSON.stringify(separator + label), + "utf8", + ); + if (encodedBytes > remaining) break; + labels.push(label); + bytes += encodedBytes; + } + const omitted = records.length - labels.length; + const listed = labels.join(", "); + return omitted > 0 ? `${listed} and ${String(omitted)} more` : listed; +}; + +/** Read one Evidence record or select from an authenticated bundle without rereading the file. */ export const readEvidenceFileSource = async ( path: string, options: EvidenceFileReadOptions = {}, + evidenceId?: string, ): Promise> => { const { signal, resources } = options; + const operation = INSPECT_ANALYSIS_VIEW_OPERATION; if (signal?.aborted) - return err( - new AnalysisCancelledError(INSPECT_ANALYSIS_VIEW_OPERATION, { - cause: signal.reason, - }), - ); - const loaded = await readJsonFile( - path, - signal, - resources, - INSPECT_ANALYSIS_VIEW_OPERATION, - ); + return err(new AnalysisCancelledError(operation, { cause: signal.reason })); + const loaded = await readJsonFile(path, signal, resources, operation); if (!loaded.ok) return loaded; if (signal?.aborted) - return err( - new AnalysisCancelledError(INSPECT_ANALYSIS_VIEW_OPERATION, { - cause: signal.reason, - }), - ); - const headroom = requireAuthenticationHeadroom(loaded.value, { - operation: INSPECT_ANALYSIS_VIEW_OPERATION, + return err(new AnalysisCancelledError(operation, { cause: signal.reason })); + const value = loaded.value; + if ( + typeof value === "object" && + value !== null && + !Array.isArray(value) && + Object.hasOwn(value, "records") + ) { + const bundle = authenticateEvidenceBundleValue(value, path, { operation }); + return bundle.ok + ? selectBundleRecord(bundle.value.records, path, evidenceId) + : bundle; + } + const headroom = requireAuthenticationHeadroom(value, { + operation, path, description: "Insufficient heap headroom to authenticate the Evidence file", }); if (!headroom.ok) return headroom; - const parsed = safeParseEvidence(loaded.value); - return parsed.ok - ? parsed - : err( - new EvidenceFileError("read", "wrong-kind", { - path, - cause: parsed.error, - contentIssues: projectInputIssues(parsed.error.issues, loaded.value), - userMessage: - "The selected file does not contain valid Evidence. Repair or recreate the file and try again.", - }), - ); + const parsed = safeParseEvidence(value); + if (!parsed.ok) + return err( + new EvidenceFileError("read", "wrong-kind", { + path, + cause: parsed.error, + contentIssues: projectInputIssues(parsed.error.issues, value), + userMessage: + "The selected file does not contain valid Evidence. Repair or recreate the file and try again.", + }), + ); + if (evidenceId !== undefined && parsed.value.evidence_id !== evidenceId) + return err( + selectionError( + `Evidence file ${path} holds ${parsed.value.evidence_id}, not selected record ${evidenceId}.`, + ), + ); + return parsed; }; diff --git a/src/cli/coreBinaryCommands.ts b/src/cli/coreBinaryCommands.ts index e40e603534..9268ecb436 100644 --- a/src/cli/coreBinaryCommands.ts +++ b/src/cli/coreBinaryCommands.ts @@ -20,6 +20,12 @@ import { import type { CliInstance } from "./types.js"; import { runCliJavaScriptApplicationAnalysis } from "./javascriptApplicationAnalysis.js"; import { withCommandCancellation } from "./commandCancellation.js"; +import { + analysisEvidenceDeliveryOptions, + cliEvidenceDeliveryInputError, + publishCliAnalysisEvidence, +} from "./analysisEvidenceDelivery.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import { javascriptApplicationOptions } from "../cliObservationOptions.js"; import { AnalysisInputError } from "../domain/analysisErrorCore.js"; import { localPathStringSchema } from "../domain/localPath.js"; @@ -380,23 +386,37 @@ const registerFunctionCommand = ( `Procedure name or address; use --procedure= for leading dashes. ${addressOrProcedureGuidance}`, ), snapshot: snapshotSelectionOption, + ...analysisEvidenceDeliveryOptions, ...nativeOpenOptionFields, }), run: ({ args, options }) => - logCliCommand(logger, "function", () => - runDirectAnalysis( - args.path, - "analyze_function", - { - procedure: resolveCliAnalysisSelector( - args.address, - options.procedure, - "procedure", - "analyze_function", - ), - }, - directAnalysisOptionsFromCli(logger, options), - ), + withCommandCancellation((signal) => + logCliCommand(logger, "function", async () => { + const delivery = { + providerId: "auto", + operation: "analyze_function", + signal, + detail: options.detail, + evidenceOutput: options.evidence_output, + }; + // Fail before analysis when summary delivery lacks its Evidence bundle sink. + const inputError = cliEvidenceDeliveryInputError(delivery); + if (inputError !== undefined) return projectAnalysisError(inputError); + const complete = await runDirectAnalysis( + args.path, + "analyze_function", + { + procedure: resolveCliAnalysisSelector( + args.address, + options.procedure, + "procedure", + "analyze_function", + ), + }, + { ...directAnalysisOptionsFromCli(logger, options), signal }, + ); + return publishCliAnalysisEvidence(complete, delivery); + }), ), }); }; diff --git a/src/contracts/analysisDetail.ts b/src/contracts/analysisDetail.ts index b6e0001645..03491440f6 100644 --- a/src/contracts/analysisDetail.ts +++ b/src/contracts/analysisDetail.ts @@ -10,6 +10,18 @@ export const analysisDetailSchema = z "complete returns the complete analysis Evidence inline. summary retains that complete Evidence in this session and returns its inspect_analysis_view summary; inspect pages, items or facets through normalized_result.parent_evidence_id.", ); +/** + * analyze_function delivery. Defaults to the token-frugal summary because a + * complete native dossier can reach several megabytes of value-flow and + * references; callers that need everything inline pass complete. + */ +export const analysisDetailSummaryDefaultSchema = z + .enum(["complete", "summary"]) + .default("summary") + .describe( + "summary (default) returns a compact dossier and retains the complete analyze_function Evidence in this session; page pseudocode, references and value-flow through normalized_result.parent_evidence_id with inspect_analysis_view. complete returns the full dossier inline.", + ); + /** * Complete analysis Evidence, or that operation's summary view of its retained * record. Callers pass the summary variant the handler actually returns so the diff --git a/src/contracts/analysisViewToolContracts.ts b/src/contracts/analysisViewToolContracts.ts index cb19115c38..744929bcab 100644 --- a/src/contracts/analysisViewToolContracts.ts +++ b/src/contracts/analysisViewToolContracts.ts @@ -17,7 +17,7 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [ ...toolContractMetadata("inspect_analysis_view"), kind: "application", description: - 'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of supported completed Evidence without repeating analysis. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for non-parameterized reads that are not entities and have one initial request line (item-shaped paths stay recorded), and a render plan per read tool: item and list views reproduce their latest recordings from the state (a list also learns the filters, search, sort and paging its recordings verify, and answers new values of them from the state), and every other read tool is a recorded view with a reason. It also reports a write plan per state-changing tool (create, update, delete or document, built from its first successful call) and write_clock; a plan is served only when replaying the capture shows a later read confirming its effect and none contradicting it. Types and identity are inferences; values are recorded data, so it refuses parents inspected with sensitive values, and a scenario parent must have selected response bodies; its values keep the capture redaction markers, and a write whose URL it redacted ends the state of every type. For a retained record use source: {kind: "retained-evidence", evidence_id: ""}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: }. A completed Evidence JSON file on this host uses {kind: "evidence-file", path: ""}: REA reads and authenticates it, then projects it like retained Evidence, so a large capture is bounded by the file reader rather than by inline-input accounting. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Typed module references use view {kind: "javascript", collection: "modules", projection: "references", where: {path: "main.js", kind: "javascript-module"}, offset: 0, limit: 64}; where matches exact retained strings (null selects unavailable paths). Rows include canonical identity, retained source-range count and observation support states. Source coverage and selection remain explicit; exhausting matching rows does not prove runtime loading or complete analysis. For exact file/native facts and processing gaps, use view {kind: "javascript-facts", where: {paths: ["package.json"]}, property_keys: ["name", "operation", "analysis_status"]}. Rows page original observations with canonical identities and Evidence; missing properties remain omitted. Paths match recorded artifact-path/source-range locations; optional kinds filter nodes. To discover field names without values, opt into include_available_property_keys: true with property_keys: []. Canonical identity and Evidence are separate from properties; keep each node_id/observation_id pair together when composing results. Inventory uses view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json"], offset: 0, limit: 64}. Inventory views page nodes, occurrences, edges or integrity contradictions. Occurrences accept exact logical_paths and occurrence_ids; nodes accept artifact_ids. Batch filters intersect (omitted means all available, [] means none); fields selects columns while retaining identity links. source_complete describes retained graph membership; source_coverage describes producer coverage, including unknown encrypted PCK directories. Page exhaustion does not prove absence in a partial source. WASM views use kind wasm and facet sections/imports/exports/candidates/glue/wat/headers/details/tool_profile; text facets use UTF-16 code units and source forms remain exact. Mach-O views use kind macho and facet entrypoints/architectures/build_metadata/load_commands/dependencies/imports/exports/segments/provenance; source_exhaustive false means unreturned rows may exist. Interface Builder views use kind interface-builder and collection documents/nodes/edges/coverage; nodes and edges accept an exact document relative_path. EVM summary returns all inferred interface candidates, carrier/decoded identities and explicit unknowns without hex or duplicate stdout. EVM views use kind evm and facet functions/bytecode_hex/stdout/stderr; function pages preserve the original candidate rows, and text offsets/limits count UTF-16 code units (hex has two characters per decoded byte). Exhausting candidate pages does not establish global interface completeness. Managed artifact summaries preserve complete identity/classification, metadata status, module/MVID, frameworks, coverage and limitations without unrelated inventories. Managed pages use kind managed and collection references/resources/attributes, retaining source coverage and build-local token scope; page exhaustion does not establish complete metadata. Managed member summaries keep identity, selection with unmatched names, coverage and per-collection counts; managed-members pages use collection types/exported_types/fields/methods/member_refs/call_edges/field_accesses/user_strings and carry the source selection and coverage. Android package summaries retain all declarations, engine identity and counts without decoded manifest text. Android manifest pages use kind android, facet manifest, offset and limit in UTF-16 code units; source_status preserves partial decoded coverage even when the retained text page is exhausted. Dart AOT summary preserves target/ABI/library identity, snapshot headers and pool lower bounds. String pages use kind dart-aot, collection package_uris/dart_uris/class_like_tokens; source coverage remains partial when the producer says partial. Protobuf views use kind protobuf, collection bodies/frames/fields, with record_ordinals, body_pointers and fields; field pages also select frame_indices, field_ids, field_numbers or schema_names. Byte identity, offset basis and body status remain present. Text and schema_value are opt-in page columns. APK signing views use kind apk-signature and collection signers/digests/lineage, with ids, signer_ids, schemes, fields and paging. Identity links remain when fields is empty. Declared identities and digest matches do not establish signature verification or authenticity. Model views use kind model-artifact and collection metadata (keys), tensors (names/types) or types (derived totals), with indices, fields and paging. Header SHA-256 scopes declarations; full-file identity remains unknown unless full hashing was selected. Arrays retain typed eight-element previews with explicit omissions. IL2CPP metadata views use kind il2cpp-metadata and collection sections/images/types/methods/fields/properties/events/parameters/string_literals/code_gen_modules/method_addresses/return_types/field_types/parameter_types. Select indices, fields and pages; sections/images/types/methods/fields/properties/events/parameters accept names, types also accept image_indices/namespaces/declaring_types, and methods/members accept declaring_types. Parameters and parameter_types accept method_indices; code_gen_modules accept image_indices; method_addresses accept addresses/states, preserving every folded alternative. Row index is scoped to the metadata digest; native pages add binary_source. Uncollected native tables have null counts and remain unavailable. Desktop directory views use kind desktop-application and collection files/omissions/candidates/components/next_probes, with fields and pagination; files and omissions accept exact paths or their stable IDs, and candidates accept families. Source coverage and stable row IDs remain available. Native inventory Evidence from list_names, list/search_procedures or list/search_strings uses view: {kind: "native-inventory", offset: 0, limit: 64, fields: ["value"]}. Rows keep source_index and address even with selected fields. Summary counts describe retained rows; provider discovery coverage remains unknown. Native function Evidence requires kind native: view: {kind: "native", facet: "procedure"}. Facets select procedure, pseudocode, assembly, basic blocks, comments, callers/callees, references, unresolved calls, referenced strings/names, native API and high-pcode. Use nonnegative offset and a positive caller-selected limit (defaults 0 and 64); pseudocode uses UTF-16 code units, other native pages use rows. Returns artifact identity, parent Evidence ID, a distinct view digest, coverage, limitations and unknowns. Actual serialized size determines MCP transport admission; choose a smaller page or export the complete retained Evidence if needed.', + 'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of supported completed Evidence without repeating analysis. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for non-parameterized reads that are not entities and have one initial request line (item-shaped paths stay recorded), and a render plan per read tool: item and list views reproduce their latest recordings from the state (a list also learns the filters, search, sort and paging its recordings verify, and answers new values of them from the state), and every other read tool is a recorded view with a reason. It also reports a write plan per state-changing tool (create, update, delete or document, built from its first successful call) and write_clock; a plan is served only when replaying the capture shows a later read confirming its effect and none contradicting it. Types and identity are inferences; values are recorded data, so it refuses parents inspected with sensitive values, and a scenario parent must have selected response bodies; its values keep the capture redaction markers, and a write whose URL it redacted ends the state of every type. For a retained record use source: {kind: "retained-evidence", evidence_id: ""}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: }. A completed Evidence JSON file on this host uses {kind: "evidence-file", path: ""}: REA reads and authenticates it, then projects it like retained Evidence, so a large capture is bounded by the file reader rather than by inline-input accounting. The file may also be an Evidence bundle written by CLI --evidence_output or export_evidence_bundle; select evidence_id when it contains more than one record. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Typed module references use view {kind: "javascript", collection: "modules", projection: "references", where: {path: "main.js", kind: "javascript-module"}, offset: 0, limit: 64}; where matches exact retained strings (null selects unavailable paths). Rows include canonical identity, retained source-range count and observation support states. Source coverage and selection remain explicit; exhausting matching rows does not prove runtime loading or complete analysis. For exact file/native facts and processing gaps, use view {kind: "javascript-facts", where: {paths: ["package.json"]}, property_keys: ["name", "operation", "analysis_status"]}. Rows page original observations with canonical identities and Evidence; missing properties remain omitted. Paths match recorded artifact-path/source-range locations; optional kinds filter nodes. To discover field names without values, opt into include_available_property_keys: true with property_keys: []. Canonical identity and Evidence are separate from properties; keep each node_id/observation_id pair together when composing results. Inventory uses view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json"], offset: 0, limit: 64}. Inventory views page nodes, occurrences, edges or integrity contradictions. Occurrences accept exact logical_paths and occurrence_ids; nodes accept artifact_ids. Batch filters intersect (omitted means all available, [] means none); fields selects columns while retaining identity links. source_complete describes retained graph membership; source_coverage describes producer coverage, including unknown encrypted PCK directories. Page exhaustion does not prove absence in a partial source. WASM views use kind wasm and facet sections/imports/exports/candidates/glue/wat/headers/details/tool_profile; text facets use UTF-16 code units and source forms remain exact. Mach-O views use kind macho and facet entrypoints/architectures/build_metadata/load_commands/dependencies/imports/exports/segments/provenance; source_exhaustive false means unreturned rows may exist. Interface Builder views use kind interface-builder and collection documents/nodes/edges/coverage; nodes and edges accept an exact document relative_path. EVM summary returns all inferred interface candidates, carrier/decoded identities and explicit unknowns without hex or duplicate stdout. EVM views use kind evm and facet functions/bytecode_hex/stdout/stderr; function pages preserve the original candidate rows, and text offsets/limits count UTF-16 code units (hex has two characters per decoded byte). Exhausting candidate pages does not establish global interface completeness. Managed artifact summaries preserve complete identity/classification, metadata status, module/MVID, frameworks, coverage and limitations without unrelated inventories. Managed pages use kind managed and collection references/resources/attributes, retaining source coverage and build-local token scope; page exhaustion does not establish complete metadata. Managed member summaries keep identity, selection with unmatched names, coverage and per-collection counts; managed-members pages use collection types/exported_types/fields/methods/member_refs/call_edges/field_accesses/user_strings and carry the source selection and coverage. Android package summaries retain all declarations, engine identity and counts without decoded manifest text. Android manifest pages use kind android, facet manifest, offset and limit in UTF-16 code units; source_status preserves partial decoded coverage even when the retained text page is exhausted. Dart AOT summary preserves target/ABI/library identity, snapshot headers and pool lower bounds. String pages use kind dart-aot, collection package_uris/dart_uris/class_like_tokens; source coverage remains partial when the producer says partial. Protobuf views use kind protobuf, collection bodies/frames/fields, with record_ordinals, body_pointers and fields; field pages also select frame_indices, field_ids, field_numbers or schema_names. Byte identity, offset basis and body status remain present. Text and schema_value are opt-in page columns. APK signing views use kind apk-signature and collection signers/digests/lineage, with ids, signer_ids, schemes, fields and paging. Identity links remain when fields is empty. Declared identities and digest matches do not establish signature verification or authenticity. Model views use kind model-artifact and collection metadata (keys), tensors (names/types) or types (derived totals), with indices, fields and paging. Header SHA-256 scopes declarations; full-file identity remains unknown unless full hashing was selected. Arrays retain typed eight-element previews with explicit omissions. IL2CPP metadata views use kind il2cpp-metadata and collection sections/images/types/methods/fields/properties/events/parameters/string_literals/code_gen_modules/method_addresses/return_types/field_types/parameter_types. Select indices, fields and pages; sections/images/types/methods/fields/properties/events/parameters accept names, types also accept image_indices/namespaces/declaring_types, and methods/members accept declaring_types. Parameters and parameter_types accept method_indices; code_gen_modules accept image_indices; method_addresses accept addresses/states, preserving every folded alternative. Row index is scoped to the metadata digest; native pages add binary_source. Uncollected native tables have null counts and remain unavailable. Desktop directory views use kind desktop-application and collection files/omissions/candidates/components/next_probes, with fields and pagination; files and omissions accept exact paths or their stable IDs, and candidates accept families. Source coverage and stable row IDs remain available. Native inventory Evidence from list_names, list/search_procedures or list/search_strings uses view: {kind: "native-inventory", offset: 0, limit: 64, fields: ["value"]}. Rows keep source_index and address even with selected fields. Summary counts describe retained rows; provider discovery coverage remains unknown. Native function Evidence accepts kind summary for identity, availability and facet counts, or kind native for selected facts: view: {kind: "native", facet: "procedure"}. Facets select procedure, pseudocode, assembly, basic blocks, comments, callers/callees, references, unresolved calls, referenced strings/names, native API and high-pcode. Use nonnegative offset and a positive caller-selected limit (defaults 0 and 64); pseudocode uses UTF-16 code units, other native pages use rows. Returns artifact identity, parent Evidence ID, a distinct view digest, coverage, limitations and unknowns. Actual serialized size determines MCP transport admission; choose a smaller page or export the complete retained Evidence if needed.', inputSchema: inspectAnalysisViewInputSchema, outputSchema: evidenceResultOf(analysisViewResultSchema), examples: [ diff --git a/src/contracts/enhancedInputs.ts b/src/contracts/enhancedInputs.ts index 4c85e2937e..22cc9b0ce1 100644 --- a/src/contracts/enhancedInputs.ts +++ b/src/contracts/enhancedInputs.ts @@ -52,6 +52,7 @@ export const enhancedInputSchemas = { }), find_xrefs_to_name: z.strictObject({ name: z.string() }), binary_overview: z.strictObject({}), + // Provider input excludes delivery detail, which belongs to the MCP tool contract. analyze_function: nativeProcedureInputSchema, inspect_native_api: nativeProcedureInputSchema, trace_feature: traceLiteralInputSchema, diff --git a/src/contracts/functionWorkflowToolContracts.ts b/src/contracts/functionWorkflowToolContracts.ts index d1380186db..dd3b1c2ae7 100644 --- a/src/contracts/functionWorkflowToolContracts.ts +++ b/src/contracts/functionWorkflowToolContracts.ts @@ -1,4 +1,5 @@ import { jsonObjectSchema } from "../domain/jsonValue.js"; +import { analysisDetailSummaryDefaultSchema } from "./analysisDetail.js"; import { enhancedInputSchemas } from "./enhancedInputs.js"; import { TOOL_EXAMPLE_OVERRIDES } from "./toolContractExamples.js"; import type { ToolContract } from "./toolContractTypes.js"; @@ -8,11 +9,23 @@ import { requireOutputSchema } from "./toolOutputSchemaPrimitives.js"; type FunctionWorkflowName = "analyze_function" | "inspect_native_api"; +/** + * MCP inputs: the provider input plus analyze_function's delivery detail, which + * the tool handler separates before any provider call. Composed workflows such + * as inspect_native_api build provider inputs from enhancedInputSchemas alone. + */ +export const functionWorkflowInputSchemas = { + analyze_function: enhancedInputSchemas.analyze_function.extend({ + detail: analysisDetailSummaryDefaultSchema, + }), + inspect_native_api: enhancedInputSchemas.inspect_native_api, +} as const; + const functionWorkflow = ( name: Name, description: string, ) => { - const inputSchema = enhancedInputSchemas[name]; + const inputSchema = functionWorkflowInputSchemas[name]; const outputSchema = requireOutputSchema(enhancedOutputSchemas, name); return { name, @@ -37,7 +50,7 @@ const functionWorkflow = ( export const FUNCTION_WORKFLOW_TOOL_CONTRACTS = [ functionWorkflow( "analyze_function", - "Build a dossier for one native function identified by symbol or provider-returned address. Returns identity with complete inclusive analyzed body ranges or explicit unknown, provider-specific pseudocode and assembly, comments, calls, references, referenced strings/names, local CFG blocks, and available native API boundary observations. Ghidra terminal-call assumptions that may cut the analyzed body are reported in limitations with available callee identities.", + "Build a dossier for one native function identified by symbol or provider-returned address. Returns identity with complete inclusive analyzed body ranges or explicit unknown, provider-specific pseudocode and assembly, comments, calls, references, referenced strings/names, local CFG blocks, and available native API boundary observations. Ghidra terminal-call assumptions that may cut the analyzed body are reported in limitations with available callee identities. Default detail summary returns a compact dossier (identity, pseudocode size, facet counts, value-flow availability) and retains the complete dossier Evidence in this session; page pseudocode, references, assembly and value-flow through normalized_result.parent_evidence_id with inspect_analysis_view. detail complete returns the full dossier inline.", ), functionWorkflow( "inspect_native_api", diff --git a/src/contracts/promptContracts.ts b/src/contracts/promptContracts.ts index e8bc22b8ac..94da41ff0e 100644 --- a/src/contracts/promptContracts.ts +++ b/src/contracts/promptContracts.ts @@ -203,7 +203,7 @@ export const PROMPT_CONTRACTS = [ { tools: ["open_binary", "analyze_function", "compare_functions"], instruction: - "When code-level localization is needed, analyze corresponding functions on both targets and compare their Evidence.", + "When code-level localization is needed, analyze corresponding functions on both targets and compare their complete dossiers: pass each summary's parent_evidence_id as a retained-evidence reference, or request detail complete.", }, { tools: ["compare_process_captures", "find_changed_behavior"], @@ -297,7 +297,7 @@ export const PROMPT_CONTRACTS = [ "analyze_function", ], instruction: - "Resolve crash text or addresses to function dossiers; do not infer a source location from symbol similarity alone.", + "Resolve crash text or addresses to function dossiers, requesting detail complete when build_call_path will consume them; do not infer a source location from symbol similarity alone.", }, { tools: ["xrefs", "procedure_callers", "build_call_path"], diff --git a/src/contracts/toolOutputSchemaGroups.ts b/src/contracts/toolOutputSchemaGroups.ts index ed95184272..37d1c56f29 100644 --- a/src/contracts/toolOutputSchemaGroups.ts +++ b/src/contracts/toolOutputSchemaGroups.ts @@ -5,6 +5,8 @@ import { nativeFunctionMatchingSchema } from "../domain/native/functionMatching. import { nativeFunctionMatchingSummaryViewResultSchema } from "../domain/analysisView/analysisView.js"; import { nativeByteReadSchema } from "../domain/native/nativeByteRead.js"; import { analysisDetailOutputSchemaOf } from "./analysisDetail.js"; +import { nativeFunctionSummaryViewResultSchema } from "../domain/analysisView/analysisView.js"; +import { functionDossierSchema } from "../domain/hopperValues.js"; import { nativeInventorySummaryViewResultSchema } from "../domain/analysisView/analysisView.js"; import { nativeInventoryRowsSchemas } from "../domain/native/nativeInventory.js"; import { z } from "zod"; @@ -404,7 +406,10 @@ export const enhancedOutputSchemas: Readonly> = { string_count: z.int().min(0), }), ), - analyze_function: functionDossierOutput, + analyze_function: analysisDetailOutputSchemaOf( + functionDossierSchema, + nativeFunctionSummaryViewResultSchema, + ), inspect_native_api: evidenceResultOf(nativeApiInspectionResultSchema), trace_feature: literalTraceOutput, trace_call_path: callPathTraceOutput, diff --git a/src/domain/analysisView/analysisView.test.ts b/src/domain/analysisView/analysisView.test.ts index 7c9e569a80..b16ca725ce 100644 --- a/src/domain/analysisView/analysisView.test.ts +++ b/src/domain/analysisView/analysisView.test.ts @@ -29,6 +29,7 @@ import { sourceMapReference } from "../javascript/javascriptApplicationGraph.fix import { createArtifactInspection } from "../artifactInspection.js"; import { javascriptModulePaths } from "./javascriptApplicationView.js"; import { + analysisViewResultSchema, inspectAnalysisViewInputSchema, projectAnalysisView, type AnalysisViewParent, @@ -100,6 +101,200 @@ it("projects native procedure and bounded assembly and high-pcode with exact par expect(flow.value).toMatchObject({ item: [{ opcode: "COPY" }] }); }); +it("projects a compact analyze_function summary that validates and retains the dossier", () => { + const parent = nativeParent(); + const summary = projectAnalysisView(parent, { kind: "summary" }); + if (!summary.ok) throw summary.error; + const validated = analysisViewResultSchema.parse(summary.value); + expect(validated).toMatchObject({ + kind: "summary", + parent_operation: "analyze_function", + parent_evidence_id: parent.evidenceId, + artifact: parent.artifact, + summary: { + procedure: { address: "0x401000" }, + pseudocode: { available: true }, + value_flow: { available: true }, + facet_availability: { references: { available: true } }, + counts: { assembly: 2, incoming_references: 0 }, + }, + }); +}); + +it("keeps omitted optional dossier records unknown in the analyze_function summary", () => { + const dossier = functionDossierSchema.parse(ghidraFunctionDossier()); + // The fixture omits unresolved_calls and Swift literals, as IDA and Kuna + // dossiers and non-Swift targets do: their counts are unknown, not zero. + const omitted = projectAnalysisView(nativeParent(dossier), { + kind: "summary", + }); + if (!omitted.ok) throw omitted.error; + expect(analysisViewResultSchema.parse(omitted.value)).toMatchObject({ + summary: { + counts: { + unresolved_calls: null, + swift_string_literals: null, + swift_string_literal_omissions: null, + }, + }, + unknowns: expect.arrayContaining([ + "unresolved_calls not recorded", + "swift_string_literals not recorded", + "swift_string_literal_omissions not recorded", + ]), + }); + const recorded = projectAnalysisView( + nativeParent({ + ...dossier, + unresolved_calls: [{ address: "0x401010", reason: "indirect call" }], + swift_string_literals: [], + swift_string_literal_omissions: [], + }), + { kind: "summary" }, + ); + if (!recorded.ok) throw recorded.error; + expect(analysisViewResultSchema.parse(recorded.value)).toMatchObject({ + summary: { + counts: { + unresolved_calls: 1, + swift_string_literals: 0, + swift_string_literal_omissions: 0, + }, + }, + }); + expect(recorded.value.unknowns).not.toContain( + "unresolved_calls not recorded", + ); + expect(recorded.value.unknowns).not.toContain( + "swift_string_literals not recorded", + ); +}); + +it("keeps value-flow counts unknown when value flow is unrecorded or unavailable", () => { + const dossier = functionDossierSchema.parse(ghidraFunctionDossier()); + const flowCounts = { + value_flow_operations: null, + value_flow_def_use: null, + value_flow_effects: null, + value_flow_parameters: null, + value_flow_parameter_uses: null, + }; + // IDA and Kuna dossiers record no value flow at all. + const unrecorded = projectAnalysisView( + nativeParent({ ...dossier, native_value_flow: null }), + { kind: "summary" }, + ); + if (!unrecorded.ok) throw unrecorded.error; + expect(analysisViewResultSchema.parse(unrecorded.value)).toMatchObject({ + summary: { + value_flow: { available: false, truncated: false }, + counts: flowCounts, + }, + unknowns: expect.arrayContaining(["native_value_flow was not recorded"]), + }); + // A Ghidra dossier can declare value flow unavailable with a reason. + const unavailable = projectAnalysisView( + nativeParent({ + ...dossier, + native_value_flow: { + available: false, + reason: "High p-code was not produced.", + limitations: [], + }, + }), + { kind: "summary" }, + ); + if (!unavailable.ok) throw unavailable.error; + expect(analysisViewResultSchema.parse(unavailable.value)).toMatchObject({ + summary: { value_flow: { available: false }, counts: flowCounts }, + unknowns: expect.arrayContaining([ + "native_value_flow unavailable: High p-code was not produced.", + ]), + }); + // Available value flow keeps its observed counts, including empty ones. + const available = projectAnalysisView(nativeParent(dossier), { + kind: "summary", + }); + if (!available.ok) throw available.error; + const flow = dossier.native_value_flow; + if (flow === null || !flow.available) throw new Error("fixture value flow"); + expect(available.value).toMatchObject({ + summary: { + value_flow: { available: true }, + counts: { + value_flow_operations: flow.operations.length, + value_flow_def_use: flow.def_use.length, + value_flow_effects: flow.effects.length, + value_flow_parameters: flow.parameters.length, + value_flow_parameter_uses: flow.parameter_uses.length, + }, + }, + }); + expect(flow.operations.length).toBeGreaterThan(0); + expect(available.value.unknowns).not.toContain( + "native_value_flow was not recorded", + ); +}); + +it("keeps provider-unavailable facets unknown in the analyze_function summary", () => { + // A Kuna-style dossier: empty rows with explicit unavailability for several facets. + const dossier = functionDossierSchema.parse(ghidraFunctionDossier()); + const unsupported = { + ...dossier, + comments: [], + incoming_references: [], + outgoing_references: [], + basic_blocks: [], + facet_availability: { + ...dossier.facet_availability, + comments: { + available: false, + reason: "Kuna exports no analyst comments.", + }, + references: { + available: false, + reason: "This provider does not report typed references.", + }, + cfg: { available: false, reason: "Kuna exports no basic blocks." }, + }, + }; + const summary = projectAnalysisView(nativeParent(unsupported), { + kind: "summary", + }); + if (!summary.ok) throw summary.error; + const validated = analysisViewResultSchema.parse(summary.value); + expect(validated).toMatchObject({ + kind: "summary", + summary: { + facet_availability: unsupported.facet_availability, + counts: { + comments: null, + incoming_references: null, + outgoing_references: null, + basic_blocks: null, + // Declared-available facets keep their observed counts. + assembly: 2, + callers: 0, + referenced_strings: dossier.referenced_strings.length, + }, + }, + unknowns: expect.arrayContaining([ + "comments unavailable: Kuna exports no analyst comments.", + "references unavailable: This provider does not report typed references.", + "cfg unavailable: Kuna exports no basic blocks.", + ]), + }); + // A dossier that declares nothing reports null availability and plain counts. + const { facet_availability: _declared, ...undeclared } = dossier; + const silent = projectAnalysisView(nativeParent(undeclared), { + kind: "summary", + }); + if (!silent.ok) throw silent.error; + expect(silent.value).toMatchObject({ + summary: { facet_availability: null, counts: { comments: 0 } }, + }); +}); + it("rejects invalid native parents, unsupported source operations and singleton offsets", () => { const view = { kind: "native", diff --git a/src/domain/analysisView/analysisView.ts b/src/domain/analysisView/analysisView.ts index c58952cc1e..6e945091fb 100644 --- a/src/domain/analysisView/analysisView.ts +++ b/src/domain/analysisView/analysisView.ts @@ -137,7 +137,10 @@ import { safeParseOwnedBinaryLayout, } from "../native/binaryLayout.js"; import { bundleResourceRootSchema } from "../apple/bundleResourceRoot.js"; -import { functionDossierSchema } from "../hopperValues.js"; +import { + functionDossierSchema, + functionFacetAvailabilitySchema, +} from "../hopperValues.js"; import { artifactGraphManifestSchema, artifactInventoryCoverageSchema, @@ -388,8 +391,13 @@ export const analysisViewSourceSchema = z.discriminatedUnion("kind", [ z.strictObject({ kind: z.literal("evidence-file"), path: absoluteLocalPathStringSchema.describe( - "Absolute local path of one completed Evidence JSON file. REA reads and authenticates it, then projects it like retained Evidence, so large captures are not charged as inline input.", + "Absolute local path of one completed Evidence JSON file, or of an Evidence bundle written by --evidence_output or export_evidence_bundle. REA reads and authenticates it, then projects it like retained Evidence, so large captures are not charged as inline input.", ), + evidence_id: prefixedDigestSchema("ev") + .exactOptional() + .describe( + "The record to project when the file is an Evidence bundle, such as a summary's normalized_result.parent_evidence_id. A bundle holding exactly one record needs no selection.", + ), }), ]); @@ -760,6 +768,99 @@ export const nativeFunctionMatchingSummaryViewResultSchema = z.strictObject({ summary: nativeFunctionMatchingViewContextSchema, }); +/** Rows observed for one facet, or null when the provider declared that facet unavailable. */ +const declaredFacetCountSchema = z + .number() + .int() + .nonnegative() + .nullable() + .describe( + "Observed rows; null when facet_availability declares the facet unavailable, so no rows is not evidence of none.", + ); + +/** Rows of an optional dossier record, or null when the dossier omitted it. */ +const recordedFacetCountSchema = z + .number() + .int() + .nonnegative() + .nullable() + .describe( + "Observed rows; null when the dossier did not record the facet (the native facet view reports it not recorded), so no rows is not evidence of none.", + ); + +/** Rows of one value-flow facet, or null when value flow is unavailable. */ +const valueFlowFacetCountSchema = z + .number() + .int() + .nonnegative() + .nullable() + .describe( + "Observed rows; null when value_flow.available is false because native_value_flow was not recorded or the provider declared it unavailable (unknowns names which), so no rows is not evidence of none.", + ); + +/** Compact analyze_function dossier; the complete dossier stays retained for facet paging. */ +export const nativeFunctionSummarySchema = z.strictObject({ + procedure: z.strictObject({ + name: z.string(), + address: z.string(), + }), + pseudocode: z.strictObject({ + available: z + .boolean() + .describe( + "Provider facet_availability for pseudocode when declared; otherwise whether decompiled text is present. The flag is authoritative over raw length.", + ), + code_units: z + .number() + .int() + .nonnegative() + .describe( + "Pseudocode length in UTF-16 code units, the unit inspect_analysis_view pages pseudocode by.", + ), + }), + value_flow: z.strictObject({ + available: z.boolean(), + truncated: z + .boolean() + .describe( + "True only when value-flow is available and the provider truncated it; false when value-flow is unavailable (see available).", + ), + }), + facet_availability: functionFacetAvailabilitySchema + .nullable() + .describe( + "The provider's declared availability per facet, or null when it declared none. Unavailable facets name their reason here and in unknowns.", + ), + counts: z.strictObject({ + assembly: declaredFacetCountSchema, + basic_blocks: declaredFacetCountSchema, + comments: declaredFacetCountSchema, + callers: declaredFacetCountSchema, + callees: declaredFacetCountSchema, + incoming_references: declaredFacetCountSchema, + outgoing_references: declaredFacetCountSchema, + unresolved_calls: recordedFacetCountSchema, + referenced_strings: declaredFacetCountSchema, + referenced_names: declaredFacetCountSchema, + swift_string_literals: recordedFacetCountSchema, + swift_string_literal_omissions: recordedFacetCountSchema, + value_flow_operations: valueFlowFacetCountSchema, + value_flow_def_use: valueFlowFacetCountSchema, + value_flow_effects: valueFlowFacetCountSchema, + value_flow_parameters: valueFlowFacetCountSchema, + value_flow_parameter_uses: valueFlowFacetCountSchema, + }), + limitation_count: z.number().int().nonnegative(), +}); + +/** Compact analyze_function summary that retains the complete native dossier Evidence. */ +export const nativeFunctionSummaryViewResultSchema = z.strictObject({ + ...summaryViewResultBase, + parent_operation: z.literal("analyze_function"), + procedure_address: z.string().nullable(), + summary: nativeFunctionSummarySchema, +}); + export const sourceToBundleSummaryViewResultSchema = z.strictObject({ ...summaryViewResultBase, artifact: z.null(), @@ -848,6 +949,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [ gpuCodeObjectsSummaryViewResultSchema, nativeInventorySummaryViewResultSchema, nativeFunctionMatchingSummaryViewResultSchema, + nativeFunctionSummaryViewResultSchema, sourceToBundleSummaryViewResultSchema, ]), z.strictObject({ diff --git a/src/domain/analysisView/nativeFunctionView.ts b/src/domain/analysisView/nativeFunctionView.ts index a3ccd02a37..0ff8a98222 100644 --- a/src/domain/analysisView/nativeFunctionView.ts +++ b/src/domain/analysisView/nativeFunctionView.ts @@ -89,14 +89,139 @@ const nativeRows = ( }[facet]; }; +type FacetAvailability = NonNullable; +type FacetAvailabilityKey = keyof FacetAvailability; + +/** Row facets whose provider availability is declared under another key. */ +const facetAvailabilityKeys: Readonly< + Partial> +> = { + incoming_references: "references", + outgoing_references: "references", + referenced_strings: "strings_names", + referenced_names: "strings_names", + basic_blocks: "cfg", +}; + +const facetAvailability = ( + dossier: FunctionDossier, + facet: string, +): FacetAvailability[FacetAvailabilityKey] | undefined => + dossier.facet_availability?.[ + facetAvailabilityKeys[facet] ?? (facet as FacetAvailabilityKey) + ]; + +/** Reasons the provider declared row facets unavailable; empty rows there are not observations. */ +const unavailableFacetUnknowns = (dossier: FunctionDossier): string[] => { + const availability = dossier.facet_availability; + if (availability === undefined) return []; + return Object.entries(availability).flatMap(([facet, status]) => + status.available ? [] : [`${facet} unavailable: ${status.reason}`], + ); +}; + +/** Optional dossier records; the native facet path reports them "not recorded" when omitted. */ +const optionalFacets = [ + "unresolved_calls", + "swift_string_literals", + "swift_string_literal_omissions", +] as const; + +/** Omitted optional records are unknown, not empty, so the summary names them. */ +const unrecordedFacetUnknowns = (dossier: FunctionDossier): string[] => + optionalFacets.flatMap((facet) => + dossier[facet] === undefined ? [`${facet} not recorded`] : [], + ); + +/** Compact dossier summary: identity, pseudocode size, facet counts and value-flow availability. */ +const summarizeDossier = (dossier: FunctionDossier) => { + const sourceFlow = dossier.native_value_flow; + const flow = sourceFlow?.available ? sourceFlow : undefined; + // A count is unknown, not zero, when the provider declared its facet unavailable. + const count = (facet: string, rows: readonly unknown[]): number | null => + facetAvailability(dossier, facet)?.available === false ? null : rows.length; + // An omitted optional record is unknown, not zero. + const recorded = (rows: readonly unknown[] | undefined): number | null => + rows === undefined ? null : rows.length; + return { + procedure: { + name: dossier.procedure.name, + address: dossier.procedure.address, + }, + pseudocode: { + available: + dossier.facet_availability?.pseudocode?.available ?? + dossier.pseudocode.length > 0, + code_units: dossier.pseudocode.length, + }, + value_flow: { + available: flow !== undefined, + truncated: flow?.truncated ?? false, + }, + facet_availability: dossier.facet_availability ?? null, + counts: { + assembly: count("assembly", dossier.assembly), + basic_blocks: count("basic_blocks", dossier.basic_blocks), + comments: count("comments", dossier.comments), + callers: count("callers", dossier.callers), + callees: count("callees", dossier.callees), + incoming_references: count( + "incoming_references", + dossier.incoming_references, + ), + outgoing_references: count( + "outgoing_references", + dossier.outgoing_references, + ), + unresolved_calls: recorded(dossier.unresolved_calls), + referenced_strings: count( + "referenced_strings", + dossier.referenced_strings, + ), + referenced_names: count("referenced_names", dossier.referenced_names), + swift_string_literals: recorded(dossier.swift_string_literals), + swift_string_literal_omissions: recorded( + dossier.swift_string_literal_omissions, + ), + // Value-flow rows are unknown, not zero, when value flow was not recorded + // or the provider declared it unavailable; parentFields names which. + value_flow_operations: flow?.operations.length ?? null, + value_flow_def_use: flow?.def_use.length ?? null, + value_flow_effects: flow?.effects.length ?? null, + value_flow_parameters: flow?.parameters.length ?? null, + value_flow_parameter_uses: flow?.parameter_uses.length ?? null, + }, + limitation_count: dossier.limitations.length, + }; +}; + +const summaryView = ( + view: Extract, + dossier: FunctionDossier, + common: ReturnType, +): UnsignedAnalysisView => ({ + kind: "summary", + view, + summary: summarizeDossier(dossier), + coverage: completeWithinViewCoverage(1, 1), + ...common, + unknowns: uniqueSorted([ + ...common.unknowns, + ...unavailableFacetUnknowns(dossier), + ...unrecordedFacetUnknowns(dossier), + ]), +}); + /** Project bounded native function facets from complete retained Evidence without reanalysis. */ export const projectNativeFunctionView = ( parent: AnalysisViewParent, dossier: FunctionDossier, view: AnalysisViewRequest, ): Result => { - if (view.kind !== "native") - return err(viewError("analyze_function Evidence requires a native view.")); + if (view.kind !== "native" && view.kind !== "summary") + return err( + viewError("analyze_function Evidence requires a native or summary view."), + ); const artifact = parent.artifact; if (!artifact) return err( @@ -110,20 +235,8 @@ export const projectNativeFunctionView = ( ]), ); const common = parentFields(parent, dossier, artifact); - const availabilityKeys: Readonly> = { - incoming_references: "references", - outgoing_references: "references", - referenced_strings: "strings_names", - referenced_names: "strings_names", - basic_blocks: "cfg", - }; - const availabilityKey = availabilityKeys[view.facet] ?? view.facet; - const availability = - dossier.facet_availability?.[ - availabilityKey as keyof NonNullable< - FunctionDossier["facet_availability"] - > - ]; + if (view.kind === "summary") return ok(summaryView(view, dossier, common)); + const availability = facetAvailability(dossier, view.facet); if (availability?.available === false) common.unknowns = uniqueSorted([...common.unknowns, availability.reason]); const singleton = (item: unknown): UnsignedAnalysisView => ({ diff --git a/src/domain/canonicalDigest.ts b/src/domain/canonicalDigest.ts index 9e24aa5367..5e7467cc95 100644 --- a/src/domain/canonicalDigest.ts +++ b/src/domain/canonicalDigest.ts @@ -11,19 +11,28 @@ export const digestCanonicalValue = ( context = "Comparison", ): string => { const hash = createHash("sha256"); - // Batch small emitted parts to cut hash.update crossings. - // Flush cuts only at part boundaries, so the hashed byte stream is unchanged. + // Batch small emitted parts to cut hash.update crossings. A large part is + // hashed as is: appending it to the buffer would hold a second copy of it + // while the concatenation flattens. Flush cuts only at part boundaries, so + // the hashed byte stream is unchanged. let buffered = ""; + const flush = (): void => { + if (buffered.length === 0) return; + hash.update(buffered); + buffered = ""; + }; const emit = (part: string): void => { - buffered += part; - if (buffered.length >= 8192) { - hash.update(buffered); - buffered = ""; + if (part.length >= 8192) { + flush(); + hash.update(part); + return; } + buffered += part; + if (buffered.length >= 8192) flush(); }; const encoded = emitCanonical(value, emit, new Set()); if (!encoded) throw new TypeError(`${context} could not canonicalize data`); - if (buffered) hash.update(buffered); + flush(); return hash.digest("hex"); }; diff --git a/src/domain/evidenceSteps.test.ts b/src/domain/evidenceSteps.test.ts index 1eadacc008..c8105d54f8 100644 --- a/src/domain/evidenceSteps.test.ts +++ b/src/domain/evidenceSteps.test.ts @@ -99,9 +99,10 @@ describe("cooperative immutable Evidence", () => { let nested: JsonValue = 1; for (let index = 0; index <= MAX_JSON_DEPTH; index += 1) nested = { nested }; + // Enough members before the over-deep one that validation yields first. const result = freezeJsonSnapshot([ - nested, ...Array(5000).fill(null), + nested, ]); const validation = jsonValueValidationSteps(result); expect(validation.next().done).toBe(false); diff --git a/src/domain/hopperValues.ts b/src/domain/hopperValues.ts index dd31eda95b..b50328e51d 100644 --- a/src/domain/hopperValues.ts +++ b/src/domain/hopperValues.ts @@ -370,6 +370,20 @@ const facetAvailabilitySchema = z.discriminatedUnion("available", [ z.object({ available: z.literal(false), reason: z.string().min(1) }).strict(), ]); +/** Provider-declared availability per dossier facet; an omitted facet declares nothing. */ +export const functionFacetAvailabilitySchema = z + .object({ + pseudocode: facetAvailabilitySchema.exactOptional(), + assembly: facetAvailabilitySchema.exactOptional(), + comments: facetAvailabilitySchema.exactOptional(), + callers: facetAvailabilitySchema.exactOptional(), + callees: facetAvailabilitySchema.exactOptional(), + references: facetAvailabilitySchema.exactOptional(), + strings_names: facetAvailabilitySchema.exactOptional(), + cfg: facetAvailabilitySchema.exactOptional(), + }) + .strict(); + export const functionDossierSchema = z .object({ procedure: procedureIdentitySchema.safeExtend({ @@ -427,19 +441,7 @@ export const functionDossierSchema = z ), native_api: nativeApiBoundarySchema.nullable(), native_value_flow: nativeValueFlowSchema.nullable(), - facet_availability: z - .object({ - pseudocode: facetAvailabilitySchema.exactOptional(), - assembly: facetAvailabilitySchema.exactOptional(), - comments: facetAvailabilitySchema.exactOptional(), - callers: facetAvailabilitySchema.exactOptional(), - callees: facetAvailabilitySchema.exactOptional(), - references: facetAvailabilitySchema.exactOptional(), - strings_names: facetAvailabilitySchema.exactOptional(), - cfg: facetAvailabilitySchema.exactOptional(), - }) - .strict() - .exactOptional(), + facet_availability: functionFacetAvailabilitySchema.exactOptional(), limitations: z.array(z.string()), }) .strict(); diff --git a/src/domain/jsonValue.test.ts b/src/domain/jsonValue.test.ts index 727ff85cfe..cc6050ea56 100644 --- a/src/domain/jsonValue.test.ts +++ b/src/domain/jsonValue.test.ts @@ -5,6 +5,8 @@ import { z } from "zod"; import { jsonObjectSchema, jsonValueSchema, + jsonValueValidationIssue, + jsonValueValidationSteps, MAX_JSON_DEPTH, } from "./jsonValue.js"; @@ -113,6 +115,59 @@ describe("jsonValueSchema depth bound", () => { }); }); +describe("jsonValueSchema width", () => { + const WIDTH = 300_000; + + it("validates wide containers member by member, including a late defect", () => { + const wide = Array.from({ length: WIDTH }, (_, index) => index); + const keyed = Object.fromEntries( + wide.slice(0, 50_000).map((index) => [`k${String(index)}`, index]), + ); + expect(jsonValueValidationIssue({ wide, keyed })).toBeUndefined(); + const holed = Array.from({ length: WIDTH }, (_, index) => index); + delete holed[WIDTH - 1]; + expect(jsonValueValidationIssue(holed)).toBe( + "JSON arrays require dense data properties without accessors", + ); + const lateAccessor: Record = { ...keyed }; + Object.defineProperty(lateAccessor, "zz", { + enumerable: true, + get: () => 1, + }); + expect(jsonValueValidationIssue(lateAccessor)).toBe( + "JSON objects cannot contain accessors", + ); + }); + + it("reports the first defect in member order, descending before moving on", () => { + const holed: unknown[] = [[Number.NaN], 1, 2]; + delete holed[2]; + expect(jsonValueValidationIssue(holed)).toBe("JSON numbers must be finite"); + expect(jsonValueValidationIssue([Number.NaN, () => 1])).toBe( + "JSON numbers must be finite", + ); + expect(jsonValueValidationIssue([() => 1, Number.NaN])).toBe( + "Invalid JSON value", + ); + }); + + it("still yields cooperatively while walking a wide array", () => { + const steps = jsonValueValidationSteps( + Array.from({ length: WIDTH }, () => true), + ); + let yields = 0; + for (;;) { + const step = steps.next(); + if (step.done) { + expect(step.value).toBeUndefined(); + break; + } + yields += 1; + } + expect(yields).toBeGreaterThanOrEqual(Math.floor(WIDTH / 4097) - 1); + }); +}); + it("preserves prototype-named members without prototype mutation", () => { const input: unknown = JSON.parse( '{"__proto__":{"preserved":7},"constructor":"ordinary","prototype":[1],"nested":{"\\u005f\\u005fproto\\u005f\\u005f":"escaped"},"list":[{"__proto__":null}]}', diff --git a/src/domain/jsonValue.ts b/src/domain/jsonValue.ts index ed464cea01..c1629837ab 100644 --- a/src/domain/jsonValue.ts +++ b/src/domain/jsonValue.ts @@ -90,7 +90,7 @@ export function* jsonValueValidationSteps( kind: "array", value, depth, - index: value.length - 1, + index: 0, }); inspect = false; } else if (isRecord(value)) { @@ -114,11 +114,11 @@ export function* jsonValueValidationSteps( const frame = pending[pending.length - 1]; if (frame === undefined) return finish(undefined); if (frame.kind === "array") { - if (frame.index < 0) { + if (frame.index >= frame.value.length) { pending.pop(); continue; } - const index = frame.index--; + const index = frame.index++; const descriptor = Object.getOwnPropertyDescriptor(frame.value, index); if (descriptor === undefined || !Object.hasOwn(descriptor, "value")) return finish( diff --git a/src/server/registerEnhancedTools.ts b/src/server/registerEnhancedTools.ts index 2b3f29cbf4..33de31ef73 100644 --- a/src/server/registerEnhancedTools.ts +++ b/src/server/registerEnhancedTools.ts @@ -32,6 +32,11 @@ import { logToolExecution } from "./toolLogging.js"; import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; import { executeFunctionAnalysisEvidence } from "../application/FunctionAnalysisEvidence.js"; import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; +import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; +import { + requireAnalysisViewRetention, + summarizeRetainedAnalysis, +} from "../application/analysisView/AnalysisViewService.js"; import { reportOperationStart, reportOperationEnd, @@ -48,6 +53,7 @@ export interface EnhancedToolRegistration { ) => AnalysisProfileCommitment | undefined) | undefined; readonly recordEvidence: EvidenceWriter["recordEvidence"] | undefined; + readonly evidenceById?: EvidenceLookup | undefined; readonly recordUnknown: UnknownRegistryPort["recordUnknown"] | undefined; readonly allowsSnapshotReplay: | ((operation: ValidatedEnhancedCall["name"]) => boolean) @@ -176,9 +182,10 @@ export const registerEnhancedTools = ( server.registerTool( analyzeFunction.name, toolRegistrationOptions(analyzeFunction), - (input, context) => - executeEnhancedTool(registration, analyzeFunction, { - validatedCall: { name: "analyze_function", input }, + ({ detail, ...providerInput }, context) => + executeFunctionTool(registration, analyzeFunction, { + detail, + parameters: jsonParameters(providerInput), context, }), ); @@ -240,12 +247,6 @@ const executeEnhancedTool = async ( await reportOperationStart(progress, name); const parsedInput = validatedCall.input; const parameters = jsonParameters(parsedInput); - if (name === "analyze_function") - return executeFunctionTool(admittedAnalysis, registration, contract, { - context, - parameters, - progress, - }); const analysis: AnalysisOperationPort = { resourceScope: admittedAnalysis.resourceScope, execute: (operation, operationParameters, executionOptions) => @@ -317,21 +318,62 @@ const executeEnhancedTool = async ( : registration.delivery.toCallToolResult(admitted, contract); }; +/** + * analyze_function owns its MCP delivery: detail selects the response shape + * and never reaches the provider, so parameters hold the provider input only. + */ const executeFunctionTool = async ( + registration: EnhancedToolRegistration & { + readonly delivery: ToolResultDelivery; + }, + contract: ToolContract, + request: { + readonly detail: "complete" | "summary"; + readonly parameters: Readonly>; + readonly context: ServerContext; + }, +): Promise => { + const { detail, context } = request; + // Fail before analysis when a retained summary cannot keep its complete source. + if (detail === "summary") { + const ready = requireAnalysisViewRetention("analyze_function", { + recordEvidence: registration.recordEvidence, + evidenceById: registration.evidenceById, + }); + if (!ready.ok) + return registration.delivery.toCallToolResult(ready, contract); + } + const admitted = await registration.withAdmittedAnalysis( + "analyze_function", + context.mcpReq.signal, + async (analysis) => { + const progress = mcpProgressReporter(context); + await reportOperationStart(progress, "analyze_function"); + return deliverFunctionAnalysis(analysis, registration, contract, { + ...request, + progress, + }); + }, + ); + return admitted.ok + ? admitted.value + : registration.delivery.toCallToolResult(admitted, contract); +}; + +const deliverFunctionAnalysis = async ( analysis: AnalysisOperationPort, registration: EnhancedToolRegistration & { readonly delivery: ToolResultDelivery; }, contract: ToolContract, request: { - readonly context: ServerContext; + readonly detail: "complete" | "summary"; readonly parameters: Readonly>; + readonly context: ServerContext; readonly progress: ProgressReporter; }, ): Promise => { - // Reuse the request reporter that already sent "started"; a second reporter - // would restart the notification sequence at 1. - const { context, parameters, progress } = request; + const { detail, parameters, context, progress } = request; const result = await logToolExecution( registration.logger, "analyze_function", @@ -349,6 +391,19 @@ const executeFunctionTool = async ( await reportOperationEnd(progress, "analyze_function", result.ok); if (!result.ok) return registration.delivery.toCallToolResult(result, contract); + if (detail === "summary") { + const summary = summarizeRetainedAnalysis(result.value, { + recordEvidence: registration.recordEvidence, + evidenceById: registration.evidenceById, + }); + if (!summary.ok) + return registration.delivery.toCallToolResult(summary, contract); + return registration.delivery.toEvidenceToolResult( + summary.value, + contract, + registration.recordEvidence?.(summary.value), + ); + } return registration.delivery.toEvidenceToolResult( result.value, contract, diff --git a/tests/boundary/cli/analyzeFunctionDetail.test.ts b/tests/boundary/cli/analyzeFunctionDetail.test.ts new file mode 100644 index 0000000000..16d3e9891a --- /dev/null +++ b/tests/boundary/cli/analyzeFunctionDetail.test.ts @@ -0,0 +1,113 @@ +import { Cli } from "incur"; +import pino from "pino"; +import { stat } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, it, vi } from "vitest"; + +import { inspectAnalysisView } from "../../../src/application/analysisView/AnalysisViewService.js"; +import { registerCoreBinaryCommands } from "../../../src/cli/coreBinaryCommands.js"; +import type { CliInstance } from "../../../src/cli/types.js"; +import { CLI_COMMANDS } from "../../../src/cliCommandNames.js"; +import { createEvidence } from "../../../src/domain/evidence.js"; +import { functionDossierSchema } from "../../../src/domain/hopperValues.js"; +import { ghidraFunctionDossier } from "../../../src/domain/ghidraValues.fixture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +// A complete analyze_function Evidence as runDirectAnalysis would return it: +// subject-bound (so the summary view has an artifact) with a schema-valid dossier. +const cannedCompleteEvidence = () => + createEvidence( + { path: "/fixtures/native.exe", format: "pe", sha256: "a".repeat(64) }, + { id: "ghidra", name: "Ghidra", version: "12.1.4" }, + { + operation: "analyze_function", + parameters: { procedure: "main" }, + result: functionDossierSchema.parse(ghidraFunctionDossier()), + rawResult: null, + }, + ); + +const harness = () => { + const runDirectAnalysis = vi.fn( + async () => cannedCompleteEvidence() as unknown, + ); + const cli = Cli.create("rea-fn-detail-test", { version: "1", sync: false }); + registerCoreBinaryCommands( + cli as unknown as CliInstance, + pino({ enabled: false }), + runDirectAnalysis as never, + {}, + ); + const command = Cli.toCommands.get(cli)?.get(CLI_COMMANDS.function); + if (command === undefined) + throw new Error("function command was not registered"); + const argsSchema = Reflect.get(command, "args"); + const optionsSchema = Reflect.get(command, "options"); + const execute = Reflect.get(command, "run"); + if (typeof execute !== "function") + throw new Error("function command has no run callback"); + const invoke = (options: Record): Promise => + execute({ + args: (argsSchema as { parse(v: unknown): unknown }).parse({ + path: "/fixtures/native.exe", + address: "main", + }), + options: (optionsSchema as { parse(v: unknown): unknown }).parse(options), + }); + return { runDirectAnalysis, invoke }; +}; + +it("defaults to the complete dossier and runs analysis", async () => { + const h = harness(); + const out = (await h.invoke({})) as { operation?: string }; + expect(h.runDirectAnalysis).toHaveBeenCalledOnce(); + expect(out.operation).toBe("analyze_function"); + expect( + (out as { normalized_result?: unknown }).normalized_result, + ).toBeDefined(); +}); + +it("rejects summary delivery without --evidence_output before any analysis", async () => { + const h = harness(); + const out = await h.invoke({ detail: "summary" }); + expect(h.runDirectAnalysis).not.toHaveBeenCalled(); + expect(JSON.stringify(out)).toContain("evidence_output"); +}); + +it("summary delivery returns the compact view and writes a bundle inspect-analysis-view pages", async () => { + const h = harness(); + const directory = await createTestTempDirectory("rea-fn-detail-"); + const destination = join(directory, "complete.json"); + const out = (await h.invoke({ + detail: "summary", + evidence_output: destination, + })) as { + summary: { + normalized_result: { kind: string; parent_evidence_id: string }; + }; + evidence_bundle: { path: string; bytes: number }; + }; + expect(h.runDirectAnalysis).toHaveBeenCalledOnce(); + expect(out.summary.normalized_result.kind).toBe("summary"); + expect(typeof out.summary.normalized_result.parent_evidence_id).toBe( + "string", + ); + expect(out.evidence_bundle.bytes).toBeGreaterThan(0); + expect((await stat(destination)).size).toBeGreaterThan(0); + // The saved bundle is the evidence-file source inspect-analysis-view reads back. + const parentId = out.summary.normalized_result.parent_evidence_id; + const page = await inspectAnalysisView({ + source: { kind: "evidence-file", path: destination, evidence_id: parentId }, + view: { kind: "native", facet: "pseudocode", offset: 0, limit: 16 }, + }); + if (!page.ok) throw page.error; + expect(page.value.parameters).toMatchObject({ parent_evidence_id: parentId }); + expect(page.value.normalized_result).toMatchObject({ + kind: "native", + item: { + text: functionDossierSchema + .parse(ghidraFunctionDossier()) + .pseudocode.slice(0, 16), + }, + }); +}); diff --git a/tests/boundary/mcp/analysisViewMcp.test.ts b/tests/boundary/mcp/analysisViewMcp.test.ts index f5562b06cd..dc47c5f394 100644 --- a/tests/boundary/mcp/analysisViewMcp.test.ts +++ b/tests/boundary/mcp/analysisViewMcp.test.ts @@ -294,22 +294,24 @@ it("discovers an oversized native dossier by exact metadata and reads its retain const retained = session.recordEvidence(record); if (!retained.ok) throw retained.error; } - const unsupportedSummary = await client.callTool({ + // The summary stays compact however large the retained pseudocode is. + const summary = await client.callTool({ name: "inspect_analysis_view", arguments: { source: { kind: "retained-evidence", evidence_id: parent.evidence_id }, view: { kind: "summary" }, }, }); - expect(parseMcpToolError(unsupportedSummary).error).toMatchObject({ - code: "invalid_request", - details: { - issues: [ - { - path: ["view"], - message: "analyze_function Evidence requires a native view.", - }, - ], + expect(summary.isError, JSON.stringify(summary.content)).not.toBe(true); + expect( + toolContract("inspect_analysis_view").outputSchema.parse( + summary.structuredContent, + ).normalized_result, + ).toMatchObject({ + kind: "summary", + parent_evidence_id: parent.evidence_id, + summary: { + pseudocode: { available: true, code_units: 6 * 1024 * 1024 }, }, }); const broad = await client.callTool({ @@ -340,9 +342,10 @@ it("discovers an oversized native dossier by exact metadata and reads its retain const result = toolContract("get_evidence_bundle").outputSchema.parse( response.structuredContent, ).result; + // Two analyze_function records plus the retained summary view. expect(result).toMatchObject({ kind: "evidence-bundle-summary", - total_retained_records: 2, + total_retained_records: 3, matching_records: 1, records: [ { @@ -958,9 +961,10 @@ it("reads successive native source pages directly from a completed-result transp ); onTestFinished(() => harness.close()); const { mcp: client, session, calls } = harness; + // Complete delivery of an oversized dossier fails transport and points at the retained parent. const failed = await client.callTool({ name: "analyze_function", - arguments: { procedure: "0x401000" }, + arguments: { procedure: "0x401000", detail: "complete" }, }); expect(failed.isError).toBe(true); const limits = z diff --git a/tests/boundary/mcp/analyzeFunctionDetail.test.ts b/tests/boundary/mcp/analyzeFunctionDetail.test.ts new file mode 100644 index 0000000000..4cb3632f44 --- /dev/null +++ b/tests/boundary/mcp/analyzeFunctionDetail.test.ts @@ -0,0 +1,189 @@ +import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; +import pino from "pino"; +import { expect, it, onTestFinished, vi } from "vitest"; + +import { EvidenceMcpServer } from "../../../src/server/EvidenceMcpServer.js"; +import { registerEnhancedTools } from "../../../src/server/registerEnhancedTools.js"; +import { createToolResultDelivery } from "../../../src/server/toolResult.js"; +import { withAdmittedAnalysis } from "../../../src/server/analysisAdmission.js"; +import type { AnalysisOperationPort } from "../../../src/application/AnalysisProvider.js"; +import { parseEvidence, type Evidence } from "../../../src/domain/evidence.js"; +import type { JsonValue } from "../../../src/domain/jsonValue.js"; +import { ok } from "../../../src/domain/result.js"; +import { functionDossierSchema } from "../../../src/domain/hopperValues.js"; +import { ghidraFunctionDossier } from "../../../src/domain/ghidraValues.fixture.js"; +import { parseGhidraFunctionInput } from "../../../src/ghidra/GhidraFunctionValues.js"; + +const dossier = functionDossierSchema.parse(ghidraFunctionDossier()); +const partialCallees = { + status: "partial" as const, + limitations: ["Callees: fixture read limit"], + limited_directions: ["callees"] as const, +}; + +// A fixed provider that returns the fixture dossier for any analyze_function +// call, after validating the request exactly as the production Ghidra client does. +const connect = async (retain = true, result = dossier) => { + const ledger = new Map(); + const record = (evidence: Evidence) => { + ledger.set(evidence.evidence_id, evidence); + return ok("added" as const); + }; + const lookup = (id: string) => ledger.get(id); + const execute = vi.fn( + async (operation: string, parameters: Record) => { + if (operation !== "analyze_function") + throw new Error(`Unexpected operation: ${operation}`); + const input = parseGhidraFunctionInput(operation, parameters); + if (!input.ok) return input; + return ok({ + result, + rawResult: null, + provider: { id: "ghidra", name: "Ghidra", version: "12.1.4" }, + limitations: [], + locations: [], + subject: { + path: "/fixtures/native.exe", + format: "pe" as const, + sha256: "a".repeat(64), + }, + relationshipCoverage: partialCallees, + }); + }, + ); + const analysis = { execute } as unknown as AnalysisOperationPort; + const delivery = createToolResultDelivery(4 * 1024 * 1024); + const server = new EvidenceMcpServer( + { name: "fn-detail", version: "1" }, + {}, + retain ? record : undefined, + delivery, + ); + registerEnhancedTools(server, { + withAdmittedAnalysis: withAdmittedAnalysis({ kind: "fixed", analysis }), + logger: pino({ enabled: false }), + activeTarget: () => undefined, + analysisProfile: undefined, + recordEvidence: retain ? record : undefined, + evidenceById: retain ? lookup : undefined, + recordUnknown: undefined, + allowsSnapshotReplay: undefined, + recordWorkflowSnapshot: undefined, + }); + const client = new Client({ name: "fn-detail-client", version: "1" }); + onTestFinished(async () => { + await client.close(); + await server.close(); + }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + await server.connect(serverTransport); + await client.connect(clientTransport); + return { client, ledger, execute }; +}; + +it("defaults to a retained summary and strips delivery detail before the provider", async () => { + const { client, ledger, execute } = await connect(); + const progress: number[] = []; + const reply = await client.callTool( + { name: "analyze_function", arguments: { procedure: "main" } }, + { onprogress: (update) => progress.push(update.progress) }, + ); + expect(progress.length).toBeGreaterThanOrEqual(2); + progress.forEach((value, index) => { + expect(value).toBeGreaterThan(progress[index - 1] ?? 0); + }); + const evidence = parseEvidence(reply.structuredContent); + expect(evidence.normalized_result).toMatchObject({ + kind: "summary", + parent_operation: "analyze_function", + }); + const parentId = ( + evidence.normalized_result as { parent_evidence_id: string } + ).parent_evidence_id; + // The complete dossier is retained in-session so the summary's parent can be paged. + expect(ledger.has(parentId)).toBe(true); + // Delivery detail never crosses into the provider request. + const providerParameters = execute.mock.calls[0]?.[1] ?? {}; + expect(providerParameters).not.toHaveProperty("detail"); + expect(providerParameters).toMatchObject({ procedure: "main" }); + // Partial caller/callee discovery stays with the summary that reports their counts. + expect(evidence.relationship_coverage).toEqual(partialCallees); + expect(ledger.get(parentId)?.relationship_coverage).toEqual(partialCallees); + // The summary envelope locates the function it describes. + expect(evidence.locations).toEqual([ + { kind: "artifact-path", path: "/fixtures/native.exe" }, + { kind: "address", address: dossier.procedure.address }, + ]); + // Optional records the fixture omits stay unknown rather than reading as empty. + expect(evidence.normalized_result).toMatchObject({ + summary: { + counts: { unresolved_calls: null, swift_string_literals: null }, + }, + }); +}); + +it("keeps value-flow counts unknown in the default summary when the provider records none", async () => { + // IDA and Kuna dossiers carry native_value_flow: null. + const { client } = await connect(true, { + ...dossier, + native_value_flow: null, + }); + const reply = await client.callTool({ + name: "analyze_function", + arguments: { procedure: "main" }, + }); + const evidence = parseEvidence(reply.structuredContent); + expect(evidence.normalized_result).toMatchObject({ + kind: "summary", + summary: { + value_flow: { available: false, truncated: false }, + counts: { + value_flow_operations: null, + value_flow_def_use: null, + value_flow_effects: null, + value_flow_parameters: null, + value_flow_parameter_uses: null, + }, + }, + unknowns: expect.arrayContaining(["native_value_flow was not recorded"]), + }); +}); + +it("composes inspect_native_api from a provider input without delivery detail", async () => { + const { client, execute } = await connect(); + const reply = await client.callTool({ + name: "inspect_native_api", + arguments: { procedure: "main" }, + }); + expect(reply.isError, JSON.stringify(reply.content)).not.toBe(true); + expect( + parseEvidence(reply.structuredContent).normalized_result, + ).toMatchObject({ procedure: { name: dossier.procedure.name } }); + expect(execute.mock.calls.map((call) => call[1])).toEqual([ + { procedure: "main" }, + ]); +}); + +it("returns the complete dossier inline when detail is complete", async () => { + const { client } = await connect(); + const reply = await client.callTool({ + name: "analyze_function", + arguments: { procedure: "main", detail: "complete" }, + }); + const evidence = parseEvidence(reply.structuredContent); + expect(evidence.normalized_result).toMatchObject({ + pseudocode: dossier.pseudocode, + }); + expect(evidence.normalized_result).toHaveProperty("native_value_flow"); +}); + +it("rejects a summary request before analysis when the server retains no Evidence", async () => { + const { client, execute } = await connect(false); + const reply = await client.callTool({ + name: "analyze_function", + arguments: { procedure: "main" }, + }); + expect(reply.isError).toBe(true); + expect(execute).not.toHaveBeenCalled(); +}); diff --git a/tests/boundary/mcp/enhancedTools.01-lists-the-complete-target-open.test.ts b/tests/boundary/mcp/enhancedTools.01-lists-the-complete-target-open.test.ts index 0de734f2ed..2a954569f5 100644 --- a/tests/boundary/mcp/enhancedTools.01-lists-the-complete-target-open.test.ts +++ b/tests/boundary/mcp/enhancedTools.01-lists-the-complete-target-open.test.ts @@ -275,7 +275,8 @@ describe("enhanced MCP tools", () => { ["analyze_swift_types", { category: "classes", pattern: "Fixture" }], ["find_xrefs_to_name", { name: "entry" }], ["binary_overview", {}], - ["analyze_function", { procedure: "0x1" }], + // This server retains no Evidence, so the complete dossier is the deliverable shape. + ["analyze_function", { procedure: "0x1", detail: "complete" }], ["inspect_native_api", { procedure: "0x1" }], ["trace_feature", { query: "hello" }], ["trace_call_path", { start: "0x1", goal: "0x2" }], diff --git a/tests/boundary/mcp/ghidraEvidenceMcp.test.ts b/tests/boundary/mcp/ghidraEvidenceMcp.test.ts index c249a8b823..610ebba9b7 100644 --- a/tests/boundary/mcp/ghidraEvidenceMcp.test.ts +++ b/tests/boundary/mcp/ghidraEvidenceMcp.test.ts @@ -71,7 +71,7 @@ it.each([ try { const analyzed = await harness.mcp.callTool({ name: "analyze_function", - arguments: { procedure: dossier.procedure.address }, + arguments: { procedure: dossier.procedure.address, detail: "complete" }, }); expect(analyzed.isError).not.toBe(true); const evidence = sessionEvidence( @@ -172,7 +172,7 @@ it("retains terminal-call qualifications across function views and snapshot repl name, arguments: name === "analyze_function" - ? { procedure: identity.address } + ? { procedure: identity.address, detail: "complete" } : { procedure: identity.address, document: "fixture" }, }; const reply = await harness.mcp.callTool(request); @@ -464,7 +464,7 @@ describe("Ghidra MCP evidence parity", () => { ( await mcp.callTool({ name: "analyze_function", - arguments: { procedure: "fixture_main" }, + arguments: { procedure: "fixture_main", detail: "complete" }, }) ).structuredContent, ); diff --git a/tests/boundary/mcp/inlineEvidenceCompositionMcp.test.ts b/tests/boundary/mcp/inlineEvidenceCompositionMcp.test.ts index a5c7dcd8df..2eb232bdc9 100644 --- a/tests/boundary/mcp/inlineEvidenceCompositionMcp.test.ts +++ b/tests/boundary/mcp/inlineEvidenceCompositionMcp.test.ts @@ -12,7 +12,7 @@ describe("inline MCP Evidence composition", () => { try { const response = await mcp.callTool({ name: "analyze_function", - arguments: { procedure: "fixture_main" }, + arguments: { procedure: "fixture_main", detail: "complete" }, }); expect(response.isError).not.toBe(true); const analyzed = toolContract("analyze_function").outputSchema.parse( diff --git a/tests/boundary/mcp/investigationAnalysisMcp.test.ts b/tests/boundary/mcp/investigationAnalysisMcp.test.ts index 5144ec1494..5040c665af 100644 --- a/tests/boundary/mcp/investigationAnalysisMcp.test.ts +++ b/tests/boundary/mcp/investigationAnalysisMcp.test.ts @@ -333,7 +333,7 @@ it.each([ expect(opened.isError, JSON.stringify(opened)).not.toBe(true); const analyzed = await client.callTool({ name: "analyze_function", - arguments: { procedure: "0x1000" }, + arguments: { procedure: "0x1000", detail: "complete" }, }); expect(analyzed.isError, JSON.stringify(analyzed)).not.toBe(true); const dossier = inlineEvidence(analyzed.structuredContent); diff --git a/tests/boundary/mcp/progressMcp.test.ts b/tests/boundary/mcp/progressMcp.test.ts index 9b471d75af..533ec45bec 100644 --- a/tests/boundary/mcp/progressMcp.test.ts +++ b/tests/boundary/mcp/progressMcp.test.ts @@ -88,7 +88,10 @@ describe("ordinary MCP progress and cancellation", () => { await server.connect(serverTransport); await client.connect(clientTransport); const result = await client.callTool( - { name: "analyze_function", arguments: { procedure: "main" } }, + { + name: "analyze_function", + arguments: { procedure: "main", detail: "complete" }, + }, { onprogress: (update) => progress.push(update.progress) }, );