fix(hopper): reconcile observed caller relationships (#7953)

* fix(hopper): reconcile observed caller relationships

* fix(hopper): retain completed caller scans on failure

Attach the native reply and collected forward facts when reconciliation
fails or is cancelled, and check scan stop codes separately from their
wording. Live observations keep REA evidence ids rather than provider
supplied ones, and the real Hopper verifier uses advertised MCP schemas.

* fix(hopper): answer caller-scan methods in the package fixture

The packaged Hopper stand-in echoed unknown bridge methods, so
analyze_function rejected rea_caller_scan_target as an invalid scan
target and the native package baseline returned unreadable_output.

* fix(hopper): share caller identities in failed scan observations

Failure partials repeated each caller identity under every destination.
The page budget counted compact replies, then schema parsing copied the
expanded form. Identities are stored once and destinations cite addresses.

* refactor(config): own the Hopper caller-scan budget default

The environment schema hard-coded the 2000 ms scan default. The domain
limit owner now names that millisecond budget and the schema reuses it.

* test(mcp): assert bundle path issues by field

compare_bundles reports each relative path as an invalid_format issue
instead of one joined diagnostic sentence.

* test(flutter): assert ABI selector refusals as structured issues

inspect_dart_aot reports the ABI label as an issue on path abi. The
message is the selector contract, without an abi: prefix.

* fix(session): retain caller observations that cannot be replayed

A missing document selector blocks snapshot replay. Backward call-graph
and trace reads still keep the native result and forward observations,
and the workflow Evidence links those records.

* fix(session): keep decompile bodies out of the in-memory ledger

Caller and callee walks still stay in Evidence without a document.
procedure_pseudo_code does not, so decompile_all leaves each body in the
file-backed export instead of retaining the whole export in memory.

* fix(hopper): own caller-scan recovery and the page byte allowance

Scan facts stay in the domain. Environment recovery, including the time
budget bounds and the fixed allowances with no runtime override, is
appended from configuration. Setup forwards the budget, and the bridge
uses one byte allowance for instructions, native callers, and pages.

* fix(hopper): retain compact caller references within the scan budget

Recovered call sites stay in the address form the scan already counted.
Expanded identity copies are measured before allocation and refused when
they exceed the retention budget. Instruction lists are not part of the
retained reply.

* fix(session): omit instruction payloads from retained caller evidence

Documentless caller and callee reads stay in the ledger for the session.
Their evidence keeps call sites and drops instruction lists the scan
already consumed.

* fix(hopper): keep the owning caller address on each compact call site

A page reference marked source_procedure "caller" belongs to that page
item's caller. Saved sites record that address, and an owner missing from
the callee inventory is stored once instead of copied onto every site.

* fix(session): leave Hopper reply shaping to the Hopper adapter

Documentless caller evidence stores the adapter's raw result unchanged.
The session no longer reads native_result.instructions.

* test(hopper): throw typed scan failures before reading results

Successful caller-scan results now throw the typed error instead of
returning after an ok check. Expected failures assert structured tags,
stop codes, and reported limits.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
This commit is contained in:
morlutoandCursor Agent authored and GitHub committed 2026-10-29 22:54:58 +08:00
1 parent 8188497f13
commit ec4a0a01c4
35 files changed
+3005 -47

No files matched your search

+161 -5
View File
@@ -22,6 +22,9 @@ HOPPER_SYMBOL_NAME_UTF16_UNITS = 1024
# that does not fit is null, not an older descriptor path.
# 536870887 is Node's 536870888-character string ceiling minus the framing LF.
# The ceiling is the encoded payload; JSON expansion is measured before it.
# Encoded JSON allowance, in bytes, shared by one instruction list, one
# native-caller list, and one caller-scan page. This release has no runtime override.
CALLER_SCAN_ALLOWANCE_BYTES = 512 * 1024
_BRIDGE_MAX_REQUEST_BYTES_SETTING = "REA_BRIDGE_MAX_REQUEST_BYTES"
DEFAULT_BRIDGE_MAX_REQUEST_BYTES = 16 * 1024 * 1024
MINIMUM_BRIDGE_MAX_REQUEST_BYTES = 7464
@@ -232,6 +235,9 @@ EXHAUSTIVE_ANALYSIS_METHODS = frozenset(
"procedure_assembly",
"procedure_callees",
"procedure_callers",
"rea_caller_scan_page",
"rea_caller_scan_target",
"rea_caller_base",
"procedure_info",
"procedure_pseudo_code",
"procedure_references",
@@ -485,19 +491,30 @@ def _instruction_addresses(procedure):
return result
def _native_calls(procedure, direction):
def _iter_native_calls(procedure, direction):
references = procedure.getAllCallees() if direction == "outgoing" else procedure.getAllCallers()
kinds = {0: "none", 1: "unknown", 2: "direct", 3: "objective_c"}
calls = {}
unresolved = []
for reference in references:
source, target, kind = reference.fromAddress(), reference.toAddress(), reference.type()
if source in BAD_ADDRESSES:
yield None, None, None
continue
if target in BAD_ADDRESSES:
unresolved.append({"address": _hex(source), "reason": "Native CallReference has no resolved target (type %s)" % kind})
yield source, None, {"address": _hex(source), "reason": "Native CallReference has no resolved target (type %s)" % kind}
continue
calls[(source, target)] = {"classification": kinds.get(kind, "unknown"), "provider_type": kind, "provenance": "hopper-public-python-api:CallReference"}
yield source, target, {"classification": kinds.get(kind, "unknown"), "provider_type": kind, "provenance": "hopper-public-python-api:CallReference"}
def _native_calls(procedure, direction):
calls = {}
unresolved = []
for source, target, detail in _iter_native_calls(procedure, direction):
if detail is None:
continue
if target is None:
unresolved.append(detail)
else:
calls[(source, target)] = detail
return calls, sorted(unresolved, key=lambda item: int(item["address"], 16))
@@ -547,6 +564,136 @@ def _procedure_map(document):
return result
def _caller_target_data(procedure):
addresses = []
seen = set()
size = 0
for block in procedure.basicBlockIterator():
for address in _block_instruction_addresses(procedure, block):
if address in seen:
continue
encoded = _hex(address)
size += len(encoded) + 4
if size > CALLER_SCAN_ALLOWANCE_BYTES:
return {"target": _procedure_identity(procedure), "instructions": addresses, "instructions_complete": False}
addresses.append(encoded)
seen.add(address)
return {"target": _procedure_identity(procedure), "instructions": addresses, "instructions_complete": True}
def _caller_scan_target(document, params):
return _caller_target_data(_procedure(document, params.get("procedure")))
def _caller_base(document, params):
procedure = _procedure(document, params.get("procedure"))
callers = []
size = 0
complete = True
for item in procedure.getAllCallerProcedures():
address = _hex(item.getEntryPoint())
size += len(address) + 4
if size > CALLER_SCAN_ALLOWANCE_BYTES:
complete = False
break
callers.append(address)
return {
**_caller_target_data(procedure),
"callers": sorted(callers, key=lambda value: int(value, 16)),
"callers_complete": complete,
}
def _caller_scan_page(document, params):
"""Observe forward relationships in one bounded, stateless bridge request."""
start = params.get("start")
if isinstance(start, bool) or not isinstance(start, int) or start < 0:
raise InvalidRequestError("Caller scan start must be a non-negative integer")
segments = document.getSegmentsList()
counts = [segment.getProcedureCount() for segment in segments]
total = sum(counts)
end = min(start + 64, total)
if start > total:
raise InvalidRequestError("Caller scan start exceeds the procedure inventory")
items = []
item_bytes = 0
page_limit = CALLER_SCAN_ALLOWANCE_BYTES
oversized = False
identities = {}
containing = {}
def identity(procedure):
entry = procedure.getEntryPoint()
if entry not in identities:
identities[entry] = _procedure_identity(procedure)
return identities[entry]
def owner(address):
if address not in containing:
procedure, _ = _containing_procedure(document, address)
containing[address] = identity(procedure) if procedure is not None else None
return containing[address]
segment_start = 0
for segment, count in zip(segments, counts):
begin, stop = max(start - segment_start, 0), min(end - segment_start, count)
segment_start += count
for index in range(begin, max(begin, stop)):
candidate = segment.getProcedureAtIndex(index)
item = {"caller": identity(candidate), "callees": [], "references": []}
size = len(json.dumps(item, ensure_ascii=True).encode("utf-8")) + 1
remaining = page_limit - item_bytes
too_large = size > remaining
if not too_large:
for callee in candidate.getAllCalleeProcedures():
address = _hex(callee.getEntryPoint())
size += len(address) + 4
if size > remaining:
too_large = True
break
item["callees"].append(address)
if not too_large:
callee_entries = set(item["callees"])
for source, destination, call in _iter_native_calls(candidate, "outgoing"):
# Account for every native record, even when its target cannot
# be associated with an observed procedure.
size += 32
if size > remaining:
too_large = True
break
if destination is None:
continue
source_procedure = item["caller"] if _hex(source) == item["caller"]["address"] else owner(source)
target_address = _hex(destination)
if target_address in callee_entries:
target_entry = target_address
else:
target_procedure = owner(destination)
if target_procedure is None:
continue
target_entry = target_procedure["address"]
edge = {
"source_address": _hex(source), "target_address": _hex(destination),
"source_procedure": "caller" if source_procedure is not None and source_procedure["address"] == item["caller"]["address"] else source_procedure,
"target_procedure": target_entry,
"call": call,
}
size += len(json.dumps(edge, ensure_ascii=True).encode("utf-8")) + 1
if size > remaining:
too_large = True
break
item["references"].append(edge)
if too_large:
oversized = not items
break
items.append(item)
item_bytes += size
else:
continue
break
return {"start": start, "next": start + len(items), "total": total, "items": items, "oversized": oversized}
def _string_record(segment, address, display):
"""Read one native typed object, retaining its shortened display separately."""
length = segment.getObjectLength(address)
@@ -887,6 +1034,15 @@ def _dispatch(method, params):
if method in EXHAUSTIVE_ANALYSIS_METHODS:
_api().require_analysis_complete(document, method)
if method == "rea_caller_scan_target":
return _caller_scan_target(document, params)
if method == "rea_caller_base":
return _caller_base(document, params)
if method == "rea_caller_scan_page":
return _caller_scan_page(document, params)
if method == "analyze_function":
return _analyze_function(document, params)
if method == "read_function_instructions":
+21
View File
@@ -343,6 +343,27 @@ Hopper regex mode.
## Bounded static traces
Hopper caller queries, incoming procedure references and function dossiers
reconcile reported reverse relationships with forward call observations. Each
standalone query or composed workflow scans the current document in pages of at
most 64 procedures and retains at most 4 MiB of encoded scan results. A composed
workflow reuses its own index and clears it when the workflow finishes. The
default cooperative scan budget is
2,000 ms; set `REA_HOPPER_CALLER_SCAN_BUDGET_MS` to a positive integer number of
milliseconds before starting REA to change it. The default limits additional
work on the serial Hopper bridge; larger documents may need a higher setting.
Results report the examined procedure count and any budget interruption.
An individual page, native caller list, and target instruction list each have
a 512 KiB encoded allowance. Hitting one reports the incomplete observation.
Cancellation is checked between pages, but cannot interrupt a Hopper API call
already executing; Hopper's native API itself materializes its reference arrays
before REA can bound their copies. Hopper exposes no document revision token, so scans are not
reused across workflows and coverage and consistency during a live scan remain
unknown even when every procedure was examined.
Hopper observations are re-observed rather than replayed from snapshots because
the provider has no document revision token. Safe observations may still be
retained as Evidence.
`trace_native_ui_action` / `rea trace-native-ui-action <target> <seed>` accepts
one authored selector/object ID, native symbol or exact function address. It
joins UI wiring to uniquely matched class/selector implementations and follows
+1
View File
@@ -184,6 +184,7 @@
"typecheck": "node scripts/run-typescript.mjs --noEmit",
"verify:test-temp-hygiene": "npm run test:prepare && node scripts/verify-test-temp-hygiene.mjs",
"verify:hopper": "npm run verify:fixtures && npm run build:cached && node scripts/verify-real-hopper.mjs",
"verify:hopper:callers": "npm run build:cached && node scripts/verify-real-hopper-callers.mjs",
"verify:hopper:deadlines": "npm run verify:fixtures && npm run build:cached && node scripts/verify-real-hopper-deadlines.mjs",
"verify:hopper:fat": "npm run build:cached && node scripts/verify-real-hopper-fat.mjs",
"verify:hopper:fat64": "npm run build:cached && node scripts/verify-real-hopper-fat.mjs --fat64",
+5
View File
@@ -0,0 +1,5 @@
#include <stdio.h>
__attribute__((noinline)) int target_fn(int x) { return x * 5; }
__attribute__((noinline)) int call_through(int (*fp)(int), int x) { return fp(x); }
__attribute__((noinline)) int call_direct(int x) { return target_fn(x); }
int main(void) { printf("%d %d\n", call_through(target_fn, 3), call_direct(4)); return 0; }
+309
View File
@@ -0,0 +1,309 @@
// Run from the repository root after `npm run build:cached` on macOS arm64,
// with Hopper and the Xcode command-line tools installed. No target is executed.
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { constants } from "node:fs";
import { access, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { createRequire } from "node:module";
import { join } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { promisify } from "node:util";
import { artifactMcpCall } from "./lib/artifact-e2e.mjs";
import { snapshotHopperRuntime } from "./lib/real-hopper-cleanup.mjs";
import { HOPPER_TARGET_LEASE_DIRECTORY } from "../dist/process/HopperLease.js";
assert.equal(process.platform, "darwin", "requires macOS");
assert.equal(
process.arch,
"arm64",
"requires macOS arm64 (fixture tail branch is ARM64)",
);
const root = process.cwd();
const require = createRequire(join(root, "package.json"));
const { Client } = await import(
pathToFileURL(require.resolve("@modelcontextprotocol/client"))
);
const { StdioClientTransport } = await import(
pathToFileURL(require.resolve("@modelcontextprotocol/client/stdio"))
);
const { parseConfig } = await import(
pathToFileURL(join(root, "dist/config/parseConfig.js"))
);
const config = parseConfig(process.env);
if (!config.ok) throw config.error;
await access(config.value.hopperLauncherPath, constants.X_OK);
const run = promisify(execFile);
const clang = (await run("xcrun", ["--find", "clang"])).stdout.trim();
const sdk = (
await run("xcrun", ["--sdk", "macosx", "--show-sdk-path"])
).stdout.trim();
const directory = await mkdtemp(join("/tmp", "rea-hopper-callers-verifier-"));
const fixture = join(directory, "tail-call");
const large = join(directory, "long-scan");
const source = fileURLToPath(
new URL("./fixtures/hopper-tail-call.c", import.meta.url),
);
const digest = async (path) =>
createHash("sha256")
.update(await readFile(path))
.digest("hex");
const sessionsBefore = await snapshotHopperRuntime(
"/tmp",
HOPPER_TARGET_LEASE_DIRECTORY,
);
const ownedProcessIds = new Set();
const client = new Client({ name: "hopper-callers-verifier", version: "1" });
const transport = new StdioClientTransport({
command: process.execPath,
args: ["dist/main.js"],
cwd: root,
env: {
...process.env,
REA_ANALYSIS_PROVIDER: "hopper",
REA_LOG_LEVEL: "silent",
REA_HOPPER_CALLER_SCAN_BUDGET_MS: "5000",
},
stderr: "pipe",
});
let stderr = "";
transport.stderr?.on("data", (chunk) => {
stderr += chunk.toString();
});
const call = async (name, args, options = {}) => {
process.stderr.write(`calling ${name}\n`);
const result = await artifactMcpCall(
client,
{ name, arguments: args },
{ timeout: 120000, ...options },
);
assert.notEqual(result.isError, true, `${name}: ${JSON.stringify(result)}`);
return (
result.structuredContent ??
JSON.parse(result.content.find((x) => x.type === "text").text)
);
};
const address = (result) => {
const match = result.normalized_result.match(/^(0x[0-9a-f]+):/mu);
assert(match, "Hopper assembly has no instruction address");
return match[1];
};
const matchingRef = (ref, site, from, to) =>
ref.source_address === site &&
ref.source_procedure?.address === from &&
ref.target_procedure?.address === to &&
ref.call?.classification === "direct";
try {
await run(clang, [
"-isysroot",
sdk,
"-arch",
"arm64",
"-O1",
source,
"-o",
fixture,
]);
const lines = ["#include <stdio.h>"];
for (let i = 0; i < 4000; i++) {
const id = String(i).padStart(4, "0");
lines.push(
`__attribute__((noinline, used)) int rea_progress_${id}(void) { puts("REA_PROGRESS_${id}"); return ${i}; }`,
);
}
lines.push("int main(void) { return rea_progress_0000(); }");
const largeSource = join(directory, "long-scan.c");
await writeFile(largeSource, lines.join("\n"));
await run(clang, [
"-isysroot",
sdk,
"-arch",
"arm64",
"-O1",
largeSource,
"-o",
large,
]);
const before = await digest(fixture),
beforeLarge = await digest(large);
await client.connect(transport);
if (transport.pid === null)
throw new Error("MCP process identity unavailable");
ownedProcessIds.add(transport.pid);
let notifications = 0;
const original = transport.onmessage;
transport.onmessage = (message) => {
if (message.method === "notifications/progress") notifications++;
original?.(message);
};
await call("open_binary", { path: fixture });
const assembly = await call("procedure_assembly", {
procedure: "_call_direct",
});
const branch = assembly.normalized_result.match(
/^(0x[0-9a-f]+):\s+b\s+_target_fn\s*$/mu,
);
assert(
branch,
`compiler did not produce expected tail branch: ${assembly.normalized_result}`,
);
const site = branch[1],
from = address(assembly);
const to = address(
await call("procedure_assembly", { procedure: "_target_fn" }),
);
assert(
(
await call("procedure_callees", { procedure: from })
).normalized_result.includes(to),
);
const callers = await call("procedure_callers", {
procedure: to,
detail: "complete",
});
assert(callers.normalized_result.includes(from), "tail caller missing");
assert.equal(callers.relationship_coverage.status, "unknown");
assert(
callers.raw_result.forward_observations.callee_inventory_callers.some(
(x) => x.address === from,
),
"forward caller provenance missing",
);
const refs = (
await call("procedure_references", { procedure: to, direction: "incoming" })
).normalized_result.references;
assert(
refs.some((x) => matchingRef(x, site, from, to)),
"exact incoming tail reference missing",
);
const dossier = (
await call("analyze_function", { procedure: to, detail: "complete" })
).normalized_result;
assert(
dossier.callers.some((x) => x.address === from),
"dossier caller missing",
);
assert(
dossier.incoming_references.some((x) => matchingRef(x, site, from, to)),
"dossier reference missing",
);
const graph = (
await call("get_call_graph", {
address: to,
direction: "backward",
detail: "complete",
})
).normalized_result.graph;
assert(
Object.values(graph)
.flat()
.some((x) => x.address === to && x.calls.includes(from)),
"backward graph edge missing",
);
const path = (
await call("trace_call_path", {
start: to,
goal: from,
direction: "backward",
})
).normalized_result;
assert.equal(path.goal_status, "reached");
assert(
path.edges.some(
(x) => x.source_address === from && x.target_address === to,
),
);
await call("close_binary", {});
assert.equal(await digest(fixture), before);
console.log(
JSON.stringify({
event: "tail_call_verified",
from,
site,
to,
sha256: before,
}),
);
await call("open_binary", { path: large });
await call("procedure_assembly", { procedure: "_rea_progress_0000" });
const query = {
address: "_rea_progress_0000",
direction: "backward",
detail: "complete",
};
const count = notifications;
await call("get_call_graph", query);
assert.equal(notifications, count, "unsolicited progress without token");
const progress = [];
await call("get_call_graph", query, { onprogress: (x) => progress.push(x) });
assert(
progress.every((x, i) => i === 0 || x.progress > progress[i - 1].progress),
"progress not monotonic",
);
assert(
progress.some((x) =>
x.message?.includes("hopper_caller_reconciliation: Examined"),
),
"missing examined-count progress",
);
const controller = new AbortController();
let partial = false;
let resolved = false;
try {
await artifactMcpCall(
client,
{ name: "get_call_graph", arguments: query },
{
timeout: 120000,
signal: controller.signal,
onprogress: (x) => {
const match = x.message?.match(
/hopper_caller_reconciliation: Examined (\d+) of (\d+) procedures/u,
);
if (match && Number(match[1]) < Number(match[2])) {
partial = true;
controller.abort("cancel during real scan");
}
},
},
);
resolved = true;
} catch (error) {
if (!controller.signal.aborted) throw error;
}
assert(
partial && !resolved,
"did not observe cancellation after partial scan progress; cancellation coverage unavailable on this host",
);
await client.ping();
const retry = await call("get_call_graph", query);
assert(retry.normalized_result.graph, "retry did not return graph");
await call("close_binary", {});
assert.equal(await digest(large), beforeLarge);
console.log(
JSON.stringify({
event: "progress_cancel_retry_verified",
progress,
sha256: beforeLarge,
}),
);
} finally {
await client.close();
await transport.close();
await new Promise((resolve) => setTimeout(resolve, 500));
const leaked = [
...(await snapshotHopperRuntime(
"/tmp",
HOPPER_TARGET_LEASE_DIRECTORY,
ownedProcessIds,
)),
].filter((path) => !sessionsBefore.has(path));
if (leaked.length === 0)
await rm(directory, { recursive: true, force: true });
if (stderr) process.stderr.write(stderr);
assert.deepEqual(
leaked,
[],
`Hopper resources remain: ${leaked.join(", ")}; fixture retained at ${directory}`,
);
}
+3
View File
@@ -1,3 +1,4 @@
import type { AnalysisWorkScope } from "./AnalysisWorkScope.js";
import type { AnalysisObservation } from "../domain/analysisObservation.js";
import type { ProviderInspectionScope } from "./ProviderInspectionScope.js";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
@@ -26,6 +27,8 @@ import {
} from "../domain/relationshipCoverage.js";
export interface ExecutionOptions {
/** Ephemeral state shared only by one composed workflow. */
readonly workScope?: AnalysisWorkScope;
readonly signal?: AbortSignal;
readonly progress?: ProgressReporter;
/**
+18
View File
@@ -0,0 +1,18 @@
/** In-memory, provider-owned intermediate facts confined to one composed task. */
export class AnalysisWorkScope {
readonly #entries = new Map<object, unknown>();
#closed = false;
get<T>(owner: object): T | undefined {
return this.#entries.get(owner) as T | undefined;
}
set<T>(owner: object, value: T): void {
if (!this.#closed) this.#entries.set(owner, value);
}
clear(): void {
this.#closed = true;
this.#entries.clear();
}
}
@@ -65,6 +65,7 @@ const SELECTED_REGISTRATION_SETTINGS = [
"REA_GHIDRA_PROJECT_PATH",
"REA_GHIDRA_PROGRAM_NAME",
"REA_GHIDRA_NATIVEAOT_JAR",
"REA_HOPPER_CALLER_SCAN_BUDGET_MS",
"REA_HOPPER_STARTUP_TIMEOUT_MS",
"REA_JEB_MCP_URL",
"REA_JEB_MCP_TIMEOUT_MS",
+59 -1
View File
@@ -188,7 +188,8 @@ it("retains unique upstream limits in canonical order across concurrent observat
const analysis: AnalysisOperationPort = {
execute: async (operation, parameters, forwarded) => {
expect(parameters).toEqual({ marker: true });
expect(forwarded).toBe(options);
expect(forwarded).toMatchObject(options);
expect(forwarded?.workScope).toBeDefined();
const call = calls++;
if (call === 0) await gate;
else release();
@@ -621,3 +622,60 @@ it.each([
});
},
);
it("shares temporary provider state only within a single completed workflow", async () => {
const owner = {};
const scopes: Array<
NonNullable<import("./AnalysisProvider.js").ExecutionOptions["workScope"]>
> = [];
const provider = { id: "fixture", name: "Fixture", version: "1" };
const analysis: AnalysisOperationPort = {
execute: async (_operation, _parameters, options) => {
const scope = options?.workScope;
if (scope === undefined) throw new Error("workflow scope missing");
scopes.push(scope);
scope.set(owner, (scope.get<number>(owner) ?? 0) + 1);
return ok(createAnalysisExecution(null, provider));
},
};
const run = () =>
executeComposedWorkflow(analysis, async (observed) => {
await observed.execute("health", {});
await observed.execute("health", {});
expect(scopes.at(-1)?.get(owner)).toBe(2);
return ok(null);
});
await run();
expect(scopes[0]?.get(owner)).toBeUndefined();
scopes[0]?.set(owner, "late completion");
expect(scopes[0]?.get(owner)).toBeUndefined();
await run();
expect(scopes[0]).toBe(scopes[1]);
expect(scopes[0]).not.toBe(scopes[2]);
expect(scopes[2]).toBe(scopes[3]);
});
it("clears workflow state when its callback throws", async () => {
const owner = {};
let captured: import("./AnalysisProvider.js").ExecutionOptions["workScope"];
const analysis: AnalysisOperationPort = {
execute: async (_operation, _parameters, options) => {
captured = options?.workScope;
captured?.set(owner, "in flight");
return ok(
createAnalysisExecution(null, {
id: "fixture",
name: "Fixture",
version: "1",
}),
);
},
};
await expect(
executeComposedWorkflow(analysis, async (observed) => {
await observed.execute("health", {});
throw new Error("failed workflow");
}),
).rejects.toThrow("failed workflow");
expect(captured?.get(owner)).toBeUndefined();
});
+11 -3
View File
@@ -1,3 +1,4 @@
import { AnalysisWorkScope } from "./AnalysisWorkScope.js";
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
import { jsonStringByteLength } from "../domain/jsonStringByteLength.js";
import { REA_WORKFLOW_PROVIDER } from "../domain/investigationProviders.js";
@@ -91,10 +92,14 @@ export const executeComposedWorkflow = async (
let perItemBytes = 0;
let omittedObservations = 0;
let cachePolicy: "live" | undefined;
const result = await execute({
const workScope = new AnalysisWorkScope();
const observed: AnalysisOperationPort = {
resourceScope: analysis.resourceScope,
execute: async (operation, parameters, options) => {
const execution = await analysis.execute(operation, parameters, options);
const execution = await analysis.execute(operation, parameters, {
...options,
workScope,
});
if (execution.ok && execution.value.evidenceId !== undefined)
evidenceLinks.add(execution.value.evidenceId);
if (execution.ok && execution.value.sourceEvidence !== undefined) {
@@ -139,7 +144,10 @@ export const executeComposedWorkflow = async (
}
return execution;
},
});
};
const result = await Promise.resolve()
.then(() => execute(observed))
.finally(() => workScope.clear());
const sources = sourceLedger.recordsForAnalysis();
if (!result.ok) return withEvidenceSources(result, sources);
for (const source of sources) evidenceLinks.add(source.evidence_id);
@@ -32,6 +32,7 @@ import {
AnalysisSnapshotCache,
SNAPSHOT_CACHE_ENTRY_CEILING,
isSnapshotCacheable,
isSourceEvidenceRetainable,
} from "./AnalysisSnapshotCache.js";
import { EvidenceLedger } from "../investigation/EvidenceLedger.js";
import { enhancedOutputSchemas } from "../../contracts/toolOutputSchemaGroups.js";
@@ -776,6 +777,59 @@ describe("analysis snapshot cache policy", () => {
);
},
);
it("retains caller observations without a document while refusing to replay them", () => {
const descriptor: CapabilityDescriptor = {
operation: "procedure_callers",
provider: ANALYSIS_SNAPSHOT_PROVIDER,
available: true,
reason: null,
effects: {
mutatesArtifact: false,
launchesProcess: false,
mayShowUi: false,
mayAccessNetwork: false,
mayWriteFilesystem: false,
changesPermissions: false,
requiresRoot: false,
},
limitations: [],
};
const parameters = { procedure: "0x1000" };
expect(
isSourceEvidenceRetainable("procedure_callers", descriptor, parameters),
).toBe(true);
expect(
isSnapshotCacheable("procedure_callers", descriptor, parameters),
).toBe(false);
expect(
isSnapshotCacheable("procedure_callers", descriptor, {
...parameters,
document: "fixture",
}),
).toBe(true);
const callees = { ...descriptor, operation: "procedure_callees" as const };
expect(
isSourceEvidenceRetainable("procedure_callees", callees, parameters),
).toBe(true);
const pseudocode = {
...descriptor,
operation: "procedure_pseudo_code" as const,
};
expect(
isSourceEvidenceRetainable(
"procedure_pseudo_code",
pseudocode,
parameters,
),
).toBe(false);
expect(
isSourceEvidenceRetainable("procedure_pseudo_code", pseudocode, {
...parameters,
document: "fixture",
}),
).toBe(true);
});
});
describe("analysis snapshot cache partitioning", () => {
@@ -87,8 +87,28 @@ const DOCUMENT_SCOPED_OPERATIONS: ReadonlySet<string> = new Set(
/** Maximum duplicated result bindings retained across direct and workflow queries. */
export const SNAPSHOT_CACHE_ENTRY_CEILING = 10_000;
/** Whether an operation is immutable and independent of provider UI state. */
export const isSnapshotCacheable = (
const replayDocumentSelected = (
operation: AnalysisOperation,
parameters: Readonly<Record<string, JsonValue>>,
): boolean =>
!DOCUMENT_SCOPED_OPERATIONS.has(operation) ||
typeof parameters.document === "string";
/**
* Caller and callee walks omit `document`. Their observations are still
* Evidence. Other document-scoped reads, including `procedure_pseudo_code`
* from `decompile_all`, already keep each body in the file-backed export.
* Copying those bodies into the in-memory ledger would retain the export.
*/
const DOCUMENTLESS_SOURCE_EVIDENCE_OPERATIONS: ReadonlySet<AnalysisOperation> =
new Set(["procedure_callers", "procedure_callees"]);
/**
* Whether a source observation is safe to retain as Evidence.
* A missing document selector blocks snapshot replay. Only caller and callee
* walks stay in the ledger without one.
*/
export const isSourceEvidenceRetainable = (
operation: AnalysisOperation,
descriptor: CapabilityDescriptor | undefined,
parameters: Readonly<Record<string, JsonValue>>,
@@ -96,15 +116,25 @@ export const isSnapshotCacheable = (
!STATEFUL_OPERATIONS.has(operation) &&
!RESOURCE_SCOPED_OPERATIONS.has(operation) &&
(!DOCUMENT_SCOPED_OPERATIONS.has(operation) ||
typeof parameters.document === "string") &&
typeof parameters.document === "string" ||
DOCUMENTLESS_SOURCE_EVIDENCE_OPERATIONS.has(operation)) &&
(!CURSOR_DEFAULT_OPERATIONS.has(operation) ||
typeof parameters.address === "string") &&
descriptor?.effects.mutatesArtifact === false &&
descriptor.cachePolicy !== "live" &&
(descriptor.cachePolicy === "snapshot" ||
descriptor.effects.mayWriteFilesystem === false) &&
descriptor.effects.changesPermissions === false;
/** Whether an observation can additionally be replayed as a snapshot. */
export const isSnapshotCacheable = (
operation: AnalysisOperation,
descriptor: CapabilityDescriptor | undefined,
parameters: Readonly<Record<string, JsonValue>>,
): descriptor is CapabilityDescriptor =>
isSourceEvidenceRetainable(operation, descriptor, parameters) &&
replayDocumentSelected(operation, parameters) &&
descriptor.cachePolicy !== "live";
/** Windows Ghidra must admit the selected source through its native snapshot owner. */
export const requiresNativeSourceAdmissionBeforeReplay = (
providerId: string | undefined,
+10 -5
View File
@@ -44,6 +44,7 @@ import { InvestigationRecords } from "../investigation/InvestigationRecords.js";
import { recordEvidenceSources } from "../investigation/recordEvidenceSources.js";
import { BinarySessionRecords } from "./BinarySessionRecords.js";
import { binarySessionStatus } from "./BinarySessionStatus.js";
import { isSourceEvidenceRetainable } from "./AnalysisSnapshotCache.js";
import type { BinarySessionDetail } from "../../contracts/sessionStatusContract.js";
import {
resolveSessionOpen,
@@ -1178,7 +1179,7 @@ export class BinarySession
execute(
name: Parameters<AnalysisOperationPort["execute"]>[0],
arguments_: Readonly<Record<string, JsonValue>>,
options?: Pick<ExecutionOptions, "signal" | "selectorOrigin">,
options?: ExecutionOptions,
): Promise<Result<AnalysisExecution, AnalysisError>> {
const generation = this.#transitionGeneration;
const call = this.#execute(name, arguments_, options, this.#transition);
@@ -1190,7 +1191,7 @@ export class BinarySession
executeIfProviderReady(
name: Parameters<AnalysisOperationPort["execute"]>[0],
arguments_: Readonly<Record<string, JsonValue>>,
options?: Pick<ExecutionOptions, "signal" | "selectorOrigin">,
options?: Pick<ExecutionOptions, "signal" | "selectorOrigin" | "workScope">,
): Promise<Result<AnalysisExecution, AnalysisError> | undefined> {
const active = this.#active;
if (
@@ -1249,7 +1250,7 @@ export class BinarySession
async #execute(
name: Parameters<AnalysisOperationPort["execute"]>[0],
arguments_: Readonly<Record<string, JsonValue>>,
options: Pick<ExecutionOptions, "signal" | "selectorOrigin"> | undefined,
options: ExecutionOptions | undefined,
transition: Promise<void>,
): Promise<Result<AnalysisExecution, AnalysisError>> {
const transitioned = await this.#waitForTransition(
@@ -1329,7 +1330,11 @@ export class BinarySession
);
if (!sourcesRecorded.ok) return sourcesRecorded;
}
if (execution.ok && cacheable && profile !== undefined) {
if (
execution.ok &&
isSourceEvidenceRetainable(name, capability, arguments_) &&
profile !== undefined
) {
const evidence = createAnalysisExecutionEvidence(
name,
arguments_,
@@ -1346,7 +1351,7 @@ export class BinarySession
const recorded = this.recordEvidence(evidence);
if (!recorded.ok) return recorded;
const retained = { ...execution.value, evidenceId: evidence.evidence_id };
if (execution.value.cachePolicy !== "live")
if (cacheable && execution.value.cachePolicy !== "live")
this.recordSnapshot({
target: active.target,
profile,
@@ -73,6 +73,47 @@ it("preserves an explicitly unbound provider observation", async () => {
}
});
it("retains live source Evidence without replaying a prior observation", async () => {
const { session, deep } = openSession({ live: true });
try {
expect((await session.open(process.execPath)).ok).toBe(true);
expect(
(
await session.execute("address_name", {
document: "0",
address: "0x9999",
})
).ok,
).toBe(true);
const child = deep.created[0];
if (child === undefined) throw new Error("expected provider child");
let reads = 0;
child.execute = () =>
Promise.resolve(
ok(createAnalysisExecution(`live ${++reads}`, deepIdentity)),
);
const parameters = { document: "0", address: "0x1000" };
const first = await session.execute("address_name", parameters);
const second = await session.execute("address_name", parameters);
expect(first.ok && first.value.result).toBe("live 1");
expect(second.ok && second.value.result).toBe("live 2");
if (!first.ok || !second.ok) return;
expect(first.value.evidenceId).toBeDefined();
expect(second.value.evidenceId).toBeDefined();
expect(first.value.evidenceId).not.toBe(second.value.evidenceId);
expect(session.exportEvidenceBundle().records).toEqual(
expect.arrayContaining([
expect.objectContaining({ normalized_result: "live 1" }),
expect.objectContaining({ normalized_result: "live 2" }),
]),
);
const snapshot = session.exportAnalysisSnapshot();
if (snapshot.ok) expect(snapshot.value.entries).toHaveLength(0);
} finally {
await session.close();
}
});
it("keeps unbound observations out of persisted target snapshots", async () => {
const { session, deep } = openSession();
const directory = await createTestTempDirectory("rea-unbound-snapshot-");
@@ -458,6 +499,7 @@ const openSession = (
failOperationInitially?: boolean;
beforeFirstClose?: () => Promise<void>;
explicitNullSubject?: boolean;
live?: boolean;
} = {},
): {
session: BinarySession;
@@ -493,6 +535,7 @@ const killableDeepProvider = (options: {
failOperationInitially?: boolean;
beforeFirstClose?: () => Promise<void>;
explicitNullSubject?: boolean;
live?: boolean;
}): {
provider: AnalysisProviderCandidate;
created: KillableChild[];
@@ -503,7 +546,12 @@ const killableDeepProvider = (options: {
const withOperationHealth = options.operationHealth !== false;
const provider: AnalysisProviderCandidate = {
identity: () => deepIdentity,
capabilities: () => [capability(deepIdentity, "address_name")],
capabilities: () => [
{
...capability(deepIdentity, "address_name"),
...(options.live ? { cachePolicy: "live" as const } : {}),
},
],
inspectAvailability: (): ProviderAvailability => ({
status: "available",
code: null,
+4
View File
@@ -21,6 +21,7 @@ import {
} from "./providerStartupTimeout.js";
import {
DEFAULT_GHIDRA_STARTUP_TIMEOUT_MS,
DEFAULT_HOPPER_CALLER_SCAN_BUDGET_MS,
DEFAULT_HOPPER_STARTUP_TIMEOUT_MS,
} from "../domain/analysisResourceLimits.js";
import { mcpInputSchemaProfileSchema } from "./mcpInputSchemaProfile.js";
@@ -100,6 +101,9 @@ const environmentSchema = z.object({
REA_HOPPER_STARTUP_TIMEOUT_MS: providerStartupTimeoutSchema.default(
DEFAULT_HOPPER_STARTUP_TIMEOUT_MS,
),
REA_HOPPER_CALLER_SCAN_BUDGET_MS: providerStartupTimeoutSchema.default(
DEFAULT_HOPPER_CALLER_SCAN_BUDGET_MS,
),
REA_SWIFTC_COMPILE_TIMEOUT_MS: providerStartupTimeoutSchema.default(
DEFAULT_SWIFTC_COMPILE_TIMEOUT_MS,
),
+92
View File
@@ -0,0 +1,92 @@
import { expect, it } from "vitest";
import { explicitSetupRegistrationEnvironment } from "../application/SetupRegistrationEnvironment.js";
import {
DEFAULT_HOPPER_CALLER_SCAN_BUDGET_MS,
MAXIMUM_HOPPER_CALLER_SCAN_BUDGET_MS,
MINIMUM_HOPPER_CALLER_SCAN_BUDGET_MS,
} from "../domain/analysisResourceLimits.js";
import {
HOPPER_CALLER_PAGE_ALLOWANCE_BYTES,
type HopperCallerScanFacts,
} from "../domain/native/hopperCallerReconciliation.js";
import { parseConfig } from "./parseConfig.js";
import {
HOPPER_CALLER_SCAN_BUDGET_SETTING,
hopperCallerScanRecovery,
} from "./hopperCallerScanBudget.js";
const facts = (
overrides: Partial<HopperCallerScanFacts> &
Pick<HopperCallerScanFacts, "stop">,
): HopperCallerScanFacts => ({
examined_procedures: 1,
procedure_total: 2,
retained_bytes: 32,
instructions_complete: true,
native_callers_complete: true,
...overrides,
});
it("records the default caller-scan budget and forwards an explicit setting", () => {
const unset = parseConfig({});
if (!unset.ok) throw unset.error;
expect(unset.value.hopperCallerScanBudget).toMatchObject({
milliseconds: DEFAULT_HOPPER_CALLER_SCAN_BUDGET_MS,
source: "default",
floorMilliseconds: MINIMUM_HOPPER_CALLER_SCAN_BUDGET_MS,
maximumMilliseconds: MAXIMUM_HOPPER_CALLER_SCAN_BUDGET_MS,
});
const selected = parseConfig({
[HOPPER_CALLER_SCAN_BUDGET_SETTING]: "250",
});
if (!selected.ok) throw selected.error;
expect(selected.value.hopperCallerScanBudget).toMatchObject({
milliseconds: 250,
source: "setting",
});
expect(
explicitSetupRegistrationEnvironment({
[HOPPER_CALLER_SCAN_BUDGET_SETTING]: "250",
UNRELATED: "dropped",
}),
).toEqual({ [HOPPER_CALLER_SCAN_BUDGET_SETTING]: "250" });
});
it("explains the time budget and that fixed byte allowances have no override", () => {
const parsed = parseConfig({ [HOPPER_CALLER_SCAN_BUDGET_SETTING]: "250" });
if (!parsed.ok) throw parsed.error;
const budget = parsed.value.hopperCallerScanBudget;
const time = hopperCallerScanRecovery(
facts({ stop: { code: "work_budget", budget_ms: 250 } }),
budget,
);
expect(time).toContain(`${HOPPER_CALLER_SCAN_BUDGET_SETTING} is 250 ms`);
expect(time).toContain(
`The default is ${String(DEFAULT_HOPPER_CALLER_SCAN_BUDGET_MS)} ms`,
);
expect(time).toContain(
`from ${String(MINIMUM_HOPPER_CALLER_SCAN_BUDGET_MS)} to ${String(MAXIMUM_HOPPER_CALLER_SCAN_BUDGET_MS)} ms`,
);
expect(time).toContain("no runtime override");
expect(time).toContain(
`${HOPPER_CALLER_SCAN_BUDGET_SETTING} does not raise them`,
);
const page = hopperCallerScanRecovery(
facts({
stop: {
code: "page_allowance",
allowance_bytes: HOPPER_CALLER_PAGE_ALLOWANCE_BYTES,
},
}),
budget,
);
expect(page).toContain("no runtime override");
expect(page).toContain(
`${HOPPER_CALLER_SCAN_BUDGET_SETTING} does not raise them`,
);
expect(page).not.toContain("is 250 ms");
expect(
hopperCallerScanRecovery(facts({ stop: { code: "complete" } }), budget),
).toBe("");
});
+70
View File
@@ -0,0 +1,70 @@
import {
DEFAULT_HOPPER_CALLER_SCAN_BUDGET_MS,
MAXIMUM_HOPPER_CALLER_SCAN_BUDGET_MS,
MINIMUM_HOPPER_CALLER_SCAN_BUDGET_MS,
} from "../domain/analysisResourceLimits.js";
import {
HOPPER_CALLER_PAGE_ALLOWANCE_BYTES,
HOPPER_CALLER_SCAN_RETENTION_BYTES,
type HopperCallerScanFacts,
} from "../domain/native/hopperCallerReconciliation.js";
export const HOPPER_CALLER_SCAN_BUDGET_SETTING =
"REA_HOPPER_CALLER_SCAN_BUDGET_MS";
/** Effective caller-scan budget and where its milliseconds came from. */
export interface HopperCallerScanBudget {
readonly setting: typeof HOPPER_CALLER_SCAN_BUDGET_SETTING;
readonly milliseconds: number;
readonly source: "default" | "setting";
readonly defaultMilliseconds: number;
readonly floorMilliseconds: number;
readonly maximumMilliseconds: number;
}
export const defaultHopperCallerScanBudget = (): HopperCallerScanBudget => ({
setting: HOPPER_CALLER_SCAN_BUDGET_SETTING,
milliseconds: DEFAULT_HOPPER_CALLER_SCAN_BUDGET_MS,
source: "default",
defaultMilliseconds: DEFAULT_HOPPER_CALLER_SCAN_BUDGET_MS,
floorMilliseconds: MINIMUM_HOPPER_CALLER_SCAN_BUDGET_MS,
maximumMilliseconds: MAXIMUM_HOPPER_CALLER_SCAN_BUDGET_MS,
});
/** Record whether the launch environment supplied the budget. */
export const hopperCallerScanBudgetFromEnvironment = (
configured: string | undefined,
milliseconds: number,
): HopperCallerScanBudget => ({
...defaultHopperCallerScanBudget(),
milliseconds,
source: configured === undefined ? "default" : "setting",
});
/** Budget record for a caller that already resolved the millisecond count. */
export const hopperCallerScanBudgetForMilliseconds = (
milliseconds: number,
): HopperCallerScanBudget =>
hopperCallerScanBudgetFromEnvironment(
milliseconds === DEFAULT_HOPPER_CALLER_SCAN_BUDGET_MS
? undefined
: String(milliseconds),
milliseconds,
);
const fixedAllowanceRecovery = `This release has no runtime override for the ${String(HOPPER_CALLER_PAGE_ALLOWANCE_BYTES)} byte instruction, native-caller, and page allowances or the ${String(HOPPER_CALLER_SCAN_RETENTION_BYTES)} byte retention budget. ${HOPPER_CALLER_SCAN_BUDGET_SETTING} does not raise them.`;
/** Environment recovery for a caller-scan limit. Domain facts stay separate. */
export const hopperCallerScanRecovery = (
facts: HopperCallerScanFacts,
budget: HopperCallerScanBudget,
): string => {
const fixedLimit =
facts.stop.code === "page_allowance" ||
facts.stop.code === "retention_budget" ||
!facts.instructions_complete ||
facts.native_callers_complete === false;
if (facts.stop.code === "work_budget")
return `${budget.setting} is ${String(budget.milliseconds)} ms (source: ${budget.source}). The default is ${String(budget.defaultMilliseconds)} ms. Allowed values are integers from ${String(budget.floorMilliseconds)} to ${String(budget.maximumMilliseconds)} ms. Set it in the environment that starts REA and restart REA. ${fixedAllowanceRecovery}`;
return fixedLimit ? fixedAllowanceRecovery : "";
};
+5
View File
@@ -9,6 +9,7 @@ import { parseEnvironment } from "./environment.js";
import { providerStartupTimeoutSchema } from "./providerStartupTimeout.js";
import { homeDirectoryFromEnvironment } from "./homeDirectory.js";
import { DEFAULT_BRIDGE_MAX_REQUEST } from "./bridgeMaxRequestBytes.js";
import { hopperCallerScanBudgetFromEnvironment } from "./hopperCallerScanBudget.js";
import { DEFAULT_REGEX_MATCHING_DEADLINE } from "./regexMatchingDeadline.js";
import { DEFAULT_REGEX_WORKER_HEAP_REQUEST } from "./regexWorkerHeapRequest.js";
import type { AppConfig } from "./types.js";
@@ -168,6 +169,10 @@ export const parseConfig = (
homeDirectoryFromEnvironment(environment, process.platform),
),
hopperStartupTimeoutMs: env.REA_HOPPER_STARTUP_TIMEOUT_MS,
hopperCallerScanBudget: hopperCallerScanBudgetFromEnvironment(
environment.REA_HOPPER_CALLER_SCAN_BUDGET_MS,
env.REA_HOPPER_CALLER_SCAN_BUDGET_MS,
),
hopperTargetPath: env.HOPPER_TARGET_PATH,
hopperTargetKind: env.HOPPER_TARGET_KIND,
hopperLoaderArgs: env.loaderArgs,
+3
View File
@@ -10,6 +10,7 @@ import type { McpOutputSchemaProfile } from "./mcpOutputSchemaProfile.js";
import type { BridgeMaxRequestBytes } from "./bridgeMaxRequestBytes.js";
import type { RegexMatchingDeadline } from "./regexMatchingDeadline.js";
import type { RegexWorkerHeapRequest } from "./regexWorkerHeapRequest.js";
import type { HopperCallerScanBudget } from "./hopperCallerScanBudget.js";
export interface AppConfig {
readonly mcpToolFamilies?: readonly McpToolFamilies[number][];
@@ -32,6 +33,8 @@ export interface AppConfig {
readonly ilspyCmdPath: string | undefined;
readonly hopperLauncherPath: string;
readonly hopperStartupTimeoutMs: number;
/** Cooperative per-operation budget for forward caller reconciliation. */
readonly hopperCallerScanBudget: HopperCallerScanBudget;
readonly hopperTargetPath: string | undefined;
readonly hopperTargetKind: "executable" | "database";
readonly hopperLoaderArgs: readonly string[];
+2
View File
@@ -251,6 +251,7 @@ import type {
} from "./process/processCapture.js";
import type { FileOffsetPartialObservation } from "./native/fileOffsetPartialObservation.js";
import type { NativeCallPartialObservation } from "./native/nativeCallPartialObservation.js";
import type { HopperCallerReconciliationPartial } from "./native/hopperCallerReconciliation.js";
import type { ElectronActivePartialObservation } from "./javascript/electronActiveObservation.js";
import type { JavaScriptRuntimeObservation } from "./javascript/javascriptRuntimeObservation.js";
import type {
@@ -314,6 +315,7 @@ export type AnalysisPartialObservation =
| FileSnapshotPartialObservation
| PartialProcessCaptureObservation
| NativeCallPartialObservation
| HopperCallerReconciliationPartial
| FileOffsetPartialObservation
| ElectronActivePartialObservation
| BrowserScenarioPartialObservation
+17
View File
@@ -31,12 +31,29 @@ export const DEFAULT_GHIDRA_STARTUP_TIMEOUT_MS = 330_000;
/** Default absolute deadline for Hopper launch, analysis, bridge, and readiness. */
export const DEFAULT_HOPPER_STARTUP_TIMEOUT_MS = 120_000;
/**
* Default wall-clock budget for one Hopper caller-scan reconciliation, in
* milliseconds. The first page always runs; a later page is not requested
* after this budget.
*/
export const DEFAULT_HOPPER_CALLER_SCAN_BUDGET_MS = 2_000;
/** Smallest caller-scan budget. Zero is not a deadline. */
export const MINIMUM_HOPPER_CALLER_SCAN_BUDGET_MS = 1;
/**
* Largest delay Node schedules exactly. Delays above this signed 32-bit
* boundary become a 1 ms timer, so startup and swiftc timeouts stop here.
*/
export const MAX_PROVIDER_STARTUP_TIMEOUT_MS = 2_147_483_647;
/**
* Largest caller-scan budget Node can schedule exactly. A delay above this
* signed 32-bit bound becomes 1 ms, so the setting stops here.
*/
export const MAXIMUM_HOPPER_CALLER_SCAN_BUDGET_MS =
MAX_PROVIDER_STARTUP_TIMEOUT_MS;
/**
* Ghidra 12.1.4 converts whole seconds to a signed int millisecond delay.
* Keep that multiplication positive and exact rather than imposing a day cap.
@@ -0,0 +1,102 @@
import { z } from "zod";
import { jsonValueSchema } from "../jsonValue.js";
import { unknownRelationshipCoverageSchema } from "../relationshipCoverage.js";
/** Hopper bridge page allowance for one procedure in a caller scan. */
export const HOPPER_CALLER_PAGE_ALLOWANCE_BYTES = 512 * 1024;
/** Encoded scan facts retained for one reconciliation. */
export const HOPPER_CALLER_SCAN_RETENTION_BYTES = 4 * 1024 * 1024;
const hopperCallerScanStopSchema = z.discriminatedUnion("code", [
z.strictObject({ code: z.literal("complete") }),
z.strictObject({
code: z.literal("work_budget"),
budget_ms: z.int().positive(),
}),
z.strictObject({ code: z.literal("inventory_changed") }),
z.strictObject({
code: z.literal("page_allowance"),
allowance_bytes: z.literal(HOPPER_CALLER_PAGE_ALLOWANCE_BYTES),
}),
z.strictObject({
code: z.literal("retention_budget"),
allowance_bytes: z.literal(HOPPER_CALLER_SCAN_RETENTION_BYTES),
}),
z.strictObject({ code: z.literal("cancelled") }),
z.strictObject({ code: z.literal("page_failure") }),
z.strictObject({ code: z.literal("invalid_page") }),
z.strictObject({ code: z.literal("invalid_progress") }),
]);
/** Structured progress for one Hopper caller scan, separate from its wording. */
export const hopperCallerScanFactsSchema = z.strictObject({
examined_procedures: z.int().nonnegative(),
procedure_total: z.int().nonnegative().nullable(),
retained_bytes: z.int().nonnegative(),
stop: hopperCallerScanStopSchema,
instructions_complete: z.boolean(),
native_callers_complete: z.boolean().nullable(),
});
export type HopperCallerScanFacts = z.infer<typeof hopperCallerScanFactsSchema>;
export type HopperCallerScanStop = HopperCallerScanFacts["stop"];
/** Counts collected before instruction and native-caller flags are known. */
export type HopperCallerScanCounts = Omit<
HopperCallerScanFacts,
"instructions_complete" | "native_callers_complete"
>;
/** Native reply and forward scan facts retained when reconciliation stops. */
export const hopperCallerReconciliationPartialSchema = z.strictObject({
kind: z.literal("hopper-caller-reconciliation"),
native_result: jsonValueSchema,
scan: hopperCallerScanFactsSchema.nullable(),
forward_destinations: jsonValueSchema.nullable(),
unread_page: jsonValueSchema.optional(),
coverage: unknownRelationshipCoverageSchema,
});
export type HopperCallerReconciliationPartial = z.infer<
typeof hopperCallerReconciliationPartialSchema
>;
const stopDetail = (stop: HopperCallerScanStop): string => {
switch (stop.code) {
case "complete":
return "";
case "work_budget":
return `the ${String(stop.budget_ms)} ms reconciliation work budget was reached`;
case "inventory_changed":
return "the procedure inventory changed during reconciliation";
case "page_allowance":
return `a single procedure exceeded the ${String(HOPPER_CALLER_PAGE_ALLOWANCE_BYTES)} byte page allowance`;
case "retention_budget":
return `the ${String(stop.allowance_bytes)} byte reconciliation retention budget was reached`;
case "cancelled":
return "cancellation was observed after completed scan observations";
case "page_failure":
return "a later scan page failed";
case "invalid_page":
return "a scan page reply did not match the scan schema";
case "invalid_progress":
return "a scan page reported inconsistent progress";
}
};
/** Wording for scan facts. Tests compare this formatter and the facts separately. */
export const formatHopperCallerScanReason = (
facts: HopperCallerScanFacts,
): string => {
const total =
facts.procedure_total === null ? "unknown" : String(facts.procedure_total);
const detail = stopDetail(facts.stop);
let reason = `Hopper caller reconciliation examined ${String(facts.examined_procedures)} of ${total} procedures and retained ${String(facts.retained_bytes)} encoded bytes${detail === "" ? "" : `; ${detail}`}. Forward and reverse API coverage and consistency during the live scan remain unknown.`;
if (!facts.instructions_complete)
reason += ` Target instruction enumeration reached the ${String(HOPPER_CALLER_PAGE_ALLOWANCE_BYTES)} byte allowance; incoming call-site coverage is incomplete.`;
if (facts.native_callers_complete === false)
reason += ` Native caller enumeration reached the ${String(HOPPER_CALLER_PAGE_ALLOWANCE_BYTES)} byte allowance.`;
return reason;
};
@@ -0,0 +1,558 @@
import { expect, it, vi } from "vitest";
import { err, ok } from "../domain/result.js";
import type { HopperClient } from "./HopperClient.js";
import { reconcileHopperCallers } from "./HopperCallerReconciliation.js";
import { AnalysisWorkScope } from "../application/AnalysisWorkScope.js";
import { formatHopperCallerScanReason } from "../domain/native/hopperCallerReconciliation.js";
import {
hopperCallerScanBudgetForMilliseconds,
hopperCallerScanRecovery,
} from "../config/hopperCallerScanBudget.js";
import {
AnalysisOutputError,
AnalysisResourceConstraintError,
} from "../domain/analysisErrorCore.js";
import { HOPPER_CALLER_SCAN_RETENTION_BYTES } from "../domain/native/hopperCallerReconciliation.js";
const identity = (address: string) => ({
address,
name: address,
symbol: null,
classification: null,
body: { available: false as const, reason: "unknown" },
});
const target = identity("0x2000");
const caller = identity("0x3000");
const edge = {
source_address: "0x3004",
target_address: "0x2000",
source_procedure: caller,
target_procedure: target,
kind: {
available: false as const,
reason:
"Hopper exposes partial CallReference classification, but not detailed reference flags",
},
call: {
classification: "direct" as const,
provider_type: 2,
provenance: "hopper-public-python-api:CallReference",
},
};
const compactEdge = {
source_address: edge.source_address,
target_address: edge.target_address,
source_procedure: "caller" as const,
target_procedure: target.address,
call: edge.call,
};
const page = (start: number) => ({
start,
next: start === 0 ? 64 : 65,
total: 65,
oversized: false,
items:
start === 0
? Array.from({ length: 64 }, (_, index) => ({
caller: identity(`0x${(0x4000 + index).toString(16)}`),
callees: [],
references: [],
}))
: [{ caller, callees: [target.address], references: [compactEdge] }],
});
const reply = (method: string, start = 0) =>
method === "rea_caller_scan_target"
? { target, instructions: [target.address], instructions_complete: true }
: page(start);
const base = {
target,
instructions: [target.address],
instructions_complete: true,
callers: [],
callers_complete: true,
};
const referenceResult = {
procedure: target,
direction: "incoming",
reference_kinds_available: false,
unresolved_calls: [],
references: [],
};
it("builds one forward index for successive queries in a workflow", async () => {
const callTool = vi
.fn()
.mockImplementation((method, args: { start: number }) =>
Promise.resolve(ok(reply(method, args.start))),
);
const client = { callTool } as unknown as HopperClient;
const scope = new AnalysisWorkScope();
const options = { workScope: scope };
const first = await reconcileHopperCallers(
client,
"procedure_callers",
{ procedure: "name" },
base,
options,
250,
);
const second = await reconcileHopperCallers(
client,
"procedure_references",
{ procedure: "name", direction: "incoming" },
referenceResult,
options,
250,
);
if (!first.ok) throw first.error;
expect(first.value.result).toEqual(["0x3000"]);
expect(first.value.rawResult).toMatchObject({
native_result: { callers: [] },
forward_observations: {
target: { address: "0x2000" },
callee_inventory_callers: [{ address: "0x3000" }],
call_references: [
{
caller_address: caller.address,
source_address: compactEdge.source_address,
target_address: compactEdge.target_address,
source_procedure: caller.address,
target_procedure: compactEdge.target_procedure,
call: compactEdge.call,
},
],
call_site_callers: [],
},
});
if (
first.value.rawResult !== null &&
typeof first.value.rawResult === "object" &&
!Array.isArray(first.value.rawResult)
) {
const observations = first.value.rawResult.forward_observations;
const sites =
observations !== null &&
typeof observations === "object" &&
!Array.isArray(observations) &&
Array.isArray(observations.call_references)
? observations.call_references
: [];
expect(sites[0]).not.toHaveProperty("name");
expect(sites[0]).not.toHaveProperty("body");
expect(first.value.rawResult).not.toHaveProperty(
"native_result.instructions",
);
}
if (!second.ok) throw second.error;
expect(second.value.result).toMatchObject({
references: [edge],
});
expect(
callTool.mock.calls.filter(([method]) => method === "rea_caller_scan_page"),
).toHaveLength(2);
scope.clear();
await reconcileHopperCallers(
client,
"procedure_callers",
{ procedure: "name" },
base,
options,
250,
);
expect(
callTool.mock.calls.filter(([method]) => method === "rea_caller_scan_page"),
).toHaveLength(4);
});
it("links each compact call site to its owning caller address", async () => {
const other = identity("0x5000");
const otherEdge = {
...compactEdge,
source_address: "0x5004",
source_procedure: "caller" as const,
};
const callTool = vi.fn().mockImplementation((method: string) =>
Promise.resolve(
ok(
method === "rea_caller_scan_target"
? reply(method)
: {
start: 0,
next: 2,
total: 2,
oversized: false,
items: [
{ caller, callees: [], references: [compactEdge] },
{
caller: other,
callees: [target.address],
references: [otherEdge],
},
],
},
),
),
);
const result = await reconcileHopperCallers(
{ callTool } as unknown as HopperClient,
"procedure_callers",
{ procedure: "name" },
base,
undefined,
250,
);
if (!result.ok) throw result.error;
expect(result.value.rawResult).toMatchObject({
forward_observations: {
callee_inventory_callers: [{ address: other.address }],
call_site_callers: [{ address: caller.address }],
call_references: [
{
caller_address: caller.address,
source_address: compactEdge.source_address,
source_procedure: caller.address,
target_address: compactEdge.target_address,
},
{
caller_address: other.address,
source_address: otherEdge.source_address,
source_procedure: other.address,
target_address: otherEdge.target_address,
},
],
},
});
expect(result.value.result).toEqual([other.address, caller.address]);
});
it("merges paged call sites and pins the selected target", async () => {
const callTool = vi
.fn()
.mockImplementation((method, args: { start: number }) =>
Promise.resolve(ok(reply(method, args.start))),
);
const client = { callTool } as unknown as HopperClient;
const result = await reconcileHopperCallers(
client,
"procedure_references",
{ procedure: "name", direction: "incoming" },
referenceResult,
undefined,
250,
);
if (!result.ok) throw result.error;
expect(result.value.result).toMatchObject({ references: [edge] });
expect(result.value.scan).toMatchObject({
examined_procedures: 65,
procedure_total: 65,
stop: { code: "complete" },
instructions_complete: true,
native_callers_complete: null,
});
if (result.value.scan === undefined) throw new Error("missing scan facts");
expect(result.value.reason).toBe(
formatHopperCallerScanReason(result.value.scan),
);
expect(callTool.mock.calls[2]?.[1]).toMatchObject({ start: 64 });
});
it("does not reuse a previous scan", async () => {
const callTool = vi
.fn()
.mockImplementation((method, args: { start: number }) =>
Promise.resolve(ok(reply(method, args.start))),
);
const client = { callTool } as unknown as HopperClient;
const result = await reconcileHopperCallers(
client,
"procedure_callers",
{ procedure: "name" },
base,
undefined,
250,
);
expect(result.ok && result.value.result).toEqual(["0x3000"]);
callTool.mockImplementation((method, args: { start: number }) =>
Promise.resolve(
ok(
method === "rea_caller_scan_target"
? reply(method)
: {
...page(args.start),
items: page(args.start).items.map((item) => ({
...item,
callees: [],
references: [],
})),
},
),
),
);
const changed = await reconcileHopperCallers(
client,
"procedure_callers",
{ procedure: "name" },
base,
undefined,
250,
);
if (!changed.ok) throw changed.error;
expect(changed.value.result).toEqual([]);
});
it("stops between pages on cancellation", async () => {
const controller = new AbortController();
const callTool = vi.fn().mockImplementation((method: string) => {
controller.abort("test cancellation");
return Promise.resolve(ok(reply(method)));
});
const result = await reconcileHopperCallers(
{ callTool } as unknown as HopperClient,
"procedure_callers",
{},
base,
{ signal: controller.signal },
250,
);
if (result.ok) throw new Error("expected the caller scan to fail");
expect(result.error._tag).toBe("AnalysisCancelledError");
expect(result.error.cause).toBe("test cancellation");
expect(result.error.partialObservation).toMatchObject({
kind: "hopper-caller-reconciliation",
native_result: base,
coverage: { status: "unknown" },
scan: {
examined_procedures: 64,
procedure_total: 65,
stop: { code: "cancelled" },
},
});
expect(callTool).toHaveBeenCalledTimes(1);
});
it("preserves cancellation while reporting the last page", async () => {
const controller = new AbortController();
const callTool = vi
.fn()
.mockImplementation((method: string) =>
Promise.resolve(
ok(
method === "rea_caller_scan_target"
? reply(method)
: { ...page(64), start: 0, next: 1, total: 1 },
),
),
);
const progress = {
report: async () => {
controller.abort("cancelled in progress");
},
};
const result = await reconcileHopperCallers(
{ callTool } as unknown as HopperClient,
"procedure_callers",
{},
base,
{ signal: controller.signal, progress },
250,
);
if (result.ok) throw new Error("expected the caller scan to fail");
expect(result.error.cause).toBe("cancelled in progress");
expect(result.error.partialObservation).toMatchObject({
kind: "hopper-caller-reconciliation",
native_result: base,
coverage: { status: "unknown" },
scan: {
examined_procedures: 1,
procedure_total: 1,
stop: { code: "cancelled" },
},
forward_destinations: {
identities: [expect.objectContaining({ address: caller.address })],
destinations: [
expect.objectContaining({
address: target.address,
caller_addresses: [caller.address],
}),
],
},
});
});
it("retains completed pages when a later scan page fails", async () => {
let pages = 0;
const callTool = vi.fn().mockImplementation((method: string) => {
if (method === "rea_caller_scan_target")
return Promise.resolve(ok(reply(method)));
pages += 1;
if (pages === 1) return Promise.resolve(ok(page(0)));
return Promise.resolve(
err(
new AnalysisOutputError("rea_caller_scan_page", "bridge page failed"),
),
);
});
const result = await reconcileHopperCallers(
{ callTool } as unknown as HopperClient,
"procedure_callers",
{},
base,
undefined,
250,
);
if (result.ok) throw new Error("expected the caller scan to fail");
expect(result.error).toMatchObject({ _tag: "AnalysisOutputError" });
expect(result.error.partialObservation).toMatchObject({
kind: "hopper-caller-reconciliation",
native_result: base,
coverage: { status: "unknown" },
scan: {
examined_procedures: 64,
procedure_total: 65,
stop: { code: "page_failure" },
},
});
});
it("reports a changing inventory as incomplete", async () => {
const callTool = vi
.fn()
.mockImplementation((method, args: { start: number }) =>
Promise.resolve(
ok(
method === "rea_caller_scan_target"
? reply(method)
: { ...page(args.start), total: args.start === 0 ? 65 : 66 },
),
),
);
const result = await reconcileHopperCallers(
{ callTool } as unknown as HopperClient,
"procedure_callers",
{},
base,
undefined,
250,
);
if (!result.ok) throw result.error;
expect(result.value.scan).toMatchObject({
examined_procedures: 64,
procedure_total: 65,
stop: { code: "inventory_changed" },
});
if (result.value.scan === undefined) throw new Error("missing scan facts");
expect(result.value.reason).toBe(
formatHopperCallerScanReason(result.value.scan),
);
});
it("retains original callers and reports a procedure that exceeds the page allowance", async () => {
const callTool = vi
.fn()
.mockResolvedValue(
ok({ start: 0, next: 0, total: 65, oversized: true, items: [] }),
);
const result = await reconcileHopperCallers(
{ callTool } as unknown as HopperClient,
"procedure_callers",
{},
{ ...base, callers: ["0x1000"] },
undefined,
250,
);
if (!result.ok) throw result.error;
expect(result.value.result).toEqual(["0x1000"]);
expect(result.value.scan).toMatchObject({
examined_procedures: 0,
procedure_total: 65,
stop: {
code: "page_allowance",
allowance_bytes: 512 * 1024,
},
});
if (result.value.scan === undefined) throw new Error("missing scan facts");
expect(result.value.reason).toBe(
`${formatHopperCallerScanReason(result.value.scan)} ${hopperCallerScanRecovery(result.value.scan, hopperCallerScanBudgetForMilliseconds(250))}`,
);
});
it("refuses expanded reference copies that exceed the retained scan budget", async () => {
const huge = "n".repeat(HOPPER_CALLER_SCAN_RETENTION_BYTES);
const heavy = { ...target, name: huge };
const callTool = vi.fn().mockImplementation((method: string) =>
Promise.resolve(
ok(
method === "rea_caller_scan_target"
? {
target: heavy,
instructions: [target.address],
instructions_complete: true,
}
: {
start: 0,
next: 1,
total: 1,
oversized: false,
items: [
{
caller,
callees: [target.address],
references: [
compactEdge,
{ ...compactEdge, source_address: "0x3008" },
],
},
],
},
),
),
);
const result = await reconcileHopperCallers(
{ callTool } as unknown as HopperClient,
"procedure_references",
{ procedure: "name", direction: "incoming" },
referenceResult,
undefined,
250,
);
if (result.ok) throw new Error("expected the caller scan to fail");
expect(result.error).toBeInstanceOf(AnalysisResourceConstraintError);
expect(result.error).toMatchObject({
_tag: "AnalysisResourceConstraintError",
reportedLimits: {
allowance_bytes: HOPPER_CALLER_SCAN_RETENTION_BYTES,
},
partialObservation: {
kind: "hopper-caller-reconciliation",
scan: {
stop: {
code: "retention_budget",
allowance_bytes: HOPPER_CALLER_SCAN_RETENTION_BYTES,
},
},
},
});
expect(JSON.stringify(result.error.partialObservation)).not.toContain(huge);
});
it("reports truncated target instructions and native callers", async () => {
const callTool = vi
.fn()
.mockResolvedValue(
ok({ start: 0, next: 0, total: 0, oversized: false, items: [] }),
);
const result = await reconcileHopperCallers(
{ callTool } as unknown as HopperClient,
"procedure_callers",
{},
{ ...base, instructions_complete: false, callers_complete: false },
undefined,
250,
);
if (!result.ok) throw result.error;
expect(result.value.scan).toMatchObject({
examined_procedures: 0,
procedure_total: 0,
stop: { code: "complete" },
instructions_complete: false,
native_callers_complete: false,
});
if (result.value.scan === undefined) throw new Error("missing scan facts");
expect(result.value.reason).toBe(
`${formatHopperCallerScanReason(result.value.scan)} ${hopperCallerScanRecovery(result.value.scan, hopperCallerScanBudgetForMilliseconds(250))}`,
);
});
+749
View File
@@ -0,0 +1,749 @@
import { performance } from "node:perf_hooks";
import { z } from "zod";
import type { ExecutionOptions } from "../application/AnalysisProvider.js";
import {
AnalysisCancelledError,
AnalysisOutputError,
AnalysisResourceConstraintError,
} from "../domain/analysisErrorCore.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { functionDossierSchema } from "../domain/native/functionDossier.js";
import {
referenceEdgeSchema,
unresolvedCallSchema,
} from "../domain/native/nativeReference.js";
import { procedureIdentitySchema } from "../domain/native/procedureInfo.js";
import { nativeCallClassificationSchema } from "../domain/native/nativeCallReference.js";
import { err, ok, type Result } from "../domain/result.js";
import type { HopperClient } from "./HopperClient.js";
import { analysisClientFailureForOperation } from "../providers/AnalysisClientFailure.js";
import {
hopperCallerScanBudgetForMilliseconds,
hopperCallerScanRecovery,
type HopperCallerScanBudget,
} from "../config/hopperCallerScanBudget.js";
import {
formatHopperCallerScanReason,
hopperCallerReconciliationPartialSchema,
HOPPER_CALLER_PAGE_ALLOWANCE_BYTES,
HOPPER_CALLER_SCAN_RETENTION_BYTES,
type HopperCallerReconciliationPartial,
type HopperCallerScanCounts,
type HopperCallerScanFacts,
type HopperCallerScanStop,
} from "../domain/native/hopperCallerReconciliation.js";
const compactEdgeSchema = z.strictObject({
source_address: z.string(),
target_address: z.string(),
source_procedure: z.union([
z.literal("caller"),
procedureIdentitySchema,
z.null(),
]),
target_procedure: z.string(),
call: nativeCallClassificationSchema,
});
const pageSchema = z.strictObject({
start: z.int().nonnegative(),
next: z.int().nonnegative(),
total: z.int().nonnegative(),
oversized: z.boolean(),
items: z.array(
z.strictObject({
caller: procedureIdentitySchema,
callees: z.array(z.string()),
references: z.array(compactEdgeSchema),
}),
),
});
const targetSchema = z.strictObject({
target: procedureIdentitySchema,
instructions: z.array(z.string()),
instructions_complete: z.boolean(),
});
const callersBaseSchema = targetSchema.extend({
callers: z.array(z.string()),
callers_complete: z.boolean(),
});
const referencesSchema = z.strictObject({
procedure: procedureIdentitySchema,
direction: z.enum(["incoming", "outgoing"]),
reference_kinds_available: z.boolean(),
unresolved_calls: z.array(unresolvedCallSchema),
references: z.array(referenceEdgeSchema),
});
type Operation =
| "procedure_callers"
| "procedure_references"
| "analyze_function";
type Identity = z.infer<typeof procedureIdentitySchema>;
type Edge = z.infer<typeof referenceEdgeSchema>;
type CompactEdge = z.infer<typeof compactEdgeSchema>;
type Destination = {
callers: Map<string, Identity>;
references: Array<{ caller: Identity; edge: CompactEdge }>;
};
type Index = {
destinations: Map<string, Destination>;
counts: HopperCallerScanCounts;
};
type ScanFailure = {
readonly error: AnalysisError;
readonly counts: HopperCallerScanCounts;
readonly destinations: Map<string, Destination>;
readonly unreadPage?: JsonValue;
};
/** A child bridge request is a phase of the enclosing user operation. */
export function reconciliationChildOptions(
options: ExecutionOptions | undefined,
): ExecutionOptions | undefined {
if (options?.progress === undefined) return options;
const progress = options.progress;
return {
...options,
progress: {
report: (update) =>
progress.report({
...update,
completed: 0,
total: null,
terminal: false,
}),
},
};
}
async function reportScanProgress(
options: ExecutionOptions | undefined,
start: number,
total: number | undefined,
final = false,
): Promise<void> {
try {
await options?.progress?.report({
phase: "hopper_caller_reconciliation",
completed: 0,
total: null,
message: `Examined ${start} of ${total ?? "unknown"} procedures`,
...(final ? { terminal: true } : {}),
});
} catch {
// Progress is observational and cannot change the analysis result.
}
}
const scanCounts = (
stop: HopperCallerScanStop,
examined: number,
procedureTotal: number | undefined,
retainedBytes: number,
): HopperCallerScanCounts => ({
examined_procedures: examined,
procedure_total: procedureTotal ?? null,
retained_bytes: retainedBytes,
stop,
});
async function buildIndex(
client: HopperClient,
operation: Operation,
document: JsonValue | undefined,
options: ExecutionOptions | undefined,
budgetMs: number,
): Promise<Result<Index, ScanFailure>> {
const destinations = new Map<string, Destination>();
const destination = (address: string): Destination => {
let value = destinations.get(address);
if (value === undefined) {
value = { callers: new Map(), references: [] };
destinations.set(address, value);
}
return value;
};
let start = 0,
total: number | undefined,
bytes = 0;
const began = performance.now();
let stop: HopperCallerScanStop = { code: "complete" };
const cancelled = (): AnalysisCancelledError =>
new AnalysisCancelledError(operation, { cause: options?.signal?.reason });
const fail = (
error: AnalysisError,
failureStop: HopperCallerScanStop,
unreadPage?: JsonValue,
): Result<Index, ScanFailure> =>
err({
error,
counts: scanCounts(failureStop, start, total, bytes),
destinations,
...(unreadPage === undefined ? {} : { unreadPage }),
});
do {
if (options?.signal?.aborted)
return fail(cancelled(), { code: "cancelled" });
if (start > 0 && performance.now() - began >= budgetMs) {
stop = { code: "work_budget", budget_ms: budgetMs };
break;
}
if (start > 0) await reportScanProgress(options, start, total);
const args: Record<string, JsonValue> = { start };
if (document !== undefined) args.document = document;
const pageOptions =
options?.progress === undefined
? options
: {
...options,
progress: {
report: (update: { readonly terminal?: boolean }) =>
update.terminal === true
? Promise.resolve()
: reportScanProgress(options, start, total),
},
};
const page = await client.callTool(
"rea_caller_scan_page",
args,
pageOptions,
);
if (!page.ok)
return options?.signal?.aborted
? fail(cancelled(), { code: "cancelled" })
: fail(
analysisClientFailureForOperation(page.error, "hopper", operation),
{ code: "page_failure" },
);
const decoded = pageSchema.safeParse(page.value);
if (!decoded.success)
return options?.signal?.aborted
? fail(cancelled(), { code: "cancelled" }, page.value)
: fail(
new AnalysisOutputError(
operation,
"Invalid Hopper caller scan page",
{ cause: decoded.error },
),
{ code: "invalid_page" },
page.value,
);
const value = decoded.data;
if (
value.start !== start ||
value.next < start ||
value.next > value.total ||
value.next - start > 64 ||
(value.next === start &&
value.next !== value.total &&
!value.oversized) ||
value.items.length !== value.next - start ||
(value.oversized && value.next !== start)
)
return options?.signal?.aborted
? fail(cancelled(), { code: "cancelled" }, page.value)
: fail(
new AnalysisOutputError(
operation,
"Invalid Hopper caller scan progress",
),
{ code: "invalid_progress" },
page.value,
);
if (total !== undefined && value.total !== total) {
stop = { code: "inventory_changed" };
break;
}
total = value.total;
if (value.oversized) {
stop = {
code: "page_allowance",
allowance_bytes: HOPPER_CALLER_PAGE_ALLOWANCE_BYTES,
};
break;
}
const pageBytes = Buffer.byteLength(JSON.stringify(page.value));
if (pageBytes > HOPPER_CALLER_SCAN_RETENTION_BYTES - bytes) {
stop = {
code: "retention_budget",
allowance_bytes: HOPPER_CALLER_SCAN_RETENTION_BYTES,
};
break;
}
bytes += pageBytes;
for (const item of value.items) {
for (const callee of item.callees)
destination(callee).callers.set(item.caller.address, item.caller);
for (const edge of item.references) {
destination(edge.target_procedure).references.push({
caller: item.caller,
edge,
});
}
}
start = value.next;
if (options?.signal?.aborted)
return fail(cancelled(), { code: "cancelled" });
await reportScanProgress(options, start, total);
if (options?.signal?.aborted)
return fail(cancelled(), { code: "cancelled" });
} while (start < (total ?? 0));
await reportScanProgress(options, start, total, true);
if (options?.signal?.aborted) return fail(cancelled(), { code: "cancelled" });
return ok({
destinations,
counts: scanCounts(stop, start, total, bytes),
});
}
/** Instruction lists are a scan filter, not part of the retained caller reply. */
const omitInstructionPayload = (value: JsonValue): JsonValue => {
if (value === null || typeof value !== "object" || Array.isArray(value))
return value;
if (!Object.hasOwn(value, "instructions")) return value;
const { instructions: _instructions, ...rest } = value;
return rest;
};
/**
* `source_procedure: "caller"` means the enclosing page item. Keep that
* owner's address on the site so the compact record does not depend on a
* separate inventory list or a copied identity.
*/
const compactCallReference = (
caller: Identity,
edge: CompactEdge,
): {
readonly caller_address: string;
readonly source_address: string;
readonly target_address: string;
readonly source_procedure: string | null;
readonly target_procedure: string;
readonly call?: CompactEdge["call"];
} => ({
caller_address: caller.address,
source_address: edge.source_address,
target_address: edge.target_address,
source_procedure:
edge.source_procedure === null || edge.source_procedure === "caller"
? edge.source_procedure === "caller"
? caller.address
: null
: edge.source_procedure.address,
target_procedure: edge.target_procedure,
...(edge.call === undefined ? {} : { call: edge.call }),
});
/** One shared identity record; destinations cite addresses instead of copies. */
const compactForwardDestinations = (
destinations: Map<string, Destination>,
): JsonValue => {
const identities = new Map<string, Identity>();
const remember = (identity: Identity): void => {
if (!identities.has(identity.address))
identities.set(identity.address, identity);
};
const rows = [...destinations.entries()].map(([address, value]) => {
const callerAddresses: string[] = [];
for (const caller of value.callers.values()) {
remember(caller);
callerAddresses.push(caller.address);
}
return {
address,
caller_addresses: callerAddresses,
references: value.references.map(({ caller, edge }) => {
remember(caller);
if (
edge.source_procedure !== null &&
edge.source_procedure !== "caller"
)
remember(edge.source_procedure);
return compactCallReference(caller, edge);
}),
};
});
return {
identities: [...identities.values()],
destinations: rows,
};
};
export const reconcileHopperCallers = async (
client: HopperClient,
operation: Operation,
parameters: Readonly<Record<string, JsonValue>>,
original: JsonValue,
options: ExecutionOptions | undefined,
budget: number | HopperCallerScanBudget,
): Promise<
Result<
{
result: JsonValue;
rawResult: JsonValue;
reason: string;
scan?: HopperCallerScanFacts;
},
AnalysisError
>
> => {
const scanBudget =
typeof budget === "number"
? hopperCallerScanBudgetForMilliseconds(budget)
: budget;
const budgetMs = scanBudget.milliseconds;
const describeScan = (facts: HopperCallerScanFacts): string => {
const observed = formatHopperCallerScanReason(facts);
const recovery = hopperCallerScanRecovery(facts, scanBudget);
return recovery === "" ? observed : `${observed} ${recovery}`;
};
const retainNative = (
error: AnalysisError,
partial: {
readonly scan: HopperCallerScanFacts | null;
readonly destinations: Map<string, Destination> | null;
readonly unreadPage?: JsonValue;
},
): AnalysisError => {
const observation: HopperCallerReconciliationPartial = {
kind: "hopper-caller-reconciliation",
native_result: original,
scan: partial.scan,
forward_destinations:
partial.destinations === null
? null
: compactForwardDestinations(partial.destinations),
coverage: {
status: "unknown",
reason:
partial.scan === null
? "Hopper caller reconciliation did not examine the procedure inventory; unexamined procedure relationships are unknown."
: describeScan(partial.scan),
limited_directions:
operation === "analyze_function"
? ["callers", "callees"]
: ["callers"],
},
...(partial.unreadPage === undefined
? {}
: { unread_page: partial.unreadPage }),
};
error.retainPartialObservation(
hopperCallerReconciliationPartialSchema.parse(observation),
);
return error;
};
const base =
operation === "procedure_callers"
? callersBaseSchema.safeParse(original)
: undefined;
if (base !== undefined && !base.success)
return err(
retainNative(
new AnalysisOutputError(
operation,
"Invalid Hopper caller observation",
{ cause: base.error },
),
{ scan: null, destinations: null },
),
);
const parsed =
operation === "procedure_callers"
? z
.array(z.string())
.safeParse(base?.success ? base.data.callers : undefined)
: operation === "analyze_function"
? functionDossierSchema.safeParse(original)
: referencesSchema.safeParse(original);
if (!parsed.success)
return err(
retainNative(
new AnalysisOutputError(
operation,
"Invalid Hopper caller observation",
{ cause: parsed.error },
),
{ scan: null, destinations: null },
),
);
const args: Record<string, JsonValue> = {};
if (parameters.document !== undefined) args.document = parameters.document;
if (parameters.procedure !== undefined) args.procedure = parameters.procedure;
const targetResponse = base?.success
? ok({
target: base.data.target,
instructions: base.data.instructions,
instructions_complete: base.data.instructions_complete,
})
: await client.callTool(
"rea_caller_scan_target",
args,
reconciliationChildOptions(options),
);
if (!targetResponse.ok)
return err(
retainNative(
analysisClientFailureForOperation(
targetResponse.error,
"hopper",
operation,
),
{ scan: null, destinations: null },
),
);
if (options?.signal?.aborted)
return err(
retainNative(
new AnalysisCancelledError(operation, {
cause: options.signal.reason,
}),
{ scan: null, destinations: null },
),
);
const selected = targetSchema.safeParse(targetResponse.value);
if (!selected.success)
return err(
retainNative(
new AnalysisOutputError(
operation,
"Invalid Hopper caller scan target",
{ cause: selected.error },
),
{ scan: null, destinations: null },
),
);
const { target, instructions, instructions_complete } = selected.data;
if (
operation !== "procedure_callers" &&
"procedure" in parsed.data &&
parsed.data.procedure.address !== target.address
)
return ok({
result: original,
rawResult: original,
reason:
"The selected procedure changed during reconciliation. Relationship coverage is unknown.",
});
const documentKey =
parameters.document === undefined
? "default"
: `document:${String(parameters.document)}`;
let indexByDocument = options?.workScope?.get<Map<string, Index>>(client);
let index = indexByDocument?.get(documentKey);
if (index === undefined) {
const scanned = await buildIndex(
client,
operation,
parameters.document,
options,
budgetMs,
);
if (!scanned.ok) {
const failure = scanned.error;
return err(
retainNative(failure.error, {
scan: {
...failure.counts,
instructions_complete,
native_callers_complete: base?.success
? base.data.callers_complete
: null,
},
destinations: failure.destinations,
...(failure.unreadPage === undefined
? {}
: { unreadPage: failure.unreadPage }),
}),
);
}
index = scanned.value;
if (options?.workScope !== undefined) {
if (indexByDocument === undefined) {
indexByDocument = new Map();
options.workScope.set(client, indexByDocument);
}
indexByDocument.set(documentKey, index);
}
}
const addresses = new Set(instructions);
addresses.add(target.address);
const selectedDestination = index.destinations.get(target.address);
const callers = new Map(selectedDestination?.callers ?? []);
const selectedReferences: Array<{ caller: Identity; edge: CompactEdge }> = [];
for (const item of selectedDestination?.references ?? []) {
if (!addresses.has(item.edge.target_address)) continue;
const source =
item.edge.source_procedure === "caller"
? item.caller
: item.edge.source_procedure;
if (source !== null) callers.set(source.address, source);
selectedReferences.push(item);
}
const unavailableKind = {
available: false as const,
reason:
"Hopper exposes partial CallReference classification, but not detailed reference flags",
};
// The scan budget counted compact pages. Expanded edges repeat each
// identity, so measure that copy before allocating it.
const expandedReferenceBytes = (): number => {
let bytes = 2;
const targetBytes = Buffer.byteLength(JSON.stringify(target));
const kindBytes = Buffer.byteLength(JSON.stringify(unavailableKind));
for (const { caller, edge } of selectedReferences) {
const source =
edge.source_procedure === "caller"
? caller
: edge.source_procedure === null
? null
: edge.source_procedure;
const sourceBytes =
source === null ? 4 : Buffer.byteLength(JSON.stringify(source));
const callBytes =
edge.call === undefined
? 0
: Buffer.byteLength(JSON.stringify(edge.call));
bytes +=
64 +
sourceBytes +
targetBytes +
kindBytes +
callBytes +
Buffer.byteLength(edge.source_address) +
Buffer.byteLength(edge.target_address);
if (bytes > HOPPER_CALLER_SCAN_RETENTION_BYTES) return bytes;
}
return bytes;
};
const retainedNativeResult = omitInstructionPayload(original);
const compactReferences = selectedReferences.map(({ caller, edge }) =>
compactCallReference(caller, edge),
);
const inventoryCallers = selectedDestination?.callers ?? new Map();
const siteCallers = new Map<string, Identity>();
for (const { caller } of selectedReferences)
if (
!inventoryCallers.has(caller.address) &&
!siteCallers.has(caller.address)
)
siteCallers.set(caller.address, caller);
if (
operation !== "procedure_callers" &&
expandedReferenceBytes() > HOPPER_CALLER_SCAN_RETENTION_BYTES
) {
const scan: HopperCallerScanFacts = {
...index.counts,
instructions_complete,
native_callers_complete: base?.success
? base.data.callers_complete
: null,
stop: {
code: "retention_budget",
allowance_bytes: HOPPER_CALLER_SCAN_RETENTION_BYTES,
},
};
return err(
retainNative(
new AnalysisResourceConstraintError(
operation,
"memory",
describeScan(scan),
{ allowance_bytes: HOPPER_CALLER_SCAN_RETENTION_BYTES },
{
retryable: false,
remediationAction: hopperCallerScanRecovery(scan, scanBudget),
},
),
{ scan, destinations: index.destinations },
),
);
}
const references = new Map<string, Edge>();
if (operation !== "procedure_callers")
for (const { caller, edge } of selectedReferences) {
const source =
edge.source_procedure === "caller" ? caller : edge.source_procedure;
references.set(`${edge.source_address}:${edge.target_address}`, {
source_address: edge.source_address,
target_address: edge.target_address,
source_procedure: source,
target_procedure: target,
kind: unavailableKind,
call: edge.call,
});
}
const mergeReferences = (
existing: z.infer<typeof referenceEdgeSchema>[],
excludeSource?: string,
) => {
const merged = new Map(
existing.map((edge) => [
`${edge.source_address}:${edge.target_address}`,
edge,
]),
);
for (const [key, edge] of references) {
if (
excludeSource !== undefined &&
edge.source_procedure?.address === excludeSource
)
continue;
const old = merged.get(key);
merged.set(
key,
old === undefined
? edge
: old.call === undefined && edge.call !== undefined
? { ...old, call: edge.call }
: old,
);
}
return [...merged.values()];
};
let result: JsonValue;
if (operation === "procedure_callers")
result = [...new Set([...(parsed.data as string[]), ...callers.keys()])];
else if (operation === "analyze_function") {
const dossier = functionDossierSchema.parse(parsed.data);
const identities = new Map(
dossier.callers.map((item) => [item.address, item]),
);
for (const [address, caller] of callers)
if (!identities.has(address)) identities.set(address, caller);
result = {
...dossier,
callers: [...identities.values()],
incoming_references: mergeReferences(
dossier.incoming_references,
dossier.procedure.address,
),
};
} else {
const facts = referencesSchema.parse(parsed.data);
result = { ...facts, references: mergeReferences(facts.references) };
}
const scan: HopperCallerScanFacts = {
...index.counts,
instructions_complete,
native_callers_complete: base?.success ? base.data.callers_complete : null,
};
return ok({
result,
rawResult: {
native_result: retainedNativeResult,
forward_observations: {
target,
callee_inventory_callers: [
...(selectedDestination?.callers.values() ?? []),
],
call_site_callers: [...siteCallers.values()],
call_references: compactReferences,
provenance:
"hopper-public-python-api:getAllCalleeProcedures and CallReference",
},
},
scan,
reason: describeScan(scan),
});
};
+205 -4
View File
@@ -2,11 +2,48 @@ import { describe, expect, it, vi } from "vitest";
import { parseConfig } from "../config/parseConfig.js";
import { silentLogger } from "../logger.js";
import { ok } from "../domain/result.js";
import { err, ok } from "../domain/result.js";
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
import { HopperClient } from "./HopperClient.js";
import { HopperProvider } from "./HopperProvider.js";
import { isSnapshotCacheable } from "../application/binary/AnalysisSnapshotCache.js";
import { routeAllowsSnapshotReplay } from "../application/binary/SessionProviderPolicy.js";
import type { SessionProviderRoute } from "../application/binary/SessionProviderRouter.js";
describe("Hopper provider capabilities", () => {
it("does not replay snapshots for mutable Hopper document observations", () => {
const config = parseConfig({});
if (!config.ok) throw config.error;
const provider = new HopperProvider(config.value, silentLogger, {});
const capabilities = provider.capabilities();
const route: SessionProviderRoute = {
identity: provider.identity(),
capabilities: new Map(capabilities.map((item) => [item.operation, item])),
profile: null,
binding: null,
selection: undefined,
createClient: () => {
throw new Error("No client needed");
},
};
const readable = capabilities.filter(
(item) => item.available && !item.effects.mutatesArtifact,
);
expect(readable.length).toBeGreaterThan(0);
for (const descriptor of readable) {
expect(descriptor.cachePolicy).toBe("live");
expect(routeAllowsSnapshotReplay(route, descriptor.operation)).toBe(
false,
);
expect(
isSnapshotCacheable(descriptor.operation, descriptor, {
document: "selected",
address: "0x1000",
}),
).toBe(false);
}
expect(routeAllowsSnapshotReplay(route, "get_call_graph")).toBe(false);
});
it("declines caller-selected gamecube-dol targets before the ELF powerpc gate", () => {
const config = parseConfig({});
if (!config.ok) throw new Error("expected valid configuration");
@@ -129,18 +166,65 @@ describe("Hopper relationship coverage", () => {
async (operation, directions) => {
const config = parseConfig({});
if (!config.ok) throw config.error;
const identity = (address: string) => ({
address,
name: address,
symbol: null,
classification: null,
body: { available: false as const, reason: "unknown" },
});
const source =
operation === "analyze_function"
? {
procedure: { ...identity("0x1000"), signature: null, locals: [] },
pseudocode: "",
assembly: [],
comments: [],
callers: [],
callees: [{ address: "0x2000", name: "tail_target" }],
callees: [identity("0x2000")],
incoming_references: [],
outgoing_references: [],
referenced_strings: [],
referenced_names: [],
basic_blocks: [],
native_api: null,
native_value_flow: null,
limitations: [],
}
: operation === "procedure_callees"
? ["0x2000"]
: [];
const callTool = vi
.spyOn(HopperClient.prototype, "callTool")
.mockResolvedValue(ok(source));
.mockImplementation((method) =>
Promise.resolve(
ok(
method === "rea_caller_base"
? {
target: identity("0x1000"),
instructions: ["0x1000"],
instructions_complete: true,
callers: [],
callers_complete: true,
}
: method === "rea_caller_scan_target"
? {
target: identity("0x1000"),
instructions: ["0x1000"],
instructions_complete: true,
}
: method === "rea_caller_scan_page"
? {
start: 0,
next: 0,
total: 0,
items: [],
oversized: false,
}
: source,
),
),
);
const client = new HopperProvider(
config.value,
silentLogger,
@@ -158,10 +242,35 @@ describe("Hopper relationship coverage", () => {
});
if (!observed.ok) throw observed.error;
expect(observed.value.result).toEqual(source);
if (operation === "procedure_callers") {
expect(observed.value.rawResult).toMatchObject({
native_result: {
callers: [],
callers_complete: true,
instructions_complete: true,
target: { address: "0x1000" },
},
forward_observations: {
call_references: [],
call_site_callers: [],
},
});
expect(observed.value.rawResult).toMatchObject({
native_result: expect.not.objectContaining({
instructions: expect.any(Array),
}),
});
}
if (operation === "procedure_callees")
expect(observed.value.rawResult).toBeNull();
expect(observed.value.relationshipCoverage).toMatchObject({
status: "unknown",
limited_directions: directions,
reason: expect.stringContaining("resolved tail calls"),
reason: expect.stringContaining(
operation === "procedure_callees"
? "resolved tail calls"
: "reconciliation examined",
),
});
} finally {
await client.close();
@@ -170,3 +279,95 @@ describe("Hopper relationship coverage", () => {
},
);
});
describe("Hopper caller reconciliation failures", () => {
it("keeps the native caller reply when a later scan page fails", async () => {
const config = parseConfig({});
if (!config.ok) throw config.error;
const identity = (address: string) => ({
address,
name: address,
symbol: null,
classification: null,
body: { available: false as const, reason: "unknown" },
});
const native = {
target: identity("0x1000"),
instructions: ["0x1000"],
instructions_complete: true,
callers: ["0x1111"],
callers_complete: true,
};
let pages = 0;
const callTool = vi
.spyOn(HopperClient.prototype, "callTool")
.mockImplementation((method) => {
if (method === "rea_caller_base") return Promise.resolve(ok(native));
if (method !== "rea_caller_scan_page") return Promise.resolve(ok(null));
pages += 1;
if (pages === 1)
return Promise.resolve(
ok({
start: 0,
next: 1,
total: 2,
oversized: false,
items: [
{
caller: identity("0x3000"),
callees: ["0x1000"],
references: [],
},
],
}),
);
return Promise.resolve(
err(
new AnalysisOutputError(
"rea_caller_scan_page",
"bridge page failed",
),
),
);
});
const client = new HopperProvider(
config.value,
silentLogger,
{},
"linux",
).createClient({
path: "/tmp/fixture.hop",
sha256: "a".repeat(64),
kind: "database",
format: "analysis-database",
});
try {
const observed = await client.execute("procedure_callers", {
procedure: "0x1000",
});
if (observed.ok) throw new Error("expected procedure_callers to fail");
expect(observed.error.partialObservation).toMatchObject({
kind: "hopper-caller-reconciliation",
native_result: native,
coverage: { status: "unknown", limited_directions: ["callers"] },
scan: {
examined_procedures: 1,
procedure_total: 2,
stop: { code: "page_failure" },
},
forward_destinations: {
identities: [expect.objectContaining({ address: "0x3000" })],
destinations: [
expect.objectContaining({
address: "0x1000",
caller_addresses: ["0x3000"],
}),
],
},
});
} finally {
await client.close();
callTool.mockRestore();
}
});
});
+61 -5
View File
@@ -44,6 +44,11 @@ import { HopperRegexSearch } from "./HopperRegexSearch.js";
import { HopperClient } from "./HopperClient.js";
import { projectHopperNameInventory } from "./HopperNameInventory.js";
import { hopperMachOImageSchema } from "./HopperMachOImage.js";
import {
reconcileHopperCallers,
reconciliationChildOptions,
} from "./HopperCallerReconciliation.js";
import { analysisClientFailureForOperation } from "../providers/AnalysisClientFailure.js";
import {
hopperCapabilities,
@@ -346,10 +351,14 @@ export class HopperProvider implements AnalysisProviderCandidate {
operation: AnalysisOperation,
normalized: JsonValue,
rawResult: JsonValue | null = null,
reconciliationReason?: string,
) =>
createAnalysisExecution(normalized, executionProvider, {
// Retain the source reply when an adapter-owned projection changes it.
rawResult,
...(reconciliationReason === undefined
? {}
: { cachePolicy: "live" as const }),
...(profile === undefined ? {} : { analysisProfile: profile }),
...(operation === "procedure_callers" ||
operation === "procedure_callees" ||
@@ -364,11 +373,16 @@ export class HopperProvider implements AnalysisProviderCandidate {
? (["callees"] as const)
: (["callers", "callees"] as const),
reason:
reconciliationReason ??
"Hopper reports caller and callee procedures through independent public API inventories. Their completeness, including resolved tail calls and agreement between reverse and forward relationships, has not been established. Observed relationships are retained.",
},
}
: {}),
limitations: [
...(reconciliationReason === undefined ||
operation !== "procedure_references"
? []
: [reconciliationReason]),
...(target.kind !== "executable"
? []
: [
@@ -395,13 +409,35 @@ export class HopperProvider implements AnalysisProviderCandidate {
execute: async (operation, parameters, options) => {
try {
return await sourceReads.run(async () => {
const willReconcile =
operation === "procedure_callers" ||
operation === "analyze_function" ||
(operation === "procedure_references" &&
parameters.direction === "incoming");
const result =
(operation === "search_strings" ||
operation === "search_procedures") &&
parameters.mode === "regex"
? await regexSearch.execute(operation, parameters, options)
: await client.callTool(operation, parameters, options);
if (!result.ok) return result;
: await client.callTool(
operation === "procedure_callers"
? "rea_caller_base"
: operation,
parameters,
willReconcile
? reconciliationChildOptions(options)
: options,
);
if (!result.ok)
return operation === "procedure_callers"
? err(
analysisClientFailureForOperation(
result.error,
"hopper",
operation,
),
)
: result;
const mapped =
operation === "address_to_file_offset"
? await mapHopperFileOffset(target, result.value, {
@@ -419,15 +455,35 @@ export class HopperProvider implements AnalysisProviderCandidate {
})
: result;
if (!mapped.ok) return mapped;
const reconciled =
operation === "procedure_callers" ||
operation === "analyze_function" ||
(operation === "procedure_references" &&
parameters.direction === "incoming")
? await reconcileHopperCallers(
client,
operation,
parameters,
mapped.value,
options,
this.config.hopperCallerScanBudget,
)
: undefined;
if (reconciled !== undefined && !reconciled.ok) return reconciled;
const reconciledResult = reconciled?.value.result ?? mapped.value;
const inventory =
operation === "list_names"
? projectHopperNameInventory(mapped.value)
: { result: mapped.value, changed: false };
? projectHopperNameInventory(reconciledResult)
: { result: reconciledResult, changed: false };
return ok(
projectExecution(
operation,
inventory.result,
inventory.changed || mapped !== result ? result.value : null,
reconciled?.value.rawResult ??
(inventory.changed || mapped !== result
? result.value
: null),
reconciled?.value.reason,
),
);
});
+2
View File
@@ -79,6 +79,8 @@ export const hopperCapabilities = (
Object.freeze({
provider: HOPPER_PROVIDER_IDENTITY,
operation,
// Hopper's GUI can change the document without a revision token.
cachePolicy: "live" as const,
...(operation === "search_bytes"
? {
available: false as const,
+42
View File
@@ -0,0 +1,42 @@
import { expect, it, vi } from "vitest";
import { mcpProgressReporter, type McpProgressContext } from "./mcpProgress.js";
it("emits monotonic MCP progress when a composed workflow starts another phase", async () => {
const notify = vi.fn().mockResolvedValue(undefined);
const reporter = mcpProgressReporter({
mcpReq: { _meta: { progressToken: "token" }, notify },
} as unknown as McpProgressContext);
await reporter.report({
phase: "native",
completed: 1,
total: 1,
message: "complete",
terminal: true,
});
await reporter.report({
phase: "reconcile",
completed: 0,
total: null,
message: "examined",
terminal: true,
});
expect(notify.mock.calls.map(([message]) => message.params.progress)).toEqual(
[1, 2],
);
expect(notify.mock.calls[1]?.[0].params.message).toBe("reconcile: examined");
});
it("does not emit progress without a negotiated token", async () => {
const notify = vi.fn();
const reporter = mcpProgressReporter({
mcpReq: { _meta: undefined, notify },
} as unknown as McpProgressContext);
await reporter.report({
phase: "native",
completed: 1,
total: 1,
message: "complete",
terminal: true,
});
expect(notify).not.toHaveBeenCalled();
});
+8 -1
View File
@@ -15,7 +15,7 @@ export const mcpProgressReporter = (
): ProgressReporter => {
const token = context.mcpReq._meta?.progressToken;
if (token === undefined) return silentProgressReporter;
return createProgressReporter(
const reporter = createProgressReporter(
async (update) => {
try {
await context.mcpReq.notify({
@@ -36,4 +36,11 @@ export const mcpProgressReporter = (
},
{ minimumIntervalMs: 100 },
);
return {
// Provider child operations can each complete before their enclosing
// workflow does. MCP progress uses the emitted sequence, not these local
// phase counts, so reset them before the shared monotonic check.
report: (update) =>
reporter.report({ ...update, completed: 0, total: null }),
};
};
@@ -292,13 +292,29 @@ it.each(["profileless", "uncacheable"] as const)(
{},
);
if (!execution.ok) return execution;
expect(execution.value.evidenceId).toBeUndefined();
if (mode === "profileless")
expect(execution.value.evidenceId).toBeUndefined();
else {
expect(execution.value.evidenceId).toEqual(expect.any(String));
expect(execution.value.evidenceId).not.toBe(spoofedEvidenceId);
}
return ok(execution.value.result);
},
);
if (!composed.ok) throw composed.error;
expect(composed.value.evidenceLinks).toBeUndefined();
expect(fixture.session.exportEvidenceBundle().records).toEqual([]);
if (mode === "profileless") {
expect(composed.value.evidenceLinks).toBeUndefined();
expect(fixture.session.exportEvidenceBundle().records).toEqual([]);
} else {
const evidenceId = composed.value.evidenceLinks?.[0];
expect(evidenceId).toEqual(expect.any(String));
expect(evidenceId).not.toBe(spoofedEvidenceId);
expect(
fixture.session
.exportEvidenceBundle()
.records.map((record) => record.evidence_id),
).toEqual([evidenceId]);
}
},
);
@@ -0,0 +1,59 @@
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { fileURLToPath } from "node:url";
import { expect, it } from "vitest";
it("pages forward caller facts across segments without retaining stale results", async () => {
const bridge = fileURLToPath(
new URL("../../../../bridge/hopper_bridge.py", import.meta.url),
);
const fixture = fileURLToPath(
new URL("../../../fixtures/hopperCallerProbe.py", import.meta.url),
);
const { stdout } = await promisify(execFile)("python3", [fixture, bridge], {
encoding: "utf8",
timeout: 3000,
});
const result = JSON.parse(stdout) as Record<string, unknown>;
expect(result.first).toMatchObject({
start: 0,
next: 64,
total: 130,
items: 64,
});
expect(result.second).toMatchObject({
start: 64,
next: 128,
total: 130,
items: 64,
caller: {
caller: { address: "0x3410" },
callees: ["0x2000"],
references: [
{
source_address: "0x3414",
target_address: "0x2000",
call: { classification: "direct" },
},
],
},
});
expect(result.third).toMatchObject({
start: 128,
next: 130,
total: 130,
items: 2,
});
expect(result.changed).toMatchObject({ callees: [], references: [] });
expect(result.oversized).toMatchObject({
start: 0,
next: 0,
total: 130,
items: [],
oversized: true,
});
expect(result.target_limited).toBe(true);
expect(result.target_count).toBeGreaterThan(0);
expect(result.target_count).toBeLessThan(0x10000);
expect(result.invalid).toBe(true);
});
@@ -0,0 +1,138 @@
import { createAnalysisExecution } from "../../../src/application/AnalysisProvider.js";
import type { AnalysisProvider } from "../../../src/application/AnalysisProvider.js";
import { executeComposedWorkflow } from "../../../src/application/WorkflowEvidence.js";
import { createWorkflowEvidence } from "../../../src/application/WorkflowEvidence.js";
import { EnhancedTools } from "../../../src/application/EnhancedTools.js";
import { createAnalysisProfile } from "../../../src/domain/analysisProfile.js";
import { ok } from "../../../src/domain/result.js";
import {
createBinarySessionTargets,
createTestBinarySession,
} from "../../fixtures/binarySession.js";
import { expect, it } from "vitest";
const identity = {
id: "fixture",
name: "Fixture analysis provider",
version: "1",
} as const;
const effects = {
mutatesArtifact: false,
launchesProcess: false,
mayShowUi: false,
mayAccessNetwork: false,
mayWriteFilesystem: false,
changesPermissions: false,
requiresRoot: false,
} as const;
it("links backward caller observations into workflow Evidence without a document", async () => {
const forward = {
native_result: { callers: ["0x2000"], callers_complete: true },
forward_observations: {
call_references: [
{
caller_address: "0x2000",
source_address: "0x2004",
target_address: "0x1000",
source_procedure: "0x2000",
call: "direct",
},
],
},
};
const provider: AnalysisProvider = {
identity: () => identity,
resolveAnalysisProfile: () =>
Promise.resolve(
ok({ profile: createAnalysisProfile(identity, { fixture: true }) }),
),
capabilities: () =>
(["procedure_address", "procedure_callers"] as const).map(
(operation) => ({
provider: identity,
operation,
available: true,
reason: null,
effects,
limitations: [],
}),
),
createClient: () => ({
execute: (operation, parameters) => {
if (operation === "procedure_address")
return Promise.resolve(
ok(createAnalysisExecution(parameters.procedure, identity)),
);
const callers = parameters.procedure === "0x1000" ? ["0x2000"] : [];
return Promise.resolve(
ok(
createAnalysisExecution(callers, identity, {
rawResult:
parameters.procedure === "0x1000"
? forward
: {
native_result: callers,
forward_observations: { call_references: [] },
},
}),
),
);
},
close: () => Promise.resolve(ok(null)),
}),
};
const session = createTestBinarySession(provider);
const [target] = await createBinarySessionTargets();
if (target === undefined) throw new Error("Missing session target");
try {
const opened = await session.open(target);
if (!opened.ok) throw opened.error;
const composed = await executeComposedWorkflow(session, (observed) =>
new EnhancedTools(observed).execute("get_call_graph", {
address: "0x1000",
direction: "backward",
}),
);
if (!composed.ok) throw composed.error;
const workflow = createWorkflowEvidence({
target: opened.value,
operation: "get_call_graph",
parameters: { address: "0x1000", direction: "backward" },
result: composed.value.result,
upstreamProfile: session.analysisProfile(),
upstreamLimitations: composed.value.limitations,
...(composed.value.evidenceLinks === undefined
? {}
: { upstreamEvidenceLinks: composed.value.evidenceLinks }),
});
const caller = composed.value.evidenceLinks
?.map((id) => session.evidenceById(id))
.find(
(evidence) =>
evidence?.operation === "procedure_callers" &&
evidence.parameters.procedure === "0x1000",
);
expect(caller).toMatchObject({
parameters: { procedure: "0x1000" },
raw_result: forward,
});
expect(caller?.raw_result).toMatchObject({
forward_observations: {
call_references: [
{
caller_address: "0x2000",
source_address: "0x2004",
target_address: "0x1000",
},
],
},
});
expect(caller?.raw_result).not.toHaveProperty("native_result.instructions");
expect(caller?.parameters).not.toHaveProperty("document");
expect(workflow.evidence_links).toContain(caller?.evidence_id);
} finally {
await session.close();
}
});
@@ -1076,12 +1076,22 @@ describe("snapshot cache eligibility", () => {
const parameters: Readonly<Record<string, JsonValue>> = live
? { address: "0x1000", document: "main" }
: {};
const evidenceIds: Array<string | undefined> = [];
for (let attempt = 0; attempt < 2; attempt += 1) {
const result = await session.execute("address_name", parameters);
expect(result).toMatchObject({ ok: true });
if (result.ok) expect(result.value.evidenceId).toBeUndefined();
if (!result.ok) throw result.error;
evidenceIds.push(result.value.evidenceId);
}
const records = session.exportEvidenceBundle().records;
if (live) {
expect(evidenceIds).toEqual([expect.any(String), evidenceIds[0]]);
expect(records.map((record) => record.evidence_id)).toEqual(
expect.arrayContaining([source.evidence_id, evidenceIds[0]]),
);
} else {
expect(evidenceIds).toEqual([undefined, undefined]);
expect(records).toEqual([source]);
}
expect(session.exportEvidenceBundle().records).toEqual([source]);
expect(session.exportAnalysisSnapshot()).toMatchObject({
ok: true,
value: { entries: [] },
@@ -1122,20 +1132,20 @@ describe("snapshot cache eligibility", () => {
await session.close();
}
});
});
it("does not snapshot reads that depend on the provider cursor", async () => {
const [first] = await createBinarySessionTargets();
const calls: string[] = [];
const session = createTestBinarySession(createCacheProvider(calls));
await openAndExecuteAddressName(session, first, {});
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(calls).toEqual(["address_name", "address_name"]);
expect(session.exportAnalysisSnapshot()).toMatchObject({
ok: true,
value: { entries: [] },
});
await session.close();
it("does not snapshot reads that depend on the provider cursor", async () => {
const [first] = await createBinarySessionTargets();
const calls: string[] = [];
const session = createTestBinarySession(createCacheProvider(calls));
await openAndExecuteAddressName(session, first, {});
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(calls).toEqual(["address_name", "address_name"]);
expect(session.exportAnalysisSnapshot()).toMatchObject({
ok: true,
value: { entries: [] },
});
await session.close();
});
describe("provider health and availability", () => {
+40
View File
@@ -112,6 +112,46 @@ const enhancedFixtureResult = (method) => {
return [{ address: "0x1000", value: "fixture" }];
case "procedure_pseudo_code":
return "return 0;";
case "rea_caller_scan_target":
return {
target: {
address: "0x1000",
name: "fixture",
symbol: null,
classification: null,
body: {
available: false,
reason: "The fixture has no function body-extent observation.",
},
},
instructions: ["0x1000"],
instructions_complete: true,
};
case "rea_caller_base":
return {
target: {
address: "0x1000",
name: "fixture",
symbol: null,
classification: null,
body: {
available: false,
reason: "The fixture has no function body-extent observation.",
},
},
instructions: ["0x1000"],
instructions_complete: true,
callers: [],
callers_complete: true,
};
case "rea_caller_scan_page":
return {
start: 0,
next: 0,
total: 0,
oversized: false,
items: [],
};
case "analyze_function": {
return {
procedure: {
+68
View File
@@ -0,0 +1,68 @@
"""Exercise the real paging owner using Hopper-shaped native objects."""
import json
from pathlib import Path
import sys
class Procedure:
def __init__(self, address):
self.address = address
self.callees = []
self.calls = {}
def getEntryPoint(self):
return self.address
def getAllCalleeProcedures(self):
return self.callees
class Segment:
def __init__(self, procedures):
self.procedures = procedures
def getProcedureCount(self):
return len(self.procedures)
def getProcedureAtIndex(self, index):
return self.procedures[index]
class Document:
def __init__(self, segments):
self.segments = segments
def getSegmentsList(self):
return self.segments
path = sys.argv[1]
bridge = {"__file__": path, "__name__": "rea_hopper_bridge"}
exec(compile(Path(path).read_text(encoding="utf-8"), path, "exec"), bridge)
target = Procedure(0x2000)
procedures = [Procedure(0x3000 + 0x10 * i) for i in range(130)]
procedures[65].callees = [target]
procedures[65].calls = {(procedures[65].address + 4, target.address): {"classification": "direct", "provider_type": 2, "provenance": "fixture"}}
document = Document([Segment(procedures[:32]), Segment(procedures[32:])])
bridge["_procedure"] = lambda document, value: target
bridge["_instruction_addresses"] = lambda procedure: [procedure.address]
bridge["_procedure_identity"] = lambda procedure: {"address": hex(procedure.address)}
bridge["_iter_native_calls"] = lambda procedure, direction: ((source, destination, call) for (source, destination), call in procedure.calls.items())
bridge["_containing_procedure"] = lambda document, address: (target if address == target.address else procedures[65], None)
read = lambda start: bridge["_caller_scan_page"](document, {"start": start})
first, second, third = read(0), read(64), read(128)
procedures[65].callees = []
procedures[65].calls = {}
changed = read(64)
try:
read(131)
except bridge["InvalidRequestError"]:
invalid = True
else:
invalid = False
original_identity = bridge["_procedure_identity"]
bridge["_procedure_identity"] = lambda procedure: {"address": hex(procedure.address), "name": "x" * 600000}
oversized = read(0)
bridge["_procedure_identity"] = original_identity
target.basicBlockIterator = lambda: [None]
bridge["_block_instruction_addresses"] = lambda procedure, block: range(0x100000000, 0x100010000)
limited_target = bridge["_caller_target_data"](target)
print(json.dumps({
"first": {"start": first["start"], "next": first["next"], "total": first["total"], "items": len(first["items"])},
"second": {"start": second["start"], "next": second["next"], "total": second["total"], "items": len(second["items"]), "caller": second["items"][1]},
"third": {"start": third["start"], "next": third["next"], "total": third["total"], "items": len(third["items"])},
"changed": changed["items"][1], "invalid": invalid, "oversized": oversized,
"target_limited": not limited_target["instructions_complete"],
"target_count": len(limited_target["instructions"]),
}))