fix(analysis): retain source evidence through directory exports (#7250)

Retain authenticated provider source records independently of snapshot replay eligibility and return them explicitly to plain application callers, including late cancellation. CLI/MCP commit source records before delivering links; function dossiers share the canonical Evidence constructor. Keep directory replies compact and preserve successful retry verification.

Real Ghidra CLI/MCP investigations verify 652-function ls, historical report provenance, cross-session resume, unchanged source/project content and complete offline Evidence. Final full 14,338-function libcrypto comparison matches all metadata/pseudocode facts and independent fresh-import observations: 14,194 written, 143 external, one recorded deadline; 494.03 s, 445.63 CPU s, 1.584 GiB sampled family peak under 2-CPU/8-GiB guards, with no OOM or cleanup failure.

144 focused tests and static/docs/site checks pass; latest-main integration passes 289 cases plus typecheck/lint and actual CLI snapshot source retrieval. Final CI source/static/build/docs/quality/package checks pass. Four ADB shard failures also occur in main CI 40349559391; cancelled coverage remains unverified and author declarations stay unchecked under the user's explicit bypass authorization. No human-review attestation is made.
This commit is contained in:
gensokyo-dev authored and GitHub committed 2026-10-26 23:56:03 +08:00
1 parent 13fba271e6
commit 5508b2affd
15 files changed
+864 -43

No files matched your search

@@ -1,5 +1,9 @@
# Native directory decompilation verification — 2026-10-23
The [2026-10-26 follow-up](https://github.com/morluto/rea/blob/72454004772f826440c5e21002f4dd52b4ad43c0/experiments/analysis-efficiency/decompile-directory.md)
adds a 14,338-function real-provider capacity run, cross-session project reuse,
and a correction preserving historical source Evidence through shared workflows.
The completed analyst task is to export the bound provider's function inventory
as one local pseudocode file per non-external function, with names, body ranges,
provider/profile identity and the limitations of each used observation. The
@@ -0,0 +1,149 @@
# Directory decompilation: capacity, resume and source Evidence
The directory-export feature requested in #6090 already shipped in #6170.
This follow-up verifies the current public workflow and fixes a source-provenance
gap found while combining it with existing Ghidra projects (#1048).
## Completed task and correctness
The analyst asks for all functions as local pseudocode files, without putting
their bodies in agent context. The committed `decompile-directory.mjs` driver
uses the production stdio MCP entrypoint and installed SDK. It discovers the
catalog, exports an initial chunk, closes/reopens the provider, resumes all
remaining work, and resumes again to verify settled files. It authenticates an
explicitly exported Evidence bundle and checks the original target/project.
Every written file is checked against its index SHA-256 and address/profile
header. Existing file hashes and modification times must survive both resumes.
Comparison facts include every address, name, status, body range and pseudocode
digest. An independent fresh-import investigation from #1048 additionally checks
the complete inventory and two nonempty pseudocode observations. This establishes
export preservation of provider observations; it does not establish discovery of
every actual function or equivalence to original source.
| Real artifact | Provider inventory | Written | External | Failed |
| ----------------------- | -----------------: | ------: | -------: | -----: |
| `/usr/bin/ls` | 652 | 541 | 111 | 0 |
| system `libcrypto.so.3` | 14,338 | 14,194 | 143 | 1 |
Target SHA-256 values are respectively
`cb30d69b24245bf2ecdc9e7f53bbad19159999970b6d82c0c00c7d32d9e37aa4` and
`aa01e614af634bc7c8b678c9250b5c4f613c61989bacf0a5d6aed6570e7dcabd`.
These installed ELF x86-64 binaries were analyzed statically, never executed.
The large run explicitly selected a five-second per-function deadline; function
`0x387f00` exceeded it, remained a recorded failure, and was retried on resume.
Later functions were still exported. No disappearance of that failure is claimed.
The resulting large tree occupies about 107 MiB on this filesystem.
## Source-provenance correction
Before this change, `BinarySession` retained provider `sourceEvidence` only inside
its snapshot-eligible branch. Ghidra procedure reads without a document selector
are not snapshot eligible. A composed directory export therefore lost saved
project reports even though the provider supplied them. The workflow also ignored
source-record IDs, and the function-dossier adapter copied observation fields
manually without those links.
The session now authenticates/retains source records independently of replay
eligibility. Plain application helpers also return authenticated immutable source
records with their outcome, including late cancellation. Both adapters explicitly
commit them before delivering links; an adapter without a retention owner refuses
the response. Source collection borrows ledger records without constructing or
hashing an intermediate bundle. The workflow links distinct source IDs without copying their payloads.
The function adapter uses the common observation-to-Evidence constructor,
preserving sources through normal and late-cancelled completion. Queries with no
source records avoid the extra recording path. Complete provider bodies remain
in the exported function files; this does not add per-function payload retention.
A real test project contains historical seed and disabled-analyzer reports saved
by REA's actual Ghidra pre-scripts. The unmodified runtime completed the export
but failed the provenance oracle: its bundle had two workflow records and no
historical source. The fixed runtime retains exactly one historical source and
links the workflow to it. All 652 inventory entries and the sampled pseudocode
still match fresh import; source project content stays unchanged. The report is
historical, with remaining original settings/completeness explicitly unknown.
## Measured costs
Baseline runtime: `19447516a4bdbc9437d112ebd3f12a5fd062db83`.
Initial fixed runtime: `f50763fd21527bdf3bab944633617b9df61dded4`, rebased on
main `f02df45a4`. Both use Node 24.22.1, locked dependencies, actual Ghidra 12.1.4
and JDK 21.0.12.1+1. The fixed driver later delegates transcript publication to
the existing cleanup/receipt owner; the protocol and task are unchanged.
| Completed task | Wall s | Family CPU s | Peak family GiB | Full-wire token estimate |
| --------------------------------- | ------: | -----------: | --------------: | -----------------------: |
| ls, before | 51.411 | 78.984 | 0.943 | 57,082 |
| ls, initial fix | 50.937 | 79.891 | 1.201 | 57,072 |
| ls with historical reports, fixed | 52.827 | 80.236 | 1.097 | 57,207 |
| libcrypto, baseline capacity run | 506.387 | 459.818 | 1.502 | 57,322 |
The plain ls pair has identical complete comparison facts. Runtime is similar;
the higher sampled memory in the fixed run is reported rather than treated as an
improvement. These sequential shared-host samples establish no generic speed or
memory ratio. [PR #7250](https://github.com/morluto/rea/pull/7250) records the final
large-run comparison and CI/review results.
The failed historical baseline is excluded from completed-task comparisons.
The complete MCP replies stayed around 11–12 KiB, including repeated textual and
structured representations, profile and caveats. The normalized result is only
the manifest. For example, the large continuation costs about 3,159 estimated
wire tokens; its complete task is 57,322 because discovery alone costs 47,049.
Both numbers include actual protocol content, rather than only pseudocode bytes.
Estimates use `cl100k_base` on the actual JSON-RPC messages, including discovery,
requests, responses and notifications, excluding only transport framing and the
driver's local transcript wrapper. They are not actual model billing. Host
projection, prompts/reasoning and later agent file reads remain unmeasured.
The historical source increases response token cost slightly while preserving
provenance. No token-reduction claim is made for this correctness fix.
CPU and memory cover the driver, REA and provider process family. Each run uses
CPU affinity 0,1 and an 8-GiB sampled family-RSS guard; Java heaps are 1536M for ls
and 3072M for crypto. RSS is sampled every 100 ms and summed: shared pages may be
counted repeatedly and short peaks may be missed. CPU is waited child usage.
Reported completed runs had no resource stop, OOM or cleanup failure.
Project preparation is excluded and documented separately in
[project reuse](ghidra-project-reuse.md). Provider starts are process-cold, with
host filesystem caches unflushed; verified resume occurs in the second session.
The index and inventory still scale with function count. This 14,338-item run
does not establish a larger capacity guarantee. Real Windows/macOS filesystems,
other decompilers and native DWARF directory layout remain unverified. The current
workflow reports unavailable DWARF layout explicitly instead of inventing it.
Actual CLI `decompile-all` calls additionally selected only `0x10a420`, reported
651 excluded entries and `debug_layout_unavailable` for requested dwarf layout,
then resumed in a new process. The existing file's hash and modification time
stayed unchanged; the returned historical source ID stayed stable. This CLI check
verifies the returned manifest/link and local function file, not offline retrieval
of the separately retained source record after the CLI process exits.
## Reproduce and inspect
Prepare an owned project using the documented
[existing-project route](../../docs/installation.md#reuse-an-existing-ghidra-project).
Use fresh receipt/output paths; the driver refuses existing output directories.
```sh
node experiments/analysis-efficiency/decompile-directory.mjs \
--runtime /absolute/rea --output /absolute/new-receipt \
--target /usr/bin/ls --project /absolute/ls-analysis.gpr --program ls \
--ghidra "$GHIDRA_INSTALL_DIR" --java "$JAVA_HOME"
```
Use `--chunk 512 --heap 3072M --decompile-seconds 5` for the crypto case.
`--control /absolute/previous-receipt` compares all exported facts;
`--golden /absolute/fresh-import-receipt` checks the #1048 driver's inventory and
pseudocode oracle. `--history required` additionally requires a linked historical
source. The driver allows failed functions to succeed on a later retry, verifies only
previously written files remain unchanged, and reloads the final index before
comparison. Complete local receipts include the transcript, resource report, index,
function files, comparison facts and authenticated Evidence export. The resource
report is supplied by the external process-family sampler, not the driver.
The current nine-file suite passed 144 tests. Against unmodified main source,
the three source-retention/constructor suites reproduce seven failures and 70
passes; the fixed full suite passes. This includes invalid source commitments,
live and cursor-dependent reads, shared source deduplication, selected/null/provider
subjects, and source links on late-cancelled completion.
@@ -0,0 +1,328 @@
/** Real-provider directory export, cross-session resume and retained provenance. */
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { execFile } from "node:child_process";
import { createWriteStream } from "node:fs";
import { mkdir, mkdtemp, readFile, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { finished } from "node:stream/promises";
import { pathToFileURL } from "node:url";
import { parseArgs, promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
const { values } = parseArgs({
options: Object.fromEntries(
[
"runtime",
"output",
"target",
"project",
"program",
"ghidra",
"java",
"control",
"heap",
"chunk",
"decompile-seconds",
"history",
"golden",
].map((key) => [key, { type: "string" }]),
),
});
for (const key of [
"runtime",
"output",
"target",
"project",
"ghidra",
"java",
]) {
assert.ok(values[key], `${key} is required`);
values[key] = resolve(values[key]);
}
const chunk = Number(values.chunk ?? 32);
assert.ok(Number.isSafeInteger(chunk) && chunk > 0);
await mkdir(values.output);
const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
const report = {
calls: [],
runtime: values.runtime,
target: values.target,
project: values.project,
};
let client, transport, account, serverLog, serverLogFinished;
// This workflow has a fixed number of public calls; pseudocode stays on disk.
const transcript = [];
let phase = "initialize";
const started = performance.now();
const tree = join(values.output, "tree");
const readIndex = async () =>
JSON.parse(await readFile(join(tree, "decompile-index.json"), "utf8"));
const fileStates = async (index) => {
const states = {};
for (const row of index.functions.filter((row) => row.status === "written")) {
const path = join(tree, row.relative_path);
const metadata = await stat(path);
states[row.address] = {
sha256: digest(await readFile(path)),
mtime_ms: metadata.mtimeMs,
};
assert.equal(states[row.address].sha256, row.sha256);
}
return states;
};
try {
report.revision = (
await promisify(execFile)("git", [
"-C",
values.runtime,
"rev-parse",
"HEAD",
])
).stdout.trim();
report.target_sha256 = digest(await readFile(values.target));
const { resolveGhidraProject } = await import(
pathToFileURL(join(values.runtime, "dist/ghidra/GhidraProjectSnapshot.js"))
.href
);
const { ArtifactResourceScope } = await import(
pathToFileURL(
join(values.runtime, "dist/artifacts/ArtifactResourceScope.js"),
).href
);
const fingerprint = async () => {
const resources = new ArtifactResourceScope();
try {
return await resolveGhidraProject(
{
path: values.project,
...(values.program ? { programName: values.program } : {}),
},
{ path: values.target },
resources,
);
} finally {
await resources.close();
}
};
const original = await fingerprint();
report.project_sha256 = original.sha256;
account = await mkdtemp(join(tmpdir(), "rea-directory-investigation-"));
transport = new StdioClientTransport({
command: process.execPath,
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
cwd: values.runtime,
maxBufferSize: 10 * 1024 * 1024,
env: {
PATH: process.env.PATH,
HOME: account,
USERPROFILE: account,
JAVA_HOME: values.java,
GHIDRA_INSTALL_DIR: values.ghidra,
GHIDRA_HEADLESS_MAXMEM: values.heap ?? "1536M",
JAVA_TOOL_OPTIONS: "-XX:ActiveProcessorCount=2",
NODE_OPTIONS: process.env.NODE_OPTIONS ?? "--max-old-space-size=3072",
REA_GHIDRA_PROJECT_PATH: values.project,
...(values.program ? { REA_GHIDRA_PROGRAM_NAME: values.program } : {}),
REA_GHIDRA_STARTUP_TIMEOUT_MS: "600000",
...(values["decompile-seconds"]
? { REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS: values["decompile-seconds"] }
: {}),
REA_MCP_TOOL_FAMILIES: "direct,enhanced,application,session",
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
REA_MCP_OUTPUT_SCHEMA_PROFILE: "compact",
},
stderr: "pipe",
});
const start = transport.start.bind(transport);
transport.start = async () => {
const receive = transport.onmessage;
transport.onmessage = (message) => {
transcript.push({ phase, direction: "response", message });
receive?.(message);
};
await start();
};
const send = transport.send.bind(transport);
transport.send = async (message) => {
transcript.push({ phase, direction: "request", message });
await send(message);
};
serverLog = createWriteStream(join(values.output, "server.log"));
serverLogFinished = finished(serverLog);
serverLogFinished.catch(() => {});
transport.stderr.pipe(serverLog);
client = new Client({ name: "rea-directory-investigation", version: "1" });
await client.connect(transport);
phase = "discovery";
const catalog = await client.listTools({}, { timeout: 15000 });
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
report.tools = catalog.tools.length;
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
const call = async (name, args, label) => {
phase = label;
const at = performance.now();
const result = await client.callTool(
{ name, arguments: args },
{ timeout: 900000, toolDefinition: definitions.get(name) },
);
report.calls.push({
name,
phase,
elapsed_ms: performance.now() - at,
response_bytes: Buffer.byteLength(JSON.stringify(result)),
is_error: result.isError ?? false,
});
assert.notEqual(
result.isError,
true,
`${name}: ${JSON.stringify(result).slice(0, 800)}`,
);
return result.structuredContent;
};
const open = () =>
call("open_binary", { path: values.target, provider_id: "ghidra" }, "open");
await open();
const first = await call(
"decompile_all",
{ output_directory: tree, limit: chunk },
"first-chunk",
);
report.first = first.normalized_result;
const firstFiles = await fileStates(await readIndex());
await call("close_binary", {}, "close-first-session");
await open();
const remaining = await call(
"decompile_all",
{ output_directory: tree, resume: true },
"cross-session-resume",
);
report.resume = remaining.normalized_result;
report.resume_evidence_links = remaining.evidence_links?.length ?? 0;
let index = await readIndex();
const allFiles = await fileStates(index);
for (const [address, state] of Object.entries(firstFiles))
assert.deepEqual(allFiles[address], state);
report.first_files_unchanged = true;
const settled = await call(
"decompile_all",
{ output_directory: tree, resume: true },
"verified-resume",
);
report.verified_resume = settled.normalized_result;
if (report.resume.counts.failed === 0)
assert.equal(report.verified_resume.counts.written, 0);
assert.equal(report.verified_resume.counts.remaining, 0);
index = await readIndex();
const settledFiles = await fileStates(index);
for (const [address, state] of Object.entries(allFiles))
assert.deepEqual(settledFiles[address], state);
assert.equal(index.functions.length, report.resume.counts.selected);
const facts = [];
for (const row of index.functions) {
const fact = {
address: row.address,
name: row.name,
status: row.status,
body: row.body,
};
if (row.status === "written") {
const text = await readFile(join(tree, row.relative_path), "utf8");
assert.ok(text.includes(`// address: ${row.address}\n`));
assert.ok(
text.includes(
`// analysis_profile: ${index.analysis_profile_digest}\n`,
),
);
fact.pseudocode_sha256 = digest(text.slice(text.indexOf("\n\n") + 2));
}
facts.push(fact);
}
report.function_count = facts.length;
report.written = facts.filter((row) => row.status === "written").length;
report.failed = facts.filter((row) => row.status === "failed").length;
report.external = facts.filter((row) => row.status === "external").length;
report.facts_sha256 = digest(JSON.stringify(facts));
await writeFile(join(values.output, "facts.json"), JSON.stringify(facts));
const evidencePath = join(values.output, "complete-evidence.json");
await call(
"export_evidence_bundle",
{ path: evidencePath },
"complete-evidence-export",
);
await call("close_binary", {}, "close");
await client.close();
await transport.close();
const { parseEvidenceBundle } = await import(
pathToFileURL(join(values.runtime, "dist/domain/evidenceBundle.js")).href
);
const bundle = parseEvidenceBundle(
JSON.parse(await readFile(evidencePath, "utf8")),
);
report.evidence_records = bundle.records.length;
const retained = bundle.records.find(
(record) => record.evidence_id === remaining.evidence_id,
);
assert.ok(retained);
report.retained_links = retained.evidence_links.length;
const byId = new Set(bundle.records.map((record) => record.evidence_id));
for (const id of retained.evidence_links) assert.ok(byId.has(id));
report.all_source_links_resolve = true;
if (values.history === "required") {
const historical = bundle.records.filter(
(record) =>
record.operation === "health" &&
record.parameters.facet === "historical-project-analysis-reports",
);
assert.equal(historical.length, 1);
assert.equal(historical[0].normalized_result.origin, "caller-project");
assert.ok(retained.evidence_links.includes(historical[0].evidence_id));
report.historical_source_retained_and_linked = true;
}
if (values.golden) {
const golden = JSON.parse(
await readFile(join(values.golden, "facts.json"), "utf8"),
);
const labels = (rows) =>
rows
.map((row) => [row.address, row.name ?? row.value])
.sort((a, b) => a[0].localeCompare(b[0], "en"));
assert.deepEqual(labels(facts), labels(golden.procedures));
for (const item of golden.pseudocode) {
const fact = facts.find((row) => row.address === item.procedure);
assert.equal(fact?.pseudocode_sha256, digest(item.result));
}
report.fresh_import_inventory_and_sampled_pseudocode_match = true;
}
if (values.control) {
const control = JSON.parse(
await readFile(join(values.control, "facts.json"), "utf8"),
);
assert.deepEqual(facts, control);
report.control_facts_match = true;
}
assert.deepEqual(await fingerprint(), original);
assert.equal(digest(await readFile(values.target)), report.target_sha256);
report.source_unchanged = true;
report.outcome = "passed";
} catch (cause) {
report.outcome = "failed";
report.failure = String(cause);
process.exitCode = 1;
} finally {
report.elapsed_ms = performance.now() - started;
const published = await finalizeRetainedQueryReceipt({
report,
client,
transport,
account,
serverLog,
serverLogFinished,
output: values.output,
transcript,
});
if (!published) process.exitCode = 1;
}
+13 -2
View File
@@ -13,6 +13,7 @@ import { silentLogger } from "../logger.js";
import type { Logger } from "pino";
import { withEvidenceCallerInputPath } from "../domain/evidence.js";
import { createImmutableEvidenceBundle } from "../domain/evidenceBundle.js";
import { retainAnalysisResultSources } from "./investigation/recordEvidenceSources.js";
import type { Evidence } from "../domain/evidence.js";
import type { NativeToolName } from "../contracts/native/nativeToolContracts.js";
import {
@@ -570,12 +571,17 @@ const executeAnalysisTool = async (input: {
if (tool === "analyze_function") {
const { executeFunctionAnalysisEvidence } =
await import("./FunctionAnalysisEvidence.js");
const result = await executeFunctionAnalysisEvidence(
let result = await executeFunctionAnalysisEvidence(
session,
input.arguments,
input.openedTarget,
{ signal },
);
result = retainAnalysisResultSources(
tool,
(source) => session.recordEvidence(source),
result,
);
return result.ok ? ok({ evidence: result.value }) : result;
}
if (
@@ -588,12 +594,17 @@ const executeAnalysisTool = async (input: {
tool === "decompile_all"
) {
const { EnhancedTools } = await import("./EnhancedTools.js");
const result = await executeComposedWorkflow(
let result = await executeComposedWorkflow(
session,
(observed) =>
new EnhancedTools(observed).execute(tool, input.arguments, signal),
composedLimitationBudget(tool),
);
result = retainAnalysisResultSources(
tool,
(source) => session.recordEvidence(source),
result,
);
if (!result.ok) return result;
const evidence = createWorkflowEvidence({
target: input.openedTarget,
@@ -2,7 +2,7 @@ import { expect, it } from "vitest";
import { createAnalysisExecution } from "./AnalysisProvider.js";
import { executeFunctionAnalysisEvidence } from "./FunctionAnalysisEvidence.js";
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
import { parseEvidence } from "../domain/evidence.js";
import { createEvidence, parseEvidence } from "../domain/evidence.js";
import { createAnalysisProfile } from "../domain/analysisProfile.js";
import { ghidraFunctionDossier } from "../domain/ghidraValues.fixture.js";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
@@ -62,6 +62,16 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
path: "/observed/provider.elf",
sha256: "b".repeat(64),
};
const source = createEvidence(
undefined,
provider,
{
operation: "health",
parameters: { facet: "historical-report" },
result: { original: "unknown" },
},
{ immutable: true },
);
for (const aborted of [false, true]) {
const controller = new AbortController();
const execution = createAnalysisExecution(dossier, provider, {
@@ -73,6 +83,7 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
rawResult: { provider_text: "preserved" },
limitations: ["Synthetic provider observation"],
locations: [{ kind: "address", address: "0x401000" }],
sourceEvidence: [source],
});
const result = await executeFunctionAnalysisEvidence(
{
@@ -85,6 +96,7 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
target,
{ signal: controller.signal },
);
expect(result.sourceEvidence).toEqual([source]);
if (aborted) {
if (result.ok) throw new Error("Expected late cancellation");
expect(result.error).toMatchObject({ _tag: "AnalysisCancelledError" });
@@ -100,6 +112,7 @@ it.each(["selected-target", "no-subject", "provider-subject"] as const)(
expect(evidence.provider).toEqual(provider);
expect(evidence.analysis_profile).toEqual(analysisProfile);
expect(evidence.relationship_coverage).toEqual(relationshipCoverage);
expect(evidence.evidence_links).toEqual([source.evidence_id]);
expect(evidence.limitations).toContain("Synthetic provider observation");
expect(evidence.locations).toContainEqual({
kind: "address",
+30 -20
View File
@@ -5,10 +5,17 @@ import type {
} from "./AnalysisProvider.js";
import { nativeProcedureInputSchema } from "../domain/native/nativeProcedureInput.js";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
import { createEvidence } from "../domain/evidence.js";
import { createAnalysisExecutionEvidence } from "../domain/analysisObservation.js";
import { parseFunctionDossier } from "../domain/hopperValues.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { completeProviderEvidence } from "./ProviderOperationExecution.js";
import type { Evidence } from "../domain/evidence.js";
import { EvidenceLedger } from "./investigation/EvidenceLedger.js";
import {
recordEvidenceSources,
withEvidenceSources,
type AnalysisResultWithSources,
} from "./investigation/recordEvidenceSources.js";
/**
* Preserve the provider's observations when presenting a validated function
@@ -20,7 +27,7 @@ export const executeFunctionAnalysisEvidence = async (
parameters: Readonly<Record<string, JsonValue>>,
target: BinaryTarget | undefined,
options?: ExecutionOptions,
) => {
): Promise<AnalysisResultWithSources<Evidence>> => {
const input = parseAnalysisInput(
"analyze_function",
parameters,
@@ -35,30 +42,33 @@ export const executeFunctionAnalysisEvidence = async (
options,
);
if (!execution.ok) return execution;
const sourceLedger = new EvidenceLedger();
const recorded = recordEvidenceSources(
(source) => sourceLedger.record(source),
execution.value.sourceEvidence ?? [],
);
const sources = sourceLedger.recordsForAnalysis();
if (!recorded.ok) return withEvidenceSources(recorded, sources);
const dossier = parseFunctionDossier(execution.value.result);
if (!dossier.ok) return dossier;
if (!dossier.ok) return withEvidenceSources(dossier, sources);
const observation = execution.value;
return completeProviderEvidence(
createEvidence(
observation.subject === undefined
? target
: (observation.subject ?? undefined),
observation.provider,
return withEvidenceSources(
completeProviderEvidence(
createAnalysisExecutionEvidence(
"analyze_function",
arguments_,
{
operation: "analyze_function",
parameters: arguments_,
...observation,
sourceEvidence: sources,
result: dossier.value,
rawResult: observation.rawResult,
limitations: observation.limitations,
locations: observation.locations,
...(observation.relationshipCoverage === undefined
? {}
: { relationshipCoverage: observation.relationshipCoverage }),
...(observation.analysisProfile === undefined
? {}
: { analysisProfile: observation.analysisProfile }),
...(observation.subject === undefined && target !== undefined
? { subject: target }
: {}),
},
"observed",
),
options?.signal,
),
sources,
);
};
+42
View File
@@ -19,6 +19,48 @@ import { projectNativeApiInspection } from "./native/NativeApiInspection.js";
import { functionDossierSchema } from "../domain/hopperValues.js";
import { ghidraFunctionDossier } from "../domain/ghidraValues.fixture.js";
import { ANALYSIS_SNAPSHOT_PROFILE } from "../domain/analysisSnapshot.fixture.js";
import { createEvidence } from "../domain/evidence.js";
it("links shared provider source Evidence once without copying its payload into the workflow", async () => {
const source = createEvidence(
undefined,
{ id: "fixture", name: "Fixture", version: "1" },
{
operation: "health",
parameters: { facet: "saved-analysis-report" },
result: { report: "historical content ".repeat(1_000) },
},
{ immutable: true },
);
const result = await executeComposedWorkflow(
{
execute: async () =>
ok(
createAnalysisExecution(
"observed",
{ id: "fixture", name: "Fixture", version: "1" },
{ sourceEvidence: [source] },
),
),
},
async (observed) => {
await observed.execute("list_procedures", {});
await observed.execute("procedure_pseudo_code", { procedure: "0x1000" });
return ok({ count: 1 });
},
);
expect(result).toEqual({
...ok({
result: { count: 1 },
limitations: [],
evidenceLinks: [source.evidence_id],
}),
sourceEvidence: [source],
});
expect(result.ok && JSON.stringify(result.value)).not.toContain(
"historical content",
);
});
it.each([false, true])(
"propagates transient=%s through composed snapshot decisions while retaining Evidence",
+24 -8
View File
@@ -23,6 +23,12 @@ import {
import type { JsonValue } from "../domain/jsonValue.js";
import type { UnknownRegistryPort } from "./investigation/InvestigationRecordPort.js";
import { UnknownRegistryError } from "../domain/unknownRegistryError.js";
import { EvidenceLedger } from "./investigation/EvidenceLedger.js";
import {
recordEvidenceSources,
withEvidenceSources,
type AnalysisResultWithSources,
} from "./investigation/recordEvidenceSources.js";
import { workflowUnknownInput } from "./investigation/WorkflowUnknown.js";
import {
renderSourceLimitations,
@@ -71,18 +77,16 @@ export const executeComposedWorkflow = async (
) => Promise<Result<JsonValue, AnalysisError>>,
budget?: ComposedLimitationBudget,
): Promise<
Result<
{
AnalysisResultWithSources<{
readonly result: JsonValue;
readonly limitations: readonly string[];
readonly cachePolicy?: "live";
readonly evidenceLinks?: readonly string[];
},
AnalysisError
>
}>
> => {
const limitations = new Map<string, SourceLimitation>();
const evidenceLinks = new Set<string>();
const sourceLedger = new EvidenceLedger();
const perItemGroups = new Set<string>();
let perItemBytes = 0;
let omittedObservations = 0;
@@ -93,6 +97,13 @@ export const executeComposedWorkflow = async (
const execution = await analysis.execute(operation, parameters, options);
if (execution.ok && execution.value.evidenceId !== undefined)
evidenceLinks.add(execution.value.evidenceId);
if (execution.ok && execution.value.sourceEvidence !== undefined) {
const recorded = recordEvidenceSources(
(source) => sourceLedger.record(source),
execution.value.sourceEvidence,
);
if (!recorded.ok) return recorded;
}
if (execution.ok && execution.value.cachePolicy === "live")
cachePolicy = "live";
if (execution.ok)
@@ -129,8 +140,11 @@ export const executeComposedWorkflow = async (
return execution;
},
});
if (!result.ok) return result;
return ok({
const sources = sourceLedger.recordsForAnalysis();
if (!result.ok) return withEvidenceSources(result, sources);
for (const source of sources) evidenceLinks.add(source.evidence_id);
return withEvidenceSources(
ok({
result: result.value,
limitations: [
...renderSourceLimitations([...limitations.values()]),
@@ -144,7 +158,9 @@ export const executeComposedWorkflow = async (
...(evidenceLinks.size === 0
? {}
: { evidenceLinks: [...evidenceLinks].sort(compareUnicodeCodePoints) }),
});
}),
sources,
);
};
/** Build the shared Evidence record for a successful composed binary workflow. */
+9 -5
View File
@@ -1254,6 +1254,15 @@ export class BinarySession
)
this.#active.nativeSourceAdmitted = true;
this.#observeRuntimeAvailability(name, execution);
// Source facts belong to the observation, even when this query cannot be
// replayed (for example a Ghidra procedure read without a document selector).
if (execution.ok && execution.value.sourceEvidence !== undefined) {
const sourcesRecorded = recordEvidenceSources(
(source) => this.recordEvidence(source),
execution.value.sourceEvidence,
);
if (!sourcesRecorded.ok) return sourcesRecorded;
}
if (
execution.ok &&
cacheable &&
@@ -1276,11 +1285,6 @@ export class BinarySession
},
"observed",
);
const sourcesRecorded = recordEvidenceSources(
(source) => this.recordEvidence(source),
execution.value.sourceEvidence ?? [],
);
if (!sourcesRecorded.ok) return sourcesRecorded;
const recorded = this.recordEvidence(evidence);
if (!recorded.ok) return recorded;
const retained = { ...execution.value, evidenceId: evidence.evidence_id };
@@ -198,6 +198,11 @@ export class EvidenceLedger {
);
}
/** Borrow immutable source records without constructing or hashing a bundle. */
recordsForAnalysis(): readonly Evidence[] {
return [...this.#records.values()];
}
/** Check whether one immutable Evidence ID is present in this ledger. */
has(evidenceId: string): boolean {
return this.#records.has(evidenceId);
@@ -0,0 +1,32 @@
import { expect, it } from "vitest";
import { createEvidence } from "../../domain/evidence.js";
import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js";
import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
import { err, ok } from "../../domain/result.js";
import {
retainAnalysisResultSources,
withEvidenceSources,
} from "./recordEvidenceSources.js";
it.each([false, true])(
"reports source recording failure without replacing earlier cancellation (cancelled=%s)",
(cancelled) => {
const source = createEvidence(
undefined,
{ id: "fixture", name: "Fixture", version: "1" },
{ operation: "health", parameters: {}, result: "historical" },
);
const primary = new AnalysisCancelledError("analyze_function", {
cause: "caller",
});
const failure = new EvidenceIntegrityError("Recorder failed");
const result = retainAnalysisResultSources(
"analyze_function",
() => err(failure),
withEvidenceSources(cancelled ? err(primary) : ok("complete"), [source]),
);
expect(result.ok).toBe(false);
if (result.ok) throw new Error("Expected recording failure");
expect(result.error).toBe(cancelled ? primary : failure);
},
);
@@ -1,7 +1,49 @@
import type { EvidenceWriter } from "./InvestigationRecordPort.js";
import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
import type { Evidence } from "../../domain/evidence.js";
import { ok, type Result } from "../../domain/result.js";
import { err, ok, type Result } from "../../domain/result.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import { AnalysisCapabilityUnavailableError } from "../../domain/analysisErrorCore.js";
/** Source facts remain available even when a completed observation is cancelled. */
export type AnalysisResultWithSources<Value> = Result<Value, AnalysisError> & {
readonly sourceEvidence?: readonly Evidence[];
};
export const withEvidenceSources = <
Outcome extends Result<unknown, AnalysisError>,
>(
result: Outcome,
sources: readonly Evidence[],
): Outcome & { readonly sourceEvidence?: readonly Evidence[] } =>
sources.length === 0 ? result : { ...result, sourceEvidence: sources };
/** Never deliver links to records a plain-port adapter cannot retain. */
export const retainAnalysisResultSources = <Value>(
operation: string,
recordEvidence: EvidenceWriter["recordEvidence"] | undefined,
result: AnalysisResultWithSources<Value>,
): Result<Value, AnalysisError> => {
const sources = result.sourceEvidence ?? [];
const retained =
sources.length > 0 && recordEvidence === undefined
? err(
new AnalysisCapabilityUnavailableError(
"rea",
operation,
"Provider source Evidence requires an explicit retention owner.",
{
userMessage:
"Use an analysis session that retains provider source Evidence, or supply an Evidence recorder when registering this adapter.",
},
),
)
: recordEvidenceSources(recordEvidence, sources);
if (retained.ok) return result;
if (result.ok) return retained;
result.error.retainSecondaryFailure(retained.error);
return result;
};
/** Record source evidence in order, stopping at the first session rejection. */
export const recordEvidenceSources = (
+13 -2
View File
@@ -34,6 +34,7 @@ import { toolRegistrationOptions } from "./toolRegistrationOptions.js";
import { executeFunctionAnalysisEvidence } from "../application/FunctionAnalysisEvidence.js";
import type { WithAdmittedAnalysis } from "./analysisAdmission.js";
import type { EvidenceLookup } from "../application/EvidenceInputResolver.js";
import { retainAnalysisResultSources } from "../application/investigation/recordEvidenceSources.js";
import {
requireAnalysisViewRetention,
summarizeRetainedAnalysis,
@@ -266,7 +267,7 @@ const executeEnhancedTool = async (
progress,
}),
};
const result = await logToolExecution(registration.logger, name, () =>
let result = await logToolExecution(registration.logger, name, () =>
executeComposedWorkflow(
analysis,
(observed) =>
@@ -277,6 +278,11 @@ const executeEnhancedTool = async (
composedLimitationBudget(name),
),
);
result = retainAnalysisResultSources(
name,
registration.recordEvidence,
result,
);
await reportOperationEnd(progress, name, result.ok);
if (result.ok) {
const evidence = createWorkflowEvidence({
@@ -399,7 +405,7 @@ const deliverFunctionAnalysis = async (
},
): Promise<CallToolResult> => {
const { detail, parameters, context, progress } = request;
const result = await logToolExecution(
let result = await logToolExecution(
registration.logger,
"analyze_function",
() =>
@@ -413,6 +419,11 @@ const deliverFunctionAnalysis = async (
},
),
);
result = retainAnalysisResultSources(
"analyze_function",
registration.recordEvidence,
result,
);
await reportOperationEnd(progress, "analyze_function", result.ok);
if (!result.ok)
return registration.delivery.toCallToolResult(result, contract);
@@ -7,7 +7,13 @@ import { registerEnhancedTools } from "../../../src/server/registerEnhancedTools
import { createToolResultDelivery } from "../../../src/server/toolResult.js";
import { withAdmittedAnalysis } from "../../../src/server/analysisAdmission.js";
import type { AnalysisOperationPort } from "../../../src/application/AnalysisProvider.js";
import { parseEvidence, type Evidence } from "../../../src/domain/evidence.js";
import {
createEvidence,
parseEvidence,
type Evidence,
} from "../../../src/domain/evidence.js";
import { join } from "node:path";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import type { JsonValue } from "../../../src/domain/jsonValue.js";
import { ok } from "../../../src/domain/result.js";
import { functionDossierSchema } from "../../../src/domain/hopperValues.js";
@@ -29,22 +35,31 @@ const connect = async (
retain = true,
result = dossier,
path = () => "/fixtures/native.exe",
sources: readonly Evidence[] = [],
) => {
const ledger = new EvidenceLedger();
const record = (evidence: Evidence) => ledger.record(evidence);
const lookup = (id: string) => ledger.forAnalysis(id);
const execute = vi.fn(
async (operation: string, parameters: Record<string, JsonValue>) => {
if (operation !== "analyze_function")
throw new Error(`Unexpected operation: ${operation}`);
let observed: JsonValue = result;
if (operation === "analyze_function") {
const input = parseGhidraFunctionInput(operation, parameters);
if (!input.ok) return input;
} else if (operation === "list_procedures") {
observed = [{ address: "0x401000", value: "main" }];
} else if (operation === "procedure_info") {
observed = null;
} else if (operation === "procedure_pseudo_code") {
observed = "int main(void) { return 0; }";
} else throw new Error(`Unexpected operation: ${operation}`);
return ok({
result,
result: observed,
rawResult: null,
provider: { id: "ghidra", name: "Ghidra", version: "12.1.4" },
limitations: [],
locations: [],
sourceEvidence: sources,
subject: {
path: path(),
format: "pe" as const,
@@ -91,6 +106,62 @@ const connect = async (
return { client, ledger, execute };
};
it.each(["analyze_function", "decompile_all"] as const)(
"retains plain-port source records before delivering %s links",
async (operation) => {
const source = createEvidence(
undefined,
{ id: "ghidra", name: "Ghidra", version: "12.1.4" },
{
operation: "health",
parameters: { facet: "historical-report" },
result: { origin: "caller-project" },
},
{ immutable: true },
);
const { client, ledger } = await connect(true, dossier, undefined, [
source,
]);
const parent = await createTestTempDirectory("rea-plain-source-");
const reply = await client.callTool({
name: operation,
arguments:
operation === "analyze_function"
? { procedure: "main", detail: "complete" }
: { output_directory: join(parent, "out") },
});
expect(reply.isError).not.toBe(true);
const evidence = parseEvidence(reply.structuredContent);
expect(evidence.evidence_links).toContain(source.evidence_id);
expect(JSON.stringify(reply)).not.toContain("caller-project");
expect(ledger.forAnalysis(source.evidence_id)).toEqual(source);
expect(
ledger
.immutableBundle()
.records.filter((record) => record.evidence_id === source.evidence_id),
).toHaveLength(1);
},
);
it("refuses source links when a plain-port adapter has no retention owner", async () => {
const source = createEvidence(
undefined,
{ id: "fixture", name: "Fixture", version: "1" },
{
operation: "health",
parameters: {},
result: "historical",
},
);
const { client } = await connect(false, dossier, undefined, [source]);
const reply = await client.callTool({
name: "analyze_function",
arguments: { procedure: "main", detail: "complete" },
});
expect(reply.isError).toBe(true);
expect(JSON.stringify(reply)).toContain("retains provider source Evidence");
});
it("keeps complete and summary delivery on the current observed path despite retained same-ID history", async () => {
let selected = "/old/native.exe";
const { client, ledger } = await connect(true, dossier, () => selected);
@@ -968,6 +968,89 @@ describe("snapshot switch rollback", () => {
});
describe("snapshot cache eligibility", () => {
it.each([false, true])(
"retains source Evidence independently of snapshot eligibility (live=%s)",
async (live) => {
const [target] = await createBinarySessionTargets();
const provider = createCacheProvider([]);
const capabilities = provider.capabilities();
provider.capabilities = () =>
capabilities.map((capability) => ({
...capability,
...(live ? { cachePolicy: "live" as const } : {}),
}));
const source = createEvidence(
undefined,
provider.identity(),
{
operation: "health",
parameters: { facet: "saved-analysis-report" },
result: { original_analysis: "unknown" },
},
{ immutable: true },
);
provider.createClient = () => ({
execute: async () =>
resultOk(
createAnalysisExecution("name", provider.identity(), {
sourceEvidence: [source],
}),
),
close: async () => resultOk(null),
});
const session = createTestBinarySession(provider);
try {
expect((await session.open(target)).ok).toBe(true);
const parameters: Readonly<Record<string, JsonValue>> = live
? { address: "0x1000", document: "main" }
: {};
for (let attempt = 0; attempt < 2; attempt += 1) {
const result = await session.execute("address_name", parameters);
expect(result).toMatchObject({ ok: true });
if (result.ok) expect(result.value.evidenceId).toBeUndefined();
}
expect(session.exportEvidenceBundle().records).toEqual([source]);
expect(session.exportAnalysisSnapshot()).toMatchObject({
ok: true,
value: { entries: [] },
});
} finally {
await session.close();
}
},
);
it("rejects invalid source Evidence on a non-replayable observation", async () => {
const [target] = await createBinarySessionTargets();
const provider = createCacheProvider([]);
const source = createEvidence(undefined, provider.identity(), {
operation: "health",
parameters: {},
result: "original",
});
provider.createClient = () => ({
execute: async () =>
resultOk(
createAnalysisExecution("name", provider.identity(), {
sourceEvidence: [{ ...source, normalized_result: "tampered" }],
}),
),
close: async () => resultOk(null),
});
const session = createTestBinarySession(provider);
try {
expect((await session.open(target)).ok).toBe(true);
const result = await session.execute("address_name", {});
expect(result).toMatchObject({
ok: false,
error: { _tag: "EvidenceIntegrityError" },
});
expect(session.exportEvidenceBundle().records).toEqual([]);
} finally {
await session.close();
}
});
it("does not snapshot reads that depend on the provider cursor", async () => {
const [first] = await createBinarySessionTargets();
const calls: string[] = [];