mirror of
https://github.com/morluto/rea.git
synced 2026-11-21 22:15:50 +00:00
feat(ida): support the official Hex-Rays IDA MCP headlessly (#4813)
* feat(ida): support the official Hex-Rays IDA MCP headlessly Hex-Rays IDA MCP serves database reads only through execute_python, so REA rejected it as an upstream. A `hexrays` headless stdio registration now leases a private idalib worker through open_database, answers the existing database-supervisor reads with a fixed, packaged, read-only query module, and releases the lease with close_database, verified by list_databases. Observations record execute_python, the query and the module digest. Tool errors are typed so a completed rejection is distinguished from an unknown outcome; an unconfirmed release is retried and a late lease on the private input is adopted. Validated IDA source and bridge tests, static checks, docs, and npm run verify:ida on Linux x64 against x86-64 and AArch64 ELF fixtures. * feat(ida): name the Hex-Rays registration for its catalog A registration without `protocol` that reaches the official Hex-Rays server failed with only the missing database-supervisor tools. Startup profile failures now report the advertised catalog, and when it is the Hex-Rays one the message and recovery name the `hexrays` registration. Refs #1416 * docs(ida): state Hex-Rays prerequisites and upstream session traces Upstream needs IDA 9.4 or later with idalib, Python 3.11 or later, uv and Git. The profile does not analyze databases open in the IDA GUI, and the server keeps its own session traces outside REA's workspace. Refs #1416 * fix(ida): skip inline data when scanning Hex-Rays calls The call scan decoded every head that FuncItems yields. Inline data such as a jump table can decode as a call and refer to a function or import, which reported false callees and call-graph edges. Skip non-code heads before decoding, as disassembly already does. Refs #1416 * fix(ida): retain Hex-Rays health reads in evidence The identity and readiness reads around each operation kept only their parsed result, dropping the bridge digest and any engine output. Keep them whole as health_observations beside the parsed metadata, in both successful and partial-failure Evidence. Refs #1416 * test(ida): list the Hex-Rays bridge in package fixtures The packed-bridge verifier has required bridge/ida/hexrays_queries.py since the Hex-Rays profile landed, but three package tests still built their member lists without it. The verifier stopped at that file, so 21 cases failed or asserted on the wrong omission. Refs #1416 * feat(ida): open existing IDA databases through the Hex-Rays profile A headless open re-ran auto-analysis from the input binary, discarding the names, types and comments of an existing database and taking far longer on large targets. Target admission now identifies IDA databases by their IDA1/IDA2 signature, and the hexrays profile opens a private copy of one; Hopper and the other IDA profiles reject them. The copy is required, not incidental: idalib repacks a database when its last lease is released, even without a save. A database reports the input it was created from, so the session is bound to the opened copy's idb_path, and health now reports the input SHA-256 it records. Validated with resolver, provider and session tests, check:fast, docs:check and test:changed, plus a real Hex-Rays MCP run against a 1.4 GB SC2 .i64 whose original file was left byte-identical. Refs #1416 * fix(ida): verify the private workspace of a database target by idb_path * test(ida): fix Hex-Rays health stand-ins and Windows path and runtime teardown * refactor(ida): resolve the registration once for target support * docs(ida): record the real Hex-Rays run on an existing .i64 * docs(ida): make the Hex-Rays registration example install-neutral * chore: keep merge-hook formatting outside the IDA change * fix(ida): retain live Hex-Rays databases this server does not lease A list_databases row with instance_id null is a live owner, not a stopped database. Mapping it to the supervisor's released-worker shape made cleanup delete files another client still held. Unleased rows stay active, so the private path must leave inventory before the workspace is removed. Accept idat inventory rows, hash the query bridge with sha256Hex, and complete databaseEngine on Hopper fixtures added on main. * fix(ida): report uncapped Hex-Rays call graphs * docs(ida): link to the Hex-Rays bridge on GitHub * fix(ida): verify Hex-Rays worker identity and cleanup * feat(ida): support Hex-Rays idat headless workers Refs #12921 * fix(ida): retain workspace after ambiguous Hex-Rays startup * fix(ida): preserve typed Hex-Rays call edges and deadlines * test(ida): verify Hex-Rays cleanup with a peer lease * test(ci): use project deadlines for the Cutter verifier * test(ida): keep verifier cleanup fixtures scoped to mocked modules * fix(ida): retain the worker backend in partial evidence --------- Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: morluto <morluto@users.noreply.github.com> Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
This commit is contained in:
42 files changed
+2627
-194
No files matched your search
@@ -0,0 +1,8 @@
|
||||
export function privateIdaInput(
|
||||
metadata: {
|
||||
readonly input_path: string;
|
||||
readonly idb_path: string | null;
|
||||
},
|
||||
target: string,
|
||||
targetIsDatabase: boolean,
|
||||
): string;
|
||||
@@ -0,0 +1,13 @@
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
/**
|
||||
* Path of REA's private copy in a headless session's metadata. A database
|
||||
* records the executable it was built from, so its copy is the opened
|
||||
* database, as IdaSessionClient binds it.
|
||||
*/
|
||||
export function privateIdaInput(metadata, target, targetIsDatabase) {
|
||||
const path = targetIsDatabase ? metadata.idb_path : metadata.input_path;
|
||||
assert.equal(typeof path, "string");
|
||||
assert.notEqual(path, target);
|
||||
return path;
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import { GHIDRA_BRIDGE_SOURCES } from "../src/ghidra/GhidraBridgeSources.ts";
|
||||
|
||||
const packedHopperBridge = "package/bridge/hopper_bridge.py";
|
||||
const packedCutterBridge = "package/bridge/cutter/rea_bridge.py";
|
||||
const packedHexRaysBridge = "package/bridge/ida/hexrays_queries.py";
|
||||
const root = fileURLToPath(new URL("../", import.meta.url));
|
||||
const packedAgentEnvSources = Object.values(AGENTENV_BRIDGE_SOURCES).map(
|
||||
(url) => `package/${relative(root, fileURLToPath(url)).split(sep).join("/")}`,
|
||||
@@ -23,6 +24,7 @@ export async function verifyPackedBridge({ tarball, packedFiles, commands }) {
|
||||
for (const path of [
|
||||
packedHopperBridge,
|
||||
packedCutterBridge,
|
||||
packedHexRaysBridge,
|
||||
...packedAgentEnvSources,
|
||||
...Object.values(GHIDRA_BRIDGE_SOURCES).map(
|
||||
(name) => `package/bridge/ghidra/${name}`,
|
||||
@@ -33,20 +35,21 @@ export async function verifyPackedBridge({ tarball, packedFiles, commands }) {
|
||||
// Read the packed source without extracting into the caller's leased root.
|
||||
// Only the interpreter's standard library is needed; caller-local modules
|
||||
// and site startup hooks must not replace the parser or execute here.
|
||||
await commands.exec("python3", [
|
||||
"-I",
|
||||
"-S",
|
||||
"-c",
|
||||
[
|
||||
"import ast, sys, tarfile",
|
||||
"with tarfile.open(sys.argv[1]) as archive:",
|
||||
" member = archive.getmember(sys.argv[2])",
|
||||
" if not member.isfile():",
|
||||
" raise ValueError(f'{member.name} is not a regular file')",
|
||||
" with archive.extractfile(member) as source:",
|
||||
" ast.parse(source.read().decode('utf-8'), filename=sys.argv[2])",
|
||||
].join("\n"),
|
||||
tarball,
|
||||
packedHopperBridge,
|
||||
]);
|
||||
for (const path of [packedHopperBridge, packedHexRaysBridge])
|
||||
await commands.exec("python3", [
|
||||
"-I",
|
||||
"-S",
|
||||
"-c",
|
||||
[
|
||||
"import ast, sys, tarfile",
|
||||
"with tarfile.open(sys.argv[1]) as archive:",
|
||||
" member = archive.getmember(sys.argv[2])",
|
||||
" if not member.isfile():",
|
||||
" raise ValueError(f'{member.name} is not a regular file')",
|
||||
" with archive.extractfile(member) as source:",
|
||||
" ast.parse(source.read().decode('utf-8'), filename=sys.argv[2])",
|
||||
].join("\n"),
|
||||
tarball,
|
||||
path,
|
||||
]);
|
||||
}
|
||||
+171
-14
@@ -3,6 +3,7 @@ import {
|
||||
assertFeatureTrace,
|
||||
findMappedNonFunction,
|
||||
} from "./lib/ida-feature-trace.mjs";
|
||||
import { privateIdaInput } from "./lib/ida-private-input.mjs";
|
||||
import { requireMcpToolError } from "./lib/mcp-verifier-results.mjs";
|
||||
import {
|
||||
verifierFailureReasons,
|
||||
@@ -13,8 +14,8 @@ import { withVerifierResourceOwner } from "./lib/verifier-file-resources.mjs";
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createReadStream } from "node:fs";
|
||||
import { access, writeFile } from "node:fs/promises";
|
||||
import { basename, dirname, join, resolve } from "node:path";
|
||||
import { access, readdir, writeFile } from "node:fs/promises";
|
||||
import { basename, dirname, isAbsolute, join, resolve } from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
@@ -29,6 +30,8 @@ const { values } = parseArgs({
|
||||
"second-marker": { type: "string" },
|
||||
"expected-string": { type: "string" },
|
||||
"expected-xref": { type: "string" },
|
||||
"expected-backend": { type: "string" },
|
||||
"verify-peer-cleanup": { type: "boolean" },
|
||||
"package-root": { type: "string" },
|
||||
report: { type: "string" },
|
||||
},
|
||||
@@ -61,6 +64,10 @@ const [
|
||||
{ readIdaConfiguration, idaCompatibilityProfile },
|
||||
{ createIdaMcpConnection },
|
||||
{ idaAddressSchema },
|
||||
{ hasIdaDatabaseSignature },
|
||||
{ hexRaysOpenSchema, hexRaysListSchema, hexRaysCloseSchema },
|
||||
{ sameIdaHostPath },
|
||||
{ isPathWithinRoot },
|
||||
] = await Promise.all([
|
||||
importBuilt("contracts/toolContracts.js"),
|
||||
importBuilt("domain/evidence.js"),
|
||||
@@ -68,6 +75,10 @@ const [
|
||||
importBuilt("ida/IdaConfiguration.js"),
|
||||
importBuilt("ida/IdaMcpConnection.js"),
|
||||
importBuilt("ida/IdaProtocolValues.js"),
|
||||
importBuilt("domain/analysisDatabase.js"),
|
||||
importBuilt("ida/IdaHexRaysProtocol.js"),
|
||||
importBuilt("ida/IdaWorkspace.js"),
|
||||
importBuilt("domain/localPath.js"),
|
||||
]);
|
||||
const config = await withVerifierResourceOwner(async (resourceScope) => {
|
||||
const registration = await readIdaConfiguration(
|
||||
@@ -85,6 +96,19 @@ if (
|
||||
throw new Error(
|
||||
"Native IDA verification requires --second-procedure and --second-marker to verify distinct-function switching; see docs/providers/ida-provider.md.",
|
||||
);
|
||||
const expectedBackend = values["expected-backend"];
|
||||
if (
|
||||
expectedBackend !== undefined &&
|
||||
(config.protocol !== "hexrays" ||
|
||||
(expectedBackend !== "idat" && expectedBackend !== "idalib"))
|
||||
)
|
||||
throw new Error(
|
||||
"--expected-backend requires a Hex-Rays registration and an idat or idalib value.",
|
||||
);
|
||||
if (values["verify-peer-cleanup"] && config.protocol !== "hexrays")
|
||||
throw new Error(
|
||||
"--verify-peer-cleanup requires a Hex-Rays stdio registration.",
|
||||
);
|
||||
const environment = Object.fromEntries(
|
||||
Object.entries({
|
||||
...process.env,
|
||||
@@ -96,6 +120,7 @@ const report = {
|
||||
passed: false,
|
||||
mode: config.mode,
|
||||
protocol: idaCompatibilityProfile(config).protocol,
|
||||
worker_backends: [],
|
||||
checks: [],
|
||||
observations: [],
|
||||
unverified: [],
|
||||
@@ -106,6 +131,7 @@ const expectedXref =
|
||||
: idaAddressSchema.parse(values["expected-xref"]);
|
||||
let phase = "target admission";
|
||||
const workspaces = new Set();
|
||||
let targetIsDatabase = false;
|
||||
const databases = new Set();
|
||||
const digest = async () => {
|
||||
const hash = createHash("sha256");
|
||||
@@ -124,12 +150,41 @@ const observe = (input) => {
|
||||
assert.ok(
|
||||
evidence.locations.some((location) => location.kind === "artifact-path"),
|
||||
);
|
||||
if (config.protocol === "hexrays") {
|
||||
const backend = evidence.raw_result.worker_backend;
|
||||
assert.ok(
|
||||
backend === "idalib" || backend === "idat",
|
||||
"Hex-Rays evidence must identify the bound worker backend.",
|
||||
);
|
||||
if (expectedBackend !== undefined)
|
||||
assert.equal(
|
||||
backend,
|
||||
expectedBackend,
|
||||
"Hex-Rays worker backend differed from the requested verification lane.",
|
||||
);
|
||||
if (!report.worker_backends.includes(backend))
|
||||
report.worker_backends.push(backend);
|
||||
}
|
||||
if (config.mode === "headless") {
|
||||
assert.notEqual(evidence.raw_result.metadata_before.input_path, target);
|
||||
workspaces.add(dirname(evidence.raw_result.metadata_before.input_path));
|
||||
workspaces.add(
|
||||
dirname(
|
||||
privateIdaInput(
|
||||
evidence.raw_result.metadata_before,
|
||||
target,
|
||||
targetIsDatabase,
|
||||
),
|
||||
),
|
||||
);
|
||||
for (const observation of evidence.raw_result.observations) {
|
||||
assert.equal(typeof observation.arguments.database, "string");
|
||||
databases.add(observation.arguments.database);
|
||||
// Hex-Rays reads select their lease by instance id through execute_python.
|
||||
const database =
|
||||
config.protocol === "hexrays"
|
||||
? observation.arguments.instance_id
|
||||
: observation.arguments.database;
|
||||
if (config.protocol === "hexrays")
|
||||
assert.equal(observation.tool, "execute_python");
|
||||
assert.equal(typeof database, "string");
|
||||
databases.add(database);
|
||||
}
|
||||
}
|
||||
report.observations.push(evidence);
|
||||
@@ -173,6 +228,68 @@ const cli = async () => {
|
||||
report.checks.push("production CLI and complete function Evidence");
|
||||
return dossier;
|
||||
};
|
||||
const verifyPeerCleanup = async (analysis, closeWithPeer, retryClose) => {
|
||||
phase = "Hex-Rays peer lease cleanup";
|
||||
const privateInput = privateIdaInput(
|
||||
analysis.raw_result.metadata_before,
|
||||
target,
|
||||
targetIsDatabase,
|
||||
);
|
||||
const workspace = dirname(privateInput);
|
||||
assert.ok(workspaces.has(workspace));
|
||||
const peer = createIdaMcpConnection(config, process.env, {
|
||||
platform: process.platform,
|
||||
});
|
||||
let instanceId;
|
||||
const releasePeer = async () => {
|
||||
if (instanceId === undefined) return;
|
||||
hexRaysCloseSchema.parse(
|
||||
await peer.call("close_database", { instance_id: instanceId }),
|
||||
);
|
||||
instanceId = undefined;
|
||||
};
|
||||
|
||||
await withVerifierCleanup(async () => {
|
||||
await peer.connect();
|
||||
const lease = hexRaysOpenSchema.parse(
|
||||
await peer.call("open_database", {
|
||||
path: privateInput,
|
||||
set_current: false,
|
||||
}),
|
||||
);
|
||||
instanceId = lease.instance_id;
|
||||
assert.equal(lease.backend, analysis.raw_result.worker_backend);
|
||||
|
||||
const retained = requireMcpToolError(await closeWithPeer());
|
||||
assert.equal(retained.code, "cleanup_incomplete");
|
||||
const resources = retained.details?.resources;
|
||||
assert.ok(
|
||||
Array.isArray(resources) && resources.includes(workspace),
|
||||
"The cleanup error must identify the retained workspace.",
|
||||
);
|
||||
await access(privateInput);
|
||||
const inventory = hexRaysListSchema.parse(
|
||||
await peer.call("list_databases", {}),
|
||||
);
|
||||
const held = inventory.instances.find(
|
||||
(row) => row.instance_id === instanceId,
|
||||
);
|
||||
assert.ok(held?.status === "current" || held?.status === "attached");
|
||||
assert.equal(held.backend, lease.backend);
|
||||
assert.ok(
|
||||
sameIdaHostPath(held.path, privateInput),
|
||||
"The peer lease must still bind the private input.",
|
||||
);
|
||||
|
||||
await releasePeer();
|
||||
await retryClose();
|
||||
await assert.rejects(access(workspace), { code: "ENOENT" });
|
||||
report.checks.push(
|
||||
"a second Hex-Rays client retains the private database until its lease is released and REA retries cleanup",
|
||||
);
|
||||
}, [releasePeer, () => peer.close()]);
|
||||
};
|
||||
|
||||
const mcp = async (cliDossier) => {
|
||||
const client = new Client({ name: "rea-real-ida-verifier", version: "1" });
|
||||
const transport = await createOwnedMcpStdioTransport({
|
||||
@@ -384,7 +501,23 @@ const mcp = async (cliDossier) => {
|
||||
});
|
||||
assert.equal(invalid.isError, true);
|
||||
assert.equal(requireMcpToolError(invalid).code, "invalid_request");
|
||||
await call("close_binary", {});
|
||||
if (values["verify-peer-cleanup"])
|
||||
await verifyPeerCleanup(
|
||||
analyzed,
|
||||
() =>
|
||||
client.callTool(
|
||||
{ name: "close_binary", arguments: {} },
|
||||
{ timeout: config.timeoutMs * 2 },
|
||||
),
|
||||
() => call("close_binary", {}),
|
||||
);
|
||||
else {
|
||||
await call("close_binary", {});
|
||||
if (config.protocol === "hexrays")
|
||||
report.unverified.push(
|
||||
"Peer-held Hex-Rays cleanup: --verify-peer-cleanup was not selected.",
|
||||
);
|
||||
}
|
||||
opened = false;
|
||||
report.checks.push(
|
||||
"one MCP connection, positive procedure inventory/search, contract-valid tool results, invalid input, and close",
|
||||
@@ -419,12 +552,26 @@ const cleanup = async () => {
|
||||
"existing GUI target remains open with the original input identity",
|
||||
);
|
||||
} else {
|
||||
const inventory = await upstream.call("idb_list", {});
|
||||
assert.ok(
|
||||
inventory.sessions.every(
|
||||
(session) => !databases.has(session.session_id),
|
||||
),
|
||||
);
|
||||
if (config.protocol === "hexrays") {
|
||||
// Leases are per server process, so a fresh server proves release by
|
||||
// listing no live database inside any REA private workspace.
|
||||
const inventory = await upstream.call("list_databases", {});
|
||||
assert.ok(
|
||||
inventory.instances.every(({ path }) =>
|
||||
[...workspaces].every(
|
||||
(workspace) =>
|
||||
!isAbsolute(path) || !isPathWithinRoot(workspace, path),
|
||||
),
|
||||
),
|
||||
);
|
||||
} else {
|
||||
const inventory = await upstream.call("idb_list", {});
|
||||
assert.ok(
|
||||
inventory.sessions.every(
|
||||
(session) => !databases.has(session.session_id),
|
||||
),
|
||||
);
|
||||
}
|
||||
for (const workspace of workspaces)
|
||||
await assert.rejects(access(workspace), { code: "ENOENT" });
|
||||
report.checks.push(
|
||||
@@ -437,10 +584,17 @@ const cleanup = async () => {
|
||||
};
|
||||
try {
|
||||
const before = await digest();
|
||||
for await (const head of createReadStream(target, { end: 3 }))
|
||||
targetIsDatabase = hasIdaDatabaseSignature(head);
|
||||
const siblings = (await readdir(dirname(target))).sort();
|
||||
await mcp(await cli());
|
||||
await cleanup();
|
||||
assert.equal(await digest(), before);
|
||||
report.checks.push("original binary unchanged and never executed");
|
||||
// Engines write databases beside their input; only the private copy may get one.
|
||||
assert.deepEqual((await readdir(dirname(target))).sort(), siblings);
|
||||
report.checks.push(
|
||||
"original target bytes and sibling directory entries unchanged",
|
||||
);
|
||||
report.passed = true;
|
||||
} catch (cause) {
|
||||
report.failure = {
|
||||
@@ -463,6 +617,9 @@ try {
|
||||
{
|
||||
passed: report.passed,
|
||||
mode: report.mode,
|
||||
...(report.worker_backends.length === 0
|
||||
? {}
|
||||
: { worker_backends: report.worker_backends }),
|
||||
checks: report.checks,
|
||||
unverified: report.unverified,
|
||||
...(report.passed
|
||||
|
||||
Reference in new issue
Block a user