mirror of
https://github.com/morluto/rea.git
synced 2026-11-22 22:45:47 +00:00
* feat(ida): support the official Hex-Rays IDA MCP headlessly Hex-Rays IDA MCP serves database reads only through execute_python, so REA rejected it as an upstream. A `hexrays` headless stdio registration now leases a private idalib worker through open_database, answers the existing database-supervisor reads with a fixed, packaged, read-only query module, and releases the lease with close_database, verified by list_databases. Observations record execute_python, the query and the module digest. Tool errors are typed so a completed rejection is distinguished from an unknown outcome; an unconfirmed release is retried and a late lease on the private input is adopted. Validated IDA source and bridge tests, static checks, docs, and npm run verify:ida on Linux x64 against x86-64 and AArch64 ELF fixtures. * feat(ida): name the Hex-Rays registration for its catalog A registration without `protocol` that reaches the official Hex-Rays server failed with only the missing database-supervisor tools. Startup profile failures now report the advertised catalog, and when it is the Hex-Rays one the message and recovery name the `hexrays` registration. Refs #1416 * docs(ida): state Hex-Rays prerequisites and upstream session traces Upstream needs IDA 9.4 or later with idalib, Python 3.11 or later, uv and Git. The profile does not analyze databases open in the IDA GUI, and the server keeps its own session traces outside REA's workspace. Refs #1416 * fix(ida): skip inline data when scanning Hex-Rays calls The call scan decoded every head that FuncItems yields. Inline data such as a jump table can decode as a call and refer to a function or import, which reported false callees and call-graph edges. Skip non-code heads before decoding, as disassembly already does. Refs #1416 * fix(ida): retain Hex-Rays health reads in evidence The identity and readiness reads around each operation kept only their parsed result, dropping the bridge digest and any engine output. Keep them whole as health_observations beside the parsed metadata, in both successful and partial-failure Evidence. Refs #1416 * test(ida): list the Hex-Rays bridge in package fixtures The packed-bridge verifier has required bridge/ida/hexrays_queries.py since the Hex-Rays profile landed, but three package tests still built their member lists without it. The verifier stopped at that file, so 21 cases failed or asserted on the wrong omission. Refs #1416 * feat(ida): open existing IDA databases through the Hex-Rays profile A headless open re-ran auto-analysis from the input binary, discarding the names, types and comments of an existing database and taking far longer on large targets. Target admission now identifies IDA databases by their IDA1/IDA2 signature, and the hexrays profile opens a private copy of one; Hopper and the other IDA profiles reject them. The copy is required, not incidental: idalib repacks a database when its last lease is released, even without a save. A database reports the input it was created from, so the session is bound to the opened copy's idb_path, and health now reports the input SHA-256 it records. Validated with resolver, provider and session tests, check:fast, docs:check and test:changed, plus a real Hex-Rays MCP run against a 1.4 GB SC2 .i64 whose original file was left byte-identical. Refs #1416 * fix(ida): verify the private workspace of a database target by idb_path * test(ida): fix Hex-Rays health stand-ins and Windows path and runtime teardown * refactor(ida): resolve the registration once for target support * docs(ida): record the real Hex-Rays run on an existing .i64 * docs(ida): make the Hex-Rays registration example install-neutral * chore: keep merge-hook formatting outside the IDA change * fix(ida): retain live Hex-Rays databases this server does not lease A list_databases row with instance_id null is a live owner, not a stopped database. Mapping it to the supervisor's released-worker shape made cleanup delete files another client still held. Unleased rows stay active, so the private path must leave inventory before the workspace is removed. Accept idat inventory rows, hash the query bridge with sha256Hex, and complete databaseEngine on Hopper fixtures added on main. * fix(ida): report uncapped Hex-Rays call graphs * docs(ida): link to the Hex-Rays bridge on GitHub * fix(ida): verify Hex-Rays worker identity and cleanup * feat(ida): support Hex-Rays idat headless workers Refs #12921 * fix(ida): retain workspace after ambiguous Hex-Rays startup * fix(ida): preserve typed Hex-Rays call edges and deadlines * test(ida): verify Hex-Rays cleanup with a peer lease * test(ci): use project deadlines for the Cutter verifier * test(ida): keep verifier cleanup fixtures scoped to mocked modules * fix(ida): retain the worker backend in partial evidence --------- Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: morluto <morluto@users.noreply.github.com> Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
274 lines
10 KiB
TypeScript
274 lines
10 KiB
TypeScript
import { execFile } from "node:child_process";
|
|
import {
|
|
access,
|
|
copyFile,
|
|
mkdir,
|
|
readFile,
|
|
readdir,
|
|
writeFile,
|
|
} from "node:fs/promises";
|
|
import { dirname, join } from "node:path";
|
|
import { promisify } from "node:util";
|
|
import { expect, it, onTestFinished } from "vitest";
|
|
import { z } from "zod";
|
|
|
|
import {
|
|
createPackageVerifierCommands,
|
|
type PackageVerifierCommands,
|
|
} from "../../../scripts/lib/verify-package-commands.mjs";
|
|
import { withVerifierCleanup } from "../../../scripts/lib/verifier-cleanup.mjs";
|
|
import { verifyPackedBridge } from "../../../scripts/verify-packed-bridge.mjs";
|
|
import {
|
|
PrivateRuntimeRoot,
|
|
privateRuntimeRootCapability,
|
|
} from "../../../src/process/PrivateRuntimeRoot.js";
|
|
import { GHIDRA_BRIDGE_SOURCES } from "../../../src/ghidra/GhidraBridgeSources.js";
|
|
import {
|
|
createTestWorkspace,
|
|
removeTestWorkspace,
|
|
} from "../../support/workspace/workspaceFixture.js";
|
|
|
|
const exec = promisify(execFile);
|
|
const packedFiles = [
|
|
"package/bridge/hopper_bridge.py",
|
|
"package/bridge/cutter/rea_bridge.py",
|
|
"package/bridge/ida/hexrays_queries.py",
|
|
"package/bridge/agentenv/capture.py",
|
|
"package/bridge/agentenv/computer.py",
|
|
"package/bridge/agentenv/solve.py",
|
|
"package/bridge/agentenv/solve_cleanup.py",
|
|
...Object.values(GHIDRA_BRIDGE_SOURCES).map(
|
|
(name) => `package/bridge/ghidra/${name}`,
|
|
),
|
|
];
|
|
const hopper = packedFiles[0]!;
|
|
const validSource =
|
|
'# Unicode source: 雨\nraise RuntimeError("Packed source must never execute")\n';
|
|
|
|
const expectedPythonCommandFailure = (diagnostic: string | RegExp) => ({
|
|
name: "VerifierCommandExitError",
|
|
code: 1,
|
|
snapshot: {
|
|
stderr: {
|
|
text:
|
|
typeof diagnostic === "string"
|
|
? expect.stringContaining(diagnostic)
|
|
: expect.stringMatching(diagnostic),
|
|
},
|
|
},
|
|
});
|
|
|
|
async function fixture(source: string = validSource, omitHopper = false) {
|
|
const scratch = await createTestWorkspace("rea-packed-bridge-");
|
|
const directory = scratch.root;
|
|
let commands: PackageVerifierCommands | undefined;
|
|
onTestFinished(() =>
|
|
commands === undefined ? removeTestWorkspace(directory) : commands.close(),
|
|
);
|
|
commands = await createPackageVerifierCommands({
|
|
workspace: { path: directory, close: () => removeTestWorkspace(directory) },
|
|
});
|
|
const staging = join(directory, "source");
|
|
const workspace = join(directory, "workspace");
|
|
await mkdir(workspace);
|
|
for (const member of packedFiles) {
|
|
if (omitHopper && member === hopper) continue;
|
|
const path = join(staging, member);
|
|
await mkdir(dirname(path), { recursive: true });
|
|
await writeFile(path, member === hopper ? source : "fixture\n");
|
|
}
|
|
const tarball = join(directory, "packed bridges.tgz");
|
|
await exec("tar", ["-czf", tarball, "-C", staging, "package"]);
|
|
return { workspace, tarball, packedFiles, commands };
|
|
}
|
|
|
|
it("parses packed Python without executing it or extracting archive members", async () => {
|
|
const input = await fixture();
|
|
const sentinel = join(input.workspace, "caller-owned.txt");
|
|
await writeFile(sentinel, "preserved\n");
|
|
await verifyPackedBridge(input);
|
|
expect(await readdir(input.workspace)).toEqual(["caller-owned.txt"]);
|
|
expect(await readFile(sentinel, "utf8")).toBe("preserved\n");
|
|
});
|
|
|
|
it.each(["cwd", "pythonpath", "sitecustomize"] as const)(
|
|
"rejects invalid packed Python without importing ambient %s code",
|
|
async (kind) => {
|
|
const input = await fixture("def broken(:\n");
|
|
const modules = join(input.workspace, "local modules");
|
|
await mkdir(modules);
|
|
const marker = join(input.workspace, "ambient-import-executed.txt");
|
|
const moduleSource = [
|
|
"import os",
|
|
"from pathlib import Path",
|
|
"Path(os.environ['REA_PACKED_IMPORT_MARKER']).write_text('executed')",
|
|
...(kind === "sitecustomize"
|
|
? ["import ast", "ast.parse = lambda *args, **kwargs: None"]
|
|
: ["def parse(*args, **kwargs): return None"]),
|
|
].join("\n");
|
|
await writeFile(
|
|
join(modules, kind === "sitecustomize" ? "sitecustomize.py" : "ast.py"),
|
|
moduleSource,
|
|
);
|
|
const verifier = new URL(
|
|
"../../../scripts/verify-packed-bridge.mjs",
|
|
import.meta.url,
|
|
).href;
|
|
// A separate Node process owns cwd/environment changes; the Vitest worker
|
|
// and its other subprocess fixtures retain their original environment.
|
|
const result = await exec(
|
|
process.execPath,
|
|
[
|
|
"--input-type=module",
|
|
"--eval",
|
|
`import { verifyPackedBridge } from ${JSON.stringify(verifier)};
|
|
import { createPackageVerifierCommands } from ${JSON.stringify(new URL("../../../scripts/lib/verify-package-commands.mjs", import.meta.url).href)};
|
|
import { withVerifierCleanup } from ${JSON.stringify(new URL("../../../scripts/lib/verifier-cleanup.mjs", import.meta.url).href)};
|
|
import { PrivateRuntimeRoot } from ${JSON.stringify(new URL("../../../dist/process/PrivateRuntimeRoot.js", import.meta.url).href)};
|
|
const root = await PrivateRuntimeRoot.create({ prefix: "rea-packed-python-command-" });
|
|
let commands;
|
|
await withVerifierCleanup(async () => {
|
|
commands = await createPackageVerifierCommands({ workspace: root });
|
|
try {
|
|
await verifyPackedBridge({ ...JSON.parse(process.env.REA_PACKED_VERIFY_INPUT), commands });
|
|
console.log(JSON.stringify({ accepted: true }));
|
|
} catch (error) {
|
|
console.log(JSON.stringify({ accepted: false, name: error.name, code: error.code, stderr: error.snapshot?.stderr?.text }));
|
|
}
|
|
}, [() => commands === undefined ? root.close() : commands.close()]);`,
|
|
],
|
|
{
|
|
cwd: kind === "cwd" ? modules : input.workspace,
|
|
env: {
|
|
...Object.fromEntries(
|
|
Object.entries(process.env).filter(
|
|
([name]) => name.toLowerCase() !== "pythonpath",
|
|
),
|
|
),
|
|
PYTHONPATH: kind === "cwd" ? "" : modules,
|
|
REA_PACKED_IMPORT_MARKER: marker,
|
|
REA_PACKED_VERIFY_INPUT: JSON.stringify({
|
|
tarball: input.tarball,
|
|
packedFiles: input.packedFiles,
|
|
}),
|
|
},
|
|
},
|
|
);
|
|
const observation = z
|
|
.object({
|
|
accepted: z.boolean(),
|
|
name: z.string().optional(),
|
|
code: z.number().optional(),
|
|
stderr: z.string().optional(),
|
|
})
|
|
.parse(JSON.parse(result.stdout));
|
|
expect(observation).toMatchObject({
|
|
accepted: false,
|
|
name: "VerifierCommandExitError",
|
|
code: 1,
|
|
stderr: expect.stringContaining("SyntaxError"),
|
|
});
|
|
await expect(access(marker)).rejects.toMatchObject({ code: "ENOENT" });
|
|
},
|
|
);
|
|
|
|
it("rejects each omitted required bridge before parsing the archive", async () => {
|
|
const input = await fixture();
|
|
for (const member of packedFiles)
|
|
await expect(
|
|
verifyPackedBridge({
|
|
...input,
|
|
tarball: join(input.workspace, "missing.tgz"),
|
|
packedFiles: packedFiles.filter((path) => path !== member),
|
|
}),
|
|
).rejects.toThrow(`package omitted ${member}`);
|
|
expect(await readdir(input.workspace)).toEqual([]);
|
|
});
|
|
|
|
it("rejects a missing actual Python member even when the listing advertises it", async () => {
|
|
const input = await fixture(validSource, true);
|
|
await expect(verifyPackedBridge(input)).rejects.toMatchObject(
|
|
expectedPythonCommandFailure(hopper),
|
|
);
|
|
expect(await readdir(input.workspace)).toEqual([]);
|
|
});
|
|
|
|
it("reports invalid packed Python syntax with the member filename", async () => {
|
|
const input = await fixture("def broken(:\n");
|
|
await expect(verifyPackedBridge(input)).rejects.toMatchObject(
|
|
expectedPythonCommandFailure(/hopper_bridge\.py[\s\S]*SyntaxError/u),
|
|
);
|
|
expect(await readdir(input.workspace)).toEqual([]);
|
|
});
|
|
|
|
it.each(["valid", "invalid"] as const)(
|
|
"parses %s source under an owned private root and preserves owner cleanup",
|
|
async (kind) => {
|
|
expect(privateRuntimeRootCapability(process.platform).available).toBe(true);
|
|
const input = await fixture(
|
|
kind === "valid" ? validSource : "def broken(:\n",
|
|
);
|
|
const root = await PrivateRuntimeRoot.create({
|
|
prefix: "rea-packed-bridge-lease-",
|
|
});
|
|
let commands: PackageVerifierCommands | undefined;
|
|
await withVerifierCleanup(async () => {
|
|
commands = await createPackageVerifierCommands({ workspace: root });
|
|
const pack = join(root.path, "pack");
|
|
await mkdir(pack);
|
|
const tarball = join(pack, "packed bridges.tgz");
|
|
await copyFile(input.tarball, tarball);
|
|
const parsed = verifyPackedBridge({ tarball, packedFiles, commands });
|
|
if (kind === "valid") await parsed;
|
|
else
|
|
await expect(parsed).rejects.toMatchObject(
|
|
expectedPythonCommandFailure("SyntaxError"),
|
|
);
|
|
expect(await readdir(root.path)).toEqual(["pack"]);
|
|
expect(
|
|
Buffer.compare(await readFile(tarball), await readFile(input.tarball)),
|
|
).toBe(0);
|
|
}, [() => (commands === undefined ? root.close() : commands.close())]);
|
|
await expect(access(root.path)).rejects.toMatchObject({ code: "ENOENT" });
|
|
},
|
|
);
|
|
|
|
it.each(["symlink", "hardlink", "directory"])(
|
|
"rejects a %s Python member without following it or extracting files",
|
|
async (kind) => {
|
|
const input = await fixture();
|
|
await exec("python3", [
|
|
"-I",
|
|
"-S",
|
|
"-c",
|
|
[
|
|
"import io, sys, tarfile",
|
|
"with tarfile.open(sys.argv[1], 'w:gz') as archive:",
|
|
" target = tarfile.TarInfo('package/bridge/target.py')",
|
|
" target.size = 5",
|
|
" archive.addfile(target, io.BytesIO(b'pass\\n'))",
|
|
" member = tarfile.TarInfo(sys.argv[2])",
|
|
" member.type = {'symlink': tarfile.SYMTYPE, 'hardlink': tarfile.LNKTYPE, 'directory': tarfile.DIRTYPE}[sys.argv[3]]",
|
|
" member.linkname = 'target.py' if sys.argv[3] == 'symlink' else target.name",
|
|
" archive.addfile(member)",
|
|
].join("\n"),
|
|
input.tarball,
|
|
hopper,
|
|
kind,
|
|
]);
|
|
await expect(verifyPackedBridge(input)).rejects.toMatchObject(
|
|
expectedPythonCommandFailure(`${hopper} is not a regular file`),
|
|
);
|
|
expect(await readdir(input.workspace)).toEqual([]);
|
|
},
|
|
);
|
|
|
|
it("rejects malformed archives without writing to the caller's workspace", async () => {
|
|
const input = await fixture();
|
|
await writeFile(input.tarball, "not a tar archive\n");
|
|
await expect(verifyPackedBridge(input)).rejects.toMatchObject(
|
|
expectedPythonCommandFailure("ReadError"),
|
|
);
|
|
expect(await readdir(input.workspace)).toEqual([]);
|
|
});
|