Files
reverse-engineering-anything/bridge/ida/test_hexrays_queries.py
T
6822ebe1f3 feat(ida): support the official Hex-Rays IDA MCP headlessly (#4813)
* feat(ida): support the official Hex-Rays IDA MCP headlessly

Hex-Rays IDA MCP serves database reads only through execute_python, so
REA rejected it as an upstream. A `hexrays` headless stdio registration
now leases a private idalib worker through open_database, answers the
existing database-supervisor reads with a fixed, packaged, read-only
query module, and releases the lease with close_database, verified by
list_databases.

Observations record execute_python, the query and the module digest.
Tool errors are typed so a completed rejection is distinguished from an
unknown outcome; an unconfirmed release is retried and a late lease on
the private input is adopted.

Validated IDA source and bridge tests, static checks, docs, and npm run
verify:ida on Linux x64 against x86-64 and AArch64 ELF fixtures.

* feat(ida): name the Hex-Rays registration for its catalog

A registration without `protocol` that reaches the official Hex-Rays
server failed with only the missing database-supervisor tools. Startup
profile failures now report the advertised catalog, and when it is the
Hex-Rays one the message and recovery name the `hexrays` registration.

Refs #1416

* docs(ida): state Hex-Rays prerequisites and upstream session traces

Upstream needs IDA 9.4 or later with idalib, Python 3.11 or later, uv
and Git. The profile does not analyze databases open in the IDA GUI, and
the server keeps its own session traces outside REA's workspace.

Refs #1416

* fix(ida): skip inline data when scanning Hex-Rays calls

The call scan decoded every head that FuncItems yields. Inline data such
as a jump table can decode as a call and refer to a function or import,
which reported false callees and call-graph edges. Skip non-code heads
before decoding, as disassembly already does.

Refs #1416

* fix(ida): retain Hex-Rays health reads in evidence

The identity and readiness reads around each operation kept only their
parsed result, dropping the bridge digest and any engine output. Keep
them whole as health_observations beside the parsed metadata, in both
successful and partial-failure Evidence.

Refs #1416

* test(ida): list the Hex-Rays bridge in package fixtures

The packed-bridge verifier has required bridge/ida/hexrays_queries.py
since the Hex-Rays profile landed, but three package tests still built
their member lists without it. The verifier stopped at that file, so 21
cases failed or asserted on the wrong omission.

Refs #1416

* feat(ida): open existing IDA databases through the Hex-Rays profile

A headless open re-ran auto-analysis from the input binary, discarding
the names, types and comments of an existing database and taking far
longer on large targets. Target admission now identifies IDA databases
by their IDA1/IDA2 signature, and the hexrays profile opens a private
copy of one; Hopper and the other IDA profiles reject them.

The copy is required, not incidental: idalib repacks a database when
its last lease is released, even without a save. A database reports
the input it was created from, so the session is bound to the opened
copy's idb_path, and health now reports the input SHA-256 it records.

Validated with resolver, provider and session tests, check:fast,
docs:check and test:changed, plus a real Hex-Rays MCP run against a
1.4 GB SC2 .i64 whose original file was left byte-identical.

Refs #1416

* fix(ida): verify the private workspace of a database target by idb_path

* test(ida): fix Hex-Rays health stand-ins and Windows path and runtime teardown

* refactor(ida): resolve the registration once for target support

* docs(ida): record the real Hex-Rays run on an existing .i64

* docs(ida): make the Hex-Rays registration example install-neutral

* chore: keep merge-hook formatting outside the IDA change

* fix(ida): retain live Hex-Rays databases this server does not lease

A list_databases row with instance_id null is a live owner, not a stopped
database. Mapping it to the supervisor's released-worker shape made cleanup
delete files another client still held. Unleased rows stay active, so the
private path must leave inventory before the workspace is removed.

Accept idat inventory rows, hash the query bridge with sha256Hex, and
complete databaseEngine on Hopper fixtures added on main.

* fix(ida): report uncapped Hex-Rays call graphs

* docs(ida): link to the Hex-Rays bridge on GitHub

* fix(ida): verify Hex-Rays worker identity and cleanup

* feat(ida): support Hex-Rays idat headless workers

Refs #12921

* fix(ida): retain workspace after ambiguous Hex-Rays startup

* fix(ida): preserve typed Hex-Rays call edges and deadlines

* test(ida): verify Hex-Rays cleanup with a peer lease

* test(ci): use project deadlines for the Cutter verifier

* test(ida): keep verifier cleanup fixtures scoped to mocked modules

* fix(ida): retain the worker backend in partial evidence

---------

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-11-20 17:58:27 -05:00

290 lines
12 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Source-owned tests for the Hex-Rays IDA MCP query bridge.
The bridge runs inside IDA through execute_python. These tests execute it the
same way, as a script in a fresh namespace, against a small stand-in database
so reply shapes, paging and call classification are checked without IDA.
"""
import json
import os
import sys
import types
import unittest
from unittest import mock
SOURCE = os.path.join(os.path.dirname(__file__), "hexrays_queries.py")
BADADDR = 0xFFFFFFFFFFFFFFFF
XREF_FAR = 1
class Func:
def __init__(self, start, end, name):
self.start_ea, self.end_ea, self.name = start, end, name
def size(self):
return self.end_ea - self.start_ea
class StringItem:
def __init__(self, ea, text):
self.ea, self.text = ea, text
def __str__(self):
return self.text
class Xref:
def __init__(self, frm, iscode, flow=False):
self.frm, self.iscode, self.flow = frm, iscode, flow
FUNCS = [Func(0x1000, 0x1018, "main"), Func(0x2000, 0x2008, "helper")]
# address -> (text, is_call, code targets, data targets)
INSNS = {
0x1000: ("push rbp", False, [], []),
0x1004: ("call helper+4", True, [0x2004], []),
0x1008: ("call cs:puts_ptr", True, [], [0x5000]),
0x100C: ("call loc_3000", True, [0x3000], []),
0x1010: ("call helper", True, [0x2000], []),
# Inline data whose bytes decode as a call and that refers to an import.
0x1014: ("dq offset exit", True, [], [0x5008]),
0x2000: ("mov eax, 1", False, [], []),
0x2004: ("retn", False, [], []),
}
DATA_HEADS = {0x1014}
IMPORTS = {0x5000: "puts", 0x5008: "exit"}
NAMES = {"main": 0x1000, "helper": 0x2000, "data_label": 0x4000}
XREFS = {0x2000: [Xref(0x1004, True), Xref(0x1ffc, True, flow=True), Xref(0x1004, True)]}
STRINGS = [(0x6000 + index, f"string {index}") for index in range(5)]
PSEUDOCODE = ["int main(void)", "{", " helper();", " return 0;", "}"]
def get_func(ea):
return next((f for f in FUNCS if f.start_ea <= ea < f.end_ea), None)
def stand_ins():
m = {name: types.ModuleType(name) for name in (
"ida_auto", "ida_funcs", "ida_hexrays", "ida_idaapi", "ida_idp", "ida_lines",
"ida_loader", "ida_nalt", "ida_name", "ida_segment", "ida_typeinf", "ida_ua",
"ida_xref", "idautils", "idc",
)}
m["ida_auto"].auto_is_ok = lambda: True
m["ida_idaapi"].BADADDR = BADADDR
m["ida_funcs"].get_func = get_func
m["ida_funcs"].get_func_name = lambda ea: next((f.name for f in FUNCS if f.start_ea == ea), "")
m["ida_funcs"].get_func_qty = lambda: len(FUNCS)
m["ida_funcs"].getn_func = lambda index: FUNCS[index]
class DecompilationFailure(Exception):
pass
class Line:
def __init__(self, line):
self.line = line
def decompile(ea):
if ea != 0x1000:
raise DecompilationFailure("decompilation failure at 0x2000")
return types.SimpleNamespace(get_pseudocode=lambda: [Line(text) for text in PSEUDOCODE])
m["ida_hexrays"].DecompilationFailure = DecompilationFailure
m["ida_hexrays"].init_hexrays_plugin = lambda: True
m["ida_hexrays"].decompile = decompile
m["ida_lines"].tag_remove = lambda text: text
m["ida_lines"].GENDSM_REMOVE_TAGS = 1
m["ida_lines"].generate_disasm_line = lambda ea, flags: INSNS[ea][0]
m["ida_loader"].PATH_TYPE_IDB = 1
m["ida_loader"].get_path = lambda kind: "/private/input.i64"
m["ida_nalt"].get_input_file_path = lambda: "/private/input"
m["ida_nalt"].retrieve_input_file_sha256 = lambda: bytes.fromhex("ab" * 32)
m["ida_nalt"].get_imagebase = lambda: 0x400000
m["ida_nalt"].get_root_filename = lambda: "input"
m["ida_nalt"].get_tinfo = lambda tif, ea: False
m["ida_nalt"].get_import_module_qty = lambda: 1
def enum_import_names(index, callback):
for ea, name in IMPORTS.items():
callback(ea, name, 0)
m["ida_nalt"].enum_import_names = enum_import_names
m["ida_name"].get_name_ea = lambda start, name: NAMES.get(name, BADADDR)
m["ida_name"].get_name = lambda ea: {0x3000: "loc_3000"}.get(ea, "")
segments = [types.SimpleNamespace(start_ea=0x1000, end_ea=0x3000, size=lambda: 0x2000)]
m["ida_segment"].get_segm_qty = lambda: len(segments)
m["ida_segment"].getnseg = lambda index: segments[index]
m["ida_typeinf"].tinfo_t = object
m["ida_typeinf"].GUESS_FUNC_OK = 1
m["ida_typeinf"].guess_tinfo = lambda tif, ea: 0
m["ida_typeinf"].func_type_data_t = object
m["ida_ua"].insn_t = lambda: types.SimpleNamespace(ea=None)
def decode_insn(insn, ea):
insn.ea = ea
return 4
m["ida_ua"].decode_insn = decode_insn
m["ida_idp"].is_call_insn = lambda insn: INSNS[insn.ea][1]
m["ida_xref"].XREF_FAR = XREF_FAR
m["idautils"].FuncItems = lambda start: [ea for ea in sorted(INSNS) if get_func(ea).start_ea == start]
m["idautils"].CodeRefsFrom = lambda ea, flow: list(INSNS[ea][2])
m["idautils"].DataRefsFrom = lambda ea: list(INSNS[ea][3])
m["ida_xref"].fl_CF = 16
m["ida_xref"].fl_CN = 17
m["ida_xref"].fl_JN = 19
m["ida_xref"].dr_O = 1
m["ida_xref"].XREF_MASK = 31
def xrefs_from(ea, flags=0):
_, _, code_refs, data_refs = INSNS[ea]
return [
types.SimpleNamespace(to=target, type=m["ida_xref"].fl_CN, iscode=True)
for target in code_refs
] + [
types.SimpleNamespace(to=target, type=m["ida_xref"].dr_O, iscode=False)
for target in data_refs
]
m["idautils"].XrefsFrom = xrefs_from
def xrefs_to(ea, flags=0):
return [x for x in XREFS.get(ea, []) if not (flags & XREF_FAR and x.flow)]
m["idautils"].XrefsTo = xrefs_to
m["idautils"].Strings = lambda: [StringItem(ea, text) for ea, text in STRINGS]
# Flags stand in for their address, so is_code can consult DATA_HEADS.
m["idc"].get_full_flags = lambda ea: ea
m["idc"].is_code = lambda flags: flags not in DATA_HEADS
return m
class BridgeTest(unittest.TestCase):
def setUp(self):
modules = stand_ins()
patcher = mock.patch.dict(sys.modules, modules)
patcher.start()
self.addCleanup(patcher.stop)
with open(SOURCE, encoding="utf-8") as source:
self.code = compile(source.read(), "<ida-mcp>", "exec")
self.namespace = {}
exec(self.code, self.namespace)
def query(self, tool, arguments):
# REA embeds JSON text in a Python string literal; json.dumps of a str
# produces the same escapes JSON.stringify does for these inputs.
line = f"_rea_query({json.dumps(tool)}, {json.dumps(json.dumps(arguments))})"
return eval(line, self.namespace)
def test_lookup_reports_entries_containment_and_missing_names(self):
rows = self.query("lookup_funcs", {"queries": ["main", "0x2004", "data_label", "absent", 'a"\\\n
']})
self.assertEqual(rows[0]["fn"], {"addr": "0x1000", "name": "main", "size": "0x18"})
self.assertEqual(rows[1]["fn"]["addr"], "0x2000")
self.assertEqual([row["error"] for row in rows[2:]], ["Not a function", "Not found", "Not found"])
self.assertEqual(rows[4]["query"], 'a"\\\n
')
def test_pages_advance_and_finish(self):
first = self.query("decompile", {"addr": "0x1000", "offset": 0, "max_lines": 3})
second = self.query("decompile", {"addr": "0x1000", "offset": 3, "max_lines": 3})
self.assertEqual(first["cursor"], {"next": 3})
self.assertEqual(second["cursor"], {"done": True})
self.assertEqual(first["code"] + "\n" + second["code"], "\n".join(PSEUDOCODE))
self.assertNotIn("error", first)
functions = self.query("list_funcs", {"queries": {"offset": 1, "count": 5}})
self.assertEqual(functions, [{"data": [{"addr": "0x2000", "name": "helper", "size": "0x8"}], "next_offset": None}])
strings = self.query("find_regex", {"pattern": "(?s).*", "offset": 0, "limit": 4})
self.assertEqual(strings["cursor"], {"next": 4})
rest = self.query("find_regex", {"pattern": "(?s).*", "offset": 4, "limit": 4})
self.assertEqual(rest, {"matches": [{"addr": "0x6004", "string": "string 4"}], "cursor": {"done": True}})
def test_snapshot_retains_only_the_latest_enumeration(self):
self.query("decompile", {"addr": "0x1000", "offset": 0, "max_lines": 1})
self.query("disasm", {"addr": "0x1000", "offset": 0, "max_instructions": 1})
self.assertEqual(list(self.namespace["_rea_pages"]), [("disasm", 0x1000)])
def test_failures_are_explicit_rows(self):
failed = self.query("decompile", {"addr": "0x2000", "offset": 0, "max_lines": 10})
self.assertEqual(failed, {"code": None, "error": "decompilation failure at 0x2000", "cursor": {"done": True}})
interior = self.query("disasm", {"addr": "0x1004", "offset": 0, "max_instructions": 10})
self.assertEqual(interior["asm"], None)
self.assertEqual(self.query("callees", {"addrs": ["0x1004"], "limit": 5}), [{"callees": None, "error": "Not a function"}])
def test_callees_keep_every_direct_target(self):
[row] = self.query("callees", {"addrs": ["0x1000"], "limit": 500})
self.assertEqual(
row,
{
"callees": [
{"addr": "0x2000", "name": "helper", "type": "internal"},
{"addr": "0x2004", "name": "helper", "type": "internal"},
{"addr": "0x3000", "name": "loc_3000", "type": "external"},
{"addr": "0x5000", "name": "puts", "type": "external"},
],
"more": False,
},
)
[capped] = self.query("callees", {"addrs": ["0x1000"], "limit": 1})
self.assertTrue(capped["more"])
self.assertEqual(len(capped["callees"]), 1)
def test_callgraph_reports_function_entries_only(self):
[row] = self.query("callgraph", {"roots": ["0x1000"], "max_depth": 0})
# Interior, non-function and import targets are not graph edges.
self.assertEqual(row["edges"], [{"from": "0x1000", "to": "0x2000", "type": "call"}])
FUNCS.extend([Func(0x4000, 0x4008, "referenced_data"), Func(0x4100, 0x4108, "jump_target")])
self.addCleanup(lambda: FUNCS.__delitem__(slice(-2, None)))
xrefs_from = sys.modules["idautils"].XrefsFrom
code_refs_from = sys.modules["idautils"].CodeRefsFrom
data_refs_from = sys.modules["idautils"].DataRefsFrom
sys.modules["idautils"].CodeRefsFrom = lambda ea, flow: (
code_refs_from(ea, flow) + ([0x4100] if ea == 0x1010 else [])
)
sys.modules["idautils"].DataRefsFrom = lambda ea: (
data_refs_from(ea) + ([0x4000] if ea == 0x1010 else [])
)
def mixed_xrefs(ea, flags):
refs = xrefs_from(ea, flags)
if ea == 0x1010:
refs.extend([
types.SimpleNamespace(to=0x4000, type=sys.modules["ida_xref"].dr_O, iscode=False),
types.SimpleNamespace(to=0x4100, type=sys.modules["ida_xref"].fl_JN, iscode=True),
])
return refs
sys.modules["idautils"].XrefsFrom = mixed_xrefs
[mixed] = self.query("callgraph", {"roots": ["0x1000"], "max_depth": 0})
self.assertEqual(mixed["edges"], row["edges"])
with self.assertRaises(ValueError):
self.query("callgraph", {"roots": ["0x1000"], "max_depth": 1})
def test_xrefs_exclude_ordinary_flow_and_duplicates(self):
[row] = self.query("xrefs_to", {"addrs": ["0x2000"], "limit": 1000})
self.assertEqual(
row,
{"xrefs": [{"addr": "0x1004", "type": "code", "fn": {"addr": "0x1000", "name": "main", "size": "0x18"}}], "more": False},
)
def test_health_and_survey_shapes(self):
self.assertEqual(
self.query("server_health", {}),
{
"status": "ok",
"input_path": "/private/input",
"input_sha256": "ab" * 32,
"idb_path": "/private/input.i64",
"imagebase": "0x400000",
"auto_analysis_ready": True,
"module": "input",
},
)
# A database that recorded no input hash reports none, not an empty digest.
sys.modules["ida_nalt"].retrieve_input_file_sha256 = lambda: None
self.assertIsNone(self.query("server_health", {})["input_sha256"])
survey = self.query("survey_binary", {"detail_level": "minimal"})
self.assertEqual(survey["segments"], [{"start": "0x1000", "end": "0x3000", "size": "0x2000"}])
self.assertEqual(survey["statistics"], {"total_segments": 1})
if __name__ == "__main__":
unittest.main()