Files
reverse-engineering-anything/tests/conformance/ghidra
Akshad Jogiandmorluto ae25050dff feat(native): surface inferred prototype instruction conflicts (#8161)
* feat(native): surface inferred prototype instruction conflicts

- Add evidence-backed x86-32 stack-read and stdcall cleanup diagnostics in ReaGhidraBridge.java without rewriting signatures or function bodies.

- Carry prototype conflicts through native API schemas, analysis summaries and retained dossier views so CLI and MCP share the observations.

- Update Ghidra analysis profile digests, documentation and MCP boundary tests for the new diagnostic contract.

* fix(native): follow framed paths and retain zero cleanup conflicts

- Traverse decoded entry-frame paths in ReaGhidraBridge.java instead of stopping at an address-ordered early return, so later reachable reads remain visible.

- Detect explicit, implicit and partial EBP writes and use decoded operands with p-code loads, preventing stale-frame warnings and recognizing real Ghidra stack reads.

- Compare immediate return cleanup with inferred stdcall stack bytes even when either side is zero, and allow zero in the native API contract.

- Add real Ghidra instruction probes and MCP/CI regressions for the reported cases and document the bounded checks.

* fix(native): attribute frame reads before register writes

- Check a decoded stack read before invalidating EBP in ReaGhidraBridge.java so instructions that read and replace the frame still report the conflict.

- Match each candidate operand's address p-code to its LOAD pointer so a POP from ESP into an EBP destination cannot masquerade as an argument read.

- Cover both decoded instruction cases in ReaPrototypeConflictProbe.java and await verifier-owned scratch cleanup on success and failure.

* fix(ghidra): recompute raw profile digests after rebase

- Recompute the 6502 and disabled-analyzer raw-binary profile digests in GhidraRawProfileDigest.test.ts so the pinned values commit to the merged shared policy on current main.

* fix(native): locate the entry frame by address and own verifier cleanup

- Find PUSH EBP; MOV EBP, ESP from the function entry address in ReaGhidraBridge.java so an owned block below the entry cannot hide a reachable argument read.

- Run analyzeHeadless in its own process group and stop that group before removing scratch in verify-ghidra-prototype-conflicts.mjs, so the JVM and decompiler children are addressed as one owned family.

- Cover the lower-owned-block case in ReaPrototypeConflictProbe.java with an entry frame preceded by a decoded lower block.

* fix(native): size prototype fixture and own Ghidra cleanup

- Allocate the instruction fixture through the lower-block offset in verify-ghidra-prototype-conflicts.mjs so the entry frame and decoded block exist for the probe.

- Check the detached process group instead of the leader exit code before cleaning up, so lingering decompiler children receive SIGTERM and SIGKILL before scratch removal.

* fix(native): own Ghidra probe cleanup with shared process owner

- Use REA's ProviderProcessSupervisor and token-verified cleanup in verify-ghidra-prototype-conflicts.mjs so the JVM and descendants are stopped before scratch removal, including failed launches.

- Preserve the source-owned x86 fixture with an entry frame and lower block while asserting verified cleanup.

- Build the pinned verifier dependency graph in the Ghidra bridge CI job and extend its timeout before running the real probe.

* fix(native): end verifier waits on interrupt before cleanup

- Install SIGINT and SIGTERM handlers before spawning Ghidra and abort the wait, so Ctrl-C no longer leaves the detached process group running.

- Keep cleanup handlers installed until the shared process owner reports verified cleanup, then remove scratch only after the process family is stopped.

* fix(native): preserve verifier cleanup interruption errors

- Keep interrupt handlers installed through process cleanup and workspace deletion, then check cancellation before reporting success.

- Preserve and report the failing cleanup step instead of replacing it with an interruption or deleting a tree that may still be unsafe.

* fix(native): retain probe failures across verifier cleanup

- Record the probe error before cleanup so a late SIGINT or SIGTERM cannot replace its stack and captured Ghidra diagnostics.

- Prefer the earlier probe failure when cleanup also fails, while preserving cleanup diagnostics and temporary state for diagnosis.

* test(kuna): raise caller matrix verifier budget for parallel CI

- The multi-target verifier spawn used a 10s timeout while sibling verifier cases use 25s.

- Under 8-way sharded CI load this intermittently exceeded 10s and produced ETIMEDOUT, failing shard 1/8 and portable tests.

- Raise to 30s to match the existing verifier budget and avoid load flakes.

* ci: retrigger checks after transient Ghidra SDK download 403

- The ghidra_bridge lane failed in upstream's untouched SDK acquisition step: curl returned 403 with 0 bytes downloaded in 0.046s from the GitHub release CDN.

- Push an empty commit to start a fresh run because rerunning on the upstream repository requires admin rights.

* fix(native): keep the cleanup lease on combined verifier failures

- Run the prototype-conflict probe through withVerifierFixture so a probe failure and a failed release surface as one AggregateError that keeps the probe error as its cause and attaches the retained fixture as cleanupLeases, instead of throwing the probe error alone and dropping the cleanup owner.

- Report the failed cleanup step and retained project only when the fixture was not released, and keep interrupt handlers installed until release finishes.

* test(native): use shared MCP prototype test deadlines

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-11-19 01:55:13 +00:00
..

Ghidra conformance corpus

These source-owned fixtures are compiled at verification time; generated binaries and Ghidra projects are never committed.

For the one-shot function bundle handoff, run npm run build:cached and then node scripts/verify-real-ghidra-function-bundle.mjs with GHIDRA_INSTALL_DIR and JAVA_HOME set. The verifier compiles its own target, creates a project with a synthetic history report, and exercises complete, summary, and no-overwrite CLI delivery. It checks the target and project files for changes.

  • ../c/riscv.c and riscv-probe.S produce source-owned RV32/RV64 ELF images with double-float ABI declarations and an independently known instruction. ../c/riscv-float-abi.c, riscv-probe.S and riscv-isa-probe.S produce typed and stripped RV32 ilp32 images whose calls cover every rule of the ilp32 policy; riscv-checks.mjs holds that policy's expected storage and the call-site oracle. npm run verify:ghidra:riscv exercises real CLI/MCP discovery, decompilation, argument storage, output schemas, profile and snapshot identity, admission and refusal diagnostics, immutable source bytes, and owned cleanup. It requires BYO Ghidra 12.2/JDK 25+ and RISC-V Clang/LLD/LLVM readelf/LLVM objdump; REA_RISCV_CLANG, REA_RISCV_LLD, REA_RISCV_READELF and REA_RISCV_OBJDUMP select existing tool commands.

  • ReaInstructionContextProbe.java inspects source-owned ARM, Thumb, MIPS32 and MIPS16 instructions through the production bridge in an ephemeral, read-only Ghidra project. It checks exact decoder-context register names, values and provenance; the ARM lane also injects an unavailable provider value. The ghidra-bridge-sources CI job runs both processor families. The input bytes are never executed.

  • ReaDecodeProbe.java decodes source-owned instruction bytes from a language's default context and checks each mnemonic, length and optional complete operand. The ghidra-bridge-sources CI job runs it for ARM:LE:32:Cortex, ARM:LE:32:v8-m and Xtensa:LE:32:default; the bytes are never executed.

  • ReaComputedCallCoverageProbe.java decodes source-owned x86-64 calls through a writable slot, a register and a direct target, injects only the slot call's recorded COMPUTED_CALL target, and reads callees through the production bridge. The recorded target stays an observed callee while callee coverage is partial and names the call site; the targetless call stays partial without that candidate limitation, and the direct call stays complete. The ghidra-bridge-sources CI job runs it; the bytes are never executed. It does not establish which references Ghidra's own analysis records for other slot shapes.

  • inventory.c produces separate x86-64 debug and stripped ELF executables. It covers imports, an external puts function, linker thunks, source symbols, direct calls, a targetless callback call, two referenced strings, and a multi-block branch.

  • entry-aliases.c preserves secondary imported entry labels, a bare-hex alias, and an interior label. The native CLI/MCP lane verifies entry selection, procedure search and feature tracing by the secondary name, annotation, interior-label rejection, and unchanged source bytes.

  • cross-format.c is freestanding so the verifier can produce AArch64 ELF, x86-64 PE, and x86-64 Mach-O targets from the same semantics. It preserves an exported entry, direct and indirect calls, a volatile string reference, and a multi-block branch across loaders.

  • relocation-symbol.c builds an x86-64 ELF with named and unnamed dynamic relocations. The cross-format verifier checks their Ghidra observations through the public CLI without executing the artifact.

  • no-return.c and ReaNoReturnProbe.java inject incorrect import flags into a real ELF or Mach-O analysis database, verify caller recovery, propagated flag repair and signature preservation, and retain genuine/unknown no-return calls and independently identified entries. Run npm run build:cached, then node scripts/verify-real-ghidra-noreturn.mjs with the Ghidra/JDK environment below on Linux or macOS x64/arm64. On Linux, use a procps-compatible process table that exposes REA's current process for owned cleanup. It also checks named limitations through the real read-only TCP bridge and provider Evidence projection; default Unix-socket transport, Windows, and the reported Amethyst artifact need separate verification. Fixture target code is never executed.

  • ReaDecompilerFailureProbe.java replaces the bridge decompiler after a real import of cross-format.c, so one function's decompilation fails. It checks that analyze_function still returns that function's assembly, callers, callees, references and basic blocks unchanged, marks pseudocode, native API and p-code value flow unavailable with the decompiler's reason, and that cancellation still ends the request. Run npm run build:cached, then node scripts/verify-real-ghidra-decompiler-failure.mjs on Linux or macOS x64/arm64. The lane also parses the degraded dossier with REA's Ghidra output contract and checks the healthy provider path through the real read-only TCP bridge. Decompiler failure marks only that observation as live: retain its Evidence, retry after backend recovery, and keep healthy dossier snapshot reuse. The injection does not establish which natural inputs make the decompiler fail.

  • node scripts/verify-real-ghidra-decompile-timeout.mjs (after npm run build:cached, Linux or macOS x64/arm64) compiles a generated shared library whose rea_slow function has thousands of branches and analyzes it with REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS=1 through the real read-only TCP provider. Ghidra's own decompiler stops at the limit with no injection: analyze_function returns the listing facets with pseudocode unavailable and a live cache policy, procedure_pseudo_code fails with a message naming the setting, and the same session then decompiles rea_small. Set REA_DECOMPILE_TIMEOUT_BRANCHES to change the fixture size. The verifier budgets 1 MiB for generated C source and derives the maximum branch count from the worst-case line size before allocating the source. Larger or invalid counts are rejected before installation inspection or compilation.

  • loader-relocation.c, loader-relocation-provider.c and ReaLoaderRelocationProbe.java exercise the production ReaGhidraBridge against a real imported Linux full-RELRO PIE ELF and a small shared library. The fixture's .data.rel.ro pointer is populated by an importer-created relocation. The probe proves that the ET_DYN program is marked relocatable by Ghidra but is still protected, checks the relocation byte range and read/write reference handling, and compares baseline/protected pseudocode. It also checks that a -fno-plt GOT caller retains identical pseudocode when its relocation blocks become volatile. Source FileBytes and block permissions must remain unchanged. npm run verify:ghidra:loader-relocation requires Linux, BYO Ghidra/JDK with its native decompiler, and a C compiler; the fixture is never executed.

scripts/verify-real-ghidra.mjs uses cc, clang, and lld-link by default; REA_CC, REA_CLANG, and REA_LLD_LINK can select alternate commands. The verifier requires the verified bring-your-own Ghidra 12.2 installation through GHIDRA_INSTALL_DIR and validates header classification before starting the provider. Provider admission accepts other 12.x (>=12.2) builds with an unverified-build limitation; this lane does not.

Conformance compares semantic facts: provider/profile identity, function classification, resolved call edges, typed references, strings/xrefs, and CFG topology. It only requires provider pseudocode and assembly to be non-empty, bounded, and address-bearing; it never compares their text with Hopper. The callback fixture also proves that an unresolved targetless flow is not silently promoted to a direct callee. Every target runs in a separate owned project and must leave no process, socket, project, or runtime root after close.

relative-switch-signed.S covers signed halfword offsets, the CBZ zero-entry route, and selector biases of 0, 2 and 10 in the AArch64 jump-table lane. The nonzero biases exercise overlapping and disjoint typed label/index ranges. REA retains Ghidra's typed labels when the compared selector has an earlier register definition; additional labels remain unknown until their relationship to the typed switch selector can be proved. The oracle rejects extra numeric labels and verifies each retained target against the source instruction immediate.