mirror of
https://github.com/morluto/rea.git
synced 2026-10-26 17:35:11 +00:00
main
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
1a48a82020 |
feat(analysis)!: default function MCP output to retained summaries (#5660)
* feat(ghidra): bound one procedure's decompile without closing the session Fixes #5340 * fix(ghidra): report decompile limit as a request timeout Keeps BinarySession from marking every Ghidra operation unavailable after one procedure's decompile reaches the limit, and names the MCP tool-call deadline in the remediation. * fix(ghidra): budget the decompile limit and report cancellation-closed sessions Adds REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS to the derived Codex/Kimi tool deadline and forwards it through setup. Marks a remote failure that settles after caller cancellation so the timeout projection no longer promises session reuse. * feat(ghidra)!: default analyze_function to a retained summary analyze_function was the one analysis tool never wired into REA's detail: summary|complete convention, so it always returned the complete native dossier inline (0.2-10 MB, dominated by value-flow and references). Agents on tool-result-capped harnesses never reached the pseudocode. Add `detail` to analyze_function, defaulting to summary: return a compact dossier (procedure identity, pseudocode size, facet counts, value-flow availability) and retain the complete dossier Evidence in the session. Page pseudocode, references, assembly and value-flow through normalized_result.parent_evidence_id with inspect_analysis_view, which already projects every native facet. detail: complete returns the full dossier inline as before. Delivery detail never reaches the provider or Evidence identity. BREAKING CHANGE: analyze_function now defaults to the compact summary view instead of the complete dossier inline. Pass detail: "complete" for the previous behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * feat(cli): expose analyze_function detail on the function command Wire `rea function` into the shared CLI detail-delivery convention (--detail, --evidence_output) so the CLI and MCP adapters offer the same summary | complete choice for analyze_function. Summary delivery writes the complete dossier Evidence to a bundle and returns the compact summary view; it fails before analysis when --evidence_output is missing, so no provider work is wasted. The CLI default stays complete because a one-shot command has no in-process Evidence retention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * refactor(ghidra): harden analyze_function summary after independent review FlatlineRoundtable cross-model review (6 lanes) converged on the summary projection's availability semantics being underspecified. Document that facet_availability is authoritative over raw pseudocode length, that code_units are UTF-16 (the unit inspect_analysis_view pages by), and that value_flow.truncated is meaningful only when value-flow is available. Parenthesize the nullish-coalescing/comparison for clarity. No behavior change; declined claims (value-flow "truncated collapse", unresolved_calls dead code) were schema-impossible or already correct. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * test(ghidra): cover analyze_function detail wiring on both adapters Close the roundtable's top convergent finding: the summary path had only domain-projection coverage, not the server/CLI wiring where the breaking behavior lives. MCP server (registerEnhancedTools): the default delivers a retained summary whose complete parent is recorded for paging; delivery detail is stripped before the provider request; detail: complete returns the full dossier inline; a registration without retention rejects a summary request before any provider work. CLI (rea function): the default returns the complete dossier; --detail summary without --evidence_output fails before analysis; with the bundle sink it returns the compact summary and writes the complete Evidence bundle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(ghidra): keep analyze_function delivery detail out of provider inputs Address the five automated review findings on #5660. P1: `detail` lived in the shared enhancedInputSchemas.analyze_function, so inspect_native_api composed `{procedure, detail: "summary"}` and the strict Ghidra analyze_function schema rejected it. Delivery detail is now owned by the analyze_function MCP tool contract; the handler separates it before any provider call, and executeFunctionAnalysisEvidence parses the provider input only. The MCP regression test validates provider input exactly as the production Ghidra client does and exercises inspect_native_api. P2: the analyze_function summary Evidence now carries the parent's relationship_coverage like native facet views, so partial or unknown caller/callee discovery is not reported as complete. P2: the summary reports the dossier's declared facet_availability, nulls the counts of facets the provider declared unavailable, and lists each reason in unknowns, so Kuna's empty references, comments and basic blocks are not mistaken for observed absence. P2: inspect-analysis-view evidence-file sources accept the Evidence bundle that --evidence_output writes, selecting a record with source.evidence_id when the bundle holds several; the CLI test reads the saved bundle back through that route. Tests that assert the complete dossier over public analyze_function now pass detail complete, and the oversized-dossier test asserts the summary view it previously expected to be unsupported. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(ghidra): keep omitted function facets unknown in the summary The analyze_function summary coerced an omitted unresolved_calls record to a count of 0 while the native facet view reports the same dossier as "not recorded", and it carried no counts at all for the pageable swift_string_literals facets. Count optional records as null when the dossier omits them, name them in unknowns, and add nullable counts for both Swift literal facets. The summary Evidence envelope also dropped the function address: createAnalysisViewEvidence now derives one function-view predicate for both the address location and relationship coverage. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(scripts): request complete analyze_function dossiers in real-provider checks The real-provider check scripts call analyze_function over MCP and then consume the complete dossier: requireFunctionDossier, functionDossierSchema, .procedure/.pseudocode reads, operation assertions, compare_functions inputs and the oversized-dossier rejection path. The summary default broke them. Every such call site now passes detail: "complete". Provider-direct callers are unchanged because detail is an MCP contract field; the admission-error sites stay unchanged because provider failures return before the summary branch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(ghidra): keep unavailable value-flow counts unknown in the summary summarizeDossier coerced the five value_flow_* counts to zero when native_value_flow was null (IDA, Kuna) or declared unavailable (Ghidra), so the summary reported empty collections while value_flow.available said otherwise. The counts are now null unless value flow is available, with the reason already named in unknowns. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(evidence): budget authentication headroom by file shape and serialized size The evidence-file headroom reserved two parsed copies, calibrated for one record, and charged strings at two bytes per character. Bundle authentication additionally holds the schema-parsed bundle, the re-parsed records and two JSON.stringify results, which expand control characters and lone surrogates sixfold: a bundle whose raw_result is a million NUL characters estimated at 2 MB parsed but needed 12 MB of serialized text. The budget now has a parsed term and a serialized term per file shape, owned by EvidenceAuthenticationHeadroom and shared by the analysis-view reader and the bundle importer, which had no guard. Two transient peaks the budget did not name are removed at their producers: the canonical digest hashes a large leaf directly instead of concatenating it into its buffer, and the JSON validator walks containers by cursor so a wide array no longer queues one entry per member. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(mcp): route analyze_function summary replies to dossier consumers in docs and prompts The MCP guide, the version-comparison and crash-triage prompt contracts, the authored skill and the Kuna, IDA and function-matching docs still told agents to pass analyze_function replies straight to compare_functions or build_call_path. Under the summary default those consumers reject the reply. They now name the retained-evidence reference for compare_functions, the observations fields of match_native_functions, and detail: "complete" for build_call_path, which accepts only inline dossiers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * test(evidence): pin validator member order and keep interrupted validation yielding Independent review of the cursor validator found the interrupted-validation fixture relied on the old last-first walk: its over-deep head was reached before the first cooperative yield. The padding now precedes the deep element so the walk yields first, which is the invariant the test guards. A new case pins first-defect-in-member-order reporting, skipped non-enumerable keys no longer count toward the yield cadence, and the import-headroom test loads the reader through a fresh module graph so it holds under the services project's shared workers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(ghidra): align deadline bounds and cancellation outcomes * refactor(ghidra): shorten deadline recovery guidance * fix(mcp): clarify the scope of inferred decompile deadlines * test(ghidra): verify deadline workflows and document measured costs Separate failure and registration cases by their distinct contracts. Record actual Linux Ghidra, CLI and MCP evidence with measured context, elapsed time, CPU and memory, and explicit remaining coverage limits. * docs(ghidra): pin integration and distinguish baseline format failures * fix(ghidra): own failure lifecycle wording in the provider * test(ghidra): budget timeout fixture generation before allocation * fix(analysis): preserve summary provenance and bound bundle guidance * fix(ghidra): align capability and traversal deadline guidance * style(config): align deadline guidance formatting * docs(ghidra): record capability and traversal verification * docs(analysis): use absolute paths for Evidence files * docs(analysis): record completed function summary measurements * docs(analysis): update main integration validation --------- Co-authored-by: Syed Rishad <rishadzzzz@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> |
||
|
|
a91412aaf9 |
feat(native): export function pseudocode to resumable directories (#6170)
* feat(decompile): write function pseudocode to a directory and return a manifest Add decompile_all and `rea decompile-all` so a caller can write one provider pseudocode file per function, or a filtered set, and receive only a small manifest. The result stays bounded when the program is larger than a local model's tool-result cap. Resume continues an export for the same target digest, provider, analysis profile, and layout. Matching files are skipped, conflicting files stay unchanged, and one function's failure does not stop the rest. Cancellation leaves files already written. Requesting dwarf layout reports that compilation-unit paths are unavailable. Simulated-provider tests cover the manifest, resume, filters, limits, cancellation, and the failure cap. MCP registration, CLI coverage, schema validity, the product catalog, check:fast, and docs:check passed. Real Ghidra was not exercised. Refs #6090 * fix(decompile): keep unknown export identity unresumable and recover torn journals Resume treated two missing target or profile digests as a match, so an export of one binary could continue into another. Resume now requires both digests, and a decompiler observation without the export's profile, or with a different target, is recorded as profile_changed. A crash during a journal append left a torn final line that made the export unresumable. Text after the last newline is now treated as uncommitted and truncated before the next append, and stale temporary files are removed. An inventory row with an empty address aborted the whole export because its failure record failed the journal schema. Such rows are now excluded and counted in the limitations. A limited resume retried the same recorded failures first and never reached new functions. Functions not yet attempted now run before retries. File path reservations come from every journal line, so an output_io record no longer releases a path. The index is rebuilt from the journal in numeric address order, and journal close and cleanup failures return a Result. The cancellation test now cancels after a file is committed and checks that file survives and is skipped on resume. Redundant snapshot-replay checks are removed, the CLI limit uses the contract constant, and --snapshot no longer claims decompile-all updates it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): refuse resume of a missing export and repair journals by byte offset A second review round found that resume against a directory that does not exist silently started a full new export and reported it as resumed. Resume now requires an existing export directory. The torn-line repair measured the committed journal from decoded text, so damaged UTF-8 could move the truncation point past the end of the file. The journal is now read as bytes and cut at the last newline byte, and an intact journal is not truncated, so a concurrent append is kept. A per-function observation for a different target is now recorded as target_changed instead of profile_changed. A failure to remove a partly created output directory is reported instead of hidden. The journal handle is closed on every path again, including an unexpected throw. The path allocator no longer gives up after 1,000 collisions and reuses a taken name. Tests now use createTestTempDirectory, assert journal lines against the journal schema, and cover resume of a missing directory, resume with an unknown analysis profile, and a target-mismatched observation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): keep a function's first file name across failed retries Reserving every journal path stopped other addresses from taking a path, but an address looked up only its latest record. After its file went missing and a retry failed without a path, the next resume moved the function to a hash-suffixed name and orphaned the original. Each address now keeps the first path the journal gave it. A matched commit still ran the decompiler but did not count toward limit. The limit now counts every function taken on after skipping verified files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): never follow a journal symlink and check procedure_info identity Torn-line repair truncates the journal, and the journal was opened through any symlink in the export directory, so repair could truncate a file outside it. The journal must now be a regular file and is opened without following symlinks where the platform supports it. Body ranges from procedure_info were written without checking that the observation belonged to this export or described the requested function. An observation with a different provider, profile or target, or another entry point, now leaves the body unavailable with that reason. File, journal and index writes now use FileHandle.writeFile, which writes the whole buffer instead of trusting one write call. The stale temporary sweep is removed: a matching file name does not prove this export created the file. The inverted-range test now uses one address space so it reaches the inversion check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): bound returned caveats and keep verified checksums Ghidra names each terminal call's address in its per-function caveats, so decompile_all copied one new caveat per function into the returned Evidence and the answer grew with the program. The returned Evidence now keeps the eight most often observed distinct per-function caveats, grouped by provider and text as they are rendered, plus a count of the rest. Inventory caveats are unchanged, and other workflows are not bounded. Each function keeps the caveats of the observations it used: a written file in its header, now including procedure_info caveats, and a failed attempt in its journal and index record. A procedure_info observation for another target or profile is foreign and contributes none. A failure record for a function that was already written replaced the record holding its checksum, so after a transient read error a resume decompiled that function again. The journal now keeps the last successful write separately from the latest attempt. Resume verifies against that write and records it again once the file verifies. A caller abort that arrives while the export finishes now returns cancellation with the caller's reason instead of a success manifest, as #6143 does for other late finalization. A publish, index or close failure stays the primary error. Also make the missing-journal resume error say what to do, and add boundary tests: one runs decompile-all through the real CLI against a fake Kuna to check flag mapping, the relative output directory, filters and limit; one runs decompile_all through production MCP registration to check that the caveat bound is applied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): keep each function's complete caveats in the index The returned Evidence keeps eight distinct per-function caveats and counts the rest, pointing readers at the export. A written function's file header truncates its caveats after 4,000 characters, and its journal record kept none, so a large function with many terminal calls could lose caveats from both the answer and the export. Every journal record, written or failed, now keeps the complete caveats of the observations its attempt used, and decompile-index.json carries them. The header stays a readable summary and, when truncated, says that the index keeps the complete list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): report external functions instead of failing them Ghidra classifies imports as external functions with an empty body and returns null pseudocode for them, as its limitations state. decompile_all recorded each one as an invalid_pseudocode failure ("not text"), so a dynamically linked program reported every import as a decompiler failure: 34 of 116 functions in /usr/bin/xxd. procedure_info owns the external classification. When it classifies a function as external, the export now records status "external", counts it in counts.external, writes no file and requests no pseudocode. The record keeps the body fact as observed. The journal header pins the target, provider and profile, so a resume skips a recorded external function without asking the provider again; a limit below the external count still leaves nothing remaining. A null pseudocode result for a function whose classification is not external is reported as no_pseudocode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): report finalization failures after a cancelled export When the caller's abort stopped the publish loop, decompile_all returned that cancellation before checking the index write and the journal close, which had already run. A failure to finalize the export was hidden, so the caller could not tell the saved index or journal was incomplete. Publish failures stay primary, then index and close failures. A cancellation counts as the caller's only when the caller's signal is aborted, and is then reported after a successful finalization with the caller's own reason. A provider's own cancellation, without a caller abort, stays a publish failure like any other. Tests cover an index failure and a close failure after the caller stops the loop, a stopped loop with a successful finalization keeping the caller's reason, and a provider cancellation staying primary over an index failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): remove a created output directory that cannot be secured createPrivateDirectory ran mkdir and chmod as one step. When mkdir succeeded and chmod failed, it returned an error before #create reached its cleanup, leaving an empty directory behind. A fresh export then refused the existing directory, and a resume refused it for having no journal, so the caller had to remove it by hand. mkdir and chmod are now separate. A mkdir failure created nothing, so nothing is removed. A chmod failure after this call created the directory removes it with a non-recursive rmdir: it is still empty, so a path another process filled or replaced meanwhile is refused rather than deleted, and that refusal is reported. A filesystem boundary test fails chmod for the new root and checks the directory is gone and a retry succeeds. A module test confirms that a function written earlier and now classified as external still reports a changed file at its path as output_exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * refactor(process): share the socket lease with owners other than Hopper The socket lease that keeps two REA processes from opening the same Hopper document lived only in HopperLease, with Hopper error messages built in. A decompile export needs the same guarantee for its output directory. OwnedLease now holds the lease mechanism unchanged: a listening local socket that the operating system ends with its process, owner identity answered by the live holder, and recovery of a stale socket serialized through a recovery directory. Callers map a tagged LeaseFailure to their own error. HopperLease keeps its exports and maps each failure to the same HopperStartError message as before. On Windows the lease is a named pipe, which needs no stale recovery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): hold the output directory per run and stop on repeated write failures Two runs could resume the same output directory at once. Each read the journal once, and the index each wrote reflected only its own view, so the last writer could drop the other run's records. A run's torn-line repair could also truncate the other run's append. DecompileExport now holds a lease on the output directory, named by its device and inode, from before the journal is read (or, for a new export, from just after the directory is created and before its journal exists) until the journal is closed. A second run is refused as a retryable capacity constraint that reports the holder's process ID. The lease ends with its process, so a crashed run leaves no lock to clear. The lease is released on every open failure, and a failed release is reported. An export also kept decompiling every remaining function when its files had stopped being writable mid-run, for example on a full disk, failing each one with output_io. After five consecutive functions whose files could not be written or read, the export now stops with an output error that carries the counts and the last failures. A verified file or any other outcome ends a run of failures. Committed files and records stay, the index is still written, and a later resume continues. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * docs(native): record directory export verification Record matched real Ghidra outputs, bounded caveat and cleanup regressions, actual catalog-inclusive context estimates, runtime measurements and coverage limits for the pinned integration. * fix(native): bound export summaries and reserve portable paths Use one case-folded reservation owner for allocation and resumed journals. Budget inline failures and provider caveats by encoded JSON bytes while retaining complete records in the output index. Align the tool contract with the collector order and verify SDK result-window behavior. * docs(native): refresh final export measurements --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> |