Add opt-in REA_GHIDRA_CHECKPOINT_DIR for owned Ghidra Program persistence through the shared CLI/MCP provider. Save completed initial analysis and metadata changes on normal close; verify and restore the last successful packed Program without auto-analysis.
Preserve generation identity, exclusive ownership, typed cancellation and cleanup retry. Refresh checkpoint-dependent profiles after retirement, including failed-switch restoration, while retaining caller selection provenance. Keep source executables and caller projects unchanged.
Validated with 720 focused tests (3 platform skips), 13 real Ghidra CLI/MCP scenarios, static/docs checks and CI. Review findings addressed. No version bump or release publication.
Refs #8165.
* fix(filesystem): settle a Windows ChangeTime before recording a baseline
Windows advances a file's ChangeTime shortly after it is written while
security components attach catalog and file-hash extended attributes. A
baseline captured during that advance disagrees with the next observation,
so an untouched file was reported as changed: "Root artifact changed before
inventory" and "Selected input changed while being read".
Dropping ctime on Windows is not an option because it is the only field
that catches a same-size rewrite whose modification time was restored.
Instead, keep ctime at full strength and stop capturing it mid-flight: on
win32 a ChangeTime still inside the drift window is observed again until two
consecutive observations agree, and the settled value becomes the baseline.
Applied where a baseline is captured and later compared: openRegularFile's
admitted metadata and the inventory root snapshot. The exhaustive process
capture walk is left alone, since settling there would delay every freshly
written file in the tree.
Refs #8975
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(filesystem): wait out the drift window instead of a repeated value
Settling on two matching observations was not enough. Two reads taken before
Windows performs its delayed ChangeTime update agree with each other and
disagree with every later one, so an unchanged file still failed a later
check: the reported drift lands 0.3-1.0 s after the write, well past a pair
of reads 25 ms apart.
Decide on the observation's age in the drift window instead, and keep
observing until the ChangeTime itself has aged out. Both the first read and
the empty-observation path now go through the same window check.
The window-exit test could not fail before: its clock was frozen, so the
window never closed and the exit branch never ran. Advance the clock while
sleeping and cover a match that precedes a delayed update.
Refs #8975
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(filesystem): configure Windows ChangeTime settle limits
REA_WINDOWS_CHANGE_TIME_DRIFT_MS, REA_WINDOWS_CHANGE_TIME_SETTLE_POLL_MS, and REA_WINDOWS_CHANGE_TIME_SETTLE_MAX_POLLS keep today's 2000 ms, 25 ms, and 40-poll defaults. Invalid values name the setting, current value, source, and range, and say to restart REA.
* fix(filesystem): settle on the selected environment and stop when cancelled
`settleChangeTimeBaseline` resolved its limits from `process.env`, but the
inventory already carries the environment its owner selected. A Windows
runtime with explicit settle settings therefore ran on ambient values or
the defaults, and an invalid ambient value could fail a runtime whose own
configuration was valid. `scanCanonical` now passes `options.environment`
through.
The wait loop also never observed the owning operation's signal, so a
cancelled inventory could keep polling for the whole drift window — up to
a minute at the configured maximum — and closing the artifact scope waited
on it. The loop now checks the signal before each poll and after the
sleep, and both call sites pass theirs.
Refs morluto/rea#8979
* fix(filesystem): keep the caller's typed error on cancellation
Checking the signal was not enough: `throwIfAborted` threw the raw
`signal.reason`, which `openRegularFile` passed through unchanged. Its
callers recognise cancellation through `RegularFileOperationCancelled`, so
a cancelled admission surfaced as an `ArtifactReaderFailure` with reason
`io` instead of a cancellation.
Let the seam supply the error a caller matches on, and have
`openRegularFile` pass its own cancellation type.
Refs morluto/rea#8979
* fix(filesystem): keep inventory cancellation typed during ChangeTime settle
scanCanonical passed the abort signal into settleChangeTimeBaseline without a cancellationError, so an aborted inventory threw the raw AbortError and analyze reported execution_failure. Inventory now throws ArtifactReaderFailure reason cancelled, which callers already translate as cancellation.
* fix(filesystem): skip ChangeTime settling for private PYZ images
External admission still waits out a fresh Windows ChangeTime. A private image REA just wrote is recorded as written, so a compressed PYZ spool does not add about a second per image.
---------
Co-authored-by: 2SinceJuly <202121119+2SinceJuly@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
Add per-connection discovery that loads canonical named tools on demand while preserving the default full catalog. Bound discovery replies before changing visibility, preserve family exclusions and Evidence workflows, and reset visibility on HTTP reconnect.
Verified with focused protocol tests and real Codex CLI artifact investigations; token costs remain task-dependent, so the profile is opt-in.
* fix(doctor): report provider settings a client registration does not record
Provider checks read the terminal running doctor, so a registration with no
env block looked ready while its server reported not_configured. Compare what
an available provider relied on with the registration's own file and report the
setting names in a non-blocking registration_environment check. The client's
effective environment stays unknown, so health and the exit status are unchanged.
Fixes#8202
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(doctor): write the Cordis fixture at its literal path
The test resolved the path through supportedClients, whose default environment
argument CodeQL traces to the fixture's writes even though the test passes {}.
It only needed the path, so spell it like the other fixtures in this file and
drop a leftover directory it never used.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(hopper): reconcile observed caller relationships
* fix(hopper): retain completed caller scans on failure
Attach the native reply and collected forward facts when reconciliation
fails or is cancelled, and check scan stop codes separately from their
wording. Live observations keep REA evidence ids rather than provider
supplied ones, and the real Hopper verifier uses advertised MCP schemas.
* fix(hopper): answer caller-scan methods in the package fixture
The packaged Hopper stand-in echoed unknown bridge methods, so
analyze_function rejected rea_caller_scan_target as an invalid scan
target and the native package baseline returned unreadable_output.
* fix(hopper): share caller identities in failed scan observations
Failure partials repeated each caller identity under every destination.
The page budget counted compact replies, then schema parsing copied the
expanded form. Identities are stored once and destinations cite addresses.
* refactor(config): own the Hopper caller-scan budget default
The environment schema hard-coded the 2000 ms scan default. The domain
limit owner now names that millisecond budget and the schema reuses it.
* test(mcp): assert bundle path issues by field
compare_bundles reports each relative path as an invalid_format issue
instead of one joined diagnostic sentence.
* test(flutter): assert ABI selector refusals as structured issues
inspect_dart_aot reports the ABI label as an issue on path abi. The
message is the selector contract, without an abi: prefix.
* fix(session): retain caller observations that cannot be replayed
A missing document selector blocks snapshot replay. Backward call-graph
and trace reads still keep the native result and forward observations,
and the workflow Evidence links those records.
* fix(session): keep decompile bodies out of the in-memory ledger
Caller and callee walks still stay in Evidence without a document.
procedure_pseudo_code does not, so decompile_all leaves each body in the
file-backed export instead of retaining the whole export in memory.
* fix(hopper): own caller-scan recovery and the page byte allowance
Scan facts stay in the domain. Environment recovery, including the time
budget bounds and the fixed allowances with no runtime override, is
appended from configuration. Setup forwards the budget, and the bridge
uses one byte allowance for instructions, native callers, and pages.
* fix(hopper): retain compact caller references within the scan budget
Recovered call sites stay in the address form the scan already counted.
Expanded identity copies are measured before allocation and refused when
they exceed the retention budget. Instruction lists are not part of the
retained reply.
* fix(session): omit instruction payloads from retained caller evidence
Documentless caller and callee reads stay in the ledger for the session.
Their evidence keeps call sites and drops instruction lists the scan
already consumed.
* fix(hopper): keep the owning caller address on each compact call site
A page reference marked source_procedure "caller" belongs to that page
item's caller. Saved sites record that address, and an owner missing from
the callee inventory is stored once instead of copied onto every site.
* fix(session): leave Hopper reply shaping to the Hopper adapter
Documentless caller evidence stores the adapter's raw result unchanged.
The session no longer reads native_result.instructions.
* test(hopper): throw typed scan failures before reading results
Successful caller-scan results now throw the typed error instead of
returning after an ok check. Expected failures assert structured tags,
stop codes, and reported limits.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Reuse the pinned SDK's combined array-element and object-member admission
before canonical validation, deferred runtime loading and tool execution.
Share a configurable finite allowance across eager/deferred HTTP and stdio,
and project refusals to bounded, actionable transport resource errors.
Preserve canonical CLI/application contracts, caller-selected input values,
ordinary validation semantics and complete Evidence access. Exercise actual
transport boundaries, real artifact/TShark workflows, packaged installation,
continued service and downstream identity/capacity regressions.
Fixes#7882
BREAKING CHANGE: MCP tool arguments now default to 1,024 combined array
elements and object members per call. Select a larger positive finite
REA_MCP_MAX_INPUT_ELEMENTS allowance and restart for larger inline inputs,
or use retained/file selectors where supported. Inputs are never truncated.
Assisted-by: Reimu (gensokyo-skills:reimu-incident-triage)
Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)
Assisted-by: Yukari (gensokyo-skills:yukari-boundary-analysis)
Reuse selected Ghidra projects through verified private read-only copies, with shared CLI/MCP configuration, committed project identity, import-SHA verification and explicit historical/unknown analysis semantics. Preserve exact cancellation and platform capability boundaries.
Actual Ghidra 12.1.4/JDK21 Linux CLI/MCP investigations matched fresh import on 652-function ls and 14,338-function libcrypto inventories and pseudocode; authenticated complete Evidence, mismatch/staleness refusal, saved historical reports and shared /tmp projects verified. 958 focused tests passed with four platform skips; build/static/docs/format passed. Real macOS/Windows reuse remains unverified; Windows reuse is unavailable. See PR verification for measured whole-task costs and final CI attribution.
Co-authored-by: akushonkamen <akushonkamen@163.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
Allow native matching representation budgets to be selected at runtime or launch while retaining the 96 MiB default and main shared-provenance accounting. Preserve actionable configuration recovery through the setting owner and verify complete CLI/MCP matches for 20k/50k synthetic inventories and real stripped LLVM 14/15 inventories.
Fixes#6406.
Based on #7112 by Kaoru0822; original authorship and co-author credit retained.
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(ghidra): bound one procedure's decompile without closing the session
Fixes#5340
* fix(ghidra): report decompile limit as a request timeout
Keeps BinarySession from marking every Ghidra operation unavailable after one procedure's decompile reaches the limit, and names the MCP tool-call deadline in the remediation.
* fix(ghidra): budget the decompile limit and report cancellation-closed sessions
Adds REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS to the derived Codex/Kimi tool deadline and forwards it through setup. Marks a remote failure that settles after caller cancellation so the timeout projection no longer promises session reuse.
* fix(ghidra): align deadline bounds and cancellation outcomes
* refactor(ghidra): shorten deadline recovery guidance
* fix(mcp): clarify the scope of inferred decompile deadlines
* test(ghidra): verify deadline workflows and document measured costs
Separate failure and registration cases by their distinct contracts.
Record actual Linux Ghidra, CLI and MCP evidence with measured context,
elapsed time, CPU and memory, and explicit remaining coverage limits.
* docs(ghidra): pin integration and distinguish baseline format failures
* fix(ghidra): own failure lifecycle wording in the provider
* test(ghidra): budget timeout fixture generation before allocation
* fix(ghidra): align capability and traversal deadline guidance
* style(config): align deadline guidance formatting
* docs(ghidra): record capability and traversal verification
---------
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
* fix(reconstruction): dedupe residual unknown ids with uniqueSorted
Current main calls uniqueBy, which is not defined, so typecheck fails.
Residual unknown ids are strings, and the adjacent lists already use
uniqueSorted.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* feat(config): make the bridge request ceiling configurable
REA_BRIDGE_MAX_REQUEST_BYTES replaces the hard-coded 16 MiB Hopper and
Cutter request payload ceiling. The default stays 16777216 bytes. Values
outside 7464 to 268435456 bytes are configuration_invalid.
Part of #6176.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): keep cleanup ownership when a sent command's reply is oversized
An authenticated command may already have changed the Cutter document
when its reply exceeds the frame ceiling. Return the existing unknown
execution, with the ceiling diagnostic in the message, so receipt
cleanup still waits. A frame that overflows before the command is sent
remains a resource constraint.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* style: format the bridge request ceiling sources
oxfmt --check failed on the five files that set and test
REA_BRIDGE_MAX_REQUEST_BYTES. Formatting only; an oversized Cutter
reply still stays an unknown execution.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): exclude the LF from the response frame budget
REA_BRIDGE_MAX_REQUEST_BYTES counts payload bytes. The Cutter response
reader and authenticated encoder were reserving one byte for the
terminating LF, so an exact-limit response was rejected. A reply that
exceeds the ceiling after the command was sent still stays an unknown
execution.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(bridge): measure escaped JSON before allocating the request
A command of U+0001 characters can expand sixfold. JSON.stringify of
that expansion can throw past Node's string ceiling before the payload
check. Count the escaped size within REA_BRIDGE_MAX_REQUEST_BYTES
first and refuse it as a resource constraint.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(config): derive the bridge request maximum from JSON expansion
A 256 MiB ceiling still lets a shorter U+0001 command expand past Node's
536870888-character string limit. The maximum is now
floor((536870888 - 296) / 7) = 76695798, the longest such command whose
authenticated frame fits in one string. Requests are still measured
before JSON.stringify.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(bridge): size the Cutter frame deadline and send the measured Hopper payload
The authenticated Cutter command is encoded after the challenge, so a
one-second read rejected an in-budget frame. That read now allows one
second per mebibyte of the ceiling and never waits longer than 25
seconds. Hello stays at one second. Hopper writes the payload it already
measured instead of serializing the same request again.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(config): bound the bridge request maximum by the serialized string
The ceiling already measures escaped JSON, so dividing Node's string limit by the seven-byte expansion rejected encoded payloads that still fit. The maximum is 536870887, one byte under that string ceiling, leaving room for the framing LF.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(hopper): serialize a bridge request when it reaches the wire
The queue kept each encoded frame, plus the caller's params, until Hopper finished the active request. A raised ceiling made a few waiting calls enough to exhaust the heap. The FIFO now keeps the params, and the frame is encoded only for the request being written.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): hash the bridge proof prefix and payload separately
An exact-limit Cutter frame plus the HMAC prefix exceeds Node's single-string ceiling, so the client threw before sending an admitted command. The prefix and payload are now separate HMAC updates, including the response proof.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): keep recovery fields when a status path exceeds the frame
A long current_file made the status reply miss the ceiling, and the fallback then dropped session_id and the completion receipts. The status now omits the path and version and still returns the session, connection proof, and up to 64 receipts. That required recovery frame is smaller than the 7464 floor.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): report an unknown path when status identity does not fit
A bounded status kept the new document generation and omitted the path, so the client filled in the descriptor's older path. The observed path is sent when it fits. When it does not, current_file is null and is not replaced with that stale path.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): count structured replies by their encoded JSON size
The response guard charged 16 bytes per primitive and two bytes per list slot, so a dense array under the configured ceiling was replaced with output_truncated. Numbers, booleans, null, and container punctuation now use their JSON lengths.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): stream authenticated responses in bounded chunks
A fitting reply was signed from retained raw bytes, escaped bytes, and a
joined frame, so a ceiling near 512 MiB could exhaust Cutter before send.
Measure and sign JSON pieces, then write the same frame in 64 KiB chunks.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(bridge): keep Cutter admission from stalling or hiding sessions
Read an unauthenticated frame off the accept loop. A command reply that
does not fit keeps completion and generation and reports an oversized path
as null. Discovery reports a ceiling mismatch instead of an absent session.
The maximum HMAC proof is opt-in, and both bridges reject non-ASCII digits.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): bound frame scans and client response copies
Search only newly read bytes for the framing LF. Unescape each socket
chunk into one response string, check the proof, and then parse it.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): keep completion, admission, and dense replies bounded
Report completed_connection only through connections that have
terminated. Cap connection threads and keep the listener up if one
fails to start. Batch scalar JSON and fail discovery when any ceiling
mismatches.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): hash requests in parts and scan escapes once
Update the client-proof prefix and request separately. Stop the response
decoder at the first quote or backslash instead of rescanning the tail.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(cutter): confirm live ceiling mismatches and omitted output
A stale descriptor from a previous ceiling is not a discovery mismatch
until its hello proves a live authenticated bridge. A response discarded
for the frame ceiling after dispatch reports output_truncated.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* test(cutter): close the unauthenticated ceiling-mismatch fixture
The listener that fails the hello proof is destroyed after its challenge
so discovery cleanup does not wait on that socket.
Co-authored-by: morluto <morluto@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>
* feat(mcp): make MCP startup phase budgets configurable
REA_MCP_INITIALIZE_BUDGET_MS, REA_MCP_TOOLS_LIST_BUDGET_MS,
REA_MCP_FIRST_CATALOG_BUDGET_MS, and REA_MCP_DOCTOR_DEADLINE_MS keep
today's defaults and refuse values outside 1 to 2147483647 ms. Codex's
30 second startup figure stays client guidance.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(mcp): enforce the first-catalog budget while phases are pending
The packaged startup probe applied each phase timeout on its own and
checked REA_MCP_FIRST_CATALOG_BUDGET_MS only after both requests
finished. A shorter first-catalog budget now clamps the in-flight
request and cancels it from one deadline measured at startup.
Co-authored-by: morluto <morluto@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>
* feat(captures): make the HAR/SAZ/qlog decoder heap configurable
REA_CAPTURE_DECODER_OLD_GENERATION_MIB keeps the 192 MiB default and
refuses values outside 8 to 16384 MiB instead of falling back.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(captures): defer decoder startup and share one parsed heap
Invalid REA_CAPTURE_DECODER_OLD_GENERATION_MIB threw while CLI commands
were registered, so doctor and help never ran. Build the capture service
only when inspect-web-network-capture runs.
Resolve the heap once in composition and pass that fact to each Node
adapter. Limitation and memory-failure reports use the selected adapter's
heap, so a decoder built with a different environment does not describe
another limit.
Mock captureDecoderProcessMemoryLimitBytes in the Node adapter test so a
Windows Bun host still exercises the unavailable-bound refusal.
Co-authored-by: morluto <morluto@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>
* feat(process): make the regex worker heap request configurable
REA_REGEX_WORKER_HEAP_REQUEST_MIB keeps the 128 MiB default and refuses
values outside 8 to 16384 MiB. A positive Node --max-old-space-size still
overrides the request and is not clamped by the setting.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(process): set the regex worker heap floor to the measured startup bound
On Node 22.23.4 a 5 MiB old-generation request aborts the regex worker
during startup, and 6 MiB starts it and returns a match. The setting floor
is that measured bound.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(process): raise the regex worker heap floor to 7 MiB
6 MiB passes configuration but returns ERR_WORKER_OUT_OF_MEMORY for a
trivial match on supported Linux Node 24 and Node 26.0.0. 7 MiB returned
that match on the measured 22, 24, and 26 lines. A clean-process test
runs the production worker at the floor.
Co-authored-by: morluto <morluto@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>
* feat(config): make the regex matching deadline configurable
REA_REGEX_MATCHING_DEADLINE_MS defaults to 5000 ms and refuses values
outside 1..2147483647. Recovery and capability text name the effective
deadline, its source, and how to change the setting.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(setup): preserve the regex matching deadline in registrations
Setup validated REA_REGEX_MATCHING_DEADLINE_MS but left it out of the
selected registration environment, so a caller override never reached
the launched server and matching fell back to 5000 ms.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* test(config): restore a stubbed regex matching deadline on failure
Kuna deadline assertions now unstub the environment in finally, so a
thrown construction or expectation cannot leave the setting stubbed.
Co-authored-by: morluto <morluto@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix: order remaining outputs independently of host locale
String.prototype.localeCompare without a pinned collation follows the
host's ICU default locale, so the same input could be ordered differently
on different machines. 614fb768 moved canonical domain projections to
compareUnicodeCodePoints; provider, application, catalog, and setup
orderings still used locale collation.
Order every remaining source site by Unicode code point, including Mach-O
symbols, firmware extraction paths, ASAR entries (previously pinned to
"en" collation), evidence ID sets, browser and Electron target, script,
and resource inventories, reconstruction obligations, capability and
provider listings, and setup environment details.
The current tool and prompt catalog names order identically under both
comparators, so the catalog identity digest is unchanged.
* fix(artifacts): order ASAR entries by their portable path
@electron/asar lists host-spelled paths, so code-point order over the raw
spelling still differed by host: "a\b" sorts after "a0" on Windows while
"a/b" sorts before it on POSIX. Pair each listing with its portable
archive path once, order by that path, and keep the host spelling for
@electron/asar calls and adapter keys.
* feat(ghidra): enable Windows x64 read-only analysis
* test(ghidra): exercise native Windows runtime boundaries
* test(ghidra): verify current Windows inventory and isolate ACL observer
* fix(ghidra): align Windows admission and address P0 review
morluto