diff --git a/.agents/skills/rea-analysis-optimization/SKILL.md b/.agents/skills/rea-analysis-optimization/SKILL.md index 764dc2eba0..3e22a0c055 100644 --- a/.agents/skills/rea-analysis-optimization/SKILL.md +++ b/.agents/skills/rea-analysis-optimization/SKILL.md @@ -10,8 +10,8 @@ equivalent completed investigations. A useful outcome can also be a measured decision to leave an implementation alone. Follow the checkout's [performance priorities](../../../AGENTS.md#performance), -[tool design](../../../docs/tool-design.md), and -[testing guide](../../../docs/testing.md). This skill supplies an optimization +[tool design](../../../docs/development/tool-design.md), and +[testing guide](../../../docs/development/testing.md). This skill supplies an optimization workflow, not additional permission to execute targets, publish, or merge. Use the current contracts, provider prerequisites, and user authorization. diff --git a/.agents/skills/rea-changelog-update/SKILL.md b/.agents/skills/rea-changelog-update/SKILL.md index 215204d829..c4e2ced530 100644 --- a/.agents/skills/rea-changelog-update/SKILL.md +++ b/.agents/skills/rea-changelog-update/SKILL.md @@ -6,7 +6,7 @@ description: Prepare or rewrite REA release changelogs and GitHub release notes # REA changelog update Use this for release-note preparation and contributor attribution. Read -[docs/releasing.md](../../../docs/releasing.md) from the repository root for +[docs/development/releasing.md](../../../docs/development/releasing.md) from the repository root for publication authority and the existing checkpoint checks. This skill prepares notes; it does not authorize merging, tagging, or publishing. diff --git a/.agents/skills/rea-ci-tests-audit/SKILL.md b/.agents/skills/rea-ci-tests-audit/SKILL.md index 4d7b44bf5c..246bb6dcae 100644 --- a/.agents/skills/rea-ci-tests-audit/SKILL.md +++ b/.agents/skills/rea-ci-tests-audit/SKILL.md @@ -9,7 +9,7 @@ Establish which checks still catch real regressions and which costs can be removed without losing that protection. The outcome may be a verified fix, an evidence-backed recommendation, or an explicit decision to stop. -Follow the checkout's `AGENTS.md`, [testing guide](../../../docs/testing.md), +Follow the checkout's `AGENTS.md`, [testing guide](../../../docs/development/testing.md), and [contribution rules](../../../CONTRIBUTING.md). Use their current commands and pinned toolchain, rather than commands remembered from an earlier audit. Preserve the requested scope: investigation alone does not authorize edits; diff --git a/.agents/skills/rea-tool-design/SKILL.md b/.agents/skills/rea-tool-design/SKILL.md index 6d36fa7aac..536600a42f 100644 --- a/.agents/skills/rea-tool-design/SKILL.md +++ b/.agents/skills/rea-tool-design/SKILL.md @@ -5,7 +5,7 @@ description: Audit, investigate, design, or change REA investigation tools, CLI/ # REA tool design -Use the [tool-design guide](../../../docs/tool-design.md) for tool boundaries, +Use the [tool-design guide](../../../docs/development/tool-design.md) for tool boundaries, contract semantics, provider ownership, and discoverability. Inspect the current contract and nearest existing tool relevant to the analyst question; source, documentation, and live catalogs may describe different versions. @@ -14,12 +14,12 @@ For provider-backed work, exercise the analyst's task through REA and compare relevant native provider and independent artifact observations where available. Check applicable partial results, recovery and session boundaries; report what was not exercised. For failure guidance, follow [design recovery around the -failure](../../../docs/tool-design.md#design-recovery-around-the-failure). -Use [workflow verification](../../../docs/testing.md#provider-backed-workflow-verification) +failure](../../../docs/development/tool-design.md#design-recovery-around-the-failure). +Use [workflow verification](../../../docs/development/testing.md#provider-backed-workflow-verification) to select evidence for the proposed contract instead of seeking provider API parity. -For an audit, follow [audit existing tools](../../../docs/tool-design.md#audit-existing-tools): +For an audit, follow [audit existing tools](../../../docs/development/tool-design.md#audit-existing-tools): scope representative tasks, exercise the live workflow, and distinguish reproduced bugs, supported limitations, design opportunities and unverified claims. Check relevant rendered MCP prompts, guided workflows and packaged skill @@ -30,7 +30,7 @@ For a design request, return the proposed boundary, input/result shape, Evidence semantics, nearest alternative, and verification approach. Implement when requested, preserving the user's scope and existing authorization. -For implementation or verification, consult [testing](../../../docs/testing.md) +For implementation or verification, consult [testing](../../../docs/development/testing.md) for the affected consumer and provider lanes, and [contributing](../../../CONTRIBUTING.md) for generated-file ownership. Read provider/workflow guides only as the task requires. diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md index 0e43a2ec25..f8e47bbb7c 100644 --- a/.agents/skills/reverse-engineer-anything/SKILL.md +++ b/.agents/skills/reverse-engineer-anything/SKILL.md @@ -108,7 +108,7 @@ explicit path or endpoint and do not need it. - Firmware image: `inspect_firmware_regions` when advertised. Use `extract_firmware` when extraction is requested, with a caller-selected new absolute output directory. These tools use caller-supplied Binwalk/Unblob on - Linux; see the [firmware guide](https://github.com/morluto/rea/blob/main/docs/firmware-analysis.md). + Linux; see the [firmware guide](https://github.com/morluto/rea/blob/main/docs/guides/firmware/firmware-analysis.md). - ESP32-family app or bootloader image (`0xE9` magic): `inspect_artifact` without a provider; see the ESP section of `references/native-and-artifacts.md`. - .NET NativeAOT PE/ELF: native Ghidra analysis. Native code can yield pseudocode; @@ -150,7 +150,7 @@ explicit path or endpoint and do not need it. old-generation heap is `REA_CAPTURE_DECODER_OLD_GENERATION_MIB` (default 192 MiB, floor 8, maximum 16384), set where REA starts. Host memory and `NODE_OPTIONS` do not raise it. See - [historical captures](https://github.com/morluto/rea/blob/main/docs/web-network-captures.md) for the exact upstream + [historical captures](https://github.com/morluto/rea/blob/main/docs/guides/web/web-network-captures.md) for the exact upstream profile and credential exclusions. Qlog support starts with aioquic 1.5.0 JSON 0.3; stream coverage and endpoint loss declarations do not recover bodies or establish network loss. Prefer detail summary, then kind qlog trace/stream @@ -166,7 +166,7 @@ explicit path or endpoint and do not need it. display filters and decode-as rules. Requires Linux, util-linux prlimit and TShark 4.x on an absolute `PATH` entry or via the optional absolute `REA_TSHARK_COMMAND` override; see - [capture guide](https://github.com/morluto/rea/blob/main/docs/packet-captures.md). + [capture guide](https://github.com/morluto/rea/blob/main/docs/guides/data/packet-captures.md). Packet limits count input packets before filtering. Output defaults to 50 matching frames; use `match_offset` / `match_limit` for later pages, or explicit null limit for complete matching output. Preserve the source digest across pages. @@ -174,7 +174,7 @@ explicit path or endpoint and do not need it. For address/DNS/SNI/protocol summaries use `inspect_pcap_capture` with MCP detail summary, then retained kind pcap pages. Requires Wireshark 4.x tshark and capinfos, with Linux prlimit or Windows Job Objects; see - [packet summaries](https://github.com/morluto/rea/blob/main/docs/pcap-analysis.md). + [packet summaries](https://github.com/morluto/rea/blob/main/docs/guides/data/pcap-analysis.md). Select an exact DNS question for address pages; page exhaustion does not prove network completeness. This reads recordings and does not acquire traffic. - Decoder discovery: `inspect_signal_decoders` searches installed decoder IDs @@ -182,7 +182,7 @@ explicit path or endpoint and do not need it. relevant details before choosing settings; this does not scan hardware. - Recorded logic/signal traces: `decode_signal_recording` with explicit decoder, channel and protocol settings. On Linux with util-linux prlimit, supply sigrok-cli 0.7.2 through - `REA_SIGROK_COMMAND`; see [recording guide](https://github.com/morluto/rea/blob/main/docs/signal-recordings.md). + `REA_SIGROK_COMMAND`; see [recording guide](https://github.com/morluto/rea/blob/main/docs/guides/signals/signal-recordings.md). Output defaults to 50 literal lines; select annotation classes and page with `line_offset` / `line_limit`, or explicit null for all remaining output. Window counts are display lines, not annotations or samples. Preserve source digests across pages. @@ -191,7 +191,7 @@ explicit path or endpoint and do not need it. through `REA_WABT_BIN_DIRECTORY`; retain exact byte/tool identity and decoded WAT. Select JavaScript glue and candidate paths explicitly and preserve URL/path ambiguity. No implicit fetch or target execution; WAT is not original source. - See the [WASM guide](https://github.com/morluto/rea/blob/main/docs/wasm-artifacts.md). + See the [WASM guide](https://github.com/morluto/rea/blob/main/docs/guides/wasm/wasm-artifacts.md). - Explicit local Android APK (or ZIP/AAB) signing: `inspect_apk_signature`. Prefer its summary or selected signer/digest/lineage page; retain complete Evidence and use `inspect_analysis_view` kind `apk-signature` for follow-ups. @@ -210,13 +210,13 @@ explicit path or endpoint and do not need it. Evidence stays retained under its original identity. Use the returned `parent_evidence_id` for exact function or `bytecode_hex`/stdout/stderr pages. Candidate coverage does not establish interface completeness. See the - [EVM bytecode guide](https://github.com/morluto/rea/blob/main/docs/evm-bytecode.md). + [EVM bytecode guide](https://github.com/morluto/rea/blob/main/docs/guides/evm/evm-bytecode.md). - Embedded Go compiler, module or build-setting metadata: `inspect_go_binary` with the explicit local file. This built-in target-free inspection requires no Go installation or disassembler. Preserve raw module bytes and source ranges; absent metadata does not prove a binary is not Go, and unparsed text records remain explicit. See the - [Go binary guide](https://github.com/morluto/rea/blob/main/docs/go-binaries.md). + [Go binary guide](https://github.com/morluto/rea/blob/main/docs/guides/native/go-binaries.md). - Embedded CUDA/HIP code objects: `inspect_gpu_code_objects`; inventory targets first, then decode selected payloads and query retained GPU Evidence. - Bundled GGUF or safetensors model declarations: `inspect_model_artifact` @@ -333,17 +333,17 @@ explicit path or endpoint and do not need it. 20 rows; request `schema_scope: "database"` for the complete stored schema. Preserve both database and existing sibling WAL identity; row samples report truncation and have unspecified order. Use a quiescent snapshot rather than assuming a live database/WAL pair was copied atomically. See the - [SQLite guide](https://github.com/morluto/rea/blob/main/docs/sqlite-databases.md). + [SQLite guide](https://github.com/morluto/rea/blob/main/docs/guides/data/sqlite-databases.md). - Explicit Linux ELF file for offline layout, symbols, relocations or static mitigation evidence: `inspect_binary_layout`. This target-free operation uses caller-supplied pwntools without opening a disassembler database. Preserve its raw locations and inference/coverage limits; see the - [offline binary guide](https://github.com/morluto/rea/blob/main/docs/binary-diagnostics.md). + [offline binary guide](https://github.com/morluto/rea/blob/main/docs/guides/native/binary-diagnostics.md). - Supplied Linux x86-64 ELF core: `inspect_recorded_crash`. Read every recorded thread, raw note and source-bound register without opening a live target. Historical PIDs do not select live processes. Request `include_debugger_context` only when core-only mapping candidates help; see the - [recorded crash guide](https://github.com/morluto/rea/blob/main/docs/recorded-crashes.md). + [recorded crash guide](https://github.com/morluto/rea/blob/main/docs/guides/native/recorded-crashes.md). - PE resource type/name/language trees and static icon group references: `inspect_pe_resources` with an explicit path. This portable operation requires no active target or engine; opaque payloads retain file ranges and SHA-256. @@ -359,9 +359,9 @@ The legacy attached profile binds the already-open GUI input by SHA-256 and leaves its database open. The modern headless profile creates and closes an owned private database without saving. Installation and registration reuse [mrexodia/ida-pro-mcp](https://github.com/mrexodia/ida-pro-mcp); see the -[IDA provider guide](https://github.com/morluto/rea/blob/main/docs/ida-provider.md). +[IDA provider guide](https://github.com/morluto/rea/blob/main/docs/providers/ida-provider.md). For migration to the separate native IDA MCP plugin, follow the guide's -[legacy attached migration](https://github.com/morluto/rea/blob/main/docs/ida-provider.md#migrate-a-legacy-attached-registration). +[legacy attached migration](https://github.com/morluto/rea/blob/main/docs/providers/ida-provider.md#migrate-a-legacy-attached-registration). Replace the upstream registration explicitly; modern mrexodia tools are not the native profile. Native attachment preserves the database but does not provide direct callers; check session availability and the guide's verification limits. @@ -407,6 +407,10 @@ describe possible providers, while top-level session capabilities describe the current route and `analysis_provider_binding` names the selected provider. Do not treat those scopes as interchangeable. +For editable Ghidra analysis that must survive a later agent process, select +the [owned Program checkpoint workflow](references/native-and-artifacts.md#continue-editable-ghidra-analysis-across-processes) +before making metadata edits. It is separate from Evidence/Unknown persistence. + For a Ghidra startup timeout on a large native target, follow [deadline recovery](references/native-and-artifacts.md#ghidra-startup-deadline-recovery) before abandoning deep analysis or repeating the same cold import. The default diff --git a/.agents/skills/reverse-engineer-anything/references/android-applications.md b/.agents/skills/reverse-engineer-anything/references/android-applications.md index cc51e22afe..f02b8329cb 100644 --- a/.agents/skills/reverse-engineer-anything/references/android-applications.md +++ b/.agents/skills/reverse-engineer-anything/references/android-applications.md @@ -22,7 +22,7 @@ Preserve slot/relocation provenance and source coverage: matched means observed candidate presence; partial scans do not establish uniqueness or absence. This reads DEX/ELF bytes without loading the APK into the decompiler or executing target code. Runtime registration and loading remain unknown. See the -[JNI guide](https://github.com/morluto/rea/blob/main/docs/android-analysis.md#link-native-declarations-to-jni-candidates). +[JNI guide](https://github.com/morluto/rea/blob/main/docs/guides/android/android-analysis.md#link-native-declarations-to-jni-candidates). For an R8/ProGuard release build with its `mapping.txt`, pass the absolute `mapping_path` to `inspect_android_class` or `inspect_android_method`. Prefer diff --git a/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md b/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md index d6491eab80..a460494298 100644 --- a/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md +++ b/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md @@ -11,7 +11,7 @@ main and an older release expose the same arguments. Tool-name prefixes belong to the client: find the canonical REA operation in its actual tool listing and server association before concluding that it is absent. A shared prefix alone does not establish which server an approval wildcard covers. See -[client tool names and discovery](https://github.com/morluto/rea/blob/main/docs/client-configuration.md#client-tool-names-and-discovery). +[client tool names and discovery](https://github.com/morluto/rea/blob/main/docs/getting-started/client-configuration.md#client-tool-names-and-discovery). ## Choose the remedy from the observed failure diff --git a/.agents/skills/reverse-engineer-anything/references/environments.md b/.agents/skills/reverse-engineer-anything/references/environments.md index 3157c81c99..327ceab171 100644 --- a/.agents/skills/reverse-engineer-anything/references/environments.md +++ b/.agents/skills/reverse-engineer-anything/references/environments.md @@ -327,7 +327,8 @@ Checks must express only the observed change: the draft suggests counts of `SECTION`, `ENDSEC` and `CIRCLE`; only `CIRCLE` equals 9 is the user's change. - Never check unchanged content, whole files, or values the app stamps: - times, ids it assigns, window sizes. REA refuses ISO 8601 date-times; the + times, ids it assigns, window sizes. REA refuses ISO 8601 and `asctime` + date-times (`Thu Nov 5 09:20:31 2026`); the rest is your judgment. - Prefer a few robust checks: whole lines over positions, rows by meaningful columns over generated ids, `at_least` when the user may add more. @@ -335,9 +336,21 @@ Checks must express only the observed change: Kinds: `file_present` and `file_absent`; `lines_present` and `lines_absent` (`lines`); `line_count` (`exact` or `contains`, and `equals`, `at_least` or `at_most`); `value_equals` and `value_absent` (a JSON `pointer` into the view, -such as `/groups/GROUP/KEY` for a key file); `row_present` and `row_absent` -(`table`, `where` by column) and `row_count`. Each names its `file` as a path -under the home. The +such as `/groups/GROUP/KEY` for a key file); `text_equals` and +`text_contains` (an XML element's whole text at a `pointer`, optional +`normalize_whitespace`); `row_present` and `row_absent` (`table`, `where` by +column) and `row_count`. Each names its `file` as a path under the home. A +document saved as a zip archive, such as FreeCAD's `.FCStd`, reads as its +entries: point into an entry's view, and name array members by selector +rather than position, as +`/entries/[name=Document.xml]/view/root/children/[name=Object][@name=Hole]`, +so the check survives an app version that writes them in another order. Give +`value_equals` a `tolerance` to compare a number the app writes as text, such +as `"5.0000000000000000"`, with a number. Check what the user wrote in a +document with `text_equals` on its paragraph element, not `value_equals` on a +text node: apps such as AbiWord and Writer split a paragraph into runs as they +save, so its first text node holds only part of it; use `text_contains` for a +value within a sentence, such as a time. The [desktop application tasks](https://github.com/morluto/rea/blob/main/docs/desktop-application-tasks.md) guide has the exact rules. diff --git a/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md b/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md index 0a83d37e79..cf97a8bde7 100644 --- a/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md +++ b/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md @@ -47,7 +47,7 @@ member comparison CLI still takes two assembly paths; process comparison takes two capture Evidence JSON paths. Check command help for the installed version. An input bundle grants no implicit file lookup: `evidence-file` and historical graph file selectors are accepted only by contracts advertising them. See the -[CLI reuse guide](https://github.com/morluto/rea/blob/main/docs/cli.md#import-export-and-compare-evidence). +[CLI reuse guide](https://github.com/morluto/rea/blob/main/docs/getting-started/cli.md#import-export-and-compare-evidence). Completed `inspect_artifact` inventory also supports a summary and `view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json", "main.js"]}`. diff --git a/.agents/skills/reverse-engineer-anything/references/javascript-applications.md b/.agents/skills/reverse-engineer-anything/references/javascript-applications.md index 34594c8bd3..e2ad9a5f12 100644 --- a/.agents/skills/reverse-engineer-anything/references/javascript-applications.md +++ b/.agents/skills/reverse-engineer-anything/references/javascript-applications.md @@ -6,7 +6,7 @@ For captured or bundled JavaScript that needs readable source recovery, use `additional_paths` when advertised, then analyze the returned `analysis_input` directory together. Each input retains its own snapshot, digest and byte-range provenance; inputs and recovered code are never executed. Recovery requires a -caller-supplied Wakaru on Linux x64; see [the recovery guide](https://github.com/morluto/rea/blob/main/docs/javascript-recovery.md). +caller-supplied Wakaru on Linux x64; see [the recovery guide](https://github.com/morluto/rea/blob/main/docs/guides/javascript/javascript-recovery.md). Use `analyze_javascript_application` directly on the operator-supplied ASAR or extracted tree. When the question concerns known source paths and the connected diff --git a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md index 761edcb0a1..b31e071f2b 100644 --- a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md +++ b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md @@ -256,7 +256,7 @@ Choose `row_limit` when a finite sample is useful and keep truncation and unspecified row order explicit. Virtual tables, shadow tables and views are schema facts, not row-read targets. REA inspects private copies and leaves the source files untouched; obtain a coherent quiescent database/WAL snapshot -before inspection. See [SQLite snapshots](https://github.com/morluto/rea/blob/main/docs/sqlite-databases.md). +before inspection. See [SQLite snapshots](https://github.com/morluto/rea/blob/main/docs/guides/data/sqlite-databases.md). ## Recorded Linux crashes @@ -271,7 +271,7 @@ The optional `include_debugger_context` facet adds mapping candidates through caller-supplied GDB/pwndbg in an owned core-only session. Display names do not establish current file identity; zero reported flags leave permissions unknown. Requested unavailable context fails with setup guidance. See -[recorded crashes](https://github.com/morluto/rea/blob/main/docs/recorded-crashes.md) +[recorded crashes](https://github.com/morluto/rea/blob/main/docs/guides/native/recorded-crashes.md) for exact upstream profiles, bounds and verification coverage. ## Offline ELF layout @@ -335,7 +335,8 @@ on the first MCP call. It retains the complete observation and returns the original selection, text availability and length without the body. Use `normalized_result.parent_evidence_id` with `inspect_analysis_view` and `view: {kind: "native", facet: "pseudocode", offset: 0, limit: 20000}`; -follow `coverage.next_offset` for later pages. Offsets and limits count UTF-16 +follow `coverage.next_offset` for later pages. Choose a smaller limit if one +page exceeds the client or transport budget. Offsets and limits count UTF-16 code units, not lines or tokens. Pages reuse the same observation without repeating decompilation. Null and empty text remain distinct. Omitted or `complete` detail preserves full inline delivery. Summary delivery requires a @@ -391,6 +392,28 @@ operation's full ordered list. Candidate capabilities, returned with complete detail, describe possible providers. Top-level session capabilities describe the current route, and `analysis_provider_binding` identifies the selected provider. +## Continue editable Ghidra analysis across processes + +For a task spanning agent sessions, check `binary_session` after selecting +Ghidra: the current provider's `capabilities[].shared_limitations` states whether +Program checkpoints are enabled or the session is ephemeral. Successful native +queries distinguish initial analysis from an actually restored packed Program; +do not infer that restoration happened merely because checkpoint mode is enabled. + +In ephemeral mode, choose a private absolute directory and set +`REA_GHIDRA_CHECKPOINT_DIR` in the environment starting REA. Restart the MCP server +after changing that launch configuration; no current tool call or runtime +analysis-setting update enables it. Stock ELF/PE/Mach-O imports on Linux/macOS +are supported; see the [checkpoint setup and limitations](https://github.com/morluto/rea/blob/main/docs/getting-started/installation.md#automatically-checkpoint-an-owned-ghidra-program). + +With the setting enabled, normal opens restore compatible saved Programs +automatically. Initial completed analysis is saved, and `close_binary` saves +later metadata edits; no separate save/restore tool is needed. Repair a reported +save failure and retry close. Cancellation or process failure can lose edits +after the last successful save; unfinished initial analysis cannot resume. +Keep the same checkpoint configuration for later processes. Evidence/Unknown +history and immutable REA query snapshots have separate persistence workflows. + ## Reuse caller-selected Ghidra analysis When the caller supplies an expanded project, configure `REA_GHIDRA_PROJECT_PATH` @@ -402,7 +425,7 @@ import overrides and repair scripts are unavailable in this mode. Treat original analysis completeness/settings as unknown and prior edits as caller-project state. The program's recorded import digest must match the target; changed project content requires reopening for a fresh profile. See the -[setup and preparation guide](https://github.com/morluto/rea/blob/main/docs/installation.md#reuse-an-existing-ghidra-project). +[setup and preparation guide](https://github.com/morluto/rea/blob/main/docs/getting-started/installation.md#reuse-an-existing-ghidra-project). ## Ghidra startup deadline recovery @@ -414,7 +437,7 @@ When the error names `ghidra.startup_timeout_ms`, call `update_analysis_settings` with `action: "set"` and a larger value within the reported bounds. Resolve retained cleanup before the next startup attempt. No REA restart is needed; an already accepted startup keeps its original -budget. A new attempt repeats import. Preserve useful Evidence before closing +budget. Follow the error's session-specific restart guidance, and preserve useful Evidence before closing and reopening the target when remediation requires that lifecycle step. For a per-function timeout, update `ghidra.decompile_timeout_seconds` and retry @@ -436,7 +459,7 @@ and memory controls in place. Disable specific expensive analyzers only when their missing coverage is acceptable and retain the resulting limitations. Exported C and embedded resources remain useful independent evidence, but a timeout does not establish complete analysis or absence of code. See the -[deadline and client guide](https://github.com/morluto/rea/blob/main/docs/mcp-contracts.md#ghidra-first-query-deadlines-and-recovery). +[deadline and client guide](https://github.com/morluto/rea/blob/main/docs/reference/mcp-contracts.md#ghidra-first-query-deadlines-and-recovery). ## Bridge request ceiling @@ -587,7 +610,7 @@ broader metadata recovery on Linux; it requires an explicitly configured NativeAOT JAR. These are separate routes. The built-in parser also runs on Windows Ghidra P0; read frozen literals from its report, because `search_strings` does not include them. See the -[NativeAOT guide](https://github.com/morluto/rea/blob/main/docs/ghidra-nativeaot.md) +[NativeAOT guide](https://github.com/morluto/rea/blob/main/docs/providers/ghidra-nativeaot.md) for setup, supported layouts, and host restrictions. Preserve binary identity and distinguish derived metadata from Ghidra database observations. Recovered types and native pseudocode are not original C# source. Windows Ghidra P0 has @@ -616,7 +639,7 @@ every executable in the bundle. It also lists missing or weak loads and earlier provides them. Use `extract_artifact` when materialized files from a supported format are needed. -It extracts the active target and accepts optional `integrity_policy`; see the [MCP contracts](https://github.com/morluto/rea/blob/main/docs/mcp-contracts.md). +It extracts the active target and accepts optional `integrity_policy`; see the [MCP contracts](https://github.com/morluto/rea/blob/main/docs/reference/mcp-contracts.md). It materializes all regular files into a fresh temporary directory chosen by REA. Symlinks and encrypted entries are inventory facts, not extractable files. The result reports the materialized directory; callers do not choose the @@ -644,7 +667,7 @@ temporary, not a retained export of the image's contents. For headerless DOS COM, explicitly open with `format: "dos-com"` (CLI `--target-format dos-com`). Inspect the load image before trusting function analysis: the entry is `0x10100`, file offset zero, with imposed real-mode segment -context. PSP/stack/device state remains unmodeled. See [DOS guide](https://github.com/morluto/rea/blob/main/docs/ghidra-dos.md). +context. PSP/stack/device state remains unmodeled. See [DOS guide](https://github.com/morluto/rea/blob/main/docs/providers/ghidra-dos.md). Use `inspect_byte_entropy` to triage high- and low-entropy ranges before extraction. @@ -658,7 +681,7 @@ otherwise. Big-endian ARM, Thumb-only, and Cortex-M variants are unsupported. `0x0000..0xffff`. 65C02, 65816, cartridge containers, bank switching, and mapper semantics are unsupported. `m68k` requires an even entry and decodes the 68040 superset of the 68000 family; Amiga/Atari containers and vector-table entry -discovery are unsupported. Windows Ghidra P0 admits raw images only for `6502`. See [raw binary guide](https://github.com/morluto/rea/blob/main/docs/raw-binary.md). +discovery are unsupported. Windows Ghidra P0 admits raw images only for `6502`. See [raw binary guide](https://github.com/morluto/rea/blob/main/docs/providers/raw-binary.md). Intel HEX, S-record, UF2 and DfuSe files are containers, not raw images: `inspect_artifact` lists address regions (UF2 per familyID, DfuSe per element @@ -670,7 +693,12 @@ numbering gaps leave coverage unknown; a DfuSe suffix CRC mismatch follows ### Function annotations in Ghidra Fresh-import sessions admit annotations; existing-project reuse is read-only. -Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier. Ghidra rejects batch addresses that request the same name or a name another selected address holds; `raw_result.failures` gives each failure's reason. +Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits. Normal close saves them when owned Program checkpoint mode is enabled; ephemeral mode discards them. CLI `annotate-native-function` follows the same setting when closing after its updated result. Select the [checkpoint workflow](#continue-editable-ghidra-analysis-across-processes) before edits that must survive a later process. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier. Ghidra rejects batch addresses that request the same name or a name another selected address holds; `raw_result.failures` gives each failure's reason. + +A provider switch or different target can retire the active Ghidra database and discard +its edits, even if the new provider fails on a later call. Same-target, +same-profile reopen keeps the active database. Retained annotation Evidence +records what happened; it does not restore the edits. For `compare_functions`, obtain `analyze_function` Evidence before and after the edit while the MCP session is open; annotation Evidence is not a comparison @@ -692,7 +720,7 @@ code pages, payload hashes and static icon-group references. Same-language icon resolution is explicit; missing references and size mismatch remain visible. Other payloads are opaque, and this operation does not decode images or simulate Windows resource loading. Consult the budgets and supported tree profile in -[offline binary diagnostics](https://github.com/morluto/rea/blob/main/docs/binary-diagnostics.md). +[offline binary diagnostics](https://github.com/morluto/rea/blob/main/docs/guides/native/binary-diagnostics.md). ## Offline PE and ELF export tables @@ -702,7 +730,7 @@ directory or ELF dynamic symbol table on any host without opening a native target. Ordinal-only, forwarded and hidden-version exports stay distinct; partial or unsupported coverage means the inventory is incomplete, not that exports are absent. Use `inspect_macho` for Mach-O. See -[offline binary diagnostics](https://github.com/morluto/rea/blob/main/docs/binary-diagnostics.md). +[offline binary diagnostics](https://github.com/morluto/rea/blob/main/docs/guides/native/binary-diagnostics.md). For exact core-module import/export differences, use `compare_wasm_interfaces` with two complete inspection Evidence records (or current-session retained diff --git a/.agents/skills/reverse-engineer-anything/references/runtime-observation.md b/.agents/skills/reverse-engineer-anything/references/runtime-observation.md index 8f5303cc9d..d87fb3cae5 100644 --- a/.agents/skills/reverse-engineer-anything/references/runtime-observation.md +++ b/.agents/skills/reverse-engineer-anything/references/runtime-observation.md @@ -41,10 +41,10 @@ behavior unknown. Record truncation, unmatched breakpoints, changed target identity and cleanup uncertainty as limitations, rather than treating no retained event as proof that the behavior never occurs. -Detailed contracts and prerequisites: [Frida](https://github.com/morluto/rea/blob/main/docs/frida-instrumentation.md), +Detailed contracts and prerequisites: [Frida](https://github.com/morluto/rea/blob/main/docs/guides/runtime/frida-instrumentation.md), [GDB/Rizin/Cutter](https://github.com/morluto/rea/blob/main/docs/reverse-engineering-tools.md), -[process capture](https://github.com/morluto/rea/blob/main/docs/process-capture.md), -and [native calls/UI](https://github.com/morluto/rea/blob/main/docs/native-investigation.md#native-desktop-observation). +[process capture](https://github.com/morluto/rea/blob/main/docs/guides/runtime/process-capture.md), +and [native calls/UI](https://github.com/morluto/rea/blob/main/docs/guides/native/native-investigation.md#native-desktop-observation). ## Browser and Electron @@ -130,5 +130,5 @@ Inspect native `.saz` snapshots with `inspect_web_network_capture` and traffic analysis, connect Everywhere's separate MCP server and use its advertised session tools with an explicit `sessionsSource` (`LiveTraffic` or `AgentCalls`). Preserve existing sessions and filters when inspecting a -recorded run. See the [capture and MCP guide](https://github.com/morluto/rea/blob/main/docs/web-network-captures.md) +recorded run. See the [capture and MCP guide](https://github.com/morluto/rea/blob/main/docs/guides/web/web-network-captures.md) for the workflow and Telerik's public setup and traffic-debugging skills. diff --git a/.github/ISSUE_TEMPLATE/documentation.yml b/.github/ISSUE_TEMPLATE/documentation.yml index a522606a6e..96489ad650 100644 --- a/.github/ISSUE_TEMPLATE/documentation.yml +++ b/.github/ISSUE_TEMPLATE/documentation.yml @@ -27,7 +27,7 @@ body: attributes: label: Documentation location description: Link the page, section, command, API, skill, or inline comment. - placeholder: README.md#quick-start or docs/mcp-contracts.md + placeholder: README.md#quick-start or docs/reference/mcp-contracts.md validations: required: true diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 5420b790ad..e222ed4c38 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -71,8 +71,8 @@ source, link, or rendered-document checks. --> - User-visible CLI/MCP output (if applicable): - Remaining proof gaps: -For boundary fixes, identify the applicable [boundary regressions](https://github.com/morluto/rea/blob/main/docs/testing.md#boundary-regressions) -and [lifecycle regressions](https://github.com/morluto/rea/blob/main/docs/testing.md#lifecycle-regressions) +For boundary fixes, identify the applicable [boundary regressions](https://github.com/morluto/rea/blob/main/docs/development/testing.md#boundary-regressions) +and [lifecycle regressions](https://github.com/morluto/rea/blob/main/docs/development/testing.md#lifecycle-regressions) covered by the evidence above, and explain relevant gaps. For evidence-bearing changes: @@ -99,7 +99,7 @@ limits, security/privacy/containment implications, and meaningful proof gaps. -- its new behavior and migration, and why compatibility/deprecation cannot preserve it. Every release increments minor, including breaking changes; keep breaking markers and migration notes so callers can assess compatibility. - See docs/releasing.md. --> + See docs/development/releasing.md. --> - Real Hopper/Ghidra, browser, or OS coverage: - Package or release metadata impact: - Security, privacy, process, or containment review: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6880245b92..d32237206e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1016,8 +1016,8 @@ jobs: uses: ./.github/actions/npm-cache - run: npm ci - run: npm run docs:check - - name: Render documentation and validate page links - run: npm run docs:build + - name: Verify published routes, source links and local anchors + run: npm run verify:docs:routes - name: Generation leaves authored files unchanged run: git diff --exit-code - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index baa6ba785e..680708bcfe 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -27,4 +27,4 @@ jobs: id: npm-cache uses: ./.github/actions/npm-cache - run: npm ci - - run: npm run docs:build + - run: npm run verify:docs:routes diff --git a/AGENTS.md b/AGENTS.md index 924159e258..2af483be32 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,13 +14,13 @@ Preserve caller-selected inputs and local evidence, including output, URLs, path ## Architecture and Cleanup -Dependencies flow inward: domain semantics and contracts, provider adapters, shared application workflows, then CLI/MCP adapters. Keep engine protocols and provider-specific code out of domain/application layers. See [docs/architecture.mermaid](docs/architecture.mermaid) when changing composition. +Dependencies flow inward: domain semantics and contracts, provider adapters, shared application workflows, then CLI/MCP adapters. Keep engine protocols and provider-specific code out of domain/application layers. See [docs/architecture/architecture.mermaid](docs/architecture/architecture.mermaid) when changing composition. Give each interpretation and resource a clear owner. Consumers must use that owner's parsed facts rather than reinterpret raw source. Preserve identity, omission, uncertainty, and source evidence across boundaries. Retain ownership of resources whose cleanup failed. Reuse `src/process/` supervision and identity primitives; a PID and executable path alone do not establish ownership after exit or reuse. Keep proof of ownership separate from validity, editability, and readiness. Malformed owned data may still be safely removable. Unresolved identity or applicability must remain unknown rather than be treated as absent, foreign, or aligned. -For lifecycle changes, follow [ownership and completion invariants](docs/tool-design.md#own-the-lifecycle) and select the applicable [lifecycle regressions](docs/testing.md#lifecycle-regressions). Include late handshakes, cancellation during final cleanup, mixed child outcomes, and cleanup retry where those stages exist; active-execution cancellation alone does not cover finalization. +For lifecycle changes, follow [ownership and completion invariants](docs/development/tool-design.md#own-the-lifecycle) and select the applicable [lifecycle regressions](docs/development/testing.md#lifecycle-regressions). Include late handshakes, cancellation during final cleanup, mixed child outcomes, and cleanup retry where those stages exist; active-execution cancellation alone does not cover finalization. Before adding complexity for a speculative or vanishingly unlikely race or security edge case, explain the concrete failure mode, likelihood, and complexity cost to the user and get their buy-in. Do not silently expand scope for theoretical robustness. @@ -68,6 +68,12 @@ do not establish fewer tokens or faster analysis. ## Development +Documentation ownership and authoring conventions live in +[docs/development/documentation.md](docs/development/documentation.md). A feature +uses `docs/specs///spec.md`, one `plan.md` and `tasks.md`; +individual architectural decisions use `docs/adr//.md`. +Consult the [documentation directory](docs/README.md) before adding a guide. + Use [.nvmrc](.nvmrc) and `package.json#packageManager` for the pinned toolchain. Cursor Cloud may place older Node.js first on `PATH`; environment setup installs the pinned toolchain under `/usr/local/bin`. Prefix `PATH="/usr/local/bin:$PATH"` before `node` and `npm`. - `npm ci`: install locked dependencies. @@ -80,15 +86,15 @@ Use [.nvmrc](.nvmrc) and `package.json#packageManager` for the pinned toolchain. Use relevant checks for the change. The complete local gate, `npm run check:pr`, is optional for broad changes or CI diagnosis. See [CONTRIBUTING.md](CONTRIBUTING.md) for development, generated-file ownership, and release conventions. When changing installers or setup recovery, follow -[Checking installers and publisher downloads](docs/testing.md#checking-installers-and-publisher-downloads). +[Checking installers and publisher downloads](docs/development/testing.md#checking-installers-and-publisher-downloads). -Coordinate expensive checks across agents. If a test unexpectedly consumes heavy CPU, stop its owned process family and profile a bounded representative case before retrying. Inspect production work, fixture generation, subprocesses, and repeated setup; preserve regression coverage when optimizing. Follow the [test resource guidance](docs/testing.md#developer-commands). +Coordinate expensive checks across agents. If a test unexpectedly consumes heavy CPU, stop its owned process family and profile a bounded representative case before retrying. Inspect production work, fixture generation, subprocesses, and repeated setup; preserve regression coverage when optimizing. Follow the [test resource guidance](docs/development/testing.md#developer-commands). `docs/public/product-catalog.json`, `docs/verification/managed-conformance-*.json`, and `skills/` are ignored outputs. Edit source contracts and `.agents/skills/reverse-engineer-anything/`, then regenerate as needed. `.cache/mcp-tool-catalog.json` is test metadata; runtime builds and source checking do not consume it. Never commit binaries, provider project documents, credentials, `dist/`, `node_modules/`, or local planning artifacts. ## Tool and Test Changes -For tool design, follow [docs/tool-design.md](docs/tool-design.md): start from the analyst question, reuse existing contracts, preserve caller choices, and return useful evidence inline. Limits must follow real format, protocol, authority, or resource constraints. Account for representation expansion before allocation; derive completeness from examined coverage. +For tool design, follow [docs/development/tool-design.md](docs/development/tool-design.md): start from the analyst question, reuse existing contracts, preserve caller choices, and return useful evidence inline. Limits must follow real format, protocol, authority, or resource constraints. Account for representation expansion before allocation; derive completeness from examined coverage. For user- or agent-facing tool problems, prioritize fixing the owning behavior, contract, and actionable result or recovery at the point of use. Document the @@ -96,7 +102,7 @@ result in product docs when explanation is needed. Add skill guidance only for workflow judgment or context that the tool cannot reasonably convey; do not use a skill as the primary repair for a discoverability or contract defect. -For test changes or provider claims, use [docs/testing.md](docs/testing.md). Select applicable [boundary regressions](docs/testing.md#boundary-regressions) for parser, inventory, snapshot, and projection fixes. Prefer real public CLI/MCP workflows, then production-boundary integration, then producer goldens. Keep module tests for distinct semantics stronger workflows cannot reliably reproduce. Preserve malformed-input, cancellation, cleanup, permission, capacity, and target-identity coverage when pruning. Report unverified coverage explicitly; simulated providers do not establish real-provider behavior. +For test changes or provider claims, use [docs/development/testing.md](docs/development/testing.md). Select applicable [boundary regressions](docs/development/testing.md#boundary-regressions) for parser, inventory, snapshot, and projection fixes. Prefer real public CLI/MCP workflows, then production-boundary integration, then producer goldens. Keep module tests for distinct semantics stronger workflows cannot reliably reproduce. Preserve malformed-input, cancellation, cleanup, permission, capacity, and target-identity coverage when pruning. Report unverified coverage explicitly; simulated providers do not establish real-provider behavior. Use Conventional Commit subjects and PR titles. Mark breaking changes with `!` or a `BREAKING CHANGE:` footer. Describe behavior changes, relevant validation, and real-provider verification scope. diff --git a/CHANGELOG.md b/CHANGELOG.md index b278878a3e..e652f26036 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -132,8 +132,8 @@ Full contribution record: [4 commits and 1 merged PR](https://github.com/morluto - **Kuna analysis:** Use a caller-installed Kuna for procedure listings, pseudocode, assembly, call relationships, exact-address xrefs and cached string searches. Project exports retain source spans and digests; repeated requests reuse completed work. Recorded real-provider coverage uses Kuna 1.833 on Linux; Windows, PE and universal Mach-O are unsupported. See [the provider guide](docs/kuna-provider.md). ([#3832](https://github.com/morluto/rea/pull/3832), [#3854](https://github.com/morluto/rea/pull/3854), [#4071](https://github.com/morluto/rea/pull/4071), [#4095](https://github.com/morluto/rea/pull/4095)). Thanks @gensokyo-dev, @kaoru0822-kitauji, @N0zoM1z0. - **Retained JavaScript facts:** Query exact application paths, selected properties and filtered module references through `inspect_analysis_view`, with paging and complete row Evidence. These views reuse completed analysis and preserve source selection and unknown coverage. ([#4265](https://github.com/morluto/rea/pull/4265), [morluto/rea#4190](https://github.com/morluto/rea/issues/4190), [#4097](https://github.com/morluto/rea/pull/4097), [morluto/rea#4092](https://github.com/morluto/rea/issues/4092)). Thanks @kaoru0822-kitauji, @N0zoM1z0. - **APK locale tables:** Request several resource locales in one decode with repeatable CLI `--locales` or MCP `locales`. Each table reports its status and coverage; existing single-locale fields remain available. ([#4804](https://github.com/morluto/rea/pull/4804), [morluto/rea#4777](https://github.com/morluto/rea/issues/4777)). Thanks @abhay-2108, @N0zoM1z0. -- **Windows ARM binaries:** Ghidra admits ARMNT/Thumb-2 and ARM64 PE applications and DLLs on the Windows x64 host lane. Recorded Ghidra 12.1.4 verification uses `WindowsResourceReference` disabled and raised startup budgets; default-analyzer completion, ARM64EC/ARM64X and Windows-on-Arm hosts remain unverified or unsupported. See [the exact scope](docs/windows-ghidra-p0.md#native-authority-and-target-identity). ([#3965](https://github.com/morluto/rea/pull/3965), [morluto/rea#2186](https://github.com/morluto/rea/issues/2186)). Thanks @N0zoM1z0, @WestLabsAI. -- **Raw m68k images:** Analyze explicitly selected big-endian m68k raw binaries using Ghidra’s fixed 68000 profile and a caller-supplied aligned entry. Real CLI/MCP verification used Ghidra 12.1.4 on macOS arm64; Linux is unverified and Windows P0 does not admit m68k. See [mapping requirements](docs/raw-binary.md). ([#4018](https://github.com/morluto/rea/pull/4018)). Thanks @icedac. +- **Windows ARM binaries:** Ghidra admits ARMNT/Thumb-2 and ARM64 PE applications and DLLs on the Windows x64 host lane. Recorded Ghidra 12.1.4 verification uses `WindowsResourceReference` disabled and raised startup budgets; default-analyzer completion, ARM64EC/ARM64X and Windows-on-Arm hosts remain unverified or unsupported. See [the exact scope](docs/providers/windows-ghidra-p0.md#native-authority-and-target-identity). ([#3965](https://github.com/morluto/rea/pull/3965), [morluto/rea#2186](https://github.com/morluto/rea/issues/2186)). Thanks @N0zoM1z0, @WestLabsAI. +- **Raw m68k images:** Analyze explicitly selected big-endian m68k raw binaries using Ghidra’s fixed 68000 profile and a caller-supplied aligned entry. Real CLI/MCP verification used Ghidra 12.1.4 on macOS arm64; Linux is unverified and Windows P0 does not admit m68k. See [mapping requirements](docs/providers/raw-binary.md). ([#4018](https://github.com/morluto/rea/pull/4018)). Thanks @icedac. - **JADX deadlines:** Set `REA_JADX_OPERATION_TIMEOUT_MS` to extend APK loading and upstream operation deadlines together. Doctor reports the effective value and setup forwards an explicit selection. ([#4466](https://github.com/morluto/rea/pull/4466), [morluto/rea#3918](https://github.com/morluto/rea/issues/3918)). Thanks @jaychou813, @SandstromPL. - **Electron window options:** Resolve stable local constant `BrowserWindow` options while retaining source provenance. Relevant writes and escapes keep the result unknown. ([#4400](https://github.com/morluto/rea/pull/4400), [morluto/rea#4366](https://github.com/morluto/rea/issues/4366)). Thanks @kaoru0822-kitauji, @N0zoM1z0. - **Language editions:** Add the Greek website edition and Malayalam documentation, with Korean typography and translation corrections across the site. ([#3520](https://github.com/morluto/rea/pull/3520), [#4792](https://github.com/morluto/rea/pull/4792), [#3821](https://github.com/morluto/rea/pull/3821), [morluto/rea#3820](https://github.com/morluto/rea/issues/3820), [#3799](https://github.com/morluto/rea/pull/3799), [morluto/rea#3798](https://github.com/morluto/rea/issues/3798), [#4344](https://github.com/morluto/rea/pull/4344), [#3856](https://github.com/morluto/rea/pull/3856), [#4166](https://github.com/morluto/rea/pull/4166)). Thanks @gensokyo-dev, @hi2242, @kalayciburak, @kaoru0822-kitauji, @kraken-503, @kukinghan, @N0zoM1z0, @yeseong0412. @@ -192,7 +192,7 @@ Full contribution record: [571 commits and 236 merged PRs](https://github.com/mo ## [6.5.0](https://github.com/morluto/rea/compare/rea-agents-6.4.0...rea-agents-6.5.0) (2026-10-13) -Provider prerequisites and recorded verification scope are listed in [the testing guide](docs/testing.md). +Provider prerequisites and recorded verification scope are listed in [the testing guide](docs/development/testing.md). ### Highlights @@ -220,7 +220,7 @@ Provider prerequisites and recorded verification scope are listed in [the testin - **HarmonyOS packages:** inventory HAP/HSP/App Pack structures without executing code. The recorded real workflow covers one pinned HAP through repository-built CLI/MCP; installed-package and real HSP/App Pack coverage remain unverified. ([#1447](https://github.com/morluto/rea/pull/1447)). Thanks @BaaRai. - **Ghidra raw inputs and references:** compatible Linux/macOS Ghidra profiles accept caller-selected raw architecture/entry information and expose typed incoming references at exact addresses. Windows Ghidra P0 admits the raw NMOS 6502 profile described below; other raw targets and the incoming-reference facet remain outside Windows P0. ([#1868](https://github.com/morluto/rea/pull/1868), [issue morluto/rea#1836](https://github.com/morluto/rea/issues/1836), [#1715](https://github.com/morluto/rea/pull/1715), [issue morluto/rea#1245](https://github.com/morluto/rea/issues/1245)). Thanks @bugstan, @casesolved-co-uk, @mvanhorn, @N0zoM1z0. - **Ghidra MIPS:** Add bounded MIPS32r2/o32 relocatable objects, soft-float integer-register storage, admitted MIPS16 shared ELF, and a narrowly admitted recordless legacy MIPS32 executable profile. The latter pins an endian-matched 32-bit language and records inferred o32 as an analysis assumption. Admission follows each documented ABI/flag/relocation boundary. ([#1949](https://github.com/morluto/rea/pull/1949), [issue morluto/rea#1937](https://github.com/morluto/rea/issues/1937), [#1950](https://github.com/morluto/rea/pull/1950), [issue morluto/rea#1938](https://github.com/morluto/rea/issues/1938), [#1991](https://github.com/morluto/rea/pull/1991), [issue morluto/rea#1851](https://github.com/morluto/rea/issues/1851), [#1898](https://github.com/morluto/rea/pull/1898), [issue morluto/rea#1620](https://github.com/morluto/rea/issues/1620)). Thanks @AurelioB, @BaaRai, @bqw371, @kaoru0822-kitauji, @N0zoM1z0. -- **Ghidra RISC-V:** admit little-endian RV32/RV64 `ET_EXEC` ELF declaring the supported double-float ABI. Relocatable/shared, big-endian and other ABI profiles remain refused; see [the profile and recorded lane](docs/ghidra-riscv.md). ([#2151](https://github.com/morluto/rea/pull/2151)). Thanks @NIU-123370. +- **Ghidra RISC-V:** admit little-endian RV32/RV64 `ET_EXEC` ELF declaring the supported double-float ABI. Relocatable/shared, big-endian and other ABI profiles remain refused; see [the profile and recorded lane](docs/providers/ghidra-riscv.md). ([#2151](https://github.com/morluto/rea/pull/2151)). Thanks @NIU-123370. - **PSP static ELF profile:** the Ghidra profile is Linux x64, bounded little-endian ELF32 `ET_EXEC` with the documented PSP/ABI declarations and a compatible caller-installed Allegrex extension. PRX, shared/relocatable images, PBP/ISO/CSO and runtime emulation are outside scope. Historical extension experiments are separate from public-provider verification; real public Ghidra PSP coverage remains unverified. ([#1599](https://github.com/morluto/rea/pull/1599)). Thanks @jaelliot. - **Ghidra analysis settings:** apply validated analyzer overrides before analysis while retaining the selected analysis profile. ([#1913](https://github.com/morluto/rea/pull/1913)). Thanks @akushonkamen, @kaoru0822-kitauji, @N0zoM1z0. - **Attached native IDA MCP:** add an optional native bridge alongside existing legacy/headless profiles. Recorded native runs use Linux IDA Free 9.4 with a navigation correction and Windows IDA Pro 9.3 with a loopback-listener change; stock native switching, native macOS and Windows IDA Free remain unverified. Attached databases remain read-only and are never saved or closed. ([#1544](https://github.com/morluto/rea/pull/1544)). Thanks @imissapixel, @N0zoM1z0. @@ -292,7 +292,7 @@ Provider prerequisites and recorded verification scope are listed in [the testin - **Source maps:** original-source and mapping rows preserve a null source name when unavailable. Handle null explicitly instead of expecting an invented unknown-source string; new index fields remain optional when reading older Evidence. ([ee50ed1](https://github.com/morluto/rea/commit/ee50ed121e38cc6e1a2ff0dc8b789f15f71c5402)). - **Streamed filtered JSON:** `--filter-output` now emits JSON `null` instead of `{}` when the selected value is null; consumers must distinguish a present null from a missing selection. Use `--full-output` when an object envelope is required. ([#2333](https://github.com/morluto/rea/pull/2333)). Thanks @Jackmeson1. - **Call graphs:** `get_call_graph.normalized_result` is now `{ graph, limitations }`; read traversal-depth groups under `graph` and retain relationship-specific caveats. Contract/fixture validation does not establish new Ghidra or IDA engine coverage. ([5dfb9d0](https://github.com/morluto/rea/commit/5dfb9d050c2cc44a61b183dae765bef4d22a53b7)). -- **MCP discovery:** `binary_session` no longer repeats `server_identity.catalog.tools`. Use the connected server tool list, grouped `tool_availability` and provider capabilities; see [MCP contracts](docs/mcp-contracts.md). Origin: [6395e8d](https://github.com/morluto/rea/commit/6395e8d3007d5ee453477460e7ee1c51cff76be0). ([#1988](https://github.com/morluto/rea/pull/1988)). Thanks @kaoru0822-kitauji, @N0zoM1z0. +- **MCP discovery:** `binary_session` no longer repeats `server_identity.catalog.tools`. Use the connected server tool list, grouped `tool_availability` and provider capabilities; see [MCP contracts](docs/reference/mcp-contracts.md). Origin: [6395e8d](https://github.com/morluto/rea/commit/6395e8d3007d5ee453477460e7ee1c51cff76be0). ([#1988](https://github.com/morluto/rea/pull/1988)). Thanks @kaoru0822-kitauji, @N0zoM1z0. - **Provider admission:** MIPS targets with unresolved GNU ABI attributes are refused. Supply supported ABI declarations. Saved Hopper databases on macOS are unavailable because the launcher cannot run the analysis bridge when opening a database; select an executable target instead. Database support requires a script-capable database launch route. ([d4bd65c](https://github.com/morluto/rea/commit/d4bd65c53c0254d0195cc7ab8da163dcadfd3b9b)). - **Segment extents:** `binary_overview` segment length may be null when Hopper coordinates cannot establish a finite extent in one address space. Preserve unknown rather than converting it to zero or a saturated value. ([36db19c](https://github.com/morluto/rea/commit/36db19c59)). @@ -372,7 +372,7 @@ Full contribution record: [336 commits and 163 merged PRs](https://github.com/mo ### ⚠ BREAKING CHANGES -* **captures:** Historical captures without current producer accounting or transaction identity, and web diffs missing accessibility or storage dimensions, are rejected. Preserve original captures and recapture with the current producer; update comparison consumers to include every dimension rather than synthesizing missing observations. See the [process capture guide](docs/process-capture.md) and [browser scenario contract](docs/browser-scenario-contract.md). ([3283a45](https://github.com/morluto/rea/commit/3283a45d09457c79e3adadb50a48620926d4bc5b)) +* **captures:** Historical captures without current producer accounting or transaction identity, and web diffs missing accessibility or storage dimensions, are rejected. Preserve original captures and recapture with the current producer; update comparison consumers to include every dimension rather than synthesizing missing observations. See the [process capture guide](docs/guides/runtime/process-capture.md) and [browser scenario contract](docs/guides/web/browser-scenario-contract.md). ([3283a45](https://github.com/morluto/rea/commit/3283a45d09457c79e3adadb50a48620926d4bc5b)) * **browser:** JSON shape paths now contain typed property and array-element segments. WebMCP input_schema_shape is replaced by input_schema and input_schema_sha256, retaining declared examples and defaults. * **process:** Filesystem digest omission entries now require system_code; budget and changed-file omissions use null, and unavailable-file omissions preserve the operating-system error code. * **contracts:** Provider analysis outputs must explicitly supply metadata previously repaired with defaults; unsupported nullable facets must be reported as null. @@ -540,7 +540,7 @@ Full contribution record: [336 commits and 163 merged PRs](https://github.com/mo ### ⚠ BREAKING CHANGES -Read the [6.1 to 6.2 migration guide](docs/migration-6.2.md) before upgrading. REA's `always-bump-minor` policy includes breaking changes; 6.2 is not a backward-compatibility promise. Preserve original saved files and regenerate missing observations rather than fabricating identities or facts. +Read the [6.1 to 6.2 migration guide](docs/migrations/migration-6.2.md) before upgrading. REA's `always-bump-minor` policy includes breaking changes; 6.2 is not a backward-compatibility promise. Preserve original saved files and regenerate missing observations rather than fabricating identities or facts. * **MCP / Evidence / snapshots / artifacts:** Success results are direct canonical Evidence; read `normalized_result` rather than the old `result`/`evidence` wrapper. Evidence requires nullable `analysis_profile`, session provider aliases are removed, snapshots require complete profile-bound query records, and artifact roles/permissions belong to occurrences. Recreate incompatible saved data; adding fields without recomputing their semantic identity is invalid. ([63bf914](https://github.com/morluto/rea/commit/63bf91405cf97b114a84f337fb1272d98a267999)) * **Process captures:** Legacy v3 conversion is removed. Explicit executable identity and `event_journal` are required; retain old captures and recapture when current comparisons are needed instead of inventing launch identity or event order. ([2a77469](https://github.com/morluto/rea/commit/2a7746998d47446c4db6d4f524e0f183d6803f3f)) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 744cb9361f..ad02d86048 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,7 +11,7 @@ incorrect user instructions warrant standalone corrections. Collect other cosmetic-only suggestions in an issue rather than separate PRs. For selected architectural owners and dependency paths, use the -[architecture map](docs/architecture.mermaid). It is not a complete directory or +[architecture map](docs/architecture/architecture.mermaid). It is not a complete directory or import inventory. For current provider wiring, start with [`src/composition/binary.ts`](src/composition/binary.ts) for deep candidates and the other `src/composition/` factories for tool services, including @@ -32,7 +32,7 @@ removing it. Preserve validation, lazy loading, caller policy, and resource acquisition and cleanup supplied by the old boundary. Published APIs, persisted formats, and independently versioned bridges require an explicit compatibility decision; an internal representation alone does not establish that obligation. -See [interpreting boundary facts once](docs/tool-design.md#interpret-boundary-facts-once) +See [interpreting boundary facts once](docs/development/tool-design.md#interpret-boundary-facts-once) for admission ownership and the distinction between repeated validation and independent checks. @@ -44,7 +44,7 @@ Use the [analysis optimization skill](.agents/skills/rea-analysis-optimization/S to locate provider and shared-workflow bottlenecks, preserve Evidence, and compare equivalent real tasks before claiming an improvement. -When adding or changing an MCP tool, follow the [tool design guide](docs/tool-design.md) and preserve the canonical contracts and generated catalog. +When adding or changing an MCP tool, follow the [tool design guide](docs/development/tool-design.md) and preserve the canonical contracts and generated catalog. When adding or changing an admission ceiling, budget, or deadline, follow [analysis settings, limits, and recovery](docs/design/limits-and-recovery.md) for ownership, operation snapshots, provider integration, and the steps for @@ -234,7 +234,7 @@ guarantee. Use the development toolchain pinned in `.nvmrc` and `package.json#packageManager`. Run `nvm use` before installing dependencies; installed-package runtime support is documented in -[installation](docs/installation.md#start-setup). +[installation](docs/getting-started/installation.md#start-setup). Use Node.js and npm for source builds and verifier scripts; the installed package's Bun runtime support does not extend to those development scripts. @@ -289,16 +289,20 @@ base revision, or run the suite on Linux or WSL, before attributing them to a change. The embedded-artifact parent-swap tests exercise the selected-path replacement when the rename succeeds; if Windows refuses it while the admitted file is open, they instead verify the refusal and cleanup. See -[testing](docs/testing.md#developer-commands) for command scopes +[testing](docs/development/testing.md#developer-commands) for command scopes and lane selection. `git submodule update --init --recursive` fails at the pinned Unblob submodule on Windows and can leave later submodules unchecked out. Apply the -[Unblob checkout workaround](docs/firmware-analysis.md#windows-submodule-checkout), +[Unblob checkout workaround](docs/guides/firmware/firmware-analysis.md#windows-submodule-checkout), then rerun the command. ## Documentation website +Use the [documentation directory](docs/README.md) and +[authoring guide](docs/development/documentation.md) for document ownership, +domain-based feature specifications and architectural decision records. + The VitePress site uses the Markdown files in `docs/`. Run `npm run docs:dev` for live editing, `npm run docs:build` to check the production build and links, and `npm run docs:preview` to preview that build at `/rea/`. @@ -352,7 +356,7 @@ npm run check:fast `test:local` runs source tests without building. `check:changed` combines cached static checks with source tests affected since the merge base with `origin/main`. Import-graph selection can miss runtime registration, bridges, and generated data; -select relevant boundary tests explicitly. See [testing](docs/testing.md#developer-commands) +select relevant boundary tests explicitly. See [testing](docs/development/testing.md#developer-commands) for commands, test placement, pruning, and real-provider prerequisites. Before requesting review, run relevant tests and `npm run check`; add @@ -427,8 +431,8 @@ Run `npm run verify:hopper` on a supported macOS host or `npm run verify:hopper:linux` on the supported Linux demo runner. These build native fixtures and verify the installed engine through CLI/MCP; package or simulated-provider checks do not establish engine behavior. See -[real-toolchain verification](docs/testing.md#end-to-end-integration-and-golden-evidence) -for prerequisites and scope, and [Hopper setup](docs/installation.md#hopper) +[real-toolchain verification](docs/development/testing.md#end-to-end-integration-and-golden-evidence) +for prerequisites and scope, and [Hopper setup](docs/getting-started/installation.md#hopper) for supported hosts and unattended startup. Use the default `build/conformance/manifest.json` unless explicitly verifying @@ -437,7 +441,7 @@ Generated fixtures and manifests remain ignored. ## Maintainer release checklist -Use the [release guide](docs/releasing.md). Main pushes automatically refresh a +Use the [release guide](docs/development/releasing.md). Main pushes automatically refresh a Release Please PR with the next version and changelog. Review its migration notes, wait for the current CI checks to pass, and merge it into main to publish. diff --git a/README.md b/README.md index 9bd7d59c5a..d29b5de108 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ **See a feature you like. Understand how it works, down to the binary level.** [![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents) -[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog) +[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/reference/mcp-contracts.md#generated-catalog) [![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#what-you-can-analyze) [![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything) [![MIT license](https://img.shields.io/badge/license-MIT-f4c430?style=flat-square)](LICENSE) @@ -72,7 +72,7 @@ Setup registers REA with your agent and installs matching workflow instructions. On Linux, Ghidra, JADX, firmware tool launches and static JavaScript/Electron analysis require a procps-compatible `ps` on REA's `PATH` for process ownership -checks. Recovery steps are in `docs/installation.md` under +checks. Recovery steps are in `docs/getting-started/installation.md` under "Linux owned-process prerequisite". ### Set up your agent @@ -91,8 +91,8 @@ available. Run package acquisition from a trusted directory outside the target project. Installation and recovery details are in the setup guide below. Setup supports Claude Code, Codex, Cursor, Gemini CLI, Grok Build and -[other agents](docs/installation.md#supported-agents). See -[installation and setup](docs/installation.md) for provider configuration and +[other agents](docs/getting-started/installation.md#supported-agents). See +[installation and setup](docs/getting-started/installation.md) for provider configuration and manual MCP registration and Claude Code/Codex plugin installation. MCP uses compact input schemas by default to reduce context overhead. Set @@ -135,7 +135,7 @@ rea --help ``` For native analysis, configure a provider first. See the -[CLI and Evidence guide](docs/cli.md) for native commands, provider selection, +[CLI and Evidence guide](docs/getting-started/cli.md) for native commands, provider selection, snapshots and scripting. ### Update REA @@ -191,45 +191,45 @@ CLI commands use the same workflows. REA requires Node.js 22.x (>=22.23.4), 24.x (>=24.22.1), or 26.12.1+, plus npm. Additional tools and host support depend on the target: -| Target | What REA returns | Requirements and guide | -| ----------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -| Desktop installations | Runtime candidates, supporting files, component markers and next probes | Built-in bounded directory/header inspection; [desktop applications](docs/desktop-applications.md) | -| Native binaries | Pseudocode, assembly, strings, symbols, calls and references | Hopper, Ghidra, IDA or Kuna; [native analysis](docs/native-investigation.md) | -| Offline ELF layout | Sections, segments, original symbols/relocations and static mitigation candidates | Caller-supplied pwntools on Linux x64; [binary diagnostics](docs/binary-diagnostics.md) | -| Go binaries | Embedded compiler version, modules, replacements and build settings | Built-in static ELF/PE/thin Mach-O inspection; no Go installation required | -| GPU code objects | CUDA/HIP targets, selected kernel parameters and resources, retained PTX text | Built-in static ELF/PE/fatbin/CUBIN/PTX inspection; [GPU code objects](docs/gpu-code-objects.md) | -| Model artifacts | GGUF / safetensors metadata, tensor declarations and derived totals | Built-in header inspection; optional full-file hash; [model files](docs/model-artifacts.md) | -| IL2CPP metadata | Unity global-metadata.dat assembly images, type and method names and index ranges | Built-in static decoding of versions 24, 27, 29, 31 and 39; no Unity required | -| Godot PCK | Archive inventory and extraction, original header/directory facts and selected content identities | Built-in static formats 2, 3 and 4; no Godot required | -| Rust binaries | Embedded cargo-auditable crate names, versions, sources and dependency edges | Built-in static ELF/PE/thin Mach-O inspection; [Rust binaries](docs/rust-binaries.md) | -| PE certificates | Authenticode WIN_CERTIFICATE entries and PKCS#7 signer/X.509 identity fields | Built-in static PE inspection; [PE certificates](docs/pe-certificates.md) | -| PE imports | Import and delay-load DLL names, ordinals, IAT RVAs and pefile import hash | Built-in static PE inspection; [PE imports](docs/pe-imports.md) | -| ELF notes | PT_NOTE/SHT_NOTE owners, types, GNU build-id and ABI-tag fields | Built-in static ELF inspection; [ELF notes](docs/elf-notes.md) | -| Binary hardening | PIE, NX, RELRO, CFG and related header flags for ELF, PE and Mach-O | Built-in static header inspection; [binary hardening](docs/binary-hardening.md) | -| Apple IPS crashes | .ips metadata, threads, images, registers and documented crash classes | Built-in static JSON inspection; [Apple IPS](docs/apple-ips.md) | -| Linux kernel modules | .modinfo keys, __versions CRCs, this_module presence, and signature trailer | Built-in static ELF inspection; [kernel modules](docs/kernel-modules.md) | -| Erlang BEAM | IFF chunks, atoms, exports, imports, lambdas and the Code header | Built-in static BEAM inspection; [Erlang BEAM](docs/erlang-beam.md) | -| WASM artifact | Validation, sections, exact import/export WAT forms and decoded WAT | Caller-supplied WABT 1.0.42; `inspect_wasm_artifact` | -| EVM bytecode | Dispatch selectors, byte offsets, inferred arguments and mutability | Local raw/hex carrier on Linux x64; [offline EVM guide](docs/evm-bytecode.md) | -| SQLite databases | Tables, columns, indexes, views, triggers and selected ordinary-table rows | Local database/WAL snapshot; Node.js 24.x >=24.15 or 26+ with native SQLite limits | -| Recorded Linux crashes | Raw notes, every recorded thread's registers/signals and optional mapping candidates | Caller-supplied pwntools on Linux x64; optional GDB/pwndbg; [recorded crashes](docs/recorded-crashes.md) | -| JavaScript / Electron | Modules, imports, source maps, routes, IPC, native add-ons, `rea audit-release-exposure` | Node.js and npm; Linux: procps-compatible `ps`; [application analysis](docs/javascript-application-workflows.md) | -| Websites | Page structure, scripts, network observations and requested screenshots | A Chrome-family browser; [browser analysis](docs/browser-observation.md) | -| Saved network captures | Requests, responses, exposed payloads and source locations | HAR or Fiddler Everywhere SAZ; mitmdump on Linux for native mitmproxy; [capture guide](docs/web-network-captures.md) | -| .NET assemblies | Metadata, CIL instructions, declared native dependencies and build comparisons | None (static only); [managed-code guide](docs/managed-code-analysis.md) | -| Android APKs | Manifest declarations, classes, decompiled methods and references | Headless JADX and a full JDK on Linux/macOS/Windows x64; [Android guide](docs/android-analysis.md) | -| APK signatures | v1/v2/v3/v3.1 signers, certificate fingerprints, key rotation and digest self-consistency | None (static only); `inspect_apk_signature` | -| Android devices | Devices, packages, processes, logs, dumps, screen captures, and APK/file transfers | Caller-supplied adb (emulator or hardware); [ADB guide](docs/adb-device-analysis.md) | -| JEB projects | Engine-backed project units, artifact digests and type/method pseudo-code | A caller-started JEB client serving MCP, selected with REA_JEB_MCP_URL; [JEB guide](docs/jeb-analysis.md) | -| Android resources | Decoded manifest, string tables, locales and version facts | Caller-supplied Apktool launcher; [resource guide](docs/apktool-resource-analysis.md) | -| Flutter apps | Dart snapshot hash, per-ABI library digests and engine build provenance | None (static APK parsing); [Flutter guide](docs/flutter-analysis.md) | -| Firmware | Regions, extraction results and native-analysis handoffs | Binwalk / Unblob on Linux; [firmware guide](docs/firmware-analysis.md) | -| Recorded packets | Selected fields, repeated values, filters and decode-as rules | Linux: TShark 4.x + prlimit; [capture guide](docs/packet-captures.md) | -| Packet summaries | Endpoints, DNS, TLS SNI, protocol distribution and HTTP requests | Caller-supplied tshark and capinfos (Linux-verified); [pcap guide](docs/pcap-analysis.md) | -| Recorded logic signals | Decoder annotations, sample ranges and diagnostics | Linux: sigrok-cli 0.7.2 + prlimit; [signal recordings](docs/signal-recordings.md) | -| Packages and resources | File inventories, digests, plists, Apple bundle anatomy and extracted resources | [Artifact and JavaScript guide](docs/javascript-artifact-reconstruction.md), [Apple applications](docs/apple-application-analysis.md) | -| Process behavior | Terminal output, interactions, exit and filesystem observations, and run comparisons | Linux/macOS with a native PTY; [process capture](docs/process-capture.md) | -| PyInstaller executables | CArchive members, Python version, entry scripts and extracted bytecode | None (static only); [Python packaged applications](docs/python-packaged-applications.md) | +| Target | What REA returns | Requirements and guide | +| ----------------------- | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Desktop installations | Runtime candidates, supporting files, component markers and next probes | Built-in bounded directory/header inspection; [desktop applications](docs/desktop-applications.md) | +| Native binaries | Pseudocode, assembly, strings, symbols, calls and references | Hopper, Ghidra, IDA or Kuna; [native analysis](docs/guides/native/native-investigation.md) | +| Offline ELF layout | Sections, segments, original symbols/relocations and static mitigation candidates | Caller-supplied pwntools on Linux x64; [binary diagnostics](docs/guides/native/binary-diagnostics.md) | +| Go binaries | Embedded compiler version, modules, replacements and build settings | Built-in static ELF/PE/thin Mach-O inspection; no Go installation required | +| GPU code objects | CUDA/HIP targets, selected kernel parameters and resources, retained PTX text | Built-in static ELF/PE/fatbin/CUBIN/PTX inspection; [GPU code objects](docs/gpu-code-objects.md) | +| Model artifacts | GGUF / safetensors metadata, tensor declarations and derived totals | Built-in header inspection; optional full-file hash; [model files](docs/model-artifacts.md) | +| IL2CPP metadata | Unity global-metadata.dat assembly images, type and method names and index ranges | Built-in static decoding of versions 24, 27, 29, 31 and 39; no Unity required | +| Godot PCK | Archive inventory and extraction, original header/directory facts and selected content identities | Built-in static formats 2, 3 and 4; no Godot required | +| Rust binaries | Embedded cargo-auditable crate names, versions, sources and dependency edges | Built-in static ELF/PE/thin Mach-O inspection; [Rust binaries](docs/rust-binaries.md) | +| PE certificates | Authenticode WIN_CERTIFICATE entries and PKCS#7 signer/X.509 identity fields | Built-in static PE inspection; [PE certificates](docs/pe-certificates.md) | +| PE imports | Import and delay-load DLL names, ordinals, IAT RVAs and pefile import hash | Built-in static PE inspection; [PE imports](docs/pe-imports.md) | +| ELF notes | PT_NOTE/SHT_NOTE owners, types, GNU build-id and ABI-tag fields | Built-in static ELF inspection; [ELF notes](docs/elf-notes.md) | +| Binary hardening | PIE, NX, RELRO, CFG and related header flags for ELF, PE and Mach-O | Built-in static header inspection; [binary hardening](docs/binary-hardening.md) | +| Apple IPS crashes | .ips metadata, threads, images, registers and documented crash classes | Built-in static JSON inspection; [Apple IPS](docs/apple-ips.md) | +| Linux kernel modules | .modinfo keys, __versions CRCs, this_module presence, and signature trailer | Built-in static ELF inspection; [kernel modules](docs/kernel-modules.md) | +| Erlang BEAM | IFF chunks, atoms, exports, imports, lambdas and the Code header | Built-in static BEAM inspection; [Erlang BEAM](docs/erlang-beam.md) | +| WASM artifact | Validation, sections, exact import/export WAT forms and decoded WAT | Caller-supplied WABT 1.0.42; `inspect_wasm_artifact` | +| EVM bytecode | Dispatch selectors, byte offsets, inferred arguments and mutability | Local raw/hex carrier on Linux x64; [offline EVM guide](docs/guides/evm/evm-bytecode.md) | +| SQLite databases | Tables, columns, indexes, views, triggers and selected ordinary-table rows | Local database/WAL snapshot; Node.js 24.x >=24.15 or 26+ with native SQLite limits | +| Recorded Linux crashes | Raw notes, every recorded thread's registers/signals and optional mapping candidates | Caller-supplied pwntools on Linux x64; optional GDB/pwndbg; [recorded crashes](docs/guides/native/recorded-crashes.md) | +| JavaScript / Electron | Modules, imports, source maps, routes, IPC, native add-ons, `rea audit-release-exposure` | Node.js and npm; Linux: procps-compatible `ps`; [application analysis](docs/guides/javascript/javascript-application-workflows.md) | +| Websites | Page structure, scripts, network observations and requested screenshots | A Chrome-family browser; [browser analysis](docs/guides/web/browser-observation.md) | +| Saved network captures | Requests, responses, exposed payloads and source locations | HAR or Fiddler Everywhere SAZ; mitmdump on Linux for native mitmproxy; [capture guide](docs/guides/web/web-network-captures.md) | +| .NET assemblies | Metadata, CIL instructions, declared native dependencies and build comparisons | None (static only); [managed-code guide](docs/guides/managed/managed-code-analysis.md) | +| Android APKs | Manifest declarations, classes, decompiled methods and references | Headless JADX and a full JDK on Linux/macOS/Windows x64; [Android guide](docs/guides/android/android-analysis.md) | +| APK signatures | v1/v2/v3/v3.1 signers, certificate fingerprints, key rotation and digest self-consistency | None (static only); `inspect_apk_signature` | +| Android devices | Devices, packages, processes, logs, dumps, screen captures, and APK/file transfers | Caller-supplied adb (emulator or hardware); [ADB guide](docs/guides/android/adb-device-analysis.md) | +| JEB projects | Engine-backed project units, artifact digests and type/method pseudo-code | A caller-started JEB client serving MCP, selected with REA_JEB_MCP_URL; [JEB guide](docs/providers/jeb-analysis.md) | +| Android resources | Decoded manifest, string tables, locales and version facts | Caller-supplied Apktool launcher; [resource guide](docs/guides/android/apktool-resource-analysis.md) | +| Flutter apps | Dart snapshot hash, per-ABI library digests and engine build provenance | None (static APK parsing); [Flutter guide](docs/guides/android/flutter-analysis.md) | +| Firmware | Regions, extraction results and native-analysis handoffs | Binwalk / Unblob on Linux; [firmware guide](docs/guides/firmware/firmware-analysis.md) | +| Recorded packets | Selected fields, repeated values, filters and decode-as rules | Linux: TShark 4.x + prlimit; [capture guide](docs/guides/data/packet-captures.md) | +| Packet summaries | Endpoints, DNS, TLS SNI, protocol distribution and HTTP requests | Caller-supplied tshark and capinfos (Linux-verified); [pcap guide](docs/guides/data/pcap-analysis.md) | +| Recorded logic signals | Decoder annotations, sample ranges and diagnostics | Linux: sigrok-cli 0.7.2 + prlimit; [signal recordings](docs/guides/signals/signal-recordings.md) | +| Packages and resources | File inventories, digests, plists, Apple bundle anatomy and extracted resources | [Artifact and JavaScript guide](docs/guides/javascript/javascript-artifact-reconstruction.md), [Apple applications](docs/guides/apple/apple-application-analysis.md) | +| Process behavior | Terminal output, interactions, exit and filesystem observations, and run comparisons | Linux/macOS with a native PTY; [process capture](docs/guides/runtime/process-capture.md) | +| PyInstaller executables | CArchive members, Python version, entry scripts and extracted bytecode | None (static only); [Python packaged applications](docs/python-packaged-applications.md) | Static JavaScript and .NET inspection read the supplied files without running the application. Runtime capture runs or interacts with the selected target @@ -243,15 +243,15 @@ or update either provider. Native formats and host support vary by provider. See -[Hopper and Ghidra setup](docs/installation.md#hopper), the -[IDA guide](docs/ida-provider.md) (including an optional native MCP bridge), the +[Hopper and Ghidra setup](docs/getting-started/installation.md#hopper), the +[IDA guide](docs/providers/ida-provider.md) (including an optional native MCP bridge), the [Kuna decompiler guide](docs/kuna-provider.md), and -[experimental Windows Ghidra support](docs/windows-ghidra-p0.md). -Ghidra also supports [16-bit DOS analysis](docs/ghidra-dos.md). +[experimental Windows Ghidra support](docs/providers/windows-ghidra-p0.md). +Ghidra also supports [16-bit DOS analysis](docs/providers/ghidra-dos.md). For large binaries, raise the provider's startup deadline with `REA_GHIDRA_STARTUP_TIMEOUT_MS` or `REA_HOPPER_STARTUP_TIMEOUT_MS`. -For provider selection, see the [CLI guide](docs/cli.md#choose-a-provider). -Check [release availability](docs/installation.md#released-package-and-main) +For provider selection, see the [CLI guide](docs/getting-started/cli.md#choose-a-provider). +Check [release availability](docs/getting-started/installation.md#released-package-and-main) for features added since the latest npm release. ## Showcases @@ -341,8 +341,8 @@ REA exports a runnable environment with a basic test to check that it starts and Which agents can use REA? Any agent that supports local MCP servers. Setup configures the -[supported agents](docs/installation.md#supported-agents); other clients can use -[manual MCP registration](docs/installation.md#mcp-registry). +[supported agents](docs/getting-started/installation.md#supported-agents); other clients can use +[manual MCP registration](docs/getting-started/installation.md#mcp-registry). @@ -352,7 +352,7 @@ Any agent that supports local MCP servers. Setup configures the Deep native analysis uses Hopper, Ghidra, IDA or Kuna, depending on the target and operation. Static JavaScript and .NET inspection work without a native analysis engine. Setup can install Hopper after approval; Ghidra, IDA and Kuna -use your existing installations. See [provider setup](docs/installation.md#hopper) +use your existing installations. See [provider setup](docs/getting-started/installation.md#hopper) for the options and their supported targets. @@ -364,7 +364,7 @@ REA starts Hopper when an operation needs it. On macOS, a first-run dialog may ask you to choose demo mode or activate your license. Complete that prompt in the same user session before your first CLI/MCP operation or unattended job. Open Hopper manually if needed. See -[Hopper startup and troubleshooting](docs/installation.md#launcher-paths-and-troubleshooting). +[Hopper startup and troubleshooting](docs/getting-started/installation.md#launcher-paths-and-troubleshooting). @@ -373,7 +373,7 @@ Hopper manually if needed. See The skill supplies investigation instructions for your agent. Use `npx rea-agents setup` to register REA's MCP server and install the matching instructions, then restart -your agent. See [skill-only installation](docs/installation.md#skill-only-installation). +your agent. See [skill-only installation](docs/getting-started/installation.md#skill-only-installation). @@ -422,12 +422,12 @@ with your REA version, target type, steps to reproduce and error output. Start with the website's [worked guides](https://rea.tools/guides/). For exact options, prerequisites and result contracts: -- [Installation and setup](docs/installation.md): agent registration, provider configuration, updates and uninstall. -- [Readiness and troubleshooting](docs/installation.md#check-readiness-for-your-task): diagnose one agent or analysis engine. -- [CLI and Evidence](docs/cli.md): commands, provider selection, snapshots, import/export and exit statuses. -- [MCP contracts](docs/mcp-contracts.md) and [agent prompts](docs/mcp-prompts.md): tool results, sessions and guided investigations. -- [Tool catalog](docs/mcp-contracts.md#generated-catalog): generated inventory of tools, providers and CLI commands. -- [Roadmap](docs/roadmap.md): planned work and capability trackers. +- [Installation and setup](docs/getting-started/installation.md): agent registration, provider configuration, updates and uninstall. +- [Readiness and troubleshooting](docs/getting-started/installation.md#check-readiness-for-your-task): diagnose one agent or analysis engine. +- [CLI and Evidence](docs/getting-started/cli.md): commands, provider selection, snapshots, import/export and exit statuses. +- [MCP contracts](docs/reference/mcp-contracts.md) and [agent prompts](docs/getting-started/mcp-prompts.md): tool results, sessions and guided investigations. +- [Tool catalog](docs/reference/mcp-contracts.md#generated-catalog): generated inventory of tools, providers and CLI commands. +- [Roadmap](docs/development/roadmap.md): planned work and capability trackers. Report vulnerabilities through [SECURITY.md](SECURITY.md). @@ -462,8 +462,8 @@ report a bug or suggest a feature, or [send a pull request](https://github.com/m to improve the code or docs. See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and checks, -[testing](docs/testing.md) for verification lanes, and the -[architecture map](docs/architecture.mermaid) for selected layers and dependency paths. +[testing](docs/development/testing.md) for verification lanes, and the +[architecture map](docs/architecture/architecture.mermaid) for selected layers and dependency paths. ## License diff --git a/SECURITY.md b/SECURITY.md index 516574c32d..110e5b86bc 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -27,7 +27,7 @@ provider-dependent behavior to be reproduced safely. REA authenticates its Hopper and Ghidra bridges with per-session random capability tokens. POSIX uses current-user Unix sockets; Windows Ghidra uses authenticated loopback TCP with a private NTFS runtime whose DACL admits only -the current user and SYSTEM. See [Windows Ghidra P0](docs/windows-ghidra-p0.md#private-runtime-transport-and-lifecycle). +the current user and SYSTEM. See [Windows Ghidra P0](docs/providers/windows-ghidra-p0.md#private-runtime-transport-and-lifecycle). Tokens are transferred through private session descriptors rather than process arguments or environment variables. This is not a sandbox and does not protect against malicious processes already diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 98d4af57fe..1e87d3e8a0 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -54,27 +54,27 @@ REA's adapters do not grant, replace, or waive these licenses. The upstream executables and their dependency graphs are supplied separately. Git submodules pin inspection sources; they are not included in the npm package. -| Project | License and source | REA relationship | -| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Ghidra | [Apache-2.0](https://github.com/NationalSecurityAgency/ghidra/blob/Ghidra_12.1.4_build/LICENSE), with the government-work and component qualifications in its [NOTICE](https://github.com/NationalSecurityAgency/ghidra/blob/Ghidra_12.1.4_build/NOTICE) | Bring your own Ghidra; see [supported versions](docs/installation.md#ghidra). Copies of the 12.1.4 [LICENSE](licenses/ghidra-12.1.4-LICENSE.txt), [NOTICE](licenses/ghidra-12.1.4-NOTICE.txt), and [DISCLAIMER](licenses/ghidra-12.1.4-DISCLAIMER.txt) are retained for reference. | -| ghidra-nativeaot | [MIT](https://github.com/Washi1337/ghidra-nativeaot/blob/effeb734fc570c32650f88b159608979dc7b423e/LICENSE.md) | Pinned inspection source; derived parser attribution above. | -| JADX | [Apache-2.0](https://github.com/skylot/jadx/blob/v1.5.6/LICENSE) | The audited jadx-headless-mcp source uses JADX 1.5.6 for Android decompilation; the engine JAR is separately supplied. | -| jadx-headless-mcp | [Apache-2.0](https://github.com/1013503897/jadx-headless-mcp/blob/5844800a486d2248fa949b74c9896285f5b54de2/LICENSE) | v0.7.1; pinned submodule and user-supplied JAR. | -| Kuna | [Apache-2.0](https://github.com/Noelo-Lab/kuna/blob/v1.990/LICENSE), with its [NOTICE](https://github.com/Noelo-Lab/kuna/blob/v1.990/NOTICE) | Bring your own `kuna`; REA verified 1.990 and never downloads, installs or bundles it. REA invokes it as a separate process. | -| Apktool | [Apache-2.0](https://github.com/iBotPeaches/Apktool/blob/master/LICENSE.md) | User-supplied Android resource decoder. | -| Binwalk | [MIT](https://github.com/ReFirmLabs/binwalk/blob/4fdab3d464d97b68e0af9088df3f9e2e1545b21c/LICENSE) | v3.1.0; pinned submodule and user-supplied firmware engine. | -| Unblob | [MIT](https://github.com/onekey-sec/unblob/blob/1fcc7a0a584a70a96c31f5a276c20944d199a089/LICENSE) | 26.6.4; pinned submodule and user-supplied firmware engine. | -| Wakaru | [Apache-2.0](https://github.com/pionxzh/wakaru/blob/8219bf5016a063d7fb215101a8c45d5959ec733f/LICENSE) | v1.14.0; pinned source and separately obtained JavaScript reconstruction tooling. | -| WABT | [Apache-2.0](https://github.com/WebAssembly/wabt/blob/ff0ef7e0009402740c805a9744c09b05be063e48/LICENSE) | v1.0.42; external WebAssembly tools; [integration provenance](third_party/wabt/README.md). | -| pwntools | [MIT, with GPL/BSD component qualifications](https://github.com/Gallopsled/pwntools/blob/3ad86ec5adab1abf9781ca7552a6f5d7cff7a33c/LICENSE-pwntools.txt) | 4.15.0; caller-supplied offline ELF analysis package. | -| pyelftools | [Public-domain dedication and component notices](https://github.com/eliben/pyelftools/blob/v0.33/LICENSE) | 0.33; includes separately licensed bundled material, including Construct. | -| Unicorn | [GPL-2.0 and component notices](https://github.com/unicorn-engine/unicorn/blob/2.1.2/COPYING) | 2.1.2; used by the caller-supplied ELF analysis profile for instruction emulation. | -| pwndbg | [MIT](third_party/pwndbg/LICENSE.md) | 2026.09.15, commit `55596aa676a2756efb8d67034f31675170eabe48`; separately obtained debugger extension. | -| GNU GDB | [GPL-3.0-or-later](https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/COPYING) | Caller-supplied debugger for the optional pwndbg core-file workflow. | -| LLDB / LLVM | [Apache-2.0 with LLVM exceptions](https://github.com/llvm/llvm-project/blob/main/LICENSE.TXT) | Caller-supplied debugger; the installed distribution's component and historical notices also apply. | -| mitmproxy | [MIT](https://github.com/mitmproxy/mitmproxy/blob/6c09d56e4c29a92f5ad01b03199977584b8ea14f/LICENSE) | 12.2.3; separately supplied network capture proxy. | -| Android Debug Bridge (adb) | [Apache-2.0](https://android.googlesource.com/platform/packages/modules/adb/+/refs/heads/main/NOTICE) | User-supplied Android device tool. A complete Android SDK download has its own [SDK license terms](https://developer.android.com/studio/terms); adb's source license does not replace them. | -| ILSpy / ICSharpCode.Decompiler | [MIT](https://github.com/icsharpcode/ILSpy/blob/master/ICSharpCode.Decompiler/Properties/AssemblyInfo.cs) | Optional separately installed .NET decompiler; not a redistributed REA dependency. | +| Project | License and source | REA relationship | +| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Ghidra | [Apache-2.0](https://github.com/NationalSecurityAgency/ghidra/blob/Ghidra_12.1.4_build/LICENSE), with the government-work and component qualifications in its [NOTICE](https://github.com/NationalSecurityAgency/ghidra/blob/Ghidra_12.1.4_build/NOTICE) | Bring your own Ghidra; see [supported versions](docs/getting-started/installation.md#ghidra). Copies of the 12.1.4 [LICENSE](licenses/ghidra-12.1.4-LICENSE.txt), [NOTICE](licenses/ghidra-12.1.4-NOTICE.txt), and [DISCLAIMER](licenses/ghidra-12.1.4-DISCLAIMER.txt) are retained for reference. | +| ghidra-nativeaot | [MIT](https://github.com/Washi1337/ghidra-nativeaot/blob/effeb734fc570c32650f88b159608979dc7b423e/LICENSE.md) | Pinned inspection source; derived parser attribution above. | +| JADX | [Apache-2.0](https://github.com/skylot/jadx/blob/v1.5.6/LICENSE) | The audited jadx-headless-mcp source uses JADX 1.5.6 for Android decompilation; the engine JAR is separately supplied. | +| jadx-headless-mcp | [Apache-2.0](https://github.com/1013503897/jadx-headless-mcp/blob/5844800a486d2248fa949b74c9896285f5b54de2/LICENSE) | v0.7.1; pinned submodule and user-supplied JAR. | +| Kuna | [Apache-2.0](https://github.com/Noelo-Lab/kuna/blob/v1.990/LICENSE), with its [NOTICE](https://github.com/Noelo-Lab/kuna/blob/v1.990/NOTICE) | Bring your own `kuna`; REA verified 1.990 and never downloads, installs or bundles it. REA invokes it as a separate process. | +| Apktool | [Apache-2.0](https://github.com/iBotPeaches/Apktool/blob/master/LICENSE.md) | User-supplied Android resource decoder. | +| Binwalk | [MIT](https://github.com/ReFirmLabs/binwalk/blob/4fdab3d464d97b68e0af9088df3f9e2e1545b21c/LICENSE) | v3.1.0; pinned submodule and user-supplied firmware engine. | +| Unblob | [MIT](https://github.com/onekey-sec/unblob/blob/1fcc7a0a584a70a96c31f5a276c20944d199a089/LICENSE) | 26.6.4; pinned submodule and user-supplied firmware engine. | +| Wakaru | [Apache-2.0](https://github.com/pionxzh/wakaru/blob/8219bf5016a063d7fb215101a8c45d5959ec733f/LICENSE) | v1.14.0; pinned source and separately obtained JavaScript reconstruction tooling. | +| WABT | [Apache-2.0](https://github.com/WebAssembly/wabt/blob/ff0ef7e0009402740c805a9744c09b05be063e48/LICENSE) | v1.0.42; external WebAssembly tools; [integration provenance](third_party/wabt/README.md). | +| pwntools | [MIT, with GPL/BSD component qualifications](https://github.com/Gallopsled/pwntools/blob/3ad86ec5adab1abf9781ca7552a6f5d7cff7a33c/LICENSE-pwntools.txt) | 4.15.0; caller-supplied offline ELF analysis package. | +| pyelftools | [Public-domain dedication and component notices](https://github.com/eliben/pyelftools/blob/v0.33/LICENSE) | 0.33; includes separately licensed bundled material, including Construct. | +| Unicorn | [GPL-2.0 and component notices](https://github.com/unicorn-engine/unicorn/blob/2.1.2/COPYING) | 2.1.2; used by the caller-supplied ELF analysis profile for instruction emulation. | +| pwndbg | [MIT](third_party/pwndbg/LICENSE.md) | 2026.09.15, commit `55596aa676a2756efb8d67034f31675170eabe48`; separately obtained debugger extension. | +| GNU GDB | [GPL-3.0-or-later](https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/COPYING) | Caller-supplied debugger for the optional pwndbg core-file workflow. | +| LLDB / LLVM | [Apache-2.0 with LLVM exceptions](https://github.com/llvm/llvm-project/blob/main/LICENSE.TXT) | Caller-supplied debugger; the installed distribution's component and historical notices also apply. | +| mitmproxy | [MIT](https://github.com/mitmproxy/mitmproxy/blob/6c09d56e4c29a92f5ad01b03199977584b8ea14f/LICENSE) | 12.2.3; separately supplied network capture proxy. | +| Android Debug Bridge (adb) | [Apache-2.0](https://android.googlesource.com/platform/packages/modules/adb/+/refs/heads/main/NOTICE) | User-supplied Android device tool. A complete Android SDK download has its own [SDK license terms](https://developer.android.com/studio/terms); adb's source license does not replace them. | +| ILSpy / ICSharpCode.Decompiler | [MIT](https://github.com/icsharpcode/ILSpy/blob/master/ICSharpCode.Decompiler/Properties/AssemblyInfo.cs) | Optional separately installed .NET decompiler; not a redistributed REA dependency. | Ghidra's original notice credits software developed at the National Security Agency. It explains that US-government-created portions are not subject to US diff --git a/bridge/agentenv/capture.py b/bridge/agentenv/capture.py new file mode 100644 index 0000000000..b1cb06354e --- /dev/null +++ b/bridge/agentenv/capture.py @@ -0,0 +1,262 @@ +"""Run REA inside an agentenv build computer for REA on the host: capture the app, draft its task and export it. + +REA sends this file's source through the computer's ``computer-exec/v1`` route as +``python3 -I -c ``; each command prints one JSON line and exits. ``start`` +leaves the capture running on its own: its stdin is a FIFO it holds open for reading and writing, so it never +sees the end of its input, and ``mark`` and ``stop`` write their requests there and wait for the capture's own +acknowledgement on its progress log. A small shell around the capture writes its exit status to a file, the +mark a later command reads. REA runs as the session user in the home, with no ``AGENTENV_*`` variable, so +neither REA nor the app it launches holds the computer's control token, and with ``APT_CONFIG`` naming the +package lists ``start`` downloads as that user, so the capture finds the package's sources. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import shutil +import subprocess +import sys +import time + +POLL_S = 0.2 +TAIL_BYTES = 4096 + + +def _say(value: dict) -> int: + print(json.dumps(value), flush=True) + return 0 + + +def _tail(path: str) -> str: + try: + with open(path, "rb") as handle: + handle.seek(max(0, os.path.getsize(path) - TAIL_BYTES)) + return handle.read().decode("utf-8", "replace") + except OSError: + return "" + + +def _environment(work: str) -> dict: + environment = {name: value for name, value in os.environ.items() if not name.startswith("AGENTENV_")} + environment["APT_CONFIG"] = os.path.join(work, "apt", "apt.conf") + return environment + + +def _apt(work: str, timeout: float) -> dict: + """Download the package lists into the work folder as the session user, until one update succeeds.""" + root = os.path.join(work, "apt") + lists = os.path.join(root, "lists") + updated = os.path.join(root, "updated") + os.makedirs(os.path.join(lists, "partial"), exist_ok=True) + os.makedirs(os.path.join(root, "cache", "archives", "partial"), exist_ok=True) + with open(os.path.join(root, "apt.conf"), "w", encoding="utf-8") as conf: + conf.write( + f'Dir::State::Lists "{lists}";\n' + f'Dir::Cache "{root}/cache";\n' + 'Debug::NoLocking "true";\n' + 'APT::Update::Error-Mode "any";\n' + ) + # apt keeps lists compressed as it fetched them, such as *_Packages.lz4 on Ubuntu. + count = lambda: len([name for name in os.listdir(lists) if re.search(r"_Packages(\.\w+)?$", name)]) # noqa: E731 + # A marker left by an update that fetched nothing is not a success; download again. + if os.path.exists(updated) and count() > 0: + return {"lists": count(), "updated": False} + log = os.path.join(root, "update.log") + try: + with open(log, "wb") as output: + code = subprocess.run(["apt-get", "update"], env=_environment(work), stdin=subprocess.DEVNULL, + stdout=output, stderr=subprocess.STDOUT, timeout=timeout).returncode + except (OSError, subprocess.TimeoutExpired) as error: + return {"lists": count(), "updated": True, "error": f"{type(error).__name__}: {error}"} + lists_fetched = count() + if code == 0 and lists_fetched > 0: + # Only a whole update that fetched lists is kept, so a failed or empty one runs again. + open(updated, "w").close() + return {"lists": lists_fetched, "updated": True, "exit_code": code} + if code == 0: + return {"lists": lists_fetched, "updated": True, "exit_code": code, + "error": "apt-get update exited 0 without a package list"} + return {"lists": lists_fetched, "updated": True, "exit_code": code, + "error": f"apt-get update exited {code}: {_tail(log)[-1500:].strip()}"} + + +def _exit_code(capture: str): + try: + with open(os.path.join(capture, "exit"), encoding="utf-8") as handle: + return int(handle.read().strip() or "-1") + except (OSError, ValueError): + return None + + +def _event_line(path: str, offset: int, kinds: set, name): + """The first progress line after ``offset`` whose event is of one of ``kinds``, for ``name`` when given.""" + try: + with open(path, "rb") as handle: + handle.seek(offset) + lines = handle.read().decode("utf-8", "replace").splitlines() + except OSError: + return None + for line in lines: + try: + event = json.loads(line)["rea_progress"]["event"] + except (ValueError, KeyError, TypeError): + continue + if isinstance(event, dict) and event.get("kind") in kinds and (name is None or event.get("name") == name): + return line + return None + + +def _wait(capture: str, offset: int, kinds: set, name, timeout: float) -> dict: + """Wait for the capture's event, or for it to exit, or for time to run out.""" + log = os.path.join(capture, "progress.log") + deadline = time.monotonic() + timeout + while True: + line = _event_line(log, offset, kinds, name) + if line is not None: + return {"line": line} + code = _exit_code(capture) + if code is not None: + return {"exited": code, "progress": _tail(log), "output": _tail(os.path.join(capture, "capture.out"))} + if time.monotonic() > deadline: + return {"timed_out": True, "progress": _tail(log)} + time.sleep(POLL_S) + + +def start(arguments: dict) -> int: + work, capture = os.path.abspath(arguments["work"]), os.path.abspath(arguments["capture"]) + rea = shutil.which("rea") + if rea is None: + return _say({"error": "REA is not installed in this computer; start the computer from a request, which installs it."}) + if os.path.exists(capture): + return _say({"error": f"the capture folder {capture} is already used"}) + apt = _apt(work, arguments.get("apt_timeout_s", 600)) + try: + os.makedirs(capture) + except FileExistsError: + return _say({"error": f"a stop claimed {capture} before the capture launched", "apt": apt}) + control = os.path.join(capture, "control") + os.mkfifo(control, 0o600) + argv = [rea, "capture-desktop-application", "--desktop-entry", arguments["desktop_entry"], + "--output", os.path.join(capture, "capture"), "--exclude", work, "--json"] + for value in arguments.get("arguments", []): + argv += ["--argument", value] + if arguments.get("package_file"): + argv += ["--package-file", arguments["package_file"]] + # Reading and writing, so the capture never reads the end of its input between requests. + reader = os.open(control, os.O_RDWR) + try: + with open(os.path.join(capture, "capture.out"), "wb") as out, \ + open(os.path.join(capture, "progress.log"), "wb") as progress: + process = subprocess.Popen( + ["sh", "-c", 'done=$1; shift; "$@"; code=$?; printf "%s" "$code" > "$done.tmp" && mv "$done.tmp" "$done"', + "sh", os.path.join(capture, "exit"), *argv], + cwd=os.path.expanduser("~"), env=_environment(work), stdin=reader, stdout=out, stderr=progress, + start_new_session=True) + finally: + os.close(reader) + waited = _wait(capture, 0, {"window_open"}, None, arguments.get("timeout_s", 120)) + return _say({"pid": process.pid, "apt": apt, **waited}) + + +def _request(capture: str, line: str) -> bool: + """Write one request to the running capture; False when nothing reads its FIFO any more.""" + try: + fd = os.open(os.path.join(capture, "control"), os.O_WRONLY | os.O_NONBLOCK) + except OSError: + return False + try: + os.write(fd, (line + "\n").encode("utf-8")) + finally: + os.close(fd) + return True + + +def mark(arguments: dict) -> int: + capture, name = os.path.abspath(arguments["capture"]), arguments["name"] + code = _exit_code(capture) + log = os.path.join(capture, "progress.log") + if code is not None: + return _say({"exited": code, "progress": _tail(log)}) + # A checkpoint already taken, whose answer was lost, is answered from the capture's own log. + taken = _event_line(log, 0, {"checkpoint"}, name) + if taken is not None: + return _say({"line": taken}) + offset = os.path.getsize(log) + if not _request(capture, f"checkpoint {name}"): + return _say({"error": "the capture is not reading its requests"}) + return _say(_wait(capture, offset, {"checkpoint", "checkpoint_refused"}, name, arguments.get("timeout_s", 600))) + + +def stop(arguments: dict) -> int: + capture = os.path.abspath(arguments["capture"]) + if _exit_code(capture) is None and not os.path.exists(os.path.join(capture, "control")): + # Never launched: claim the folder, so a start still on its way finds it used and launches nothing. + os.makedirs(capture, exist_ok=True) + return _say({"absent": True}) + if _exit_code(capture) is None: + _request(capture, "stop") + deadline = time.monotonic() + arguments.get("timeout_s", 600) + while (code := _exit_code(capture)) is None and time.monotonic() < deadline: + time.sleep(POLL_S) + log = os.path.join(capture, "progress.log") + if code is None: + return _say({"timed_out": True, "progress": _tail(log)}) + output = os.path.join(capture, "capture.out") + return _say({"exit_code": code, "progress": _tail(log), "output_bytes": os.path.getsize(output)}) + + +def run(arguments: dict) -> int: + """Run one REA command with its output in a file, after writing the files it reads.""" + rea = shutil.which("rea") + if rea is None: + return _say({"error": "REA is not installed in this computer"}) + for path, text in arguments.get("files", {}).items(): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as handle: + handle.write(text) + with open(arguments["stdout"], "wb") as out: + try: + done = subprocess.run([rea, *arguments["argv"]], cwd=os.path.expanduser("~"), + env=_environment(os.path.abspath(arguments["work"])), stdin=subprocess.DEVNULL, stdout=out, + stderr=subprocess.PIPE, timeout=arguments.get("timeout_s", 1800)) + except subprocess.TimeoutExpired: + return _say({"timed_out": True}) + return _say({"exit_code": done.returncode, "stdout_bytes": os.path.getsize(arguments["stdout"]), + "stderr": done.stderr[-TAIL_BYTES:].decode("utf-8", "replace")}) + + +def listing(arguments: dict) -> int: + """Every regular file under a folder, by its path relative to the home, with its size and SHA-256. + + The list goes to the file ``output`` names, outside the folder, and the answer gives its size: a large + bundle's list outgrows the 1 MiB of stdout a command answers with, so REA reads it through ``data/get``. + """ + home = os.path.expanduser("~") + files, skipped = [], [] + for directory, folders, names in os.walk(os.path.abspath(arguments["root"])): + folders.sort() + for name in sorted(names): + path = os.path.join(directory, name) + relative = os.path.relpath(path, home) + if os.path.islink(path) or not os.path.isfile(path): + skipped.append(relative) + continue + digest = hashlib.sha256() + with open(path, "rb") as handle: + for chunk in iter(lambda: handle.read(1 << 20), b""): + digest.update(chunk) + files.append({"path": relative, "size": os.path.getsize(path), "sha256": digest.hexdigest()}) + output = os.path.abspath(arguments["output"]) + os.makedirs(os.path.dirname(output), exist_ok=True) + with open(output, "w", encoding="utf-8") as handle: + json.dump({"files": files, "skipped": skipped}, handle) + return _say({"listing_bytes": os.path.getsize(output)}) + + +COMMANDS = {"start": start, "mark": mark, "stop": stop, "run": run, "list": listing} + +if __name__ == "__main__": + sys.exit(COMMANDS[sys.argv[1]](json.loads(sys.argv[2]))) diff --git a/bridge/agentenv/computer.py b/bridge/agentenv/computer.py new file mode 100644 index 0000000000..fc1540e29b --- /dev/null +++ b/bridge/agentenv/computer.py @@ -0,0 +1,83 @@ +"""Keep one agentenv computer up for REA until stdin closes or SIGTERM arrives. + +REA runs ``computer.py [task]`` with the agentenv runner's own interpreter, so ``agent_env`` +imports. It deploys the bundle's task, ``smoke`` by default, on this machine with +``run_bundle(keep=True, emulate=False)``, prints one JSON line naming the computer's address, MCP URL, control +token and sandbox id, or ``{"error": ...}``, and tears the computer down once its stdin closes or it gets SIGTERM +or SIGINT. Everything else it or agentenv prints, a build's output included, goes to stderr. It exits 0 once +the teardown left nothing up, 1 when it did, and 2 when no computer came up. +""" + +from __future__ import annotations + +import json +import os +import signal +import sys +import threading + + +def _protocol(): + """A private copy of stdout for the JSON line; fd 1 then points at stderr, as do children that inherit it.""" + protocol = os.fdopen(os.dup(1), "w", encoding="utf-8") + os.dup2(2, 1) + sys.stdout = sys.stderr + return protocol + + +def _say(protocol, value: dict) -> None: + protocol.write(json.dumps(value) + "\n") + protocol.flush() + + +def _computer(result): + """The run's deployed computer, or why there is none.""" + run = result.runs[0] if result.runs else None + if run is None: + return None, "the bundle ran no task" + if run.error is not None or run.failed_steps: + return None, f"the task failed: {run.error or run.failed_steps}" + for env in run.context.deployed_envs: + if env.environment_url and env.mcp_url and getattr(env, "sandbox_id", None): + return env, None + return None, "the task deployed no computer" + + +def main(argv: list[str]) -> int: + protocol = _protocol() + bundle, task = argv[1], argv[2] if len(argv) > 2 else "smoke" + try: + from agent_env.bundle import run_bundle + except ImportError as error: + _say(protocol, {"error": f"agent_env does not import with {sys.executable}: {error}"}) + return 2 + try: + result = run_bundle(bundle, tasks=[task], keep=True, emulate=False, sandbox="local", + on_progress=lambda line: print(line, file=sys.stderr, flush=True)) + except BaseException as error: # noqa: BLE001 - a signal's RunInterrupted too; run_bundle tore its runs down + _say(protocol, {"error": f"{type(error).__name__}: {error}"}) + return 2 + stop = threading.Event() + for signum in (signal.SIGTERM, signal.SIGINT): + signal.signal(signum, lambda *_: stop.set()) + env, problem = _computer(result) + try: + if env is None: + _say(protocol, {"error": problem}) + else: + _say(protocol, {"base_url": env.environment_url, "mcp_url": env.mcp_url, + "control_token": env.control_token, "sandbox_id": env.sandbox_id}) + threading.Thread(target=lambda: (sys.stdin.buffer.read(), stop.set()), daemon=True).start() + while not stop.wait(0.5): + pass + finally: + torn_down = result.teardown() + still_up = [sandbox for run in torn_down.runs if run.torn_down is not None for sandbox in run.torn_down.still_up] + if still_up: + print(f"still up after the teardown: {still_up}", file=sys.stderr, flush=True) + return 1 + return 0 if env is not None else 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/bridge/ghidra/ReaGhidraBridge.java b/bridge/ghidra/ReaGhidraBridge.java index 05cbb01781..1c97f91bd1 100644 --- a/bridge/ghidra/ReaGhidraBridge.java +++ b/bridge/ghidra/ReaGhidraBridge.java @@ -187,6 +187,8 @@ public final class ReaGhidraBridge extends HeadlessScript { private static final int MAX_DECOMPILE_TIMEOUT_SECONDS = Integer.MAX_VALUE / 1000; private final ReaGhidraExtensions analysisExtensions = new ReaGhidraExtensions(); private static AddressSpace sessionDefaultAddressSpace; + private Path checkpointDirectory; + private long checkpointModification; @Override public void run() throws Exception { @@ -195,6 +197,7 @@ public final class ReaGhidraBridge extends HeadlessScript { throw new IllegalArgumentException("REA bridge requires one session descriptor"); } Path descriptorPath = Path.of(arguments[0]); + checkpointDirectory = descriptorPath.toAbsolutePath().normalize().getParent(); SessionDescriptor descriptor = readDescriptor(descriptorPath); decompileTimeoutSeconds = descriptor.decompileTimeoutSeconds(); // Windows retains a native immutable bearer descriptor lease until @@ -247,6 +250,7 @@ public final class ReaGhidraBridge extends HeadlessScript { throw new IOException("Windows Ghidra P0 does not admit metadata recovery or database mutation."); analysisExtensions.analyze(descriptor.analysisExtensions, currentProgram, monitor); initializeDecompiler(); + checkpointModification = currentProgram.getModificationNumber(); serve(descriptor); } finally { @@ -451,6 +455,7 @@ public final class ReaGhidraBridge extends HeadlessScript { if (descriptor.existingProject && isMutatingOperation(request.method)) throw new RequestFailure("method_unavailable", "Existing Ghidra project sessions are read-only"); return switch (request.method) { + case "checkpoint_program" -> checkpointProgram(request.params, descriptor); case "read_binary_inventory_summary" -> binaryInventorySummary(request.params); case "inspect_native_load_image" -> inspectNativeLoadImage(request.params); case "read_bytes" -> readMemoryBytes(request.params); @@ -506,6 +511,31 @@ public final class ReaGhidraBridge extends HeadlessScript { return method.equals("annotate_native_function") || method.equals("set_address_name") || method.equals("set_addresses_names"); } + /** Internal lifecycle request: the destination is inside this owned runtime. */ + private JsonObject checkpointProgram(JsonObject params, SessionDescriptor descriptor) throws Exception { + requireKeys(params, Set.of("path", "force")); + if (descriptor.existingProject || !descriptor.transport.equals("unix-socket")) + throw new RequestFailure("method_unavailable", "This Program has no checkpoint authority"); + Path output = Path.of(requireString(params, "path")).toAbsolutePath().normalize(); + if (!output.getParent().equals(checkpointDirectory) || + !output.getFileName().toString().matches("checkpoint-[a-f0-9-]{36}\\.gzf")) + throw new RequestFailure("invalid_request", "Checkpoint destination must belong to this runtime"); + JsonElement forced = params.get("force"); + if (!forced.isJsonPrimitive() || !forced.getAsJsonPrimitive().isBoolean()) + throw new RequestFailure("invalid_request", "Checkpoint force must be a boolean"); + boolean force = forced.getAsBoolean(); + JsonObject result = new JsonObject(); + boolean changed = currentProgram.getModificationNumber() != checkpointModification; + result.addProperty("saved", force || changed); + if (force || changed) { + if (Files.exists(output)) throw new RequestFailure("invalid_request", "Checkpoint destination already exists"); + currentProgram.saveToPackedFile(output.toFile(), monitor); + checkpointModification = currentProgram.getModificationNumber(); + result.addProperty("path", output.toString()); + } + return result; + } + private JsonObject sessionInfo(SessionDescriptor descriptor) throws Exception { boolean timedOut = analysisTimeoutOccurred(); JsonObject provider = new JsonObject(); @@ -1384,8 +1414,29 @@ public final class ReaGhidraBridge extends HeadlessScript { flow.addProperty("conditional", false); flow.addProperty("computed", false); if (instruction == null) { - result.addProperty("status", !currentProgram.getMemory().contains(address) ? "outside-memory" : currentProgram.getListing().getInstructionContaining(address) != null - ? "not-instruction-boundary" : currentProgram.getListing().getDefinedDataContaining(address) != null ? "data" : "not-in-listing"); + Instruction containing = currentProgram.getListing().getInstructionContaining(address); + boolean inMemory = currentProgram.getMemory().contains(address); + Data containingData = inMemory && containing == null + ? currentProgram.getListing().getDefinedDataContaining(address) : null; + result.addProperty("status", !inMemory ? "outside-memory" : containing != null + ? "not-instruction-boundary" : containingData != null ? "data" : "not-in-listing"); + if (inMemory && containing == null && containingData != null) { + JsonObject parentData = new JsonObject(); + parentData.addProperty("address", canonicalAddress(containingData.getAddress())); + parentData.addProperty("length", containingData.getLength()); + parentData.addProperty("type_id", containingData.getDataType().getPathName()); + parentData.addProperty("provenance", "ghidra-listing"); + result.add("containing_data", parentData); + } + if (containing != null && currentProgram.getMemory().contains(address)) { + JsonObject containingInstruction = new JsonObject(); + containingInstruction.addProperty("address", canonicalAddress(containing.getAddress())); + containingInstruction.addProperty("length", containing.getLength()); + containingInstruction.addProperty("mnemonic", containing.getMnemonicString()); + containingInstruction.addProperty("raw_disassembly", containing.toString()); + containingInstruction.addProperty("provenance", "ghidra-listing"); + result.add("containing_instruction", containingInstruction); + } for (String key : List.of("bytes", "length", "mnemonic", "raw_disassembly")) result.add(key, JsonNull.INSTANCE); } else { result.addProperty("status", "decoded"); @@ -2346,7 +2397,7 @@ public final class ReaGhidraBridge extends HeadlessScript { if (!match.matched()) { continue; } - JsonObject item = resultItem(entry.item, null); + JsonObject item = resultItem(entry.item, "procedure"); if (!match.aliases().isEmpty()) { JsonArray aliases = new JsonArray(); for (String alias : match.aliases()) aliases.add(alias); @@ -2361,7 +2412,7 @@ public final class ReaGhidraBridge extends HeadlessScript { if (!matcher.matches(item.value)) { continue; } - items.add(resultItem(item, null)); + items.add(resultItem(item, procedureSearch ? "procedure" : "string")); } return items; } diff --git a/bridge/ghidra/ReaGhidraRiscvCallingConvention.java b/bridge/ghidra/ReaGhidraRiscvCallingConvention.java index 89fbbc276c..7e857bb029 100644 --- a/bridge/ghidra/ReaGhidraRiscvCallingConvention.java +++ b/bridge/ghidra/ReaGhidraRiscvCallingConvention.java @@ -37,7 +37,7 @@ import org.xml.sax.InputSource; /** * Apply the committed RISC-V calling-convention policy after analysis. Ghidra's generic RV32 * compiler specification models ilp32d. Each policy derives a program-specific prototype from the - * program's own default prototype, as its {@link Policy} row and docs/ghidra-riscv.md state. The + * program's own default prototype, as its {@link Policy} row and docs/providers/ghidra-riscv.md state. The * script installs the prototype as a specification extension and assigns it, with dynamic * storage, to every non-thunk function, external function and function definition. */ diff --git a/bridge/native/rea_lldb_tracer.py b/bridge/native/rea_lldb_tracer.py index 11e365b8cc..b6f5fef42b 100644 --- a/bridge/native/rea_lldb_tracer.py +++ b/bridge/native/rea_lldb_tracer.py @@ -243,6 +243,20 @@ def _hex(value): return "0x%x" % value +def _source_location(address): + """Preserve available line-entry facets without acquiring source files.""" + entry = address.GetLineEntry() + if not entry.IsValid(): + return None + file_spec = entry.GetFileSpec() + file_path = str(file_spec) if file_spec.IsValid() else None + line = entry.GetLine() or None + column = entry.GetColumn() or None + if not file_path and line is None and column is None: + return None + return {"file_path": file_path or None, "line": line, "column": column} + + def _location(address, target): """Load and file address, module and symbol of one code address.""" module = address.GetModule() @@ -256,6 +270,7 @@ def _location(address, target): "module": module.GetFileSpec().GetFilename() if module.IsValid() else None, "module_path": str(module.GetFileSpec()) if module.IsValid() else None, "symbol": symbol.GetName() if symbol.IsValid() else None, + "source_location": _source_location(address), } return location diff --git a/bridge/native/test_rea_lldb_tracer.py b/bridge/native/test_rea_lldb_tracer.py index e1c3e66e97..5fd7c0d1f2 100644 --- a/bridge/native/test_rea_lldb_tracer.py +++ b/bridge/native/test_rea_lldb_tracer.py @@ -25,6 +25,45 @@ spec.loader.exec_module(tracer) class NativeTraceBudgetTests(unittest.TestCase): + def test_source_coordinates_preserve_partial_metadata_without_reading_source(self): + def address(valid=True, file_valid=True, line=23, column=7): + class SourceFileSpec: + def IsValid(self): + return file_valid + + def __str__(self): + return "/missing/é space/fixture.pas" + + file_spec = SourceFileSpec() + entry = types.SimpleNamespace( + IsValid=lambda: valid, + GetFileSpec=lambda: file_spec, + GetLine=lambda: line, + GetColumn=lambda: column, + ) + return types.SimpleNamespace(GetLineEntry=lambda: entry) + + self.assertEqual( + tracer._source_location(address()), + {"file_path": "/missing/é space/fixture.pas", "line": 23, "column": 7}, + ) + self.assertEqual( + tracer._source_location(address(column=0)), + {"file_path": "/missing/é space/fixture.pas", "line": 23, "column": None}, + ) + self.assertEqual( + tracer._source_location(address(file_valid=False)), + {"file_path": None, "line": 23, "column": 7}, + ) + self.assertEqual( + tracer._source_location(address(line=0)), + {"file_path": "/missing/é space/fixture.pas", "line": None, "column": 7}, + ) + self.assertIsNone(tracer._source_location(address(valid=False))) + self.assertIsNone( + tracer._source_location(address(file_valid=False, line=0, column=0)) + ) + def test_invalid_launch_process_reports_explicit_no_creation(self): class InvalidProcess: def IsValid(self): @@ -390,6 +429,9 @@ class NativeTraceBudgetTests(unittest.TestCase): return FileSpec() class Address: + def GetLineEntry(self): + return types.SimpleNamespace(IsValid=lambda: False) + def GetModule(self): return Module() @@ -491,7 +533,24 @@ class NativeTraceBudgetTests(unittest.TestCase): def GetFileSpec(self): return FileSpec() + class SourceFileSpec: + def IsValid(self): + return True + + def __str__(self): + return source_path + class Address: + def GetLineEntry(self): + if source_path is None: + return types.SimpleNamespace(IsValid=lambda: False) + return types.SimpleNamespace( + IsValid=lambda: True, + GetFileSpec=SourceFileSpec, + GetLine=lambda: 15, + GetColumn=lambda: 3, + ) + def GetModule(self): return Module() @@ -518,19 +577,35 @@ class NativeTraceBudgetTests(unittest.TestCase): breakpoint_count = 1000 created = [[Breakpoint()] for _ in range(breakpoint_count)] config = {"breakpoints": [{"kind": "objc-method"}] * breakpoint_count} - report, locations_truncated = tracer._breakpoint_report( - object(), created, config - ) - retained_bytes = sum( - len(json.dumps(location).encode("utf-8")) - + (1 if index > 0 else 0) - for row in report - for index, location in enumerate(row["locations"]) - ) - self.assertTrue(locations_truncated) - self.assertEqual(sum(row["location_count"] for row in report), 64_000) - self.assertLess(sum(len(row["locations"]) for row in report), 64_000) - self.assertLessEqual(retained_bytes, tracer.MAX_BREAKPOINT_LOCATION_BYTES) + for source_path in (None, "/tmp/" + "源🚀" * 3000 + "/calls.pas"): + with self.subTest(source_path_present=source_path is not None): + report, locations_truncated = tracer._breakpoint_report( + object(), created, config + ) + retained_bytes = sum( + len(json.dumps(location).encode("utf-8")) + + (1 if index > 0 else 0) + for row in report + for index, location in enumerate(row["locations"]) + ) + self.assertTrue(locations_truncated) + self.assertEqual( + sum(row["location_count"] for row in report), 64_000 + ) + self.assertLess( + sum(len(row["locations"]) for row in report), 64_000 + ) + self.assertLessEqual( + retained_bytes, tracer.MAX_BREAKPOINT_LOCATION_BYTES + ) + if source_path is not None: + self.assertTrue( + any( + location["source_location"]["file_path"] == source_path + for row in report + for location in row["locations"] + ) + ) if __name__ == "__main__": diff --git a/bridge/pwndbg/test_core_context.py b/bridge/pwndbg/test_core_context.py index 3e61455a91..42f09d901e 100644 --- a/bridge/pwndbg/test_core_context.py +++ b/bridge/pwndbg/test_core_context.py @@ -109,7 +109,7 @@ class CoreContextTests(unittest.TestCase): self.assertFalse(result["ok"]) self.assertEqual(result["reason"], "output-limit") self.assertNotIn("exceeds64", result["message"]) - self.assertEqual(result["message"], "Complete debugger context exceeds the 64 MiB reply budget.") + self.assertIn("64 MiB", result["message"]) self.assertEqual(reads, []) self.assertEqual(queries, ["maps"]) diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 2a69c3072d..ed51121c51 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -1,4 +1,18 @@ import { defineConfig } from "vitepress"; +import { dirname, relative, resolve, sep } from "node:path"; +import { fileURLToPath } from "node:url"; +import { + documentationRewrites, + documentationSections, +} from "./documentation.js"; + +const documentationRoot = fileURLToPath(new URL("..", import.meta.url)); +const rewrites: Record = { + ...documentationRewrites, + "README.md": "documentation.md", + "specs/README.md": "specs/index.md", + "adr/README.md": "adr/index.md", +}; export default defineConfig({ title: "REA", @@ -6,128 +20,92 @@ export default defineConfig({ "Reverse engineer binaries, applications, and runtime behavior with one CLI and MCP server.", lang: "en-US", base: "/rea/", + rewrites, + markdown: { + config(md) { + // Repository links resolve from their physical source. Render them from + // the stable public route instead, including links across categories. + md.core.ruler.after("inline", "documentation-source-links", (state) => { + const physicalPath = + state.env.realPath ?? + (state.env.relativePath + ? resolve(documentationRoot, state.env.relativePath) + : null); + if (physicalPath === null) return; + const renderedPath = + state.env.path ?? + resolve( + documentationRoot, + rewrites[state.env.relativePath] ?? state.env.relativePath, + ); + for (const block of state.tokens) { + for (const token of block.children ?? []) { + const attribute = + token.type === "link_open" + ? "href" + : token.type === "image" + ? "src" + : null; + if (attribute === null) continue; + const destination = token.attrGet(attribute); + if ( + !destination || + /^(?:[a-z][a-z\d+.-]*:|[/#])/iu.test(destination) + ) + continue; + const separator = destination.search(/[?#]/u); + const path = + separator === -1 ? destination : destination.slice(0, separator); + const suffix = separator === -1 ? "" : destination.slice(separator); + const source = relative( + documentationRoot, + resolve(dirname(physicalPath), path), + ) + .split(sep) + .join("/"); + const rendered = relative( + dirname(renderedPath), + resolve( + documentationRoot, + source.startsWith("public/") + ? source.slice("public/".length) + : (rewrites[source] ?? source), + ), + ) + .split(sep) + .join("/"); + token.attrSet(attribute, rendered + suffix); + } + } + }); + }, + }, srcExclude: ["readme/**"], themeConfig: { nav: [ { text: "Get started", link: "/installation" }, + { text: "Documentation", link: "/documentation" }, { text: "Reference", link: "/mcp-contracts" }, { text: "Roadmap", link: "/roadmap" }, ], sidebar: [ { - text: "Getting started", + text: "Documentation directory", items: [ - { text: "Installation and setup", link: "/installation" }, - { text: "Client configuration", link: "/client-configuration" }, - { text: "CLI and Evidence", link: "/cli" }, - { text: "Agent prompts", link: "/mcp-prompts" }, - { text: "Roadmap", link: "/roadmap" }, - ], - }, - { - text: "Investigation guides", - items: [ - { text: "Native binaries", link: "/native-investigation" }, - { text: "Go build metadata", link: "/go-binaries" }, - { text: "IL2CPP metadata", link: "/il2cpp-metadata" }, - { text: "Model artifacts", link: "/model-artifacts" }, - { text: "Rust dependencies", link: "/rust-binaries" }, - { text: "PE certificates", link: "/pe-certificates" }, - { text: "PE imports", link: "/pe-imports" }, - { text: "PE TLS/load config", link: "/pe-load-config" }, - { text: "ELF notes", link: "/elf-notes" }, - { text: "Binary hardening", link: "/binary-hardening" }, - { text: "Apple IPS crash reports", link: "/apple-ips" }, - { text: "Linux kernel modules", link: "/kernel-modules" }, - { text: "CPython pyc files", link: "/python-pyc" }, - { text: "Erlang BEAM", link: "/erlang-beam" }, - { - text: "JavaScript and Electron", - link: "/javascript-artifact-reconstruction", - }, - { - text: "JavaScript workflows", - link: "/javascript-application-workflows", - }, - { text: "Browser observation", link: "/browser-observation" }, - { - text: "Web network captures", - link: "/web-network-captures", - }, - { - text: "Build RL environments", - link: "/agent-environments-quickstart", - }, - { text: "Environment reference", link: "/agent-environments" }, - { text: "Electron observation", link: "/electron-observation" }, - { text: "Electron fuses", link: "/electron-fuses" }, - { text: "Release exposure", link: "/release-exposure" }, - { text: "Process capture", link: "/process-capture" }, - { - text: "Desktop application capture", - link: "/desktop-application-capture", - }, - { - text: "Desktop application state", - link: "/desktop-application-state", - }, - { - text: "Desktop application export", - link: "/desktop-application-export", - }, - { - text: "Desktop application tasks", - link: "/desktop-application-tasks", - }, - { text: ".NET assemblies", link: "/managed-code-analysis" }, - { - text: "Python packaged applications", - link: "/python-packaged-applications", - }, - { text: "Android applications", link: "/android-analysis" }, - { text: "Apple applications", link: "/apple-application-analysis" }, - { text: "Desktop installations", link: "/desktop-applications" }, - { text: "Firmware", link: "/firmware-analysis" }, - { text: "iNES containers", link: "/ines-containers" }, - { text: "SQLite databases", link: "/sqlite-databases" }, - { text: "Packet captures", link: "/packet-captures" }, - { text: "Signal recordings", link: "/signal-recordings" }, - ], - }, - { - text: "Provider guides", - collapsed: true, - items: [ - { text: "IDA", link: "/ida-provider" }, - { text: "Kuna", link: "/kuna-provider" }, - { text: "Ghidra on Windows", link: "/windows-ghidra-p0" }, - { text: "Ghidra DOS analysis", link: "/ghidra-dos" }, - { text: "Ghidra raw binary", link: "/raw-binary" }, - { text: "Ghidra NativeAOT", link: "/ghidra-nativeaot" }, - { text: "Ghidra PowerPC", link: "/ghidra-ppc" }, - { text: "Ghidra GameCube DOL", link: "/ghidra-dol" }, - { text: "Ghidra semantics", link: "/ghidra-provider" }, - ], - }, - { - text: "Reference and development", - collapsed: true, - items: [ - { text: "MCP contracts", link: "/mcp-contracts" }, - { text: "Tool design", link: "/tool-design" }, - { - text: "Admission limits and recovery", - link: "/design/limits-and-recovery", - }, - { text: "Testing", link: "/testing" }, - { text: "Client verification", link: "/client-verification" }, - { - text: "Reconstruction readiness", - link: "/reconstruction-readiness", - }, - { text: "Releasing", link: "/releasing" }, + { text: "All guides", link: "/documentation" }, + { text: "Authoring conventions", link: "/development/documentation" }, + { text: "Feature specifications", link: "/specs/" }, + { text: "Architecture decisions", link: "/adr/" }, ], }, + ...documentationSections.map((section) => ({ + text: section.text, + collapsed: section.text !== "Getting started", + items: section.items.map((item) => ({ + text: item.text, + link: item.route, + })), + })), ], search: { provider: "local" }, socialLinks: [ diff --git a/docs/.vitepress/documentation.ts b/docs/.vitepress/documentation.ts new file mode 100644 index 0000000000..9636ac6b70 --- /dev/null +++ b/docs/.vitepress/documentation.ts @@ -0,0 +1,659 @@ +// Authored guide ownership: physical sources and stable public routes. +export const documentationSections = [ + { + text: "Getting started", + items: [ + { + text: "Installation and setup", + source: "getting-started/installation.md", + route: "/installation", + }, + { + text: "Client configuration and troubleshooting", + source: "getting-started/client-configuration.md", + route: "/client-configuration", + }, + { + text: "CLI and Evidence", + source: "getting-started/cli.md", + route: "/cli", + }, + { + text: "Guided MCP prompts and completion", + source: "getting-started/mcp-prompts.md", + route: "/mcp-prompts", + }, + ], + }, + { + text: "Native binaries", + items: [ + { + text: "Native binary investigation", + source: "guides/native/native-investigation.md", + route: "/native-investigation", + }, + { + text: "Go binary build metadata", + source: "guides/native/go-binaries.md", + route: "/go-binaries", + }, + { + text: "Offline binary diagnostics", + source: "guides/native/binary-diagnostics.md", + route: "/binary-diagnostics", + }, + { + text: "Recorded crash evidence", + source: "guides/native/recorded-crashes.md", + route: "/recorded-crashes", + }, + ], + }, + { + text: "JavaScript and Electron", + items: [ + { + text: "JavaScript Application Graph", + source: "guides/javascript/javascript-application-graph.md", + route: "/javascript-application-graph", + }, + { + text: "Cross-layer JavaScript application workflows", + source: "guides/javascript/javascript-application-workflows.md", + route: "/javascript-application-workflows", + }, + { + text: "JavaScript artifact reconstruction", + source: "guides/javascript/javascript-artifact-reconstruction.md", + route: "/javascript-artifact-reconstruction", + }, + { + text: "JavaScript source recovery", + source: "guides/javascript/javascript-recovery.md", + route: "/javascript-recovery", + }, + { + text: "Passive Node and Electron runtime observation", + source: "guides/javascript/javascript-runtime-observation.md", + route: "/javascript-runtime-observation", + }, + { + text: "JavaScript static/runtime reconciliation", + source: "guides/javascript/javascript-runtime-reconciliation.md", + route: "/javascript-runtime-reconciliation", + }, + { + text: "Electron file-page observation", + source: "guides/javascript/electron-observation.md", + route: "/electron-observation", + }, + ], + }, + { + text: "Browser and captured websites", + items: [ + { + text: "Website observation with CDP", + source: "guides/web/browser-observation.md", + route: "/browser-observation", + }, + { + text: "Browser scenario contract", + source: "guides/web/browser-scenario-contract.md", + route: "/browser-scenario-contract", + }, + { + text: "Website runtime source attribution", + source: "guides/web/web-runtime.md", + route: "/web-runtime", + }, + { + text: "Historical web network captures", + source: "guides/web/web-network-captures.md", + route: "/web-network-captures", + }, + { + text: "HTTP/3 capture lab", + source: "http3-capture-lab.md", + route: "/http3-capture-lab", + }, + { + text: "Captured website source locations", + source: "guides/web/web-source-location.md", + route: "/web-source-location", + }, + { + text: "Captured website module relationships", + source: "guides/web/website-module-trace.md", + route: "/website-module-trace", + }, + { + text: "Captured website scripts and static JavaScript analysis", + source: "guides/web/website-script-export.md", + route: "/website-script-export", + }, + ], + }, + { + text: "Android applications", + items: [ + { + text: "Static Android APK analysis", + source: "guides/android/android-analysis.md", + route: "/android-analysis", + }, + { + text: "Apktool resource decoding", + source: "guides/android/apktool-resource-analysis.md", + route: "/apktool-resource-analysis", + }, + { + text: "ADB device inspection and acquisition", + source: "guides/android/adb-device-analysis.md", + route: "/adb-device-analysis", + }, + { + text: "Flutter build identification", + source: "guides/android/flutter-analysis.md", + route: "/flutter-analysis", + }, + ], + }, + { + text: "Apple applications", + items: [ + { + text: "Apple Application Inventory Projection", + source: "guides/apple/apple-application-analysis.md", + route: "/apple-application-analysis", + }, + ], + }, + { + text: ".NET assemblies", + items: [ + { + text: "Managed-code analysis", + source: "guides/managed/managed-code-analysis.md", + route: "/managed-code-analysis", + }, + ], + }, + { + text: "Firmware", + items: [ + { + text: "Firmware analysis", + source: "guides/firmware/firmware-analysis.md", + route: "/firmware-analysis", + }, + ], + }, + { + text: "HarmonyOS packages", + items: [ + { + text: "Static HarmonyOS package inventory", + source: "guides/harmony/harmony-analysis.md", + route: "/harmony-analysis", + }, + ], + }, + { + text: "WebAssembly", + items: [ + { + text: "Offline WebAssembly artifacts", + source: "guides/wasm/wasm-artifacts.md", + route: "/wasm-artifacts", + }, + ], + }, + { + text: "EVM contracts", + items: [ + { + text: "Offline EVM bytecode", + source: "guides/evm/evm-bytecode.md", + route: "/evm-bytecode", + }, + ], + }, + { + text: "Databases and recorded packets", + items: [ + { + text: "SQLite database snapshots", + source: "guides/data/sqlite-databases.md", + route: "/sqlite-databases", + }, + { + text: "Recorded packet capture inspection", + source: "guides/data/packet-captures.md", + route: "/packet-captures", + }, + { + text: "Packet capture summaries", + source: "guides/data/pcap-analysis.md", + route: "/pcap-analysis", + }, + ], + }, + { + text: "Process capture and instrumentation", + items: [ + { + text: "Process capture", + source: "guides/runtime/process-capture.md", + route: "/process-capture", + }, + { + text: "Frida instrumentation", + source: "guides/runtime/frida-instrumentation.md", + route: "/frida-instrumentation", + }, + ], + }, + { + text: "Provider setup and formats", + items: [ + { + text: "Ghidra provider semantics", + source: "providers/ghidra-provider.md", + route: "/ghidra-provider", + }, + { + text: "DOS MZ and COM analysis with Ghidra", + source: "providers/ghidra-dos.md", + route: "/ghidra-dos", + }, + { + text: "Optional NativeAOT metadata recovery", + source: "providers/ghidra-nativeaot.md", + route: "/ghidra-nativeaot", + }, + { + text: "MIPS ELF analysis through Ghidra", + source: "providers/ghidra-mips.md", + route: "/ghidra-mips", + }, + { + text: "PSP static ELF analysis through Ghidra", + source: "providers/ghidra-psp.md", + route: "/ghidra-psp", + }, + { + text: "RISC-V ELF analysis through Ghidra", + source: "providers/ghidra-riscv.md", + route: "/ghidra-riscv", + }, + { + text: "Windows Ghidra P0", + source: "providers/windows-ghidra-p0.md", + route: "/windows-ghidra-p0", + }, + { + text: "Headerless raw-binary analysis with Ghidra", + source: "providers/raw-binary.md", + route: "/raw-binary", + }, + { + text: "IDA provider", + source: "providers/ida-provider.md", + route: "/ida-provider", + }, + { + text: "JEB instrumentation and analysis", + source: "providers/jeb-analysis.md", + route: "/jeb-analysis", + }, + ], + }, + { + text: "Contracts and evidence", + items: [ + { + text: "MCP runtime contracts", + source: "reference/mcp-contracts.md", + route: "/mcp-contracts", + }, + { + text: "Reconstruction obligation ledgers", + source: "reference/reconstruction-obligation-ledgers.md", + route: "/reconstruction-obligation-ledgers", + }, + ], + }, + { + text: "Development and verification", + items: [ + { + text: "Settings and recovery", + source: "design/limits-and-recovery.md", + route: "/design/limits-and-recovery", + }, + { + text: "MCP tool design", + source: "development/tool-design.md", + route: "/tool-design", + }, + { + text: "Testing REA", + source: "development/testing.md", + route: "/testing", + }, + { + text: "Client and agent verification", + source: "development/client-verification.md", + route: "/client-verification", + }, + { + text: "Reconstruction readiness conformance", + source: "development/reconstruction-readiness.md", + route: "/reconstruction-readiness", + }, + { + text: "Releasing REA", + source: "development/releasing.md", + route: "/releasing", + }, + { + text: "REA roadmap", + source: "development/roadmap.md", + route: "/roadmap", + }, + ], + }, + { + text: "Release migrations", + items: [ + { + text: "Upgrading from REA 6.1 to 6.2", + source: "migrations/migration-6.2.md", + route: "/migration-6.2", + }, + ], + }, + { + text: "Signal recordings", + items: [ + { + text: "Recorded signal inspection", + source: "guides/signals/signal-recordings.md", + route: "/signal-recordings", + }, + ], + }, + { + text: "Additional guides", + items: [ + { + text: "Offline API contract inventory", + source: "api-contracts.md", + route: "/api-contracts", + }, + { + text: "Bun standalone executables", + source: "bun-standalone.md", + route: "/bun-standalone", + }, + { + text: "Capability composition and ownership", + source: "capability-composition.md", + route: "/capability-composition", + }, + { + text: ".NET single-file bundles", + source: "dotnet-bundles.md", + route: "/dotnet-bundles", + }, + { + text: "ESP flash images", + source: "esp-flash-image.md", + route: "/esp-flash-image", + }, + { + text: "U-Boot FIT images", + source: "fit-images.md", + route: "/fit-images", + }, + { + text: "Measure MCP catalog costs in Codex", + source: "mcp-catalog-evaluation.md", + route: "/mcp-catalog-evaluation", + }, + { + text: "Offline Playwright traces", + source: "playwright-traces.md", + route: "/playwright-traces", + }, + { + text: "Testing lanes and provider verification", + source: "testing-lanes.md", + route: "/testing-lanes", + }, + { + text: "UF2 and DfuSe images", + source: "uf2-dfuse.md", + route: "/uf2-dfuse", + }, + { + text: "WeChat mini-program wxapkg archives", + source: "wxapkg.md", + route: "/wxapkg", + }, + { + text: "Binary hardening", + source: "binary-hardening.md", + route: "/binary-hardening", + }, + { + text: "Desktop application capture", + source: "desktop-application-capture.md", + route: "/desktop-application-capture", + }, + { + text: "Desktop application export", + source: "desktop-application-export.md", + route: "/desktop-application-export", + }, + { + text: "Desktop application state", + source: "desktop-application-state.md", + route: "/desktop-application-state", + }, + { + text: "Desktop application tasks", + source: "desktop-application-tasks.md", + route: "/desktop-application-tasks", + }, + { + text: "Desktop build computer", + source: "desktop-build-computer.md", + route: "/desktop-build-computer", + }, + { + text: "Desktop build witness", + source: "desktop-build-witness.md", + route: "/desktop-build-witness", + }, + { + text: "iNES containers", + source: "ines-containers.md", + route: "/ines-containers", + }, + { + text: "Build RL environments with REA", + source: "agent-environments-quickstart.md", + route: "/agent-environments-quickstart", + }, + { + text: "Agent environments from recorded captures", + source: "agent-environments.md", + route: "/agent-environments", + }, + { + text: "APK signature schemes", + source: "apk-signature.md", + route: "/apk-signature", + }, + { + text: "Apple IPS crash reports", + source: "apple-ips.md", + route: "/apple-ips", + }, + { + text: "Static analysis in GitHub Actions", + source: "ci-analysis.md", + route: "/ci-analysis", + }, + { + text: "Desktop installation directories", + source: "desktop-applications.md", + route: "/desktop-applications", + }, + { + text: "Flattened devicetree inspection", + source: "devicetree.md", + route: "/devicetree", + }, + { + text: "Electron fuse state", + source: "electron-fuses.md", + route: "/electron-fuses", + }, + { text: "ELF note tables", source: "elf-notes.md", route: "/elf-notes" }, + { + text: "Erlang BEAM files", + source: "erlang-beam.md", + route: "/erlang-beam", + }, + { text: "ESP images", source: "esp-image.md", route: "/esp-image" }, + { + text: "GameCube DOL analysis through Ghidra", + source: "ghidra-dol.md", + route: "/ghidra-dol", + }, + { + text: "PowerPC ELF analysis through Ghidra", + source: "ghidra-ppc.md", + route: "/ghidra-ppc", + }, + { + text: "Godot PCK archives", + source: "godot-pck.md", + route: "/godot-pck", + }, + { + text: "Static CUDA and HIP code objects", + source: "gpu-code-objects.md", + route: "/gpu-code-objects", + }, + { + text: "Intel HEX and S-record images", + source: "hex-records.md", + route: "/hex-records", + }, + { + text: "IL2CPP metadata", + source: "il2cpp-metadata.md", + route: "/il2cpp-metadata", + }, + { + text: "Static JavaScript route inventory", + source: "javascript-route-inventory.md", + route: "/javascript-route-inventory", + }, + { + text: "Linux kernel modules", + source: "kernel-modules.md", + route: "/kernel-modules", + }, + { + text: "Kuna provider", + source: "kuna-provider.md", + route: "/kuna-provider", + }, + { + text: "Model file declarations", + source: "model-artifacts.md", + route: "/model-artifacts", + }, + { + text: "Match native functions across builds", + source: "native-function-matching.md", + route: "/native-function-matching", + }, + { + text: "Node.js single executable applications", + source: "node-sea.md", + route: "/node-sea", + }, + { + text: "PE Authenticode certificate tables", + source: "pe-certificates.md", + route: "/pe-certificates", + }, + { + text: "PE import and delay-load tables", + source: "pe-imports.md", + route: "/pe-imports", + }, + { + text: "PE TLS callbacks and load configuration", + source: "pe-load-config.md", + route: "/pe-load-config", + }, + { + text: "Python packaged applications", + source: "python-packaged-applications.md", + route: "/python-packaged-applications", + }, + { + text: "CPython pyc inspection", + source: "python-pyc.md", + route: "/python-pyc", + }, + { + text: "Reconstruction coverage", + source: "reconstruction-coverage.md", + route: "/reconstruction-coverage", + }, + { + text: "Release exposure", + source: "release-exposure.md", + route: "/release-exposure", + }, + { + text: "Reverse-engineering tool integrations", + source: "reverse-engineering-tools.md", + route: "/reverse-engineering-tools", + }, + { + text: "Rust binary dependency metadata", + source: "rust-binaries.md", + route: "/rust-binaries", + }, + { + text: "U-Boot legacy uImage containers", + source: "uimage.md", + route: "/uimage", + }, + { + text: "UnityFS AssetBundle archives", + source: "unityfs.md", + route: "/unityfs", + }, + ], + }, +]; + +export const documentationRewrites = Object.fromEntries( + documentationSections.flatMap((section) => + section.items.flatMap((item) => { + const target = `${item.route.slice(1)}.md`; + return item.source === target ? [] : [[item.source, target]]; + }), + ), +); diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000000..a092412d4d --- /dev/null +++ b/docs/README.md @@ -0,0 +1,196 @@ +# Documentation directory + +REA documentation is organized by reader task and document owner. This directory +links every authored guide; the [site](https://rea.tools/) also serves separate +product pages. The VitePress documentation source is this `docs/` tree. + +## Getting started + +- [Installation and setup](getting-started/installation.md) +- [Client configuration and troubleshooting](getting-started/client-configuration.md) +- [CLI and Evidence](getting-started/cli.md) +- [Guided MCP prompts and completion](getting-started/mcp-prompts.md) + +## Native binaries + +- [Native binary investigation](guides/native/native-investigation.md) +- [Go binary build metadata](guides/native/go-binaries.md) +- [Offline binary diagnostics](guides/native/binary-diagnostics.md) +- [Recorded crash evidence](guides/native/recorded-crashes.md) + +## JavaScript and Electron + +- [JavaScript Application Graph](guides/javascript/javascript-application-graph.md) +- [Cross-layer JavaScript application workflows](guides/javascript/javascript-application-workflows.md) +- [JavaScript artifact reconstruction](guides/javascript/javascript-artifact-reconstruction.md) +- [JavaScript source recovery](guides/javascript/javascript-recovery.md) +- [Passive Node and Electron runtime observation](guides/javascript/javascript-runtime-observation.md) +- [JavaScript static/runtime reconciliation](guides/javascript/javascript-runtime-reconciliation.md) +- [Electron file-page observation](guides/javascript/electron-observation.md) + +## Browser and captured websites + +- [Website observation with CDP](guides/web/browser-observation.md) +- [Browser scenario contract](guides/web/browser-scenario-contract.md) +- [Website runtime source attribution](guides/web/web-runtime.md) +- [Historical web network captures](guides/web/web-network-captures.md) +- [HTTP/3 capture lab](http3-capture-lab.md) +- [Captured website source locations](guides/web/web-source-location.md) +- [Captured website module relationships](guides/web/website-module-trace.md) +- [Captured website scripts and static JavaScript analysis](guides/web/website-script-export.md) + +## Android applications + +- [Static Android APK analysis](guides/android/android-analysis.md) +- [Apktool resource decoding](guides/android/apktool-resource-analysis.md) +- [ADB device inspection and acquisition](guides/android/adb-device-analysis.md) +- [Flutter build identification](guides/android/flutter-analysis.md) + +## Apple applications + +- [Apple Application Inventory Projection](guides/apple/apple-application-analysis.md) + +## .NET assemblies + +- [Managed-code analysis](guides/managed/managed-code-analysis.md) + +## Firmware + +- [Firmware analysis](guides/firmware/firmware-analysis.md) + +## HarmonyOS packages + +- [Static HarmonyOS package inventory](guides/harmony/harmony-analysis.md) + +## WebAssembly + +- [Offline WebAssembly artifacts](guides/wasm/wasm-artifacts.md) + +## EVM contracts + +- [Offline EVM bytecode](guides/evm/evm-bytecode.md) + +## Databases and recorded packets + +- [SQLite database snapshots](guides/data/sqlite-databases.md) +- [Recorded packet capture inspection](guides/data/packet-captures.md) +- [Packet capture summaries](guides/data/pcap-analysis.md) + +## Process capture and instrumentation + +- [Process capture](guides/runtime/process-capture.md) +- [Frida instrumentation](guides/runtime/frida-instrumentation.md) + +## Signal recordings + +- [Recorded signal inspection](guides/signals/signal-recordings.md) + +## Provider setup and formats + +- [Ghidra provider semantics](providers/ghidra-provider.md) +- [DOS MZ and COM analysis with Ghidra](providers/ghidra-dos.md) +- [Optional NativeAOT metadata recovery](providers/ghidra-nativeaot.md) +- [MIPS ELF analysis through Ghidra](providers/ghidra-mips.md) +- [PSP static ELF analysis through Ghidra](providers/ghidra-psp.md) +- [RISC-V ELF analysis through Ghidra](providers/ghidra-riscv.md) +- [Windows Ghidra P0](providers/windows-ghidra-p0.md) +- [Headerless raw-binary analysis with Ghidra](providers/raw-binary.md) +- [IDA provider](providers/ida-provider.md) +- [JEB instrumentation and analysis](providers/jeb-analysis.md) + +## Contracts and evidence + +- [MCP runtime contracts](reference/mcp-contracts.md) +- [Reconstruction obligation ledgers](reference/reconstruction-obligation-ledgers.md) + +## Development and verification + +- [Settings and recovery](design/limits-and-recovery.md) +- [MCP tool design](development/tool-design.md) +- [Testing REA](development/testing.md) +- [Client and agent verification](development/client-verification.md) +- [Reconstruction readiness conformance](development/reconstruction-readiness.md) +- [Releasing REA](development/releasing.md) +- [REA roadmap](development/roadmap.md) + +## Release migrations + +- [Upgrading from REA 6.1 to 6.2](migrations/migration-6.2.md) + +## Architecture and authoring + +- [Architecture map](https://github.com/morluto/rea/blob/main/docs/architecture/architecture.mermaid) +- [Documentation ownership and contribution model](development/documentation.md) +- [Feature specifications](specs/README.md) +- [Architectural decision records](adr/README.md) + +## Machine-owned documentation + +| Path | Owner and update command | +| ----------------------------------------- | ----------------------------------------------------------------------------------------- | +| `public/product-catalog.json` | Ignored contract projection; `npm run docs:generate` | +| `verification/managed-conformance-*.json` | Ignored managed conformance projections; `npm run evidence:generate` | +| `error-contract.schema.json` | Tracked schema from contracts; `npm run docs:error-schema` | +| `user-visible-errors.csv` | Authored error classification inventory; preserve its schema | +| `releases/.contributions.json` | Durable release contribution record; follow the [release guide](development/releasing.md) | +| `assets/` | Shared authored images referenced by guides | + +Do not edit generated catalogs or evidence to change product facts. Change their +canonical contracts or producer and regenerate. Historical release records retain +the exact commit messages and references from the release they describe. + +## Additional guides + +- [Offline API contract inventory](api-contracts.md) +- [Bun standalone executables](bun-standalone.md) +- [Capability composition and ownership](capability-composition.md) +- [.NET single-file bundles](dotnet-bundles.md) +- [ESP flash images](esp-flash-image.md) +- [U-Boot FIT images](fit-images.md) +- [Measure MCP catalog costs in Codex](mcp-catalog-evaluation.md) +- [Offline Playwright traces](playwright-traces.md) +- [Testing lanes and provider verification](testing-lanes.md) +- [UF2 and DfuSe images](uf2-dfuse.md) +- [WeChat mini-program wxapkg archives](wxapkg.md) +- [Binary hardening](binary-hardening.md) +- [Desktop application capture](desktop-application-capture.md) +- [Desktop application export](desktop-application-export.md) +- [Desktop application state](desktop-application-state.md) +- [Desktop application tasks](desktop-application-tasks.md) +- [Desktop build computer](desktop-build-computer.md) +- [Desktop build witness](desktop-build-witness.md) +- [iNES containers](ines-containers.md) +- [Build RL environments with REA](agent-environments-quickstart.md) +- [Agent environments from recorded captures](agent-environments.md) +- [APK signature schemes](apk-signature.md) +- [Apple IPS crash reports](apple-ips.md) +- [Static analysis in GitHub Actions](ci-analysis.md) +- [Desktop installation directories](desktop-applications.md) +- [Flattened devicetree inspection](devicetree.md) +- [Electron fuse state](electron-fuses.md) +- [ELF note tables](elf-notes.md) +- [Erlang BEAM files](erlang-beam.md) +- [ESP images](esp-image.md) +- [GameCube DOL analysis through Ghidra](ghidra-dol.md) +- [PowerPC ELF analysis through Ghidra](ghidra-ppc.md) +- [Godot PCK archives](godot-pck.md) +- [Static CUDA and HIP code objects](gpu-code-objects.md) +- [Intel HEX and S-record images](hex-records.md) +- [IL2CPP metadata](il2cpp-metadata.md) +- [Static JavaScript route inventory](javascript-route-inventory.md) +- [Linux kernel modules](kernel-modules.md) +- [Kuna provider](kuna-provider.md) +- [Model file declarations](model-artifacts.md) +- [Match native functions across builds](native-function-matching.md) +- [Node.js single executable applications](node-sea.md) +- [PE Authenticode certificate tables](pe-certificates.md) +- [PE import and delay-load tables](pe-imports.md) +- [PE TLS callbacks and load configuration](pe-load-config.md) +- [Python packaged applications](python-packaged-applications.md) +- [CPython pyc inspection](python-pyc.md) +- [Reconstruction coverage](reconstruction-coverage.md) +- [Release exposure](release-exposure.md) +- [Reverse-engineering tool integrations](reverse-engineering-tools.md) +- [Rust binary dependency metadata](rust-binaries.md) +- [U-Boot legacy uImage containers](uimage.md) +- [UnityFS AssetBundle archives](unityfs.md) diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 0000000000..16c75811f4 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,9 @@ +# Architectural decision records + +Store one actual decision per `docs/adr//.md`, without a +global sequence number. Use meaningful names and the Michael Nygard sections: +Status, Context, Decision and Consequences. Link each decision from its owning +feature specification. + +See [the authoring guide](../development/documentation.md#feature-lifecycle) for +the distinction between specifications, roadmaps and isolated decisions. diff --git a/docs/agent-environments-quickstart.md b/docs/agent-environments-quickstart.md index 4e64305714..aca00484c3 100644 --- a/docs/agent-environments-quickstart.md +++ b/docs/agent-environments-quickstart.md @@ -11,14 +11,14 @@ with your RL framework to set task goals, check rewards, and run training. ## Before you start -Install [REA and its agent integration](installation.md). To record an app, +Install [REA and its agent integration](getting-started/installation.md). To record an app, you also need a running test instance and a Chrome-family browser. Choose an instance where the agent is allowed to create, edit, and delete data. REA records the app you provide; you start it and prepare its test data. To run the exported environment, install the [AgentEnv framework](https://github.com/scaleapi/agentenv-framework#install) -and start Docker. REA's [compatibility check](testing.md#exported-environment-bundles) +and start Docker. REA's [compatibility check](development/testing.md#exported-environment-bundles) uses AgentEnv version `0.9.1431` with Python 3.12. There are two ways to build an environment: ask your agent to record the app, @@ -52,7 +52,7 @@ work and which need more evidence. The app is used during recording; the exported environment runs from the saved data. After your agent exports the folder, skip to -[run the environment](#3-run-the-environment). The next section is an +[run the environment](#run-the-environment). The next section is an alternative for an existing recording. ## 2. Or build from a saved recording @@ -91,6 +91,8 @@ A successful build means the folder was created. Check the report to confirm it supports the operations your tasks need. `coverage: null` means the build could not prepare state for this environment and it will use replay. + + ## 3. Run the environment Whether your agent exported the folder or you built it from a HAR, run it diff --git a/docs/agent-environments.md b/docs/agent-environments.md index be11df05db..f35a51fedd 100644 --- a/docs/agent-environments.md +++ b/docs/agent-environments.md @@ -30,7 +30,7 @@ rea environment-serve manifest.json --evidence capture.json --port 8765 ``` The first command inspects the capture; the second projects the -[environment manifest](web-network-captures.md#environment-manifest) view from +[environment manifest](guides/web/web-network-captures.md#environment-manifest) view from it, where `view.json` wraps the capture Evidence as the inline source: ```json @@ -163,7 +163,7 @@ at info level; the report is the only output on stdout: `coverage` is the state's read views and write plans summarized, and `null` when the state is unavailable and the bundle serves replay. The -[environment coverage](web-network-captures.md#environment-coverage) view +[environment coverage](guides/web/web-network-captures.md#environment-coverage) view reports the same summary for any capture without building, so an agent can check it after each recording. The view summarizes the state as derived; it does not apply the serve-time admission checks, so for an environment whose @@ -235,7 +235,7 @@ name `export_environment_bundle`; the others name their stage as ## From a browser scenario The capture can also come from REA itself: a -[browser scenario](browser-scenario-contract.md) records a scripted session +[browser scenario](guides/web/browser-scenario-contract.md) records a scripted session in a browser REA launches, and its Evidence is a manifest parent like a HAR inspection. Select the `network` event family and response bodies. A replayed reply carries up to 256 KiB of a recorded body, but the scenario keeps at most @@ -321,7 +321,7 @@ declared secret replaced by its redaction marker (a body that redaction had to empty replays no body). To record writes, sign in through the UI and then send the application's API -calls with [request steps](browser-scenario-contract.md#request-steps): each +calls with [request steps](guides/web/browser-scenario-contract.md#request-steps): each sends one request to the start origin through the page's own `fetch`, with the session's cookies, and a header can carry the CSRF cookie or stored token the application's own requests send. A write is served only when the capture shows @@ -481,7 +481,7 @@ captured text without changing the tool names or schema fields clients use. Replay answers each call from one recording, so the answers reflect whichever moment of the capture recorded them. The -[environment state](web-network-captures.md#environment-state) view extracts +[environment state](guides/web/web-network-captures.md#environment-state) view extracts instead the data the capture's reads observed before the first related write: entity types with an identity field, entities with their recorded values, documents for reads that are not entities, and a render plan per read tool, @@ -503,14 +503,14 @@ see [Serve from the state](#serve-from-the-state). `--host` and `--port` default to `MCP_HOST` and `MCP_PORT` from the environment, then to `127.0.0.1` and an ephemeral port, exactly as -[`rea mcp --http`](cli.md#serve-mcp-over-streamable-http) resolves them. The +[`rea mcp --http`](getting-started/cli.md#serve-mcp-over-streamable-http) resolves them. The bound MCP URL is written to stderr as one JSON log line (`"msg":"MCP transport listening"` with `url`); `REA_LOG_LEVEL` overrides that log level. `ENVIRONMENT_NAME` renames the served environment; otherwise the manifest's `name` is used, and a set but blank value is refused as `configuration_invalid` before binding rather than silently falling back. The listener serves one MCP client at a time with the same session, `Host` header -and shutdown contract as [transports](mcp-contracts.md#transports) describes. +and shutdown contract as [transports](reference/mcp-contracts.md#transports) describes. It runs until SIGINT or SIGTERM, then closes the listener, prints one `{"status":"stopped", …}` document whose `state` member says whether it served from the state, and exits `130` or `143`. @@ -585,7 +585,7 @@ names the method and path template, then what the tool returns, one level deep: `returns a JSON object with fields …`, `returns a JSON array of objects with fields …` or `returns a JSON object whose lists objects with fields …`, where a list is an array of objects found by the -[environment state](web-network-captures.md#environment-state)'s list rule (the +[environment state](guides/web/web-network-captures.md#environment-state)'s list rule (the root array, or the one array of objects an envelope carries). Field and member names are written as JSON strings, so a captured name cannot read as more of the description; a member name too long to show whole is a quoted prefix marked @@ -641,7 +641,7 @@ rea environment-serve manifest.json --evidence capture.json ``` Before it binds, the server derives the -[environment state](web-network-captures.md#environment-state) the capture +[environment state](guides/web/web-network-captures.md#environment-state) the capture observed: it runs the `environment-state` view on the capture in-process with the manifest's `template_rules` in the same order, exactly as `inspect-analysis-view` projects it from the same capture, so the state @@ -733,7 +733,7 @@ query model may describe supported paging; an empty later page cannot start returning earlier entities merely because it was empty in the capture. `POST`, `PUT`, `PATCH` and `DELETE` tools are served only when the state's -[write plan](web-network-captures.md#environment-state) for them is served: +[write plan](guides/web/web-network-captures.md#environment-state) for them is served: the capture's later reads confirmed the effect the plan models. Other write tools are not served, so no call claims a change it does not make; each one's plan states why in its `reason`. A served write changes the live state, which @@ -1187,7 +1187,7 @@ read, not that the write had no other effect. ## Compatibility verification `npm run verify:agentenv:environment` is the hosted `real-agentenv` lane -described in [testing](testing.md#exported-environment-bundles). It projects +described in [testing](development/testing.md#exported-environment-bundles). It projects the Grafana-shaped source-owned capture through the built CLI, packs the checkout, exports a bundle with `--package-tarball`, runs it with the pinned `agentenv-framework` release on the runner's Docker and requires the smoke task diff --git a/docs/api-contracts.md b/docs/api-contracts.md new file mode 100644 index 0000000000..bc9e752db8 --- /dev/null +++ b/docs/api-contracts.md @@ -0,0 +1,178 @@ +# Offline API contract inventory + +`inspect_api_contract` / `rea inspect-api-contract` supports local OpenAPI JSON +and GraphQL SDL (`format: "graphql-sdl"`). Neither profile contacts endpoints. + +## OpenAPI JSON + +The default profile reads one local OpenAPI +3.0, 3.1 or 3.2 JSON document. It inventories inline operation slots under +`paths`; the facts describe declarations, not implemented, reachable or +authorized runtime behavior. YAML, AsyncAPI, callbacks/webhooks, schema graphs, +contract diffs and traffic correlation are outside this version. + +The OpenAPI importer never resolves references, opens sibling documents, evaluates +examples or contacts declared servers. OpenAPI 3.2 `query` and +`additionalOperations` follow the [Path Item contract](https://spec.openapis.org/oas/v3.2.1.html#path-item-object), +including case-sensitive additional HTTP method spelling. + +## Select a useful answer + +MCP defaults to a summary retaining the complete inventory. An optional view +returns selected facts immediately, taking precedence over `detail`: + +```json +{ + "path": "/artifacts/openapi.json", + "view": { + "kind": "api-contract", + "collection": "operations", + "methods": ["GET"], + "limit": 20 + } +} +``` + +`operations` supports exact `operation_ids`, `operation_keys`, `methods`, +`paths`, `declared_ids` and `states` filters. Filters combine by intersection; +empty arrays select nothing. `fields` chooses columns; identity, source pointer, +state and unknown reason remain. The other collections are +`path-item-references` (optional `reference_ids`) and `diagnostics` (optional +`codes`), also with fields and offset/limit. The default page contains 64 rows. + +Query a retained result with `inspect_analysis_view`, using the same view and +`source: {"kind":"retained-evidence","evidence_id":"..."}`. Copy the complete +parent ID from `normalized_result.parent_evidence_id`, not the summary's own ID. +These pages use immutable retained facts without rereading the specification. +Complete Evidence remains available through `get_evidence_bundle`. + +CLI defaults to complete Evidence. For a selected answer, save that complete +Evidence in an absent bundle path: + +```sh +rea inspect-api-contract ./openapi.json --detail summary \ + --view '{"kind":"api-contract","collection":"operations","methods":["GET"],"limit":20}' \ + --evidence_output ./contract.evidence.json --json +``` + +Later `rea inspect-analysis-view` can read that bundle through an exact +`evidence-file` source selection, even after the original specification is +unavailable. Publication never overwrites an existing file. CLI preserves its +original path spelling in `caller_input_path`; analysis locations and MCP inputs +use absolute filesystem paths. + +## Identity, provenance and coverage + +Each operation has an artifact-scoped `operation_id`, an exact method/path +`operation_key`, an optional `declared_id` from `operationId`, and its original +JSON Pointer. Keys can correlate exact declarations in a caller-selected pair +of revisions; they do not establish equivalence across unrelated APIs, renamed +path templates or runtime handlers. Declared IDs are not assumed unique. +The artifact includes the verified byte length, SHA-256 and selected path. + +Malformed siblings retain useful declarations plus diagnostics. A Path Item +`$ref`, including a local fragment, remains unresolved; adjacent inline slots +are unknown because referenced fields may overlap. Missing or malformed paths +are explicit, and missing required OpenAPI 3.0 paths does not become a complete +empty inventory. `source_coverage` is limited to inline paths and records that +reference resolution and full schema validation were not performed. Page +exhaustion describes the selected rows and never upgrades source coverage. + +Structured transport userinfo credentials are excluded from reference URIs; +`credential_redacted` reports that change, and an ambiguous credential span leaves +the displayed reference unknown. Ordinary reference query/fragment text and +names are retained. Original bytes remain in the caller's artifact; complete +inventory Evidence does not embed the entire source JSON. Duplicate JSON members +use the shared parser's last-value-wins behavior, recorded in source coverage. + +If decoding or later validation fails after a verified read, the typed error +retains a compact `file-snapshot-hash` observation with the exact path, digest +and byte length. Syntax diagnostics do not quote source JSON. Cancellation +does not replace a parser failure that has already completed. + +## Resource bounds and verification + +The file-byte allowance and conservative expanded-facts allowance are each +32 MiB resource policies, not OpenAPI format limits. `max_file_bytes` can lower +the input allowance. Native JSON parsing additionally checks available heap +against the shared representation-expansion estimate before decoding. The +shared JSON nesting limit is 1,000, including uninterpreted schema regions. +Capacity refusals are typed; reply paging does not increase parser allowances. + +The committed [Petstore fixture](https://github.com/morluto/rea/blob/b7c8caedd4166e6032c49024cdc6fb1154f1a2df/tests/fixtures/openapi/README.md) is an +original document from the published application, replayed offline through +public CLI/MCP tests. These check advertised schemas, complete/selected parity, +retained paging, partial references, zero requests to a loopback reference +server and unchanged source bytes. They do not establish runtime endpoint +behavior or full OpenAPI conformance. + +## GraphQL SDL and operation documents + +Select `format: "graphql-sdl"` for SDL. Optionally supply `operations_path` to +one local document containing operations and fragments. Both MCP paths are +absolute; CLI accepts relative spellings: + +```sh +rea inspect-api-contract ./schema.graphql --contractFormat graphql-sdl \ + --operationsPath ./queries.graphql --detail summary \ + --view '{"kind":"graphql-contract","collection":"references","limit":20}' \ + --evidence_output ./graphql.evidence.json --json +``` + +Collections are `types`, `fields`, `definitions`, `references`, `fragment-spreads` +and `diagnostics`. Each supports `fields`, `offset` and `limit` (default 64). +Exact batch filters include type/field/definition/reference IDs, type names, +`field_keys` such as `Query.viewer`, owning type names, binding/spread states +and diagnostic codes. Filters intersect; an empty array selects nothing. +Identity, source and applicable uncertainty fields always remain. Retained pages +and bundle-based queries use the same `graphql-contract` views without rereading +SDL or rebuilding its schema model. + +Types and fields retain individual declaration occurrences, including extensions, +list/non-null wrappers, implemented interfaces and union member names. Operation +and fragment definitions own field references with aliases and parent-reference +IDs. Each reference links to a supplied SDL field occurrence, remains `model-only` +for a library-provided field such as `__typename`, or records an exact unknown +reason. Rejected schema construction keeps declarations but cannot supply trusted +bindings. Missing or duplicate fragment names remain explicit. Fragments are +visited once in their declared type scope and linked by spread edges; cycles +are never recursively expanded. + +The versioned `graphql-sdl-field-bindings-v1` profile records the GraphQL library +version. It performs syntax parsing and schema construction for field resolution; +it does not establish whole-schema/operation validity, spread applicability, +reachability, authorization or endpoint behavior. Arguments, defaults, +descriptions, directive behavior, enum value inventory, introspection JSON, +federation composition and version comparison remain outside this profile. + +Every fact has a document role, SHA-256, UTF-16 start/end offsets and parser token +line/column. Offsets address decoded UTF-8 text, including an optional BOM, rather +than byte positions. The two files are independently verified snapshots, not an +atomic pair. Fact IDs identify source occurrences; `field_key` separately supports +exact name-based joins and does not assert equivalence across revisions. CLI +retains its auxiliary path spelling in `caller_input_paths.operations_path`, +outside semantic identity, as well as its primary `caller_input_path`. + +Malformed operation text retains the SDL inventory with partial coverage and a +source-located diagnostic. A primary parser refusal or later resource/cancellation +failure retains known source identities and available interpreted Evidence; +syntax diagnostics do not quote source text. Complete Evidence contains facts, +not original document bytes. An exhausted page never upgrades source coverage. + +Each file has the same configurable maximum of 32 MiB. Conservatively charged +facts share a 32 MiB allowance; text/AST allocation is admitted against available +heap. The parser admits at most 262,144 lexical tokens and 65,536 comments per +document; subsequent iterative AST admission limits traversal depth to 128. +These are REA resource policies, not GraphQL language limits. Very deep input can +be refused by the parser before AST admission. Selected replies reduce transport +and retained queries reuse immutable facts; they do not avoid the initial parse. + +The committed SWAPI SDL is an unchanged public application fixture with provenance +in `tests/fixtures/graphql/README.md`. It supplies 53 types and 243 fields; +verification does not execute its API. + +Large complete inventories can exceed the default MCP response budget even when +selected pages fit. Use `export_evidence_bundle` with a caller-selected local +path for complete retained Evidence, or CLI complete JSON/bundle publication. +A transport refusal does not erase a completed retained analysis. Narrow pages +and source coverage remain available on the same continuing connection. diff --git a/docs/apple-ips.md b/docs/apple-ips.md index 796256d7f0..375427fddd 100644 --- a/docs/apple-ips.md +++ b/docs/apple-ips.md @@ -87,7 +87,7 @@ that matches none of those rules is `unclassified`. Opened Mach-O and dSYM UUID joins remain unobserved. Symbol names printed inside the report are recorded claims and are not verified. Linux ELF cores -use the separate [recorded crash](recorded-crashes.md) workflow. +use the separate [recorded crash](guides/native/recorded-crashes.md) workflow. ## Limits and failures diff --git a/docs/architecture.mermaid b/docs/architecture/architecture.mermaid similarity index 100% rename from docs/architecture.mermaid rename to docs/architecture/architecture.mermaid diff --git a/docs/ci-analysis.md b/docs/ci-analysis.md index 153f2d0db8..bec6951d00 100644 --- a/docs/ci-analysis.md +++ b/docs/ci-analysis.md @@ -65,14 +65,14 @@ cancellation can still prevent upload. that analysis was complete. A successful operation can retain partial analysis, unknowns, and limitations. Exit zero is not a security verdict, proof of complete coverage, or a policy gate for those findings. Inspect the saved Evidence and -coverage before making such a decision. See [output and exit status](cli.md#output-and-exit-status). +coverage before making such a decision. See [output and exit status](getting-started/cli.md#output-and-exit-status). A nonzero exit can represent invalid input, unavailable tooling, an unsupported host/capability, or another operation failure. Keep the original JSON error (when emitted), stderr, and exit code, and use their category/message to diagnose it. An early startup failure may leave `analysis.json` empty; do not parse that as successful analysis. Select a supported operation or explicitly provision its -[documented prerequisites](installation.md); do not silently switch providers, +[documented prerequisites](getting-started/installation.md); do not silently switch providers, execute the artifact, or ignore the failure to obtain a green job. To exercise the failure path, temporarily set `ARTIFACT_PATH` to diff --git a/docs/design/limits-and-recovery.md b/docs/design/limits-and-recovery.md index c6ea9dde98..d34917ba7f 100644 --- a/docs/design/limits-and-recovery.md +++ b/docs/design/limits-and-recovery.md @@ -3,7 +3,7 @@ This guide explains the runtime settings design for maintainers and developers: who owns each value, when an update takes effect, how providers enforce it, and how to extend the system. The public tool contract is in -[MCP contracts](../mcp-contracts.md#editable-analysis-settings); recorded runtime +[MCP contracts](../reference/mcp-contracts.md#editable-analysis-settings); recorded runtime checks and their limitations are in the [verification report](../verification/runtime-analysis-settings.md). @@ -125,6 +125,19 @@ raise REA's reserved string limit or Node's string ceiling. Like the regex deadline, it is in `SELECTED_REGISTRATION_SETTINGS`, so setup forwards an explicit selection into the client registration. +Windows ChangeTime admission waits out a fresh timestamp before recording a +baseline. `REA_WINDOWS_CHANGE_TIME_DRIFT_MS` (default 2000 ms, floor 1 ms, +maximum 60000 ms), `REA_WINDOWS_CHANGE_TIME_SETTLE_POLL_MS` (default 25 ms, +floor 1 ms, maximum 1000 ms), and `REA_WINDOWS_CHANGE_TIME_SETTLE_MAX_POLLS` +(default 40, floor 1, maximum 1000) are launch settings. Unset keeps those +defaults. An invalid value is a configuration error: it names the setting, the +rejected value, source `setting`, the default, the floor, and the maximum, and +says to set a positive decimal integer in the environment that starts REA and +then restart REA. There is no runtime override. Setup forwards an explicit +selection. When the poll budget runs out, admission keeps the last observation +instead of failing the readable file. Raising a setting only changes how long +that wait can last. It does not change Windows ChangeTime itself. + A per-call budget can have a runtime default without having an environment setting. JavaScript `max_heap_mb` (CLI `--max-heap-mb`) is an integer from 128 to 16384 MiB, default 1024. It @@ -386,7 +399,7 @@ application points, and resulting behavior. Check that adapters deliver the canonical recovery guidance, with independent assertions for its underlying facts. Avoid freezing prose or deriving all expected values from the same registry used by the implementation; either can hide the wrong failure. -See [test design](../testing.md#test-design-and-pruning) for the assertion policy. +See [test design](../development/testing.md#test-design-and-pruning) for the assertion policy. Runtime checks cover atomic validation, stale writes, reset, runtime isolation, accepted-operation snapshots, and explicit argument precedence. Public MCP diff --git a/docs/desktop-application-capture.md b/docs/desktop-application-capture.md index b9f56a3db9..f68d8da4e8 100644 --- a/docs/desktop-application-capture.md +++ b/docs/desktop-application-capture.md @@ -52,7 +52,10 @@ application is already open, or if the window list cannot be read. A launch beside a running instance can hand off to it, and that instance's unsaved state would be missing from the baseline. It then records the starting state, launches the application in its own process group with a run token in its -environment, and reads commands from stdin, one per line: +environment, and reads commands from stdin, one per line. The application gets +REA's environment without the `AGENTENV_*` variables an agent environment sets, +such as its control token: an app the exported computer launches never has +them, and an app captured inside a computer must not hold a credential to it. | Line | Effect | | ------------------------------------------- | ----------------------------------------------------------- | @@ -65,6 +68,15 @@ session. A name that repeats an earlier checkpoint, or is one of the capture's own names (`baseline`, `stop`, `time-limit`, `exit`), is refused with a progress line on stderr; no checkpoint is taken and the session continues. +Name the checkpoints a later step reads by what they mark. The +[export](desktop-application-export.md) seeds the bundle from the baseline +unless `--seed-checkpoint` names another, and a +[task](desktop-application-tasks.md) is checked against the last checkpoint +unless `--end-checkpoint` names another. So one capture can build a seed and +then the change: launch the app, create and save the document, send +`checkpoint seed`, make the edit and save it, send `checkpoint end`, then +`stop`. + The session also finishes when the launched process has exited and the application's processes and windows are all gone in two consecutive samples, or when `--max-duration-seconds` passes. Before deciding the processes are @@ -72,8 +84,18 @@ gone, the capture looks for any process still carrying the run token, so a worker whose launcher exited before it was sampled keeps the session open. A sample whose window list could not be read is not taken as an empty one; the limitations say how many there were. One queued line is handled between -samples, and reading stdin pauses while 16 lines wait. Progress lines on stderr say when -the window has opened and when each checkpoint is taken. +samples, and reading stdin pauses while 16 lines wait. + +Progress lines on stderr are JSON, `{"rea_progress": {"phase": +"desktop_capture", "message": ...}}`. Those a driver waits on also carry an +`event`: `{"kind": "window_open"}` once the window opens; `{"kind": +"checkpoint", "name", "trigger", "at_ms", "entries", "changes", +"retained_files", "retained_bytes"}` once a checkpoint has been taken, the +baseline, `stop` and `exit` included, with the paths walked, the paths +changed since the checkpoint before and the files retained; and `{"kind": +"checkpoint_refused", "name", "reason"}` for a requested name it refused. A +driver that sends `checkpoint ` waits for the event naming it rather +than for a fixed time. When the session ends the capture sends SIGTERM to the launched process, its process group and every process that carries the run token, waits up to five @@ -122,7 +144,13 @@ command makes it private (mode 0700). It holds: distinct content. The Evidence operation is `capture_desktop_application` and its predicate type -is `rea.desktop-application-capture`. Its result has four parts. +is `rea.desktop-application-capture`. Its `host` names the platform, +architecture, display and home directory, and the host's name (`hostname`) +and the user's (`user`), which captured state can hold, as in a mail +template's `ada@e81232be457d`; the export removes them from the bundle. A +capture written before they were recorded has neither, and one is left out +when the system cannot say, as for a user with no passwd entry. The rest of +the result has four parts. **Application.** The desktop entry (id, path, name, `Exec`, icon, `StartupWMClass`, categories, MIME types), the launch command and working @@ -132,7 +160,39 @@ architecture and the apt sources that list the installed version, with the origin (`o=`) apt reports for each source in `source_origins`, such as `Ubuntu` for the Ubuntu archive and its mirrors or `LP-PPA-` for a PPA; an empty source list means the version came from a local `.deb` or the package lists -are missing, which `package_lists_present` tells apart. +are missing, which `package_lists_present` tells apart. apt's lists folder, +and the trusted keys below, are where `apt-config` resolves +`Dir::State::Lists`, `Dir::Etc::TrustedParts` and `Dir::Etc::Trusted` under +`Dir`, so an `APT_CONFIG` file that moves them is followed: on a computer +without lists, an ordinary user can run `apt-get update` with an `APT_CONFIG` +naming writable `Dir::State::Lists` and `Dir::Cache` folders and capture with +the same `APT_CONFIG`. Without `apt-config`, or when it fails, they are +`/var/lib/apt/lists`, `/etc/apt/trusted.gpg.d` and `/etc/apt/trusted.gpg`. + +The Debian package is the one `dpkg-query -S` names as the owner of the +desktop entry the capture launched, or else of the executable, since the +entry's package is what installs the application: LibreOffice Writer's entry +belongs to `libreoffice-writer`, while the `/usr/bin/libreoffice` it runs +belongs to `libreoffice-common`, which does not bring Writer with it. When the +two differ, `executable_package` names the executable's package, which the +export installs beside the entry's, since the entry's package need not depend +on it (`vim-common` owns Vim's entry but does not depend on `vim`). +`executable_package_reason` says why the computer could not install it by +name, by the rule for library packages below: neither the Ubuntu 24.04 archive +nor a vendor repository the export declares offers it, or `apt-cache policy` +did not say. When a package owns the entry but none owns its program, such as +a `~/.local/bin/` found first on `PATH`, the entry's package is recorded +without `executable_package`, and `executable_package_reason` says no package +owns the program, so the export refuses rather than install a package that +does not bring it. A program link that dpkg does not own counts as owned when +dpkg owns the file it resolves to, as an alternatives link does. An +`--executable` launch, or an entry no package owns, such as +one in `~/.local/share/applications`, records the executable's package. When +`dpkg-query` does not finish or exits with an error for the executable or the +entry, the package is `unknown` and the limitations say which path, since a +failed lookup is not proof that no package owns it. The sources, vendor +repositories and `--package-file` match below are the recorded package's; the +library walk still reads the executable. Each source whose origin is not `Ubuntu` is also recorded in `apt_repositories`. `apt-get indextargets` names the sources file entry apt @@ -148,24 +208,30 @@ and in `apt-get indextargets` and `apt-cache policy` output alike, so entries match whether or not apt prints it, and sets `uri_credentials_removed` on a repository whose URI held one (a capture written before this was recorded omits the field). The key is found by verification, not by name: the candidates are the -entry's `signed-by` keyrings, or else every `.gpg` and `.asc` key in -`/etc/apt/trusted.gpg.d` and `/etc/apt/trusted.gpg`, and each is checked with +entry's `signed-by` keyrings, or else every `.gpg` and `.asc` key in apt's trusted +parts folder and its trusted keyring, and each is checked with `gpgv` against the release's `InRelease` (or `Release` with `Release.gpg`) -beside its index in apt's lists folder. A key counts when gpgv reports a -valid signature by it and no bad, expired, revoked or unverifiable signature -by it, whatever gpgv's exit status: a release also signed by a key the host -lacks makes gpgv fail. Each candidate is copied, an armored key decoded to a -binary keyring, into a private temporary folder that is removed afterwards; a -folder REA cannot remove fails the capture with the folder as its cleanup -resource. The key that verifies is recorded in `signing_key` with its path, +beside its index in apt's lists folder. A deb822 entry whose `Signed-By` field +holds the armored key itself, as `add-apt-repository ppa:/` +writes it on Ubuntu 24.04, has that key as its only candidate, as apt trusts +no other for it: REA reads the field's continuation lines as deb822 writes +them, a `.` line standing for an empty one, from the key block's `BEGIN` line +to its last `END` line, and records the key's path as the entry's sources +file. A key counts when gpgv reports a valid signature by it and no bad, +expired, revoked or unverifiable signature by it, whatever gpgv's exit status: +a release also signed by a key the host lacks makes gpgv fail. Each candidate +is copied, an armored key decoded to a binary keyring, into a private +temporary folder that is removed afterwards; a folder REA cannot remove +fails the capture with the folder as its cleanup resource. The key that +verifies is recorded in `signing_key` with its path, SHA-256, format (`binary` or `armored`) and primary fingerprint, and its bytes are retained under `content/`. A key file holding one key is retained as it is; from one holding several, such as a legacy `/etc/apt/trusted.gpg`, only the transferable key with the verifying fingerprint is retained, in binary, so the bundle trusts no other key. When no candidate verifies, `gpgv` is -missing, apt holds no signed release, no sources entry matches, the entry's -`Signed-By` holds the key itself, or the key is larger than -`--max-file-bytes` or the content budget, `signing_key` is null and +missing, apt holds no signed release, no sources entry matches, the key the +entry's `Signed-By` holds does not verify the release, or the key is larger +than `--max-file-bytes` or the content budget, `signing_key` is null and `signing_key_reason` says why. Ubuntu Pro's ESM repositories (origins such as `UbuntuESMApps`, host `esm.ubuntu.com`) are not checked: apt reaches them only with the host's Ubuntu Pro credentials, which a computer does not have. @@ -187,7 +253,12 @@ listed, and neither is the application's own package. One batched recorded vendor repository with a verified key offers, in any component its sources entry lists, since the export declares them all; the rest, such as packages from a PPA or a local `.deb`, are listed in -`uninstallable_library_packages`. Sonames found nowhere are listed in +`uninstallable_library_packages`. The same query checks `executable_package`. +Each recorded repository's `supplies` lists the kept packages, and the +executable's package, that the archive does not offer and its package files +list, so the export can leave out those only a repository it does not declare +supplies, such as one whose key the content budget later dropped. Sonames +found nowhere are listed in `missing_libraries`, and libraries no package owns in `unpackaged_libraries`. `library_packages_reason` says why the lists are incomplete: the executable is not an ELF image (a launcher script, for example), `ldconfig`, `dpkg-query` or @@ -223,7 +294,10 @@ the fuse sentinel in their binary or a `resources/app.asar` beside it. State directories are the XDG config, data, state and cache folders and the dot-folder named after the package, the executable, the desktop entry's full id and its last part, and the entry's name, with whether each existed at the -start. Dotfiles in the home named after the application (`.rc`, or +start. A folder in those places whose name is one of these in another letter +case joins them, such as GIMP's `~/.config/GIMP` for the `gimp` package; only +the whole name is compared, so `~/.config/GIMPshop` does not. Dotfiles in the +home named after the application (`.rc`, or `.` followed by `-`, `.` or `_`, for a name of three or more characters) that exist at the start join them, since an application may only read such a file during the capture. A `flatpak run` launch is named after its application diff --git a/docs/desktop-application-export.md b/docs/desktop-application-export.md index ab687e849a..83fca6bc1a 100644 --- a/docs/desktop-application-export.md +++ b/docs/desktop-application-export.md @@ -22,10 +22,16 @@ the capture, once REA has validated it against the capture; see [desktop application tasks](desktop-application-tasks.md). `--package-url` says where the computer downloads the package of an application installed from a `.deb` file; see [Applications from a .deb file](#applications-from-a-deb-file). +`--seed-checkpoint` names the checkpoint whose state seeds the bundle, the +baseline by default, and `--end-checkpoint` the one a task must pass on, the +capture's last by default; see [The seed checkpoint](#the-seed-checkpoint). The command returns Evidence with operation `export_desktop_application` and predicate type `rea.desktop-application-export`, listing the environment name, -owned paths, seed files and written files. A failed export, cancellation +the seed and end `checkpoints`, owned paths, seed files, the seeds it +rewrote to remove the capture host's name and user (`scrubbed_files`), the +bundle files that still hold them (`unscrubbed_files`), and written files, +the last three by their path in the bundle. A failed export, cancellation included, removes the bundle it had not yet published, so no partial copy of the user's data remains. If the bundle was published but its output handles could not be closed, the command reports a cleanup error that carries the export @@ -38,7 +44,7 @@ Evidence; the bundle itself is complete. | `envs//env.toml` | `type = "computer_app"`, the module and class, `computer = "ubuntu-24.04"`, `driver = "reference"` | | `envs//.py` | The `ComputerAppEnvironment` class | | `envs//SKILL.md` | What the app is, its window class and where its files live | -| `envs//seed/` | The application's files from before the capture, by home-relative path | +| `envs//seed/` | The application's files at the seed checkpoint, by home-relative path | | `envs//apt-keys/` | The signing key of each vendor apt repository the app installs from | | `envs//install-package.sh` | With `--package-url`: the run script that installs the downloaded package with dpkg | | `tasks/smoke.json` | A task that deploys the app | @@ -63,8 +69,18 @@ The class declares: leaves out dependencies on the libraries it loads, as Typora's Electron package does, still starts; library packages the capture found neither the Ubuntu 24.04 archive nor a declared repository offers are left out, with a - limitation. An archive package's declared dependencies are trusted as they - are. The package must come from the Ubuntu 24.04 archive or a mirror of it, + limitation, and so are those only repositories the export leaves out supply + (a repository's `supplies`), such as one whose key the capture could not + retain. An archive package's declared dependencies are trusted as they + are. When the desktop entry's package is not the one that owns the program + it runs, as `vim-common`'s entry runs `vim`'s program, the program's package + (`executable_package`) follows the app's by name, since the entry's package + need not depend on it; a capture that found no archive or declared + repository offering it records why, and the export refuses it with that + reason, as it does when only repositories it leaves out supply it, or when + no package owns the program at all (`executable_package_reason` without + `executable_package`). The package must come from the Ubuntu 24.04 archive + or a mirror of it, or from a vendor apt repository whose signing key the capture verified and retained. A source counts as the archive when apt reports its origin as `Ubuntu` for a noble suite; a capture made before origins were recorded @@ -72,9 +88,11 @@ The class declares: - For a vendor repository, `sources=[AptSource(apt=..., suite=..., components=[...], key="apt-keys/.gpg")]`, with the retained key copied to that path (`.asc` when it is armored, which agentenv tells by its - content). apt takes one key per URI and suite, so sources that share them, - such as one repository listed for two architectures, are declared once, with - their components merged; one with a different key is left out. A repository + content). A PPA added with `add-apt-repository`, whose sources entry holds its + key in `Signed-By`, exports that key as `apt-keys/.asc`. apt takes one + key per URI and suite, so sources that share them, such as one repository + listed for two architectures, are declared once, with their components + merged; one with a different key is left out. A repository whose key the capture did not verify, an Ubuntu Pro (ESM) repository, one whose sources entry's URI held a user name or password (which the capture did not record and the export never writes; the reason names neither them @@ -100,18 +118,23 @@ The class declares: taskbar or says it is a splash, and an app whose splash shows first would otherwise count as ready before its own window opens. Arguments that are the captured home or under it, and `