Entrpiandgensokyo-dev
64196b08ca
* feat(artifacts): inventory and extract ESP32 firmware images ESP32-family app and bootloader images (magic 0xE9) had no reader, so open_binary refused OTA downloads, app partitions and bootloaders. Add the esp-image container format behind inspect_artifact and extract_artifact. A pure decoder reads each byte once and keeps only fixed-size fields. Derived facts cover the chip, from a table cited to ESP-IDF v6.1; segment classification as ESP-IDF's should_map and should_load; checksum and appended SHA-256 results; app and bootloader descriptors; Secure Boot v2 signature blocks, observed and unverified; and undescribed trailing byte ranges. Truncation and undescribed bytes make coverage unknown. ESP8266 images are refused with a specific reason, and ciphertext stays unrecognized. Add the format to every target, Evidence, snapshot, live-summary and artifact vocabulary, document it, and add an optional verify:esp-images lane that compares esptool-built images with esptool image-info and espsecure signature-info-v2. Refs #3615 * test(artifacts): run the ESP image lane on esptool v5.6.0 The lane compared descriptor fields only when esptool image-info printed one, so a descriptor that REA reports and esptool does not print passed without comparison. Compare the descriptor kind in both directions. Record that the lane passed for all 13 images with esptool and espsecure v5.6.0, the version the layout facts cite, as well as with 4.12.0. * refactor(artifacts): define ESP image layout sizes once The decoder and the derived facts each defined the segment-header, appended-digest and flash-sector sizes, the erased-byte scan and the offset formatter. Define them once with the other fixed-layout structures and import them in both places. The signature slot count is the one constant that drives the slot scan. Correct the observedEnd comment: it is the end of the last chunk read. * fix(artifacts): report ESP-IDF's segment length limit by chip ESP-IDF's verify_segment_header rejects a segment of 16 MiB or more unless the chip maps flash through 32-bit cache addresses (SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP: ESP32-S3, P4, C5, C61, H4 and S31 in v6.1). Record that capability in the chip table and add a limitation for such segments; unknown chips get none. `loaded` repeats should_load. It does not mean the bootloader accepts the segment: verify_load_addresses still rejects one outside the chip's RAM, which REA does not model. Say so in the docs, the skill reference and the inventory limitation. * test(artifacts): check ESP lane tools first and verify extraction The lane started work before checking its tools, so a missing RISC-V target, ld.lld, esptool or espsecure surfaced as a raw spawn or command error. Check each one before building and name the variable to set. Warn when esptool is not v5.6.0, the release the layout facts cite. Assert each segment's class for the ESP32-C3 app, its variants and the bootloader, as the Xtensa image already did. Extract the app and the signed image through the CLI and compare every file with the bytes at esptool's segment offsets or in the appended signature sector; the source's SHA-256 must be unchanged and equal the reported root_sha256. * docs(artifacts): state what the ESP image lane does not establish The Verification section listed the lane's checks but not its gaps: only ESP32-C3 and ESP32 images come from a real producer, no real project build, OTA download or flash dump is inspected, no signature is verified, and the lane does not run in CI. Record the timing of the final lane run on esptool v5.6.0. * fix(artifacts): compare ESP segment declarations Segment entries hash only their data, so compare_artifacts reported a segment that moved to another load address with the same bytes as unchanged; only the image itself changed. Compare each segment's load_addr and data_len as its declaration, as Node SEA and PyInstaller members are compared, and count a changed declaration as a metadata change, also when content is unavailable. Header offsets are layout and the classification is derived, so neither is compared. * test(artifacts): run ESP lane commands as owned process trees The lane ran its tools through execFile. A timeout killed only the direct child, so a linker or assembler that Clang or GCC had started could keep running while the lane removed its workspace. Run every command through createVerifierCommands, check the tools inside withVerifierCleanup, and remove the workspace only after commands.close() has stopped every tree; a failed close keeps the workspace. Only a nonzero exit now counts as a refusal or an expected esptool failure; timeouts and spawn failures propagate. Supervising each command roughly doubles the lane's run time. * test(artifacts): cancel ESP decoding after its first decoded chunk The cancellation test requested cancel or close on setImmediate, which can land while the file is still being opened and checked, so it never showed that decoding had started. It also passed against a reader that ignored both requests while decoding. Acknowledge the decoder's first chunk, then cancel or close, join the operation, and assert that no later chunk reaches the decoder and that the reader ends closed. * docs(artifacts): record which ESP metadata comparison ignores Segments are compared by their declaration. The image's and the signature blocks' format_metadata are not: every field comes from their own bytes, which content comparison already covers, and a block's image_digest_matches follows the image's content. * docs(artifacts): state that ESP mmu_page_size is a base-2 logarithm The app descriptor's mmu_page_size keeps ESP-IDF's name and value, the MMU page size in log base 2 as esp_app_desc_t declares it, so 16 means 64 KiB. Say so in docs/esp-image.md and the schema comment; a reader could otherwise take 16 as 16 KiB. * fix(esp-image): retain member consumption and share lifecycle regressions * test(artifacts): scope MCP catalogs and budget full-schema validation * test(artifacts): verify compact output profiles and document bounded catalogs * test(artifacts): close readers when cleanup-test admission fails * test(artifacts): compare ESP image limitations with their owners The decoder and reader tests froze complete limitation sentences, so a harmless rewording failed them. The wording now has two owners: ESP_IMAGE_LIMITATIONS in espImageFacts.ts, which the decoder, the inventory and the reader use, and ESP_DESCRIPTOR_LIMITATIONS beside the descriptor decoder. Tests compare delivered limitations with the owner, called with the facts each case establishes (segment index, declared and available bytes, offsets, chip, page range), next to the structured assertions they already make. Delivered text is unchanged. * test(esp-image): assert structured facts and owned guidance * fix(esp-image): keep spi_flash_mode and descriptor fields before a truncated tail ESP-IDF v6.1's esp_app_desc_t declares spi_flash_mode at byte 181, after mmu_page_size, as the committed reference layout shows; the descriptor now keeps it as stored. The decoder also required all 256 descriptor bytes, though it reads none past byte 181, so a file ending in the reserved tail lost the version, project, digest and revision facts it does hold. The app descriptor now needs only its fields; the segment's truncation is reported as before. * test(esp-image): pin the P-192 public key digest to espsecure's layout espsecure packs ECDSA signature blocks as "<BBBx32sB64s64s1031x" and zero-pads a 48-byte P-192 point to 64 bytes, so the signature starts at byte 101 for every SHA-256 curve. signature_info_v2 hashes sig_block[36:101] and _digest_sbv2_public_key the curve ID with the padded point. No test built a P-192 block; this one checks public_key_sha256 against that definition. * fix(esp-image): keep each app descriptor field the file holds in full A file can end anywhere inside the 256-byte esp_app_desc_t. The decoder needed every field byte, so a cut at byte 176 dropped the version, project and app_elf_sha256 it does hold. Once the 4-byte magic identifies an app descriptor, each field is now kept when the file holds all of its bytes and is null otherwise; observed_length records where the descriptor ends, so a cut field is told apart from text that is not UTF-8. The fields after the magic become nullable in the metadata schema. * fix(esp-image): report a signature block whose image digest does not match A CRC-valid Secure Boot v2 block can carry another image's digest. The block recorded image_digest_matches false, but no limitation said so, unlike a checksum or appended-SHA mismatch, so CLI and MCP summaries showed only the generic unverified note. The mismatch is now a limitation; coverage stays complete and signature_status stays unverified. The three other signature-sector limitations move into ESP_IMAGE_LIMITATIONS with their text unchanged, so all ESP image wording has one owner. * refactor(esp-image): remove redundant descriptor byte guards --------- Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
17 KiB
Firmware analysis
REA accepts Binwalk 3.1.x for region inspection and Unblob 26.6.x for explicit extraction. The report parsers are verified with Binwalk 3.1.0 and Unblob 26.6.4. Another build on those lines is accepted, the observed version is reported, and a limitation records that the parser was not verified against that build. This first firmware milestone is verified on Linux x64. No device, root privilege, mount, emulation or execution of extracted programs is needed. Arbitrary firmware bytes do not pass native executable admission; select a returned ELF separately for supported Ghidra analysis.
On Linux, Binwalk and Unblob run as REA-owned processes and require a
procps-compatible ps on REA's PATH; see the shared
Linux process prerequisite.
Caller-supplied tools
Provide existing tools using absolute executable paths, scoped to your shell:
export REA_BINWALK_COMMAND=/absolute/path/to/binwalk
export REA_UNBLOB_COMMAND=/absolute/path/to/unblob
REA does not install these tools, their dependencies or system packages. It
checks the tool's --version banner when executing a request. Binwalk must
print binwalk <version> and Unblob must print the version alone. Inspection and
extraction have independent prerequisites; one does not require the other.
Both require util-linux prlimit, normally /usr/bin/prlimit; override its
absolute path with REA_FIRMWARE_PRLIMIT_COMMAND if needed. rea doctor does
not currently inspect firmware readiness. Engine selection/version checks run
when the firmware operation is requested; successful general doctor checks do
not establish firmware tool or external-extractor availability.
REA retains at most 8 MiB of combined stdout and stderr per command. Exceeding that diagnostic budget stops analysis with an explicit truncated-output error; child process limits do not bound parent-side pipe retention. Completed commands drain their output streams before recording diagnostics. Failed worker cleanup retains the worker and workspace for retry before another request or at shutdown.
Unblob needs the external extractor for the selected format on its inherited
PATH: for example, debugfs for ext2/ext3/ext4. Its default Landlock sandbox
and skip rules stay enabled. Sandbox startup failures remain actionable errors;
REA does not retry with --no-sandbox. Missing extractors reported in a valid
analysis report retain partial results and dependency names inline.
Upstream sources are preserved unmodified as optional pinned submodules:
git submodule update --init third_party/binwalk third_party/unblob
Normal REA builds and npm packages do not include or build these sources or engine binaries. See upstream provenance.
Windows submodule checkout
On Windows, the Unblob checkout stops with error: invalid path. Several
upstream test fixture names under tests/integration/ contain characters,
trailing dots, or trailing spaces that NTFS does not allow; REA does not use
those fixtures. Keep the clone, exclude that directory, and retry from the
repository root in PowerShell:
git -C third_party/unblob config core.protectNTFS false
git -C third_party/unblob sparse-checkout set --no-cone '/*' '!/tests/integration/'
git submodule update third_party/unblob
Both settings are required. While core.protectNTFS is enabled, Git for
Windows also rejects index paths that the sparse pattern excludes; the pattern
keeps the invalid names out of the worktree. The setting applies only to the
Unblob submodule. In Git Bash, prefix the sparse-checkout command with
MSYS_NO_PATHCONV=1; otherwise !/tests/integration/ is rewritten as a
Windows path.
The extraction destination must be absent with an existing writable parent.
REA checks that selection before starting an engine and still acquires it
exclusively during publication. Invalid sources and destinations identify the
caller argument; failures inside an acquired workspace keep their own cause.
Published file relative_path preserves producer spelling for lineage; path
uses the writer's actual normalized filename. An extracted filename that cannot
be represented losslessly as UTF-8 fails publication explicitly with its raw
name bytes in hexadecimal instead of silently omitting the entry.
CLI and MCP
rea inspect-firmware-regions /path/to/firmware.bin
rea extract-firmware /path/to/firmware.bin /path/to/new-output
rea extract-firmware /path/to/firmware.bin /path/to/selected-output \
--offset 64 --length 4096 --max-depth 3
The corresponding MCP tools accept the same intent:
{
"name": "inspect_firmware_regions",
"arguments": { "path": "/path/to/firmware.bin" }
}
{
"name": "extract_firmware",
"arguments": {
"path": "/path/to/firmware.bin",
"output_directory": "/path/to/new-output",
"range": { "offset": 64, "length": 4096 },
"max_depth": 3
}
}
Both return Evidence with the original path and SHA-256, raw producer reports, actual executable path/digest, version and limitations. Binwalk signature UUIDs and Unblob blob IDs are producer-local identities; compare content hashes and source ranges across runs. A matching version does not attest an installed build's source revision, so that field remains unknown.
Interpret the results
- Region offsets refer to bytes in the original firmware.
reported_sizepreserves Binwalk's output;size_basis=provider_reported_validation_unknownreflects that Binwalk can replace unknown lengths with the next signature or EOF before serialization. Neither a signature nor its confidence byte proves a valid partition, encryption or runtime behavior. - Extraction returns regular file paths, independently verified SHA-256 values,
producer hashes when available, chunks, diagnostics and task derivations.
Unblob's task report is unordered; REA resolves relationships by paths and
blob IDs. Logical references use
$input,$output, or$output/<relative-path>so a real file named$inputstays distinct from the original input. Chunk offsets belong to the task's input file. Only chunks in the selected input have a knownroot_file_range, adjusted by the selected offset. - Decompressed children have unknown original byte ranges and runtime addresses. Task lineage does not establish a byte-for-byte mapping through decompression.
partialmarks unknown/encrypted chunks, selected input bytes outside all reported chunk intervals, reported extraction problems, unpublished links/special files or reached depth limits.completemeans the selected input is covered by reported chunk intervals and the engine finished its configured extraction without those reported conditions; default skip rules and format coverage still apply. It does not establish complete firmware reconstruction.extraction_gapsmakes those conditions recoverable: a row identifies an unextracted region or a failed task/stage. A chunk retains its identity and byte range, plus its original-file range when it belongs to the selected input.unclassified-rangerows identify selected input intervals with no reported chunk; their provider identity and handler stay null. An empty chunk inventory therefore cannot establish complete extraction of a nonempty selected input. A multi-file stage retains its reported ID and handler with unknown byte ranges; a standalone task failure keeps unavailable stage identity null. Rows retain exact dependency names from the producer's missing-dependency reports; an accepted report with no names remains a failure with unknown dependencies; no executable identity or specific installation step is inferred. Each row'scontinuation_idreferences one instruction inextraction_continuations, so many regions share recovery prose without losing any region identity. Instructions use already configured capabilities where one exists. Every extraction rerun requires a fresh absentoutput_directory, preserving the prior evidence. Unblob's declaredERRORseverity establishes an extraction failure, including standaloneUnknownErrortask exceptions. Warnings and resolved extraction problems remain diagnostics with partial coverage; they do not establish failed extractor stages. Standalone errors do not acquire a nearby chunk's handler or byte range. Nested unknown chunks useinspect_firmware_regionson the actual published child rather than repeating the same extraction with unchanged handlers. Inspection requires separately configured Binwalk (REA_BINWALK_COMMAND); without it, the inspection step is unavailable and the region remains unexamined. A listed gap does not mean the target is corrupt or that a retained region is empty; REA installs nothing.- The destination must be absent, absolute, with an existing parent. Publication uses exclusive, symlink-resistant writes and durable digest readback. Existing destinations are never overwritten. This is not an atomic directory rename: a new tree can be visible while files are written. Failed uncommitted output is rolled back. Symlinks and special files are recorded without publication; empty directories, ownership and executable permission bits are not reproduced.
Select a verified native child for the existing tools:
{
"name": "open_binary",
"arguments": {
"path": "/path/from/extraction/usr/bin/program",
"provider_id": "ghidra"
}
}
Then use analyze_function, inspect_native_api or the other native primitives.
Native architecture and executable validation still apply. Generic
MIPS ELF and RISC-V ELF lanes are separate
documents with their own admission boundaries. This milestone does
not add PPC decompilation, raw flash load profiles, bootloader/device
tree semantics, vulnerability scanning, emulation or hardware runtime capture.
Resource and lifecycle policy
- Nonempty regular firmware inputs: at most 128 MiB, copied and hashed into a private workspace before analysis. Growth beyond that allowance during copying receives the same typed input refusal as an already oversized file. Supply a stable copy within the allowance. No in-place source modification.
- One queued heavy operation per provider instance; Binwalk one thread and Unblob one worker. No automatic parallel extraction engines.
- Analysis defaults to 120 seconds, from tool resolution and input snapshot
through inspection and its final private-workspace cleanup, or extraction
through report consumption and inventory, excluding queue waiting and retained
cleanup admission. Inspection has no publication phase. Select
analysis_timeout_msin either MCP operation or--analysis-timeout-msin either CLI command for slower analysis. Current results retain the actual budget; an absent historical value means unknown. Extraction publication and its final cleanup start a separate 120-second budget; selectpublication_timeout_msin MCP or--publication-timeout-msin CLI for a slow destination. Valid budgets are integers from 1 through 2,147,483,647 ms (the timer representation limit). Publication timeouts identify their phase and verified file progress; verified writes before atomic commit are not a claim that output remains published after rollback. Caller cancellation applies to both phases and final cleanup. Phase handoff, publication commit and final completion also check monotonic elapsed time, so delayed timer callbacks cannot revive an expired budget. - Analysis timeout recovery names the selected budget and the analysis control, not the later publication control. Keep the client call timeout above analysis, publication and cleanup headroom; keep the input/range/question unchanged. Resolve any reported incomplete cleanup before retrying. Raising wall time does not raise fixed CPU, memory, report, diagnostic or input limits.
- Inherited 1 GiB per-process address-space limit,
120 CPU seconds and zero core-dump allowance through
prlimit. - JSON reports: at most 8 MiB and valid UTF-8; malformed encoding and
oversized reports fail without repairing or truncating producer data.
Exit status 0 is not an empty result: a missing, empty, malformed or
oversized report fails with
unreadable_output, itsreport_state, the command, exit status and captured output. REA does not retry automatically. Diagnostic output is bounded with exact captured byte counts. - Default staging budget: 256 MiB and 10,000 entries, configurable using
max_output_bytes/max_output_files(CLI--max-output-bytes/--max-output-files). Exceeding either selected staging budget returns a nonretryable resource constraint naming the selected limit. Depth defaults to 3, allowed range 1–10.
Staging budgets are polled and rechecked after exit; short-lived overshoot is possible. The per-file process limit is the larger of the selected payload budget and the fixed report/diagnostic file allowances, so a small payload selection still permits the mandatory report. A tighter inherited host limit still applies. The inherited file-size limit bounds each file, not total storage; address-space limits are per process, not an aggregate memory quota. Worker count is not CPU affinity for every external extractor. These limits do not claim cgroup/container containment. Cancellation cleans the owned process group before removing its workspace; uncertain cleanup retains resources and blocks further operations in that provider instance.
Verification
Generate small public, source-derived fixtures into ignored _reference/:
npm run fixtures:firmware
npm run verify:firmware
Fixture generation requires existing Python 3 and a host C compiler (cc);
override with REA_FIRMWARE_PYTHON / REA_FIRMWARE_CC. The independent oracle
records gzip offset/length and extracted file hashes. Real verification checks
CLI/MCP, full-file and selected-range extraction, unknown chunks, depth limits,
content parity and unchanged source bytes. No binary fixture is committed.
The optional missing-extractor lane asserts dependency names and the owning
handler in extraction_gaps. Those assertions must be run against the pinned
Unblob release to establish real-provider coverage; synthetic producer fixtures
alone do not verify Unblob's report nesting or stage identities. The gap ownership
claims remain unverified against real Unblob until that lane runs.
Additional lanes are explicit, so the base lane stays small:
# Existing mke2fs and debugfs required; put debugfs on the scoped PATH.
REA_FIRMWARE_VERIFY_EXT4=1 npm run fixtures:firmware
REA_FIRMWARE_VERIFY_EXT4=1 npm run verify:firmware
# Existing Ghidra 12.x (>=12.1) and its declared JDK required.
REA_FIRMWARE_VERIFY_GHIDRA=1 npm run verify:firmware
REA_FIRMWARE_MKE2FS_COMMAND selects the optional ext4 fixture producer.
REA_FIRMWARE_FIXTURE_ROOT selects generated data storage.
REA_FIRMWARE_TEST_ENTRYPOINT can select an isolated installed npm package's
scripts/rea.mjs to verify the packaged CLI/MCP workflow. Real coverage here is
generated gzip/USTAR/ext4 with a Linux x64 ELF child; SquashFS, UBIFS, proprietary
firmware, other hosts and architectures remain unverified.
Inspect ESP images
Use inspect-artifact / inspect_artifact on an ESP32-family app or
bootloader image, such as an OTA download or app partition, for its chip,
classified segments, checksum and SHA-256 results, descriptor and Secure Boot
v2 signature blocks. extract-artifact / extract_artifact writes one file
per segment and signature block. Reading is portable and needs no external
tool. See ESP images.
Inspect extracted devicetrees
Use inspect_devicetree / rea inspect-devicetree on an
extracted DTB for its node/property tree, reservations, hardware summary and
selected phandle relationships. FDT v16/v17 inspection is portable and requires
no external tool or disassembler.
Inspect Intel HEX and S-record images
Use inspect-artifact / inspect_artifact on an Intel HEX or Motorola S-record
file for its address regions, declared start address and record facts, then
extract-artifact / extract_artifact for one raw file per region. Decoding is
portable and requires no external tool. See
Intel HEX and S-record images.
Inspect U-Boot legacy uImages
inspect-artifact / extract-artifact recognize a legacy uImage by its
0x27051956 magic and inventory it without running or decompressing it.
The root facts record the 64-byte header (architecture, OS, type, and
compression as raw values and names, load and entry addresses, image name,
and timestamp) with the header and data CRC32 verified; the stored payload
is extractable as payload.bin, and multi-file images additionally expose
each sub-image as image-<n>.bin from the stored size table. Declared
addresses are facts from the header, not proven entry points, and compressed
payloads stay as stored. Intel HEX/S-record and FIT images are separate
readers and are not covered here. See U-Boot legacy uImage for
component padding, preserved names, resource policies and verification scope.