mirror of
https://github.com/morluto/rea.git
synced 2026-11-10 00:36:35 +00:00
bf271ba3fb4bb882846c5ec9b34931bb356ea89c
37
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
5990ff0324 |
feat(artifacts): read WeChat mini-program wxapkg containers (#8148)
* feat(artifacts): read WeChat mini-program wxapkg containers Add a read-only reader for plaintext wxapkg packages: the 0xBE/0xED 18-byte big-endian header, the bounded name/offset/size member table, and verbatim member bytes at absolute carrier offsets. - Members retain declared names, absolute offsets and sizes, so source_location reports real carrier positions. - V1MMWX PC-client packages are detected as encrypted coverage with null member facts; extraction refuses with a typed error and no decryption is attempted. - Bounded table and directory budgets are admitted before allocation; malformed marks, truncated tables, invalid UTF-8 names, member extents inside the header or beyond the file, and counts that cannot fit their table are typed format or limit failures. - Member reads stream in bounded 64 KiB chunks with carrier freshness checks; the reader joins outstanding streams before close and bounds active streams and repeated member I/O. Refs #8121. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(artifacts): bound wxapkg members to the declared data region Normalize the two indexLength conventions before the table bounds check and the table read. Member ranges must then lie in that declared data region, which has to fit in the file. Record wxapkg format metadata as not compared: header facts repeat the package bytes, and entry source paths keep a producer root prefix. * fix(artifacts): choose the wxapkg index after member ranges fit Four trailing bytes made the rows-only layout end on the file and dropped the count-inclusive layout. Keep every admitted layout until the parsed rows and member ranges select one. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Entrpiandmorluto
|
b43c329df7 |
feat(artifacts): inventory and extract UF2 and DfuSe firmware containers (#8230)
* feat(artifacts): inventory and extract UF2 and DfuSe regions open_binary refused UF2 and DfuSe firmware containers as an unsupported format, so inspect_artifact and extract_artifact could not show which bytes a drag-and-drop or DFU update writes at which addresses. UF2 is recognized by a complete first block with all three magic numbers. Valid main-flash payloads are joined into address-contiguous regions per familyID, named family-0x<id>/0x<start>-0x<end>.bin, and known IDs are named from a small table copied from Microsoft's uf2families.json at a pinned commit. Numbering is counted per familyID and block count: absent, repeated, out-of-order and out-of-range block numbers are reported, and absent blocks leave coverage unknown. Not-main-flash blocks are excluded and reported; file-container blocks are reported and leave coverage unknown; invalid blocks and a partial final block are excluded with their position. A rewrite of one address with different bytes fails closed, naming both block spans. DfuSe is recognized only by its prefix together with a DFU 1.1a suffix. Each element becomes one region at its file range with its target's alternate setting and name. Structural faults fail closed with a byte offset; overlapping elements stay separate and are reported. A suffix CRC mismatch follows integrity_policy: fail refuses with a dfu-crc32 integrity error, and record-and-continue records a root contradiction that extraction also carries. HEX, UF2 and DfuSe share one retained-region reader and one run merger, so the HEX decoder and reader lose their private copies. * docs(artifacts): document UF2 and DfuSe images and their esptool lane Describe recognition, region names, metadata, numbering, failures, the DfuSe suffix CRC under integrity_policy, and limits, and link the page from the CLI and firmware guides and the agent skill reference. npm run verify:uf2 runs esptool merge-bin --format uf2 on generated inputs at known addresses, with and without MD5, with 256-byte payloads, and on a concatenation of two chips' outputs. Inspection and extraction through the built CLI must return every input at its address under the familyID esptool itself reports. * fix(artifacts): accept both DfuSe image size conventions UM0391 calls DFUImageSize the file length, and dfu-util's dfuse-pack.py writes that value, while ST's DfuFileMgr, MicroPython's dfu.py and pydfu leave the 16-byte suffix out. The decoder accepted only the second form, so it refused dfuse-pack.py output as malformed. Accept either value and keep the declared one in image_size; any other value still fails with the suffix offset and the file length. The tests pin both producers' output for one image, CRC included, from their build() functions transcribed and run under Python 3. * fix(artifacts): keep reused UF2 block numbers from hiding missing blocks esptool numbers every merged input from 0, so two inputs with the same block count share one numbering group. A block lost from one input left its number present in the other, and the file reported complete coverage. Each block number now records the target addresses it was seen at: equal address counts are complete copies, a resend to the same address is one block, and unequal counts make coverage unknown with a limitation and ambiguous_numbering_group_count. sequence_count becomes numbering_group_count, and region block positions become first_file_block and last_file_block. Review follow-ups in the same change: - compare_artifacts compares a DfuSe element's alternate setting and target name, so bytes moved to another device memory are changed; - the shared reader keeps no region bytes outside its map after close; - one domain type replaces four identical decoder failure types; - the UF2 decoder returns a typed limit instead of throwing on bytes beyond its admitted size, and caps retained regions, not runs; - DfuSe caps elements with bytes only, names empty elements and element-less targets, drops the constant bVersion and bcdDFU fields, and says whether the CRC matches when DFUImageSize fits neither convention; - classification evidence carries a suffix only when the probe saw the end, and root classification reads the tail only for a DfuSe prefix; - the UF2 domain region owns family_name, and the root records the family registry revision; the table's MIT notice is preserved; - docs pin the producer revisions (dfu-util ee6b393d and f9a537c9, MicroPython cb99ca98), give the MD5 structure's real meaning, and describe container contradictions in the MCP contract guide; - tests and the esptool lane add the equal-count inputs and a block lost from one of them, accepted boundaries beside each refusal, a DfuSe root larger than the classification prefix, and CLI refusals. * feat(artifacts): name the UF2 families that LibreTiny builds for LibreTiny writes UF2 for Realtek, Beken and Bouffalo Lab modules. The Microsoft registry revision REA already cites (cf3bd7f9) lists eight of those families: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U, BK7231N, BK7251 and BL602. Their regions now carry those names. LN882H, which that revision omits, keeps its familyID with no name. * fix(artifacts): count DfuSe declarations without bytes instead of listing them Empty DfuSe elements have no entry, so DFUSE_MAX_ELEMENTS does not cap them, and the target walk kept a description string for each one. At 8 bytes per element, a file under the 128 MiB ceiling can declare about 16.7 million of them. The walk now counts every declaration without bytes and builds only the first NAMED_ELEMENTS descriptions, which is all that the limitation shows. The limitation text and empty_element_count do not change. The guide now says that both DfuSe limitations name the first four. On this host, decoding 4,000,000 empty elements (a 32 MB file) ran out of memory under a 128 MB V8 heap before this change, and it peaked at 1.59 GB RSS with the default heap. It now decodes in 114 ms with a 220 MB peak. --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> |
||
|
|
22e5aa97af |
feat(artifacts): inspect classic iNES container regions (#2816)
* feat(artifacts): inspect classic iNES container regions
* fix(artifacts): preserve iNES source disappearance as integrity failure
* fix(artifacts): revalidate iNES source before parsing header
* fix(artifacts): validate admitted roots before classification
* fix(artifacts): normalize source loss during root hashing
* fix(artifacts): bind iNES extraction admission to inventory
* test(artifacts): clarify non-executing inspection wording
* fix: preserve binary plist detection with extended magic probe
* test(browser): wait for popup document loads before capture
* fix(artifacts): honor cancellation across source validation
* fix(artifacts): prefer artifact magic over implicit database suffix
* fix(artifacts): select extraction readers from scanned format
* fix(artifacts): complete prefix reads and share ZIP format admission
* fix(browser): budget scenario cleanup for content drain and shutdown
* test(artifacts): admit valid universal inputs before routing
* test(artifacts): verify iNES workflows on macOS
* test(ci): own iNES macOS coverage and portable fixture paths
* fix: preserve cleanup retries and avoid duplicate inventory parsing
* fix(contracts): share ZIP target formats across MCP lifecycle schemas
* fix(target): preserve typed cancellation during descriptor admission
* refactor(artifacts): use JSON-safe inventory producer output directly
* test(artifacts): share file read setup and satisfy CI checks
* fix(artifacts): classify changed admission as integrity failure
* ci: run Mach-O summary delivery coverage on macOS
* test(javascript): align parameter provenance with main contract
* test(browser): await discoverable fixture pages before runtime verification
* fix(artifacts): pass target metadata to format detection
* fix(artifacts): bind extraction to opened target
* fix: repair the static-check failures on the default branch (#2895)
* fix: repair the static-check failures on the default branch
The default branch fails its own required `Static checks` job three ways, so
every pull request reports that job as failed regardless of its own change.
- `npm run format:check` reports `src/domain/functionComparison.test.ts` and
`src/ida/IdaOperationRunner.ts`.
- `npm run lint` reports a duplicate `import { join } from "node:path"` in
`tests/boundary/mcp/ghidraInventoryMcp.test.ts` (lines 4 and 17).
- `npm run typecheck` reports `format: "devin"` missing from the client format
union in `src/application/SupportedClients.ts`, and an unchecked index in
`tests/boundary/cli/clientProfileLifecycle.test.ts`, whose pair array is
inferred as `string[][]`.
Apply the pinned formatter, drop the duplicate import, list the `"devin"` format
that its own definition already uses, and mark the pair array `as const`.
Validation: `npm run check:fast` and `npm run format:check` pass.
* fix(setup): keep Devin on the JSON configuration dialect
---------
Co-authored-by: 吴佳浩 <wujiahao@supremind.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
(cherry picked from commit
|
||
|
Entrpi
|
e669365c3b |
feat(ghidra): admit RISC-V ilp32 ELF with a derived calling convention (#8229)
* feat(ghidra): record the RISC-V float-ABI calling-convention decision Decision D3 (#7477), provisional: admit ELF32 soft-float (ilp32) RISC-V images through a lane that keeps the pinned RISCV:LE:32:default language and corrects the calling convention in the bridge. This commit is the contract only; admission is unchanged. - The domain owns the e_flags float-ABI vocabulary and parses .riscv.attributes and Tag_RISCV_arch into a declared, absent or unavailable ISA declaration. - The Ghidra profile owns the lane table (class and float ABI), lane ids built from fields, the ilp32-prototype-v1 policy and its __rea_ilp32 prototype, and the extensions Ghidra 12.1.4 decodes with p-code. - The session handshake accepts the bridge's calling-convention report. - docs/ghidra-riscv.md keeps the decision record and per-ABI policies for ilp32 and ilp32f; a test keeps it in step with the tables. - tests/conformance/c/riscv-float-abi.c is the source the policies were measured on. * feat(ghidra): admit RISC-V ilp32 ELF with a derived calling convention ESP32-C2, C3, C5, C6, C61, H2 and H21 builds are ELF32 soft-float (ilp32) images, which the RISC-V lane refused at its double-float check. Ghidra 12.1.4's generic RV32 compiler specification models ilp32d, so its stock prototype would read float and double arguments from fa0-fa7. - Target resolution reads the one SHT_RISCV_ATTRIBUTES section through the same bounded ELF declaration reader as range validation, and records Tag_RISCV_arch as declared, absent or unavailable. - Admission selects a lane from the class and float-ABI table. The soft-float lane also requires an RV32I Tag_RISCV_arch that declares only extensions Ghidra decodes with p-code; the refusal names any others (Zcb, Zcmp, Zcmt, Zb*, xesp*). - Soft-float profiles commit float_abi, arch, riscv_calling_convention ilp32-prototype-v1 and the lane elf32-exec-le-soft-float-generic-v1. Double-float profiles are unchanged. - ReaGhidraRiscvCallingConvention.java runs after analysis. It derives __rea_ilp32 from the default prototype without floating-point entries and rules, with fs0-fs11 killed by calls, self-checks its storage and assigns it to every non-thunk function, external function and function definition. Its report is written last. - The provider client requires that report exactly when the profile commits the policy, before any operation, and keeps it as session limitations and caller-project history. ELF32 single-float (ilp32f), quad-float and ELF64 soft- or single-float images stay refused, now with a reason built from the lane table. * test(ghidra): verify RISC-V ilp32 against real Ghidra verify:ghidra:riscv now also builds tests/conformance/c/riscv-float-abi.c with ESP32-C3's -march=rv32imc_zicsr_zifencei -mabi=ilp32, typed with DWARF and linked again with --strip-all. Every allocated section of the stripped image must keep its address and bytes. - riscv-checks.mjs holds the ilp32 storage of each fixture function together with the words the compiler's call site writes there. An LLVM objdump reading of the caller must find every word at its stated location: an unaligned named double, the a7/stack split, two floats in an integer pair, a large structure by reference, aligned variadic pairs and an indirect call. - Through CLI and stdio MCP, the soft-float builds must report the soft-float lane and ilp32-prototype-v1, __rea_ilp32 with exactly that storage (one integer word per location when stripped), caller pseudocode that passes the source constants and names no floating-point register, the prototype outcome in retained Evidence, and decoded fence.i and csrr. - The RV32 double-float image relabelled as soft-float, refused before, is admitted through the new lane. RV64 soft-float, single-float and unsupported bits are refused with their reasons. - riscv-checks.test.ts runs the checks offline against recorded LLVM and Ghidra output, including Ghidra's stock ilp32d prototypes as a negative control. * fix(ghidra): pass RISC-V ilp32 values over 8 bytes by reference The psABI passes any value wider than 2 x XLEN by reference, scalars included. __rea_ilp32 converted only structures and unions over 8 bytes to pointers, so a long double (16 bytes) was joined into a1-a4. Measured on Ghidra 12.1.4, rea_riscv_abi_quad(long double value, int sel) read value from a4:4,a3:4,a2:4,a1:4 and sel from a5, where clang passes a pointer in a1 and sel in a2. - ReaGhidraRiscvCallingConvention.java now has one derivation driven by a per-policy row (floating-point entry size, preserved floating-point registers). Every policy inserts a rule before the input list's final join that converts any value of 9 bytes or more to a pointer; a return that size already becomes a caller buffer. The self-check requires a 16-byte value to pass by reference. - The fixture gains rea_riscv_abi_quad. The call-site oracle now checks the words the by-reference pointers point at, for the large structure and the long double, and the recorded data comes from an LLVM 19 build. - docs/ghidra-riscv.md: the decision record and the ilp32 policy row. ilp32-prototype-v1 keeps its id: no release carries it yet. * test(ghidra): check RISC-V lane limitations by their facts The limitation tests froze a full explanatory sentence, the note count and array positions, so a harmless rewording failed them: the pattern Greptile flagged on #7486. Assert the facts the notes must carry instead: the pinned language, the static-linkage boundary, one note naming the declared ABI, the ABI the pinned language models and the prototype REA applies, and the declared ISA. * test(ghidra): check return storage in stripped RISC-V ilp32 builds The stripped branch of assertRiscvIlp32Storage compared parameter storage only, so an inferred return that lost the high word of a register pair, such as rea_riscv_abi_wide at a0:4 instead of a1:4,a0:4, still passed. Stripped returns must now use the typed build's storage, or none when the result goes through caller memory. Recovered type names are still not compared: a test accepts other type names and refuses the dropped high word. * docs(ghidra): record decision D3 as decided in #7477 |
||
|
Entrpi
|
e6d0618979 |
feat(artifacts): inventory and extract ESP flash layouts (#8228)
* feat(artifacts): inventory and extract ESP32 firmware images ESP32-family app and bootloader images (magic 0xE9) had no reader, so open_binary refused OTA downloads, app partitions and bootloaders. Add the esp-image container format behind inspect_artifact and extract_artifact. A pure decoder reads each byte once and keeps only fixed-size fields. Derived facts cover the chip, from a table cited to ESP-IDF v6.1; segment classification as ESP-IDF's should_map and should_load; checksum and appended SHA-256 results; app and bootloader descriptors; Secure Boot v2 signature blocks, observed and unverified; and undescribed trailing byte ranges. Truncation and undescribed bytes make coverage unknown. ESP8266 images are refused with a specific reason, and ciphertext stays unrecognized. Add the format to every target, Evidence, snapshot, live-summary and artifact vocabulary, document it, and add an optional verify:esp-images lane that compares esptool-built images with esptool image-info and espsecure signature-info-v2. Refs #3615 * test(artifacts): run the ESP image lane on esptool v5.6.0 The lane compared descriptor fields only when esptool image-info printed one, so a descriptor that REA reports and esptool does not print passed without comparison. Compare the descriptor kind in both directions. Record that the lane passed for all 13 images with esptool and espsecure v5.6.0, the version the layout facts cite, as well as with 4.12.0. * refactor(artifacts): define ESP image layout sizes once The decoder and the derived facts each defined the segment-header, appended-digest and flash-sector sizes, the erased-byte scan and the offset formatter. Define them once with the other fixed-layout structures and import them in both places. The signature slot count is the one constant that drives the slot scan. Correct the observedEnd comment: it is the end of the last chunk read. * fix(artifacts): report ESP-IDF's segment length limit by chip ESP-IDF's verify_segment_header rejects a segment of 16 MiB or more unless the chip maps flash through 32-bit cache addresses (SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP: ESP32-S3, P4, C5, C61, H4 and S31 in v6.1). Record that capability in the chip table and add a limitation for such segments; unknown chips get none. `loaded` repeats should_load. It does not mean the bootloader accepts the segment: verify_load_addresses still rejects one outside the chip's RAM, which REA does not model. Say so in the docs, the skill reference and the inventory limitation. * test(artifacts): check ESP lane tools first and verify extraction The lane started work before checking its tools, so a missing RISC-V target, ld.lld, esptool or espsecure surfaced as a raw spawn or command error. Check each one before building and name the variable to set. Warn when esptool is not v5.6.0, the release the layout facts cite. Assert each segment's class for the ESP32-C3 app, its variants and the bootloader, as the Xtensa image already did. Extract the app and the signed image through the CLI and compare every file with the bytes at esptool's segment offsets or in the appended signature sector; the source's SHA-256 must be unchanged and equal the reported root_sha256. * docs(artifacts): state what the ESP image lane does not establish The Verification section listed the lane's checks but not its gaps: only ESP32-C3 and ESP32 images come from a real producer, no real project build, OTA download or flash dump is inspected, no signature is verified, and the lane does not run in CI. Record the timing of the final lane run on esptool v5.6.0. * fix(artifacts): compare ESP segment declarations Segment entries hash only their data, so compare_artifacts reported a segment that moved to another load address with the same bytes as unchanged; only the image itself changed. Compare each segment's load_addr and data_len as its declaration, as Node SEA and PyInstaller members are compared, and count a changed declaration as a metadata change, also when content is unavailable. Header offsets are layout and the classification is derived, so neither is compared. * test(artifacts): run ESP lane commands as owned process trees The lane ran its tools through execFile. A timeout killed only the direct child, so a linker or assembler that Clang or GCC had started could keep running while the lane removed its workspace. Run every command through createVerifierCommands, check the tools inside withVerifierCleanup, and remove the workspace only after commands.close() has stopped every tree; a failed close keeps the workspace. Only a nonzero exit now counts as a refusal or an expected esptool failure; timeouts and spawn failures propagate. Supervising each command roughly doubles the lane's run time. * test(artifacts): cancel ESP decoding after its first decoded chunk The cancellation test requested cancel or close on setImmediate, which can land while the file is still being opened and checked, so it never showed that decoding had started. It also passed against a reader that ignored both requests while decoding. Acknowledge the decoder's first chunk, then cancel or close, join the operation, and assert that no later chunk reaches the decoder and that the reader ends closed. * docs(artifacts): record which ESP metadata comparison ignores Segments are compared by their declaration. The image's and the signature blocks' format_metadata are not: every field comes from their own bytes, which content comparison already covers, and a block's image_digest_matches follows the image's content. * feat(artifacts): inventory and extract ESP flash layouts A whole-flash dump of an ESP32-family chip was read as its bootloader image, with the partition table, apps and data left as undescribed bytes, or refused when it began with erased flash. Read it as an esp-flash-image container instead. The partition table is decoded as esp_partition_table_verify accepts it and is found at the caller's partition_table_offset, at the default 0x8000, or by scanning 0x1000-aligned sectors for a table whose MD5 verifies, which the inventory reports as derived. The bootloader is located at its chip's ROM bootloader offset. Each partition is an entry with its declaration in flat format_metadata; the bootloader and app partitions carry ESP image facts from the existing streaming decoder; the otadata records are reported with a selection derived without rollback or anti-rollback, and the running partition stays unknown. One unreadable partition does not hide the others, and extraction writes every partition, nvs_keys included, as ordinary data. Flash layouts are detected before single ESP images. The scan runs only when an ESP image sits at a ROM bootloader offset, and is refused when a bootloader at another chip's offset shows that the dump does not start at flash address 0. open_binary gains flash_offset and partition_table_offset, which require format esp-flash-image and are part of the snapshot target identity; the CLI takes them with --target-format esp-flash-image. Comparison treats a partition's label, type, subtype, range and flags as its declaration. * test(artifacts): verify ESP flash layouts with gen_esp32part, merge-bin and otatool Add the optional verify:esp-flash lane. It builds default and relocated partition tables with ESP-IDF's gen_esp32part.py, assembles dumps with esptool merge-bin, including one written from flash 0x1000, and writes OTA records with otatool through flash_file_tool.py, which replaces parttool's esptool calls with reads and writes of the dump file. It compares REA's bootloader and app facts with esptool image-info, every partition with gen_esp32part.py's decoding of the extracted table, and the OTA records and derived slot with otatool read_otadata after each switch. It also checks extracted bytes and source integrity, the scan and caller table offsets, flash_offset on the 0x1000 dump, a corrupted app beside valid partitions and a truncated dump. ESP32 linker scripts place the conformance bootloader and app in ESP32 address ranges. * fix(artifacts): read ESP flash layouts as the bootloader does Read the partition table as bootloader_utility_load_partition_table does: the last factory, otadata and same-subtype OTA declaration wins, and a limitation names repeats. Plan each otadata record within its partition and the dump. An otadata partition smaller than two sectors selects none, the bootloader's INVALID_INDEX; a record the dump lacks leaves the selection unknown while the other record stays reported. The fallbacks follow the bootloader's indexes: ota_0-fallback and factory-fallback name no partition when the table declares none, and the assumptions add GPIO holds and deep-sleep wake. Refuse a dump whose bootloader contradicts its origin before the default table offset is tried, whether flash_offset was omitted or given, and keep the ESP image reader's ESP8266 refusal. A table never lies inside the located bootloader image, so single app images are no longer scanned. An explicit esp-flash-image selection is decided before other container signatures, and the native commands' --target-format no longer takes it. Say when no bootloader confirms the default origin. Exempt the primary bootloader and partition_table entries of ESP-IDF 5.3 from placement limits, report every region inside a longer one, and name partitions by index when labels repeat ignoring case. Keep an undecodable label's bytes in label_hex and compare them. Word undescribed ranges as starting at a byte that is not erased, and treat a short erased region as erased. Add an MCP compare_artifacts test for partition declarations, reader identity and close tests, scan edge cases, the 0x2000 ROM offset and a literal otatool CRC. * test(artifacts): verify ESP primary layout entries and the otadata fallback Build a table with the primary bootloader and partition_table entries that gen_esp32part.py fills from --primary-bootloader-offset, and check that REA reads it with complete coverage, no overlap or misplacement limitation, and a table that agrees with gen_esp32part's decoding. Erase the otadata with otatool after the slot switches and check the records against read_otadata and the factory fallback. * fix(artifacts): keep ESP flash coverage complete for a dump cut short An esptool merge-bin dump ends after the last part it wrote, usually inside its last partition. Such a partition made the layout's coverage unknown, so artifact comparison of two merge-bin dumps reported every unchanged partition as unknown. The partition table names every partition whatever the file's length, so a partition cut short or left out by the end of the file hides no entry. As the ESP image reader does for a cut-short segment, its entry holds the bytes the file has and an entry limitation gives the declared size; a partition with no bytes stays in partitions_outside_dump. Coverage stays complete unless undescribed bytes remain or no bootloader is located. An MCP compare_artifacts test of two dumps that end inside ota_0 now classifies every partition, and fails on the old rule. * fix(artifacts): report written bytes in ESP partitions, erased slots included An app slot is erased for ESP-IDF's bootloader when its image header bytes are erased flash, and REA reported only that status. A slot can still hold data further in: a dump read from a device held a complete older app image 1 MiB into an OTA slot reported as erased, with no limitation. Each partition entry now records where the bytes the dump holds are written, relative to the partition: written_offset, the first byte that is not 0xFF, written_end, just past the last, and written_bytes, how many are not 0xFF. They are null for a partition outside the dump. The status stays erased, which is the bootloader's view, and a limitation names an erased app slot that holds written bytes and says they are not read as an image. The reader reads every partition byte once more to find them. * fix(artifacts): derive the ESP boot selection for every partition table The boot selection lived inside otadata, so a table without an otadata partition, the common factory-only layout, had none. ESP-IDF's bootloader_utility_get_selected_boot_partition returns the factory index when the table declares no otadata partition. Move it to a top-level boot_selection that every layout has. Its basis names the bootloader's branch: otadata, factory-fallback and ota_0-fallback from the records, no-otadata for a table without otadata, none for the bootloader's INVALID_INDEX, and unknown when the dump lacks a record the bootloader reads, which replaces the null selection. running_partition moves with it, and otadata keeps its partition index and records. The assumptions name what each branch leaves out: no records assumption without otadata, and none for none or unknown. The real-tool lane checks no-otadata on a table gen_esp32part built without otadata. * fix(artifacts): say when an ESP image is a flash dump read from 0x1000 An ESP32 dump read from flash 0x1000, as read-flash 0x1000 or merge-bin --target-offset 0x1000 writes it, opened automatically as its bootloader image with a megabyte of undescribed bytes and no next probe. The flash reader refuses that origin, but nothing told an agent which flash_offset to pass. espFlashDumpOrigin owns the inference: an image of a chip whose ROM loads the bootloader from flash R, at file offset F, is the bootloader of a dump read from R - F when a partition table sits where that origin puts 0x8000. The ESP image reader asks it for a non-app image at file offset 0, so the result does not depend on how the reader was created, and records flash_dump {flash_offset, table_file_offset}, null otherwise, with a limitation that says to reopen the file as esp-flash-image with that flash_offset. inspect_artifact proposes the same open_binary call as a next probe, and the flash reader's misplaced-origin refusal names the offset. * perf(artifacts): summarize nested ESP images in the flash layout summary view The summary view returned an ESP flash layout's root facts whole, so each app partition and the bootloader carried every segment and signature block, about half of a dump's summary. The summary view now gives each nested image a summary instead of its facts: status, chip, entry point, segment count, load ranges (each mapped, loaded or unclassified segment's address and declared length), the checksum and appended SHA-256 statuses, coverage and limitations. Complete detail keeps the full facts, every segment included, and the retained Evidence is unchanged. On a 16 MiB device dump the summary's root falls from 5,141 to 2,657 bytes, and the summary Evidence from 10,038 to 7,554. * test(artifacts): expect the misplaced image the ESP bootloader search reports |
||
|
morlutoandCursor Agent
|
c0c3902207 |
fix(native): preserve Mach-O target selection in macOS tools (#7831)
* fix(native): preserve Mach-O target selection in macOS tools * fix(native): retain exact Mach-O slice signature coverage * fix(native): admit x86_64 LIB64 signature selectors * test(native): match signature checks to actual slices * fix(native): recognize vtool refusals on private Mach-O aliases Match vtool's container refusal to the operand the command received so a parenthesized universal path keeps completed facts and the refusal limitation. Expect codesign -a only when the target format is universal. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
|
d39b3000ec |
feat(artifacts): inventory and extract U-Boot FIT images (#7771)
* feat(artifacts): inventory and extract U-Boot FIT images open_binary refused U-Boot Flattened Image Tree files as an unsupported format, so inspect_artifact and extract_artifact could not list or recover a FIT's kernel, devicetree, ramdisk and loadable images. Recognize an FDT v16/v17 with a top-level /images node and parse it with the existing parseDevicetree. Each /images child becomes an inventory entry named images/<unit-name>.bin that extracts its stored bytes from embedded data, data-offset (past the 4-aligned devicetree end) or data-position (from the file start). Image format_metadata records the declared type, arch, os, compression, load and entry; root metadata records the description, timestamp and configurations. crc32, md5, sha1, sha256, sha384 and sha512 hash nodes are verified in one streamed pass per image; other algorithms are reported unverified. Signatures are reported, not verified, and nothing is decompressed, decrypted or executed. Data outside the file, overlapping or duplicate image data, a missing or repeated data source and hash mismatches fail closed. The devicetree is bounded by the fixed 16 MiB allowance; external image data is streamed from disk. Refs #7129 * fix(artifacts): bound FIT configuration references and suggest analysis only for plain images * fix(artifacts): retain buffered FIT stream cancellation * test(artifacts): gate FIT faults on the observed read phase * fix(artifacts): verify FIT links and summary identities * fix(artifacts): bound FIT diagnostics and retain unknown signatures * fix(artifacts): compare FIT declarations and resolved data links * fix(artifacts): admit FIT raw probes through the shared loader contract * fix(artifacts): preserve FIT field availability independently * fix(artifacts): isolate nested entry metadata from reader facts --------- Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> Co-authored-by: gensokyo-dev <342580529+users.noreply.github.com> |
||
|
Entrpiandgensokyo-dev
|
64196b08ca |
feat(artifacts): inventory and extract ESP32 firmware images (#7498)
* feat(artifacts): inventory and extract ESP32 firmware images ESP32-family app and bootloader images (magic 0xE9) had no reader, so open_binary refused OTA downloads, app partitions and bootloaders. Add the esp-image container format behind inspect_artifact and extract_artifact. A pure decoder reads each byte once and keeps only fixed-size fields. Derived facts cover the chip, from a table cited to ESP-IDF v6.1; segment classification as ESP-IDF's should_map and should_load; checksum and appended SHA-256 results; app and bootloader descriptors; Secure Boot v2 signature blocks, observed and unverified; and undescribed trailing byte ranges. Truncation and undescribed bytes make coverage unknown. ESP8266 images are refused with a specific reason, and ciphertext stays unrecognized. Add the format to every target, Evidence, snapshot, live-summary and artifact vocabulary, document it, and add an optional verify:esp-images lane that compares esptool-built images with esptool image-info and espsecure signature-info-v2. Refs #3615 * test(artifacts): run the ESP image lane on esptool v5.6.0 The lane compared descriptor fields only when esptool image-info printed one, so a descriptor that REA reports and esptool does not print passed without comparison. Compare the descriptor kind in both directions. Record that the lane passed for all 13 images with esptool and espsecure v5.6.0, the version the layout facts cite, as well as with 4.12.0. * refactor(artifacts): define ESP image layout sizes once The decoder and the derived facts each defined the segment-header, appended-digest and flash-sector sizes, the erased-byte scan and the offset formatter. Define them once with the other fixed-layout structures and import them in both places. The signature slot count is the one constant that drives the slot scan. Correct the observedEnd comment: it is the end of the last chunk read. * fix(artifacts): report ESP-IDF's segment length limit by chip ESP-IDF's verify_segment_header rejects a segment of 16 MiB or more unless the chip maps flash through 32-bit cache addresses (SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP: ESP32-S3, P4, C5, C61, H4 and S31 in v6.1). Record that capability in the chip table and add a limitation for such segments; unknown chips get none. `loaded` repeats should_load. It does not mean the bootloader accepts the segment: verify_load_addresses still rejects one outside the chip's RAM, which REA does not model. Say so in the docs, the skill reference and the inventory limitation. * test(artifacts): check ESP lane tools first and verify extraction The lane started work before checking its tools, so a missing RISC-V target, ld.lld, esptool or espsecure surfaced as a raw spawn or command error. Check each one before building and name the variable to set. Warn when esptool is not v5.6.0, the release the layout facts cite. Assert each segment's class for the ESP32-C3 app, its variants and the bootloader, as the Xtensa image already did. Extract the app and the signed image through the CLI and compare every file with the bytes at esptool's segment offsets or in the appended signature sector; the source's SHA-256 must be unchanged and equal the reported root_sha256. * docs(artifacts): state what the ESP image lane does not establish The Verification section listed the lane's checks but not its gaps: only ESP32-C3 and ESP32 images come from a real producer, no real project build, OTA download or flash dump is inspected, no signature is verified, and the lane does not run in CI. Record the timing of the final lane run on esptool v5.6.0. * fix(artifacts): compare ESP segment declarations Segment entries hash only their data, so compare_artifacts reported a segment that moved to another load address with the same bytes as unchanged; only the image itself changed. Compare each segment's load_addr and data_len as its declaration, as Node SEA and PyInstaller members are compared, and count a changed declaration as a metadata change, also when content is unavailable. Header offsets are layout and the classification is derived, so neither is compared. * test(artifacts): run ESP lane commands as owned process trees The lane ran its tools through execFile. A timeout killed only the direct child, so a linker or assembler that Clang or GCC had started could keep running while the lane removed its workspace. Run every command through createVerifierCommands, check the tools inside withVerifierCleanup, and remove the workspace only after commands.close() has stopped every tree; a failed close keeps the workspace. Only a nonzero exit now counts as a refusal or an expected esptool failure; timeouts and spawn failures propagate. Supervising each command roughly doubles the lane's run time. * test(artifacts): cancel ESP decoding after its first decoded chunk The cancellation test requested cancel or close on setImmediate, which can land while the file is still being opened and checked, so it never showed that decoding had started. It also passed against a reader that ignored both requests while decoding. Acknowledge the decoder's first chunk, then cancel or close, join the operation, and assert that no later chunk reaches the decoder and that the reader ends closed. * docs(artifacts): record which ESP metadata comparison ignores Segments are compared by their declaration. The image's and the signature blocks' format_metadata are not: every field comes from their own bytes, which content comparison already covers, and a block's image_digest_matches follows the image's content. * docs(artifacts): state that ESP mmu_page_size is a base-2 logarithm The app descriptor's mmu_page_size keeps ESP-IDF's name and value, the MMU page size in log base 2 as esp_app_desc_t declares it, so 16 means 64 KiB. Say so in docs/esp-image.md and the schema comment; a reader could otherwise take 16 as 16 KiB. * fix(esp-image): retain member consumption and share lifecycle regressions * test(artifacts): scope MCP catalogs and budget full-schema validation * test(artifacts): verify compact output profiles and document bounded catalogs * test(artifacts): close readers when cleanup-test admission fails * test(artifacts): compare ESP image limitations with their owners The decoder and reader tests froze complete limitation sentences, so a harmless rewording failed them. The wording now has two owners: ESP_IMAGE_LIMITATIONS in espImageFacts.ts, which the decoder, the inventory and the reader use, and ESP_DESCRIPTOR_LIMITATIONS beside the descriptor decoder. Tests compare delivered limitations with the owner, called with the facts each case establishes (segment index, declared and available bytes, offsets, chip, page range), next to the structured assertions they already make. Delivered text is unchanged. * test(esp-image): assert structured facts and owned guidance * fix(esp-image): keep spi_flash_mode and descriptor fields before a truncated tail ESP-IDF v6.1's esp_app_desc_t declares spi_flash_mode at byte 181, after mmu_page_size, as the committed reference layout shows; the descriptor now keeps it as stored. The decoder also required all 256 descriptor bytes, though it reads none past byte 181, so a file ending in the reserved tail lost the version, project, digest and revision facts it does hold. The app descriptor now needs only its fields; the segment's truncation is reported as before. * test(esp-image): pin the P-192 public key digest to espsecure's layout espsecure packs ECDSA signature blocks as "<BBBx32sB64s64s1031x" and zero-pads a 48-byte P-192 point to 64 bytes, so the signature starts at byte 101 for every SHA-256 curve. signature_info_v2 hashes sig_block[36:101] and _digest_sbv2_public_key the curve ID with the padded point. No test built a P-192 block; this one checks public_key_sha256 against that definition. * fix(esp-image): keep each app descriptor field the file holds in full A file can end anywhere inside the 256-byte esp_app_desc_t. The decoder needed every field byte, so a cut at byte 176 dropped the version, project and app_elf_sha256 it does hold. Once the 4-byte magic identifies an app descriptor, each field is now kept when the file holds all of its bytes and is null otherwise; observed_length records where the descriptor ends, so a cut field is told apart from text that is not UTF-8. The fields after the magic become nullable in the metadata schema. * fix(esp-image): report a signature block whose image digest does not match A CRC-valid Secure Boot v2 block can carry another image's digest. The block recorded image_digest_matches false, but no limitation said so, unlike a checksum or appended-SHA mismatch, so CLI and MCP summaries showed only the generic unverified note. The mismatch is now a limitation; coverage stays complete and signature_status stays unverified. The three other signature-sector limitations move into ESP_IMAGE_LIMITATIONS with their text unchanged, so all ESP image wording has one owner. * refactor(esp-image): remove redundant descriptor byte guards --------- Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> |
||
|
Cipherandgensokyo-dev
|
18a8e71c9f |
feat(artifacts): add U-Boot legacy uImage reader (part of #7129) (#7234)
* feat(artifacts): add U-Boot legacy uImage reader (part of #7129) Recognize legacy uImages by their 0x27051956 magic and inventory them as containers: 64-byte header facts (arch/os/type/compression raw and named, load/entry addresses, name, timestamp) with header and data CRC32 verified, stored payload extractable as payload.bin, and multi-file images split from the stored size table into image-<n>.bin. Compressed payloads stay as stored; nothing is executed or decompressed. Wires the format through classification, the artifact reader dispatch, extraction, and the evidence/snapshot format vocabularies. Related to #7129 (uImage part only); HEX/SREC and FIT are out of scope. * fix(uimage): preserve aligned component ranges and exact header facts * fix(uimage): retain buffered members through consumption and cancellation * fix(uimage): preserve active read and validation failure precedence --------- Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> |
||
|
blankymandgensokyo-dev
|
e78ac1a55b |
feat(artifacts): read UnityFS AssetBundle containers (#7699)
* feat(artifacts): read UnityFS AssetBundle containers inspect-artifact and extract-artifact now recognize the UnityFS signature and treat format versions 6 through 9 as containers. Header facts (format version, player/engine strings, declared size, dataflags, blocks-info bounds and the declared 16-byte data hash) are retained alongside a directory of CAB-* and .resS members. Member bytes stream on demand: only the blocks covering the requested member are read and decompressed with bounded expansion and cancellation. Bundles declaring asset-bundle encryption keep header facts and report unknown directory coverage. LZMA blocks-info reports unknown coverage; LZMA member reads fail as unavailable while the directory stays readable. Malformed counts, out-of-stream members and declared sizes beyond the file fail as format errors. UnityFS classifies by bytes in root detection, inventory graph construction and desktop inspection, which emits a unityfs-header fact. Validation: tsc --noEmit, oxlint (no new findings on touched files), 17 reader tests covering LZ4HC/uncompressed tables, blocks-info at end, format 6 without padding alignment, multi-block members, malformed input, unsupported compression, cancellation and abandoned streams; full artifacts/domain/conformance suites pass. Verified against one real YooAsset bundle (Unity 2021.3.32f1, format 8, LZ4HC): the streamed SerializedFile contained its managed assembly byte-identical to a UnityPy extraction. Other producer variants unverified. * fix(unityfs): own bounded member reads and preserve partial evidence * refactor(unityfs): clarify producer revision flag decisions * fix(artifacts): preserve coverage authority and bound UnityFS block work --------- Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> |
||
|
Aadityaandgensokyo-dev
|
c36971953c |
feat(artifacts): inventory and extract Intel HEX and Motorola S-record regions (#7136)
* feat(artifacts): inventory and extract Intel HEX and Motorola S-record regions open_binary refused Intel HEX and S-record programming files as an unsupported format, so inspect_artifact and extract_artifact could not answer which bytes load at which addresses. Recognize both encodings by a fully valid first record and decode them statically into address-contiguous regions, each an inventory entry named 0x<start>-0x<end>.bin and extractable as raw bytes. Root format_metadata records the encoding and variant, record counts, unique and duplicate bytes, address bounds, termination, declared start address, S0 header and validated S5/S6 count. Gaps are never filled; start addresses are declared facts, not proven entry points. Checksum, record-type, byte-count, line-length, overlap-conflict, address wrap and post-termination errors fail closed with the line number; a missing termination record leaves coverage unknown. Decoding is bounded by a 128 MiB file ceiling (decoded bytes are at most half the file and are reserved after the size check) and 65,536 address-discontiguous runs. Refs #7129 * fix(artifacts): refuse Intel HEX extended address records with a non-zero address * fix(artifacts): bound HEX region copies and retain stream cancellation --------- Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> |
||
|
morluto
|
ace4dbf36e |
fix(contracts): align live target classification with admitted facts
Share exact live target variants across open and session responses, preserving historical Evidence and snapshot admission. Consolidate equivalent ELF metadata aliases at their owner. |
||
|
morluto
|
c27560e434 | fix(artifacts): classify selected package names | ||
|
morluto
|
779d548c4e | fix(artifacts): preserve selected ASAR companion paths | ||
|
Ariel Fikru Avicennaandgensokyo-dev
|
cab78de506 |
feat(ghidra): analyze GameCube/Wii DOL executables through Ghidra (#6240)
* feat(domain): parse and validate GameCube/Wii DOL headers Add a pure parser for the 0x100-byte DOL header (7 text and 11 data slots, BSS, entry point). Used slots must lie after the header and inside the file and the 0x80000000..0x81800000 load window; text slots are 4-byte aligned; sections may not overlap in memory or in the file; the entry point must be inside a text section. BSS may overlap data sections, as .sdata/.sdata2 do in shipped DOLs. Refs #1741 * feat(binary): admit caller-selected GameCube/Wii DOL targets * feat(ghidra): import GameCube/Wii DOL targets through a section-map pre-script * test(ghidra): cover DOL launcher arguments, attestation and raw profile digests * test(ghidra): add the real verify:ghidra:dol lane and DOL docs * fix(ghidra): read only the DOL header and section ranges instead of the whole file * test(ghidra): derive DOL lane addresses from readelf and admit Gekko from the install catalog * fix(ghidra): attest DOL block permissions, space, overlay and load state * fix(ghidra): keep load-image permissions optional for older inspections * fix(contracts): accept DOL targets in binary_session output * fix(ghidra): adapt the DOL lane to the current upstream address schema Validate DOL load-image coordinates with the upstream observed-address schema, drop a duplicate import left by the rebase, and re-pin the raw profile digests to the unmodified upstream resolver. * fix(ghidra): preserve bridge inventory symbol provenance * fix(ghidra): align DOL verification with shared workflow contracts * feat(domain): parse and validate GameCube/Wii DOL headers Add a pure parser for the 0x100-byte DOL header (7 text and 11 data slots, BSS, entry point). Used slots must lie after the header and inside the file and the 0x80000000..0x81800000 load window; text slots are 4-byte aligned; sections may not overlap in memory or in the file; the entry point must be inside a text section. BSS may overlap data sections, as .sdata/.sdata2 do in shipped DOLs. Refs #1741 * feat(binary): admit caller-selected GameCube/Wii DOL targets * feat(ghidra): import GameCube/Wii DOL targets through a section-map pre-script * test(ghidra): cover DOL launcher arguments, attestation and raw profile digests * test(ghidra): add the real verify:ghidra:dol lane and DOL docs * fix(ghidra): read only the DOL header and section ranges instead of the whole file * test(ghidra): derive DOL lane addresses from readelf and admit Gekko from the install catalog * fix(ghidra): attest DOL block permissions, space, overlay and load state * fix(ghidra): keep load-image permissions optional for older inspections * fix(contracts): accept DOL targets in binary_session output * fix(ghidra): adapt the DOL lane to the current upstream address schema Validate DOL load-image coordinates with the upstream observed-address schema, drop a duplicate import left by the rebase, and re-pin the raw profile digests to the unmodified upstream resolver. * docs(ghidra): qualify the DOL memory window * refactor(ghidra): own DOL BSS ranges in the parsed header * docs(ghidra): record real DOL workflows and task costs * docs(ghidra): record final DOL integration verification * fix(mcp): share the native procedure search row contract * docs(ghidra): record public procedure search verification --------- Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> |
||
|
Lewis WangandClaude Opus 5.5
|
861e17a3a7 |
fix(native): admit thin arm64_32 Mach-O targets for static inspection (#5788)
* fix(native): admit thin arm64_32 Mach-O targets for static inspection Target admission mapped Mach-O CPU types only to x86, x86_64, arm and arm64, so a thin arm64_32 object was rejected as "unsupported Mach-O architecture" before list-architectures, inspect-macho or inspect-signature could run, although the same slice inside a FAT file was inventoried. Recognize CPU_TYPE_ARM64_32 as its own Mach-O-only architecture with a dedicated thin target variant, so ELF, PE and raw-binary paths cannot carry it. Add its slice policy so 32-bit dispatch metadata refuses it like other 32-bit Mach-O images instead of reporting a CPU mismatch. Ghidra, Hopper and IDA have no verified arm64_32 lane and decline it explicitly; Hopper's declined families now share one predicate. Fixes #5713 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(apple): run the thin arm64_32 CLI regression on macOS The regression compiles an arm64_32 object with Apple clang, so it skips on the Ubuntu jobs. List it in the macOS Apple artifact job and its change scope so the job runs it and changes to it select that job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5fa4defcbc |
feat(artifacts): inspect and extract Godot PCK with retained facts (#3634)
* feat(artifacts): read Godot PCK packs Add a Godot Engine PCK artifact reader for standalone packs and packs embedded behind an executable trailer (formats 2 and 3). Members are uncompressed byte ranges verified against their declared MD5 on read. Encrypted directories, sparse bundles, patch removal entries, unknown flags and format 1 fail closed as unavailable; encrypted members are reported without being read. inspect-artifact and extract-artifact recognize standalone packs by their GDPC magic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(artifacts): bound Godot PCK metadata and own member streams Cap member paths at 4096 bytes and the retained directory at 64 MiB before allocation. Read members with positional chunk reads through pipeline so cancellation is typed, abandoned consumers release their source, and the shared descriptor stays reader-owned. Retain the root descriptor for godot-pck inventory, as for ZIP and universal Mach-O. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(artifacts): complete Godot PCK facts, integrity and selected views * fix(artifacts): preserve atomic-root inventory Evidence * fix(artifacts): bound aggregate PCK member expansion --------- Co-authored-by: Samucahub <151155051+Samucahub@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> |
||
|
Prasad Navale
|
3c1cce78ee |
fix(domain): admit Qualcomm Hexagon (EM_HEXAGON, 164) ELF binaries (#4819)
* fix(domain): admit Qualcomm Hexagon (EM_HEXAGON, 164) ELF binaries Resolve missing ELF admission case for Qualcomm Hexagon DSP targets (e_machine = 164, EM_HEXAGON). - Add 'hexagon' to BINARY_ARCHITECTURES and HeaderArchitecture - Map EM_HEXAGON (164) in elfArchitecture() to 'hexagon' - Include 'hexagon' in GhidraProvider's SUPPORTED_ARCHITECTURES - Gracefully decline 'hexagon' with Ghidra-directed diagnostic in Hopper and IDA - Add unit tests for Hexagon ELF parsing and adapter admission checks Fixes #4781 * fix(hopper,ghidra): add Ghidra guidance to Hopper Hexagon refusal and add Ghidra admission regression test |
||
|
zhugehyukandClaude Opus 5.5
|
d62d3082c2 |
feat(ghidra): support raw big-endian m68k analysis (#4018)
* feat(ghidra): support raw big-endian m68k analysis Admit `m68k` for headerless raw-binary targets. Admission requires a 2-byte aligned entry and a mapping within 0x0..0xffffffff, and Ghidra imports with BinaryLoader using 68000:BE:32:default, the default compiler specification, and the `ram` default address space. `6502` and `m68k` now share one RAW_ONLY_ARCHITECTURES owner, so detected targets, Evidence subjects, snapshots, and lifecycle output contracts exclude both instead of repeating a 6502 literal. Ghidra's default 68000 language decodes the 68040 superset; separate 68020/68030/CPU32/ColdFire variants, Amiga/Atari containers, and reset vector discovery remain unsupported. Windows Ghidra P0 still admits raw 6502 only. Verified with Ghidra 12.1.4 on macOS against an 8-byte self-built fixture (nop; nop; moveq #1,d0; rts at 0x1000): load image verified in `ram`, entry decompiles to `return 1`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(ghidra): verify raw m68k analysis with real Ghidra Add npm run verify:ghidra:m68k. The verifier loads a source-owned 22-byte 68000 fixture (annotated hex, no assembler) at 0x1000 and checks, through the compiled CLI and stdio MCP server: the 68000:BE:32:default BinaryLoader import, load-image parity, exact mapped bytes, both functions, the bsr.b call reference, decompilation by address and file offset, rejection of a misaligned entry and of a mapping past 0xffffffff without replacing the active session, unchanged source bytes, and runtime cleanup. Run with Ghidra 12.1.4 and JDK 21 on macOS arm64. Linux has not been exercised; Windows Ghidra P0 does not admit m68k. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ysopwnable
|
2b5ee72d26 |
feat(ghidra): admit ARM PE targets on Windows (#3965)
Admit Windows ARMNT/Thumb-2 and ARM64 PE targets using preserved COFF machine facts. Keep unsupported machine gates explicit and distinguish loader-selected versus configured language provenance. Verified with real Windows Ghidra 12.1.4 CLI/MCP workflows, ARM fixtures and Microsoft-signed Coreinfo ARM64 / MSVCR120 ARM inputs. Resource limits, analyzer overrides, startup timeout observations and verification gaps are documented in the PR. Final CI and automated review passed. Fixes #2186 |
||
|
morlutoandAurelio Bernal
|
9387282274 |
fix(ghidra): reconcile GNU MIPS ABI declarations
Parse bounded GNU file-scope FP/MSA attributes once, retain absent versus uninspected declarations, reject conflicts and unsupported argument ABIs, and include observed facts in profile identity. LE/BE filesystem and profile regressions pass; real Ghidra/MIPS behavior remains unverified. Fixes #3096 Co-authored-by: Aurelio Bernal <aurelio@bernalr.com> |
||
|
Ariel Fikru Avicennaandmorluto
|
c86b09a109 |
feat(ghidra): admit 32-bit big-endian PowerPC ELF (#1747)
* feat(ghidra): admit 32-bit big-endian PowerPC ELF Recognize EM_PPC as a provider-neutral `powerpc` family that keeps its ELF class, byte order, type and flags, and admit ELF32 big-endian ET_EXEC through a bounded Ghidra lane. ELF64, little-endian, relocatable/shared objects and EF_PPC_EMB are refused with a specific reason; Hopper and IDA do not gain PowerPC support, and Windows Ghidra P0 is unchanged. Ghidra's ELF loader selects PowerPC:BE:32:e500:default for a plain EM_PPC header, which misreads 750-family code. The lane binds PowerPC:BE:32:default/default in the profile (language_selection "admitted-lane-v1"), imports with it and checks the reported language. A language set through REA_GHIDRA_LANGUAGE_ID still replaces the lane default. Add a source-owned fixture, fixed-encoding probe, GNU readelf cross-check and the optional `verify:ghidra:ppc` real-provider lane, plus docs/ghidra-ppc.md. Refs #1741 * style(ghidra): format shared PowerPC profile integration * fix(ghidra): preserve caller-selected PowerPC languages --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> |
||
|
|
729ca2ab6a |
fix(ghidra): report unapplied PE CodeView PDB limitation (#2283)
* fix(ghidra): report unapplied PE CodeView PDB limitation (#2259) When Ghidra analyzes a PE target that references companion debug symbols in its CodeView (IMAGE_DEBUG_TYPE_CODEVIEW / RSDS or NB10) debug directory, Ghidra's headless analyzer cannot automatically locate or apply the companion PDB from isolated runtime directories without companion admission. Procedure names remain analysis-generated (FUN_<address>). Parse CodeView debug metadata (RSDS GUID/age and NB10 age) during PE target resolution, and report an Evidence limitation informing callers and agents that referenced debug symbols were not applied and procedure names are analysis-generated. Fixes #2259 * test(pe): group CodeView header coverage separately * fix(ghidra): avoid contradictory PDB limitations * fix(android): make DEX descriptor pattern portable * fix(ci): isolate Linux Hopper process fixture inventory * fix(comparison): recheck historical calls-only changes (#2223) * fix(comparison): preserve incomplete function call evidence Keep missing, unknown, partial and unresolved call relationships unknown in function comparisons and reconstruction claims. Aggregate IDA caller and callee coverage without overwriting uncertainty. Validate with 144 focused tests, typecheck/lint, docs checks and test discovery. Real providers were not verified; the full headless IDA session suite requires unavailable Windows native controls. * fix(comparison): recheck historical calls-only changes --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> * style: format identity comparison and IDA runner --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> Co-authored-by: Lucien <164857476+dakjdakd@users.noreply.github.com> |
||
|
记得吃中饭
|
9c14999efd |
feat(ghidra): support raw NMOS 6502 analysis (#2079)
* feat(ghidra): admit raw NMOS 6502 analysis * test(ghidra): require verified POSIX verifier cleanup * test(ghidra): verify 6502 low-memory raw imports * docs(skill): document raw NMOS 6502 support * fix(ghidra): preserve requested Windows raw source paths * docs(ghidra): include raw 6502 in Windows installation scope * test(ghidra): verify Windows raw junction rejection through CLI and MCP * fix(validation): make fixtures portable and preserve explicit Codex home * fix(ghidra): attest raw mapping state and address space * fix(ghidra): align diagnostics and regression groups * fix(ghidra): bind raw attestation to language address space * chore(ghidra): avoid verifier insertion conflict * docs(ghidra): align linked raw admission guides * fix(ghidra): retain 6502 verifier workspace on failure * test(ghidra): isolate Windows authority doctor fixture * fix(binary): re-admit Windows raw paths on session reopen * test(binary): require fresh provider execution after path change * Constrain 6502 to raw lifecycle and snapshot targets * Bypass saved reads for unadmitted Windows paths * Guard direct CLI snapshot replay on Windows paths * fix(snapshot): gate Windows Ghidra replay on native admission * fix(snapshot): constrain nested 6502 subjects to raw images * test: make dyld slice fixtures portable |
||
|
Ysopwnableandfactory-droid[bot]
|
a86deecac5 |
fix: align MCP archive contracts and documentation (#2478)
Reuse the domain ZIP format list for session schemas, verify full Harmony CLI/MCP lifecycles, and correct native view guidance and audited docs drift. Refresh locked dependency notices and gate inventory/provenance consistency, including notice inputs in the cached documentation gate. Fixes #2454 Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> |
||
|
The one -Niuandmorluto
|
0baad58e60 |
feat(ghidra): admit RISC-V ELF analysis through a pinned generic lane (#2151)
* feat(ghidra): admit RISC-V ELF analysis through a pinned generic lane Refs #2063. Recognize `EM_RISCV` (243) as a provider-neutral family that carries its own ELF facts, and bound the first Ghidra lane to little-endian `ET_EXEC` images that declare the double-float ABI, pinned to `RISCV:LE:32:default` or `RISCV:LE:64:default` with the `gcc` compiler specification. Ghidra 12.1.4 is not reliable without the pin: its `riscv.opinion` pins `variant="default"` for the 64-bit constraint and leaves the 32-bit constraint with the comment that the loader "must choose between 'default' and 'AndeStar' variants". Measured on the installed release tree, a source-owned generic RV32 ELF with no Andes declaration imports as `RISCV:LE:32:AndeStar_v5:gcc`, while RV64 resolves to `RISCV:LE:64:default:gcc`. The lane therefore states the language per width and refuses any caller-selected processor outside it instead of inheriting the header guess. Refusals name the declaration rather than falling back to a guessed profile: big-endian images (Ghidra 12.1.4 ships no big-endian RISC-V language), non-`ET_EXEC` images, `EF_RISCV_RVE`, `EF_RISCV_TSO`, and every float ABI other than double, which the pinned generic compiler specifications model. The raw e_flags bits stay in Evidence as declarations, not as instruction semantics. Family recognition now has one owner for the ELF-metadata families: `ELF_METADATA_ARCHITECTURES`, `isElfMetadataArchitecture`, and `HeaderArchitecture` replace ten `Exclude<BinaryArchitecture, "mips">` sites that would each have needed a second hand-edited member. `ghidraPinnedLanguage` is that owner for the import commitment: the session readback previously re-derived only the MIPS pin, so a profile committing any second pinned language failed with a language-mismatch error before the bridge started. Hopper and IDA refuse RISC-V by name, matching the MIPS wording. Caller-supplied raw-binary entry keeps its existing four-family list; this lane is ELF-header admission only. Verification on macOS 15.7.7 arm64, Node 24.18.0, Ghidra 12.1.4 PUBLIC and JDK 21.0.12.1, with the host decompiler built from the release sources for `os/mac_arm_64`: `npm run check` passes; `npm run test:fast` passes 396 test files; `npm run verify:test-discovery` discovers all 917 test files exactly once. Source-owned RV32 and RV64 double-float `ET_EXEC` fixtures complete `rea analyze --provider ghidra` and `rea decompile rea_riscv_entry`, committing `RISCV:LE:32:default` and `RISCV:LE:64:default` with `language_selection: provider-pinned-v1`; the soft-float RV64 shape is refused with the named float-ABI reason, and the full refusal matrix including the RV32 and width-mismatch shapes is covered by unit tests. No `verify:ghidra:riscv` lane script is included in this change; the runs above were performed manually against the installed engine. Fork pull requests receive no CI check runs, so the real-provider scope above is the evidence available for review. * fix(ghidra): bound the RISC-V lane by declared flags and document it Address review feedback on #2151. `e_flags` bits outside `EF_RISCV_RVC | EF_RISCV_FLOAT_ABI` are now refused with the offending value named, so a reserved or newer producer declaration cannot enter the verified generic lane by falling through the RVE, TSO and float-ABI checks. Admission reads `e_type` only, so the refusal text no longer calls `ET_EXEC` static: a fixed-address declaration does not establish static linkage, and `PT_INTERP` and dynamic dependencies are not inspected. The session records that boundary as a limitation rather than inferring a load model. Add `docs/ghidra-riscv.md` in the shape of the MIPS lane guide (admission boundary, refusal table, why the language is pinned per class, Evidence fields, and what the fixtures do not establish), and correct `docs/firmware-analysis.md`, which still told users that MIPS and RISC-V decompilation had not been added. Verification after the change: `npm run check` and `npm run docs:check` pass; the RISC-V module tests pass (17 lane cases plus 9 parser cases); on Ghidra 12.1.4 with JDK 21.0.12.1 the RV32 and RV64 double-float fixtures still analyze and decompile with `RISCV:LE:32:default` / `RISCV:LE:64:default` and `language_selection: provider-pinned-v1`, the soft-float fixture is still refused by name, and the new linkage limitation appears in the emitted Evidence. * fix(ghidra): align RISC-V pinning with admission Reuse the complete lane admission boundary when selecting a language, retain unsigned unsupported flag bits in diagnostics, and derive float ABI metadata from the source declaration. * test(ghidra): verify RISC-V CLI and MCP workflows Build source-owned RV32/RV64 fixtures with Clang and LLD, establish independent ELF and symbol facts with LLVM readelf, and verify real provider evidence, instruction bytes, calls, decompilation, schema projections, snapshot replay, refusal diagnostics, source immutability and cleanup. Verified on macOS ARM64 with Ghidra 12.1.4, JDK 21 and LLVM/LLD 23.1.2. Dynamic linkage, big-endian, Andes and RVE/TSO workflows remain unverified; ISA attributes are not inspected. --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> |
||
|
akushonkamen
|
10400fdb57 |
fix(ida): thin the selected FAT slice into the private headless input (#2201)
Universal Mach-O admission reported the selected slice architecture while the IDA provider's private input copy handed the whole FAT container to upstream idalib, which then picked its own slice (x86_64 for /bin/cat) while Evidence labeled the session arm64. FAT parsing now exports the selected slice's byte range as universalSlice, and the private input copy writes exactly those bytes: the whole source digest still anchors the copy against admission-time changes, and the thinned result anchors the post-operation identity check. Thin targets keep the whole-file copy, and mach-o-universal remains the container format across Evidence, snapshots, and provider routing. Validation: typecheck, oxlint (no new warnings), oxfmt, and targeted vitest runs — ida adapters 12 files/116 tests, domain and services projects 60 files/736 tests, ghidra/hopper behavior 2 files/50 tests, filesystem and universal-extraction boundary 2 files/49 tests, native signature MCP 1 file/2 tests, plus a new thinning suite (5 tests, red before the fix: 3 failed). The thinned product was verified byte-identical to the `clang -arch arm64` input of a `lipo -create` universal fixture. Controlled fixtures only; no live IDA analysis was performed for this change. Closes #2139 |
||
|
morluto
|
b38429cb6a |
fix(native): preserve universal Mach-O container identity
Keep FAT container format distinct from selected slice architecture through target admission, Evidence, snapshots, and CLI/MCP lifecycle results. Preserve native, Hopper, Ghidra, extraction, and dylib routing for the Mach-O family. Reject selected FAT ranges overlapping their table or too short for a Mach-O header. Replace empty-slice fixtures with in-bounds executable headers and cover thin, universal, single-slice, FAT64, and swapped containers. Validation: static and architecture checks, generated docs, targeted producer and public MCP tests. Provider routing uses controlled fixtures; no live Hopper, IDA, JADX, or Ghidra analysis was performed for this change. |
||
|
|
e9fe6437a7 |
feat(ghidra): admit verified MIPS16 shared ELF images
Admit the verified ELF32 o32 MIPS32r2 MIPS16 shared-image profile after complete source load and relocation inspection. Preserve caller-selected bytes, pin the endian-matched language, and report per-address decoder context. Refuse unverified relocation shapes with explicit reasons. Verified on Linux x64 with Ghidra 12.1.4: eight source-owned public CLI/MCP controls, original-artifact instruction/pseudocode/export replay, snapshot restore and owned cleanup. 152 integrated focused tests, static checks and documentation checks passed. Fixes #1851 Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> |
||
|
Matt Van Hornandmorluto
|
7c207c7951 |
feat(native): admit raw binaries with caller-supplied architecture and entry (#1868)
* feat(native): admit raw binaries with caller-supplied architecture and entry * fix(native): honor raw-binary preparation and entry identity * fix(ghidra): stream raw snapshot attestation with cancellation * fix(native): preserve and validate raw lifecycle entries --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> |
||
|
BaaRaiandmorluto
|
f2150eb9a7 |
feat(harmony): project HarmonyOS package inventory evidence (#1447)
* feat(harmony): project HarmonyOS package inventory evidence First HarmonyOS increment from #1431: HAP, HSP, and App Pack roots are classified through the hardened ZIP reader and projected into an execution-free application graph with packaging-model hints, exact component paths and hashes, App Pack children by path, runtime-family hints, and path-based N-API bridge hypotheses. HarmonyOS .har stays unclassified by suffix because .har also names HTTP Archive JSON. Adds the project_harmony_application_graph tool (CLI and MCP), synthetic composition coverage, a pinned real-package verify lane, and docs. * refactor(domain): share application graph projection scaffolding The HarmonyOS projection mirrors the Android and Apple projections by design, which pushed jscpd duplication over its 2.5 percent budget. Extract the identical pieces (inventory input schema, projected component schema and occurrence flattening, bridge candidate schema factory, coverage assembly, comparator, canonical digest) into domain/mobileApplicationGraphProjection.ts and route all three projections plus the composition fixtures through it. Duplication drops from 2.52 to 2.49 percent with no behavior change: projection ids, classifications, and limitations are byte-identical. * fix(harmony): address review, stable app-pack roots, and rotated fixture - rebased onto main; the shared projection scaffolding now keeps main's nativeRuntimeConvention loop and the harmony runtime families adopt the same react-native/flutter/unity path conventions - the .app App Pack suffix is now a root-input rule only (verified ZIP magic in classifyRootFormat and BinaryTargetResolver); nested .app ZIP members inside other packages keep the generic zip classification, with a regression test, so existing inventories do not change - the signing limitation now states that an empty signing list does not mean unsigned because packaging-tool signatures live in the ZIP signing block, and the app-pack root rule is disclosed as suffix-based - repinned the real-package fixture to the current VHome 2.6.14-beta asset after the upstream deleted v5.0.111-beta and replaced the v2.6.14-beta asset within hours; the pin records both observed digests --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> |
||
|
Ysopwnable
|
d6f698d8af |
fix(ghidra): admit carved MIPS ELF with incomplete section tables (#1629)
* fix(ghidra): inspect carved MIPS ABI declarations without missing sections * test: share filesystem fixtures and cleanup assertions |
||
|
Jay-Alexander Elliotandgithub-actions[bot]
|
bddddccdee |
feat(native): admit bounded MIPS ELF analysis through Ghidra (#1362)
* feat(native): admit bounded MIPS ELF analysis through Ghidra Retain MIPS ELF identity and interpretation through Evidence and snapshots. Keep provider admission bounded to ELF32 ET_EXEC o32 MIPS32r2, without implicitly widening Hopper, IDA or Windows P0 coverage. Add source-owned fixtures and an optional real-provider verifier; real Ghidra verification remains pending in this contribution environment. Refs #718 * fix(ghidra): bound MIPS floating-point mode admission Reject explicit EF_MIPS_FP64 and EF_MIPS_NAN2008 declarations until their semantics have a real-provider verification lane. Add PSP PRX/Allegrex, o64/EABI and floating-point-mode refusal regressions. Document that the header gate is not a full .MIPS.abiflags check and advance the profile lane identity to invalidate older interpretation bindings. Validation: 24 standalone checks against the transpiled production profile module passed; the two new flag cases failed before this change. Clang 17 and LLD compiled the source-owned fixture for both endian targets. The complete Vitest/typecheck suite and real Ghidra verification remain pending. Refs morluto/rea#718; PSP support remains follow-up morluto/rea#1330. * fix(native): validate MIPS ABI records before Ghidra admission Read bounded ELF32 PT_MIPS_ABIFLAGS and SHT_MIPS_ABIFLAGS records through the already-open target handle. Preserve declared ABI fields, reject conflicting/duplicate/truncated records, and keep missing records unknown. Wire inspection into target resolution before provider selection. Require compatible declarations for the deliberately narrow Ghidra lane and bind all ABI fields into profile identity (v3) for snapshot invalidation. Clang 17 produced eight source-owned fixtures spanning both byte orders and FPXX/FP32/FP64/soft-float modes with identical ELF header flags. The new reader and policy distinguish them. 24 focused regression test bodies passed after TypeScript transpilation using node:test in place of Vitest; strict isolated TypeScript checking of the reader and target/result types passed. These are limited checks, not full Vitest/project typechecking or real-Ghidra verification, which remain pending. Advances morluto/rea#718. PSP/Allegrex remains morluto/rea#1330 follow-up. * fix(native): repair MIPS integration boundaries and test resolver behavior Keep ELF targets out of Hopper Mach-O preparation and scope existing PE fixture types correctly. Move ABI tests into the filesystem boundary lane, use shared cleanup, add resolver regressions, apply the pinned formatter, and register the real MIPS verifier through the existing package-command pattern. Validated with repository-pinned Node/npm, check:ci, actual focused Vitest, discovery and docs:check. Real Ghidra conformance remains pending. * fix(test): honor caller-selected JDK in real MIPS verification * test(native): strengthen MIPS Ghidra conformance Cross-check the source-owned fixture with bounded GNU readelf inspection, compare production ELF/ABI interpretation and independent symbol addresses, and verify effective Ghidra language/compiler identity through existing public operations. Add a source-owned fixed-encoding probe for immediate and conditional branch checks plus global/string byte assertions. Add 16 assertion/producer-boundary regression cases without widening production admission, changing provider contracts, or introducing proprietary dependencies. Local focused suite: 99 passed, one existing root-permission test skipped. Typecheck, lint/module boundaries, formatting and 700-file discovery pass; full hosted checks and real Ghidra verification remain pending for this checkpoint. * fix(ghidra): preserve native-tool architecture checks after MIPS integration The conflict-free upstream merge failed TypeScript compilation because includes on the union of MIPS-only and other architecture arrays requires never. Use equality-based membership without casts or changing accepted native hosts. Extend the existing native-decompiler rejection table with a MIPS executable at a Linux x64 tool path; retain upstream FAT and unknown-header behavior. Hosted verification follows this commit; no success is claimed yet. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Kaoru0822andClaude Opus 5.5
|
418a5562f6 |
fix(native): keep the resolved bundle plist and permission failures
- Carry the Info.plist that declared the program file on the target as bundleInfoPlist, so inspect_plist defaults to it even when the program file is a symlink into another bundle directory. - Accept a readable Info.plist symlink, as reading the plist directly did. - Treat only a missing path as an absent layout or wrapper; a permission denial now reaches the resolver's permission-denied classification. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
N0zoM1z0
|
5bcb6645b5 |
feat(ghidra): admit explicit DOS COM analysis (#702)
* feat(ghidra): admit explicit DOS COM analysis Carry caller-selected COM interpretation through CLI, MCP and target identity, seed real-mode entry/context before Ghidra analysis, and verify complete loaded bytes, source mappings and measured register context. Keep original binaries unchanged and PSP/runtime coverage explicit. Assisted-by: Nitori (gensokyo-skills:nitori-reverse-engineering) * fix(verification): validate COM package selection before allocating fixtures |
||
|
N0zoM1z0
|
4f099a10f8 |
feat(ghidra): add DOS MZ analysis and complete function extents (#556)
Validate DOS MZ load modules and import them with Ghidra's 16-bit real-mode profile. Preserve complete observed inclusive function body ranges through inventory, function analysis, Evidence and snapshots. Add a public source-generated DOS CLI/MCP verification lane and regressions for malformed headers, body membership and cache profiles. Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing) Assisted-by: Suika (gensokyo-skills:suika-scatter-gather-planning) |
||
|
morluto
|
b8a9da0c0f |
refactor(mcp): make analysis states unrepresentable (#468)
* refactor(config): parse capability settings into exact states * refactor(runtime): encode replay and process outcomes exactly * refactor: make analysis states unrepresentable * refactor(mcp): register canonical schemas with SDK * fix: preserve capability and catalog invariants * fix(build): remove stale compiled modules * fix(replay): update real verifier policy dependency * fix(package): verify the complete MCP catalog * fix(test): supply complete replay reload policy * refactor(contracts): centralize output schema lookup * refactor(mcp): delegate protocol boundaries to SDK * fix(browser): approve real source capture * fix(mcp): preserve catalog guidance |