* fix(firmware): preserve symlink target bytes
* fix(apple): preserve undecodable dylib link targets
* style(tests): fix CI formatting
* test(firmware): select the named contract directly
* test(firmware): record raw bytes for a decodable symlink target
* test(mcp): assert bundle path issues by field
compare_bundles reports each relative path as an invalid_format issue
instead of one joined diagnostic sentence.
* test(firmware): stat raw links without following them
access follows a symlink, so a published dangling raw-link still looked
absent. lstat and readlink inspect the link itself.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* refactor(macho): extract non-throwing container and slice load-command readers
Split readMachoImage into readMachoContainer, which classifies a file as
thin, FAT or not Mach-O and keeps FAT range issues per slice, and
readSliceLoadCommands, which reads one slice header and frames its load
commands, returning every command framed before a failure.
readMachoImage keeps its strict policy on top of them: commands framed
before a framing defect still decode first, an unsupported FAT slice
still yields to a later malformed one, and the first FAT range issue is
still reported. A characterization test pins that precedence for inputs
with several defects; it passes against both the previous and the new
implementation.
Part of #4306.
* feat(macho): report per-slice encryption status in inspect_macho
inspect_macho now reports the LC_ENCRYPTION_INFO(_64) state of every
slice in a new `encryption` collection, read in process from load
commands without reading code bytes.
- Each slice is encrypted, not-encrypted, absent, malformed or unknown.
Absent requires a completed load-command walk; malformed and unknown
carry a reason and keep commands read before the failure.
- The named range is end-exclusive in slice and input-file coordinates
and maps to file-backed segment fragments, VM addresses and the
overlapping nonempty sections.
- A ciphertext range adds a limitation naming its input-file bytes.
- The in-process read and the native tools are bracketed by one regular
file descriptor and stat identity checks; a change reports
artifact_changed. A missing or inaccessible target keeps the tool
facts and reports encryption as non-exhaustive, never absent.
- Retained Evidence recorded before this field is parsed by the owning
schema and reports encryption as unknown in the macho view and in
managed/native verification.
Fixes#4306
* fix(macho): keep slice facts and failures under large tables and close errors
- Frame load commands through a visitor instead of retaining every
command, restoring the pre-refactor memory profile (2M eight-byte
commands: peak heap 370 MiB -> 0.9 MiB).
- Confirm a large FAT container from a valid slice's Mach-O header so a
sibling's range fault stays per slice instead of hiding healthy slices.
- Name only sections whose bytes the range and their segment both cover.
- Throw inspection failures inside the owned descriptor callback so a
close failure keeps them as its primary error.
* fix(macho): map encrypted ranges only within segment VM extents
mapRange gave every file-backed segment fragment VM addresses, even for a
segment with no VM size, whose file bytes dyld never maps, and past vmsize
when a segment declared more file bytes than memory.
Read vmsize with the segment. A segment with no VM size keeps its fragment
with null VM addresses. A file size beyond the VM size, or a VM range past
the address space, makes the mapping unknown with a reason while the slice
keeps its encryption status.
The docs also claimed the encryption read never reads code bytes. Like
every Mach-O container read, a 0xcafebabe input of at most 16 MiB is read
whole to rule out a JVM class file; say so.
* fix(macho): read encryption through the admitted descriptor facts
Main's withArtifactRegularFile now hands callers the admitted handle and
its initial stats. Use those facts instead of re-stating the handle, and
pass the legacy Evidence payload straight to createEvidence.
* fix(macho): retain a finished report when the target close fails
The file owner keeps the completed callback value when the descriptor close
fails or cancellation arrives during close. inspect_macho now validates that
value and retains the report as the error's partial observation, as
inspect_signature does, instead of returning only the cleanup or
cancellation error.
* test(native): supply readEncryption in merged dyld refusal cases
main's new dyld refusal cases build NativeMachoInspectionContext without
the encryption reader this branch requires.
* test(views): bind the Mach-O encryption page to its facet
main's facet-ownership case builds inspect_macho Evidence without the
encryption coverage this branch requires; the fixture now carries one
slice row and the loop checks the encryption page like the other facets.
* fix(macho): bound encryption commands and range mapping across slices
A slice retains at most two encryption load commands, since a second one
already makes it malformed; the reason keeps the full count. Range mapping
in every slice shares one budget of segment and section records, charged
from each segment command's capacity before it is decoded. A slice past
the budget keeps its encryption status and reports an unknown mapping.
* style(macho): format encryption reader and binding test
* fix(macho): stop retaining segments past the mapping budget
Segment commands are collected during framing only while their mapping
records still fit in the budget shared by all slices, and the budget is
charged only when the slice maps its range.
* docs(macho): clarify retained mapping allowance
* fix(ci): own the Mach-O encryption test in Apple scope
---------
Co-authored-by: t <t@t>
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
TYPE_NAMES stopped at flat_dt (8), while the U-Boot include/image.h
commit that the tables are pinned to defines image types 0-47. A header
declaring a later type, such as kernel_noload (14) or tfa-bl31 (45),
was reported with type_name "unknown".
Add types 9-47 with their mkimage -T short names from boot/image.c at
the same commit. IH_TYPE_LOADABLE (22) has no short name there and is
named "loadable", as other tables already name identifiers U-Boot does
not list. Identifiers past the enum stay "unknown".
* feat(artifacts): inventory and extract U-Boot FIT images
open_binary refused U-Boot Flattened Image Tree files as an unsupported
format, so inspect_artifact and extract_artifact could not list or
recover a FIT's kernel, devicetree, ramdisk and loadable images.
Recognize an FDT v16/v17 with a top-level /images node and parse it with
the existing parseDevicetree. Each /images child becomes an inventory
entry named images/<unit-name>.bin that extracts its stored bytes from
embedded data, data-offset (past the 4-aligned devicetree end) or
data-position (from the file start). Image format_metadata records the
declared type, arch, os, compression, load and entry; root metadata
records the description, timestamp and configurations.
crc32, md5, sha1, sha256, sha384 and sha512 hash nodes are verified in
one streamed pass per image; other algorithms are reported unverified.
Signatures are reported, not verified, and nothing is decompressed,
decrypted or executed. Data outside the file, overlapping or duplicate
image data, a missing or repeated data source and hash mismatches fail
closed. The devicetree is bounded by the fixed 16 MiB allowance;
external image data is streamed from disk.
Refs #7129
* fix(artifacts): bound FIT configuration references and suggest analysis only for plain images
* fix(artifacts): retain buffered FIT stream cancellation
* test(artifacts): gate FIT faults on the observed read phase
* fix(artifacts): verify FIT links and summary identities
* fix(artifacts): bound FIT diagnostics and retain unknown signatures
* fix(artifacts): compare FIT declarations and resolved data links
* fix(artifacts): admit FIT raw probes through the shared loader contract
* fix(artifacts): preserve FIT field availability independently
* fix(artifacts): isolate nested entry metadata from reader facts
---------
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
Co-authored-by: gensokyo-dev <342580529+users.noreply.github.com>
U-Boot's mkimage writes the multi-file size table for script images
(`mkimage -T script`, the boot.scr format) exactly as it does for multi
images, `mkimage -l` lists the script as "Image 0", and the `source`
command skips the table before running the script ("scripts are just
multi-image files with one component", boot/image-board.c).
The uImage reader parsed the table only for IH_TYPE_MULTI, so a script
image reported sub_image_count 0 and exposed only payload.bin, whose
first bytes are the binary size table rather than the script text.
Treat IH_TYPE_SCRIPT like IH_TYPE_MULTI: parse the same bounded table,
expose image-<n>.bin components and report the size-table limitation.
payload.bin is unchanged. A script payload without a valid table now
fails closed, as a multi image already does.
* feat(artifacts): inventory and extract ESP32 firmware images
ESP32-family app and bootloader images (magic 0xE9) had no reader, so
open_binary refused OTA downloads, app partitions and bootloaders.
Add the esp-image container format behind inspect_artifact and
extract_artifact. A pure decoder reads each byte once and keeps only
fixed-size fields. Derived facts cover the chip, from a table cited to
ESP-IDF v6.1; segment classification as ESP-IDF's should_map and
should_load; checksum and appended SHA-256 results; app and bootloader
descriptors; Secure Boot v2 signature blocks, observed and unverified;
and undescribed trailing byte ranges. Truncation and undescribed bytes
make coverage unknown. ESP8266 images are refused with a specific
reason, and ciphertext stays unrecognized.
Add the format to every target, Evidence, snapshot, live-summary and
artifact vocabulary, document it, and add an optional verify:esp-images
lane that compares esptool-built images with esptool image-info and
espsecure signature-info-v2.
Refs #3615
* test(artifacts): run the ESP image lane on esptool v5.6.0
The lane compared descriptor fields only when esptool image-info printed
one, so a descriptor that REA reports and esptool does not print passed
without comparison. Compare the descriptor kind in both directions.
Record that the lane passed for all 13 images with esptool and espsecure
v5.6.0, the version the layout facts cite, as well as with 4.12.0.
* refactor(artifacts): define ESP image layout sizes once
The decoder and the derived facts each defined the segment-header,
appended-digest and flash-sector sizes, the erased-byte scan and the
offset formatter. Define them once with the other fixed-layout structures
and import them in both places. The signature slot count is the one
constant that drives the slot scan.
Correct the observedEnd comment: it is the end of the last chunk read.
* fix(artifacts): report ESP-IDF's segment length limit by chip
ESP-IDF's verify_segment_header rejects a segment of 16 MiB or more
unless the chip maps flash through 32-bit cache addresses
(SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP: ESP32-S3, P4, C5, C61, H4 and
S31 in v6.1). Record that capability in the chip table and add a
limitation for such segments; unknown chips get none.
`loaded` repeats should_load. It does not mean the bootloader accepts the
segment: verify_load_addresses still rejects one outside the chip's RAM,
which REA does not model. Say so in the docs, the skill reference and the
inventory limitation.
* test(artifacts): check ESP lane tools first and verify extraction
The lane started work before checking its tools, so a missing RISC-V
target, ld.lld, esptool or espsecure surfaced as a raw spawn or command
error. Check each one before building and name the variable to set. Warn
when esptool is not v5.6.0, the release the layout facts cite.
Assert each segment's class for the ESP32-C3 app, its variants and the
bootloader, as the Xtensa image already did. Extract the app and the
signed image through the CLI and compare every file with the bytes at
esptool's segment offsets or in the appended signature sector; the
source's SHA-256 must be unchanged and equal the reported root_sha256.
* docs(artifacts): state what the ESP image lane does not establish
The Verification section listed the lane's checks but not its gaps: only
ESP32-C3 and ESP32 images come from a real producer, no real project
build, OTA download or flash dump is inspected, no signature is verified,
and the lane does not run in CI.
Record the timing of the final lane run on esptool v5.6.0.
* fix(artifacts): compare ESP segment declarations
Segment entries hash only their data, so compare_artifacts reported a
segment that moved to another load address with the same bytes as
unchanged; only the image itself changed. Compare each segment's
load_addr and data_len as its declaration, as Node SEA and PyInstaller
members are compared, and count a changed declaration as a metadata
change, also when content is unavailable.
Header offsets are layout and the classification is derived, so neither
is compared.
* test(artifacts): run ESP lane commands as owned process trees
The lane ran its tools through execFile. A timeout killed only the direct
child, so a linker or assembler that Clang or GCC had started could keep
running while the lane removed its workspace. Run every command through
createVerifierCommands, check the tools inside withVerifierCleanup, and
remove the workspace only after commands.close() has stopped every tree;
a failed close keeps the workspace.
Only a nonzero exit now counts as a refusal or an expected esptool
failure; timeouts and spawn failures propagate. Supervising each command
roughly doubles the lane's run time.
* test(artifacts): cancel ESP decoding after its first decoded chunk
The cancellation test requested cancel or close on setImmediate, which
can land while the file is still being opened and checked, so it never
showed that decoding had started. It also passed against a reader that
ignored both requests while decoding. Acknowledge the decoder's first
chunk, then cancel or close, join the operation, and assert that no
later chunk reaches the decoder and that the reader ends closed.
* docs(artifacts): record which ESP metadata comparison ignores
Segments are compared by their declaration. The image's and the
signature blocks' format_metadata are not: every field comes from their
own bytes, which content comparison already covers, and a block's
image_digest_matches follows the image's content.
* docs(artifacts): state that ESP mmu_page_size is a base-2 logarithm
The app descriptor's mmu_page_size keeps ESP-IDF's name and value, the
MMU page size in log base 2 as esp_app_desc_t declares it, so 16 means
64 KiB. Say so in docs/esp-image.md and the schema comment; a reader
could otherwise take 16 as 16 KiB.
* fix(esp-image): retain member consumption and share lifecycle regressions
* test(artifacts): scope MCP catalogs and budget full-schema validation
* test(artifacts): verify compact output profiles and document bounded catalogs
* test(artifacts): close readers when cleanup-test admission fails
* test(artifacts): compare ESP image limitations with their owners
The decoder and reader tests froze complete limitation sentences, so a
harmless rewording failed them. The wording now has two owners:
ESP_IMAGE_LIMITATIONS in espImageFacts.ts, which the decoder, the inventory
and the reader use, and ESP_DESCRIPTOR_LIMITATIONS beside the descriptor
decoder. Tests compare delivered limitations with the owner, called with the
facts each case establishes (segment index, declared and available bytes,
offsets, chip, page range), next to the structured assertions they already
make. Delivered text is unchanged.
* test(esp-image): assert structured facts and owned guidance
* fix(esp-image): keep spi_flash_mode and descriptor fields before a truncated tail
ESP-IDF v6.1's esp_app_desc_t declares spi_flash_mode at byte 181, after
mmu_page_size, as the committed reference layout shows; the descriptor now
keeps it as stored. The decoder also required all 256 descriptor bytes,
though it reads none past byte 181, so a file ending in the reserved tail
lost the version, project, digest and revision facts it does hold. The app
descriptor now needs only its fields; the segment's truncation is reported
as before.
* test(esp-image): pin the P-192 public key digest to espsecure's layout
espsecure packs ECDSA signature blocks as "<BBBx32sB64s64s1031x" and
zero-pads a 48-byte P-192 point to 64 bytes, so the signature starts at
byte 101 for every SHA-256 curve. signature_info_v2 hashes sig_block[36:101]
and _digest_sbv2_public_key the curve ID with the padded point. No test built
a P-192 block; this one checks public_key_sha256 against that definition.
* fix(esp-image): keep each app descriptor field the file holds in full
A file can end anywhere inside the 256-byte esp_app_desc_t. The decoder
needed every field byte, so a cut at byte 176 dropped the version, project
and app_elf_sha256 it does hold. Once the 4-byte magic identifies an app
descriptor, each field is now kept when the file holds all of its bytes and
is null otherwise; observed_length records where the descriptor ends, so a
cut field is told apart from text that is not UTF-8. The fields after the
magic become nullable in the metadata schema.
* fix(esp-image): report a signature block whose image digest does not match
A CRC-valid Secure Boot v2 block can carry another image's digest. The
block recorded image_digest_matches false, but no limitation said so, unlike
a checksum or appended-SHA mismatch, so CLI and MCP summaries showed only
the generic unverified note. The mismatch is now a limitation; coverage
stays complete and signature_status stays unverified. The three other
signature-sector limitations move into ESP_IMAGE_LIMITATIONS with their
text unchanged, so all ESP image wording has one owner.
* refactor(esp-image): remove redundant descriptor byte guards
---------
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
* feat(artifacts): add U-Boot legacy uImage reader (part of #7129)
Recognize legacy uImages by their 0x27051956 magic and inventory them as containers: 64-byte header facts (arch/os/type/compression raw and named, load/entry addresses, name, timestamp) with header and data CRC32 verified, stored payload extractable as payload.bin, and multi-file images split from the stored size table into image-<n>.bin. Compressed payloads stay as stored; nothing is executed or decompressed. Wires the format through classification, the artifact reader dispatch, extraction, and the evidence/snapshot format vocabularies. Related to #7129 (uImage part only); HEX/SREC and FIT are out of scope.
* fix(uimage): preserve aligned component ranges and exact header facts
* fix(uimage): retain buffered members through consumption and cancellation
* fix(uimage): preserve active read and validation failure precedence
---------
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
* feat(artifacts): read UnityFS AssetBundle containers
inspect-artifact and extract-artifact now recognize the UnityFS signature
and treat format versions 6 through 9 as containers. Header facts (format
version, player/engine strings, declared size, dataflags, blocks-info
bounds and the declared 16-byte data hash) are retained alongside a
directory of CAB-* and .resS members. Member bytes stream on demand:
only the blocks covering the requested member are read and decompressed
with bounded expansion and cancellation.
Bundles declaring asset-bundle encryption keep header facts and report
unknown directory coverage. LZMA blocks-info reports unknown coverage;
LZMA member reads fail as unavailable while the directory stays
readable. Malformed counts, out-of-stream members and declared sizes
beyond the file fail as format errors. UnityFS classifies by bytes in
root detection, inventory graph construction and desktop inspection,
which emits a unityfs-header fact.
Validation: tsc --noEmit, oxlint (no new findings on touched files),
17 reader tests covering LZ4HC/uncompressed tables, blocks-info at end,
format 6 without padding alignment, multi-block members, malformed
input, unsupported compression, cancellation and abandoned streams;
full artifacts/domain/conformance suites pass. Verified against one
real YooAsset bundle (Unity 2021.3.32f1, format 8, LZ4HC): the
streamed SerializedFile contained its managed assembly byte-identical
to a UnityPy extraction. Other producer variants unverified.
* fix(unityfs): own bounded member reads and preserve partial evidence
* refactor(unityfs): clarify producer revision flag decisions
* fix(artifacts): preserve coverage authority and bound UnityFS block work
---------
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
* feat(artifacts): inventory and extract Intel HEX and Motorola S-record regions
open_binary refused Intel HEX and S-record programming files as an
unsupported format, so inspect_artifact and extract_artifact could not
answer which bytes load at which addresses.
Recognize both encodings by a fully valid first record and decode them
statically into address-contiguous regions, each an inventory entry named
0x<start>-0x<end>.bin and extractable as raw bytes. Root format_metadata
records the encoding and variant, record counts, unique and duplicate
bytes, address bounds, termination, declared start address, S0 header and
validated S5/S6 count. Gaps are never filled; start addresses are declared
facts, not proven entry points.
Checksum, record-type, byte-count, line-length, overlap-conflict, address
wrap and post-termination errors fail closed with the line number; a
missing termination record leaves coverage unknown. Decoding is bounded by
a 128 MiB file ceiling (decoded bytes are at most half the file and are
reserved after the size check) and 65,536 address-discontiguous runs.
Refs #7129
* fix(artifacts): refuse Intel HEX extended address records with a non-zero address
* fix(artifacts): bound HEX region copies and retain stream cancellation
---------
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
Replace overlapping traversals with one bounded relationship owner. Preserve superview direction, treat conflicting or unsupported registry placement as unknown, and retain source facts and coverage.