OpenSource/reverse-engineering-anything
1
0 Fork 0
mirror of https://github.com/morluto/rea.git synced 2026-11-04 22:06:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
6e9ee97a69cb16c7150d8b550e705cba055ee4f0
reverse-engineering-anything/docs/uf2-dfuse.md
T
Entrpiandmorluto b43c329df7 feat(artifacts): inventory and extract UF2 and DfuSe firmware containers (#8230) 
* feat(artifacts): inventory and extract UF2 and DfuSe regions

open_binary refused UF2 and DfuSe firmware containers as an unsupported
format, so inspect_artifact and extract_artifact could not show which
bytes a drag-and-drop or DFU update writes at which addresses.

UF2 is recognized by a complete first block with all three magic
numbers. Valid main-flash payloads are joined into address-contiguous
regions per familyID, named family-0x<id>/0x<start>-0x<end>.bin, and
known IDs are named from a small table copied from Microsoft's
uf2families.json at a pinned commit. Numbering is counted per familyID
and block count: absent, repeated, out-of-order and out-of-range block
numbers are reported, and absent blocks leave coverage unknown.
Not-main-flash blocks are excluded and reported; file-container blocks
are reported and leave coverage unknown; invalid blocks and a partial
final block are excluded with their position. A rewrite of one address
with different bytes fails closed, naming both block spans.

DfuSe is recognized only by its prefix together with a DFU 1.1a suffix.
Each element becomes one region at its file range with its target's
alternate setting and name. Structural faults fail closed with a byte
offset; overlapping elements stay separate and are reported. A suffix
CRC mismatch follows integrity_policy: fail refuses with a dfu-crc32
integrity error, and record-and-continue records a root contradiction
that extraction also carries.

HEX, UF2 and DfuSe share one retained-region reader and one run
merger, so the HEX decoder and reader lose their private copies.

* docs(artifacts): document UF2 and DfuSe images and their esptool lane

Describe recognition, region names, metadata, numbering, failures, the
DfuSe suffix CRC under integrity_policy, and limits, and link the page
from the CLI and firmware guides and the agent skill reference.

npm run verify:uf2 runs esptool merge-bin --format uf2 on generated
inputs at known addresses, with and without MD5, with 256-byte
payloads, and on a concatenation of two chips' outputs. Inspection and
extraction through the built CLI must return every input at its address
under the familyID esptool itself reports.

* fix(artifacts): accept both DfuSe image size conventions

UM0391 calls DFUImageSize the file length, and dfu-util's dfuse-pack.py
writes that value, while ST's DfuFileMgr, MicroPython's dfu.py and pydfu
leave the 16-byte suffix out. The decoder accepted only the second form,
so it refused dfuse-pack.py output as malformed.

Accept either value and keep the declared one in image_size; any other
value still fails with the suffix offset and the file length. The tests
pin both producers' output for one image, CRC included, from their
build() functions transcribed and run under Python 3.

* fix(artifacts): keep reused UF2 block numbers from hiding missing blocks

esptool numbers every merged input from 0, so two inputs with the same
block count share one numbering group. A block lost from one input left
its number present in the other, and the file reported complete
coverage. Each block number now records the target addresses it was
seen at: equal address counts are complete copies, a resend to the same
address is one block, and unequal counts make coverage unknown with a
limitation and ambiguous_numbering_group_count. sequence_count becomes
numbering_group_count, and region block positions become
first_file_block and last_file_block.

Review follow-ups in the same change:
- compare_artifacts compares a DfuSe element's alternate setting and
  target name, so bytes moved to another device memory are changed;
- the shared reader keeps no region bytes outside its map after close;
- one domain type replaces four identical decoder failure types;
- the UF2 decoder returns a typed limit instead of throwing on bytes
  beyond its admitted size, and caps retained regions, not runs;
- DfuSe caps elements with bytes only, names empty elements and
  element-less targets, drops the constant bVersion and bcdDFU fields,
  and says whether the CRC matches when DFUImageSize fits neither
  convention;
- classification evidence carries a suffix only when the probe saw the
  end, and root classification reads the tail only for a DfuSe prefix;
- the UF2 domain region owns family_name, and the root records the
  family registry revision; the table's MIT notice is preserved;
- docs pin the producer revisions (dfu-util ee6b393d and f9a537c9,
  MicroPython cb99ca98), give the MD5 structure's real meaning, and
  describe container contradictions in the MCP contract guide;
- tests and the esptool lane add the equal-count inputs and a block
  lost from one of them, accepted boundaries beside each refusal, a
  DfuSe root larger than the classification prefix, and CLI refusals.

* feat(artifacts): name the UF2 families that LibreTiny builds for

LibreTiny writes UF2 for Realtek, Beken and Bouffalo Lab modules. The
Microsoft registry revision REA already cites (cf3bd7f9) lists eight of
those families: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U, BK7231N,
BK7251 and BL602. Their regions now carry those names. LN882H, which
that revision omits, keeps its familyID with no name.

* fix(artifacts): count DfuSe declarations without bytes instead of listing them

Empty DfuSe elements have no entry, so DFUSE_MAX_ELEMENTS does not cap
them, and the target walk kept a description string for each one. At 8
bytes per element, a file under the 128 MiB ceiling can declare about
16.7 million of them. The walk now counts every declaration without bytes
and builds only the first NAMED_ELEMENTS descriptions, which is all that
the limitation shows. The limitation text and empty_element_count do not
change. The guide now says that both DfuSe limitations name the first four.

On this host, decoding 4,000,000 empty elements (a 32 MB file) ran out of
memory under a 128 MB V8 heap before this change, and it peaked at 1.59 GB
RSS with the default heap. It now decodes in 114 ms with a 220 MB peak.

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-31 16:27:00 +08:00

225 lines
17 KiB
Markdown
Raw Blame History

# UF2 and DfuSe images
REA inventories UF2 (Microsoft's USB Flashing Format) and DfuSe (ST's DFU
update container, `.dfu`) files through `inspect-artifact` /
`inspect_artifact`, and materializes their decoded bytes through
`extract-artifact` / `extract_artifact`. UF2 is the drag-and-drop format of the
RP2040 and RP2350 bootloaders, of TinyUF2 on ESP32-S2 and ESP32-S3, and of
`esptool merge-bin --format uf2`. Decoding is static and needs no external tool.
```sh
rea inspect-artifact ./firmware.uf2 --json
rea inspect-artifact ./firmware.dfu --detail summary --evidence_output evidence.json --json
rea extract-artifact ./firmware.uf2 --json
```
For MCP, open the file with `open_binary`, then call `inspect_artifact`. Summary
delivery retains the complete Evidence; use `parent_evidence_id` with
`inspect_analysis_view` to read selected regions and fields. `extract_artifact`
returns a fresh `output_root` containing one raw file per region.
## Recognition
A file is UF2 only when its first 512 bytes are a complete block with all three
magic numbers: `0x0A324655` and `0x9E5D5157` at the start and `0x0AB16F30` at
the end. A file is DfuSe only when it begins with `DfuSe` and format version 1
and ends with a 16-byte DFU 1.1a suffix: `bcdDFU` `0x011A`, the `UFD` signature
and `bLength` 16. The suffix CRC is not part of recognition. The file name does
not matter, and a file with only one of the DfuSe markers stays unrecognized.
The container formats are `uf2` and `dfuse`.
## UF2 regions and metadata
Each entry is one address-contiguous region of one family's main-flash
payloads, named `family-0x<familyID>/0x<start>-0x<end>.bin` with an inclusive
end address, for example `family-0xe48bff56/0x10000000-0x1000ffff.bin`. Blocks
without the familyID flag (`0x2000`) go under `no-family/`. Families are
separate address spaces: a multi-family file, such as a Pico SDK image for both
RP2040 and RP2350, has a region per family even at the same addresses. Gaps are
never filled.
Region `format_metadata` (`uf2-region`) records `family_id` (or `null`),
`family_name`, `start_address`, `end_address`, `length`, `block_count`, and
`first_file_block`/`last_file_block`: the zero-based positions in the file of
the contributing blocks, not their UF2 block numbers. `family_name` is REA's
label, not file content. It comes from a small table copied from Microsoft's
`uf2families.json` at commit
[`cf3bd7f9`](https://github.com/microsoft/uf2/blob/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/utils/uf2families.json),
which the root's `family_name_registry` names. The table covers the ESP32
family, ESP8266, RP2040, the RP2350 and RP2XXX download families, STM32F4,
nRF52840, SAMD21 and SAMD51. It also covers the families that LibreTiny builds
for and the registry lists: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U,
BK7231N, BK7251 and BL602. Other IDs, LibreTiny's LN882H among them, keep their
value with `family_name: null`. A familyID names a target family; it does not
establish the CPU state, memory map or entry point.
Root `format_metadata` (`uf2`) records:
| Field | Meaning |
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `block_count`, `trailing_bytes` | Complete 512-byte blocks, and bytes after the last complete block |
| `invalid_block_count` | Blocks with a wrong magic number, a payload over 476 bytes, or a payload that wraps past 4 GiB |
| `flash_block_count` | Valid main-flash blocks, the only source of region bytes |
| `not_main_flash_block_count`, `file_container_block_count` | Blocks with flag `0x1`, and blocks with flag `0x1000` |
| `md5_block_count`, `extension_tag_block_count` | Blocks that declare an MD5 flash region (`0x4000`) or extension tags (`0x8000`) |
| `family_count`, `family_name_registry` | Distinct familyIDs among main-flash blocks, and the registry revision of `family_name` |
| `numbering_group_count` | Groups of valid blocks that share a familyID (or none) and a declared block count |
| `missing_block_count` | Block numbers below a group's block count that no valid block carries |
| `ambiguous_numbering_group_count` | Groups whose numbers appear at unequal counts of target addresses |
| `repeated_block_count`, `out_of_order_block_count` | Blocks whose number already appeared in their group, and blocks numbered at or below the previous block of their group |
| `block_number_out_of_range_count` | Blocks numbered at or above their block count, or with a block count of 0 |
| `region_count`, `data_bytes`, `duplicate_bytes` | Regions, unique decoded bytes, and bytes rewritten with identical values |
Numbering is grouped by familyID because a bootloader ignores every field of
another family's block, including its numbering. Within a group, a block number
seen at several target addresses belongs to several copies of a sequence:
esptool numbers each merged input file from 0, so inputs with equal block
counts share a group, and each number then appears once per input. A block sent
twice to the same address is one block. A group is complete when every number
below its block count is present at the same count of addresses. If some numbers
appear at more addresses than others, a copy may have lost a block, so coverage
is unknown and a limitation names the group.
## UF2 facts, limitations and failures
| Situation | Outcome |
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Blocks out of address order | Sorted into regions |
| Blocks numbered at or below the previous block of their group | Counted in `out_of_order_block_count`; the spec lets blocks arrive in any order |
| An address written twice with the same value in one family | Accepted and counted in `duplicate_bytes` |
| An address written twice with different values in one family | Fails, naming the family, the address and the block spans of both writes |
| The same addresses in different families | Separate regions; never a conflict |
| Numbered blocks absent from a group | Regions are returned; coverage is unknown and a limitation names the groups |
| Reused numbers at unequal address counts | Regions are returned; coverage is unknown and a limitation names the groups |
| An invalid block | Excluded; coverage is unknown and a limitation names the first one and its offset |
| A partial final block | Excluded; coverage is unknown and the file may be truncated |
| Block numbers at or above the block count | Payloads are kept; coverage is unknown |
| Not-main-flash blocks (`0x1`) | Excluded from regions; their count and payload byte count are reported; coverage is unaffected |
| File-container blocks (`0x1000`) | Excluded from regions; their files are not reconstructed, so coverage is unknown |
| MD5 (`0x4000`) and extension-tag (`0x8000`) blocks | Counted and reported as a limitation; neither is decoded or verified |
Unlike a malformed Intel HEX record, which fails the whole file, an invalid UF2
block is excluded and reported: each block carries its own framing and address,
so its neighbours stay readable.
The MD5 structure is not checked because it is not a checksum of the block's
bytes in the file. It names a region of device flash, usually aligned to erase
pages, whose MD5 a flashing tool compares to skip writing. Producers also
disagree on its place: the specification puts it in the last 24 bytes of the
data area, while esptool v4.12.0 and v5.6.0 write it right after a shorter final
payload, so those 24 bytes hold zeros or part of that structure.
## DfuSe regions and metadata
Each element is one entry at its own file range, named
`target-<t>/element-<e>-0x<start>-0x<end>.bin` with zero-based target and
element indexes, for example `target-0/element-0-0x08000000-0x0800ffff.bin`.
Elements are never merged. Each occurrence's `source_location` is the byte range
of the element data in the file.
Element `format_metadata` (`dfuse-element`) records `target_index`,
`alternate_setting`, `target_name`, `element_index`, `start_address`,
`end_address` and `length`. `target_name` is `null` when `bTargetNamed` is 0;
otherwise it is `szTargetName` up to its first NUL, one character per byte.
`compare_artifacts` treats an element's alternate setting and target name as
its declaration, so the same bytes moved to another device memory are
`changed` in `metadata`.
Root `format_metadata` (`dfuse`) records:
| Field | Meaning |
| ---------------------------------------------------- | ------------------------------------------------------------------------------- |
| `image_size`, `target_count` | `DFUImageSize` and `bTargets` from the prefix |
| `element_count`, `empty_element_count`, `data_bytes` | All elements, elements of 0 bytes (which have no entry), and element data bytes |
| `overlapping_element_count` | Elements that overlap an earlier element of the same alternate setting |
| `device_version`, `product_id`, `vendor_id` | `bcdDevice`, `idProduct` and `idVendor` from the suffix |
| `suffix_crc`, `computed_crc` | The stored `dwCRC`, and the same CRC computed over every byte before it |
Recognition already requires `bVersion` 1 and `bcdDFU` `0x011A`, so they are
not repeated. The DFU suffix CRC is CRC-32 without the final inversion, as
dfu-util stores and checks it.
Producers disagree on `DFUImageSize`. UM0391 calls it the file length, and
dfu-util's `dfuse-pack.py` writes that value since commit `ee6b393d`
(2020-10-08). Earlier `dfuse-pack.py` and MicroPython's `tools/dfu.py`
(checked at `cb99ca98`) write the length before the suffix, and an ST community
report describes ST's DfuFileMgr the same way; REA has not run DfuFileMgr.
dfu-util's own parser ignores the field. REA accepts either value and keeps the
declared one in `image_size`.
## DfuSe facts, limitations and failures
| Situation | Outcome |
| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `dwCRC` differs from the computed CRC | Follows `integrity_policy`, below |
| A target without the `Target` signature | Fails with its byte offset |
| `DFUImageSize` that equals neither the file length nor the bytes before the suffix | Fails with the declared size, the suffix offset, the file length and whether the suffix CRC matches |
| A target size or an element size that disagrees with the bytes that hold it | Fails with the byte offset and the declared and available sizes |
| An element whose address range wraps past `0xFFFFFFFF` | Fails with its byte offset |
| Overlapping elements of one alternate setting | Kept as separate entries; a limitation names the first four; coverage is unaffected |
| The same addresses in different alternate settings | Separate entries; an alternate setting selects a device memory |
| Elements of 0 bytes, and targets with no elements | No entry; counted; a limitation names the first four, with address and alternate setting |
A CRC mismatch is an integrity contradiction, never proof of a cause. With the
default `integrity_policy` `fail`, inspection and extraction fail with
`artifact_integrity_mismatch`, whose details carry `algorithm: "dfu-crc32"`,
`declared_crc32` and `calculated_crc32`. With `record-and-continue`
(CLI `--integrity-policy record-and-continue`), the root occurrence becomes
`mismatched`, `integrity_contradictions` records the pair for logical path `.`,
and extraction carries that contradiction, because every element came from the
contradicted container. The root occurrence has no `declared_integrity`: that
field holds entry digests, so the declared CRC appears in the contradiction and,
as a number, in `suffix_crc`. Under `fail`, the elements are hashed before the
mismatch is raised.
## Limits
Each file must be at most 128 MiB. The UF2 decoder reserves at most 476 bytes
per complete block before reading, merges payloads into regions of at most the
same total size, and keeps one numbering entry per valid block, so its peak is
about twice the decoded bytes. It retains at most 65,536 regions as entries; the
order of blocks does not count against that bound. The DfuSe decoder holds the
file once, serves elements as views of it, and retains at most 65,536 elements
with bytes as entries; empty elements are only counted. These are fixed
resource bounds, not format limits, and fail as limits (`file-size` and
`capacity`) with no runtime override. Configurable ceilings are tracked in
[#6176](https://github.com/morluto/rea/issues/6176).
Decoded regions are cached only for the reader's lifetime, and each open checks
the admitted carrier identity again. These readers share the Intel HEX
reader's region streams, cancellation and close behavior.
## Analyzing a region
Inventory does not establish the CPU, byte order or memory map, or an entry
point. To disassemble a region, extract it and open the region file with
`open_binary` using format `raw-binary`, an architecture you choose, and
`entry_point` set to the region start address; raw-binary maps file offset 0 at
`entry_point`. Each region is opened separately, so references between regions
are not resolved.
## Verification
Fixtures written from the UF2 specification and from ST UM0391 cover block and
element layouts, numbering groups and reused numbers, families, rewrites and
conflicts, invalid and partial blocks, not-main-flash and file-container blocks,
structural faults, overlaps, both integrity policies, accepted boundaries beside
each refusal, resource refusals, inventory, extraction, comparison and reader
identity checks. The reader lifecycle (cancellation, close and stream
ownership) is covered by the Intel HEX reader tests, which exercise the shared
reader. Two DfuSe fixtures are byte-identical, CRC included, to what the
`build()` functions of dfu-util's `dfuse-pack.py` (at `f9a537c9`) and
MicroPython's `tools/dfu.py` (at `cb99ca98`) produce for the same image, one for
each `DFUImageSize` convention. Those functions were transcribed and run under
Python 3; the scripts themselves were not run. CLI and stdio MCP tests exercise
the same artifact provider and advertised schemas.
`npm run verify:uf2` runs esptool's `merge-bin --format uf2` on generated
inputs at known addresses, with and without MD5, with 256-byte payloads, with
two inputs of equal block count, and on a concatenation of two chips' outputs.
Inspection and extraction must return every input at its address under
esptool's own familyID. It passed with esptool v4.12.0 and v5.6.0 on macOS. Set
`REA_ESPTOOL_PYTHON` to a Python that runs `-m esptool`. Pico SDK, picotool,
`uf2conv.py`, TinyUF2 and real DfuSe producers remain unverified, and no device
accepted these files in REA's tests.
Reference in new issue View Git Blame Copy Permalink