Files
reverse-engineering-anything/src/artifacts/artifactFormats.ts
T
Entrpiandmorluto b43c329df7 feat(artifacts): inventory and extract UF2 and DfuSe firmware containers (#8230)
* feat(artifacts): inventory and extract UF2 and DfuSe regions

open_binary refused UF2 and DfuSe firmware containers as an unsupported
format, so inspect_artifact and extract_artifact could not show which
bytes a drag-and-drop or DFU update writes at which addresses.

UF2 is recognized by a complete first block with all three magic
numbers. Valid main-flash payloads are joined into address-contiguous
regions per familyID, named family-0x<id>/0x<start>-0x<end>.bin, and
known IDs are named from a small table copied from Microsoft's
uf2families.json at a pinned commit. Numbering is counted per familyID
and block count: absent, repeated, out-of-order and out-of-range block
numbers are reported, and absent blocks leave coverage unknown.
Not-main-flash blocks are excluded and reported; file-container blocks
are reported and leave coverage unknown; invalid blocks and a partial
final block are excluded with their position. A rewrite of one address
with different bytes fails closed, naming both block spans.

DfuSe is recognized only by its prefix together with a DFU 1.1a suffix.
Each element becomes one region at its file range with its target's
alternate setting and name. Structural faults fail closed with a byte
offset; overlapping elements stay separate and are reported. A suffix
CRC mismatch follows integrity_policy: fail refuses with a dfu-crc32
integrity error, and record-and-continue records a root contradiction
that extraction also carries.

HEX, UF2 and DfuSe share one retained-region reader and one run
merger, so the HEX decoder and reader lose their private copies.

* docs(artifacts): document UF2 and DfuSe images and their esptool lane

Describe recognition, region names, metadata, numbering, failures, the
DfuSe suffix CRC under integrity_policy, and limits, and link the page
from the CLI and firmware guides and the agent skill reference.

npm run verify:uf2 runs esptool merge-bin --format uf2 on generated
inputs at known addresses, with and without MD5, with 256-byte
payloads, and on a concatenation of two chips' outputs. Inspection and
extraction through the built CLI must return every input at its address
under the familyID esptool itself reports.

* fix(artifacts): accept both DfuSe image size conventions

UM0391 calls DFUImageSize the file length, and dfu-util's dfuse-pack.py
writes that value, while ST's DfuFileMgr, MicroPython's dfu.py and pydfu
leave the 16-byte suffix out. The decoder accepted only the second form,
so it refused dfuse-pack.py output as malformed.

Accept either value and keep the declared one in image_size; any other
value still fails with the suffix offset and the file length. The tests
pin both producers' output for one image, CRC included, from their
build() functions transcribed and run under Python 3.

* fix(artifacts): keep reused UF2 block numbers from hiding missing blocks

esptool numbers every merged input from 0, so two inputs with the same
block count share one numbering group. A block lost from one input left
its number present in the other, and the file reported complete
coverage. Each block number now records the target addresses it was
seen at: equal address counts are complete copies, a resend to the same
address is one block, and unequal counts make coverage unknown with a
limitation and ambiguous_numbering_group_count. sequence_count becomes
numbering_group_count, and region block positions become
first_file_block and last_file_block.

Review follow-ups in the same change:
- compare_artifacts compares a DfuSe element's alternate setting and
  target name, so bytes moved to another device memory are changed;
- the shared reader keeps no region bytes outside its map after close;
- one domain type replaces four identical decoder failure types;
- the UF2 decoder returns a typed limit instead of throwing on bytes
  beyond its admitted size, and caps retained regions, not runs;
- DfuSe caps elements with bytes only, names empty elements and
  element-less targets, drops the constant bVersion and bcdDFU fields,
  and says whether the CRC matches when DFUImageSize fits neither
  convention;
- classification evidence carries a suffix only when the probe saw the
  end, and root classification reads the tail only for a DfuSe prefix;
- the UF2 domain region owns family_name, and the root records the
  family registry revision; the table's MIT notice is preserved;
- docs pin the producer revisions (dfu-util ee6b393d and f9a537c9,
  MicroPython cb99ca98), give the MD5 structure's real meaning, and
  describe container contradictions in the MCP contract guide;
- tests and the esptool lane add the equal-count inputs and a block
  lost from one of them, accepted boundaries beside each refusal, a
  DfuSe root larger than the classification prefix, and CLI refusals.

* feat(artifacts): name the UF2 families that LibreTiny builds for

LibreTiny writes UF2 for Realtek, Beken and Bouffalo Lab modules. The
Microsoft registry revision REA already cites (cf3bd7f9) lists eight of
those families: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U, BK7231N,
BK7251 and BL602. Their regions now carry those names. LN882H, which
that revision omits, keeps its familyID with no name.

* fix(artifacts): count DfuSe declarations without bytes instead of listing them

Empty DfuSe elements have no entry, so DFUSE_MAX_ELEMENTS does not cap
them, and the target walk kept a description string for each one. At 8
bytes per element, a file under the 128 MiB ceiling can declare about
16.7 million of them. The walk now counts every declaration without bytes
and builds only the first NAMED_ELEMENTS descriptions, which is all that
the limitation shows. The limitation text and empty_element_count do not
change. The guide now says that both DfuSe limitations name the first four.

On this host, decoding 4,000,000 empty elements (a 32 MB file) ran out of
memory under a 128 MB V8 heap before this change, and it peaked at 1.59 GB
RSS with the default heap. It now decodes in 114 ms with a 220 MB peak.

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-31 16:27:00 +08:00

452 lines
17 KiB
TypeScript

import type { AsarHeaderAdmission } from "../config/asarHeaderAdmission.js";
import {
ARTIFACT_FORMAT_IDS,
type ArtifactFormatId,
} from "../domain/artifactGraph.js";
import type { EspFlashSelection } from "../domain/firmware/espFlashLocation.js";
import { ESP_IMAGE_LIMITATIONS } from "../domain/firmware/espImageFacts.js";
import {
ZIP_PACKAGE_FORMATS,
type ZipPackageFormat,
} from "../domain/zipPackageFormat.js";
import {
ArtifactReaderFailure,
type ArtifactReader,
} from "./ArtifactReader.js";
import type { ArtifactReaderAdmission } from "./ArtifactReaderAdmission.js";
import { AsarArtifactReader } from "./AsarArtifactReader.js";
import type { BunStandaloneAdmission } from "./bun/BunStandaloneAdmission.js";
import { BunStandaloneArtifactReader } from "./BunStandaloneArtifactReader.js";
import { DfuSeArtifactReader } from "./DfuSeArtifactReader.js";
import { DirectoryArtifactReader } from "./DirectoryArtifactReader.js";
import type { DotnetBundleAdmission } from "./dotnet/DotnetBundleAdmission.js";
import { DotnetBundleArtifactReader } from "./DotnetBundleArtifactReader.js";
import {
EspFlashImageArtifactReader,
type EspFlashAdmission,
} from "./EspFlashImageArtifactReader.js";
import { EspImageArtifactReader } from "./EspImageArtifactReader.js";
import { FitArtifactReader } from "./FitArtifactReader.js";
import type { GodotPckAdmission } from "./GodotPckLocation.js";
import { GodotPckArtifactReader } from "./GodotPckArtifactReader.js";
import { InesArtifactReader, INES_LIMITATION } from "./InesArtifactReader.js";
import { HexRecordArtifactReader } from "./HexRecordArtifactReader.js";
import { MachOSliceArtifactReader } from "./MachOSliceArtifactReader.js";
import { NativeDmgArtifactReader } from "./NativeDmgArtifactReader.js";
import type { NodeSeaAdmission } from "./nodeSea/NodeSeaAdmission.js";
import { NodeSeaArtifactReader } from "./NodeSeaArtifactReader.js";
import type { PyInstallerAdmission } from "./pyinstaller/PyInstallerAdmission.js";
import { PyInstallerArtifactReader } from "./PyInstallerArtifactReader.js";
import { Uf2ArtifactReader } from "./Uf2ArtifactReader.js";
import { UImageArtifactReader } from "./UImageArtifactReader.js";
import { UnityFsArtifactReader } from "./UnityFsArtifactReader.js";
import { ZipArtifactReader } from "./ZipArtifactReader.js";
/** How inventory describes a format when no reader was constructed. */
export type AbsentInventory = "platform" | "universal" | "unavailable" | "root";
export interface ArtifactReaderConstruction {
readonly environment: Readonly<NodeJS.ProcessEnv>;
readonly unreadableDirectories: "fail" | "omit";
readonly admission?: ArtifactReaderAdmission | undefined;
readonly asarHeaderAdmission?: AsarHeaderAdmission | undefined;
/** Caller inputs for an explicitly selected ESP flash image. */
readonly espFlashSelection?: EspFlashSelection | undefined;
/** Caller-selected root spelling; only ASAR companion lookup consumes this. */
readonly selectedPath?: string | undefined;
}
export interface FormatDescriptor {
readonly id: ArtifactFormatId;
/** Every ZIP package family shares this row of capabilities. */
readonly zipFamily: boolean;
/** Root occurrence kind is container, including directories and zip packages. */
readonly occurrenceContainer: boolean;
/** Path-role classification treats the format as a container of its own. */
readonly roleContainer: boolean;
/** Classification keeps the admitted source for a later reader. */
readonly retainClassifiedSource: boolean;
/** Extraction can open this format without an embedded-pack probe. */
readonly extractable: boolean;
/** Reader construction is refused off macOS. */
readonly requiresDarwin: boolean;
readonly absentInventory: AbsentInventory;
/** Extra limitation when a reader was constructed. */
readonly readerLimitation?: string;
/** Absent for formats with no entry reader, including `unknown`. */
readonly createReader?:
| ((path: string, options: ArtifactReaderConstruction) => ArtifactReader)
| undefined;
}
const sourceOf = (options: ArtifactReaderConstruction) => {
const admission = options.admission;
return admission !== undefined && "kind" in admission
? admission.source
: admission;
};
function admittedAs(
kind: "dotnet-bundle",
options: ArtifactReaderConstruction,
): DotnetBundleAdmission | ReturnType<typeof sourceOf>;
function admittedAs(
kind: "bun-standalone",
options: ArtifactReaderConstruction,
): BunStandaloneAdmission | ReturnType<typeof sourceOf>;
function admittedAs(
kind: "godot-pck",
options: ArtifactReaderConstruction,
): GodotPckAdmission | ReturnType<typeof sourceOf>;
function admittedAs(
kind: "pyinstaller",
options: ArtifactReaderConstruction,
): PyInstallerAdmission | ReturnType<typeof sourceOf>;
function admittedAs(
kind: "node-sea",
options: ArtifactReaderConstruction,
): NodeSeaAdmission | ReturnType<typeof sourceOf>;
function admittedAs(
kind: "esp-flash-image",
options: ArtifactReaderConstruction,
): EspFlashAdmission | ReturnType<typeof sourceOf>;
function admittedAs(
kind: string,
options: ArtifactReaderConstruction,
): ArtifactReaderAdmission | ReturnType<typeof sourceOf> {
const admission = options.admission;
return admission !== undefined &&
"kind" in admission &&
admission.kind === kind
? admission
: sourceOf(options);
}
/** One capability row for every ZIP-backed package family. */
export const ZIP_FAMILY_CAPABILITIES = {
zipFamily: true,
occurrenceContainer: true,
roleContainer: false,
retainClassifiedSource: true,
extractable: true,
requiresDarwin: false,
absentInventory: "unavailable",
} as const satisfies Omit<FormatDescriptor, "id" | "createReader">;
type FormatCapabilities = Omit<FormatDescriptor, "id" | "createReader">;
type ReaderFn = (
path: string,
options: ArtifactReaderConstruction,
) => ArtifactReader;
const withReader = <Id extends ArtifactFormatId>(
id: Id,
capabilities: FormatCapabilities,
createReader: ReaderFn,
) => ({ id, ...capabilities, createReader });
const withoutReader = <Id extends ArtifactFormatId>(
id: Id,
capabilities: FormatCapabilities,
) => ({ id, ...capabilities });
const zipDescriptor = (id: ZipPackageFormat) =>
withReader(
id,
ZIP_FAMILY_CAPABILITIES,
(path, options) =>
new ZipArtifactReader(path, id, undefined, sourceOf(options)),
);
function described<Id extends ArtifactFormatId>(
id: Id,
descriptor: FormatCapabilities & { readonly createReader: ReaderFn },
): ReturnType<typeof withReader<Id>>;
function described<Id extends ArtifactFormatId>(
id: Id,
descriptor: FormatCapabilities,
): ReturnType<typeof withoutReader<Id>>;
function described<Id extends ArtifactFormatId>(
id: Id,
descriptor: FormatCapabilities & { readonly createReader?: ReaderFn },
) {
const { createReader, ...capabilities } = descriptor;
return createReader === undefined
? withoutReader(id, capabilities)
: withReader(id, capabilities, createReader);
}
const plain = {
zipFamily: false,
occurrenceContainer: false,
roleContainer: false,
retainClassifiedSource: false,
extractable: false,
requiresDarwin: false,
absentInventory: "root",
} as const satisfies Omit<FormatDescriptor, "id" | "createReader">;
function container<Id extends ArtifactFormatId>(
id: Id,
descriptor: Partial<FormatCapabilities> & { readonly createReader: ReaderFn },
): ReturnType<typeof withReader<Id>>;
function container<Id extends ArtifactFormatId>(
id: Id,
descriptor?: Partial<FormatCapabilities>,
): ReturnType<typeof withoutReader<Id>>;
function container<Id extends ArtifactFormatId>(
id: Id,
descriptor: Partial<FormatCapabilities> & {
readonly createReader?: ReaderFn;
} = {},
) {
const { createReader, ...rest } = descriptor;
const capabilities = {
...plain,
occurrenceContainer: true as const,
...rest,
};
return createReader === undefined
? withoutReader(id, capabilities)
: withReader(id, capabilities, createReader);
}
const NON_ZIP_FORMATS = {
"dotnet-bundle": container("dotnet-bundle", {
retainClassifiedSource: true,
absentInventory: "root",
createReader: (path, options) =>
new DotnetBundleArtifactReader(
path,
admittedAs("dotnet-bundle", options),
),
}),
"bun-standalone": container("bun-standalone", {
retainClassifiedSource: true,
absentInventory: "unavailable",
createReader: (path, options) =>
new BunStandaloneArtifactReader(
path,
admittedAs("bun-standalone", options),
),
}),
dmg: container("dmg", {
requiresDarwin: true,
absentInventory: "platform",
createReader: (path, options) =>
new NativeDmgArtifactReader(path, options.environment),
}),
pkg: container("pkg", { absentInventory: "platform" }),
asar: container("asar", {
extractable: true,
absentInventory: "unavailable",
createReader: (path, options) =>
new AsarArtifactReader(path, {
...(options.asarHeaderAdmission === undefined
? {}
: { headerAdmission: options.asarHeaderAdmission }),
...(options.selectedPath === undefined
? {}
: { selectedPath: options.selectedPath }),
}),
}),
ines: container("ines", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation: INES_LIMITATION,
createReader: (path, options) => {
const source = sourceOf(options);
if (source === undefined)
throw new ArtifactReaderFailure(
"integrity",
"iNES inventory requires its admitted source descriptor",
);
return new InesArtifactReader(path, source);
},
}),
"godot-pck": container("godot-pck", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation:
"PCK inventory establishes packaged bytes and metadata; scenes, binary resources, GDScript bytecode and runtime signal behavior were not decoded.",
createReader: (path, options) =>
new GodotPckArtifactReader(path, admittedAs("godot-pck", options)),
}),
unityfs: container("unityfs", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation:
"UnityFS inventory establishes packaged bytes and metadata; serialized objects, textures, meshes and runtime asset behavior were not decoded. LZMA-compressed members are unsupported.",
createReader: (path, options) =>
new UnityFsArtifactReader(path, sourceOf(options)),
}),
uimage: container("uimage", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation:
"uImage inventory establishes the 64-byte legacy header facts and stored payload bytes. Compressed payloads are not decompressed, and the image is not executed.",
createReader: (path, options) =>
new UImageArtifactReader(path, sourceOf(options)),
}),
"javascript-bundle": described("javascript-bundle", plain),
"source-map": described("source-map", plain),
"mach-o-universal": described("mach-o-universal", {
...plain,
retainClassifiedSource: true,
extractable: true,
requiresDarwin: true,
absentInventory: "universal",
createReader: (path, options) =>
new MachOSliceArtifactReader(
path,
options.environment,
undefined,
sourceOf(options),
),
}),
"mach-o": described("mach-o", plain),
elf: described("elf", plain),
pe: described("pe", plain),
"dos-mz": described("dos-mz", plain),
"dos-com": described("dos-com", plain),
"raw-binary": described("raw-binary", plain),
"gamecube-dol": described("gamecube-dol", plain),
hopper: described("hopper", plain),
"analysis-database": described("analysis-database", plain),
plist: described("plist", plain),
entitlements: described("entitlements", plain),
directory: container("directory", {
absentInventory: "unavailable",
createReader: (path, options) =>
new DirectoryArtifactReader(path, options.unreadableDirectories),
}),
file: described("file", plain),
pyinstaller: container("pyinstaller", {
retainClassifiedSource: true,
createReader: (path, options) =>
new PyInstallerArtifactReader(path, admittedAs("pyinstaller", options)),
}),
"node-sea": container("node-sea", {
retainClassifiedSource: true,
absentInventory: "unavailable",
readerLimitation:
"Node SEA inventory establishes the injected blob, flags, fuse sentinel, main script, optional code cache, snapshot, and assets. V8 snapshots and code cache are not deserialized, and the application is not executed.",
createReader: (path, options) =>
new NodeSeaArtifactReader(path, admittedAs("node-sea", options)),
}),
"intel-hex": container("intel-hex", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation:
"Intel HEX/S-record inventory decodes data records into address-contiguous regions. It does not establish CPU, byte order, memory map, or that a declared start address is executable; gaps are unwritten, not zero.",
createReader: (path, options) =>
new HexRecordArtifactReader(path, "intel-hex", sourceOf(options)),
}),
srec: container("srec", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation:
"Intel HEX/S-record inventory decodes data records into address-contiguous regions. It does not establish CPU, byte order, memory map, or that a declared start address is executable; gaps are unwritten, not zero.",
createReader: (path, options) =>
new HexRecordArtifactReader(path, "srec", sourceOf(options)),
}),
"esp-image": container("esp-image", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation: ESP_IMAGE_LIMITATIONS.inventoryScope,
createReader: (path, options) =>
new EspImageArtifactReader(path, sourceOf(options)),
}),
uf2: container("uf2", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation:
"UF2 inventory decodes main-flash block payloads into address-contiguous regions per familyID. It does not establish CPU, byte order, memory map, or an entry point; a familyID names a target family, and gaps are unwritten, not zero.",
createReader: (path, options) =>
new Uf2ArtifactReader(path, sourceOf(options)),
}),
dfuse: container("dfuse", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation:
"DfuSe inventory lists each target element as one region at its declared address. It does not establish CPU, byte order, memory map, or an entry point; an alternate setting selects a device memory, not an address space REA can interpret.",
createReader: (path, options) =>
new DfuSeArtifactReader(path, sourceOf(options)),
}),
"esp-flash-image": container("esp-flash-image", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation:
"ESP flash layout inventory reads a dump's bootloader region, partition table and partitions and decodes the bootloader and app partitions as ESP images. The OTA selection is derived without app rollback or anti-rollback, and the running partition is unknown. Partitions, nvs_keys included, are extracted as ordinary data; nothing is decrypted.",
createReader: (path, options) =>
new EspFlashImageArtifactReader(
path,
admittedAs("esp-flash-image", options),
options.espFlashSelection,
),
}),
fit: container("fit", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation:
"FIT inventory reports each /images node's stored bytes and its declared type, architecture, OS, compression, load and entry. Image data is not decompressed or decrypted, signatures are not verified, nothing is executed, and load and entry addresses are declarations.",
createReader: (path, options) =>
new FitArtifactReader(path, sourceOf(options)),
}),
unknown: described("unknown", plain),
} satisfies Record<
Exclude<ArtifactFormatId, ZipPackageFormat>,
FormatDescriptor
>;
const ZIP_FORMATS = Object.fromEntries(
ZIP_PACKAGE_FORMATS.map((id) => [id, zipDescriptor(id)]),
) as {
readonly [Id in ZipPackageFormat]: ReturnType<typeof zipDescriptor>;
};
/**
* One descriptor per domain format id. `unknown` has no reader. ZIP package
* families share {@link ZIP_FAMILY_CAPABILITIES}.
*/
const ARTIFACT_FORMAT_REGISTRY = {
...NON_ZIP_FORMATS,
...ZIP_FORMATS,
};
export const ARTIFACT_FORMATS: {
readonly [Id in ArtifactFormatId]: (typeof ARTIFACT_FORMAT_REGISTRY)[Id];
} = ARTIFACT_FORMAT_REGISTRY;
const listedFormatIds: readonly ArtifactFormatId[] = ARTIFACT_FORMAT_IDS;
/** Descriptor for a recognized format id, when the value is one. */
export const artifactFormatDescriptor = (
format: string,
): FormatDescriptor | undefined =>
listedFormatIds.some((id) => id === format)
? ARTIFACT_FORMATS[format as ArtifactFormatId]
: undefined;