feat(artifacts): inventory and extract ESP flash layouts

A whole-flash dump of an ESP32-family chip was read as its bootloader
image, with the partition table, apps and data left as undescribed bytes,
or refused when it began with erased flash. Read it as an esp-flash-image
container instead.

The partition table is decoded as esp_partition_table_verify accepts it
and is found at the caller's partition_table_offset, at the default
0x8000, or by scanning 0x1000-aligned sectors for a table whose MD5
verifies, which the inventory reports as derived. The bootloader is
located at its chip's ROM bootloader offset. Each partition is an entry
with its declaration in flat format_metadata; the bootloader and app
partitions carry ESP image facts from the existing streaming decoder; the
otadata records are reported with a selection derived without rollback or
anti-rollback, and the running partition stays unknown. One unreadable
partition does not hide the others, and extraction writes every partition,
nvs_keys included, as ordinary data.

Flash layouts are detected before single ESP images. The scan runs only
when an ESP image sits at a ROM bootloader offset, and is refused when a
bootloader at another chip's offset shows that the dump does not start at
flash address 0. open_binary gains flash_offset and partition_table_offset,
which require format esp-flash-image and are part of the snapshot target
identity; the CLI takes them with --target-format esp-flash-image.
Comparison treats a partition's label, type, subtype, range and flags as
its declaration.
This commit is contained in:
Entrpi committed 2026-10-28 03:10:07 +11:00
1 parent ea2b3864dd
commit b2e69728e3
49 files changed
+3918 -89

No files matched your search

@@ -673,3 +673,19 @@ refusals. ESP8266 images are refused, and flash-encrypted images are not
recognized. Segments are not disassembled, and RISC-V or Xtensa segments
cannot be opened as raw-binary. See
[ESP images](https://github.com/morluto/rea/blob/main/docs/esp-image.md).
## ESP flash images
A whole-flash dump opens as `esp-flash-image`. `root_format_metadata` holds
`partition_table` (its `method`: `default`, `caller` or a derived
`scan`), `bootloader` and `apps` ESP image facts, `otadata` with a derived
`selected` slot and its `assumptions`, and `partitions_outside_dump`;
`running_partition` is always `unknown`. Partition rows
(`partitions/<label>.bin`) carry the declared `type`, `subtype`, names,
`flash_address`, `size` and flags; `encrypted` is a storage flag, not
proof of ciphertext. For a dump read from another flash address, call
`open_binary` with `format: "esp-flash-image"` and `flash_offset`; a
relocated table can be named with `partition_table_offset`. No table found
means unknown, often flash encryption. Extraction writes `nvs_keys` as
ordinary data. See
[ESP flash images](https://github.com/morluto/rea/blob/main/docs/esp-flash-image.md).
+12
View File
@@ -252,6 +252,18 @@ The inventory lists each segment with its ESP-IDF load classification, the
checksum and SHA-256 results, the descriptor and any Secure Boot v2 signature
blocks. See [ESP images](esp-image.md) for recognition, coverage and refusals.
Inspect an ESP flash dump, or one read from a flash address other than 0:
```sh
rea inspect-artifact flash.bin --json
rea inspect-artifact dump.bin --target-format esp-flash-image --flash-offset 0x1000 --json
rea extract-artifact flash.bin --json
```
The inventory lists the bootloader, the partition table and each partition,
with ESP image facts for app partitions and the OTA records. See
[ESP flash images](esp-flash-image.md) for table location, inputs and limits.
Inspect a Node.js SEA carrier without running it:
```sh
+166
View File
@@ -0,0 +1,166 @@
# ESP flash images
REA reads a dump of an ESP32-family SPI flash, such as the output of
`esptool read-flash 0 ALL dump.bin` or `esptool merge-bin`, through
`inspect_artifact` and `extract_artifact`. The reader is static: no device,
esptool or ESP-IDF is needed, and no code is run.
The container format is `esp-flash-image`. Entries are the bootloader region,
the partition table sector and one file per partition. The bootloader and
each app partition are also decoded as [ESP images](esp-image.md), so their
headers, segments, integrity and descriptors appear in the root
`format_metadata`. Layout facts are cited to ESP-IDF v6.1
(`esp_flash_partitions.h`, `flash_partitions.c`, `gen_esp32part.py`,
`bootloader_common_loader.c`, `bootloader_utility.c` and each chip's
`esp_rom_caps.h`).
## Recognition
A file is a flash image when REA finds a partition table that ESP-IDF's
bootloader accepts (`esp_partition_table_verify`): 32-byte `0x50AA` entries,
at most one `0xEBEB` MD5 entry over the entries before it, and a terminating
entry within `0xC00` bytes. The table is located in this order:
1. at the caller's `partition_table_offset`, when given; no scan follows;
2. at the default `0x8000`;
3. by scanning every `0x1000`-aligned flash sector for a table whose MD5
verifies. This location is derived, and the inventory says so.
Without a caller selection, the scan runs only when an ESP image sits at a
ROM bootloader offset, so ordinary files are not scanned. A C3 or S3 dump
begins with its bootloader, so flash layouts are detected before a file is
read as one ESP image. An image at another chip's ROM offset, such as an
ESP32 bootloader at the start of a dump read from `0x1000`, means the dump's
origin is not flash address 0. REA then does not scan, because a table found
that way would place every partition at the wrong file offset; pass
`flash_offset`.
When no table is found the layout is not recognized. This is an unknown
outcome, not a malformed file: flash encryption, which encrypts the
bootloader and the table, is one cause.
## Caller inputs
| Input | Meaning |
| ------------------------- | ----------------------------------------------------------------------------- |
| `format: esp-flash-image` | Read the file as a flash dump; with no table found, refuse it |
| `flash_offset` | Flash address (`0x…`) of the dump's first byte; default 0 |
| `partition_table_offset` | Flash address (`0x…`, a multiple of `0x1000`) of the table; disables the scan |
MCP takes these on `open_binary`; the CLI takes `--target-format
esp-flash-image`, `--flash-offset` and `--partition-table-offset` on
`inspect-artifact` and `extract-artifact`. The offsets require the format, and
the format takes no `architecture` or `entry_point`. The inputs are part of
the target's identity, so reopening the same bytes with other inputs reads
them again instead of reusing the earlier reading. The `open_binary` summary
reports them as `flash_offset` and `partition_table_offset`, null when
omitted.
## Layout facts
The root `format_metadata` (`esp-flash-image`) records:
- `flash_offset`: its `value` and whether the `caller` gave it or it is the
`default`;
- `chip`: from the bootloader's `chip_id`, or null without a located
bootloader;
- `bootloader`: the region from the chip's ROM bootloader offset to the
table, with its ESP image facts;
- `partition_table`: its flash address, the `method` that found it
(`default`, `caller` or `scan`), the partition count and the MD5 `status`
(`valid`, `invalid` or `absent`) with the stored and computed digests;
- `apps`: ESP image facts for each app partition the dump holds;
- `otadata`: the OTA records and the derived selection;
- `partitions_outside_dump`: declared partitions with no bytes in the file;
- `undescribed`: byte ranges outside every region that are not erased flash.
The ROM loads the bootloader from `ESP_ROM_BOOTLOADER_OFFSET_FLASH`:
`0x1000` for ESP32 and ESP32-S2; `0x0` for ESP32-S3, C2, C3, C6, C61, H2 and
H21; and `0x2000` for ESP32-C5, H4, P4 and S31. A bootloader is located only
where its own `chip_id` says the ROM loads it.
### Partitions
Each partition entry is `partitions/<label>.bin` when the label is a safe,
unique name of letters, digits, `.`, `_` and `-` with at least one letter or
underscore, and `partitions/<index>.bin` otherwise; `source_path` keeps the
label. Its `format_metadata` (`esp-flash-partition`) records the table
`index`, `label` (null when not UTF-8), `type`, `subtype`, ESP-IDF's
`type_name` and `subtype_name`, `flash_address`, `size`, the `encrypted` and
`readonly` flags and any `unknown_flags`. The `encrypted` flag says how a
device with flash encryption stores the partition; it does not establish
that the dump's bytes are ciphertext.
A partition that the file holds only in part is the bytes the file holds,
with a limitation. One unreadable partition does not hide the others:
partitions outside the dump, before its first byte or overlapping another
are each reported while the rest stay readable.
### App partitions and the bootloader
Each app partition and the bootloader region carry ESP image `facts`:
`status` is `esp-image`, `erased` (the image header bytes are erased flash,
as in an empty OTA slot) or `unrecognized` (ciphertext or other data). For an
image they hold the [ESP image](esp-image.md) root facts, `segments`,
`signature_blocks`, `coverage` and `limitations`, with offsets relative to
the region. Extract a partition and inspect it as an ESP image for its
segments as entries.
### OTA data
The otadata partition holds two `esp_ota_select_entry_t` records, one at the
start of each 4 KiB sector. REA reports each record's `ota_seq`,
`seq_label`, `ota_state` and its name, `crc`, whether the CRC matches (CRC-32
of `ota_seq` seeded with `0xFFFFFFFF`) and whether the bootloader would treat
it as valid.
`selected` is derived as `bootloader_utility_get_selected_boot_partition`
does without app rollback or anti-rollback: the valid record with the higher
`ota_seq` selects slot `(ota_seq - 1) % ota_app_count`, indexed by OTA
subtype. With no usable record or no OTA app, the factory app is selected,
else `ota_0`. Its `assumptions` list what the derivation leaves out.
`running_partition` is always `unknown`: at boot the bootloader may first
mark a pending-verify record aborted, check the app's secure version against
eFuse, and fall back to another partition when an app fails to load.
## Coverage and extraction
Erased flash (`0xFF`) between regions is described; any other byte outside
the bootloader region, the table sector and the partitions is listed in
`undescribed` and makes coverage `unknown`. Coverage is also `unknown` when a
partition is truncated or outside the dump, or when no bootloader is
located.
`extract_artifact` writes every entry as an ordinary file, `nvs_keys`
partitions included. On a device without flash encryption an `nvs_keys`
partition holds the NVS encryption keys in plain text, and a dump of it
contains them. REA decrypts nothing, recovers no keys and does not interpret
NVS.
## Comparison
Artifact comparison uses each partition's `label`, `type`, `subtype`,
`flash_address`, `size` and flags as its declaration, so a partition that
moves or changes type with unchanged bytes is `changed` in `metadata`.
Partitions pair by path. The root facts derive from the dump's bytes, which
content comparison covers.
## Limits
- A flash dump inside another container, such as a ZIP, is classified from
its first bytes and so reads as an ESP image, not as a flash layout; extract
it first.
- ESP8266 dumps, Secure Boot v1 bootloader digests and flash-encrypted
layouts are not read.
- NVS, FAT, SPIFFS and other data partitions are extracted, not decoded.
- An image nested in a partition is decoded for its facts; its segments are
not inventory entries.
## Verification
Synthetic fixtures with computed MD5s and CRCs cover the table rules, every
way the table is located, the misplaced-origin guard, the ROM bootloader
offsets, overlaps, truncation and partitions outside the dump, index names
for unsafe labels, the otadata states and the bootloader's selection rules.
CLI and stdio MCP tests exercise the inputs, their identity, extraction and
comparison.
+5 -3
View File
@@ -208,9 +208,11 @@ first byte that is not erased starts a range to the end of that region. A
slot that is neither a block nor erased is reported whole. Adjacent ranges
are merged.
A whole-flash dump of a chip whose bootloader is at offset 0, such as
ESP32-C3 or ESP32-S3, is recognized as that bootloader image; the partition
table, apps and data follow as undescribed bytes.
A whole-flash dump whose partition table REA finds is read as an
[ESP flash image](esp-flash-image.md). Without one, a dump of a chip whose
bootloader is at offset 0, such as ESP32-C3 or ESP32-S3, is recognized as that
bootloader image, and the partition table, apps and data follow as
undescribed bytes.
## Limits
+11
View File
@@ -295,6 +295,17 @@ v2 signature blocks. `extract-artifact` / `extract_artifact` writes one file
per segment and signature block. Reading is portable and needs no external
tool. See [ESP images](esp-image.md).
## Inspect ESP flash dumps
Use the same tools on a whole-flash dump, such as `esptool read-flash 0 ALL`
output. The inventory reports the bootloader, the partition table and how it
was found, each partition with its declared type, flags and digest, ESP image
facts for the bootloader and each app partition, and the OTA records with a
derived selection; the running partition stays unknown. Extraction writes
each partition, `nvs_keys` included, as an ordinary file. For a dump that
does not start at flash address 0, select `esp-flash-image` with
`flash_offset`. See [ESP flash images](esp-flash-image.md).
## Inspect extracted devicetrees
Use [`inspect_devicetree`](devicetree.md) / `rea inspect-devicetree` on an
+14 -14
View File
@@ -47,20 +47,20 @@ For runtime instrumentation, see the [Frida guide](./frida-instrumentation.md).
## Choose a guide
| Task | Guides |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Identify a desktop installation | [Runtime candidates, evidence and follow-up files](desktop-applications.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Linux kernel modules](kernel-modules.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [GameCube DOL](ghidra-dol.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [Kuna](kuna-provider.md), [JEB](jeb-analysis.md) |
| Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) |
| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) |
| Inspect assemblies, packages, or firmware | [Godot PCK](godot-pck.md), [.NET](managed-code-analysis.md), [Python packages](python-packaged-applications.md), [Android](android-analysis.md), [APK signatures](apk-signature.md), [Apktool resources](apktool-resource-analysis.md), [ADB acquisition](adb-device-analysis.md), [Apple](apple-application-analysis.md), [Harmony](harmony-analysis.md), [firmware](firmware-analysis.md), [ESP images](esp-image.md), [WASM](wasm-artifacts.md), [EVM](evm-bytecode.md) |
| Observe or compare a browser scenario | [CDP observation](browser-observation.md), [scenarios](browser-scenario-contract.md), [runtime evidence](web-runtime.md), [historical captures](web-network-captures.md) |
| Serve recorded behavior as an agent environment | [Agent environments](agent-environments.md) |
| Trace captured website source | [Source locations](web-source-location.md), [module traces](website-module-trace.md), [script export](website-script-export.md) |
| Capture and compare a command | [Process capture](process-capture.md) |
| Use CLI/MCP and retain evidence | [CLI](cli.md), [MCP contracts](mcp-contracts.md), [agent prompts](mcp-prompts.md), [obligation ledgers](reconstruction-obligation-ledgers.md) |
| Task | Guides |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Identify a desktop installation | [Runtime candidates, evidence and follow-up files](desktop-applications.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Linux kernel modules](kernel-modules.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [GameCube DOL](ghidra-dol.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [Kuna](kuna-provider.md), [JEB](jeb-analysis.md) |
| Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) |
| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) |
| Inspect assemblies, packages, or firmware | [Godot PCK](godot-pck.md), [.NET](managed-code-analysis.md), [Python packages](python-packaged-applications.md), [Android](android-analysis.md), [APK signatures](apk-signature.md), [Apktool resources](apktool-resource-analysis.md), [ADB acquisition](adb-device-analysis.md), [Apple](apple-application-analysis.md), [Harmony](harmony-analysis.md), [firmware](firmware-analysis.md), [ESP images](esp-image.md), [ESP flash images](esp-flash-image.md), [WASM](wasm-artifacts.md), [EVM](evm-bytecode.md) |
| Observe or compare a browser scenario | [CDP observation](browser-observation.md), [scenarios](browser-scenario-contract.md), [runtime evidence](web-runtime.md), [historical captures](web-network-captures.md) |
| Serve recorded behavior as an agent environment | [Agent environments](agent-environments.md) |
| Trace captured website source | [Source locations](web-source-location.md), [module traces](website-module-trace.md), [script export](website-script-export.md) |
| Capture and compare a command | [Process capture](process-capture.md) |
| Use CLI/MCP and retain evidence | [CLI](cli.md), [MCP contracts](mcp-contracts.md), [agent prompts](mcp-prompts.md), [obligation ledgers](reconstruction-obligation-ledgers.md) |
For development, use [testing](testing.md), [client verification](client-verification.md),
[tool design](tool-design.md), [admission limits and recovery](design/limits-and-recovery.md),
+80 -1
View File
@@ -75,6 +75,11 @@ import {
} from "../domain/apple/appleArchiveFormat.js";
import { hasDiskImageSignature } from "../domain/apple/diskImageFormat.js";
import { hasGodotPckHeader } from "../domain/godotPckFormat.js";
import {
locateEspFlashLayout,
type EspFlashSelection,
} from "../domain/firmware/espFlashLocation.js";
import { espFlashSelectionError } from "../domain/firmware/espFlashSelection.js";
import {
ESP8266_REFUSAL,
recognizeEspImage,
@@ -127,6 +132,8 @@ export interface BinaryTargetResolverOptions {
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
/** Inputs for format=esp-flash-image; rejected with any other format. */
readonly espFlash?: EspFlashSelection;
readonly signal?: AbortSignal;
readonly resources?: ArtifactResourceScope;
}
@@ -264,6 +271,18 @@ const readOpenedBinaryTarget = async (
const hostArchitecture = options.hostArchitecture ?? process.arch;
const { targetKind, formatHint, signal, architecture, entryPoint } = options;
throwIfTargetResolutionCancelled(signal);
// Flash inputs pair with one format; check before any hint branch.
const flashPairing = espFlashSelectionError({
format: formatHint,
flashOffset: options.espFlash?.flashOffset,
partitionTableOffset: options.espFlash?.partitionTableOffset,
});
if (flashPairing !== undefined)
return err(
new BinaryTargetError(path, flashPairing, {
constraint: "load_selection_rejected",
}),
);
if (formatHint === "raw-binary") {
if (targetKind !== undefined && targetKind !== "executable")
return err(
@@ -351,6 +370,45 @@ const readOpenedBinaryTarget = async (
dol: admitted.value.header,
});
}
if (formatHint === "esp-flash-image") {
if (targetKind !== undefined && targetKind !== "archive")
return err(
new BinaryTargetError(
path,
"ESP flash image format requires an archive target kind",
{ constraint: "load_selection_rejected" },
),
);
const selection = options.espFlash ?? {};
// An encrypted dump has no readable table: unknown, not malformed.
const located = await locateEspFlashLayout(
metadata.size,
(position, length) => readHandleRange(handle, position, length, signal),
selection,
{ explicit: true, ...(signal === undefined ? {} : { signal }) },
);
if (!located.ok)
return err(
new BinaryTargetError(
path,
`ESP flash image layout is not recognized: ${located.error}`,
{ constraint: "file_format_rejected" },
),
);
return ok({
path,
sourcePath,
artifactSelectedPath: selectedPath,
sha256: await sha256Handle(handle, path, metadata, signal),
kind: "archive",
format: "esp-flash-image",
espFlash: {
selected: true,
flashOffset: selection.flashOffset ?? null,
partitionTableOffset: selection.partitionTableOffset ?? null,
},
});
}
if (formatHint === "dos-com") {
if (targetKind !== undefined && targetKind !== "executable")
return err(
@@ -403,6 +461,17 @@ const readOpenedBinaryTarget = async (
artifactSelectedPath: selectedPath,
sha256: await sha256Handle(handle, path, metadata, signal),
};
if (artifactFormat === "esp-flash-image")
return ok({
...identity,
kind: "archive",
format: artifactFormat,
espFlash: {
selected: false,
flashOffset: null,
partitionTableOffset: null,
},
});
return isArchiveFormat(artifactFormat)
? ok({ ...identity, kind: "archive", format: artifactFormat })
: ok({ ...identity, kind: "artifact", format: artifactFormat });
@@ -669,6 +738,14 @@ const detectArtifactFormat = async (
);
if (hasZipSignature(magic)) return zipRootPackageFormat(lower);
if (hasGodotPckHeader(magic)) return "godot-pck";
// A C3 or S3 dump begins with its bootloader, so a layout comes first.
const flash = await locateEspFlashLayout(
initial.size,
(position, length) => readHandleRange(handle, position, length, signal),
{},
{ explicit: false, ...(signal === undefined ? {} : { signal }) },
);
if (flash.ok) return "esp-flash-image";
const esp = recognizeEspImage(magic, initial.size);
if (esp === "esp8266")
throw new BinaryTargetError(path, ESP8266_REFUSAL, {
@@ -742,7 +819,9 @@ const isArchiveFormat = (
>,
): format is Extract<BinaryTarget, { kind: "archive" }>["format"] =>
isZipPackageFormat(format) ||
["asar", "godot-pck", "dmg", "pkg", "esp-image"].includes(format);
["asar", "godot-pck", "dmg", "pkg", "esp-image", "esp-flash-image"].includes(
format,
);
/** Whether the leading bytes begin any executable header family, valid or not. */
const claimsLeadingExecutableHeader = async (
+53 -46
View File
@@ -1,3 +1,4 @@
import type { EspFlashSelection } from "../domain/firmware/espFlashLocation.js";
import type { ExecutableFormatHint } from "../domain/dosCom.js";
import { dispatchQueuedInterrupts } from "../process/queuedInterrupts.js";
import type {
@@ -132,6 +133,7 @@ export const runDirectAnalysis = async (
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
readonly espFlash?: EspFlashSelection;
} = {},
): Promise<Result<Evidence, JsonValue>> =>
withProcessCancellation(options.signal, (signal) =>
@@ -139,25 +141,50 @@ export const runDirectAnalysis = async (
logger: options.logger ?? silentLogger,
snapshotPath: options.snapshotPath,
signal,
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...targetSelection(options),
...(options.providerId === undefined
? {}
: { providerId: options.providerId }),
}),
);
/** How the caller asks an artifact target to be read. */
export interface ArtifactTargetSelection {
readonly formatHint?: ExecutableFormatHint;
readonly espFlash?: EspFlashSelection;
}
/** The caller's target interpretation, forwarded unchanged to every open. */
const targetSelection = (options: {
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
readonly espFlash?: EspFlashSelection;
}) => ({
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...(options.espFlash === undefined ? {} : { espFlash: options.espFlash }),
});
/** Execute one provider-native semantic operation with atomic provenance. */
export const runProviderAnalysis = async (
dependencies: DirectAnalysisDependencies,
...[path, tool, arguments_, logger = silentLogger, signal, delivery]:
...[
path,
tool,
arguments_,
logger = silentLogger,
signal,
delivery,
selection,
]:
| readonly [
path: string,
tool: NativeToolName | ArtifactAnalysisOperation | ManagedToolName,
@@ -176,6 +203,15 @@ export const runProviderAnalysis = async (
readonly evidenceBundlePath?: string;
},
]
| readonly [
path: string,
tool: ArtifactAnalysisOperation,
arguments_: Readonly<Record<string, JsonValue>>,
logger: Logger | undefined,
signal: AbortSignal | undefined,
delivery: undefined,
selection: ArtifactTargetSelection,
]
): Promise<Result<Evidence, JsonValue>> => {
if (delivery !== undefined && tool !== "inspect_managed_members")
return err(
@@ -247,6 +283,7 @@ export const runProviderAnalysis = async (
snapshotPath: undefined,
signal: operationSignal,
evidenceParameters,
...targetSelection(selection ?? {}),
...(ARTIFACT_ANALYSIS_OPERATION_SET.has(tool)
? { createSession: dependencies.createArtifactBinarySession }
: NATIVE_INSPECTION_OPERATIONS.has(tool)
@@ -327,6 +364,7 @@ const runAnalysis = async (
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
readonly espFlash?: EspFlashSelection;
},
): Promise<Result<Evidence, JsonValue>> => {
const { logger, signal, snapshotPath } = options;
@@ -373,15 +411,7 @@ const runAnalysis = async (
path,
snapshotPath,
signal,
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...targetSelection(options),
});
if (!prepared.ok) return err(prepared.error);
const { snapshot } = prepared.value;
@@ -390,15 +420,7 @@ const runAnalysis = async (
const preview = await session.previewTarget(prepared.value.target, {
signal,
snapshot,
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...targetSelection(options),
...(options.providerId === undefined
? {}
: { providerId: options.providerId }),
@@ -437,15 +459,7 @@ const runAnalysis = async (
}
const openOptions = {
signal,
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...targetSelection(options),
...(snapshot === undefined ? {} : { snapshot }),
...(options.providerId === undefined
? {}
@@ -703,6 +717,7 @@ const prepareSnapshot = async (options: {
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
readonly espFlash?: EspFlashSelection;
readonly snapshotPath: string | undefined;
readonly signal: AbortSignal;
}): Promise<
@@ -721,15 +736,7 @@ const prepareSnapshot = async (options: {
if (!loaded.ok) return loaded;
const target = await session.resolveTarget(path, {
signal: options.signal,
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...targetSelection(options),
});
if (!target.ok) return target;
if (!snapshotMatchesTarget(loaded.value.target, target.value))
@@ -1,3 +1,4 @@
import type { EspFlashSelection } from "../../domain/firmware/espFlashLocation.js";
import type { ExecutableFormatHint } from "../../domain/dosCom.js";
import type {
BinaryArchitecture,
@@ -38,6 +39,8 @@ export interface BinarySessionOpenOptions {
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
/** Inputs for format=esp-flash-image. */
readonly espFlash?: EspFlashSelection;
readonly snapshot?: AnalysisSnapshot;
/**
* Load the snapshot inside the open's turn instead of before it, so calls
@@ -101,6 +104,7 @@ export const resolveSessionOpen = async (
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...(options.espFlash === undefined ? {} : { espFlash: options.espFlash }),
...(options.signal === undefined ? {} : { signal: options.signal }),
});
if (!parsed.ok) return parsed;
@@ -1,3 +1,4 @@
import type { EspFlashSelection } from "../../domain/firmware/espFlashLocation.js";
import type { InvestigationRecordPort } from "../investigation/InvestigationRecordPort.js";
import type { ExecutableFormatHint } from "../../domain/dosCom.js";
import type {
@@ -64,6 +65,7 @@ export interface BinarySessionPort
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
readonly espFlash?: EspFlashSelection;
readonly snapshot?: AnalysisSnapshot;
readonly loadSnapshot?: AnalysisSnapshotLoader;
readonly providerId?: AnalysisProviderSelector;
@@ -13,4 +13,10 @@ export const binaryTargetSummary = (target: BinaryTarget) =>
...(target.format === "raw-binary" || target.format === "gamecube-dol"
? { entry_point: target.entryPoint }
: {}),
...(target.format === "esp-flash-image"
? {
flash_offset: target.espFlash.flashOffset,
partition_table_offset: target.espFlash.partitionTableOffset,
}
: {}),
});
+14
View File
@@ -1,3 +1,4 @@
import type { EspFlashSelection } from "../domain/firmware/espFlashLocation.js";
import type { AnalysisSettingsReader } from "../domain/analysisSettings.js";
import { runtimeAsarHeaderAdmission } from "../config/runtimeAnalysisSettings.js";
import { artifactTargetRefusal } from "../domain/apple/artifactTargetApplicability.js";
@@ -332,6 +333,7 @@ class ArtifactClient implements AnalysisClient {
: undefined,
selectedPath: this.target.artifactSelectedPath,
inputFormat: this.target.format,
espFlashSelection: espFlashSelection(this.target),
outputRoot: parsed.output_root,
environment: this.environment,
asarHeaderAdmission: runtimeAsarHeaderAdmission(
@@ -542,12 +544,24 @@ class ArtifactClient implements AnalysisClient {
this.environment,
this.settings,
),
espFlashSelection: espFlashSelection(this.target),
...(options?.signal === undefined ? {} : { signal: options.signal }),
integrity: parsed.integrity_policy,
});
}
}
/** Caller inputs of an explicitly selected ESP flash image, for its readers. */
const espFlashSelection = (
target: BinaryTarget,
): EspFlashSelection | undefined =>
target.format === "esp-flash-image" && target.espFlash.selected
? {
flashOffset: target.espFlash.flashOffset ?? undefined,
partitionTableOffset: target.espFlash.partitionTableOffset ?? undefined,
}
: undefined;
const isArtifactOperation = (
operation: AnalysisOperation,
): operation is ArtifactAnalysisOperation =>
+1
View File
@@ -180,6 +180,7 @@ export interface ArtifactReader {
| "pyinstaller"
| "node-sea"
| "esp-image"
| "esp-flash-image"
| "file";
entries(
signal?: AbortSignal,
@@ -0,0 +1,371 @@
import { readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { describe, expect, it, vi } from "vitest";
import { EspFlashImageArtifactReader } from "./EspFlashImageArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import { extractArtifact } from "./extraction/ArtifactExtraction.js";
import * as espImageDecoding from "../domain/firmware/espImage.js";
import {
TWO_OTA_PARTITIONS,
espFlashDump,
otaData,
otaRecord,
partitionTable,
} from "../domain/firmware/espFlash.fixture.js";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
espImage,
pattern,
} from "../domain/firmware/espImage.fixture.js";
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
import { scanCanonicalArtifactInventory } from "../../tests/fixtures/artifactInventory.js";
const INVENTORY_LIMITATION =
"ESP flash layout inventory reads a dump's bootloader region, partition table and partitions and decodes the bootloader and app partitions as ESP images. The OTA selection is derived without app rollback or anti-rollback, and the running partition is unknown. Partitions, nvs_keys included, are extracted as ordinary data; nothing is decrypted.";
const BOOTLOADER = espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS });
const APP = espImage({ segments: ESP32C3_APP_SEGMENTS });
/** A C3 dump: factory and ota_0 hold the app, ota_1 is erased, otadata selects ota_0. */
const DUMP = espFlashDump({
bootloader: { at: 0, bytes: BOOTLOADER },
parts: [
{ at: 0xd000, bytes: otaData(otaRecord({ seq: 1 })) },
{ at: 0x1_0000, bytes: APP },
{ at: 0x2_0000, bytes: APP },
],
});
const writeDump = async (bytes: Uint8Array, name = "flash.bin") => {
const path = join(await createTestTempDirectory("rea-esp-flash-"), name);
await writeFile(path, bytes);
return path;
};
const root = (
snapshot: Awaited<ReturnType<typeof scanCanonicalArtifactInventory>>,
) =>
snapshot.occurrences.find(({ logical_path }) => logical_path === ".")
?.format_metadata;
describe("ESP flash layout inventory", () => {
it("inventories the bootloader, table and partitions with app and otadata facts", async () => {
const snapshot = await scanCanonicalArtifactInventory(
await writeDump(DUMP),
);
expect(snapshot.manifest.root_format).toBe("esp-flash-image");
expect(snapshot.coverage).toEqual({
scope: "container-entries",
status: "complete",
});
expect(snapshot.limitations).toEqual([INVENTORY_LIMITATION]);
expect(
snapshot.occurrences.map(({ logical_path }) => logical_path),
).toEqual([
".",
"bootloader.bin",
"partition-table.bin",
"partitions/factory.bin",
"partitions/nvs.bin",
"partitions/ota_0.bin",
"partitions/ota_1.bin",
"partitions/otadata.bin",
"partitions/phy_init.bin",
]);
expect(root(snapshot)).toMatchObject({
kind: "esp-flash-image",
flash_offset: { value: 0, source: "default" },
chip: { target: "esp32c3", isa: "riscv" },
bootloader: {
flash_address: 0,
region_size: 0x8000,
facts: {
status: "esp-image",
image: { descriptor: { kind: "bootloader" } },
coverage: "complete",
},
},
partition_table: {
flash_address: 0x8000,
method: "default",
partition_count: 6,
md5: { status: "valid" },
},
partitions_outside_dump: [],
otadata: {
partition_index: 1,
selected: {
active_record: 0,
slot: 0,
partition_index: 4,
basis: "otadata",
},
running_partition: "unknown",
},
undescribed: [],
});
const apps = (
root(snapshot) as {
apps: { partition_index: number; facts: { status: string } }[];
}
).apps;
expect(
apps.map(({ partition_index, facts }) => [partition_index, facts.status]),
).toEqual([
[3, "esp-image"],
[4, "esp-image"],
[5, "erased"],
]);
const factory = snapshot.occurrences.find(
({ logical_path }) => logical_path === "partitions/factory.bin",
);
expect(factory).toMatchObject({
source_path: "factory",
artifact_format: "esp-image",
format_metadata: {
kind: "esp-flash-partition",
index: 3,
label: "factory",
type_name: "app",
subtype_name: "factory",
flash_address: 0x1_0000,
size: 0x1_0000,
},
});
});
it("finds a relocated table by scanning and says the location is derived", async () => {
const snapshot = await scanCanonicalArtifactInventory(
await writeDump(
espFlashDump({
bootloader: { at: 0, bytes: BOOTLOADER },
tableAt: 0xa000,
}),
),
);
expect(root(snapshot)).toMatchObject({
partition_table: { flash_address: 0xa000, method: "scan" },
});
expect(snapshot.limitations).toContain(
"The partition table at 0xa000 was found by scanning 0x1000-aligned flash sectors for a table whose MD5 verifies; the dump does not record the bootloader's configured table offset.",
);
});
});
describe("ESP flash layout coverage and selection", () => {
it("keeps usable partitions beside a truncated and a missing one", async () => {
const cut = DUMP.subarray(0, 0x2_8000);
const snapshot = await scanCanonicalArtifactInventory(await writeDump(cut));
expect(snapshot.coverage.status).toBe("unknown");
const ota0 = snapshot.occurrences.find(
({ logical_path }) => logical_path === "partitions/ota_0.bin",
);
expect(ota0).toMatchObject({
declared_size: 0x8000,
limitations: [
"partitions/ota_0.bin is truncated: the dump holds 32768 of its 65536 bytes.",
],
});
expect(root(snapshot)).toMatchObject({
partitions_outside_dump: [{ index: 5, label: "ota_1" }],
});
expect(
snapshot.occurrences.some(
({ logical_path }) => logical_path === "partitions/factory.bin",
),
).toBe(true);
});
it("reads a dump that starts at the caller's flash offset", async () => {
const esp32 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0000,
});
const whole = espFlashDump({ bootloader: { at: 0x1000, bytes: esp32 } });
const snapshot = await scanCanonicalArtifactInventory(
await writeDump(whole.subarray(0x1000)),
{
espFlashSelection: { flashOffset: 0x1000 },
},
);
expect(root(snapshot)).toMatchObject({
flash_offset: { value: 0x1000, source: "caller" },
chip: { target: "esp32" },
bootloader: { flash_address: 0x1000, region_size: 0x7000 },
partition_table: { flash_address: 0x8000, method: "default" },
});
const nvs = snapshot.occurrences.find(
({ logical_path }) => logical_path === "partitions/nvs.bin",
);
expect(nvs?.source_location).toEqual({ offset: 0x8000, length: 0x4000 });
});
it("refuses a selected dump without a readable table as not recognized", async () => {
const reader = new EspFlashImageArtifactReader(
await writeDump(pattern(0x2_0000, 7)),
);
try {
await expect(
reader.entries()[Symbol.asyncIterator]().next(),
).rejects.toMatchObject({
reason: "unavailable",
message: expect.stringContaining(
"ESP flash image layout is not recognized",
),
});
} finally {
await reader.close();
}
});
});
describe("ESP flash layout extraction and lifecycle", () => {
it("extracts every partition byte-exactly, nvs_keys included", async () => {
const parts = [
...TWO_OTA_PARTITIONS.slice(0, 3),
{
label: "keys",
type: 0x01,
subtype: 0x04,
offset: 0x1_0000,
size: 0x1000,
flags: 1,
},
];
const keys = pattern(0x1000, 3);
const dump = espFlashDump({
size: 0x1_1000,
bootloader: { at: 0, bytes: BOOTLOADER },
table: partitionTable(parts),
parts: [{ at: 0x1_0000, bytes: keys }],
});
const directory = await createTestTempDirectory("rea-esp-flash-extract-");
const path = join(directory, "flash.bin");
await writeFile(path, dump);
const resourceScope = new ArtifactResourceScope();
try {
const extracted = await extractArtifact({
inputPath: path,
inputFormat: "esp-flash-image",
outputRoot: join(directory, "out"),
environment: process.env,
integrityPolicy: "fail",
resourceScope,
});
expect(extracted.containment_verified).toBe(true);
expect(
extracted.artifacts.map(({ relative_path }) => relative_path).sort(),
).toEqual([
"bootloader.bin",
"partition-table.bin",
"partitions/keys.bin",
"partitions/nvs.bin",
"partitions/otadata.bin",
"partitions/phy_init.bin",
]);
for (const [name, offset, length] of [
["bootloader.bin", 0, 0x8000],
["partition-table.bin", 0x8000, 0x1000],
["partitions/keys.bin", 0x1_0000, 0x1000],
] as const)
expect(
Buffer.compare(
await readFile(join(extracted.output_root, name)),
dump.subarray(offset, offset + length),
),
).toBe(0);
} finally {
await resourceScope.close();
}
});
it.each(["cancel", "close"] as const)(
"settles app decoding when the caller requests %s after the first decoded chunk",
async (action) => {
const large = espImage({
segments: [{ address: 0x4038_0000, data: Buffer.alloc(8 << 20, 0x5a) }],
});
const parts = [
{
label: "factory",
type: 0,
subtype: 0,
offset: 0x1_0000,
size: 0x90_0000,
},
];
const reader = new EspFlashImageArtifactReader(
await writeDump(
espFlashDump({
size: 0x91_0000,
bootloader: { at: 0, bytes: BOOTLOADER },
table: partitionTable(parts),
parts: [{ at: 0x1_0000, bytes: large }],
}),
),
);
const controller = new AbortController();
const reason = { stage: "decoding" };
let completion: Promise<void> | undefined;
let decoded = 0;
let acknowledge = (): void => undefined;
const decoding = new Promise<void>((resolve) => {
acknowledge = resolve;
});
const create = espImageDecoding.createEspImageDecoder;
// The bootloader decodes in one chunk; acknowledge the app's second chunk.
const observer = vi
.spyOn(espImageDecoding, "createEspImageDecoder")
.mockImplementation((size) => {
const decoder = create(size);
let writes = 0;
return {
write: (chunk) => {
decoder.write(chunk);
decoded += 1;
writes += 1;
if (size > 0x10_0000 && writes === 1) acknowledge();
},
get satisfied() {
return decoder.satisfied;
},
finish: () => decoder.finish(),
};
});
try {
const pending = reader
.entries(controller.signal)
[Symbol.asyncIterator]()
.next();
await Promise.race([
decoding,
pending.then(() => {
throw new Error("Preparation settled before app decoding began");
}),
]);
const requestedAt = decoded;
if (action === "cancel") controller.abort(reason);
else completion = reader.close();
await expect(pending).rejects.toMatchObject(
action === "cancel"
? { reason: "cancelled", cause: reason }
: { reason: "unavailable" },
);
await completion;
expect(decoded).toBe(requestedAt);
await reader.close();
await expect(
reader.entries()[Symbol.asyncIterator]().next(),
).rejects.toMatchObject({
reason: "unavailable",
});
} finally {
observer.mockRestore();
await reader.close();
}
},
);
});
@@ -0,0 +1,436 @@
import { Readable, Transform, pipeline } from "node:stream";
import type { FileHandle } from "node:fs/promises";
import type { EspFlashImageMetadata } from "../domain/artifactFormatMetadata.js";
import { ESP_CHIP_TABLE_SOURCE } from "../domain/firmware/espChips.js";
import {
espFlashPartitionFacts,
finishEspFlashLayout,
planEspFlashLayout,
type EspFlashLayoutFacts,
type EspFlashPlan,
type EspFlashRegion,
} from "../domain/firmware/espFlashLayout.js";
import {
locateEspFlashLayout,
type EspFlashLocation,
type EspFlashSelection,
} from "../domain/firmware/espFlashLocation.js";
import { offsetText } from "../domain/firmware/espImageStructures.js";
import { ESP_OTA_SELECT_ENTRY_BYTES } from "../domain/firmware/espOtaData.js";
import { isEspAppPartition } from "../domain/firmware/espPartitionTable.js";
import {
ArtifactReaderFailure,
copyArtifactEntry,
sameArtifactEntry,
type ArtifactEntry,
type ArtifactReader,
type ArtifactReaderTraversal,
type ArtifactTraversalOptions,
} from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import { ArtifactFileSource } from "./ArtifactFileSource.js";
import {
decodeEspFlashImage,
firstUnerasedInRange,
readEspFlashRange,
type EspFlashReadContext,
} from "./EspFlashRegionReads.js";
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js";
const LABEL = "ESP flash image";
/** A located layout borrowed with the descriptor that produced it. */
export interface EspFlashAdmission {
readonly kind: "esp-flash-image";
readonly source: StableRegularFileDescriptor;
readonly location: EspFlashLocation;
}
type EspFlashImageFacts = NonNullable<
EspFlashImageMetadata["bootloader"]
>["facts"];
/**
* Inventory an ESP flash dump as its bootloader region, partition table and
* partitions. The bootloader and app partitions are decoded as ESP images;
* each entry is a positional read of the carrier, rechecked against its
* identity.
*/
export class EspFlashImageArtifactReader implements ArtifactReader {
readonly format = "esp-flash-image" as const;
readonly #scope = new ArtifactResourceScope();
readonly #entries = new Map<string, ArtifactEntry>();
readonly #streams = new Set<Readable>();
readonly #source: ArtifactFileSource;
readonly #located: EspFlashLocation | undefined;
readonly #selection: EspFlashSelection;
#prepared: Promise<void> | undefined;
#metadata: EspFlashImageMetadata | undefined;
#coverage: ArtifactReaderTraversal | undefined;
#closing = false;
constructor(
path: string,
admitted?: EspFlashAdmission | StableRegularFileDescriptor,
selection: EspFlashSelection = {},
) {
const admission =
admitted !== undefined && "kind" in admitted ? admitted : undefined;
this.#source = new ArtifactFileSource(
path,
LABEL,
admission?.source ??
(admitted as StableRegularFileDescriptor | undefined),
);
this.#located = admission?.location;
this.#selection = selection;
}
formatMetadata(): EspFlashImageMetadata | undefined {
return this.#metadata === undefined
? undefined
: structuredClone(this.#metadata);
}
async prepareContainer(
expectedSha256: string,
signal?: AbortSignal,
): Promise<void> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
await this.#source.bindSha256(expectedSha256, signal);
});
}
async *entries(
signal?: AbortSignal,
options?: ArtifactTraversalOptions,
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
this.#prepared ??= this.#prepare(signal);
await this.#prepared;
await this.#source.read(signal);
});
if (this.#coverage !== undefined)
options?.onCoverage?.({
...this.#coverage,
limitations: [...this.#coverage.limitations],
});
for (const entry of this.#entries.values()) {
this.#assertOpen(signal);
yield copyArtifactEntry(entry);
}
}
async open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
return this.#scope.run(async () => {
this.#assertOpen(signal);
const stored = this.#entries.get(entry.adapterKey);
if (
stored === undefined ||
!sameArtifactEntry(entry, stored) ||
stored.byteOffset === null ||
stored.declaredSize === null
)
throw new ArtifactReaderFailure(
"integrity",
`${LABEL} entry does not match its owning inventory`,
);
const { handle } = await this.#source.read(signal);
this.#assertOpen(signal);
const source =
stored.declaredSize === 0
? Readable.from([])
: readFileHandleChunks(handle, {
start: stored.byteOffset,
end: stored.byteOffset + stored.declaredSize - 1,
signal,
cancellationFailure: (reason) =>
new ArtifactReaderFailure(
"cancelled",
`${LABEL} read cancelled`,
{ cause: reason },
),
});
const output = this.#entryStream(stored, signal);
// Retirement can destroy the stream before its consumer listens.
output.on("error", () => undefined);
this.#streams.add(output);
output.once("close", () => this.#streams.delete(output));
void this.#scope
.run(
() =>
new Promise<void>((resolve) => {
pipeline(source, output, () => {
resolve();
});
}),
)
.catch((cause: unknown) =>
output.destroy(
cause instanceof Error
? cause
: new Error(`${LABEL} stream ownership failed`, { cause }),
),
);
return output;
});
}
async close(): Promise<void> {
this.#closing = true;
for (const stream of this.#streams)
stream.destroy(
new ArtifactReaderFailure("unavailable", `${LABEL} reader closed`),
);
await this.#scope.close();
await this.#source.close();
this.#entries.clear();
}
provenance(): readonly [] {
return [];
}
#assertOpen(signal?: AbortSignal): void {
assertArtifactReadActive(signal);
if (this.#closing)
throw new ArtifactReaderFailure(
"unavailable",
`${LABEL} reader is closed`,
);
}
async #prepare(signal?: AbortSignal): Promise<void> {
const { handle, initial } = await this.#source.read(signal);
const fileSize = initial.size;
const location =
this.#located ?? (await this.#locate(handle, fileSize, signal));
const plan = planEspFlashLayout({
fileSize,
origin: location.origin,
bootloader: location.bootloader,
tableAddress: location.tableAddress,
tableMethod: location.tableMethod,
table: location.table,
});
const context: EspFlashReadContext = { handle, fileSize, signal };
const facts = await this.#readFacts(context, plan);
const images = await this.#decodeImages(context, plan);
// A changed carrier is an identity failure, not a different layout.
await this.#source.read(signal);
for (const region of plan.regions) {
const entry = regionEntry(region);
this.#entries.set(entry.adapterKey, entry);
}
this.#metadata = layoutMetadata(location, plan, facts, images);
const limitations = [
...location.limitations,
...(location.tableMethod === "scan"
? [
`The partition table at ${offsetText(location.tableAddress)} was found by scanning 0x1000-aligned flash sectors for a table whose MD5 verifies; the dump does not record the bootloader's configured table offset.`,
]
: []),
...facts.limitations,
];
if (facts.coverage === "unknown" || limitations.length > 0)
this.#coverage = { status: facts.coverage, limitations };
}
async #locate(
handle: FileHandle,
fileSize: number,
signal?: AbortSignal,
): Promise<EspFlashLocation> {
const located = await locateEspFlashLayout(
fileSize,
(position, length) =>
readEspFlashRange({ handle, fileSize, signal }, position, length),
this.#selection,
{ explicit: true, ...(signal === undefined ? {} : { signal }) },
);
if (!located.ok)
throw new ArtifactReaderFailure(
"unavailable",
`${LABEL} layout is not recognized: ${located.error}`,
);
return located.value;
}
async #readFacts(
context: EspFlashReadContext,
plan: EspFlashPlan,
): Promise<EspFlashLayoutFacts> {
const record = (position: number) =>
readEspFlashRange(context, position, ESP_OTA_SELECT_ENTRY_BYTES);
const otadataRecords =
plan.otadata === undefined
? undefined
: ([
await record(plan.otadata.records[0]),
await record(plan.otadata.records[1]),
] as const);
const gapFirstUnerased: (number | undefined)[] = [];
for (const gap of plan.gaps) {
this.#assertOpen(context.signal);
gapFirstUnerased.push(
await firstUnerasedInRange(context, gap.offset, gap.length),
);
}
return finishEspFlashLayout(plan, { otadataRecords, gapFirstUnerased });
}
/** ESP image facts for the bootloader region and each app partition. */
async #decodeImages(
context: EspFlashReadContext,
plan: EspFlashPlan,
): Promise<ReadonlyMap<EspFlashRegion, EspFlashImageFacts>> {
const images = new Map<EspFlashRegion, EspFlashImageFacts>();
for (const region of plan.regions) {
const image =
region.role === "bootloader" ||
(region.partition !== undefined && isEspAppPartition(region.partition));
if (!image) continue;
this.#assertOpen(context.signal);
images.set(
region,
await decodeEspFlashImage(
context,
{ start: region.fileOffset, length: region.available },
() => this.#assertOpen(context.signal),
),
);
}
return images;
}
#entryStream(entry: ArtifactEntry, signal?: AbortSignal): Transform {
let bytes = 0;
const check = async (): Promise<void> => {
if (bytes !== entry.declaredSize)
throw new ArtifactReaderFailure(
"integrity",
`${LABEL} entry ended early: ${entry.path}`,
);
await this.#source.read(signal);
};
return new Transform({
transform(chunk: Buffer, _encoding, callback) {
bytes += chunk.length;
callback(null, chunk);
},
flush(callback) {
void check().then(
() => callback(),
(cause: unknown) =>
callback(
cause instanceof Error
? cause
: new Error(`${LABEL} read failed`, { cause }),
),
);
},
});
}
}
/** One positional entry; a partition's declared flags stay metadata. */
const regionEntry = (region: EspFlashRegion): ArtifactEntry => ({
path: region.path,
...(region.partition?.label === null || region.partition === undefined
? {}
: { sourcePath: region.partition.label }),
kind: "file",
declaredSize: region.available,
compressedSize: null,
executable: false,
// The table's encrypted flag says how a device stores the partition, not
// that these bytes are ciphertext.
encrypted: false,
byteOffset: region.fileOffset,
declaredIntegrity: null,
unpacked: false,
limitations:
region.available < region.length
? [
`${region.path} is truncated: the dump holds ${String(region.available)} of its ${String(region.length)} bytes.`,
]
: [],
adapterKey: `esp-flash-image:${region.path}`,
...(region.partition === undefined
? {}
: { formatMetadata: espFlashPartitionFacts(region.partition) }),
});
const layoutMetadata = (
location: EspFlashLocation,
plan: EspFlashPlan,
facts: EspFlashLayoutFacts,
images: ReadonlyMap<EspFlashRegion, EspFlashImageFacts>,
): EspFlashImageMetadata => {
const bootloaderRegion = plan.regions.find(
({ role }) => role === "bootloader",
);
const bootloaderFacts =
bootloaderRegion === undefined ? undefined : images.get(bootloaderRegion);
const placed = new Set(
plan.regions.flatMap(({ partition }) =>
partition === undefined ? [] : [partition.index],
),
);
const { bootloader, table } = location;
return {
kind: "esp-flash-image",
flash_offset: location.origin,
chip:
bootloader === undefined
? null
: {
target: bootloader.chip.target,
isa: bootloader.chip.isa,
source: ESP_CHIP_TABLE_SOURCE,
},
bootloader:
bootloaderRegion === undefined || bootloaderFacts === undefined
? null
: {
flash_address: bootloaderRegion.flashAddress,
region_size: bootloaderRegion.length,
facts: bootloaderFacts,
},
partition_table: {
flash_address: location.tableAddress,
method: location.tableMethod,
partition_count: table.partitions.length,
md5: {
status: table.md5.status,
stored: table.md5.stored,
computed: table.md5.computed,
},
},
partitions_outside_dump: table.partitions
.filter(({ index }) => !placed.has(index))
.map(espFlashPartitionFacts),
apps: [...images].flatMap(([region, image]) =>
region.partition === undefined
? []
: [{ partition_index: region.partition.index, facts: image }],
),
otadata:
facts.otadata === null
? null
: {
...facts.otadata,
records: facts.otadata.records.map((record) => ({ ...record })),
selected: {
...facts.otadata.selected,
assumptions: [...facts.otadata.selected.assumptions],
},
},
undescribed: [...facts.undescribed],
};
};
+132
View File
@@ -0,0 +1,132 @@
import type { FileHandle } from "node:fs/promises";
import type { EspFlashImageMetadata } from "../domain/artifactFormatMetadata.js";
import {
ESP8266_REFUSAL,
createEspImageDecoder,
recognizeEspImage,
} from "../domain/firmware/espImage.js";
import { firstUnerased } from "../domain/firmware/espImageStructures.js";
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
import { ArtifactReaderFailure } from "./ArtifactReader.js";
const LABEL = "ESP flash image";
const HEADER_BYTES = 24;
const PROBE_BYTES = 32;
/** The carrier being read, its size and the caller's cancellation. */
export interface EspFlashReadContext {
readonly handle: FileHandle;
readonly fileSize: number;
readonly signal?: AbortSignal | undefined;
}
type EspFlashImageFacts = NonNullable<
EspFlashImageMetadata["bootloader"]
>["facts"];
const chunks = (
{ handle, signal }: EspFlashReadContext,
start: number,
length: number,
): AsyncIterable<Buffer> =>
readFileHandleChunks(handle, {
start,
end: start + length - 1,
signal,
cancellationFailure: (reason) =>
new ArtifactReaderFailure("cancelled", `${LABEL} read cancelled`, {
cause: reason,
}),
}) as AsyncIterable<Buffer>;
/** Up to `length` bytes at a file offset; fewer at the end of the file. */
export const readEspFlashRange = async (
context: EspFlashReadContext,
position: number,
length: number,
): Promise<Buffer> => {
const available = Math.min(length, context.fileSize - position);
if (position < 0 || available <= 0) return Buffer.alloc(0);
const parts: Buffer[] = [];
for await (const chunk of chunks(context, position, available))
parts.push(chunk);
return Buffer.concat(parts);
};
/** The file offset of the first byte in a range that is not erased flash. */
export const firstUnerasedInRange = async (
context: EspFlashReadContext,
start: number,
length: number,
): Promise<number | undefined> => {
if (length <= 0) return undefined;
let position = start;
for await (const chunk of chunks(context, start, length)) {
const found = firstUnerased(chunk, position);
if (found !== undefined) return found;
position += chunk.length;
}
return undefined;
};
const noImage = (
status: "erased" | "unrecognized",
limitations: readonly string[],
): EspFlashImageFacts => ({
status,
image: null,
segments: [],
signature_blocks: [],
coverage: null,
limitations: [...limitations],
});
/**
* Decode an ESP image at the start of a region, as the bootloader reads a
* partition: one streaming pass with the ESP image decoder. Offsets in the
* facts are relative to the region; `erased` means the header bytes are
* erased flash.
*/
export const decodeEspFlashImage = async (
context: EspFlashReadContext,
region: { readonly start: number; readonly length: number },
assertOpen: () => void,
): Promise<EspFlashImageFacts> => {
// The region's end bounds the read, so a short region stays short.
const prefix = await readEspFlashRange(
{ ...context, fileSize: region.start + region.length },
region.start,
PROBE_BYTES,
);
if (
prefix.length >= HEADER_BYTES &&
prefix.subarray(0, HEADER_BYTES).every((byte) => byte === 0xff)
)
return noImage("erased", []);
const recognized = recognizeEspImage(prefix, region.length);
if (recognized === "esp8266")
return noImage("unrecognized", [ESP8266_REFUSAL]);
if (recognized !== "esp-image")
return noImage("unrecognized", [
"The region does not begin with a recognized ESP image header; it may be encrypted or hold other data.",
]);
const decoder = createEspImageDecoder(region.length);
for await (const chunk of chunks(context, region.start, region.length)) {
assertOpen();
decoder.write(chunk);
if (decoder.satisfied) break;
}
const image = decoder.finish();
if (!image.ok) return noImage("unrecognized", [image.error.message]);
return {
status: "esp-image",
image: image.value.metadata,
segments: image.value.segments.map(({ metadata }) => ({ ...metadata })),
signature_blocks: image.value.signatureBlocks.map(({ metadata }) => ({
...metadata,
})),
coverage: image.value.coverage,
limitations: [...image.value.limitations],
};
};
+19
View File
@@ -6,6 +6,11 @@ import { AsarArtifactReader } from "./AsarArtifactReader.js";
import type { ArtifactReader } from "./ArtifactReader.js";
import { DirectoryArtifactReader } from "./DirectoryArtifactReader.js";
import { EspImageArtifactReader } from "./EspImageArtifactReader.js";
import {
EspFlashImageArtifactReader,
type EspFlashAdmission,
} from "./EspFlashImageArtifactReader.js";
import type { EspFlashSelection } from "../domain/firmware/espFlashLocation.js";
import { GodotPckArtifactReader } from "./GodotPckArtifactReader.js";
import { MachOSliceArtifactReader } from "./MachOSliceArtifactReader.js";
import { NativeDmgArtifactReader } from "./NativeDmgArtifactReader.js";
@@ -29,6 +34,7 @@ export type ArtifactReaderFormat =
| "mach-o-universal"
| "dmg"
| "esp-image"
| "esp-flash-image"
| ZipPackageFormat;
interface ArtifactReaderConstruction {
@@ -39,8 +45,11 @@ interface ArtifactReaderConstruction {
| NodeSeaAdmission
| PyInstallerAdmission
| GodotPckAdmission
| EspFlashAdmission
| undefined;
readonly asarHeaderAdmission?: AsarHeaderAdmission | undefined;
/** Caller inputs for an explicitly selected ESP flash image. */
readonly espFlashSelection?: EspFlashSelection | undefined;
/** Caller-selected root spelling; only ASAR companion lookup consumes this. */
readonly selectedPath?: string | undefined;
}
@@ -118,6 +127,16 @@ export function createArtifactReader(
return new NativeDmgArtifactReader(path, options.environment);
case "esp-image":
return new EspImageArtifactReader(path, rootSource);
case "esp-flash-image":
return new EspFlashImageArtifactReader(
path,
admission !== undefined &&
"kind" in admission &&
admission.kind === "esp-flash-image"
? admission
: rootSource,
options.espFlashSelection,
);
default:
return undefined;
}
+11 -1
View File
@@ -7,6 +7,7 @@ import {
import { hashStableRootArtifactHandle } from "../inventory/hashStableRootArtifact.js";
import { absoluteCallerPath } from "../../filesystem/callerPath.js";
import type { AsarHeaderAdmission } from "../../config/asarHeaderAdmission.js";
import type { EspFlashSelection } from "../../domain/firmware/espFlashLocation.js";
import { createArtifactReader } from "../createArtifactReader.js";
import { resolveAsarHeaderAdmission } from "../AsarHeader.js";
import {
@@ -59,6 +60,8 @@ import type { ArtifactIntegrityPolicyName } from "../../domain/artifactIntegrity
/** Local extraction input with the output root chosen by the adapter. */
export interface ArtifactExtractionInput {
readonly asarHeaderAdmission?: AsarHeaderAdmission;
/** Caller inputs when the target was explicitly selected as an ESP flash image. */
readonly espFlashSelection?: EspFlashSelection | undefined;
readonly inputPath: string;
readonly expectedRootSha256?: string | undefined;
/** Selected artifact spelling; packed I/O remains on inputPath. */
@@ -99,6 +102,7 @@ const extractArtifactInScope = async (
asarHeaderAdmission:
input.asarHeaderAdmission ??
resolveAsarHeaderAdmission(input.environment),
espFlashSelection: input.espFlashSelection,
integrity: input.integrityPolicy,
},
probeFacts,
@@ -578,6 +582,7 @@ const requireExtractableFormat = async (
format !== "asar" &&
format !== "godot-pck" &&
format !== "esp-image" &&
format !== "esp-flash-image" &&
format !== "mach-o-universal" &&
!isZipPackageFormat(format)
) {
@@ -624,7 +629,11 @@ const createReader = (
path: string,
input: Pick<
ArtifactExtractionInput,
"inputPath" | "selectedPath" | "environment" | "asarHeaderAdmission"
| "inputPath"
| "selectedPath"
| "environment"
| "asarHeaderAdmission"
| "espFlashSelection"
>,
containerFormat: ArtifactGraphManifest["root_format"],
transferredSource?: ActiveArtifactInventorySource,
@@ -640,6 +649,7 @@ const createReader = (
input.selectedPath ?? input.inputPath,
),
admission: transferredSource?.admission,
espFlashSelection: input.espFlashSelection,
asarHeaderAdmission:
containerFormat === "asar"
? (input.asarHeaderAdmission ??
@@ -284,6 +284,7 @@ export const rootOccurrenceFor = (
"pyinstaller",
"node-sea",
"esp-image",
"esp-flash-image",
].includes(metadata.format)
? "container"
: artifactRoleForFormat(metadata.path, metadata.format).kind,
@@ -574,7 +575,11 @@ const artifactRoleForFormat = (
format: zipPackageFormatForPath(path) ?? format,
};
if (format === "plist") return { kind: "plist", format };
if (format === "godot-pck" || format === "esp-image")
if (
format === "godot-pck" ||
format === "esp-image" ||
format === "esp-flash-image"
)
return { kind: "container", format };
return {
kind: ["native-addon", "dynamic-library"].includes(byPath.kind)
+30 -1
View File
@@ -1,4 +1,9 @@
import type { GodotPckAdmission } from "../GodotPckLocation.js";
import type { EspFlashAdmission } from "../EspFlashImageArtifactReader.js";
import {
locateEspFlashLayout,
type EspFlashSelection,
} from "../../domain/firmware/espFlashLocation.js";
import type { PyInstallerAdmission } from "../pyinstaller/PyInstallerAdmission.js";
import { assertArtifactSelectedSource } from "../ArtifactSelectedSource.js";
import { artifactNameForClassification } from "../../domain/artifactNameSelection.js";
@@ -55,6 +60,7 @@ type RootInventoryClassification = RootClassification & {
| NodeSeaAdmission
| PyInstallerAdmission
| GodotPckAdmission
| EspFlashAdmission
| undefined;
};
@@ -75,6 +81,7 @@ export const classifyAndHashRootForInventory = async (
| "asarHeaderAdmission"
| "selectedPath"
| "expectedRootSha256"
| "espFlashSelection"
>,
probeFacts?: ArtifactEmbeddedFormatProbeFacts,
): Promise<RootInventoryClassification> => {
@@ -150,6 +157,7 @@ export const classifyAndHashRootForInventory = async (
options.signal,
options.asarHeaderAdmission,
options.selectedPath,
options.espFlashSelection,
);
const format =
typeof classified === "string" ? classified : classified.kind;
@@ -167,7 +175,8 @@ export const classifyAndHashRootForInventory = async (
format === "godot-pck" ||
format === "pyinstaller" ||
format === "node-sea" ||
format === "esp-image";
format === "esp-image" ||
format === "esp-flash-image";
outcome = {
kind: "completed",
value: {
@@ -207,11 +216,13 @@ const classifyRootFormat = async (
signal?: AbortSignal,
asarHeaderAdmission?: AsarHeaderAdmission,
selectedPath?: string,
espFlashSelection?: EspFlashSelection,
): Promise<
| ArtifactOccurrence["artifact_format"]
| NodeSeaAdmission
| PyInstallerAdmission
| GodotPckAdmission
| EspFlashAdmission
> => {
const { handle } = source;
const probe = createArtifactClassificationProbe(asarHeaderAdmission);
@@ -267,6 +278,24 @@ const classifyRootFormat = async (
return "dmg";
const pck = await embeddedProbes.godotPck(evidence.prefix);
if (pck !== undefined) return { kind: "godot-pck", source, location: pck };
// A C3 or S3 dump begins with its bootloader, so the layout is located
// before the bytes are read as one ESP image.
const flash = await locateEspFlashLayout(
archiveBytes,
read,
espFlashSelection ?? {},
{
explicit: espFlashSelection !== undefined,
...(signal === undefined ? {} : { signal }),
},
);
if (flash.ok)
return { kind: "esp-flash-image", source, location: flash.value };
if (espFlashSelection !== undefined)
throw new ArtifactReaderFailure(
"unavailable",
`ESP flash image layout is not recognized: ${flash.error}`,
);
const fatEvidence = await readRootFatSliceHeaders(handle, evidence, signal);
const mzWindowsHeaderSignature = await readRootMzWindowsHeaderSignature(
handle,
+10 -2
View File
@@ -1,4 +1,5 @@
import type { GodotPckAdmission } from "../GodotPckLocation.js";
import type { EspFlashAdmission } from "../EspFlashImageArtifactReader.js";
import type { PyInstallerAdmission } from "../pyinstaller/PyInstallerAdmission.js";
import type { NodeSeaAdmission } from "../nodeSea/NodeSeaAdmission.js";
import type { AsarHeaderAdmission } from "../../config/asarHeaderAdmission.js";
@@ -25,7 +26,8 @@ export const createReader = (
| StableRegularFileDescriptor
| NodeSeaAdmission
| PyInstallerAdmission
| GodotPckAdmission,
| GodotPckAdmission
| EspFlashAdmission,
asarHeaderAdmission?: AsarHeaderAdmission,
selectedPath?: string,
): ArtifactReader | undefined => {
@@ -69,6 +71,11 @@ export const inventoryCoverage = (
"Node SEA inventory establishes the injected blob, flags, fuse sentinel, main script, optional code cache, snapshot, and assets. V8 snapshots and code cache are not deserialized, and the application is not executed.",
]
: []),
...(reader.format === "esp-flash-image"
? [
"ESP flash layout inventory reads a dump's bootloader region, partition table and partitions and decodes the bootloader and app partitions as ESP images. The OTA selection is derived without app rollback or anti-rollback, and the running partition is unknown. Partitions, nvs_keys included, are extracted as ordinary data; nothing is decrypted.",
]
: []),
...(reader.format === "esp-image"
? [
"ESP image inventory reads the header-declared segments, checksum, appended SHA-256, descriptor and Secure Boot v2 signature blocks. Segment classification follows ESP-IDF v6.1 should_map and should_load for a cold boot and does not check RAM ranges; signatures are not cryptographically verified, and code is not disassembled.",
@@ -98,7 +105,8 @@ export const inventoryCoverage = (
format === "asar" ||
format === "godot-pck" ||
format === "node-sea" ||
format === "esp-image"
format === "esp-image" ||
format === "esp-flash-image"
)
return {
coverage: { scope: "container-entries", status: "unknown" },
+6
View File
@@ -1,3 +1,4 @@
import type { EspFlashSelection } from "../../domain/firmware/espFlashLocation.js";
import type { ArtifactIntegrityPolicyName } from "../../domain/artifactIntegrityPolicy.js";
import type { AsarHeaderAdmission } from "../../config/asarHeaderAdmission.js";
import type { ArtifactResourceScope } from "../ArtifactResourceScope.js";
@@ -14,4 +15,9 @@ export interface ArtifactInventoryOptions {
readonly environment?: Readonly<NodeJS.ProcessEnv> | undefined;
/** ASAR header/index admission resolved from the owner's environment; default when absent. */
readonly asarHeaderAdmission?: AsarHeaderAdmission | undefined;
/**
* Caller inputs when the target was explicitly selected as an ESP flash
* image; detection with defaults applies otherwise.
*/
readonly espFlashSelection?: EspFlashSelection | undefined;
}
+57
View File
@@ -1,3 +1,8 @@
import {
espFlashAddressSchema,
parseEspFlashAddress,
} from "../domain/firmware/espFlashSelection.js";
import type { ArtifactTargetSelection } from "../application/DirectAnalysis.js";
import { projectCliAnalysisResult } from "./projectCliAnalysisResult.js";
import {
analysisEvidenceDeliveryOptions,
@@ -194,6 +199,51 @@ const registerInterfaceBuilderCommand = (
});
};
/** Options that read a dump as an ESP flash image with caller inputs. */
const espFlashSelectionOptions = {
targetFormat: z
.enum(["esp-flash-image"])
.optional()
.describe(
"Read the target as an ESP flash dump; required with --flash-offset or --partition-table-offset",
),
flashOffset: espFlashAddressSchema
.optional()
.describe("Flash address (0xhex) of the dump's first byte"),
partitionTableOffset: espFlashAddressSchema
.optional()
.describe(
"Flash address (0xhex, a multiple of 0x1000) of the partition table; disables the table scan",
),
};
const espFlashTargetSelection = (options: {
readonly targetFormat?: "esp-flash-image" | undefined;
readonly flashOffset?: string | undefined;
readonly partitionTableOffset?: string | undefined;
}): ArtifactTargetSelection => ({
...(options.targetFormat === undefined
? {}
: { formatHint: options.targetFormat }),
...(options.flashOffset === undefined &&
options.partitionTableOffset === undefined
? {}
: {
espFlash: {
...(options.flashOffset === undefined
? {}
: { flashOffset: parseEspFlashAddress(options.flashOffset) }),
...(options.partitionTableOffset === undefined
? {}
: {
partitionTableOffset: parseEspFlashAddress(
options.partitionTableOffset,
),
}),
},
}),
});
const registerExtractionCommand = (
cli: CliInstance,
logger: Logger,
@@ -210,6 +260,7 @@ const registerExtractionCommand = (
.enum(["fail", "record-and-continue"])
.default("fail")
.describe("Behavior when declared artifact integrity does not match"),
...espFlashSelectionOptions,
}),
run: ({ args, options }) =>
logCliCommand(logger, "extract-artifact", () =>
@@ -221,6 +272,9 @@ const registerExtractionCommand = (
output_root: createArtifactExtractionDestination(),
},
logger,
undefined,
undefined,
espFlashTargetSelection(options),
).then(projectCliAnalysisResult),
),
});
@@ -243,6 +297,7 @@ const registerInspectionCommand = (
.enum(["fail", "record-and-continue"])
.default("fail")
.describe("Behavior when declared artifact integrity does not match"),
...espFlashSelectionOptions,
}),
run: ({ args, options }) =>
logCliCommand(logger, "inspect-artifact", () =>
@@ -262,6 +317,8 @@ const registerInspectionCommand = (
{ integrity_policy: options.integrityPolicy },
logger,
signal,
undefined,
espFlashTargetSelection(options),
);
return publishCliAnalysisEvidence(complete, delivery);
}),
+37 -1
View File
@@ -6,6 +6,11 @@ import {
rawBinarySelectionError,
} from "../domain/rawBinary.js";
import { z } from "zod";
import {
espFlashAddressSchema,
espFlashSelectionError,
parseEspFlashAddress,
} from "../domain/firmware/espFlashSelection.js";
import { analysisProviderSelectorSchema } from "../domain/providerSelection.js";
/** Caller-supplied analysis snapshot path; must name its destination absolutely. */
@@ -25,7 +30,7 @@ export const openBinaryInputSchema = z
format: executableFormatHintSchema
.optional()
.describe(
"Explicit interpretation: dos-com or raw-binary (headerless), or gamecube-dol (GameCube/Wii DOL, never auto-detected). Omission preserves header-based detection.",
"Explicit interpretation: dos-com or raw-binary (headerless), gamecube-dol (GameCube/Wii DOL, never auto-detected), or esp-flash-image (an ESP flash dump read with flash_offset and partition_table_offset). Omission preserves header-based detection.",
),
architecture: binaryArchitectureSchema
.optional()
@@ -37,10 +42,41 @@ export const openBinaryInputSchema = z
.describe(
"Caller-supplied load and entry address (0xhex). File offset 0 is mapped here. Required with format=raw-binary; rejected otherwise. Optional with format=gamecube-dol; must equal the DOL header entry.",
),
flash_offset: espFlashAddressSchema
.optional()
.describe(
"Flash address (0xhex) of the dump's first byte, when it does not start at flash address 0. Only with format=esp-flash-image.",
),
partition_table_offset: espFlashAddressSchema
.optional()
.describe(
"Flash address (0xhex, a multiple of 0x1000) of the partition table, which disables the table scan. Only with format=esp-flash-image.",
),
provider_id: analysisProviderSelectorSchema.optional(),
snapshot_path: snapshotPathSchema.optional(),
})
.superRefine((value, context) => {
const flash = espFlashSelectionError({
format: value.format,
flashOffset:
value.flash_offset === undefined
? undefined
: parseEspFlashAddress(value.flash_offset),
partitionTableOffset:
value.partition_table_offset === undefined
? undefined
: parseEspFlashAddress(value.partition_table_offset),
});
if (flash !== undefined)
context.addIssue({
code: "custom",
path: [
value.partition_table_offset === undefined
? "flash_offset"
: "partition_table_offset",
],
message: flash,
});
const pairing = rawBinarySelectionError({
...(value.format === undefined ? {} : { format: value.format }),
...(value.architecture === undefined
+34 -8
View File
@@ -77,6 +77,7 @@ const formatSchema = z.enum([
"dmg",
"pkg",
"esp-image",
"esp-flash-image",
"plist",
"javascript",
"source-map",
@@ -96,7 +97,10 @@ const targetSchema = z.discriminatedUnion("format", [
targetIdentitySchema.extend({
format: z.enum(
formatSchema.options.filter(
(format) => format !== "raw-binary" && format !== "gamecube-dol",
(format) =>
format !== "raw-binary" &&
format !== "gamecube-dol" &&
format !== "esp-flash-image",
),
),
architecture: architectureSchema
@@ -115,6 +119,14 @@ const targetSchema = z.discriminatedUnion("format", [
architecture: z.literal("powerpc"),
entry_point: rawBinaryEntryPointSchema,
}),
// The caller's inputs select how the dump is read, so they are identity.
targetIdentitySchema.extend({
format: z.literal("esp-flash-image"),
kind: z.literal("archive"),
architecture: z.null(),
flash_offset: z.number().int().min(0).max(0xffff_ffff).nullable(),
partition_table_offset: z.number().int().min(0).max(0xffff_ffff).nullable(),
}),
]);
const bindingSchema = z
.object({
@@ -383,12 +395,21 @@ export const snapshotTarget = (target: BinaryTarget): AnalysisSnapshotTarget =>
format: target.format,
entry_point: target.entryPoint,
}
: {
sha256: target.sha256,
kind: target.kind,
architecture: target.architecture ?? null,
format: target.format,
};
: target.format === "esp-flash-image"
? {
sha256: target.sha256,
kind: target.kind,
architecture: null,
format: target.format,
flash_offset: target.espFlash.flashOffset,
partition_table_offset: target.espFlash.partitionTableOffset,
}
: {
sha256: target.sha256,
kind: target.kind,
architecture: target.architecture ?? null,
format: target.format,
};
/** Build the immutable provider/profile binding used by a cached snapshot. */
export const snapshotBinding = (
@@ -413,7 +434,12 @@ export const snapshotMatchesTarget = (
snapshot.entry_point === target.entryPoint)) &&
(snapshot.format !== "gamecube-dol" ||
(target.format === "gamecube-dol" &&
snapshot.entry_point === target.entryPoint));
snapshot.entry_point === target.entryPoint)) &&
(snapshot.format !== "esp-flash-image" ||
(target.format === "esp-flash-image" &&
snapshot.flash_offset === target.espFlash.flashOffset &&
snapshot.partition_table_offset ===
target.espFlash.partitionTableOffset));
/** Compare a snapshot binding with the selected concrete analysis profile. */
export const snapshotMatchesProfile = (
@@ -18,7 +18,8 @@ export const artifactInventorySummaryProjectionIds = (
format === "godot-pck" ||
format === "pyinstaller" ||
format === "node-sea" ||
format === "esp-image"
format === "esp-image" ||
format === "esp-flash-image"
)
return [current.evidence_id];
const { view_digest: _digest, ...projection } = result;
@@ -267,7 +267,8 @@ export const projectArtifactInventoryView = (
inventory.manifest.root_format === "godot-pck" ||
inventory.manifest.root_format === "pyinstaller" ||
inventory.manifest.root_format === "node-sea" ||
inventory.manifest.root_format === "esp-image";
inventory.manifest.root_format === "esp-image" ||
inventory.manifest.root_format === "esp-flash-image";
const counts = inventoryCounts(source);
const limitations = uniqueSorted([
...parent.limitations,
+13 -1
View File
@@ -15,6 +15,7 @@ import {
type InventorySet,
} from "./artifactInventoryEvidence.js";
import {
espFlashPartitionDeclaration,
espImageSegmentDeclaration,
nodeSeaEntryDeclaration,
pyInstallerEntryDeclaration,
@@ -448,6 +449,7 @@ interface OccurrenceMetadata {
readonly asarPacking: MetadataComparison;
readonly nodeSea: MetadataComparison;
readonly espImageSegment: MetadataComparison;
readonly espFlashPartition: MetadataComparison;
readonly pyInstaller: MetadataComparison;
}
@@ -455,7 +457,8 @@ interface OccurrenceMetadata {
const declarationChanged = (metadata: OccurrenceMetadata): boolean =>
metadata.asarPacking === "changed" ||
metadata.nodeSea === "changed" ||
metadata.espImageSegment === "changed";
metadata.espImageSegment === "changed" ||
metadata.espFlashPartition === "changed";
const compareOptionalMetadata = <T>(
left: T | undefined,
@@ -497,6 +500,15 @@ const occurrenceMetadata = (input: OccurrencePair): OccurrenceMetadata => {
: undefined,
espImageSegmentDeclaration,
),
espFlashPartition: compareOptionalMetadata(
left?.format_metadata?.kind === "esp-flash-partition"
? left.format_metadata
: undefined,
right?.format_metadata?.kind === "esp-flash-partition"
? right.format_metadata
: undefined,
espFlashPartitionDeclaration,
),
pyInstaller: compareOptionalMetadata(
isPyInstallerEntry(left) ? left.format_metadata : undefined,
isPyInstallerEntry(right) ? right.format_metadata : undefined,
+143 -7
View File
@@ -132,6 +132,13 @@ const uint16 = z.number().int().min(0).max(0xffff);
const sha256Hex = z.string().regex(/^[0-9a-f]{64}$/u);
const byteRange = z.strictObject({ offset, length: offset });
/** Derived from `chip_id` through the cited table. */
const espChipFactsSchema = z.strictObject({
target: z.string(),
isa: z.enum(["xtensa", "riscv"]),
source: z.string(),
});
/** `esp_app_desc_t` fields, as ESP-IDF names them; strings stop at NUL. */
const espAppDescriptorSchema = z.strictObject({
kind: z.literal("app"),
@@ -183,13 +190,7 @@ export const espImageMetadataSchema = z.strictObject({
hash_appended: z.boolean(),
}),
/** Derived from `chip_id` through the cited table; null for unknown IDs. */
chip: z
.strictObject({
target: z.string(),
isa: z.enum(["xtensa", "riscv"]),
source: z.string(),
})
.nullable(),
chip: espChipFactsSchema.nullable(),
checksum: z.strictObject({
offset: offset.nullable(),
stored: uint8.nullable(),
@@ -266,6 +267,139 @@ export type EspImageSignatureBlockMetadata = z.infer<
typeof espImageSignatureBlockMetadataSchema
>;
/** One partition table entry as declared, with ESP-IDF's type names. */
export const espFlashPartitionMetadataSchema = z.strictObject({
kind: z.literal("esp-flash-partition"),
/** Position among the table's partition entries. */
index: uint8,
/** The label up to its first NUL; null when it is not UTF-8. */
label: z.string().nullable(),
type: uint8,
subtype: uint8,
/** `gen_esp32part.py` names; null for values it does not name. */
type_name: z.string().nullable(),
subtype_name: z.string().nullable(),
flash_address: uint32,
size: uint32,
encrypted: z.boolean(),
readonly: z.boolean(),
/** Flag bits other than `encrypted` and `readonly`, which ESP-IDF ignores. */
unknown_flags: uint32,
});
export type EspFlashPartitionMetadata = z.infer<
typeof espFlashPartitionMetadataSchema
>;
/** What a partition entry declares; its index is layout. */
export const espFlashPartitionDeclaration = (
metadata: EspFlashPartitionMetadata,
) => ({
label: metadata.label,
type: metadata.type,
subtype: metadata.subtype,
flash_address: metadata.flash_address,
size: metadata.size,
encrypted: metadata.encrypted,
readonly: metadata.readonly,
unknown_flags: metadata.unknown_flags,
});
/** ESP image facts for a region of a dump; offsets are relative to the region. */
const espFlashImageFactsSchema = z.strictObject({
/** An ESP image, erased flash, or bytes that are not one, such as ciphertext. */
status: z.enum(["esp-image", "erased", "unrecognized"]),
image: espImageMetadataSchema.nullable(),
segments: z.array(espImageSegmentMetadataSchema),
signature_blocks: z.array(espImageSignatureBlockMetadataSchema),
coverage: z.enum(["complete", "unknown"]).nullable(),
limitations: z.array(z.string()),
});
const espOtaRecordSchema = z.strictObject({
index: z.union([z.literal(0), z.literal(1)]),
ota_seq: uint32,
seq_label: z.string().regex(/^[0-9a-f]{40}$/u),
ota_state: uint32,
state_name: z
.enum(["new", "pending-verify", "valid", "invalid", "aborted", "undefined"])
.nullable(),
crc: uint32,
/** CRC-32 of `ota_seq` seeded with 0xFFFFFFFF, as the bootloader checks it. */
crc_matches: z.boolean(),
valid: z.boolean(),
});
/**
* An ESP flash dump: where it sits in flash, its bootloader, partition table,
* partitions and OTA data. Entries are the table's partitions; app and
* bootloader regions carry their ESP image facts here.
*/
export const espFlashImageMetadataSchema = z.strictObject({
kind: z.literal("esp-flash-image"),
/** Flash address of the file's first byte: 0 unless the caller gave one. */
flash_offset: z.strictObject({
value: uint32,
source: z.enum(["default", "caller"]),
}),
/** From the bootloader's `chip_id`; null without a located bootloader. */
chip: espChipFactsSchema.nullable(),
/** The image at its chip's ROM offset, through the partition table. */
bootloader: z
.strictObject({
flash_address: uint32,
region_size: offset,
facts: espFlashImageFactsSchema,
})
.nullable(),
partition_table: z.strictObject({
flash_address: uint32,
/** 0x8000, the caller's offset, or a scan for a verified MD5 (derived). */
method: z.enum(["default", "caller", "scan"]),
partition_count: uint8,
md5: z.strictObject({
status: z.enum(["valid", "invalid", "absent"]),
stored: z
.string()
.regex(/^[0-9a-f]{32}$/u)
.nullable(),
computed: z
.string()
.regex(/^[0-9a-f]{32}$/u)
.nullable(),
}),
}),
/** Declared partitions with no bytes in the dump, which have no entry. */
partitions_outside_dump: z.array(espFlashPartitionMetadataSchema),
/** ESP image facts for each app partition the dump holds. */
apps: z.array(
z.strictObject({ partition_index: uint8, facts: espFlashImageFactsSchema }),
),
otadata: z
.strictObject({
partition_index: uint8,
records: z.array(espOtaRecordSchema),
/** Derived from the records under the stated assumptions. */
selected: z.strictObject({
active_record: z.union([z.literal(0), z.literal(1)]).nullable(),
slot: uint8.nullable(),
partition_index: uint8.nullable(),
basis: z.enum([
"otadata",
"factory-fallback",
"ota_0-fallback",
"none",
]),
assumptions: z.array(z.string()),
}),
/** Rollback, anti-rollback and load fallback decide it at boot. */
running_partition: z.literal("unknown"),
})
.nullable(),
/** Bytes outside every described region that are not erased flash. */
undescribed: z.array(byteRange),
});
export type EspFlashImageMetadata = z.infer<typeof espFlashImageMetadataSchema>;
/** Declaration identity that is stable across archive layout changes. */
export const pyInstallerEntryDeclaration = (
metadata: PyInstallerEntryMetadata,
@@ -286,6 +420,8 @@ export const artifactFormatMetadataSchema = z.discriminatedUnion("kind", [
espImageMetadataSchema,
espImageSegmentMetadataSchema,
espImageSignatureBlockMetadataSchema,
espFlashImageMetadataSchema,
espFlashPartitionMetadataSchema,
]);
export type ArtifactFormatMetadata = z.infer<
typeof artifactFormatMetadataSchema
+1
View File
@@ -87,6 +87,7 @@ const artifactFormatSchema = z.enum([
"pyinstaller",
"node-sea",
"esp-image",
"esp-flash-image",
"unknown",
]);
+17
View File
@@ -262,12 +262,29 @@ export type BinaryTarget =
| "pkg"
| "esp-image";
})
| (BinaryTargetIdentity &
NonExecutableMetadata & {
readonly kind: "archive";
readonly format: "esp-flash-image";
/** How the dump is read; null inputs take their defaults. */
readonly espFlash: EspFlashTargetSelection;
})
| (BinaryTargetIdentity &
NonExecutableMetadata & {
readonly kind: "artifact";
readonly format: "plist" | "javascript" | "source-map";
});
/** Caller inputs that select how an ESP flash dump is read. */
export interface EspFlashTargetSelection {
/** The caller selected the format, so the table scan runs without a bootloader. */
readonly selected: boolean;
/** Flash address of the dump's first byte. */
readonly flashOffset: number | null;
/** Flash address of the partition table. */
readonly partitionTableOffset: number | null;
}
/** Whether an executable target contains a thin or universal Mach-O image. */
export const isMachOFormat = (
format: unknown,
+5 -1
View File
@@ -9,8 +9,12 @@ export const executableFormatHintSchema = z.enum([
"dos-com",
"raw-binary",
"gamecube-dol",
"esp-flash-image",
]);
/** Explicit executable interpretation for formats with no identifying header. */
/**
* Explicit interpretation for formats with no identifying header, or, for
* esp-flash-image, a dump read with caller inputs.
*/
export type ExecutableFormatHint = z.infer<typeof executableFormatHintSchema>;
/** Validate the single 64 KiB COM segment, excluding its 256-byte PSP prefix. */
+1
View File
@@ -39,6 +39,7 @@ export const evidenceSubjectFormatSchema = z.enum([
"pyinstaller",
"node-sea",
"esp-image",
"esp-flash-image",
]);
/** Format-owned architecture declarations shared by Evidence and persisted executions. */
+22
View File
@@ -9,6 +9,8 @@
* `components/soc/<target>/include/soc/soc.h`;
* - page sizes: `SOC_MMU_PAGE_SIZE_8KB_SUPPORTED` in `soc_caps.h` and
* `components/soc/<target>/Kconfig.mmu`;
* - bootloader flash offset: `ESP_ROM_BOOTLOADER_OFFSET_FLASH` in
* `components/esp_rom/<target>/esp_rom_caps.h`, which the ROM fixes;
* - segment length limit: `SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP` in
* `soc_caps.h`, which sets `ESP_IMAGE_MAX_FLASH_ADDR_SIZE` in
* `components/bootloader_support/src/esp_image_format.c`.
@@ -31,6 +33,8 @@ export interface EspChip {
/** ESP-IDF `IDF_TARGET` name, as `esptool --chip` also spells it. */
readonly target: string;
readonly isa: EspChipIsa;
/** Where the ROM loads the second-stage bootloader from flash. */
readonly romBootloaderOffset: number;
/** `SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP`, which lifts the 16 MiB segment limit. */
readonly cache32BitAddressMap: boolean;
/** `should_map` ranges with fixed bounds. */
@@ -56,6 +60,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0000,
target: "esp32",
isa: "xtensa",
romBootloaderOffset: 0x1000,
cache32BitAddressMap: false,
mapped: [range(0x400d_0000, 0x4040_0000), range(0x3f40_0000, 0x3f80_0000)],
},
@@ -63,6 +68,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0002,
target: "esp32s2",
isa: "xtensa",
romBootloaderOffset: 0x1000,
cache32BitAddressMap: false,
mapped: [range(0x4008_0000, 0x4080_0000), range(0x3f00_0000, 0x3ff8_0000)],
},
@@ -70,6 +76,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0005,
target: "esp32c3",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: false,
mapped: [range(0x4200_0000, 0x4280_0000), range(0x3c00_0000, 0x3c80_0000)],
},
@@ -77,6 +84,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0009,
target: "esp32s3",
isa: "xtensa",
romBootloaderOffset: 0x0,
cache32BitAddressMap: true,
mapped: [range(0x4200_0000, 0x4400_0000), range(0x3c00_0000, 0x3e00_0000)],
},
@@ -84,6 +92,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x000c,
target: "esp32c2",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: false,
mapped: [range(0x4200_0000, 0x4240_0000), range(0x3c00_0000, 0x3c40_0000)],
},
@@ -91,6 +100,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x000d,
target: "esp32c6",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: false,
mapped: [],
pageSizedMapping: { low: 0x4200_0000, pageSizes: C6_H2_PAGE_SIZES },
@@ -99,6 +109,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0010,
target: "esp32h2",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: false,
mapped: [],
pageSizedMapping: { low: 0x4200_0000, pageSizes: C6_H2_PAGE_SIZES },
@@ -107,6 +118,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0012,
target: "esp32p4",
isa: "riscv",
romBootloaderOffset: 0x2000,
cache32BitAddressMap: true,
mapped: [range(0x4000_0000, 0x4400_0000), range(0x4800_0000, 0x4c00_0000)],
},
@@ -114,6 +126,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0014,
target: "esp32c61",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: true,
mapped: [range(0x4200_0000, 0x4400_0000)],
},
@@ -121,6 +134,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0017,
target: "esp32c5",
isa: "riscv",
romBootloaderOffset: 0x2000,
cache32BitAddressMap: true,
mapped: [range(0x4200_0000, 0x4400_0000)],
},
@@ -128,6 +142,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0019,
target: "esp32h21",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: false,
mapped: [range(0x4200_0000, 0x4300_0000)],
},
@@ -135,6 +150,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x001c,
target: "esp32h4",
isa: "riscv",
romBootloaderOffset: 0x2000,
cache32BitAddressMap: true,
mapped: [range(0x4200_0000, 0x4400_0000)],
},
@@ -142,6 +158,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0020,
target: "esp32s31",
isa: "riscv",
romBootloaderOffset: 0x2000,
cache32BitAddressMap: true,
mapped: [range(0x4000_0000, 0x4400_0000), range(0x5000_0000, 0x5400_0000)],
},
@@ -152,6 +169,11 @@ const BY_ID = new Map(CHIPS.map((chip) => [chip.chipId, chip]));
export const espChipForId = (chipId: number): EspChip | undefined =>
BY_ID.get(chipId);
/** Every flash offset a ROM loads a bootloader from, lowest first. */
export const ESP_ROM_BOOTLOADER_OFFSETS: readonly number[] = [
...new Set(CHIPS.map(({ romBootloaderOffset }) => romBootloaderOffset)),
].sort((left, right) => left - right);
const SIXTEEN_MIB = 0x100_0000;
/**
+140
View File
@@ -0,0 +1,140 @@
import { createHash } from "node:crypto";
import { crc32 } from "node:zlib";
/** One `esp_partition_info_t` as `gen_esp32part.py` writes it. */
export interface EspPartitionFixture {
readonly label: string;
readonly type: number;
readonly subtype: number;
readonly offset: number;
readonly size: number;
readonly flags?: number;
}
export const partitionEntry = (part: EspPartitionFixture): Buffer => {
const entry = Buffer.alloc(32);
entry.writeUInt16LE(0x50aa, 0);
entry.writeUInt8(part.type, 2);
entry.writeUInt8(part.subtype, 3);
entry.writeUInt32LE(part.offset, 4);
entry.writeUInt32LE(part.size, 8);
entry.write(part.label, 12, 16, "utf8");
entry.writeUInt32LE(part.flags ?? 0, 28);
return entry;
};
/** The entries, an MD5 entry unless omitted, the terminator and 0xFF fill. */
export const partitionTable = (
parts: readonly EspPartitionFixture[],
{ md5 = "valid" }: { readonly md5?: "valid" | "corrupt" | "absent" } = {},
): Buffer => {
const entries = Buffer.concat(parts.map(partitionEntry));
const digest = createHash("md5").update(entries).digest();
if (md5 === "corrupt") digest.writeUInt8(digest.readUInt8(0) ^ 0xff, 0);
const md5Entry =
md5 === "absent"
? Buffer.alloc(0)
: Buffer.concat([
Buffer.from([0xeb, 0xeb]),
Buffer.alloc(14, 0xff),
digest,
]);
const table = Buffer.concat([entries, md5Entry]);
return Buffer.concat([table, Buffer.alloc(0xc00 - table.length, 0xff)]);
};
/** The layout `idf.py` builds for two OTA apps, from `partitions_two_ota.csv`. */
export const TWO_OTA_PARTITIONS: readonly EspPartitionFixture[] = [
{ label: "nvs", type: 0x01, subtype: 0x02, offset: 0x9000, size: 0x4000 },
{ label: "otadata", type: 0x01, subtype: 0x00, offset: 0xd000, size: 0x2000 },
{
label: "phy_init",
type: 0x01,
subtype: 0x01,
offset: 0xf000,
size: 0x1000,
},
{
label: "factory",
type: 0x00,
subtype: 0x00,
offset: 0x1_0000,
size: 0x1_0000,
},
{
label: "ota_0",
type: 0x00,
subtype: 0x10,
offset: 0x2_0000,
size: 0x1_0000,
},
{
label: "ota_1",
type: 0x00,
subtype: 0x11,
offset: 0x3_0000,
size: 0x1_0000,
},
];
/** One `esp_ota_select_entry_t`; the CRC is computed unless given. */
export const otaRecord = ({
seq,
state = 0xffff_ffff,
crc,
}: {
readonly seq: number;
readonly state?: number;
readonly crc?: number;
}): Buffer => {
const record = Buffer.alloc(32, 0xff);
record.writeUInt32LE(seq >>> 0, 0);
record.writeUInt32LE(state >>> 0, 24);
const computed = crc32(record.subarray(0, 4), 0xffff_ffff) >>> 0;
record.writeUInt32LE((crc ?? computed) >>> 0, 28);
return record;
};
/** Two 4 KiB sectors with one record at the start of each; omitted is erased. */
export const otaData = (first?: Buffer, second?: Buffer): Buffer => {
const data = Buffer.alloc(0x2000, 0xff);
first?.copy(data, 0);
second?.copy(data, 0x1000);
return data;
};
/** A flash dump of `size` bytes of erased flash with the given contents. */
export const flashImage = (
size: number,
contents: readonly { readonly at: number; readonly bytes: Uint8Array }[],
): Buffer => {
const image = Buffer.alloc(size, 0xff);
for (const { at, bytes } of contents) Buffer.from(bytes).copy(image, at);
return image;
};
/** Where a synthetic dump places each part, by flash address. */
export interface EspFlashDumpOptions {
readonly size?: number;
readonly bootloader?: { readonly at: number; readonly bytes: Uint8Array };
readonly tableAt?: number;
readonly table?: Uint8Array;
readonly parts?: readonly {
readonly at: number;
readonly bytes: Uint8Array;
}[];
}
/** A whole-flash dump: erased flash with a bootloader, a table and contents. */
export const espFlashDump = ({
size = 0x4_0000,
bootloader,
tableAt = 0x8000,
table = partitionTable(TWO_OTA_PARTITIONS),
parts = [],
}: EspFlashDumpOptions = {}): Buffer =>
flashImage(size, [
...(bootloader === undefined ? [] : [bootloader]),
{ at: tableAt, bytes: table },
...parts,
]);
+74
View File
@@ -0,0 +1,74 @@
import {
ESP_ROM_BOOTLOADER_OFFSETS,
espChipForId,
type EspChip,
} from "./espChips.js";
import { recognizeEspImage } from "./espImage.js";
import { offsetText, u16 } from "./espImageStructures.js";
/** Bytes that recognition needs at a candidate bootloader offset. */
export const ESP_BOOTLOADER_PROBE_BYTES = 32;
/** Where the dump holds bytes for each ROM bootloader offset. */
export interface EspBootloaderProbe {
/** Flash address the ROM loads from. */
readonly flashAddress: number;
/** The same address in the file, given the dump's flash offset. */
readonly fileOffset: number;
/** Up to `ESP_BOOTLOADER_PROBE_BYTES` from that file offset. */
readonly prefix: Uint8Array;
/** Bytes from that file offset to the end of the file. */
readonly available: number;
}
export interface EspBootloaderLocation {
readonly flashAddress: number;
readonly fileOffset: number;
readonly chip: EspChip;
}
/**
* The bootloader is the ESP image at the flash address its chip's ROM loads
* from (`ESP_ROM_BOOTLOADER_OFFSET_FLASH`). An image at another chip's ROM
* offset is reported, not located: the dump may not start where its flash
* offset says.
*/
export const locateEspBootloader = (
probes: readonly EspBootloaderProbe[],
): {
readonly location: EspBootloaderLocation | undefined;
readonly limitations: readonly string[];
} => {
const limitations: string[] = [];
for (const probe of probes) {
if (recognizeEspImage(probe.prefix, probe.available) !== "esp-image")
continue;
const chipId = u16(probe.prefix, 12);
const chip = espChipForId(chipId);
if (chip?.romBootloaderOffset === probe.flashAddress)
return {
location: {
flashAddress: probe.flashAddress,
fileOffset: probe.fileOffset,
chip,
},
limitations: [],
};
limitations.push(
chip === undefined
? `An ESP image at flash ${offsetText(probe.flashAddress)} has chip_id 0x${chipId.toString(16).padStart(4, "0")}, which is not in the chip table; it is not read as the bootloader.`
: `An ${chip.target} image is at flash ${offsetText(probe.flashAddress)}, but its ROM loads the bootloader from ${offsetText(chip.romBootloaderOffset)}; the dump may not start at the given flash offset.`,
);
}
return { location: undefined, limitations };
};
/** The file offsets that hold each ROM bootloader offset, given the dump's origin. */
export const espBootloaderProbeOffsets = (
flashOffset: number,
fileSize: number,
): readonly { readonly flashAddress: number; readonly fileOffset: number }[] =>
ESP_ROM_BOOTLOADER_OFFSETS.map((flashAddress) => ({
flashAddress,
fileOffset: flashAddress - flashOffset,
})).filter(({ fileOffset }) => fileOffset >= 0 && fileOffset < fileSize);
+279
View File
@@ -0,0 +1,279 @@
import { describe, expect, it } from "vitest";
import {
decodeEspPartitionTable,
espPartitionSubtypeName,
espPartitionTypeName,
type EspPartition,
} from "./espPartitionTable.js";
import { decodeEspOtaRecord, selectEspOtaSlot } from "./espOtaData.js";
import {
espBootloaderProbeOffsets,
locateEspBootloader,
} from "./espFlashBootloader.js";
import {
TWO_OTA_PARTITIONS,
otaRecord,
partitionEntry,
partitionTable,
} from "./espFlash.fixture.js";
import { ESP32C3_BOOTLOADER_SEGMENTS, espImage } from "./espImage.fixture.js";
const table = (bytes: Uint8Array) => {
const decoded = decodeEspPartitionTable(bytes);
if (!decoded.ok) throw new Error(decoded.error);
return decoded.value;
};
describe("ESP partition tables", () => {
it("decodes entries, flags and a verified MD5 as gen_esp32part writes them", () => {
const decoded = table(
partitionTable([
...TWO_OTA_PARTITIONS,
{
label: "keys",
type: 0x01,
subtype: 0x04,
offset: 0x4_0000,
size: 0x1000,
flags: 0b11,
},
]),
);
expect(decoded.md5.status).toBe("valid");
expect(decoded.md5.entryOffset).toBe(7 * 32);
expect(decoded.length).toBe(9 * 32);
expect(decoded.limitations).toEqual([]);
expect(decoded.partitions.map(({ label }) => label)).toEqual([
"nvs",
"otadata",
"phy_init",
"factory",
"ota_0",
"ota_1",
"keys",
]);
expect(decoded.partitions.at(-1)).toMatchObject({
index: 6,
entryOffset: 6 * 32,
type: 0x01,
subtype: 0x04,
offset: 0x4_0000,
size: 0x1000,
encrypted: true,
readonly: true,
unknownFlags: 0,
});
});
it("accepts a table without an MD5 entry, as the bootloader does", () => {
const decoded = table(
partitionTable(TWO_OTA_PARTITIONS, { md5: "absent" }),
);
expect(decoded.md5).toEqual({
status: "absent",
entryOffset: null,
stored: null,
computed: null,
});
expect(decoded.partitions).toHaveLength(6);
});
it("keeps the entries of a table whose MD5 does not match", () => {
const decoded = table(
partitionTable(TWO_OTA_PARTITIONS, { md5: "corrupt" }),
);
expect(decoded.md5.status).toBe("invalid");
expect(decoded.partitions).toHaveLength(6);
expect(decoded.limitations).toEqual([
"The partition table MD5 at 0xc0 does not match the entries before it; ESP-IDF's bootloader rejects the table.",
]);
});
it.each([
["erased flash", Buffer.alloc(0xc00, 0xff), "the first entry is empty"],
[
"a foreign entry",
Buffer.concat([
partitionEntry(TWO_OTA_PARTITIONS[0]!),
Buffer.alloc(32, 0),
Buffer.alloc(0xbc0, 0xff),
]),
"entry 1 at 0x20 has magic 0x0, not a partition, MD5 or terminating entry",
],
[
"no terminator",
Buffer.concat(
Array.from({ length: 96 }, () =>
partitionEntry(TWO_OTA_PARTITIONS[0]!),
),
),
"no terminating entry within 96 entries (0xc00 bytes)",
],
[
"short bytes",
partitionEntry(TWO_OTA_PARTITIONS[0]!),
"the bytes end at 0x20, inside entry 1",
],
])("rejects %s as not a table", (_name, bytes, reason) => {
expect(decodeEspPartitionTable(bytes)).toEqual({
ok: false,
error: reason,
});
});
it("names types and subtypes as ESP-IDF does, and keeps unknown flags and labels", () => {
expect(espPartitionTypeName(0x00)).toBe("app");
expect(espPartitionSubtypeName(0x00, 0x1f)).toBe("ota_15");
expect(espPartitionSubtypeName(0x01, 0x04)).toBe("nvs_keys");
expect(espPartitionTypeName(0x40)).toBeNull();
expect(espPartitionSubtypeName(0x40, 0x00)).toBeNull();
const entry = partitionEntry({
label: "x",
type: 0x40,
subtype: 0,
offset: 0x1_0000,
size: 0x1000,
flags: 0x8000_0001,
});
entry.writeUInt8(0xff, 13);
const [decoded] = table(
Buffer.concat([entry, Buffer.alloc(0xbe0, 0xff)]),
).partitions;
expect(decoded).toMatchObject({
label: null,
encrypted: true,
unknownFlags: 0x8000_0000,
});
});
});
const partitions = (): readonly EspPartition[] =>
table(partitionTable(TWO_OTA_PARTITIONS)).partitions;
describe("ESP otadata selection", () => {
it("decodes records and checks the CRC the bootloader computes", () => {
expect(
decodeEspOtaRecord(otaRecord({ seq: 3, state: 2 }), 0),
).toMatchObject({
index: 0,
ota_seq: 3,
ota_state: 2,
state_name: "valid",
crc_matches: true,
valid: true,
});
expect(decodeEspOtaRecord(otaRecord({ seq: 3, crc: 0 }), 1)).toMatchObject({
crc_matches: false,
valid: false,
});
expect(decodeEspOtaRecord(Buffer.alloc(32, 0xff), 1)).toMatchObject({
ota_seq: 0xffff_ffff,
state_name: "undefined",
valid: false,
});
});
const select = (first: Buffer, second: Buffer, parts = partitions()) =>
selectEspOtaSlot(
[decodeEspOtaRecord(first, 0), decodeEspOtaRecord(second, 1)],
parts,
);
const erased = Buffer.alloc(32, 0xff);
it("selects the higher valid sequence and maps it onto the OTA slots", () => {
expect(select(otaRecord({ seq: 4 }), otaRecord({ seq: 5 }))).toMatchObject({
active_record: 1,
slot: 0,
partition_index: 4,
basis: "otadata",
});
// An invalid state removes a record; a tie keeps record 0.
expect(
select(otaRecord({ seq: 6, state: 3 }), otaRecord({ seq: 5 })),
).toMatchObject({ active_record: 1, slot: 0 });
expect(select(otaRecord({ seq: 2 }), otaRecord({ seq: 2 }))).toMatchObject({
active_record: 0,
slot: 1,
partition_index: 5,
});
});
it("falls back as the bootloader does without usable records", () => {
expect(select(erased, erased)).toMatchObject({
basis: "factory-fallback",
partition_index: 3,
slot: null,
});
const noFactory = partitions().filter(({ label }) => label !== "factory");
expect(select(erased, erased, noFactory)).toMatchObject({
basis: "ota_0-fallback",
slot: 0,
partition_index: 4,
});
// A set sequence with a bad CRC is not unusable, but no record is valid.
expect(
select(otaRecord({ seq: 2, crc: 1 }), erased, noFactory),
).toMatchObject({ basis: "none", partition_index: null });
expect(select(otaRecord({ seq: 2 }), erased).assumptions).toHaveLength(3);
});
it("indexes OTA slots by subtype, so a missing subtype selects no partition", () => {
const gapped = partitions().filter(({ label }) => label !== "ota_0");
// Two OTA apps would be counted, but only ota_1 exists: seq 1 selects slot 0.
const withSecond = [
...gapped,
{ ...gapped.at(-1)!, index: 9, subtype: 0x12, label: "ota_2" },
];
expect(select(otaRecord({ seq: 1 }), erased, withSecond)).toMatchObject({
slot: 0,
partition_index: null,
basis: "otadata",
});
});
});
describe("ESP bootloader location", () => {
const bootloader = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0005,
});
const probe = (flashAddress: number, bytes: Uint8Array) => ({
flashAddress,
fileOffset: flashAddress,
prefix: bytes.subarray(0, 32),
available: bytes.length,
});
it("locates the image at its chip's ROM offset", () => {
expect(locateEspBootloader([probe(0, bootloader)])).toMatchObject({
location: { flashAddress: 0, fileOffset: 0, chip: { target: "esp32c3" } },
limitations: [],
});
});
it("reports an image at another chip's ROM offset instead of locating it", () => {
const esp32 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0000,
});
expect(locateEspBootloader([probe(0, esp32)])).toEqual({
location: undefined,
limitations: [
"An esp32 image is at flash 0x0, but its ROM loads the bootloader from 0x1000; the dump may not start at the given flash offset.",
],
});
});
it("probes only the ROM offsets the file holds after its flash offset", () => {
expect(espBootloaderProbeOffsets(0, 0x10_0000)).toEqual([
{ flashAddress: 0, fileOffset: 0 },
{ flashAddress: 0x1000, fileOffset: 0x1000 },
{ flashAddress: 0x2000, fileOffset: 0x2000 },
]);
expect(espBootloaderProbeOffsets(0x1000, 0x10_0000)).toEqual([
{ flashAddress: 0x1000, fileOffset: 0 },
{ flashAddress: 0x2000, fileOffset: 0x1000 },
]);
});
});
+335
View File
@@ -0,0 +1,335 @@
import { offsetText } from "./espImageStructures.js";
import type { EspBootloaderLocation } from "./espFlashBootloader.js";
import {
ESP_PARTITION_TABLE_SECTOR_BYTES,
espPartitionSubtypeName,
espPartitionTypeName,
isEspOtaData,
type EspPartition,
type EspPartitionTable,
} from "./espPartitionTable.js";
import {
ESP_OTA_DATA_SECTOR_BYTES,
decodeEspOtaRecord,
selectEspOtaSlot,
type EspOtaRecord,
type EspOtaSelection,
} from "./espOtaData.js";
/** Where the file's first byte sits in flash, and who said so. */
export interface EspFlashOrigin {
readonly value: number;
readonly source: "default" | "caller";
}
export type EspPartitionTableMethod = "default" | "caller" | "scan";
/** One described byte range of the dump, in flash and in the file. */
export interface EspFlashRegion {
readonly path: string;
readonly role: "bootloader" | "partition-table" | "partition";
readonly flashAddress: number;
/** Length the layout gives the region. */
readonly length: number;
readonly fileOffset: number;
/** Bytes of the region the file holds, from `fileOffset`. */
readonly available: number;
readonly partition: EspPartition | undefined;
}
export interface EspFlashPlanInput {
readonly fileSize: number;
readonly origin: EspFlashOrigin;
readonly bootloader: EspBootloaderLocation | undefined;
readonly tableAddress: number;
readonly tableMethod: EspPartitionTableMethod;
readonly table: EspPartitionTable;
}
/** Regions to inventory and the byte ranges the adapter must still read. */
export interface EspFlashPlan {
readonly input: EspFlashPlanInput;
readonly regions: readonly EspFlashRegion[];
/** File ranges no region describes; erased flash there is described. */
readonly gaps: readonly {
readonly offset: number;
readonly length: number;
}[];
/** File offsets of the two otadata records, when the file holds them. */
readonly otadata:
| {
readonly partition: EspPartition;
readonly records: readonly [number, number];
}
| undefined;
readonly limitations: readonly string[];
}
const SAFE_LABEL = /^(?=.*[A-Za-z_])[A-Za-z0-9_][A-Za-z0-9._-]{0,15}$/u;
/**
* Partition paths use the label when it is a safe, unique path segment
* with a letter or underscore, else the table index, which then cannot
* collide with a label.
*/
const partitionPaths = (
partitions: readonly EspPartition[],
): ReadonlyMap<number, string> => {
const counts = new Map<string, number>();
for (const { label } of partitions)
if (label !== null) counts.set(label, (counts.get(label) ?? 0) + 1);
return new Map(
partitions.map((part) => {
const label = part.label;
const named =
label !== null && SAFE_LABEL.test(label) && counts.get(label) === 1;
return [
part.index,
`partitions/${named ? label : String(part.index)}.bin`,
];
}),
);
};
const region = (
input: EspFlashPlanInput,
fields: Omit<EspFlashRegion, "fileOffset" | "available">,
): EspFlashRegion => {
const fileOffset = fields.flashAddress - input.origin.value;
const start = Math.max(0, fileOffset);
const end = Math.min(input.fileSize, fileOffset + fields.length);
return { ...fields, fileOffset, available: Math.max(0, end - start) };
};
const rangeText = (address: number, length: number): string =>
`${offsetText(address)}-${offsetText(address + length)}`;
/** Lay the bootloader, table and partitions over the dump. */
export const planEspFlashLayout = (input: EspFlashPlanInput): EspFlashPlan => {
const limitations: string[] = [...input.table.limitations];
const regions: EspFlashRegion[] = [];
const tableEnd = input.tableAddress + ESP_PARTITION_TABLE_SECTOR_BYTES;
const { bootloader } = input;
if (bootloader !== undefined) {
if (bootloader.flashAddress < input.tableAddress)
regions.push(
region(input, {
path: "bootloader.bin",
role: "bootloader",
// ESP_BOOTLOADER_SIZE: from the bootloader to the partition table.
flashAddress: bootloader.flashAddress,
length: input.tableAddress - bootloader.flashAddress,
partition: undefined,
}),
);
else
limitations.push(
`The partition table at ${offsetText(input.tableAddress)} does not follow the bootloader at ${offsetText(bootloader.flashAddress)}; the bootloader region is not described.`,
);
}
regions.push(
region(input, {
path: "partition-table.bin",
role: "partition-table",
flashAddress: input.tableAddress,
length: ESP_PARTITION_TABLE_SECTOR_BYTES,
partition: undefined,
}),
);
const paths = partitionPaths(input.table.partitions);
for (const part of input.table.partitions) {
const placed = region(input, {
path: paths.get(part.index) ?? `partitions/${String(part.index)}.bin`,
role: "partition",
flashAddress: part.offset,
length: part.size,
partition: part,
});
const name = part.label ?? `#${String(part.index)}`;
if (part.offset < tableEnd)
limitations.push(
`Partition ${name} at ${offsetText(part.offset)} starts before the end of the partition table at ${offsetText(tableEnd)}; gen_esp32part.py rejects such a table.`,
);
if (placed.fileOffset < 0)
limitations.push(
`Partition ${name} at ${offsetText(part.offset)} begins before the dump's first byte at flash ${offsetText(input.origin.value)}; it is not read.`,
);
else if (placed.available === 0)
limitations.push(
`Partition ${name} at ${rangeText(part.offset, part.size)} lies outside the dump, which ends at flash ${offsetText(input.origin.value + input.fileSize)}.`,
);
else if (placed.available < part.size)
limitations.push(
`Partition ${name} at ${rangeText(part.offset, part.size)} is truncated: the dump holds ${String(placed.available)} of its ${String(part.size)} bytes.`,
);
if (placed.fileOffset >= 0 && placed.available > 0) regions.push(placed);
}
limitations.push(...overlaps(regions));
const otadataPartition = input.table.partitions.find(isEspOtaData);
const otadataStart =
otadataPartition === undefined
? undefined
: otadataPartition.offset - input.origin.value;
const otadata =
otadataPartition === undefined || otadataStart === undefined
? undefined
: otadataStart >= 0 &&
otadataStart + ESP_OTA_DATA_SECTOR_BYTES + 32 <= input.fileSize
? {
partition: otadataPartition,
records: [
otadataStart,
otadataStart + ESP_OTA_DATA_SECTOR_BYTES,
] as const,
}
: undefined;
if (otadataPartition !== undefined && otadata === undefined)
limitations.push(
`The dump does not hold both otadata records of partition ${otadataPartition.label ?? `#${String(otadataPartition.index)}`}; the OTA selection is unknown.`,
);
return {
input,
regions,
gaps: gapsBetween(regions, input.fileSize),
otadata,
limitations,
};
};
/** Regions that share bytes, which gen_esp32part.py rejects for partitions. */
const overlaps = (regions: readonly EspFlashRegion[]): string[] => {
const ordered = [...regions].sort(
(left, right) => left.flashAddress - right.flashAddress,
);
const found: string[] = [];
for (let index = 1; index < ordered.length; index += 1) {
const previous = ordered[index - 1];
const current = ordered[index];
if (
previous !== undefined &&
current !== undefined &&
current.flashAddress < previous.flashAddress + previous.length
)
found.push(
`${current.path} at ${offsetText(current.flashAddress)} overlaps ${previous.path}, which ends at ${offsetText(previous.flashAddress + previous.length)}.`,
);
}
return found;
};
/** File ranges outside every region, in order. */
const gapsBetween = (
regions: readonly EspFlashRegion[],
fileSize: number,
): { offset: number; length: number }[] => {
const covered = regions
.map(({ fileOffset, available }) => ({
start: Math.max(0, fileOffset),
end: Math.max(0, fileOffset) + available,
}))
.filter(({ end, start }) => end > start)
.sort((left, right) => left.start - right.start);
const gaps: { offset: number; length: number }[] = [];
let cursor = 0;
for (const { start, end } of covered) {
if (start > cursor) gaps.push({ offset: cursor, length: start - cursor });
cursor = Math.max(cursor, end);
}
if (fileSize > cursor)
gaps.push({ offset: cursor, length: fileSize - cursor });
return gaps;
};
export interface EspFlashOtaFacts {
readonly partition_index: number;
readonly records: readonly EspOtaRecord[];
/** Derived from the records; see its assumptions. */
readonly selected: EspOtaSelection;
/** The bootloader's rollback and fallback decide it at boot. */
readonly running_partition: "unknown";
}
/** Facts the adapter read for the plan: the otadata records and the gaps. */
export interface EspFlashReads {
readonly otadataRecords: readonly [Uint8Array, Uint8Array] | undefined;
/** The first non-erased file offset in each gap, in plan order. */
readonly gapFirstUnerased: readonly (number | undefined)[];
}
export interface EspFlashLayoutFacts {
readonly otadata: EspFlashOtaFacts | null;
readonly undescribed: readonly { offset: number; length: number }[];
readonly coverage: "complete" | "unknown";
readonly limitations: readonly string[];
}
/** Complete the plan with the bytes the adapter read. */
export const finishEspFlashLayout = (
plan: EspFlashPlan,
reads: EspFlashReads,
): EspFlashLayoutFacts => {
const limitations = [...plan.limitations];
const undescribed: { offset: number; length: number }[] = [];
plan.gaps.forEach((gap, index) => {
const first = reads.gapFirstUnerased[index];
if (first !== undefined)
undescribed.push({
offset: first,
length: gap.offset + gap.length - first,
});
});
const total = undescribed.reduce((sum, { length }) => sum + length, 0);
if (undescribed.length > 0)
limitations.push(
`${String(total)} bytes of the dump, in ${String(undescribed.length)} range(s), lie outside the bootloader region, the partition table and every partition and are not erased flash.`,
);
const otadata =
plan.otadata === undefined || reads.otadataRecords === undefined
? null
: otaFacts(plan, reads.otadataRecords);
const incomplete = plan.input.table.partitions.some((part) => {
const region = plan.regions.find((item) => item.partition === part);
return region === undefined || region.available < part.size;
});
return {
otadata,
undescribed,
coverage:
undescribed.length > 0 ||
incomplete ||
plan.input.bootloader === undefined
? "unknown"
: "complete",
limitations,
};
};
const otaFacts = (
plan: EspFlashPlan,
[first, second]: readonly [Uint8Array, Uint8Array],
): EspFlashOtaFacts | null => {
if (plan.otadata === undefined) return null;
const records = [decodeEspOtaRecord(first, 0), decodeEspOtaRecord(second, 1)];
return {
partition_index: plan.otadata.partition.index,
records,
selected: selectEspOtaSlot(records, plan.input.table.partitions),
running_partition: "unknown",
};
};
/** Flat entry metadata for one partition, as the table declares it. */
export const espFlashPartitionFacts = (part: EspPartition) => ({
kind: "esp-flash-partition" as const,
index: part.index,
label: part.label,
type: part.type,
subtype: part.subtype,
type_name: espPartitionTypeName(part.type),
subtype_name: espPartitionSubtypeName(part.type, part.subtype),
flash_address: part.offset,
size: part.size,
encrypted: part.encrypted,
readonly: part.readonly,
unknown_flags: part.unknownFlags,
});
@@ -0,0 +1,142 @@
import { describe, expect, it } from "vitest";
import {
locateEspFlashLayout,
type EspFlashSelection,
} from "./espFlashLocation.js";
import {
TWO_OTA_PARTITIONS,
flashImage,
partitionTable,
} from "./espFlash.fixture.js";
import { ESP32C3_BOOTLOADER_SEGMENTS, espImage } from "./espImage.fixture.js";
const c3Bootloader = espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS });
const esp32Bootloader = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0000,
});
const locate = (
bytes: Uint8Array,
selection: EspFlashSelection = {},
explicit = false,
) =>
locateEspFlashLayout(
bytes.length,
async (position, length) =>
bytes.subarray(Math.max(0, position), Math.max(0, position) + length),
selection,
{ explicit },
);
const dump = (
tableAt: number,
{
bootloader = c3Bootloader,
bootloaderAt = 0,
md5 = "valid",
}: {
readonly bootloader?: Uint8Array;
readonly bootloaderAt?: number;
readonly md5?: "valid" | "corrupt" | "absent";
} = {},
) =>
flashImage(0x2_0000, [
{ at: bootloaderAt, bytes: bootloader },
{ at: tableAt, bytes: partitionTable(TWO_OTA_PARTITIONS, { md5 }) },
]);
describe("ESP flash layout location", () => {
it("finds the bootloader and the table at their default offsets", async () => {
const located = await locate(dump(0x8000));
expect(located.ok && located.value).toMatchObject({
origin: { value: 0, source: "default" },
bootloader: { flashAddress: 0, chip: { target: "esp32c3" } },
tableAddress: 0x8000,
tableMethod: "default",
table: { md5: { status: "valid" } },
limitations: [],
});
});
it("scans for a relocated table only when its MD5 verifies", async () => {
const located = await locate(dump(0xa000));
expect(located.ok && located.value).toMatchObject({
tableAddress: 0xa000,
tableMethod: "scan",
});
expect(await locate(dump(0xa000, { md5: "absent" }))).toEqual({
ok: false,
error:
"no partition table at 0x8000 or in any 0x1000-aligned flash sector with a verified MD5; flash encryption is one cause",
});
});
it("uses the caller's table offset and does not scan", async () => {
const located = await locate(dump(0xa000), {
partitionTableOffset: 0xa000,
});
expect(located.ok && located.value).toMatchObject({
tableAddress: 0xa000,
tableMethod: "caller",
});
expect(
await locate(dump(0xa000), { partitionTableOffset: 0x9000 }),
).toEqual({
ok: false,
error:
"no partition table at the given offset 0x9000: the first entry is empty",
});
});
it("reads a dump that starts after flash address 0 at the caller's origin", async () => {
// An ESP32 dump read from 0x1000: its bootloader is the file's first byte.
const partial = dump(0x8000, {
bootloader: esp32Bootloader,
bootloaderAt: 0x1000,
}).subarray(0x1000);
const located = await locate(partial, { flashOffset: 0x1000 });
expect(located.ok && located.value).toMatchObject({
origin: { value: 0x1000, source: "caller" },
bootloader: {
flashAddress: 0x1000,
fileOffset: 0,
chip: { target: "esp32" },
},
tableAddress: 0x8000,
tableMethod: "default",
});
// Without the origin, the bootloader's chip says the dump is misplaced, so
// a scan would misplace every partition by 0x1000; none is attempted.
const misplaced = await locate(partial);
expect(misplaced.ok).toBe(false);
expect(!misplaced.ok && misplaced.error).toContain("Pass flash_offset");
});
it("does not scan arbitrary files without a bootloader unless the format was selected", async () => {
const bare = flashImage(0x2_0000, [
{ at: 0xa000, bytes: partitionTable(TWO_OTA_PARTITIONS) },
]);
expect(await locate(bare)).toEqual({
ok: false,
error: "no partition table at 0x8000 and no bootloader at a ROM offset",
});
const explicit = await locate(bare, {}, true);
expect(explicit.ok && explicit.value).toMatchObject({
bootloader: undefined,
tableAddress: 0xa000,
tableMethod: "scan",
});
});
it("rejects a table offset outside the dump", async () => {
expect(
await locate(dump(0x8000), { partitionTableOffset: 0x10_0000 }),
).toEqual({
ok: false,
error:
"the partition table offset 0x100000 is outside the dump, which holds flash 0x0-0x20000",
});
});
});
+181
View File
@@ -0,0 +1,181 @@
import { err, ok, type Result } from "../result.js";
import {
ESP_BOOTLOADER_PROBE_BYTES,
espBootloaderProbeOffsets,
locateEspBootloader,
type EspBootloaderLocation,
} from "./espFlashBootloader.js";
import type {
EspFlashOrigin,
EspPartitionTableMethod,
} from "./espFlashLayout.js";
import { offsetText } from "./espImageStructures.js";
import {
ESP_PARTITION_TABLE_DEFAULT_OFFSET,
ESP_PARTITION_TABLE_MAX_BYTES,
ESP_PARTITION_TABLE_SECTOR_BYTES,
decodeEspPartitionTable,
type EspPartitionTable,
} from "./espPartitionTable.js";
/** Caller inputs that select how a dump is read; omitted means the default. */
export interface EspFlashSelection {
/** Flash address of the dump's first byte. */
readonly flashOffset?: number | undefined;
/** Flash address of the partition table; it disables the scan. */
readonly partitionTableOffset?: number | undefined;
}
/** Where the layout sits in the dump, found once and shared by its readers. */
export interface EspFlashLocation {
readonly origin: EspFlashOrigin;
readonly bootloader: EspBootloaderLocation | undefined;
readonly tableAddress: number;
readonly tableMethod: EspPartitionTableMethod;
readonly table: EspPartitionTable;
readonly limitations: readonly string[];
}
/** Read up to `length` bytes at a file offset; fewer means end of file. */
export type EspFlashRead = (
position: number,
length: number,
) => Promise<Uint8Array>;
const SCAN_CHUNK_BYTES = 0x1_0000;
const MAGIC_LOW = 0xaa;
const MAGIC_HIGH = 0x50;
const tableAt = async (
read: EspFlashRead,
fileOffset: number,
): Promise<Result<EspPartitionTable, string>> =>
decodeEspPartitionTable(
await read(fileOffset, ESP_PARTITION_TABLE_MAX_BYTES),
);
/**
* The first sector-aligned table whose MD5 verifies; requiring the MD5
* keeps arbitrary bytes that begin with the magic from passing.
*/
const scanForTable = async (
read: EspFlashRead,
fileSize: number,
origin: number,
signal?: AbortSignal,
): Promise<{ address: number; table: EspPartitionTable } | undefined> => {
const skew =
(ESP_PARTITION_TABLE_SECTOR_BYTES -
(origin % ESP_PARTITION_TABLE_SECTOR_BYTES)) %
ESP_PARTITION_TABLE_SECTOR_BYTES;
for (let chunk = 0; chunk < fileSize; chunk += SCAN_CHUNK_BYTES) {
signal?.throwIfAborted();
// One extra byte lets a sector head at the chunk's last byte be checked.
const bytes = await read(chunk, SCAN_CHUNK_BYTES + 1);
const first =
chunk +
((skew -
(chunk % ESP_PARTITION_TABLE_SECTOR_BYTES) +
ESP_PARTITION_TABLE_SECTOR_BYTES) %
ESP_PARTITION_TABLE_SECTOR_BYTES);
for (
let position = first;
position < chunk + SCAN_CHUNK_BYTES && position < fileSize;
position += ESP_PARTITION_TABLE_SECTOR_BYTES
) {
const at = position - chunk;
if (bytes[at] !== MAGIC_LOW || bytes[at + 1] !== MAGIC_HIGH) continue;
const decoded = await tableAt(read, position);
if (decoded.ok && decoded.value.md5.status === "valid")
return { address: origin + position, table: decoded.value };
}
}
return undefined;
};
/**
* Locate the bootloader and partition table: the caller's table offset
* when given, else the default `0x8000`, else a scan of 0x1000-aligned
* flash sectors for an MD5-verified table (a derived location). The scan
* runs only when a bootloader sits at its chip's ROM offset or the caller
* selected this format, and never when a bootloader at another chip's ROM
* offset says the dump's origin is wrong.
*/
export const locateEspFlashLayout = async (
fileSize: number,
read: EspFlashRead,
selection: EspFlashSelection,
{
explicit,
signal,
}: { readonly explicit: boolean; readonly signal?: AbortSignal },
): Promise<Result<EspFlashLocation, string>> => {
const origin: EspFlashOrigin =
selection.flashOffset === undefined
? { value: 0, source: "default" }
: { value: selection.flashOffset, source: "caller" };
const probes = await Promise.all(
espBootloaderProbeOffsets(origin.value, fileSize).map(
async ({ flashAddress, fileOffset }) => ({
flashAddress,
fileOffset,
prefix: await read(fileOffset, ESP_BOOTLOADER_PROBE_BYTES),
available: fileSize - fileOffset,
}),
),
);
const bootloader = locateEspBootloader(probes);
const located = (
tableAddress: number,
tableMethod: EspPartitionTableMethod,
table: EspPartitionTable,
): Result<EspFlashLocation, string> =>
ok({
origin,
bootloader: bootloader.location,
tableAddress,
tableMethod,
table,
limitations: bootloader.limitations,
});
const fileOffsetOf = (address: number): number => address - origin.value;
if (selection.partitionTableOffset !== undefined) {
const address = selection.partitionTableOffset;
if (fileOffsetOf(address) < 0 || fileOffsetOf(address) >= fileSize)
return err(
`the partition table offset ${offsetText(address)} is outside the dump, which holds flash ${offsetText(origin.value)}-${offsetText(origin.value + fileSize)}`,
);
const decoded = await tableAt(read, fileOffsetOf(address));
return decoded.ok
? located(address, "caller", decoded.value)
: err(
`no partition table at the given offset ${offsetText(address)}: ${decoded.error}`,
);
}
const defaultOffset = fileOffsetOf(ESP_PARTITION_TABLE_DEFAULT_OFFSET);
if (defaultOffset >= 0 && defaultOffset < fileSize) {
const decoded = await tableAt(read, defaultOffset);
if (decoded.ok)
return located(
ESP_PARTITION_TABLE_DEFAULT_OFFSET,
"default",
decoded.value,
);
}
const misplaced =
bootloader.location === undefined && bootloader.limitations.length > 0;
if (misplaced && selection.flashOffset === undefined)
return err(
`no partition table at ${offsetText(ESP_PARTITION_TABLE_DEFAULT_OFFSET)}, and ${bootloader.limitations.join(" ")} Pass flash_offset to read the dump from another flash address`,
);
if (bootloader.location === undefined && !explicit)
return err(
`no partition table at ${offsetText(ESP_PARTITION_TABLE_DEFAULT_OFFSET)} and no bootloader at a ROM offset`,
);
const scanned = await scanForTable(read, fileSize, origin.value, signal);
return scanned === undefined
? err(
`no partition table at ${offsetText(ESP_PARTITION_TABLE_DEFAULT_OFFSET)} or in any 0x1000-aligned flash sector with a verified MD5; flash encryption is one cause`,
)
: located(scanned.address, "scan", scanned.table);
};
+219
View File
@@ -0,0 +1,219 @@
import { describe, expect, it } from "vitest";
import { espChipForId } from "./espChips.js";
import {
espFlashPartitionFacts,
finishEspFlashLayout,
planEspFlashLayout,
type EspFlashPlanInput,
} from "./espFlashLayout.js";
import {
decodeEspPartitionTable,
type EspPartitionTable,
} from "./espPartitionTable.js";
import {
TWO_OTA_PARTITIONS,
otaRecord,
partitionTable,
type EspPartitionFixture,
} from "./espFlash.fixture.js";
const table = (parts: readonly EspPartitionFixture[]): EspPartitionTable => {
const decoded = decodeEspPartitionTable(partitionTable(parts));
if (!decoded.ok) throw new Error(decoded.error);
return decoded.value;
};
const c3 = espChipForId(0x0005)!;
const input = (fields: Partial<EspFlashPlanInput> = {}): EspFlashPlanInput => ({
fileSize: 0x4_0000,
origin: { value: 0, source: "default" },
bootloader: { flashAddress: 0, fileOffset: 0, chip: c3 },
tableAddress: 0x8000,
tableMethod: "default",
table: table(TWO_OTA_PARTITIONS),
...fields,
});
const erasedGaps = (count: number) =>
Array.from({ length: count }, () => undefined);
describe("ESP flash layout plans", () => {
it("describes the bootloader region, the table sector and every partition", () => {
const plan = planEspFlashLayout(input());
expect(
plan.regions.map(({ path, flashAddress, length, available }) => [
path,
flashAddress,
length,
available,
]),
).toEqual([
["bootloader.bin", 0, 0x8000, 0x8000],
["partition-table.bin", 0x8000, 0x1000, 0x1000],
["partitions/nvs.bin", 0x9000, 0x4000, 0x4000],
["partitions/otadata.bin", 0xd000, 0x2000, 0x2000],
["partitions/phy_init.bin", 0xf000, 0x1000, 0x1000],
["partitions/factory.bin", 0x1_0000, 0x1_0000, 0x1_0000],
["partitions/ota_0.bin", 0x2_0000, 0x1_0000, 0x1_0000],
["partitions/ota_1.bin", 0x3_0000, 0x1_0000, 0x1_0000],
]);
expect(plan.gaps).toEqual([]);
expect(plan.otadata?.records).toEqual([0xd000, 0xe000]);
expect(plan.limitations).toEqual([]);
});
it("falls back to the index for unsafe or repeated labels", () => {
const parts: EspPartitionFixture[] = [
{ label: "../x", type: 1, subtype: 2, offset: 0x9000, size: 0x1000 },
{ label: "dup", type: 1, subtype: 2, offset: 0xa000, size: 0x1000 },
{ label: "dup", type: 1, subtype: 2, offset: 0xb000, size: 0x1000 },
{ label: "7", type: 1, subtype: 2, offset: 0xc000, size: 0x1000 },
];
expect(
planEspFlashLayout(input({ table: table(parts) }))
.regions.map(({ path }) => path)
.slice(2),
).toEqual([
"partitions/0.bin",
"partitions/1.bin",
"partitions/2.bin",
"partitions/3.bin",
]);
});
it("keeps a usable partition beside truncated, missing and early ones", () => {
const plan = planEspFlashLayout(
input({
fileSize: 0x3_8000,
origin: { value: 0x1000, source: "caller" },
bootloader: undefined,
table: table([
{ label: "early", type: 1, subtype: 2, offset: 0x0, size: 0x1000 },
{
label: "app",
type: 0,
subtype: 0,
offset: 0x1_0000,
size: 0x1_0000,
},
{
label: "cut",
type: 0,
subtype: 0x10,
offset: 0x3_0000,
size: 0x1_0000,
},
{
label: "gone",
type: 1,
subtype: 0x82,
offset: 0x10_0000,
size: 0x1000,
},
]),
}),
);
expect(
plan.regions.map(({ path, fileOffset, available }) => [
path,
fileOffset,
available,
]),
).toEqual([
["partition-table.bin", 0x7000, 0x1000],
["partitions/app.bin", 0xf000, 0x1_0000],
["partitions/cut.bin", 0x2_f000, 0x9000],
]);
expect(plan.limitations).toEqual([
"Partition early at 0x0 starts before the end of the partition table at 0x9000; gen_esp32part.py rejects such a table.",
"Partition early at 0x0 begins before the dump's first byte at flash 0x1000; it is not read.",
"Partition cut at 0x30000-0x40000 is truncated: the dump holds 36864 of its 65536 bytes.",
"Partition gone at 0x100000-0x101000 lies outside the dump, which ends at flash 0x39000.",
]);
const facts = finishEspFlashLayout(plan, {
otadataRecords: undefined,
gapFirstUnerased: erasedGaps(plan.gaps.length),
});
expect(facts.coverage).toBe("unknown");
});
it("reports overlapping regions", () => {
const plan = planEspFlashLayout(
input({
table: table([
{ label: "a", type: 1, subtype: 2, offset: 0x9000, size: 0x2000 },
{ label: "b", type: 1, subtype: 2, offset: 0xa000, size: 0x1000 },
]),
}),
);
expect(plan.limitations).toEqual([
"partitions/b.bin at 0xa000 overlaps partitions/a.bin, which ends at 0xb000.",
]);
});
});
describe("ESP flash layout facts", () => {
it("describes erased gaps and reports the rest as undescribed", () => {
const plan = planEspFlashLayout(
input({
table: table(TWO_OTA_PARTITIONS.slice(0, 3)),
fileSize: 0x2_0000,
}),
);
expect(plan.gaps).toEqual([{ offset: 0x1_0000, length: 0x1_0000 }]);
const erased = finishEspFlashLayout(plan, {
otadataRecords: undefined,
gapFirstUnerased: [undefined],
});
expect(erased).toMatchObject({ coverage: "complete", undescribed: [] });
const written = finishEspFlashLayout(plan, {
otadataRecords: undefined,
gapFirstUnerased: [0x1_8000],
});
expect(written.undescribed).toEqual([{ offset: 0x1_8000, length: 0x8000 }]);
expect(written.coverage).toBe("unknown");
expect(written.limitations.at(-1)).toBe(
"32768 bytes of the dump, in 1 range(s), lie outside the bootloader region, the partition table and every partition and are not erased flash.",
);
});
it("derives the selected app from the otadata records and leaves the running one unknown", () => {
const plan = planEspFlashLayout(input());
const facts = finishEspFlashLayout(plan, {
otadataRecords: [otaRecord({ seq: 2 }), Buffer.alloc(32, 0xff)],
gapFirstUnerased: [],
});
expect(facts.otadata).toMatchObject({
partition_index: 1,
records: [{ ota_seq: 2, valid: true }, { valid: false }],
selected: {
active_record: 0,
slot: 1,
partition_index: 5,
basis: "otadata",
},
running_partition: "unknown",
});
expect(facts.coverage).toBe("complete");
});
it("projects a partition's declaration with ESP-IDF names", () => {
const [nvs] = table(TWO_OTA_PARTITIONS).partitions;
expect(espFlashPartitionFacts(nvs!)).toEqual({
kind: "esp-flash-partition",
index: 0,
label: "nvs",
type: 1,
subtype: 2,
type_name: "data",
subtype_name: "nvs",
flash_address: 0x9000,
size: 0x4000,
encrypted: false,
readonly: false,
unknown_flags: 0,
});
});
});
+32
View File
@@ -0,0 +1,32 @@
import { z } from "zod";
import type { ExecutableFormatHint } from "../dosCom.js";
import { ESP_PARTITION_TABLE_SECTOR_BYTES } from "./espPartitionTable.js";
/** A 32-bit flash address written as `0x`-prefixed hexadecimal. */
export const espFlashAddressSchema = z.string().regex(/^0x[0-9a-fA-F]{1,8}$/u, {
message: "must be a hexadecimal flash address of the form 0x…",
});
export const parseEspFlashAddress = (text: string): number =>
Number.parseInt(text.slice(2), 16);
/** Describe the format and flash-input pairing without performing I/O. */
export const espFlashSelectionError = (input: {
readonly format?: ExecutableFormatHint | undefined;
readonly flashOffset?: number | undefined;
readonly partitionTableOffset?: number | undefined;
}): string | undefined => {
if (
(input.flashOffset !== undefined ||
input.partitionTableOffset !== undefined) &&
input.format !== "esp-flash-image"
)
return "flash_offset and partition_table_offset apply only with format=esp-flash-image";
if (
input.partitionTableOffset !== undefined &&
input.partitionTableOffset % ESP_PARTITION_TABLE_SECTOR_BYTES !== 0
)
return "partition_table_offset must be a multiple of 0x1000, as ESP-IDF requires";
return undefined;
};
+165
View File
@@ -0,0 +1,165 @@
import { crc32 } from "node:zlib";
import { hex, u32 } from "./espImageStructures.js";
import {
espOtaAppSlot,
isEspFactoryApp,
type EspPartition,
} from "./espPartitionTable.js";
/*
* OTA data, from ESP-IDF v6.1 `esp_flash_partitions.h`
* (`esp_ota_select_entry_t`, `esp_ota_img_states_t`),
* `bootloader_common_loader.c` and `bootloader_utility.c`
* (`bootloader_utility_get_selected_boot_partition`), and `otatool.py`.
*/
/** The otadata partition holds one record at the start of each of two sectors. */
export const ESP_OTA_DATA_SECTOR_BYTES = 0x1000;
export const ESP_OTA_SELECT_ENTRY_BYTES = 32;
const UNSET = 0xffff_ffff;
export type EspOtaStateName =
| "new"
| "pending-verify"
| "valid"
| "invalid"
| "aborted"
| "undefined";
const STATE_NAMES: Readonly<Record<number, EspOtaStateName>> = {
0x0: "new",
0x1: "pending-verify",
0x2: "valid",
0x3: "invalid",
0x4: "aborted",
[UNSET]: "undefined",
};
/** One `esp_ota_select_entry_t`, observed as written. */
export interface EspOtaRecord {
readonly index: 0 | 1;
readonly ota_seq: number;
readonly seq_label: string;
readonly ota_state: number;
readonly state_name: EspOtaStateName | null;
readonly crc: number;
/** CRC-32 of `ota_seq`, seeded with `0xFFFFFFFF`, as the bootloader computes it. */
readonly crc_matches: boolean;
/** `bootloader_common_ota_select_valid`: set sequence, usable state, matching CRC. */
readonly valid: boolean;
}
export const decodeEspOtaRecord = (
bytes: Uint8Array,
index: 0 | 1,
): EspOtaRecord => {
const sequence = bytes.subarray(0, 4);
const otaSeq = u32(bytes, 0);
const state = u32(bytes, 24);
const crc = u32(bytes, 28);
const crcMatches = crc32(sequence, UNSET) >>> 0 === crc;
return {
index,
ota_seq: otaSeq,
seq_label: hex(bytes.subarray(4, 24)),
ota_state: state,
state_name: STATE_NAMES[state] ?? null,
crc,
crc_matches: crcMatches,
valid: otaSeq !== UNSET && state !== 0x3 && state !== 0x4 && crcMatches,
};
};
export const ESP_OTA_SELECTION_ASSUMPTIONS = [
"Records are used as written: the bootloader may first mark a pending-verify record aborted when app rollback is enabled.",
"Anti-rollback, which compares the app's secure_version with eFuse, is not applied.",
"The selected app is not checked for a loadable image; the bootloader falls back to other partitions when it fails to load.",
] as const;
export interface EspOtaSelection {
/** The record the bootloader reads as active: the higher valid `ota_seq`. */
readonly active_record: 0 | 1 | null;
/** `(ota_seq - 1) % ota_app_count`, the bootloader's slot arithmetic. */
readonly slot: number | null;
/** The table index of the app partition for that slot or the fallback. */
readonly partition_index: number | null;
readonly basis: "otadata" | "factory-fallback" | "ota_0-fallback" | "none";
readonly assumptions: readonly string[];
}
/** `bootloader_common_ota_select_invalid`, which does not check the CRC. */
const unusable = ({ ota_seq, ota_state }: EspOtaRecord): boolean =>
ota_seq === UNSET || ota_state === 0x3 || ota_state === 0x4;
/**
* Derive the app partition the bootloader would select from these records,
* as `bootloader_utility_get_selected_boot_partition` does without app
* rollback or anti-rollback:
* - both records unusable, or no OTA app: the factory app, else `ota_0`;
* - otherwise the valid record with the higher `ota_seq` (record 0 on a
* tie), whose slot is `(ota_seq - 1) % ota_app_count`;
* - no valid record (a CRC mismatch): the factory app, else none, because
* the bootloader then tries every partition.
*/
export const selectEspOtaSlot = (
records: readonly EspOtaRecord[],
partitions: readonly EspPartition[],
): EspOtaSelection => {
// The bootloader counts every OTA app but indexes them by subtype, and a
// later entry with the same subtype replaces an earlier one.
let appCount = 0;
const bySlot = new Map<number, EspPartition>();
for (const part of partitions) {
const slot = espOtaAppSlot(part);
if (slot === null) continue;
appCount += 1;
bySlot.set(slot, part);
}
const factory = partitions.find(isEspFactoryApp);
const assumptions = [...ESP_OTA_SELECTION_ASSUMPTIONS];
const none = {
active_record: null,
slot: null,
partition_index: null,
basis: "none",
assumptions,
} as const;
const factoryFallback =
factory === undefined
? undefined
: ({
...none,
partition_index: factory.index,
basis: "factory-fallback",
} as const);
if (records.every(unusable) || appCount === 0) {
if (factoryFallback !== undefined) return factoryFallback;
const first = bySlot.get(0);
return first === undefined
? none
: {
...none,
slot: 0,
partition_index: first.index,
basis: "ota_0-fallback",
};
}
const valid = records.filter(({ valid }) => valid);
const [first, second] = valid;
const active =
first !== undefined &&
second !== undefined &&
second.ota_seq > first.ota_seq
? second
: first;
if (active === undefined) return factoryFallback ?? none;
const slot = ((active.ota_seq - 1) >>> 0) % appCount;
return {
active_record: active.index,
slot,
// A gap in the OTA subtypes selects a slot with no partition.
partition_index: bySlot.get(slot)?.index ?? null,
basis: "otadata",
assumptions,
};
};
+234
View File
@@ -0,0 +1,234 @@
import { createHash } from "node:crypto";
import { err, ok, type Result } from "../result.js";
import { hex, offsetText, u16, u32 } from "./espImageStructures.js";
/*
* The partition table, from ESP-IDF v6.1 `esp_flash_partitions.h`,
* `bootloader_support/src/flash_partitions.c` (`esp_partition_table_verify`,
* which the bootloader applies) and `partition_table/gen_esp32part.py`.
*/
export const ESP_PARTITION_TABLE_DEFAULT_OFFSET = 0x8000;
/** The table occupies one flash sector (`ESP_PARTITION_TABLE_SIZE`). */
export const ESP_PARTITION_TABLE_SECTOR_BYTES = 0x1000;
/** `ESP_PARTITION_TABLE_MAX_LEN`: 96 entries, the MD5 and the terminator included. */
export const ESP_PARTITION_TABLE_MAX_BYTES = 0xc00;
const ENTRY_BYTES = 32;
const MAX_ENTRIES = ESP_PARTITION_TABLE_MAX_BYTES / ENTRY_BYTES;
const PARTITION_MAGIC = 0x50aa;
const MD5_MAGIC = 0xebeb;
const END_MAGIC = 0xffff;
const END_TYPE = 0xff;
/** `ESP_PARTITION_MD5_OFFSET`: the digest follows 16 bytes into its entry. */
const MD5_DIGEST_OFFSET = 16;
const KNOWN_FLAGS = 0b11;
/** One `esp_partition_info_t`, as the table declares it. */
export interface EspPartition {
/** Position among the partition entries; the MD5 entry is not counted. */
readonly index: number;
/** Byte offset of the entry from the start of the table. */
readonly entryOffset: number;
readonly type: number;
readonly subtype: number;
/** Flash address. */
readonly offset: number;
readonly size: number;
/** The label up to its first NUL, or null when it is not UTF-8. */
readonly label: string | null;
readonly encrypted: boolean;
readonly readonly: boolean;
/** Flag bits other than `encrypted` and `readonly`, which ESP-IDF ignores. */
readonly unknownFlags: number;
}
export interface EspPartitionTableMd5 {
readonly status: "valid" | "invalid" | "absent";
readonly entryOffset: number | null;
readonly stored: string | null;
readonly computed: string | null;
}
export interface EspPartitionTable {
readonly partitions: readonly EspPartition[];
readonly md5: EspPartitionTableMd5;
/** Bytes through the terminating entry. */
readonly length: number;
readonly limitations: readonly string[];
}
const UTF8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
const label = (bytes: Uint8Array): string | null => {
const end = bytes.indexOf(0);
try {
return UTF8.decode(end < 0 ? bytes : bytes.subarray(0, end));
} catch {
return null;
}
};
const partition = (
entry: Uint8Array,
index: number,
entryOffset: number,
): EspPartition => {
const flags = u32(entry, 28);
return {
index,
entryOffset,
type: entry[2] ?? 0,
subtype: entry[3] ?? 0,
offset: u32(entry, 4),
size: u32(entry, 8),
label: label(entry.subarray(12, 28)),
encrypted: (flags & 1) !== 0,
readonly: (flags & 2) !== 0,
unknownFlags: (flags & ~KNOWN_FLAGS) >>> 0,
};
};
/**
* Decode a table as ESP-IDF's bootloader verifies it: 32-byte entries with
* the `0x50AA` magic, at most one `0xEBEB` MD5 entry over the entries before
* it, and a terminator whose magic, type and subtype are all `0xFF`, which
* cannot be the first entry. A structural fault means the bytes are not a
* table; an MD5 mismatch, which the bootloader also rejects, is reported as
* an integrity fact so that the declared entries stay visible.
*/
export const decodeEspPartitionTable = (
bytes: Uint8Array,
): Result<EspPartitionTable, string> => {
const partitions: EspPartition[] = [];
const limitations: string[] = [];
let md5: EspPartitionTableMd5 = {
status: "absent",
entryOffset: null,
stored: null,
computed: null,
};
for (let slot = 0; slot < MAX_ENTRIES; slot += 1) {
const at = slot * ENTRY_BYTES;
const entry = bytes.subarray(at, at + ENTRY_BYTES);
if (entry.length < ENTRY_BYTES)
return err(
`the bytes end at ${offsetText(at + entry.length)}, inside entry ${String(slot)}`,
);
const magic = u16(entry, 0);
if (magic === PARTITION_MAGIC) {
partitions.push(partition(entry, partitions.length, at));
continue;
}
if (magic === MD5_MAGIC) {
const computed = createHash("md5")
.update(bytes.subarray(0, at))
.digest("hex");
const stored = hex(entry.subarray(MD5_DIGEST_OFFSET, ENTRY_BYTES));
if (md5.status !== "absent") {
limitations.push(
`The partition table has a second MD5 entry at ${offsetText(at)}; ESP-IDF allows one and rejects the table.`,
);
md5 = { ...md5, status: "invalid" };
continue;
}
md5 = {
status: stored === computed ? "valid" : "invalid",
entryOffset: at,
stored,
computed,
};
if (stored !== computed)
limitations.push(
`The partition table MD5 at ${offsetText(at)} does not match the entries before it; ESP-IDF's bootloader rejects the table.`,
);
continue;
}
if (
slot > 0 &&
magic === END_MAGIC &&
entry[2] === END_TYPE &&
entry[3] === END_TYPE
)
return ok({ partitions, md5, length: at + ENTRY_BYTES, limitations });
return err(
slot === 0 && magic === END_MAGIC
? "the first entry is empty"
: `entry ${String(slot)} at ${offsetText(at)} has magic ${offsetText(magic)}, not a partition, MD5 or terminating entry`,
);
}
return err(
`no terminating entry within ${String(MAX_ENTRIES)} entries (${offsetText(ESP_PARTITION_TABLE_MAX_BYTES)} bytes)`,
);
};
/** ESP-IDF's names for partition types (`gen_esp32part.py` `TYPES`). */
const TYPE_NAMES: Readonly<Record<number, string>> = {
0x00: "app",
0x01: "data",
0x02: "bootloader",
0x03: "partition_table",
};
/** ESP-IDF's subtype names per type (`gen_esp32part.py` `SUBTYPES`). */
const SUBTYPE_NAMES: Readonly<
Record<number, Readonly<Record<number, string>>>
> = {
0x00: {
0x00: "factory",
0x20: "test",
...Object.fromEntries(
Array.from({ length: 16 }, (_, slot) => [
0x10 + slot,
`ota_${String(slot)}`,
]),
),
0x30: "tee_0",
0x31: "tee_1",
},
0x01: {
0x00: "ota",
0x01: "phy",
0x02: "nvs",
0x03: "coredump",
0x04: "nvs_keys",
0x05: "efuse",
0x06: "undefined",
0x80: "esphttpd",
0x81: "fat",
0x82: "spiffs",
0x83: "littlefs",
0x90: "tee_ota",
},
0x02: { 0x00: "primary", 0x01: "ota", 0x02: "recovery" },
0x03: { 0x00: "primary", 0x01: "ota" },
};
export const espPartitionTypeName = (type: number): string | null =>
TYPE_NAMES[type] ?? null;
export const espPartitionSubtypeName = (
type: number,
subtype: number,
): string | null => SUBTYPE_NAMES[type]?.[subtype] ?? null;
const APP_TYPE = 0x00;
const OTA_DATA_SUBTYPE = 0x00;
const DATA_TYPE = 0x01;
const OTA_APP_SUBTYPE_FLAG = 0x10;
const OTA_APP_SUBTYPE_MASK = 0x0f;
/** The OTA slot of an app partition (`ota_0` to `ota_15`), if it has one. */
export const espOtaAppSlot = (part: EspPartition): number | null =>
part.type === APP_TYPE &&
(part.subtype & ~OTA_APP_SUBTYPE_MASK) === OTA_APP_SUBTYPE_FLAG
? part.subtype & OTA_APP_SUBTYPE_MASK
: null;
export const isEspFactoryApp = (part: EspPartition): boolean =>
part.type === APP_TYPE && part.subtype === 0x00;
export const isEspOtaData = (part: EspPartition): boolean =>
part.type === DATA_TYPE && part.subtype === OTA_DATA_SUBTYPE;
export const isEspAppPartition = (part: EspPartition): boolean =>
part.type === APP_TYPE;
+24
View File
@@ -77,6 +77,30 @@ export const liveBinaryTargetSummaryVariants = [
]),
architecture: z.null(),
}),
z.object({
...identity,
kind: z.literal("archive"),
format: z.literal("esp-flash-image"),
architecture: z.null(),
flash_offset: z
.number()
.int()
.min(0)
.max(0xffff_ffff)
.nullable()
.describe(
"Caller-selected flash address of the dump's first byte; null reads it as flash address 0.",
),
partition_table_offset: z
.number()
.int()
.min(0)
.max(0xffff_ffff)
.nullable()
.describe(
"Caller-selected partition table address; null tries 0x8000, then a scan.",
),
}),
z.object({
...identity,
kind: z.literal("artifact"),
+20
View File
@@ -1,3 +1,4 @@
import { parseEspFlashAddress } from "../domain/firmware/espFlashSelection.js";
import { registerNativeFunctionMatchingTool } from "./registerNativeFunctionMatchingTool.js";
import type { EvidenceMcpServer } from "./EvidenceMcpServer.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
@@ -192,6 +193,25 @@ const registerOpenLifecycleTool = ({
...(input.entry_point === undefined
? {}
: { entryPoint: input.entry_point }),
...(input.flash_offset === undefined &&
input.partition_table_offset === undefined
? {}
: {
espFlash: {
...(input.flash_offset === undefined
? {}
: {
flashOffset: parseEspFlashAddress(input.flash_offset),
}),
...(input.partition_table_offset === undefined
? {}
: {
partitionTableOffset: parseEspFlashAddress(
input.partition_table_offset,
),
}),
},
}),
...(input.provider_id === undefined
? {}
: { providerId: input.provider_id }),
+136
View File
@@ -0,0 +1,136 @@
import { readFile, rm, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, onTestFinished } from "vitest";
import { cliTest } from "../support/cli/cliFixture.js";
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
espImage,
} from "../../src/domain/firmware/espImage.fixture.js";
import {
espFlashDump,
otaData,
otaRecord,
} from "../../src/domain/firmware/espFlash.fixture.js";
import { parseEvidence } from "../../src/domain/evidence.js";
import { artifactInspectionResultSchema } from "../../src/domain/artifactInspection.js";
import {
artifactExtractionResultSchema,
artifactInventoryResultSchema,
} from "../../src/domain/artifactGraph.js";
const rootMetadata = (json: unknown) => {
const graph = artifactInventoryResultSchema.parse(
artifactInspectionResultSchema.parse(parseEvidence(json).normalized_result)
.substeps[0]?.evidence.normalized_result,
);
return {
graph,
root: graph.occurrences.find(({ logical_path }) => logical_path === ".")
?.format_metadata,
};
};
cliTest(
"inventories and extracts a whole-flash dump through the compiled CLI",
async ({ cli }) => {
const directory = await createTestTempDirectory("rea-esp-flash-cli-");
const source = join(directory, "flash.bin");
const app = espImage({ segments: ESP32C3_APP_SEGMENTS });
const dump = espFlashDump({
bootloader: {
at: 0,
bytes: espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS }),
},
parts: [
{ at: 0xd000, bytes: otaData(otaRecord({ seq: 2 })) },
{ at: 0x2_0000, bytes: app },
{ at: 0x3_0000, bytes: app },
],
});
await writeFile(source, dump);
const inspected = await cli.run({
arguments: ["inspect-artifact", source, "--json"],
});
expect(inspected.exitCode, inspected.stdout + inspected.stderr).toBe(0);
const { graph, root } = rootMetadata(inspected.json);
expect(graph.manifest.root_format).toBe("esp-flash-image");
expect(root).toMatchObject({
kind: "esp-flash-image",
chip: { target: "esp32c3" },
partition_table: { method: "default", md5: { status: "valid" } },
otadata: {
selected: { slot: 1, partition_index: 5, basis: "otadata" },
running_partition: "unknown",
},
});
const extracted = await cli.run({
arguments: ["extract-artifact", source, "--json"],
environment: { TMPDIR: directory, TMP: directory, TEMP: directory },
});
expect(extracted.exitCode, extracted.stdout + extracted.stderr).toBe(0);
const result = artifactExtractionResultSchema.parse(
parseEvidence(extracted.json).normalized_result,
);
onTestFinished(() =>
rm(result.output_root, { recursive: true, force: true }),
);
expect(
Buffer.compare(
await readFile(join(result.output_root, "partitions/ota_1.bin")),
dump.subarray(0x3_0000, 0x4_0000),
),
).toBe(0);
},
);
cliTest(
"reads a dump from a caller-selected flash offset and refuses unpaired inputs",
async ({ cli }) => {
const directory = await createTestTempDirectory(
"rea-esp-flash-cli-origin-",
);
const source = join(directory, "from-0x1000.bin");
const esp32 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0000,
});
await writeFile(
source,
espFlashDump({ bootloader: { at: 0x1000, bytes: esp32 } }).subarray(
0x1000,
),
);
const unpaired = await cli.run({
arguments: [
"inspect-artifact",
source,
"--flash-offset",
"0x1000",
"--json",
],
});
expect(unpaired.exitCode).not.toBe(0);
expect(unpaired.stdout + unpaired.stderr).toContain(
"flash_offset and partition_table_offset apply only with format=esp-flash-image",
);
const inspected = await cli.run({
arguments: [
"inspect-artifact",
source,
"--target-format",
"esp-flash-image",
"--flash-offset",
"0x1000",
"--json",
],
});
expect(inspected.exitCode, inspected.stdout + inspected.stderr).toBe(0);
expect(rootMetadata(inspected.json).root).toMatchObject({
flash_offset: { value: 0x1000, source: "caller" },
chip: { target: "esp32" },
bootloader: { flash_address: 0x1000 },
});
},
);
@@ -35,9 +35,11 @@ import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { ESP8266_REFUSAL } from "../../../src/domain/firmware/espImage.js";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
esp8266Image,
espImage,
} from "../../../src/domain/firmware/espImage.fixture.js";
import { espFlashDump } from "../../../src/domain/firmware/espFlash.fixture.js";
import { javaClassFile } from "../../fixtures/javaClassFile.js";
import {
MAX_JVM_CLASS_IDENTITY_BYTES,
@@ -564,6 +566,104 @@ describe("binary target I/O: package classification", () => {
});
});
describe("binary target I/O: ESP flash layouts", () => {
it("detects a flash layout before the bootloader it begins with", async () => {
const directory = await createTestTempDirectory("rea-esp-flash-target-");
const dump = join(directory, "flash.bin");
const bootloader = join(directory, "bootloader.bin");
const image = espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS });
await Promise.all([
writeFile(dump, espFlashDump({ bootloader: { at: 0, bytes: image } })),
writeFile(bootloader, image),
]);
expect(await parseBinaryTarget(dump)).toMatchObject({
ok: true,
value: {
kind: "archive",
format: "esp-flash-image",
espFlash: {
selected: false,
flashOffset: null,
partitionTableOffset: null,
},
},
});
expect(await parseBinaryTarget(bootloader)).toMatchObject({
ok: true,
value: { kind: "archive", format: "esp-image" },
});
});
it("reads a selected flash dump with caller inputs and refuses bad pairings", async () => {
const directory = await createTestTempDirectory("rea-esp-flash-select-");
const esp32 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0,
});
const partial = join(directory, "from-0x1000.bin");
const noise = join(directory, "noise.bin");
await Promise.all([
writeFile(
partial,
espFlashDump({ bootloader: { at: 0x1000, bytes: esp32 } }).subarray(
0x1000,
),
),
writeFile(noise, Buffer.alloc(0x1_0000, 0x5a)),
]);
expect(
await parseBinaryTarget(partial, {
formatHint: "esp-flash-image",
espFlash: { flashOffset: 0x1000 },
}),
).toMatchObject({
ok: true,
value: {
kind: "archive",
format: "esp-flash-image",
espFlash: {
selected: true,
flashOffset: 0x1000,
partitionTableOffset: null,
},
},
});
for (const [options, reason] of [
[
{ espFlash: { flashOffset: 0x1000 } },
"flash_offset and partition_table_offset apply only with format=esp-flash-image",
],
[
{
formatHint: "esp-flash-image",
espFlash: { partitionTableOffset: 0x8800 },
},
"partition_table_offset must be a multiple of 0x1000, as ESP-IDF requires",
],
] as const)
expect(await parseBinaryTarget(partial, options)).toMatchObject({
ok: false,
error: {
_tag: "BinaryTargetError",
constraint: "load_selection_rejected",
reason,
},
});
// Ciphertext is not recognized as a layout: unknown, not malformed.
expect(
await parseBinaryTarget(noise, { formatHint: "esp-flash-image" }),
).toMatchObject({
ok: false,
error: {
constraint: "file_format_rejected",
reason: expect.stringContaining(
"ESP flash image layout is not recognized",
),
},
});
});
});
describe("binary target I/O: filesystem and app bundle target resolution", () => {
it("resolves relative Hopper databases and rejects unknown or unreadable paths", async () => {
const directory = await createTestTempDirectory("rea-target-");
@@ -0,0 +1,98 @@
import { readFile, rm, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import { z } from "zod";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js";
import {
ESP32C3_BOOTLOADER_SEGMENTS,
espImage,
} from "../../../src/domain/firmware/espImage.fixture.js";
import { espFlashDump } from "../../../src/domain/firmware/espFlash.fixture.js";
import { parseEvidence } from "../../../src/domain/evidence.js";
import { artifactExtractionResultSchema } from "../../../src/domain/artifactGraph.js";
const summarySchema = z.object({
summary: z.object({ root_format_metadata: z.unknown() }),
});
// The compiled stdio fixture checks advertised schemas against their dialect
// and validates actual tool responses.
it.each(["full", "compact"] as const)(
"reads a flash dump at the caller's origin and keeps inputs in its identity (%s)",
async (profile) => {
const root = await createTestTempDirectory("rea-esp-flash-mcp-");
const path = join(root, "from-0x1000.bin");
const esp32 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0000,
});
const dump = espFlashDump({ bootloader: { at: 0x1000, bytes: esp32 } });
await writeFile(path, dump.subarray(0x1000));
const { call } = await connectLocalToolsMcp({
REA_MCP_INPUT_SCHEMA_PROFILE: profile,
});
let outputRoot: string | undefined;
try {
const rejected = await call("open_binary", {
path,
flash_offset: "0x1000",
});
expect(rejected.isError).toBe(true);
expect(JSON.stringify(rejected)).toContain(
"flash_offset and partition_table_offset apply only with format=esp-flash-image",
);
const opened = await call("open_binary", {
path,
format: "esp-flash-image",
flash_offset: "0x1000",
});
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
expect(JSON.stringify(opened.structuredContent)).toContain(
'"partition_table_offset":null',
);
const reply = await call("inspect_artifact", { detail: "summary" });
expect(reply.isError, JSON.stringify(reply)).not.toBe(true);
expect(
summarySchema.parse(
parseEvidence(reply.structuredContent).normalized_result,
).summary.root_format_metadata,
).toMatchObject({
kind: "esp-flash-image",
flash_offset: { value: 0x1000, source: "caller" },
chip: { target: "esp32" },
partition_table: { flash_address: 0x8000, method: "default" },
});
const extracted = await call("extract_artifact", {});
expect(extracted.isError, JSON.stringify(extracted)).not.toBe(true);
const result = artifactExtractionResultSchema.parse(
parseEvidence(extracted.structuredContent).normalized_result,
);
outputRoot = result.output_root;
// nvs at flash 0x9000 is file offset 0x8000 in a dump read from 0x1000.
expect(
Buffer.compare(
await readFile(join(result.output_root, "partitions/nvs.bin")),
dump.subarray(0x9000, 0xd000),
),
).toBe(0);
// Reopening the same bytes with other inputs is a different target.
const reopened = await call("open_binary", {
path,
format: "esp-flash-image",
flash_offset: "0x1000",
partition_table_offset: "0x8000",
});
expect(reopened.isError, JSON.stringify(reopened)).not.toBe(true);
const again = await call("inspect_artifact", { detail: "summary" });
expect(
summarySchema.parse(
parseEvidence(again.structuredContent).normalized_result,
).summary.root_format_metadata,
).toMatchObject({ partition_table: { method: "caller" } });
} finally {
if (outputRoot !== undefined)
await rm(outputRoot, { recursive: true, force: true });
}
},
);