fix(artifacts): read ESP flash layouts as the bootloader does

Read the partition table as bootloader_utility_load_partition_table
does: the last factory, otadata and same-subtype OTA declaration wins,
and a limitation names repeats. Plan each otadata record within its
partition and the dump. An otadata partition smaller than two sectors
selects none, the bootloader's INVALID_INDEX; a record the dump lacks
leaves the selection unknown while the other record stays reported. The
fallbacks follow the bootloader's indexes: ota_0-fallback and
factory-fallback name no partition when the table declares none, and the
assumptions add GPIO holds and deep-sleep wake.

Refuse a dump whose bootloader contradicts its origin before the default
table offset is tried, whether flash_offset was omitted or given, and keep
the ESP image reader's ESP8266 refusal. A table never lies inside the
located bootloader image, so single app images are no longer scanned. An
explicit esp-flash-image selection is decided before other container
signatures, and the native commands' --target-format no longer takes it.
Say when no bootloader confirms the default origin.

Exempt the primary bootloader and partition_table entries of ESP-IDF 5.3
from placement limits, report every region inside a longer one, and name
partitions by index when labels repeat ignoring case. Keep an undecodable
label's bytes in label_hex and compare them. Word undescribed ranges as
starting at a byte that is not erased, and treat a short erased region as
erased.

Add an MCP compare_artifacts test for partition declarations, reader
identity and close tests, scan edge cases, the 0x2000 ROM offset and a
literal otatool CRC.
This commit is contained in:
Entrpi committed 2026-10-28 04:04:35 +11:00
1 parent 43247a2afb
commit 4561779b58
19 files changed
+1188 -330

No files matched your search

@@ -679,7 +679,8 @@ cannot be opened as raw-binary. See
A whole-flash dump opens as `esp-flash-image`. `root_format_metadata` holds
`partition_table` (its `method`: `default`, `caller` or a derived
`scan`), `bootloader` and `apps` ESP image facts, `otadata` with a derived
`selected` slot and its `assumptions`, and `partitions_outside_dump`;
`selected` slot (null when a record is missing) and its `assumptions`, and
`partitions_outside_dump`;
`running_partition` is always `unknown`. Partition rows
(`partitions/<label>.bin`) carry the declared `type`, `subtype`, names,
`flash_address`, `size` and flags; `encrypted` is a storage flag, not
+109 -52
View File
@@ -16,28 +16,45 @@ headers, segments, integrity and descriptors appear in the root
## Recognition
A file is a flash image when REA finds a partition table that ESP-IDF's
bootloader accepts (`esp_partition_table_verify`): 32-byte `0x50AA` entries,
at most one `0xEBEB` MD5 entry over the entries before it, and a terminating
entry within `0xC00` bytes. The table is located in this order:
A file is a flash image when REA finds a partition table in the entry
structure ESP-IDF's bootloader verifies (`esp_partition_table_verify`):
32-byte `0x50AA` entries, at most one `0xEBEB` MD5 entry over the entries
before it, and a terminating entry within `0xC00` bytes. ESP-IDF accepts a
table with no entries, but REA requires one so that erased flash is not a
table. A table whose MD5 does not match, or that has a second MD5 entry, is
still read, with `md5.status` `invalid` and a limitation, because the
bootloader would refuse it. The table is located in this order:
1. at the caller's `partition_table_offset`, when given; no scan follows;
2. at the default `0x8000`;
3. by scanning every `0x1000`-aligned flash sector for a table whose MD5
verifies. This location is derived, and the inventory says so.
Without a caller selection, the scan runs only when an ESP image sits at a
ROM bootloader offset, so ordinary files are not scanned. A C3 or S3 dump
begins with its bootloader, so flash layouts are detected before a file is
read as one ESP image. An image at another chip's ROM offset, such as an
ESP32 bootloader at the start of a dump read from `0x1000`, means the dump's
origin is not flash address 0. REA then does not scan, because a table found
that way would place every partition at the wrong file offset; pass
`flash_offset`.
A table never lies inside the located bootloader image, so the default
offset and the scan are tried only after the image's last segment. Without
a caller selection, the scan runs only when an ESP image sits at a ROM
bootloader offset, and a single app or bootloader image ends at its last
segment, so neither ordinary files nor single images are scanned. A C3 or
S3 dump begins with its bootloader, so flash layouts are detected before a
file is read as one ESP image.
Unless the caller names the table with `partition_table_offset`, REA
refuses two kinds of dump before looking for a table:
- an image of a known chip at another chip's ROM offset, such as an ESP32
bootloader at the start of a dump read from `0x1000`. The dump's origin
is not the flash address REA assumed, given or default, and a table found
there would place every partition at the wrong file offset; pass the
`flash_offset` where the dump starts.
- an ESP8266 image at a ROM offset. ESP8266 dumps can hold the same table
format, and they keep the [ESP image](esp-image.md) reader's ESP8266
refusal.
When no table is found the layout is not recognized. This is an unknown
outcome, not a malformed file: flash encryption, which encrypts the
bootloader and the table, is one cause.
bootloader and the table, is one cause. When a table is found without a
located bootloader and `flash_offset` is omitted, a limitation says that
each partition's file offset assumes the dump starts at flash `0x0`.
## Caller inputs
@@ -72,7 +89,8 @@ The root `format_metadata` (`esp-flash-image`) records:
- `apps`: ESP image facts for each app partition the dump holds;
- `otadata`: the OTA records and the derived selection;
- `partitions_outside_dump`: declared partitions with no bytes in the file;
- `undescribed`: byte ranges outside every region that are not erased flash.
- `undescribed`: ranges outside every region, each from a byte that is not
erased flash to the end of its gap.
The ROM loads the bootloader from `ESP_ROM_BOOTLOADER_OFFSET_FLASH`:
`0x1000` for ESP32 and ESP32-S2; `0x0` for ESP32-S3, C2, C3, C6, C61, H2 and
@@ -81,15 +99,25 @@ where its own `chip_id` says the ROM loads it.
### Partitions
Each partition entry is `partitions/<label>.bin` when the label is a safe,
unique name of letters, digits, `.`, `_` and `-` with at least one letter or
underscore, and `partitions/<index>.bin` otherwise; `source_path` keeps the
label. Its `format_metadata` (`esp-flash-partition`) records the table
`index`, `label` (null when not UTF-8), `type`, `subtype`, ESP-IDF's
`type_name` and `subtype_name`, `flash_address`, `size`, the `encrypted` and
`readonly` flags and any `unknown_flags`. The `encrypted` flag says how a
device with flash encryption stores the partition; it does not establish
that the dump's bytes are ciphertext.
Each partition entry is `partitions/<label>.bin` when the label is a safe
name of up to 16 letters, digits, `.`, `_` and `-` that starts with a
letter, digit or underscore and has at least one letter or underscore, and
no other label repeats it, ignoring case as some file systems do; it is
`partitions/<index>.bin` otherwise. `source_path` keeps the label, the
table's own name for the partition. Its `format_metadata`
(`esp-flash-partition`) records the table `index`, `label` (null when not
UTF-8), `label_hex` (the label's bytes when it is not UTF-8, else null),
`type`, `subtype`, ESP-IDF's `type_name` and `subtype_name`,
`flash_address`, `size`, the `encrypted` and `readonly` flags and any
`unknown_flags`. The `encrypted` flag says how a device with flash
encryption stores the partition; it does not establish that the dump's
bytes are ciphertext.
ESP-IDF 5.3 and later may declare primary `bootloader` and `partition_table`
entries over the bootloader region and the table sector. They are
partitions like any other, whose bytes are also `bootloader.bin` and
`partition-table.bin`, and they are exempt from the rule that partitions
follow the table, as in `gen_esp32part.py`.
A partition that the file holds only in part is the bytes the file holds,
with a limitation. One unreadable partition does not hide the others:
@@ -109,27 +137,44 @@ segments as entries.
### OTA data
The otadata partition holds two `esp_ota_select_entry_t` records, one at the
start of each 4 KiB sector. REA reports each record's `ota_seq`,
`seq_label`, `ota_state` and its name, `crc`, whether the CRC matches (CRC-32
of `ota_seq` seeded with `0xFFFFFFFF`) and whether the bootloader would treat
start of each 4 KiB sector. Like the bootloader, REA reads the last otadata
partition the table declares, and the last factory app and OTA app of each
subtype; a limitation names any repeated declaration. REA reports each
record the partition and the dump hold: its `ota_seq`, `seq_label`,
`ota_state` and its name, `crc`, whether the CRC matches (CRC-32 of
`ota_seq` seeded with `0xFFFFFFFF`) and whether the bootloader would treat
it as valid.
`selected` is derived as `bootloader_utility_get_selected_boot_partition`
does without app rollback or anti-rollback: the valid record with the higher
`ota_seq` selects slot `(ota_seq - 1) % ota_app_count`, indexed by OTA
subtype. With no usable record or no OTA app, the factory app is selected,
else `ota_0`. Its `assumptions` list what the derivation leaves out.
`running_partition` is always `unknown`: at boot the bootloader may first
mark a pending-verify record aborted, check the app's secure version against
eFuse, and fall back to another partition when an app fails to load.
does without app rollback or anti-rollback. `basis` names its branch:
- `otadata`: the valid record with the higher `ota_seq` (record 0 on a tie)
selects slot `(ota_seq - 1) % ota_app_count`, indexed by OTA subtype;
- `factory-fallback`: no record is usable, or the table has no OTA app, and
a factory app is declared; or records are set but none is valid, as after
a CRC mismatch, which selects the factory index whether or not a factory
app is declared;
- `ota_0-fallback`: no record is usable and no factory app is declared, so
the bootloader selects boot index 0;
- `none`: the otadata partition is smaller than two 4 KiB sectors, so the
bootloader cannot read it and resets without selecting an app.
`partition_index` is null when the table declares no partition for the
selected slot or fallback; the bootloader then tries the other apps.
`selected` is null when the dump lacks a record the bootloader reads. Its
`assumptions` list what the derivation leaves out, among them factory-reset
and test-app GPIO holds and a wake from deep sleep. `running_partition` is
always `unknown`: at boot the bootloader may first mark a pending-verify
record aborted, check the app's secure version against eFuse, and fall back
to another partition when an app fails to load.
## Coverage and extraction
Erased flash (`0xFF`) between regions is described; any other byte outside
the bootloader region, the table sector and the partitions is listed in
`undescribed` and makes coverage `unknown`. Coverage is also `unknown` when a
partition is truncated or outside the dump, or when no bootloader is
located.
Erased flash (`0xFF`) between regions is described. In a gap between
regions, the first byte that is not erased starts an `undescribed` range to
the end of the gap, and makes coverage `unknown`; the rest of that range is
not examined. Coverage is also `unknown` when a partition is truncated or
outside the dump, or when no bootloader is located.
`extract_artifact` writes every entry as an ordinary file, `nvs_keys`
partitions included. On a device without flash encryption an `nvs_keys`
@@ -139,31 +184,40 @@ NVS.
## Comparison
Artifact comparison uses each partition's `label`, `type`, `subtype`,
`flash_address`, `size` and flags as its declaration, so a partition that
moves or changes type with unchanged bytes is `changed` in `metadata`.
Partitions pair by path. The root facts derive from the dump's bytes, which
content comparison covers.
Artifact comparison uses each partition's `label`, `label_hex`, `type`,
`subtype`, `flash_address`, `size` and flags as its declaration, so a
partition that moves, changes type or flags, or is renamed with unchanged
bytes is `changed` in `metadata`. Partitions pair by path. The root facts
are not compared as a declaration: they derive from the dump's bytes, which
content comparison covers, and from the caller's `flash_offset` and
`partition_table_offset`, which are part of the target's identity and
decide each partition's byte range.
## Limits
- A flash dump inside another container, such as a ZIP, is classified from
its first bytes and so reads as an ESP image, not as a flash layout; extract
it first.
- ESP8266 dumps, Secure Boot v1 bootloader digests and flash-encrypted
layouts are not read.
- ESP8266 dumps are refused; Secure Boot v1 bootloader digests and
flash-encrypted layouts are not read.
- Partitions are not checked against the flash chip size, which the
bootloader also verifies.
- NVS, FAT, SPIFFS and other data partitions are extracted, not decoded.
- An image nested in a partition is decoded for its facts; its segments are
not inventory entries.
## Verification
Synthetic fixtures with computed MD5s and CRCs cover the table rules, every
way the table is located, the misplaced-origin guard, the ROM bootloader
offsets, overlaps, truncation and partitions outside the dump, index names
for unsafe labels, the otadata states and the bootloader's selection rules.
CLI and stdio MCP tests exercise the inputs, their identity, extraction and
comparison.
Synthetic fixtures with computed MD5s and CRCs, and one CRC otatool wrote,
cover the table rules, every way the table is located, the scan's chunk and
sector edges, the misplaced-origin guard with the origin omitted or given,
the ESP8266 refusal, the `0x0`, `0x1000` and `0x2000` ROM bootloader
offsets, single images that are not scanned, overlaps, primary entries,
truncation and partitions outside the dump, index names for unsafe labels,
repeated declarations, the otadata states and records, and each branch of
the bootloader's selection. CLI and stdio MCP tests exercise the inputs and
their identity and extraction, and an MCP `compare_artifacts` test compares
partition declarations, labels that are not UTF-8 included.
The optional real lane builds its dumps with esptool and ESP-IDF's own tools:
@@ -207,7 +261,10 @@ The lane does not establish:
- dumps read from a device: its dumps come from `merge-bin`;
- chips other than ESP32-C3 and ESP32;
- OTA states the bootloader writes itself, app rollback or anti-rollback;
- `ota_state` and record validity, which `read_otadata` does not print, or
OTA states the bootloader writes itself, app rollback or anti-rollback;
- selection with gaps or repeated OTA subtypes, which otatool does not
write;
- flash-encrypted dumps, which are not recognized by design.
It does not run in CI.
@@ -5,6 +5,7 @@ import { describe, expect, it, vi } from "vitest";
import { EspFlashImageArtifactReader } from "./EspFlashImageArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import type { ArtifactEntry } from "./ArtifactReader.js";
import { extractArtifact } from "./extraction/ArtifactExtraction.js";
import * as espImageDecoding from "../domain/firmware/espImage.js";
import {
@@ -51,6 +52,12 @@ const root = (
snapshot.occurrences.find(({ logical_path }) => logical_path === ".")
?.format_metadata;
const list = async (reader: EspFlashImageArtifactReader) => {
const entries: ArtifactEntry[] = [];
for await (const entry of reader.entries()) entries.push(entry);
return entries;
};
describe("ESP flash layout inventory", () => {
it("inventories the bootloader, table and partitions with app and otadata facts", async () => {
const snapshot = await scanCanonicalArtifactInventory(
@@ -219,6 +226,87 @@ describe("ESP flash layout coverage and selection", () => {
} finally {
await reader.close();
}
// The caller's selection decides before another container's signature.
const zipLike = await writeDump(
Buffer.concat([Buffer.from("PK\x03\x04"), pattern(0x1_0000, 3)]),
"dump.zip",
);
await expect(
scanCanonicalArtifactInventory(zipLike, { espFlashSelection: {} }),
).rejects.toThrow(/ESP flash image layout is not recognized/u);
});
});
describe("ESP flash layout reader identity", () => {
it("rejects a changed carrier during extraction preparation", async () => {
const reader = new EspFlashImageArtifactReader(await writeDump(DUMP));
try {
await expect(
reader.prepareContainer("0".repeat(64)),
).rejects.toMatchObject({ reason: "integrity" });
} finally {
await reader.close();
}
});
it("rejects forged entries and changed carrier identity, then closes idempotently", async () => {
const path = await writeDump(DUMP);
const reader = new EspFlashImageArtifactReader(path);
try {
const entry = (await list(reader)).find(
({ path: name }) => name === "partitions/nvs.bin",
);
if (entry === undefined) throw new Error("Missing nvs partition");
await expect(
reader.open({ ...entry, declaredSize: 1 }),
).rejects.toMatchObject({ reason: "integrity" });
await expect(
reader.open({ ...entry, byteOffset: 0 }),
).rejects.toMatchObject({ reason: "integrity" });
await expect(
reader.open({
...entry,
adapterKey: "esp-flash-image:partitions/x.bin",
}),
).rejects.toMatchObject({ reason: "integrity" });
await writeFile(path, DUMP.subarray(0, 0x1_0000));
await expect(reader.open(entry)).rejects.toMatchObject({
reason: "integrity",
});
} finally {
await reader.close();
await reader.close();
}
});
it("settles an active entry stream when closed", async () => {
const reader = new EspFlashImageArtifactReader(await writeDump(DUMP));
try {
const [entry] = await list(reader);
if (entry === undefined) throw new Error("Missing entry");
const stream = await reader.open(entry);
await reader.close();
expect(stream.destroyed).toBe(true);
await expect(
reader.entries()[Symbol.asyncIterator]().next(),
).rejects.toMatchObject({ reason: "unavailable" });
} finally {
await reader.close();
}
});
it("honors cancellation before admitting a file with the exact reason", async () => {
const reader = new EspFlashImageArtifactReader(await writeDump(DUMP));
const controller = new AbortController();
const reason = { stage: "before admission" };
controller.abort(reason);
try {
await expect(
reader.entries(controller.signal)[Symbol.asyncIterator]().next(),
).rejects.toMatchObject({ reason: "cancelled", cause: reason });
} finally {
await reader.close();
}
});
});
+18 -13
View File
@@ -265,15 +265,17 @@ export class EspFlashImageArtifactReader implements ArtifactReader {
context: EspFlashReadContext,
plan: EspFlashPlan,
): Promise<EspFlashLayoutFacts> {
const record = (position: number) =>
readEspFlashRange(context, position, ESP_OTA_SELECT_ENTRY_BYTES);
const otadataRecords =
plan.otadata === undefined
? undefined
: ([
await record(plan.otadata.records[0]),
await record(plan.otadata.records[1]),
] as const);
const otadataRecords: (Uint8Array | undefined)[] = [];
for (const { fileOffset, available } of plan.otadata?.records ?? [])
otadataRecords.push(
available
? await readEspFlashRange(
context,
fileOffset,
ESP_OTA_SELECT_ENTRY_BYTES,
)
: undefined,
);
const gapFirstUnerased: (number | undefined)[] = [];
for (const gap of plan.gaps) {
this.#assertOpen(context.signal);
@@ -426,10 +428,13 @@ const layoutMetadata = (
: {
...facts.otadata,
records: facts.otadata.records.map((record) => ({ ...record })),
selected: {
...facts.otadata.selected,
assumptions: [...facts.otadata.selected.assumptions],
},
selected:
facts.otadata.selected === null
? null
: {
...facts.otadata.selected,
assumptions: [...facts.otadata.selected.assumptions],
},
},
undescribed: [...facts.undescribed],
};
+2 -1
View File
@@ -99,8 +99,9 @@ export const decodeEspFlashImage = async (
region.start,
PROBE_BYTES,
);
// A region shorter than a header is erased when all of its bytes are.
if (
prefix.length >= HEADER_BYTES &&
prefix.length > 0 &&
prefix.subarray(0, HEADER_BYTES).every((byte) => byte === 0xff)
)
return noImage("erased", []);
+27 -25
View File
@@ -242,17 +242,6 @@ const classifyRootFormat = async (
const archiveBytes = (await handle.stat()).size;
const evidence = probe.finish(archiveBytes);
abortIfNeeded(signal);
const classificationPath = artifactNameForClassification(
selectedPath ?? path,
path,
);
const lower = classificationPath.toLowerCase();
if (hasZipSignature(evidence.prefix)) return zipRootPackageFormat(lower);
if (
lower.endsWith(".pkg") &&
evidence.prefix.subarray(0, 4).equals(Buffer.from("xar!"))
)
return "pkg";
// Fill each range: a short result is end of file, not a partial host read.
const read = async (position: number, length: number): Promise<Buffer> => {
const bytes = Buffer.alloc(length);
@@ -271,6 +260,32 @@ const classifyRootFormat = async (
}
return bytes.subarray(0, filled);
};
const locateFlash = (selection: EspFlashSelection, explicit: boolean) =>
locateEspFlashLayout(archiveBytes, read, selection, {
explicit,
...(signal === undefined ? {} : { signal }),
});
// The caller's format selection decides before any other signature.
if (espFlashSelection !== undefined) {
const selected = await locateFlash(espFlashSelection, true);
if (!selected.ok)
throw new ArtifactReaderFailure(
"unavailable",
`ESP flash image layout is not recognized: ${selected.error}`,
);
return { kind: "esp-flash-image", source, location: selected.value };
}
const classificationPath = artifactNameForClassification(
selectedPath ?? path,
path,
);
const lower = classificationPath.toLowerCase();
if (hasZipSignature(evidence.prefix)) return zipRootPackageFormat(lower);
if (
lower.endsWith(".pkg") &&
evidence.prefix.subarray(0, 4).equals(Buffer.from("xar!"))
)
return "pkg";
if (
lower.endsWith(".dmg") &&
(await hasDiskImageSignature(archiveBytes, read))
@@ -280,22 +295,9 @@ const classifyRootFormat = async (
if (pck !== undefined) return { kind: "godot-pck", source, location: pck };
// A C3 or S3 dump begins with its bootloader, so the layout is located
// before the bytes are read as one ESP image.
const flash = await locateEspFlashLayout(
archiveBytes,
read,
espFlashSelection ?? {},
{
explicit: espFlashSelection !== undefined,
...(signal === undefined ? {} : { signal }),
},
);
const flash = await locateFlash({}, false);
if (flash.ok)
return { kind: "esp-flash-image", source, location: flash.value };
if (espFlashSelection !== undefined)
throw new ArtifactReaderFailure(
"unavailable",
`ESP flash image layout is not recognized: ${flash.error}`,
);
const fatEvidence = await readRootFatSliceHeaders(handle, evidence, signal);
const mzWindowsHeaderSignature = await readRootMzWindowsHeaderSignature(
handle,
+5 -1
View File
@@ -33,8 +33,12 @@ export const snapshotSelectionOption = z
.optional()
.describe("Load and update a local analysis snapshot");
/** Explicit interpretation for headerless native executable formats. */
/**
* Explicit interpretation for headerless native executable formats. Flash
* dumps take their offsets on the artifact commands instead.
*/
export const formatSelectionOption = executableFormatHintSchema
.exclude(["esp-flash-image"])
.optional()
.describe(
"Explicit interpretation: dos-com or raw-binary (headerless), or gamecube-dol (GameCube/Wii DOL, never auto-detected); omission preserves header-based detection",
+33 -15
View File
@@ -274,6 +274,11 @@ export const espFlashPartitionMetadataSchema = z.strictObject({
index: uint8,
/** The label up to its first NUL; null when it is not UTF-8. */
label: z.string().nullable(),
/** The label's bytes up to its first NUL in hex, when it is not UTF-8. */
label_hex: z
.string()
.regex(/^(?:[0-9a-f]{2}){1,16}$/u)
.nullable(),
type: uint8,
subtype: uint8,
/** `gen_esp32part.py` names; null for values it does not name. */
@@ -295,6 +300,7 @@ export const espFlashPartitionDeclaration = (
metadata: EspFlashPartitionMetadata,
) => ({
label: metadata.label,
label_hex: metadata.label_hex,
type: metadata.type,
subtype: metadata.subtype,
flash_address: metadata.flash_address,
@@ -374,28 +380,40 @@ export const espFlashImageMetadataSchema = z.strictObject({
apps: z.array(
z.strictObject({ partition_index: uint8, facts: espFlashImageFactsSchema }),
),
/** The otadata partition the bootloader uses: the last one declared. */
otadata: z
.strictObject({
partition_index: uint8,
records: z.array(espOtaRecordSchema),
/** Derived from the records under the stated assumptions. */
selected: z.strictObject({
active_record: z.union([z.literal(0), z.literal(1)]).nullable(),
slot: uint8.nullable(),
partition_index: uint8.nullable(),
basis: z.enum([
"otadata",
"factory-fallback",
"ota_0-fallback",
"none",
]),
assumptions: z.array(z.string()),
}),
/** The records the dump holds, at most one per sector. */
records: z.array(espOtaRecordSchema).max(2),
/**
* Derived from both records under the stated assumptions; null when
* the dump lacks one. `partition_index` is null when the table
* declares no partition for the selected slot or fallback, and `none`
* is the bootloader's INVALID_INDEX, after which it resets.
*/
selected: z
.strictObject({
active_record: z.union([z.literal(0), z.literal(1)]).nullable(),
slot: uint8.nullable(),
partition_index: uint8.nullable(),
basis: z.enum([
"otadata",
"factory-fallback",
"ota_0-fallback",
"none",
]),
assumptions: z.array(z.string()),
})
.nullable(),
/** Rollback, anti-rollback and load fallback decide it at boot. */
running_partition: z.literal("unknown"),
})
.nullable(),
/** Bytes outside every described region that are not erased flash. */
/**
* Ranges outside every described region, each from a byte that is not
* erased flash to the end of its gap.
*/
undescribed: z.array(byteRange),
});
export type EspFlashImageMetadata = z.infer<typeof espFlashImageMetadataSchema>;
+52 -13
View File
@@ -3,8 +3,13 @@ import {
espChipForId,
type EspChip,
} from "./espChips.js";
import { recognizeEspImage } from "./espImage.js";
import { offsetText, u16 } from "./espImageStructures.js";
import { ESP_IMAGE_HEADER_BYTES, recognizeEspImage } from "./espImage.js";
import {
ESP_IMAGE_SEGMENT_HEADER_BYTES,
offsetText,
u16,
u32,
} from "./espImageStructures.js";
/** Bytes that recognition needs at a candidate bootloader offset. */
export const ESP_BOOTLOADER_PROBE_BYTES = 32;
@@ -27,22 +32,31 @@ export interface EspBootloaderLocation {
readonly chip: EspChip;
}
export interface EspBootloaderSearch {
readonly location: EspBootloaderLocation | undefined;
readonly limitations: readonly string[];
/** A known chip's image sits at another chip's ROM offset. */
readonly misplaced: boolean;
/** An ESP8266 image sits at a ROM offset. */
readonly esp8266: boolean;
}
/**
* The bootloader is the ESP image at the flash address its chip's ROM loads
* from (`ESP_ROM_BOOTLOADER_OFFSET_FLASH`). An image at another chip's ROM
* offset is reported, not located: the dump may not start where its flash
* offset says.
* from (`ESP_ROM_BOOTLOADER_OFFSET_FLASH`). Without one, an image of a known
* chip at another chip's ROM offset says the dump may not start where its
* flash offset says, and an ESP8266 image says it is an ESP8266 dump.
*/
export const locateEspBootloader = (
probes: readonly EspBootloaderProbe[],
): {
readonly location: EspBootloaderLocation | undefined;
readonly limitations: readonly string[];
} => {
): EspBootloaderSearch => {
const limitations: string[] = [];
let misplaced = false;
let esp8266 = false;
for (const probe of probes) {
if (recognizeEspImage(probe.prefix, probe.available) !== "esp-image")
continue;
const recognized = recognizeEspImage(probe.prefix, probe.available);
if (recognized === "esp8266") esp8266 = true;
if (recognized !== "esp-image") continue;
const chipId = u16(probe.prefix, 12);
const chip = espChipForId(chipId);
if (chip?.romBootloaderOffset === probe.flashAddress)
@@ -53,14 +67,39 @@ export const locateEspBootloader = (
chip,
},
limitations: [],
misplaced: false,
esp8266: false,
};
if (chip !== undefined) misplaced = true;
limitations.push(
chip === undefined
? `An ESP image at flash ${offsetText(probe.flashAddress)} has chip_id 0x${chipId.toString(16).padStart(4, "0")}, which is not in the chip table; it is not read as the bootloader.`
: `An ${chip.target} image is at flash ${offsetText(probe.flashAddress)}, but its ROM loads the bootloader from ${offsetText(chip.romBootloaderOffset)}; the dump may not start at the given flash offset.`,
: `An ${chip.target} image is at flash ${offsetText(probe.flashAddress)}, but its ROM loads the bootloader from ${offsetText(chip.romBootloaderOffset)}; the dump may not start at flash ${offsetText(probe.flashAddress - probe.fileOffset)}.`,
);
}
return { location: undefined, limitations };
return { location: undefined, limitations, misplaced, esp8266 };
};
/**
* The file offset just past the last segment of the image at `fileOffset`,
* found by following its segment headers, or the file size when they run
* past the end. No partition table lies before it.
*/
export const espImageSegmentsEnd = async (
read: (position: number, length: number) => Promise<Uint8Array>,
fileOffset: number,
fileSize: number,
): Promise<number> => {
const header = await read(fileOffset, ESP_IMAGE_HEADER_BYTES);
if (header.length < ESP_IMAGE_HEADER_BYTES) return fileSize;
let cursor = fileOffset + ESP_IMAGE_HEADER_BYTES;
for (let index = 0; index < (header[1] ?? 0); index += 1) {
const segment = await read(cursor, ESP_IMAGE_SEGMENT_HEADER_BYTES);
if (segment.length < ESP_IMAGE_SEGMENT_HEADER_BYTES) return fileSize;
cursor += ESP_IMAGE_SEGMENT_HEADER_BYTES + u32(segment, 4);
if (cursor >= fileSize) return fileSize;
}
return cursor;
};
/** The file offsets that hold each ROM bootloader offset, given the dump's origin. */
+116 -7
View File
@@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest";
import {
decodeEspPartitionTable,
espBootPartitions,
espPartitionSubtypeName,
espPartitionTypeName,
type EspPartition,
@@ -9,6 +10,7 @@ import {
import { decodeEspOtaRecord, selectEspOtaSlot } from "./espOtaData.js";
import {
espBootloaderProbeOffsets,
espImageSegmentsEnd,
locateEspBootloader,
} from "./espFlashBootloader.js";
import {
@@ -17,7 +19,11 @@ import {
partitionEntry,
partitionTable,
} from "./espFlash.fixture.js";
import { ESP32C3_BOOTLOADER_SEGMENTS, espImage } from "./espImage.fixture.js";
import {
ESP32C3_BOOTLOADER_SEGMENTS,
esp8266Image,
espImage,
} from "./espImage.fixture.js";
const table = (bytes: Uint8Array) => {
const decoded = decodeEspPartitionTable(bytes);
@@ -142,10 +148,37 @@ describe("ESP partition tables", () => {
).partitions;
expect(decoded).toMatchObject({
label: null,
// The source bytes stay comparable when the label is not UTF-8.
labelHex: "78ff",
encrypted: true,
unknownFlags: 0x8000_0000,
});
});
it("keeps the partitions the bootloader keeps: the last of each role", () => {
const [nvs, otadata, , factory, ota0] = TWO_OTA_PARTITIONS;
const boot = espBootPartitions(
table(
partitionTable([
...TWO_OTA_PARTITIONS,
{ ...factory!, label: "factory2", offset: 0x4_0000 },
{ ...otadata!, label: "otadata2", offset: 0x5_0000 },
{ ...ota0!, label: "again", offset: 0x6_0000 },
{ ...nvs!, label: "nvs2", offset: 0x7_0000 },
]),
).partitions,
);
expect(boot.factory?.label).toBe("factory2");
expect(boot.otadata?.label).toBe("otadata2");
// A same-subtype OTA entry replaces the slot but still counts.
expect(boot.otaSlots.get(0)?.label).toBe("again");
expect(boot.otaAppCount).toBe(3);
expect(boot.limitations).toEqual([
"The table declares otadata 2 times (partitions #1, #7); ESP-IDF's bootloader uses the last.",
"The table declares factory 2 times (partitions #3, #6); ESP-IDF's bootloader uses the last.",
"The table declares ota_0 2 times (partitions #4, #8); ESP-IDF's bootloader uses the last.",
]);
});
});
const partitions = (): readonly EspPartition[] =>
@@ -167,6 +200,10 @@ describe("ESP otadata selection", () => {
crc_matches: false,
valid: false,
});
// The CRC otatool v5.5.2 wrote for ota_seq 1 in the real-tool lane.
expect(
decodeEspOtaRecord(otaRecord({ seq: 1, crc: 0x4743_989a }), 0),
).toMatchObject({ crc_matches: true, valid: true });
expect(decodeEspOtaRecord(Buffer.alloc(32, 0xff), 1)).toMatchObject({
ota_seq: 0xffff_ffff,
state_name: "undefined",
@@ -174,10 +211,16 @@ describe("ESP otadata selection", () => {
});
});
const select = (first: Buffer, second: Buffer, parts = partitions()) =>
const select = (
first: Buffer,
second: Buffer,
parts = partitions(),
otadataSize = 0x2000,
) =>
selectEspOtaSlot(
[decodeEspOtaRecord(first, 0), decodeEspOtaRecord(second, 1)],
parts,
espBootPartitions(parts),
otadataSize,
);
const erased = Buffer.alloc(32, 0xff);
@@ -197,6 +240,12 @@ describe("ESP otadata selection", () => {
slot: 1,
partition_index: 5,
});
// ota_seq 0 is set; (0 - 1) wraps to 0xFFFFFFFF, and 0xFFFFFFFF % 2 is 1.
expect(select(otaRecord({ seq: 0 }), erased)).toMatchObject({
active_record: 0,
slot: 1,
partition_index: 5,
});
});
it("falls back as the bootloader does without usable records", () => {
@@ -211,11 +260,37 @@ describe("ESP otadata selection", () => {
slot: 0,
partition_index: 4,
});
// A set sequence with a bad CRC is not unusable, but no record is valid.
// Boot index 0 stands without an ota_0 partition.
const onlyOta1 = noFactory.filter(({ label }) => label !== "ota_0");
expect(select(erased, erased, onlyOta1)).toMatchObject({
basis: "ota_0-fallback",
slot: 0,
partition_index: null,
});
// A set sequence with a bad CRC is not unusable, but no record is valid:
// the factory index, with no factory partition to name.
expect(
select(otaRecord({ seq: 2, crc: 1 }), erased, noFactory),
).toMatchObject({ basis: "none", partition_index: null });
expect(select(otaRecord({ seq: 2 }), erased).assumptions).toHaveLength(3);
).toMatchObject({ basis: "factory-fallback", partition_index: null });
expect(select(otaRecord({ seq: 2 }), erased)?.assumptions).toHaveLength(4);
});
it("selects nothing from a short otadata partition and nothing known from one record", () => {
expect(
select(
otaRecord({ seq: 1 }),
otaRecord({ seq: 2 }),
partitions(),
0x1000,
),
).toMatchObject({ basis: "none", slot: null, partition_index: null });
expect(
selectEspOtaSlot(
[decodeEspOtaRecord(otaRecord({ seq: 1 }), 0)],
espBootPartitions(partitions()),
0x2000,
),
).toBeNull();
});
it("indexes OTA slots by subtype, so a missing subtype selects no partition", () => {
@@ -260,11 +335,45 @@ describe("ESP bootloader location", () => {
expect(locateEspBootloader([probe(0, esp32)])).toEqual({
location: undefined,
limitations: [
"An esp32 image is at flash 0x0, but its ROM loads the bootloader from 0x1000; the dump may not start at the given flash offset.",
"An esp32 image is at flash 0x0, but its ROM loads the bootloader from 0x1000; the dump may not start at flash 0x0.",
],
misplaced: true,
esp8266: false,
});
});
it("tells an unknown chip and an ESP8266 image from a misplaced origin", () => {
const unknown = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x7777,
});
expect(locateEspBootloader([probe(0, unknown)])).toMatchObject({
location: undefined,
misplaced: false,
esp8266: false,
});
expect(locateEspBootloader([probe(0, esp8266Image())])).toMatchObject({
location: undefined,
limitations: [],
misplaced: false,
esp8266: true,
});
});
it("finds where a located image's segments end by their headers", async () => {
const read = async (position: number, length: number) =>
bootloader.subarray(position, position + length);
const declared = ESP32C3_BOOTLOADER_SEGMENTS.reduce(
(end, { data }) => end + 8 + data.length,
24,
);
expect(await espImageSegmentsEnd(read, 0, bootloader.length)).toBe(
declared,
);
// Segments that run past the file end at the file.
expect(await espImageSegmentsEnd(read, 0, 40)).toBe(40);
});
it("probes only the ROM offsets the file holds after its flash offset", () => {
expect(espBootloaderProbeOffsets(0, 0x10_0000)).toEqual([
{ flashAddress: 0, fileOffset: 0 },
+118 -52
View File
@@ -2,14 +2,18 @@ import { offsetText } from "./espImageStructures.js";
import type { EspBootloaderLocation } from "./espFlashBootloader.js";
import {
ESP_PARTITION_TABLE_SECTOR_BYTES,
espBootPartitions,
espPartitionSubtypeName,
espPartitionTypeName,
isEspOtaData,
espPrimaryLayoutRole,
type EspBootPartitions,
type EspPartition,
type EspPartitionTable,
} from "./espPartitionTable.js";
import {
ESP_OTA_DATA_MIN_BYTES,
ESP_OTA_DATA_SECTOR_BYTES,
ESP_OTA_SELECT_ENTRY_BYTES,
decodeEspOtaRecord,
selectEspOtaSlot,
type EspOtaRecord,
@@ -46,6 +50,13 @@ export interface EspFlashPlanInput {
readonly table: EspPartitionTable;
}
/** One otadata record within its partition, and whether the file holds it. */
export interface EspOtaRecordPlan {
readonly index: 0 | 1;
readonly fileOffset: number;
readonly available: boolean;
}
/** Regions to inventory and the byte ranges the adapter must still read. */
export interface EspFlashPlan {
readonly input: EspFlashPlanInput;
@@ -55,11 +66,13 @@ export interface EspFlashPlan {
readonly offset: number;
readonly length: number;
}[];
/** File offsets of the two otadata records, when the file holds them. */
/** The partitions the bootloader keeps from the table. */
readonly boot: EspBootPartitions;
/** The otadata partition the bootloader uses and its records. */
readonly otadata:
| {
readonly partition: EspPartition;
readonly records: readonly [number, number];
readonly records: readonly EspOtaRecordPlan[];
}
| undefined;
readonly limitations: readonly string[];
@@ -68,21 +81,26 @@ export interface EspFlashPlan {
const SAFE_LABEL = /^(?=.*[A-Za-z_])[A-Za-z0-9_][A-Za-z0-9._-]{0,15}$/u;
/**
* Partition paths use the label when it is a safe, unique path segment
* with a letter or underscore, else the table index, which then cannot
* collide with a label.
* Partition paths use the label when it is a safe path segment with a
* letter or underscore that no other label repeats, ignoring case as some
* file systems do; else the table index, which cannot collide with a label.
*/
const partitionPaths = (
partitions: readonly EspPartition[],
): ReadonlyMap<number, string> => {
const counts = new Map<string, number>();
for (const { label } of partitions)
if (label !== null) counts.set(label, (counts.get(label) ?? 0) + 1);
if (label !== null) {
const folded = label.toLowerCase();
counts.set(folded, (counts.get(folded) ?? 0) + 1);
}
return new Map(
partitions.map((part) => {
const label = part.label;
const named =
label !== null && SAFE_LABEL.test(label) && counts.get(label) === 1;
label !== null &&
SAFE_LABEL.test(label) &&
counts.get(label.toLowerCase()) === 1;
return [
part.index,
`partitions/${named ? label : String(part.index)}.bin`,
@@ -104,6 +122,39 @@ const region = (
const rangeText = (address: number, length: number): string =>
`${offsetText(address)}-${offsetText(address + length)}`;
const partitionName = (part: EspPartition): string =>
part.label ?? `#${String(part.index)}`;
/** The records the otadata partition holds, each where the file holds it. */
const otadataPlan = (
input: EspFlashPlanInput,
partition: EspPartition,
limitations: string[],
): NonNullable<EspFlashPlan["otadata"]> => {
const name = partitionName(partition);
const small = partition.size < ESP_OTA_DATA_MIN_BYTES;
if (small)
limitations.push(
`The otadata partition ${name} is ${offsetText(partition.size)} bytes, smaller than the two 4 KiB sectors ESP-IDF's bootloader reads; the bootloader resets without selecting an app.`,
);
const records: EspOtaRecordPlan[] = [];
for (const index of [0, 1] as const) {
const within = index * ESP_OTA_DATA_SECTOR_BYTES;
// A record past the declared size belongs to whatever follows.
if (within + ESP_OTA_SELECT_ENTRY_BYTES > partition.size) continue;
const fileOffset = partition.offset + within - input.origin.value;
const available =
fileOffset >= 0 &&
fileOffset + ESP_OTA_SELECT_ENTRY_BYTES <= input.fileSize;
if (!available)
limitations.push(
`otadata record ${String(index)} of partition ${name} at flash ${offsetText(partition.offset + within)} is outside the dump${small ? "." : "; the OTA selection is unknown."}`,
);
records.push({ index, fileOffset, available });
}
return { partition, records };
};
/** Lay the bootloader, table and partitions over the dump. */
export const planEspFlashLayout = (input: EspFlashPlanInput): EspFlashPlan => {
const limitations: string[] = [...input.table.limitations];
@@ -145,8 +196,8 @@ export const planEspFlashLayout = (input: EspFlashPlanInput): EspFlashPlan => {
length: part.size,
partition: part,
});
const name = part.label ?? `#${String(part.index)}`;
if (part.offset < tableEnd)
const name = partitionName(part);
if (part.offset < tableEnd && espPrimaryLayoutRole(part) === undefined)
limitations.push(
`Partition ${name} at ${offsetText(part.offset)} starts before the end of the partition table at ${offsetText(tableEnd)}; gen_esp32part.py rejects such a table.`,
);
@@ -165,54 +216,59 @@ export const planEspFlashLayout = (input: EspFlashPlanInput): EspFlashPlan => {
if (placed.fileOffset >= 0 && placed.available > 0) regions.push(placed);
}
limitations.push(...overlaps(regions));
const otadataPartition = input.table.partitions.find(isEspOtaData);
const otadataStart =
otadataPartition === undefined
? undefined
: otadataPartition.offset - input.origin.value;
const otadata =
otadataPartition === undefined || otadataStart === undefined
? undefined
: otadataStart >= 0 &&
otadataStart + ESP_OTA_DATA_SECTOR_BYTES + 32 <= input.fileSize
? {
partition: otadataPartition,
records: [
otadataStart,
otadataStart + ESP_OTA_DATA_SECTOR_BYTES,
] as const,
}
: undefined;
if (otadataPartition !== undefined && otadata === undefined)
limitations.push(
`The dump does not hold both otadata records of partition ${otadataPartition.label ?? `#${String(otadataPartition.index)}`}; the OTA selection is unknown.`,
);
const boot = espBootPartitions(input.table.partitions);
limitations.push(...boot.limitations);
return {
input,
regions,
gaps: gapsBetween(regions, input.fileSize),
otadata,
boot,
otadata:
boot.otadata === undefined
? undefined
: otadataPlan(input, boot.otadata, limitations),
limitations,
};
};
const regionEnd = ({ flashAddress, length }: EspFlashRegion): number =>
flashAddress + length;
/** A primary bootloader or table entry declared over exactly its region. */
const declaresRegion = (
first: EspFlashRegion,
second: EspFlashRegion,
): boolean => {
const [region, entry] =
first.partition === undefined ? [first, second] : [second, first];
return (
region.partition === undefined &&
entry.partition !== undefined &&
espPrimaryLayoutRole(entry.partition) === region.role &&
region.flashAddress === entry.flashAddress &&
region.length === entry.length
);
};
/** Regions that share bytes, which gen_esp32part.py rejects for partitions. */
const overlaps = (regions: readonly EspFlashRegion[]): string[] => {
const ordered = [...regions].sort(
(left, right) => left.flashAddress - right.flashAddress,
);
const found: string[] = [];
for (let index = 1; index < ordered.length; index += 1) {
const previous = ordered[index - 1];
const current = ordered[index];
// The region reaching furthest so far, so a long one shows every overlap.
let furthest: EspFlashRegion | undefined;
for (const current of ordered) {
if (
previous !== undefined &&
current !== undefined &&
current.flashAddress < previous.flashAddress + previous.length
furthest !== undefined &&
current.flashAddress < regionEnd(furthest) &&
!declaresRegion(furthest, current)
)
found.push(
`${current.path} at ${offsetText(current.flashAddress)} overlaps ${previous.path}, which ends at ${offsetText(previous.flashAddress + previous.length)}.`,
`${current.path} at ${offsetText(current.flashAddress)} overlaps ${furthest.path}, which ends at ${offsetText(regionEnd(furthest))}.`,
);
if (furthest === undefined || regionEnd(current) > regionEnd(furthest))
furthest = current;
}
return found;
};
@@ -242,16 +298,18 @@ const gapsBetween = (
export interface EspFlashOtaFacts {
readonly partition_index: number;
/** The records the dump holds, each with its index. */
readonly records: readonly EspOtaRecord[];
/** Derived from the records; see its assumptions. */
readonly selected: EspOtaSelection;
/** Derived from the records; see its assumptions. Null when unknown. */
readonly selected: EspOtaSelection | null;
/** The bootloader's rollback and fallback decide it at boot. */
readonly running_partition: "unknown";
}
/** Facts the adapter read for the plan: the otadata records and the gaps. */
export interface EspFlashReads {
readonly otadataRecords: readonly [Uint8Array, Uint8Array] | undefined;
/** The bytes of each planned otadata record the file holds, in plan order. */
readonly otadataRecords: readonly (Uint8Array | undefined)[];
/** The first non-erased file offset in each gap, in plan order. */
readonly gapFirstUnerased: readonly (number | undefined)[];
}
@@ -281,12 +339,12 @@ export const finishEspFlashLayout = (
const total = undescribed.reduce((sum, { length }) => sum + length, 0);
if (undescribed.length > 0)
limitations.push(
`${String(total)} bytes of the dump, in ${String(undescribed.length)} range(s), lie outside the bootloader region, the partition table and every partition and are not erased flash.`,
`${String(undescribed.length)} range(s) of the dump, ${String(total)} bytes in all, lie outside the bootloader region, the partition table and every partition and are not described; each runs from a byte that is not erased flash to the end of its gap.`,
);
const otadata =
plan.otadata === undefined || reads.otadataRecords === undefined
plan.otadata === undefined
? null
: otaFacts(plan, reads.otadataRecords);
: otaFacts(plan, plan.otadata, reads.otadataRecords);
const incomplete = plan.input.table.partitions.some((part) => {
const region = plan.regions.find((item) => item.partition === part);
return region === undefined || region.available < part.size;
@@ -306,14 +364,21 @@ export const finishEspFlashLayout = (
const otaFacts = (
plan: EspFlashPlan,
[first, second]: readonly [Uint8Array, Uint8Array],
): EspFlashOtaFacts | null => {
if (plan.otadata === undefined) return null;
const records = [decodeEspOtaRecord(first, 0), decodeEspOtaRecord(second, 1)];
otadata: NonNullable<EspFlashPlan["otadata"]>,
bytes: readonly (Uint8Array | undefined)[],
): EspFlashOtaFacts => {
const records = otadata.records.flatMap(({ index, available }, position) => {
const read = bytes[position];
return available &&
read !== undefined &&
read.length >= ESP_OTA_SELECT_ENTRY_BYTES
? [decodeEspOtaRecord(read, index)]
: [];
});
return {
partition_index: plan.otadata.partition.index,
partition_index: otadata.partition.index,
records,
selected: selectEspOtaSlot(records, plan.input.table.partitions),
selected: selectEspOtaSlot(records, plan.boot, otadata.partition.size),
running_partition: "unknown",
};
};
@@ -323,6 +388,7 @@ export const espFlashPartitionFacts = (part: EspPartition) => ({
kind: "esp-flash-partition" as const,
index: part.index,
label: part.label,
label_hex: part.labelHex,
type: part.type,
subtype: part.subtype,
type_name: espPartitionTypeName(part.type),
+148 -21
View File
@@ -4,12 +4,18 @@ import {
locateEspFlashLayout,
type EspFlashSelection,
} from "./espFlashLocation.js";
import { ESP8266_REFUSAL } from "./espImage.js";
import {
TWO_OTA_PARTITIONS,
flashImage,
partitionTable,
} from "./espFlash.fixture.js";
import { ESP32C3_BOOTLOADER_SEGMENTS, espImage } from "./espImage.fixture.js";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
esp8266Image,
espImage,
} from "./espImage.fixture.js";
const c3Bootloader = espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS });
const esp32Bootloader = espImage({
@@ -21,11 +27,17 @@ const locate = (
bytes: Uint8Array,
selection: EspFlashSelection = {},
explicit = false,
reads: [number, number][] = [],
) =>
locateEspFlashLayout(
bytes.length,
async (position, length) =>
bytes.subarray(Math.max(0, position), Math.max(0, position) + length),
async (position, length) => {
reads.push([position, length]);
return bytes.subarray(
Math.max(0, position),
Math.max(0, position) + length,
);
},
selection,
{ explicit },
);
@@ -47,6 +59,15 @@ const dump = (
{ at: tableAt, bytes: partitionTable(TWO_OTA_PARTITIONS, { md5 }) },
]);
/** An ESP32 dump read from flash 0x1000: its bootloader is the file's first byte. */
const fromSecondSector = (tableAt: number) =>
dump(tableAt, { bootloader: esp32Bootloader, bootloaderAt: 0x1000 }).subarray(
0x1000,
);
const UNCONFIRMED =
"No bootloader at a ROM offset confirms that the dump starts at flash 0x0; each partition's file offset assumes it. Pass flash_offset if the dump starts elsewhere.";
describe("ESP flash layout location", () => {
it("finds the bootloader and the table at their default offsets", async () => {
const located = await locate(dump(0x8000));
@@ -60,6 +81,20 @@ describe("ESP flash layout location", () => {
});
});
it("locates a bootloader at the 0x2000 ROM offset of the chips that use it", async () => {
const c5 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x17,
});
const located = await locate(
dump(0x8000, { bootloader: c5, bootloaderAt: 0x2000 }),
);
expect(located.ok && located.value).toMatchObject({
bootloader: { flashAddress: 0x2000, chip: { target: "esp32c5" } },
tableMethod: "default",
});
});
it("scans for a relocated table only when its MD5 verifies", async () => {
const located = await locate(dump(0xa000));
expect(located.ok && located.value).toMatchObject({
@@ -69,10 +104,48 @@ describe("ESP flash layout location", () => {
expect(await locate(dump(0xa000, { md5: "absent" }))).toEqual({
ok: false,
error:
"no partition table at 0x8000 or in any 0x1000-aligned flash sector with a verified MD5; flash encryption is one cause",
"no partition table at 0x8000 or in any 0x1000-aligned flash sector after the bootloader image with a verified MD5; flash encryption is one cause",
});
});
it.each([0x1_0000, 0x1_1000])(
"scans across read chunks to a table at %s",
async (tableAt) => {
const located = await locate(dump(tableAt));
expect(located.ok && located.value).toMatchObject({
tableAddress: tableAt,
tableMethod: "scan",
});
},
);
it("scans the sectors of a dump whose origin is not sector-aligned", async () => {
// Flash 0x9000 is file offset 0x8800 when the dump starts at flash 0x800.
const bytes = flashImage(0x2_0000, [
{ at: 0x8800, bytes: partitionTable(TWO_OTA_PARTITIONS) },
]);
const located = await locate(bytes, { flashOffset: 0x800 }, true);
expect(located.ok && located.value).toMatchObject({
origin: { value: 0x800, source: "caller" },
tableAddress: 0x9000,
tableMethod: "scan",
limitations: [],
});
});
it("does not look for a table inside a single app image", async () => {
const app = espImage({
segments: [
...ESP32C3_APP_SEGMENTS,
{ address: 0x4038_0000, data: Buffer.alloc(0x2_0000, 0x5a) },
],
});
const reads: [number, number][] = [];
expect((await locate(app, {}, false, reads)).ok).toBe(false);
// Probes and segment headers only: no table read, no scan chunk.
expect(reads.every(([, length]) => length <= 32)).toBe(true);
});
it("uses the caller's table offset and does not scan", async () => {
const located = await locate(dump(0xa000), {
partitionTableOffset: 0xa000,
@@ -90,13 +163,22 @@ describe("ESP flash layout location", () => {
});
});
it("rejects a table offset outside the dump", async () => {
expect(
await locate(dump(0x8000), { partitionTableOffset: 0x10_0000 }),
).toEqual({
ok: false,
error:
"the partition table offset 0x100000 is outside the dump, which holds flash 0x0-0x20000",
});
});
});
describe("ESP flash layout origins", () => {
it("reads a dump that starts after flash address 0 at the caller's origin", async () => {
// An ESP32 dump read from 0x1000: its bootloader is the file's first byte.
const partial = dump(0x8000, {
bootloader: esp32Bootloader,
bootloaderAt: 0x1000,
}).subarray(0x1000);
const located = await locate(partial, { flashOffset: 0x1000 });
const located = await locate(fromSecondSector(0x8000), {
flashOffset: 0x1000,
});
expect(located.ok && located.value).toMatchObject({
origin: { value: 0x1000, source: "caller" },
bootloader: {
@@ -106,12 +188,35 @@ describe("ESP flash layout location", () => {
},
tableAddress: 0x8000,
tableMethod: "default",
limitations: [],
});
});
it.each([
["omitted", {}],
["given as 0x0", { flashOffset: 0 }],
] as const)(
"refuses an origin the bootloader contradicts when it is %s",
async (_name, selection) => {
// The table sits at flash 0x9000, so the default 0x8000 would find it
// at file 0x8000 and misplace every partition by 0x1000.
const refused = await locate(fromSecondSector(0x9000), selection);
expect(refused).toEqual({
ok: false,
error:
"An esp32 image is at flash 0x0, but its ROM loads the bootloader from 0x1000; the dump may not start at flash 0x0. Pass the flash_offset where the dump starts, or partition_table_offset",
});
},
);
it("finds the table after the bootloader once the origin is right", async () => {
const located = await locate(fromSecondSector(0x9000), {
flashOffset: 0x1000,
});
expect(located.ok && located.value).toMatchObject({
tableAddress: 0x9000,
tableMethod: "scan",
});
// Without the origin, the bootloader's chip says the dump is misplaced, so
// a scan would misplace every partition by 0x1000; none is attempted.
const misplaced = await locate(partial);
expect(misplaced.ok).toBe(false);
expect(!misplaced.ok && misplaced.error).toContain("Pass flash_offset");
});
it("does not scan arbitrary files without a bootloader unless the format was selected", async () => {
@@ -127,16 +232,38 @@ describe("ESP flash layout location", () => {
bootloader: undefined,
tableAddress: 0xa000,
tableMethod: "scan",
limitations: [UNCONFIRMED],
});
});
it("rejects a table offset outside the dump", async () => {
expect(
await locate(dump(0x8000), { partitionTableOffset: 0x10_0000 }),
).toEqual({
it("reads an unknown chip's dump at its default table without calling it misplaced", async () => {
const unknown = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x7777,
});
const located = await locate(dump(0x8000, { bootloader: unknown }));
expect(located.ok && located.value).toMatchObject({
bootloader: undefined,
tableMethod: "default",
limitations: [
"An ESP image at flash 0x0 has chip_id 0x7777, which is not in the chip table; it is not read as the bootloader.",
UNCONFIRMED,
],
});
});
it("refuses an ESP8266 dump as the ESP image reader does", async () => {
const esp8266 = flashImage(0x2_0000, [
{ at: 0, bytes: esp8266Image() },
{ at: 0x8000, bytes: partitionTable(TWO_OTA_PARTITIONS) },
]);
expect(await locate(esp8266)).toEqual({
ok: false,
error:
"the partition table offset 0x100000 is outside the dump, which holds flash 0x0-0x20000",
error: ESP8266_REFUSAL,
});
expect(await locate(esp8266, {}, true)).toEqual({
ok: false,
error: ESP8266_REFUSAL,
});
});
});
+73 -43
View File
@@ -2,6 +2,7 @@ import { err, ok, type Result } from "../result.js";
import {
ESP_BOOTLOADER_PROBE_BYTES,
espBootloaderProbeOffsets,
espImageSegmentsEnd,
locateEspBootloader,
type EspBootloaderLocation,
} from "./espFlashBootloader.js";
@@ -9,6 +10,7 @@ import type {
EspFlashOrigin,
EspPartitionTableMethod,
} from "./espFlashLayout.js";
import { ESP8266_REFUSAL } from "./espImage.js";
import { offsetText } from "./espImageStructures.js";
import {
ESP_PARTITION_TABLE_DEFAULT_OFFSET,
@@ -55,40 +57,40 @@ const tableAt = async (
);
/**
* The first sector-aligned table whose MD5 verifies; requiring the MD5
* keeps arbitrary bytes that begin with the magic from passing.
* The first table whose MD5 verifies at a flash sector boundary from the
* file offset `start`; requiring the MD5 keeps arbitrary bytes that begin
* with the magic from passing. Chunks start at sector boundaries, so each
* holds whole sector heads.
*/
const scanForTable = async (
read: EspFlashRead,
fileSize: number,
origin: number,
{
fileSize,
origin,
start,
}: { fileSize: number; origin: number; start: number },
signal?: AbortSignal,
): Promise<{ address: number; table: EspPartitionTable } | undefined> => {
const skew =
(ESP_PARTITION_TABLE_SECTOR_BYTES -
(origin % ESP_PARTITION_TABLE_SECTOR_BYTES)) %
ESP_PARTITION_TABLE_SECTOR_BYTES;
for (let chunk = 0; chunk < fileSize; chunk += SCAN_CHUNK_BYTES) {
signal?.throwIfAborted();
// One extra byte lets a sector head at the chunk's last byte be checked.
const bytes = await read(chunk, SCAN_CHUNK_BYTES + 1);
const first =
chunk +
((skew -
(chunk % ESP_PARTITION_TABLE_SECTOR_BYTES) +
ESP_PARTITION_TABLE_SECTOR_BYTES) %
ESP_PARTITION_TABLE_SECTOR_BYTES);
for (
let position = first;
position < chunk + SCAN_CHUNK_BYTES && position < fileSize;
position += ESP_PARTITION_TABLE_SECTOR_BYTES
) {
const at = position - chunk;
if (bytes[at] !== MAGIC_LOW || bytes[at + 1] !== MAGIC_HIGH) continue;
const decoded = await tableAt(read, position);
if (decoded.ok && decoded.value.md5.status === "valid")
return { address: origin + position, table: decoded.value };
const sector = ESP_PARTITION_TABLE_SECTOR_BYTES;
// The file offset of the first sector boundary at or after `start`.
const boundary = (sector - (origin % sector)) % sector;
let position =
start <= boundary
? boundary
: boundary + Math.ceil((start - boundary) / sector) * sector;
let chunkStart = position;
let bytes: Uint8Array = new Uint8Array(0);
for (; position < fileSize; position += sector) {
if (position - chunkStart + 2 > bytes.length) {
signal?.throwIfAborted();
chunkStart = position;
bytes = await read(position, SCAN_CHUNK_BYTES);
}
const at = position - chunkStart;
if (bytes[at] !== MAGIC_LOW || bytes[at + 1] !== MAGIC_HIGH) continue;
const decoded = await tableAt(read, position);
if (decoded.ok && decoded.value.md5.status === "valid")
return { address: origin + position, table: decoded.value };
}
return undefined;
};
@@ -96,10 +98,12 @@ const scanForTable = async (
/**
* Locate the bootloader and partition table: the caller's table offset
* when given, else the default `0x8000`, else a scan of 0x1000-aligned
* flash sectors for an MD5-verified table (a derived location). The scan
* runs only when a bootloader sits at its chip's ROM offset or the caller
* selected this format, and never when a bootloader at another chip's ROM
* offset says the dump's origin is wrong.
* flash sectors for an MD5-verified table (a derived location). A table
* never lies inside the located bootloader image. The scan runs only when a
* bootloader sits at its chip's ROM offset or the caller selected this
* format. Without a caller table offset, an ESP8266 image at a ROM offset
* refuses the dump, and a known chip's image at another chip's ROM offset
* refuses it as read from the wrong flash offset, given or default.
*/
export const locateEspFlashLayout = async (
fileSize: number,
@@ -125,6 +129,15 @@ export const locateEspFlashLayout = async (
),
);
const bootloader = locateEspBootloader(probes);
// Without a bootloader, nothing in the dump confirms a default origin.
const unconfirmed =
bootloader.location === undefined &&
!bootloader.misplaced &&
origin.source === "default"
? [
`No bootloader at a ROM offset confirms that the dump starts at flash ${offsetText(origin.value)}; each partition's file offset assumes it. Pass flash_offset if the dump starts elsewhere.`,
]
: [];
const located = (
tableAddress: number,
tableMethod: EspPartitionTableMethod,
@@ -136,7 +149,7 @@ export const locateEspFlashLayout = async (
tableAddress,
tableMethod,
table,
limitations: bootloader.limitations,
limitations: [...bootloader.limitations, ...unconfirmed],
});
const fileOffsetOf = (address: number): number => address - origin.value;
if (selection.partitionTableOffset !== undefined) {
@@ -152,8 +165,24 @@ export const locateEspFlashLayout = async (
`no partition table at the given offset ${offsetText(address)}: ${decoded.error}`,
);
}
if (bootloader.location === undefined && bootloader.esp8266)
return err(ESP8266_REFUSAL);
// A table found at an origin the bootloader contradicts would place every
// partition at the wrong file offset.
if (bootloader.location === undefined && bootloader.misplaced)
return err(
`${bootloader.limitations.join(" ")} Pass the flash_offset where the dump starts, or partition_table_offset`,
);
const imageEnd =
bootloader.location === undefined
? 0
: await espImageSegmentsEnd(
read,
bootloader.location.fileOffset,
fileSize,
);
const defaultOffset = fileOffsetOf(ESP_PARTITION_TABLE_DEFAULT_OFFSET);
if (defaultOffset >= 0 && defaultOffset < fileSize) {
if (defaultOffset >= imageEnd && defaultOffset < fileSize) {
const decoded = await tableAt(read, defaultOffset);
if (decoded.ok)
return located(
@@ -162,20 +191,21 @@ export const locateEspFlashLayout = async (
decoded.value,
);
}
const misplaced =
bootloader.location === undefined && bootloader.limitations.length > 0;
if (misplaced && selection.flashOffset === undefined)
return err(
`no partition table at ${offsetText(ESP_PARTITION_TABLE_DEFAULT_OFFSET)}, and ${bootloader.limitations.join(" ")} Pass flash_offset to read the dump from another flash address`,
);
if (bootloader.location === undefined && !explicit)
return err(
`no partition table at ${offsetText(ESP_PARTITION_TABLE_DEFAULT_OFFSET)} and no bootloader at a ROM offset`,
[
`no partition table at ${offsetText(ESP_PARTITION_TABLE_DEFAULT_OFFSET)} and no bootloader at a ROM offset`,
...bootloader.limitations,
].join("; "),
);
const scanned = await scanForTable(read, fileSize, origin.value, signal);
const scanned = await scanForTable(
read,
{ fileSize, origin: origin.value, start: imageEnd },
signal,
);
return scanned === undefined
? err(
`no partition table at ${offsetText(ESP_PARTITION_TABLE_DEFAULT_OFFSET)} or in any 0x1000-aligned flash sector with a verified MD5; flash encryption is one cause`,
`no partition table at ${offsetText(ESP_PARTITION_TABLE_DEFAULT_OFFSET)} or in any 0x1000-aligned flash sector${bootloader.location === undefined ? "" : " after the bootloader image"} with a verified MD5; flash encryption is one cause`,
)
: located(scanned.address, "scan", scanned.table);
};
+114 -8
View File
@@ -60,7 +60,10 @@ describe("ESP flash layout plans", () => {
["partitions/ota_1.bin", 0x3_0000, 0x1_0000, 0x1_0000],
]);
expect(plan.gaps).toEqual([]);
expect(plan.otadata?.records).toEqual([0xd000, 0xe000]);
expect(plan.otadata?.records).toEqual([
{ index: 0, fileOffset: 0xd000, available: true },
{ index: 1, fileOffset: 0xe000, available: true },
]);
expect(plan.limitations).toEqual([]);
});
@@ -70,6 +73,9 @@ describe("ESP flash layout plans", () => {
{ label: "dup", type: 1, subtype: 2, offset: 0xa000, size: 0x1000 },
{ label: "dup", type: 1, subtype: 2, offset: 0xb000, size: 0x1000 },
{ label: "7", type: 1, subtype: 2, offset: 0xc000, size: 0x1000 },
// Case-insensitive file systems would extract these to one file.
{ label: "Data", type: 1, subtype: 2, offset: 0xd000, size: 0x1000 },
{ label: "data", type: 1, subtype: 2, offset: 0xe000, size: 0x1000 },
];
expect(
planEspFlashLayout(input({ table: table(parts) }))
@@ -80,6 +86,39 @@ describe("ESP flash layout plans", () => {
"partitions/1.bin",
"partitions/2.bin",
"partitions/3.bin",
"partitions/4.bin",
"partitions/5.bin",
]);
});
it("accepts the primary bootloader and table entries ESP-IDF 5.3 declares", () => {
const plan = planEspFlashLayout(
input({
table: table([
{
label: "PrimaryBTLDR",
type: 2,
subtype: 0,
offset: 0,
size: 0x8000,
},
{
label: "PrimaryPrtTable",
type: 3,
subtype: 0,
offset: 0x8000,
size: 0x1000,
},
...TWO_OTA_PARTITIONS,
]),
}),
);
expect(plan.limitations).toEqual([]);
expect(plan.regions.map(({ path }) => path).slice(0, 4)).toEqual([
"bootloader.bin",
"partition-table.bin",
"partitions/PrimaryBTLDR.bin",
"partitions/PrimaryPrtTable.bin",
]);
});
@@ -133,27 +172,93 @@ describe("ESP flash layout plans", () => {
"Partition gone at 0x100000-0x101000 lies outside the dump, which ends at flash 0x39000.",
]);
const facts = finishEspFlashLayout(plan, {
otadataRecords: undefined,
otadataRecords: [],
gapFirstUnerased: erasedGaps(plan.gaps.length),
});
expect(facts.coverage).toBe("unknown");
});
it("reports overlapping regions", () => {
it("reports every region inside a longer one", () => {
const plan = planEspFlashLayout(
input({
table: table([
{ label: "a", type: 1, subtype: 2, offset: 0x9000, size: 0x2000 },
{ label: "a", type: 1, subtype: 2, offset: 0x9000, size: 0x8000 },
{ label: "b", type: 1, subtype: 2, offset: 0xa000, size: 0x1000 },
{ label: "c", type: 1, subtype: 2, offset: 0xc000, size: 0x1000 },
]),
}),
);
expect(plan.limitations).toEqual([
"partitions/b.bin at 0xa000 overlaps partitions/a.bin, which ends at 0xb000.",
"partitions/b.bin at 0xa000 overlaps partitions/a.bin, which ends at 0x11000.",
"partitions/c.bin at 0xc000 overlaps partitions/a.bin, which ends at 0x11000.",
]);
});
});
describe("ESP flash layout otadata", () => {
const [, otadata] = TWO_OTA_PARTITIONS;
it("reads the otadata partition the bootloader uses: the last declared", () => {
const plan = planEspFlashLayout(
input({
table: table([
...TWO_OTA_PARTITIONS,
{ ...otadata!, label: "otadata2", offset: 0x3_0000 + 0x1_0000 },
]),
fileSize: 0x5_0000,
}),
);
expect(plan.otadata?.partition.label).toBe("otadata2");
expect(plan.otadata?.records.map(({ fileOffset }) => fileOffset)).toEqual([
0x4_0000, 0x4_1000,
]);
expect(plan.limitations).toContain(
"The table declares otadata 2 times (partitions #1, #6); ESP-IDF's bootloader uses the last.",
);
});
it("does not read a record past a short otadata partition and selects none", () => {
const short = TWO_OTA_PARTITIONS.map((part) =>
part.label === "otadata" ? { ...part, size: 0x1000 } : part,
);
const plan = planEspFlashLayout(input({ table: table(short) }));
expect(plan.otadata?.records).toEqual([
{ index: 0, fileOffset: 0xd000, available: true },
]);
expect(plan.limitations).toContain(
"The otadata partition otadata is 0x1000 bytes, smaller than the two 4 KiB sectors ESP-IDF's bootloader reads; the bootloader resets without selecting an app.",
);
const facts = finishEspFlashLayout(plan, {
otadataRecords: [otaRecord({ seq: 1 })],
gapFirstUnerased: erasedGaps(plan.gaps.length),
});
expect(facts.otadata).toMatchObject({
records: [{ index: 0, ota_seq: 1 }],
selected: { basis: "none", partition_index: null },
});
});
it("keeps the record a dump holds when it ends between the two", () => {
const plan = planEspFlashLayout(input({ fileSize: 0xe000 }));
expect(plan.otadata?.records).toEqual([
{ index: 0, fileOffset: 0xd000, available: true },
{ index: 1, fileOffset: 0xe000, available: false },
]);
expect(plan.limitations).toContain(
"otadata record 1 of partition otadata at flash 0xe000 is outside the dump; the OTA selection is unknown.",
);
const facts = finishEspFlashLayout(plan, {
otadataRecords: [otaRecord({ seq: 3 }), undefined],
gapFirstUnerased: erasedGaps(plan.gaps.length),
});
expect(facts.otadata).toMatchObject({
records: [{ index: 0, ota_seq: 3, valid: true }],
selected: null,
running_partition: "unknown",
});
});
});
describe("ESP flash layout facts", () => {
it("describes erased gaps and reports the rest as undescribed", () => {
const plan = planEspFlashLayout(
@@ -164,18 +269,18 @@ describe("ESP flash layout facts", () => {
);
expect(plan.gaps).toEqual([{ offset: 0x1_0000, length: 0x1_0000 }]);
const erased = finishEspFlashLayout(plan, {
otadataRecords: undefined,
otadataRecords: [],
gapFirstUnerased: [undefined],
});
expect(erased).toMatchObject({ coverage: "complete", undescribed: [] });
const written = finishEspFlashLayout(plan, {
otadataRecords: undefined,
otadataRecords: [],
gapFirstUnerased: [0x1_8000],
});
expect(written.undescribed).toEqual([{ offset: 0x1_8000, length: 0x8000 }]);
expect(written.coverage).toBe("unknown");
expect(written.limitations.at(-1)).toBe(
"32768 bytes of the dump, in 1 range(s), lie outside the bootloader region, the partition table and every partition and are not erased flash.",
"1 range(s) of the dump, 32768 bytes in all, lie outside the bootloader region, the partition table and every partition and are not described; each runs from a byte that is not erased flash to the end of its gap.",
);
});
@@ -205,6 +310,7 @@ describe("ESP flash layout facts", () => {
kind: "esp-flash-partition",
index: 0,
label: "nvs",
label_hex: null,
type: 1,
subtype: 2,
type_name: "data",
+8 -3
View File
@@ -56,11 +56,16 @@ const allZero = (bytes: Uint8Array): boolean =>
const UTF8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
/** A NUL-terminated C string in a fixed array, or null when not UTF-8. */
const cString = (bytes: Uint8Array): string | null => {
/** The bytes of a NUL-terminated C string in a fixed array. */
export const cStringBytes = (bytes: Uint8Array): Uint8Array => {
const end = bytes.indexOf(0);
return end < 0 ? bytes : bytes.subarray(0, end);
};
/** A NUL-terminated C string in a fixed array, or null when not UTF-8. */
export const cString = (bytes: Uint8Array): string | null => {
try {
return UTF8.decode(end < 0 ? bytes : bytes.subarray(0, end));
return UTF8.decode(cStringBytes(bytes));
} catch {
return null;
}
+42 -54
View File
@@ -1,11 +1,7 @@
import { crc32 } from "node:zlib";
import { hex, u32 } from "./espImageStructures.js";
import {
espOtaAppSlot,
isEspFactoryApp,
type EspPartition,
} from "./espPartitionTable.js";
import type { EspBootPartitions } from "./espPartitionTable.js";
/*
* OTA data, from ESP-IDF v6.1 `esp_flash_partitions.h`
@@ -15,6 +11,8 @@ import {
*/
/** The otadata partition holds one record at the start of each of two sectors. */
export const ESP_OTA_DATA_SECTOR_BYTES = 0x1000;
/** `read_otadata` refuses a smaller otadata partition. */
export const ESP_OTA_DATA_MIN_BYTES = 2 * ESP_OTA_DATA_SECTOR_BYTES;
export const ESP_OTA_SELECT_ENTRY_BYTES = 32;
const UNSET = 0xffff_ffff;
@@ -73,7 +71,8 @@ export const decodeEspOtaRecord = (
export const ESP_OTA_SELECTION_ASSUMPTIONS = [
"Records are used as written: the bootloader may first mark a pending-verify record aborted when app rollback is enabled.",
"Anti-rollback, which compares the app's secure_version with eFuse, is not applied.",
"The selected app is not checked for a loadable image; the bootloader falls back to other partitions when it fails to load.",
"The selected app is not checked for a loadable image; the bootloader falls back to other partitions when it fails to load or is not declared.",
"Factory-reset and test-app GPIO holds and a wake from deep sleep, which can override the selection, are not applied.",
] as const;
export interface EspOtaSelection {
@@ -81,8 +80,9 @@ export interface EspOtaSelection {
readonly active_record: 0 | 1 | null;
/** `(ota_seq - 1) % ota_app_count`, the bootloader's slot arithmetic. */
readonly slot: number | null;
/** The table index of the app partition for that slot or the fallback. */
/** The table index of the selected app; null when none is declared. */
readonly partition_index: number | null;
/** The bootloader's branch; `none` is its INVALID_INDEX, which resets. */
readonly basis: "otadata" | "factory-fallback" | "ota_0-fallback" | "none";
readonly assumptions: readonly string[];
}
@@ -92,58 +92,48 @@ const unusable = ({ ota_seq, ota_state }: EspOtaRecord): boolean =>
ota_seq === UNSET || ota_state === 0x3 || ota_state === 0x4;
/**
* Derive the app partition the bootloader would select from these records,
* as `bootloader_utility_get_selected_boot_partition` does without app
* Derive the app partition the bootloader would select, as
* `bootloader_utility_get_selected_boot_partition` does without app
* rollback or anti-rollback:
* - both records unusable, or no OTA app: the factory app, else `ota_0`;
* - an otadata partition smaller than two sectors: none, because
* `read_otadata` fails and the bootloader resets;
* - both records unusable, or no OTA app: the factory app when one is
* declared, else boot index 0 (`ota_0`);
* - otherwise the valid record with the higher `ota_seq` (record 0 on a
* tie), whose slot is `(ota_seq - 1) % ota_app_count`;
* - no valid record (a CRC mismatch): the factory app, else none, because
* the bootloader then tries every partition.
* - no valid record, as after a CRC mismatch: the factory index, whether
* or not a factory app is declared.
* Unknown (null) when the dump lacks a record the bootloader would read.
*/
export const selectEspOtaSlot = (
records: readonly EspOtaRecord[],
partitions: readonly EspPartition[],
): EspOtaSelection => {
// The bootloader counts every OTA app but indexes them by subtype, and a
// later entry with the same subtype replaces an earlier one.
let appCount = 0;
const bySlot = new Map<number, EspPartition>();
for (const part of partitions) {
const slot = espOtaAppSlot(part);
if (slot === null) continue;
appCount += 1;
bySlot.set(slot, part);
}
const factory = partitions.find(isEspFactoryApp);
boot: EspBootPartitions,
otadataSize: number,
): EspOtaSelection | null => {
const assumptions = [...ESP_OTA_SELECTION_ASSUMPTIONS];
const none = {
const selection = (
basis: EspOtaSelection["basis"],
fields: Partial<Omit<EspOtaSelection, "basis" | "assumptions">> = {},
): EspOtaSelection => ({
active_record: null,
slot: null,
partition_index: null,
basis: "none",
...fields,
basis,
assumptions,
} as const;
const factoryFallback =
factory === undefined
? undefined
: ({
...none,
partition_index: factory.index,
basis: "factory-fallback",
} as const);
if (records.every(unusable) || appCount === 0) {
if (factoryFallback !== undefined) return factoryFallback;
const first = bySlot.get(0);
return first === undefined
? none
: {
...none,
});
const factory = selection("factory-fallback", {
partition_index: boot.factory?.index ?? null,
});
if (otadataSize < ESP_OTA_DATA_MIN_BYTES) return selection("none");
if (records.length < 2) return null;
if (records.every(unusable) || boot.otaAppCount === 0)
return boot.factory !== undefined
? factory
: selection("ota_0-fallback", {
slot: 0,
partition_index: first.index,
basis: "ota_0-fallback",
};
}
partition_index: boot.otaSlots.get(0)?.index ?? null,
});
const valid = records.filter(({ valid }) => valid);
const [first, second] = valid;
const active =
@@ -152,14 +142,12 @@ export const selectEspOtaSlot = (
second.ota_seq > first.ota_seq
? second
: first;
if (active === undefined) return factoryFallback ?? none;
const slot = ((active.ota_seq - 1) >>> 0) % appCount;
return {
if (active === undefined) return factory;
const slot = ((active.ota_seq - 1) >>> 0) % boot.otaAppCount;
return selection("otadata", {
active_record: active.index,
slot,
// A gap in the OTA subtypes selects a slot with no partition.
partition_index: bySlot.get(slot)?.index ?? null,
basis: "otadata",
assumptions,
};
partition_index: boot.otaSlots.get(slot)?.index ?? null,
});
};
+93 -21
View File
@@ -1,7 +1,14 @@
import { createHash } from "node:crypto";
import { err, ok, type Result } from "../result.js";
import { hex, offsetText, u16, u32 } from "./espImageStructures.js";
import {
cString,
cStringBytes,
hex,
offsetText,
u16,
u32,
} from "./espImageStructures.js";
/*
* The partition table, from ESP-IDF v6.1 `esp_flash_partitions.h`,
@@ -36,6 +43,8 @@ export interface EspPartition {
readonly size: number;
/** The label up to its first NUL, or null when it is not UTF-8. */
readonly label: string | null;
/** The label's bytes up to its first NUL, in hex, when it is not UTF-8. */
readonly labelHex: string | null;
readonly encrypted: boolean;
readonly readonly: boolean;
/** Flag bits other than `encrypted` and `readonly`, which ESP-IDF ignores. */
@@ -57,23 +66,14 @@ export interface EspPartitionTable {
readonly limitations: readonly string[];
}
const UTF8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
const label = (bytes: Uint8Array): string | null => {
const end = bytes.indexOf(0);
try {
return UTF8.decode(end < 0 ? bytes : bytes.subarray(0, end));
} catch {
return null;
}
};
const partition = (
entry: Uint8Array,
index: number,
entryOffset: number,
): EspPartition => {
const flags = u32(entry, 28);
const labelBytes = entry.subarray(12, 28);
const label = cString(labelBytes);
return {
index,
entryOffset,
@@ -81,7 +81,8 @@ const partition = (
subtype: entry[3] ?? 0,
offset: u32(entry, 4),
size: u32(entry, 8),
label: label(entry.subarray(12, 28)),
label,
labelHex: label === null ? hex(cStringBytes(labelBytes)) : null,
encrypted: (flags & 1) !== 0,
readonly: (flags & 2) !== 0,
unknownFlags: (flags & ~KNOWN_FLAGS) >>> 0,
@@ -89,12 +90,14 @@ const partition = (
};
/**
* Decode a table as ESP-IDF's bootloader verifies it: 32-byte entries with
* the `0x50AA` magic, at most one `0xEBEB` MD5 entry over the entries before
* it, and a terminator whose magic, type and subtype are all `0xFF`, which
* cannot be the first entry. A structural fault means the bytes are not a
* table; an MD5 mismatch, which the bootloader also rejects, is reported as
* an integrity fact so that the declared entries stay visible.
* Decode a table in the entry structure ESP-IDF's bootloader verifies:
* 32-byte entries with the `0x50AA` magic, at most one `0xEBEB` MD5 entry
* over the entries before it, and a terminator whose magic, type and subtype
* are all `0xFF`. A structural fault means the bytes are not a table. An MD5
* mismatch or a second MD5 entry, which the bootloader rejects, is reported
* as an integrity fact so that the declared entries stay visible. ESP-IDF
* accepts an empty table, but REA requires one entry before the terminator
* so that erased flash is not read as a table.
*/
export const decodeEspPartitionTable = (
bytes: Uint8Array,
@@ -214,6 +217,9 @@ export const espPartitionSubtypeName = (
const APP_TYPE = 0x00;
const OTA_DATA_SUBTYPE = 0x00;
const DATA_TYPE = 0x01;
const BOOTLOADER_TYPE = 0x02;
const PARTITION_TABLE_TYPE = 0x03;
const PRIMARY_SUBTYPE = 0x00;
const OTA_APP_SUBTYPE_FLAG = 0x10;
const OTA_APP_SUBTYPE_MASK = 0x0f;
@@ -224,11 +230,77 @@ export const espOtaAppSlot = (part: EspPartition): number | null =>
? part.subtype & OTA_APP_SUBTYPE_MASK
: null;
export const isEspFactoryApp = (part: EspPartition): boolean =>
const isEspFactoryApp = (part: EspPartition): boolean =>
part.type === APP_TYPE && part.subtype === 0x00;
export const isEspOtaData = (part: EspPartition): boolean =>
const isEspOtaData = (part: EspPartition): boolean =>
part.type === DATA_TYPE && part.subtype === OTA_DATA_SUBTYPE;
export const isEspAppPartition = (part: EspPartition): boolean =>
part.type === APP_TYPE;
/**
* The region a primary `bootloader` or `partition_table` entry declares.
* ESP-IDF 5.3 and later may declare them over the bootloader region and the
* table sector, and `gen_esp32part.py` exempts them from its rule that
* partitions follow the table.
*/
export const espPrimaryLayoutRole = (
part: EspPartition,
): "bootloader" | "partition-table" | undefined => {
if (part.subtype !== PRIMARY_SUBTYPE) return undefined;
if (part.type === BOOTLOADER_TYPE) return "bootloader";
return part.type === PARTITION_TABLE_TYPE ? "partition-table" : undefined;
};
/** The partitions the bootloader keeps from a table, and the OTA app count. */
export interface EspBootPartitions {
readonly factory: EspPartition | undefined;
readonly otadata: EspPartition | undefined;
/** Each OTA slot's partition, indexed by subtype. */
readonly otaSlots: ReadonlyMap<number, EspPartition>;
/** Every OTA app entry, duplicates included, as `bs->app_count` counts. */
readonly otaAppCount: number;
readonly limitations: readonly string[];
}
/**
* Read the table as `bootloader_utility_load_partition_table` does: a later
* factory, otadata or same-subtype OTA entry replaces an earlier one, and
* every OTA app entry adds to the app count.
*/
export const espBootPartitions = (
partitions: readonly EspPartition[],
): EspBootPartitions => {
let factory: EspPartition | undefined;
let otadata: EspPartition | undefined;
let otaAppCount = 0;
const otaSlots = new Map<number, EspPartition>();
const roles = new Map<string, EspPartition[]>();
for (const part of partitions) {
const slot = espOtaAppSlot(part);
const role = isEspFactoryApp(part)
? "factory"
: isEspOtaData(part)
? "otadata"
: slot === null
? undefined
: `ota_${String(slot)}`;
if (role === undefined) continue;
roles.set(role, [...(roles.get(role) ?? []), part]);
if (role === "factory") factory = part;
else if (role === "otadata") otadata = part;
else if (slot !== null) {
otaSlots.set(slot, part);
otaAppCount += 1;
}
}
const limitations = [...roles].flatMap(([role, parts]) =>
parts.length < 2
? []
: [
`The table declares ${role} ${String(parts.length)} times (partitions ${parts.map(({ index }) => `#${String(index)}`).join(", ")}); ESP-IDF's bootloader uses the last.`,
],
);
return { factory, otadata, otaSlots, otaAppCount, limitations };
};
@@ -0,0 +1,124 @@
import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import {
ESP32C3_BOOTLOADER_SEGMENTS,
espImage,
} from "../../src/domain/firmware/espImage.fixture.js";
import {
TWO_OTA_PARTITIONS,
espFlashDump,
partitionTable,
} from "../../src/domain/firmware/espFlash.fixture.js";
import { artifactComparisonResultSchema } from "../../src/domain/artifactComparison.js";
import { artifactInspectionResultSchema } from "../../src/domain/artifactInspection.js";
import { artifactInventoryResultSchema } from "../../src/domain/artifactGraph.js";
import { parseEvidence, type Evidence } from "../../src/domain/evidence.js";
import { connectLocalToolsMcp } from "../fixtures/localToolsMcp.js";
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
const BOOTLOADER = {
at: 0,
bytes: espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS }),
};
/** A table without an MD5 whose first entry has these label bytes. */
const labelBytes = (bytes: readonly number[]): Buffer => {
const table = Buffer.from(
partitionTable(TWO_OTA_PARTITIONS, { md5: "absent" }),
);
table.fill(0, 12, 28);
Buffer.from(bytes).copy(table, 12);
return table;
};
it("reports an ESP partition whose declaration alone changed", async () => {
const root = await createTestTempDirectory("rea-esp-flash-comparison-");
const { call } = await connectLocalToolsMcp();
const inspect = async (name: string, table: Buffer): Promise<Evidence> => {
const path = join(root, name);
await writeFile(path, espFlashDump({ bootloader: BOOTLOADER, table }));
const opened = await call("open_binary", { path });
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
const response = await call("inspect_artifact", {});
expect(response.isError, JSON.stringify(response)).not.toBe(true);
const inventory = artifactInspectionResultSchema
.parse(parseEvidence(response.structuredContent).normalized_result)
.substeps.find(
({ operation }) => operation === "inventory_artifact",
)?.evidence;
if (inventory === undefined) throw new Error("Missing inventory Evidence");
return parseEvidence(inventory);
};
const compare = async (left: Evidence, right: Evidence) => {
const response = await call("compare_artifacts", { left, right });
expect(response.isError, JSON.stringify(response)).not.toBe(true);
return artifactComparisonResultSchema.parse(
parseEvidence(response.structuredContent).normalized_result,
);
};
const occurrence = (evidence: Evidence, path: string) => {
const found = artifactInventoryResultSchema
.parse(evidence.normalized_result)
.occurrences.find(({ logical_path }) => logical_path === path);
if (found === undefined) throw new Error(`Missing ${path}`);
return found;
};
const changedPaths = (result: { changes: { logical_path: string }[] }) =>
result.changes
.map(({ logical_path }) => logical_path)
.filter((path) => path !== ".")
.sort();
// The same partition bytes, with nvs declared read-only.
const readonly = TWO_OTA_PARTITIONS.map((part) =>
part.label === "nvs" ? { ...part, flags: 0x2 } : part,
);
const left = await inspect("flash.bin", partitionTable(TWO_OTA_PARTITIONS));
const right = await inspect("readonly.bin", partitionTable(readonly));
const leftNvs = occurrence(left, "partitions/nvs.bin");
const rightNvs = occurrence(right, "partitions/nvs.bin");
expect(rightNvs.artifact_id).toBe(leftNvs.artifact_id);
expect(leftNvs.format_metadata).toMatchObject({ readonly: false });
expect(rightNvs.format_metadata).toMatchObject({ readonly: true });
const changed = await compare(left, right);
expect(
changed.changes.find(
({ logical_path }) => logical_path === "partitions/nvs.bin",
),
).toEqual({
classification: "changed",
logical_path: "partitions/nvs.bin",
dimensions: ["metadata"],
left_occurrence_id: leftNvs.occurrence_id,
right_occurrence_id: rightNvs.occurrence_id,
left_artifact_id: leftNvs.artifact_id,
right_artifact_id: rightNvs.artifact_id,
evidence_links: [left.evidence_id, right.evidence_id],
});
// Only the table's bytes and the declaration it gives nvs changed.
expect(changedPaths(changed)).toEqual([
"partition-table.bin",
"partitions/nvs.bin",
]);
// Labels that are not UTF-8 compare by their bytes.
const first = await inspect("label-a.bin", labelBytes([0xff, 0x01]));
const second = await inspect("label-b.bin", labelBytes([0xfe, 0x01]));
expect(occurrence(first, "partitions/0.bin").format_metadata).toMatchObject({
label: null,
label_hex: "ff01",
});
expect(changedPaths(await compare(first, second))).toEqual([
"partition-table.bin",
"partitions/0.bin",
]);
// Equal declarations leave every partition unchanged.
const same = await compare(
left,
await inspect("copy.bin", partitionTable(TWO_OTA_PARTITIONS)),
);
expect(same.changes).toEqual([]);
});
@@ -594,6 +594,22 @@ describe("binary target I/O: ESP flash layouts", () => {
});
});
it("keeps refusing an ESP8266 dump that has an ESP-IDF-style table", async () => {
const directory = await createTestTempDirectory("rea-esp8266-flash-");
const dump = join(directory, "esp8266-flash.bin");
await writeFile(
dump,
espFlashDump({ bootloader: { at: 0, bytes: esp8266Image() } }),
);
expect(await parseBinaryTarget(dump)).toMatchObject({
ok: false,
error: {
constraint: "file_format_rejected",
reason: ESP8266_REFUSAL,
},
});
});
it("reads a selected flash dump with caller inputs and refuses bad pairings", async () => {
const directory = await createTestTempDirectory("rea-esp-flash-select-");
const esp32 = espImage({