diff --git a/docs/desktop-application-export.md b/docs/desktop-application-export.md index 851642bda9..91e22f3d99 100644 --- a/docs/desktop-application-export.md +++ b/docs/desktop-application-export.md @@ -562,7 +562,10 @@ as their entries, read with Python's `zipfile` from the file in place, with REA's entry, size, compression ratio and archive limits and its wording for an entry it does not read. The module reads the central directory itself, as REA does, and shows `zipfile` only the records it lists, reading each entry in -bounded chunks no further than a byte past its declared size. A gzip file is +bounded chunks no further than a byte past its declared size. The reader +follows a Zip64 locator to the complete end-record span and skips its +extension bytes before passing the bounded directory to `zipfile`. +A gzip file is the view of the document inside it, with `"compression": "gzip"` beside its format, decompressed with Python's `zlib` within the same entry size and ratio limits, its members joined as REA joins them. A JSON file diff --git a/docs/desktop-application-state.md b/docs/desktop-application-state.md index a3e9644c5c..82cc57b213 100644 --- a/docs/desktop-application-state.md +++ b/docs/desktop-application-state.md @@ -29,6 +29,9 @@ A path absent from the capture is an input error. `sqlite_tables`, `sqlite_rows`, `sqlite_view_bytes`, `text_lines`, `xml_nodes`, `xml_depth`, `parsed_bytes`, `json_depth`, `zip_entries`, `zip_entry_bytes`, `zip_bytes`, and `zip_ratio`. Unspecified controls keep the defaults described below. +A request that raises `zip_entries` past the classic ZIP count range still uses +the selected entry limit. The host and exported readers encode the retained +directory's exact count with Zip64 fields when the classic fields are full. `xml_depth` admits at most 494 and `json_depth` at most 991: these bounds follow the fixed Evidence JSON nesting limit, including the captured-version and ZIP-member wrappers and XML's element/children representation. A request @@ -138,16 +141,16 @@ Views hold the captured bytes as they are, the capture host's name and user included; the export removes those from the bundle. A version records whether the file was present, the SHA-256 of its retained bytes, and a view: -| Format | Detected by | View | -| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -| `sqlite` | The `SQLite format 3` header | Every ordinary table's columns, primary key and up to 500 rows | -| `dconf` | The `GVariant` header | Groups whose path names the application, plus the names of the other groups | -| `zip` | A zip local file header, `PK\x03\x04`, whatever the name | Its file entries in archive order, each with its own format and view | -| `json` | A `.json` name or a leading `{` or `[` that parses | The parsed value, from files up to 4 MiB nested up to 500 levels | -| `xml` | An XML name, a declaration or a leading element that parses | An element tree (`name`, `attributes`, `children`); text kept exactly, DTDs never read | -| `keyfile` | Only `[group]`, `key=value` and comment lines, with at least one group | Groups of `key` → string value, from files up to 4 MiB | -| `text` | Other decoded text | Lines, up to 5,000 | -| `binary` | Content that cannot be read as text | None | +| Format | Detected by | View | +| --------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- | +| `sqlite` | The `SQLite format 3` header | Every ordinary table's columns, primary key and up to 500 rows | +| `dconf` | The `GVariant` header | Groups whose path names the application, plus the names of the other groups | +| `zip` | A ZIP local file header or an empty archive signature, whatever the name | Its file entries in archive order, each with its own format and view | +| `json` | A `.json` name or a leading `{` or `[` that parses | The parsed value, from files up to 4 MiB nested up to 500 levels | +| `xml` | An XML name, a declaration or a leading element that parses | An element tree (`name`, `attributes`, `children`); text kept exactly, DTDs never read | +| `keyfile` | Only `[group]`, `key=value` and comment lines, with at least one group | Groups of `key` → string value, from files up to 4 MiB | +| `text` | Other decoded text | Lines, up to 5,000 | +| `binary` | Content that cannot be read as text | None | Text views decode UTF-8 or strictly valid UTF-16 with a leading byte-order mark. UTF-16 containing a NUL character stays binary. These rules also apply @@ -275,8 +278,9 @@ database, or an archive, inside an archive keeps its format and `sha256` and has a `view_error` instead of a view: it is not read. The view lists at most 1,000 entries; `entry_count` counts them all, and `truncated` is true when some are not listed. REA reads the central directory itself, a record at a time, -through the Zip64 end record whenever a Zip64 locator precedes the end record, -as Python's `zipfile` does, and hands the zip reader only the records it lists, +and follows a Zip64 locator to the complete end-record span, including any +extension bytes. The fixed-record fallback supports prepended archives +whose offsets omit the prefix. REA hands the zip reader only the records it lists, so an archive of hundreds of thousands of entries costs no more to read than one of a thousand. An archive whose central directory cannot be read has a `view_error` of its own. diff --git a/docs/specs/desktop-app-environments/document-views/plan.md b/docs/specs/desktop-app-environments/document-views/plan.md index 73e31d6bf9..45d4cee6fc 100644 --- a/docs/specs/desktop-app-environments/document-views/plan.md +++ b/docs/specs/desktop-app-environments/document-views/plan.md @@ -1,6 +1,7 @@ # Plan -1. Format and reader: `zip` detected by its local file header; the text, JSON, +1. Format and reader: `zip` detected by its local file header or an empty + classic/Zip64 archive signature; the text, JSON, key file and XML readers shared as `textualView`, so a file and an entry read alike; a domain module for the entry limits, the entry view and the archive view, and an application reader that walks the verified bytes with diff --git a/docs/specs/desktop-app-environments/document-views/spec.md b/docs/specs/desktop-app-environments/document-views/spec.md index a765cf6f0b..af4012add2 100644 --- a/docs/specs/desktop-app-environments/document-views/spec.md +++ b/docs/specs/desktop-app-environments/document-views/spec.md @@ -13,8 +13,9 @@ evaluators, REA's and the exported verifier's, agree. ## Requirements -- Format: a file starting with a zip local file header (`PK\x03\x04`) is - `zip`, whatever its name. Its view is +- Format: a file starting with a zip local file header (`PK\x03\x04`), or + a classic/Zip64 empty-archive signature with at least 22 bytes, is `zip`, + whatever its name. A signature is not proof of directory validity. Its view is `{entries, entry_count, truncated}`: the file entries in archive order, directories left out, each `{name, size, sha256, format, view, view_error}`. - Reading: from the retained bytes already checked against their digest, in diff --git a/docs/specs/desktop-app-environments/state-view/spec.md b/docs/specs/desktop-app-environments/state-view/spec.md index 0e56b7f86e..3e980fc124 100644 --- a/docs/specs/desktop-app-environments/state-view/spec.md +++ b/docs/specs/desktop-app-environments/state-view/spec.md @@ -40,7 +40,7 @@ generation reads the kinds of change. only when it has no internal subset and its DTD never read ([XML text checks](../xml-text-checks/spec.md)); key files up to 4 MiB whose every line parses; zip archives by - their local file header, as their entries with each entry's own format and + their local file header or empty archive signature, as their entries with each entry's own format and view ([document views](../document-views/spec.md)); text; binary. A gzip file, told by its magic bytes, is read as the document inside, its members joined, within a zip entry's 16 MiB and 100-times limits, and its version diff --git a/scripts/verify/baselines/raw-buffer-read.json b/scripts/verify/baselines/raw-buffer-read.json index c59fbc9609..2c0060e39b 100644 --- a/scripts/verify/baselines/raw-buffer-read.json +++ b/scripts/verify/baselines/raw-buffer-read.json @@ -13,7 +13,7 @@ "src/artifacts/nodeSea/NodeSeaLocation.ts": 2, "src/artifacts/pyinstaller/CArchiveLayout.ts": 8, "src/domain/apkSignature/apkSignatureSchemes.ts": 1, - "src/domain/apkSignature/apkSigningBlock.ts": 7, + "src/domain/apkSignature/apkSigningBlock.ts": 2, "src/domain/apkSignature/apkV1Signature.ts": 15, "src/domain/apple/appleMachoFixups.ts": 22, "src/domain/apple/appleMachoSelection.ts": 2, @@ -32,7 +32,6 @@ "src/domain/binaryTarget.ts": 10, "src/domain/desktop/elfDependencies.ts": 4, "src/domain/desktop/openPgpKeys.ts": 2, - "src/domain/desktop/zipDirectory.ts": 14, "src/domain/dosMz.ts": 10, "src/domain/elfDynamicEntry.ts": 4, "src/domain/elfFixedHeader.ts": 4, diff --git a/src/domain/apkSignature/apkSigningBlock.ts b/src/domain/apkSignature/apkSigningBlock.ts index 9d328de67e..ae77febf38 100644 --- a/src/domain/apkSignature/apkSigningBlock.ts +++ b/src/domain/apkSignature/apkSigningBlock.ts @@ -1,6 +1,12 @@ +import { BinaryReader } from "../binaryContainer.js"; import { + readZip64DirectoryRecord, zipEndOfCentralDirectoryCandidates, + zipEndUsesSingleVolume, + ZIP_CLASSIC_DIRECTORY_LIMITS, ZIP_EOCD_BYTES, + ZIP64_END, + ZIP64_LOCATOR, type ZipEndOfCentralDirectory, } from "../zipEndOfCentralDirectory.js"; @@ -20,11 +26,6 @@ import { */ import { type Result, err, ok } from "../result.js"; -const ZIP64_EOCD_LOCATOR_SIGNATURE = 0x07064b50; -const ZIP64_EOCD_SIGNATURE = 0x06064b50; -const ZIP64_SENTINEL = 0xffffffff; -/** The 16-bit EOCD entry count saturates here when Zip64 carries the real count. */ -const ZIP64_ENTRIES_SENTINEL = 0xffff; /** "APK Sig Block 42" — the 16-byte block magic (JAR/APK signing block spec). */ const BLOCK_MAGIC = "APK Sig Block 42"; /** A single pair is at least 12 bytes (uint64 length + uint32 id); cap the count. */ @@ -123,7 +124,7 @@ interface CentralDirectory { const classicDirectory = ( eocd: ZipEndOfCentralDirectory, ): Result => { - if (eocd.disk !== 0 || eocd.directoryDisk !== 0) + if (!zipEndUsesSingleVolume(eocd)) return err({ kind: "multi-disk-unsupported", detail: `classic EOCD names disk ${String(eocd.disk)} and central-directory disk ${String(eocd.directoryDisk)}`, @@ -147,7 +148,6 @@ const centralDirectoryOffset = ( bytes: Uint8Array, eocd: ZipEndOfCentralDirectory, ): Result => { - const dv = view(bytes); const { offset: eocdOffset, totalEntries: entries, @@ -155,19 +155,24 @@ const centralDirectoryOffset = ( directoryOffset: offset, } = eocd; const saturated = - offset === ZIP64_SENTINEL || - size === ZIP64_SENTINEL || - entries === ZIP64_ENTRIES_SENTINEL; + offset === ZIP_CLASSIC_DIRECTORY_LIMITS.offset || + size === ZIP_CLASSIC_DIRECTORY_LIMITS.size || + entries === ZIP_CLASSIC_DIRECTORY_LIMITS.count; if (!saturated) return classicDirectory(eocd); + const reader = new BinaryReader( + Buffer.from(bytes.buffer, bytes.byteOffset, bytes.byteLength), + true, + "zip", + ); // Zip64: a locator precedes the EOCD, pointing at the Zip64 EOCD record. - const locator = eocdOffset - 20; - if ( - locator < 0 || - dv.getUint32(locator, true) !== ZIP64_EOCD_LOCATOR_SIGNATURE - ) { + const locator = eocdOffset - ZIP64_LOCATOR.bytes; + if (locator < 0 || reader.u32(locator) !== ZIP64_LOCATOR.signature) { // Exactly 65,535 entries saturates the count without Zip64; that archive // is still described completely by its 32-bit EOCD fields. - if (offset !== ZIP64_SENTINEL && size !== ZIP64_SENTINEL) + if ( + offset !== ZIP_CLASSIC_DIRECTORY_LIMITS.offset && + size !== ZIP_CLASSIC_DIRECTORY_LIMITS.size + ) return classicDirectory(eocd); return err({ kind: "malformed-zip", @@ -177,8 +182,8 @@ const centralDirectoryOffset = ( } // APPNOTE 6.3.10 §4.3.15 makes the record offset relative to its disk. // A selected file cannot resolve another disk's offset as a local span. - const recordDisk = dv.getUint32(locator + 4, true); - const totalDisks = dv.getUint32(locator + 16, true); + const recordDisk = reader.u32(locator + 4); + const totalDisks = reader.u32(locator + 16); if (totalDisks === 0) return err({ kind: "malformed-zip", @@ -196,46 +201,51 @@ const centralDirectoryOffset = ( kind: "zip64-unsupported", detail: "multi-disk Zip64 archives are not supported", }); - const recordOffset = u64(dv, locator + 8); - if (recordOffset === null) + const offsetOnDisk = reader.word(locator + ZIP64_LOCATOR.recordOffset, 64); + const maximumExact = BigInt(Number.MAX_SAFE_INTEGER); + if (offsetOnDisk > maximumExact) return err({ kind: "zip64-unsupported", detail: "Zip64 EOCD offset exceeds safe range", }); - if (recordOffset + 56 > locator) - return err({ - kind: "malformed-zip", - offset: locator + 8, - detail: "Zip64 EOCD record extends beyond its locator", - }); - if (dv.getUint32(recordOffset, true) !== ZIP64_EOCD_SIGNATURE) - return err({ - kind: "malformed-zip", - offset: recordOffset, - detail: "invalid Zip64 EOCD signature", - }); - // The size excludes the signature and size field. Check the declared - // body before using its fixed fields to locate the central directory. - const recordSize = dv.getBigUint64(recordOffset + 4, true); - if (recordSize < 44n || recordSize > BigInt(locator - recordOffset - 12)) - return err({ - kind: "malformed-zip", - offset: recordOffset + 4, - detail: "invalid Zip64 EOCD record size", - }); - const z64Offset = u64(dv, recordOffset + 48); - const z64Size = u64(dv, recordOffset + 40); - // The Zip64 record's total entry count; the 16-bit EOCD field saturates. - const z64Entries = u64(dv, recordOffset + 32); - if (z64Offset === null || z64Size === null || z64Entries === null) + const recordOffset = Number(offsetOnDisk); + const record = readZip64DirectoryRecord(reader, recordOffset, locator); + if (!record.ok) { + switch (record.error) { + case "fixed-fields": + return err({ + kind: "malformed-zip", + offset: locator + ZIP64_LOCATOR.recordOffset, + detail: "Zip64 EOCD record extends beyond its locator", + }); + case "signature": + return err({ + kind: "malformed-zip", + offset: recordOffset, + detail: "invalid Zip64 EOCD signature", + }); + case "size": + return err({ + kind: "malformed-zip", + offset: recordOffset + ZIP64_END.sizeOffset, + detail: "invalid Zip64 EOCD record size", + }); + } + } + const { directoryOffset, directorySize, totalEntries } = record.value; + if ( + directoryOffset > maximumExact || + directorySize > maximumExact || + totalEntries > maximumExact + ) return err({ kind: "zip64-unsupported", detail: "Zip64 sizes exceed safe range", }); return ok({ - offset: z64Offset, - size: z64Size, - entries: z64Entries, + offset: Number(directoryOffset), + size: Number(directorySize), + entries: Number(totalEntries), end: recordOffset, }); }; diff --git a/src/domain/desktop/appExportPython.ts b/src/domain/desktop/appExportPython.ts index f6dbe30dcb..0a758e815a 100644 --- a/src/domain/desktop/appExportPython.ts +++ b/src/domain/desktop/appExportPython.ts @@ -1,3 +1,5 @@ +import { ZIP_EOCD_BYTES } from "../zipEndOfCentralDirectory.js"; +import { ZIP_SIGNATURES } from "../zipPackageFormat.js"; import { XML_ELEMENT_OPENING } from "../xmlDocument.js"; import { DESKTOP_VIEW_DEFAULTS, @@ -90,7 +92,9 @@ LINE_BREAK = re.compile(${JSON.stringify(DESKTOP_LINE_BREAK)}) ECMASCRIPT_TRIM_CHARACTERS = ${PYTHON_ECMASCRIPT_TRIM_CHARACTERS} SQLITE_HEADER = b"SQLite format 3\x00" GVDB_HEADER = b"GVariant" -ZIP_HEADER = b"PK\x03\x04" +ZIP_SIGNATURES = (${ZIP_SIGNATURES.map((signature) => `bytes(${JSON.stringify(signature)})`).join(", ")}) +ZIP_HEADER = ZIP_SIGNATURES[0] +ZIP_EOCD_BYTES = ${String(ZIP_EOCD_BYTES)} GZIP_HEADER = b"\x1f\x8b\x08" XML_SPACE = "xml:space" MAX_SAFE_INTEGER = 2**53 - 1 @@ -428,6 +432,11 @@ def _bytes_view(name: str, data: bytes) -> dict: return _text_view(name, text, len(data)) +def _is_zip(data: bytes) -> bool: + """Recognize storage signatures before the directory reader checks validity.""" + return data.startswith(ZIP_HEADER) or (len(data) >= ZIP_EOCD_BYTES and data.startswith(ZIP_SIGNATURES)) + + def _view(path: Path) -> dict: """A typed JSON view of one file in the shapes of REA's state view; truncated says the view is partial.""" too_large = {"format": None, "view": None, "error": VIEW_TOO_LARGE} @@ -446,7 +455,7 @@ def _view(path: Path) -> dict: return too_large if data.startswith(SQLITE_HEADER): return _sqlite(path) - if data.startswith(ZIP_HEADER): + if _is_zip(data): with path.open("rb") as file: return _zip(file, os.fstat(file.fileno()).st_size) if data.startswith(GZIP_HEADER): diff --git a/src/domain/desktop/appExportPythonZip.ts b/src/domain/desktop/appExportPythonZip.ts index 3f6dab1edc..df45f66538 100644 --- a/src/domain/desktop/appExportPythonZip.ts +++ b/src/domain/desktop/appExportPythonZip.ts @@ -1,3 +1,12 @@ +import { + ZIP_CLASSIC_DIRECTORY_LIMITS, + ZIP_CLASSIC_DISK_OFFSETS, + ZIP_SINGLE_VOLUME, + ZIP64_END, + ZIP64_LOCATOR, +} from "../zipEndOfCentralDirectory.js"; +import { ZIP_EXTRA_FIELD } from "../zip/zipExtraFields.js"; + /** * The exported module's zip and gzip readers: the central directory read a * record at a time, entries read in bounded chunks, a gzip file's document @@ -12,9 +21,14 @@ ZIP_END = b"PK\x05\x06" ZIP_CENTRAL = b"PK\x01\x02" ZIP64_LOCATOR = b"PK\x06\x07" ZIP64_END = b"PK\x06\x06" +ZIP64_END_LAYOUT = ${JSON.stringify(ZIP64_END)} +ZIP64_LOCATOR_LAYOUT = ${JSON.stringify(ZIP64_LOCATOR)} +ZIP_CLASSIC_DIRECTORY_LIMITS = ${JSON.stringify(ZIP_CLASSIC_DIRECTORY_LIMITS)} +ZIP_CLASSIC_DISK_OFFSETS = ${JSON.stringify(Object.values(ZIP_CLASSIC_DISK_OFFSETS))} +ZIP_SINGLE_VOLUME = ${JSON.stringify(ZIP_SINGLE_VOLUME)} +ZIP_EXTRA_FIELD = ${JSON.stringify(ZIP_EXTRA_FIELD)} #: Formats an archive entry may have whose reader needs a tool, a worker or another archive opened. -NOT_READ_INSIDE = ((SQLITE_HEADER, "sqlite", "a SQLite database"), (GVDB_HEADER, "dconf", "a dconf database"), - (ZIP_HEADER, "zip", "an archive")) +NOT_READ_INSIDE = ((SQLITE_HEADER, "sqlite", "a SQLite database"), (GVDB_HEADER, "dconf", "a dconf database")) def _entry_view(name: str, data: bytes) -> dict: @@ -23,6 +37,8 @@ def _entry_view(name: str, data: bytes) -> dict: for header, form, inside in NOT_READ_INSIDE: if data.startswith(header): return {"format": form, "view": None, "error": f"{inside} inside an archive is not read"} + if _is_zip(data): + return {"format": "zip", "view": None, "error": "an archive inside an archive is not read"} return _bytes_view(name, data) @@ -74,11 +90,24 @@ def _zip_entry(archive, info, refusal, name): "view": view.get("view"), "view_error": view.get("error")} +def _zip64_directory_record(file, record: int, locator: int): + """Read the fixed fields only, after the entire declared record fits before the locator.""" + fixed = ZIP64_END_LAYOUT["fixedBytes"] + prefix = ZIP64_END_LAYOUT["sizePrefixBytes"] + if record < 0 or record + fixed > locator: + return None + file.seek(record) + header = file.read(fixed) + if len(header) != fixed or header[:4] != ZIP64_END: + return None + body = struct.unpack_from("= 5 and len(data) == length and data[0] == 1 and not flags & 0x800 +def _zip_extras(extra: bytes): + """Check all framed fields before selection and return the last Unicode Path and the other original bytes.""" + at, copied, selected, kept = 0, 0, None, None + header = ZIP_EXTRA_FIELD["headerBytes"] + while at + header <= len(extra): + payload = at + header + size = struct.unpack_from(" len(extra): + raise zipfile.BadZipFile("Corrupt extra field") + kind = struct.unpack_from("= 5 and data[0] == 1 and not flags & 0x800 and int.from_bytes(data[1:5], "little") == zlib.crc32(raw)) return data[5:] if usable else None @@ -125,24 +187,13 @@ def _zip_name(raw: bytes, flags: int, unicode_path): return raw.decode("cp437") if unicode_path is None else unicode_path.decode("utf-8") -def _zip_kept_record(fixed: bytes, rest: bytes, name_length: int, extra_length: int): +def _zip_kept_record(fixed: bytes, raw: bytes, extra, comment: bytes): """Remove already-interpreted Unicode claims from the bounded decoder copy. zipfile otherwise interprets each occurrence, retaining an earlier valid name or rejecting invalid UTF-8 that zip.js already superseded. All other fields, comments and the original artifact bytes remain unchanged.""" - raw, extra = rest[:name_length], rest[name_length:name_length + extra_length] - kept, at = bytearray(), 0 - while at + 4 <= len(extra): - kind, length = struct.unpack_from(" len(extra): - raise zipfile.BadZipFile("Corrupt extra field") - if kind != 0x7075: - kept += extra[at:end] - at = end - kept += extra[at:] header = bytearray(fixed) - struct.pack_into(" end: return None - unicode_path = _zip_unicode_path(rest[:name_length], flags, rest[name_length:name_length + extra_length]) - name = _zip_name(rest[:name_length], flags, unicode_path) + extra_end = name_length + extra_length + raw = rest[:name_length] + selected, extra = _zip_extras(rest[name_length:extra_end]) + name = _zip_name(raw, flags, _zip_unicode_path(raw, flags, selected)) if not (name.endswith("/") or (external >> 16) & 0o170000 == 0o040000 or (made_by >> 8 == 0 and external & 0x10)): count += 1 if len(records) < keep: - records.append(_zip_kept_record(fixed, rest, name_length, extra_length)) + records.append(_zip_kept_record(fixed, raw, extra, rest[extra_end:])) names.append(name) return start, declared, records, count, names @@ -222,16 +275,22 @@ def _zip(file, size: int) -> dict: if directory is None: raise zipfile.BadZipFile("no readable central directory") start, declared, records, count, names = directory - kept = b"".join(records) - offset = declared if declared <= 0xFFFFFFFF else start - # zipfile follows a Zip64 locator signature wherever one sits before the end record, and the kept records - # may end in those bytes, so a real Zip64 end record and locator for them come first. + kept, retained = b"".join(records), len(records) + limit = ZIP_CLASSIC_DIRECTORY_LIMITS + offset = declared if declared <= limit["offset"] else start + classic_count = min(retained, limit["count"]) + # zipfile follows a Zip64 locator signature wherever one sits before the end record, including in + # a kept record's comment. Give it real Zip64 records with the exact selected directory facts. tail = (kept - + struct.pack("<4sQ2H2I4Q", ZIP64_END, 44, 45, 45, 0, 0, len(records), len(records), len(kept), offset) + + struct.pack("<4sQ2H2I4Q", ZIP64_END, 44, 45, 45, 0, 0, retained, retained, len(kept), offset) + struct.pack("<4sIQI", ZIP64_LOCATOR, 0, offset + len(kept), 1) - + struct.pack("<4s4H2IH", ZIP_END, 0, 0, len(records), len(records), len(kept), offset, 0)) + + struct.pack("<4s4H2IH", ZIP_END, 0, 0, classic_count, classic_count, + min(len(kept), limit["size"]), min(offset, limit["offset"]), 0)) with zipfile.ZipFile(_BoundedArchive(file, start, tail)) as archive: - for info, name in zip(archive.infolist(), names, strict=True): + infos = archive.infolist() + if len(infos) != len(names): + raise zipfile.BadZipFile("retained member count differs from decoded member count") + for info, name in zip(infos, names): refusal = _zip_refusal(info, spent) spent += info.file_size if refusal is None else 0 entries.append(_zip_entry(archive, info, refusal, name)) @@ -279,8 +338,8 @@ def _gzip(name: str, data: bytes) -> dict: if MAX_ZIP_RATIO * len(data) < MAX_ZIP_ENTRY_BYTES else f"larger than the {_view_byte_limit(MAX_ZIP_ENTRY_BYTES)} a compressed document's view reads"} for header, form, inside in NOT_READ_INSIDE: - if form != "zip" and document.startswith(header): + if document.startswith(header): return {**compressed, "format": form, "error": f"{inside} inside a gzip file is not read"} - view = _zip(io.BytesIO(document), len(document)) if document.startswith(ZIP_HEADER) else _bytes_view(name, document) + view = _zip(io.BytesIO(document), len(document)) if _is_zip(document) else _bytes_view(name, document) return {**view, "compression": "gzip"} `; diff --git a/src/domain/desktop/stateFormats.ts b/src/domain/desktop/stateFormats.ts index b9d3ceda1e..b2e3ff59ab 100644 --- a/src/domain/desktop/stateFormats.ts +++ b/src/domain/desktop/stateFormats.ts @@ -9,6 +9,8 @@ import { } from "./desktopViewPolicy.js"; import { err, ok, type Result } from "../result.js"; import { safeParseJson } from "../safeJson.js"; +import { hasZipLocalHeader, hasZipSignature } from "../zipPackageFormat.js"; +import { ZIP_EOCD_BYTES } from "../zipEndOfCentralDirectory.js"; import { declaredDocumentType, isWellFormedXml, @@ -42,8 +44,6 @@ export const SQLITE_SIDECARS = [ { suffix: "-journal", label: "rollback journal" }, ] as const; const GVDB_HEADER = "GVariant"; -/** A zip archive's local file header, which FreeCAD, ODF, OOXML, EPUB and JAR files begin with. */ -const ZIP_HEADER = "PK\u0003\u0004"; const XML_EXTENSIONS = /\.(?:xml|xbel|plist|svg|ui)$/iu; /** A DOCTYPE that only names its DTD reads as XML; one with an internal subset, which can declare entities, does not. */ export const XML_POLICY = { @@ -306,7 +306,11 @@ export const detectStateFormat = ( const header = new TextDecoder("latin1").decode(bytes.subarray(0, 16)); if (header === SQLITE_HEADER) return "sqlite"; if (header.startsWith(GVDB_HEADER)) return "dconf"; - if (header.startsWith(ZIP_HEADER)) return "zip"; + if ( + hasZipLocalHeader(bytes) || + (bytes.length >= ZIP_EOCD_BYTES && hasZipSignature(bytes)) + ) + return "zip"; const text = decodeStateText(bytes); if (text === undefined) return "binary"; const trimmed = text.trimStart(); diff --git a/src/domain/desktop/zipArchive.fixture.ts b/src/domain/desktop/zipArchive.fixture.ts index 78acfc811f..d186e1f8e4 100644 --- a/src/domain/desktop/zipArchive.fixture.ts +++ b/src/domain/desktop/zipArchive.fixture.ts @@ -173,20 +173,25 @@ export const emptyEntriesArchive = (count: number): Buffer => { * and a classic one that defers to them, or with `sentinels` false keeps its * own counts, size and offset, as a writer may when they fit. */ -export const withZip64End = (archive: Buffer, sentinels = true): Buffer => { +export const withZip64End = ( + archive: Buffer, + sentinels = true, + extension: Uint8Array = new Uint8Array(), +): Buffer => { const end = archive.length - 22; const entries = archive.readUInt16LE(end + 10); const size = archive.readUInt32LE(end + 12); const offset = archive.readUInt32LE(end + 16); - const record = Buffer.alloc(56); + const record = Buffer.alloc(56 + extension.length); record.writeUInt32LE(0x06064b50, 0); - record.writeBigUInt64LE(44n, 4); + record.writeBigUInt64LE(BigInt(44 + extension.length), 4); record.writeUInt16LE(45, 12); record.writeUInt16LE(45, 14); record.writeBigUInt64LE(BigInt(entries), 24); record.writeBigUInt64LE(BigInt(entries), 32); record.writeBigUInt64LE(BigInt(size), 40); record.writeBigUInt64LE(BigInt(offset), 48); + record.set(extension, 56); const locator = Buffer.alloc(20); locator.writeUInt32LE(0x07064b50, 0); locator.writeBigUInt64LE(BigInt(end), 8); diff --git a/src/domain/desktop/zipDirectory.ts b/src/domain/desktop/zipDirectory.ts index fba7a595de..ba2518f985 100644 --- a/src/domain/desktop/zipDirectory.ts +++ b/src/domain/desktop/zipDirectory.ts @@ -1,20 +1,22 @@ import { crc32 } from "node:zlib"; +import { BinaryReader } from "../binaryContainer.js"; import { + ZIP_CLASSIC_DIRECTORY_LIMITS, + ZIP64_END, + ZIP64_LOCATOR, ZIP_EOCD_BYTES, - ZIP_MAX_COMMENT_BYTES, + readZip64DirectoryRecord, + zip64EndUsesSingleVolume, zipEndOfCentralDirectoryCandidates, + zipEndUsesSingleVolume, } from "../zipEndOfCentralDirectory.js"; import { strictUtf8Decoder } from "../strictUtf8.js"; +import { ZIP_EXTRA_FIELD, lastZipExtraField } from "../zip/zipExtraFields.js"; const CENTRAL_HEADER = 0x02014b50; const CENTRAL_HEADER_BYTES = 46; -const ZIP64_LOCATOR = 0x07064b50; -const ZIP64_LOCATOR_BYTES = 20; -const ZIP64_END = 0x06064b50; -const ZIP64_END_BYTES = 56; const UTF8_NAME = 0x800; -const UNICODE_PATH = 0x7075; const UNIX_TYPE = 0o170000; const UNIX_DIRECTORY = 0o040000; const MSDOS_DIRECTORY = 0x10; @@ -33,41 +35,45 @@ export interface ZipDirectory { } /** Where the directory ends, how long it is and the offset it declares, from the end record and any Zip64 one. */ -const directoryEnd = (view: DataView) => { - const window = Math.max( - 0, - view.byteLength - ZIP_EOCD_BYTES - ZIP_MAX_COMMENT_BYTES, - ); - const found = zipEndOfCentralDirectoryCandidates( - new Uint8Array( - view.buffer, - view.byteOffset + window, - view.byteLength - window, - ), - ).next(); +const directoryEnd = (reader: BinaryReader) => { + const found = zipEndOfCentralDirectoryCandidates(reader.bytes).next(); if (found.done === true) return undefined; - const end = window + found.value.offset; - const { totalEntries, directorySize, directoryOffset } = found.value; - const classic = { end, size: directorySize, declaredOffset: directoryOffset }; - const record = end - ZIP64_LOCATOR_BYTES - ZIP64_END_BYTES; - // Like Python's zipfile, a Zip64 locator is followed whether or not the classic record defers to it. - if ( - record < 0 || - view.getUint32(end - ZIP64_LOCATOR_BYTES, true) !== ZIP64_LOCATOR - ) + const eocd = found.value; + const end = eocd.offset; + const { totalEntries, directorySize, directoryOffset } = eocd; + const classic = zipEndUsesSingleVolume(eocd) + ? { end, size: directorySize, declaredOffset: directoryOffset } + : undefined; + const locator = end - ZIP64_LOCATOR.bytes; + // Like Python's zipfile, follow a locator whether or not the classic record defers to it. + if (locator < 0 || reader.u32(locator) !== ZIP64_LOCATOR.signature) return classic; - // Read just before its locator, where zipfile and zip.js look when the locator's own offset holds no Zip64 record. - if (view.getUint32(record, true) === ZIP64_END) + const declared = reader.word(locator + ZIP64_LOCATOR.recordOffset, 64); + const fallback = locator - ZIP64_END.fixedBytes; + // Keep the fixed-record fallback for archives with an unadjusted prepended offset. + for (const at of [ + declared <= BigInt(fallback) ? Number(declared) : -1, + fallback, + ]) { + const record = readZip64DirectoryRecord(reader, at, locator); + if (!record.ok || record.value.end !== locator) continue; + if ( + !zipEndUsesSingleVolume(eocd, "zip64Classic") || + !zip64EndUsesSingleVolume(reader, locator, at) + ) + return undefined; return { - end: record, - size: Number(view.getBigUint64(record + 40, true)), - declaredOffset: Number(view.getBigUint64(record + 48, true)), + end: at, + size: Number(record.value.directorySize), + declaredOffset: Number(record.value.directoryOffset), }; - // Without one the classic values stand, as zip.js reads them, unless they are sentinels deferring to it. + } + // Keep the classic values for a stray locator when the classic record is complete. + const limits = ZIP_CLASSIC_DIRECTORY_LIMITS; const deferred = - totalEntries === 0xffff || - directorySize === 0xffffffff || - directoryOffset === 0xffffffff; + totalEntries === limits.count || + directorySize === limits.size || + directoryOffset === limits.offset; return deferred ? undefined : classic; }; @@ -79,45 +85,44 @@ interface RecordName { readonly flags: number; } -/** An Info-ZIP Unicode Path when zip.js uses it: version 1, no UTF-8 flag, and the CRC-32 of the stored name. */ +/** The usable last Unicode Path, or null when the extra block is malformed. */ const unicodePath = ( - bytes: Uint8Array, - view: DataView, + reader: BinaryReader, { name, extra, comment, flags }: RecordName, -): Uint8Array | undefined => { +): Uint8Array | undefined | null => { // zip.js indexes extra fields by ID with Map.set: the last occurrence // replaces earlier ones, even when that last claim fails validation. - let selected: { at: number; length: number } | undefined; - for (let at = extra; at + 4 <= comment;) { - const length = view.getUint16(at + 2, true); - if (view.getUint16(at, true) === UNICODE_PATH) selected = { at, length }; - at += 4 + length; - } - if (selected === undefined) return undefined; - const { at, length } = selected; - return length >= 5 && - at + 4 + length <= comment && - bytes[at + 4] === 1 && + const selected = lastZipExtraField( + reader, + extra, + comment, + ZIP_EXTRA_FIELD.unicodePath, + ); + if (!selected.ok) return null; + if (selected.value === undefined) return undefined; + const { start, end } = selected.value; + const { bytes } = reader; + return end - start >= 5 && + bytes[start] === 1 && (flags & UTF8_NAME) === 0 && - view.getUint32(at + 5, true) === crc32(bytes.subarray(name, extra)) - ? bytes.subarray(at + 9, at + 4 + length) + reader.u32(start + 1) === crc32(bytes.subarray(name, extra)) + ? bytes.subarray(start + 5, end) : undefined; }; /** * Whether an entry's name, as REA decodes it, ends in a slash; undefined when - * it must be UTF-8 and is not. A plain CP437 name is judged by its last byte, - * so most records are read without allocating. + * its extra block is malformed or its name must be UTF-8 and is not. A plain + * CP437 name is judged by its last byte, so most records are read without allocating. */ const endsInSlash = ( - bytes: Uint8Array, - view: DataView, + reader: BinaryReader, record: RecordName, ): boolean | undefined => { const unicode = - record.comment > record.extra - ? unicodePath(bytes, view, record) - : undefined; + record.comment > record.extra ? unicodePath(reader, record) : undefined; + if (unicode === null) return undefined; + const { bytes } = reader; const utf8 = unicode ?? ((record.flags & UTF8_NAME) === 0 @@ -143,8 +148,12 @@ export const scanZipDirectory = ( bytes: Uint8Array, keep: number, ): ZipDirectory | undefined => { - const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); - const located = directoryEnd(view); + const reader = new BinaryReader( + Buffer.from(bytes.buffer, bytes.byteOffset, bytes.byteLength), + true, + "zip", + ); + const located = directoryEnd(reader); if (located === undefined) return undefined; const start = located.end - located.size; if (start < 0) return undefined; @@ -153,27 +162,26 @@ export const scanZipDirectory = ( for (let at = start; at < located.end;) { if ( at + CENTRAL_HEADER_BYTES > located.end || - view.getUint32(at, true) !== CENTRAL_HEADER + reader.u32(at) !== CENTRAL_HEADER ) return undefined; const name = at + CENTRAL_HEADER_BYTES; - const extra = name + view.getUint16(at + 28, true); - const comment = extra + view.getUint16(at + 30, true); - const next = comment + view.getUint16(at + 32, true); + const extra = name + reader.u16(at + 28); + const comment = extra + reader.u16(at + 30); + const next = comment + reader.u16(at + 32); if (next > located.end) return undefined; - const slash = endsInSlash(bytes, view, { + const slash = endsInSlash(reader, { name, extra, comment, - flags: view.getUint16(at + 8, true), + flags: reader.u16(at + 8), }); if (slash === undefined) return undefined; - const external = view.getUint32(at + 38, true); + const external = reader.u32(at + 38); const directory = slash || ((external >>> 16) & UNIX_TYPE) === UNIX_DIRECTORY || - (view.getUint16(at + 4, true) >> 8 === 0 && - (external & MSDOS_DIRECTORY) !== 0); + (reader.u16(at + 4) >> 8 === 0 && (external & MSDOS_DIRECTORY) !== 0); if (!directory) { count += 1; if (records.length < keep) records.push(bytes.subarray(at, next)); @@ -189,24 +197,57 @@ export const boundedDirectoryTail = (directory: ZipDirectory): Uint8Array => { (total, record) => total + record.length, 0, ); - const tail = new Uint8Array(size + ZIP_EOCD_BYTES); + const { + count: maxCount, + size: maxSize, + offset: maxOffset, + } = ZIP_CLASSIC_DIRECTORY_LIMITS; + const count = directory.records.length; + // A declared offset past 32 bits cannot be right for a view-sized archive. + const offset = + directory.declaredOffset <= maxOffset + ? directory.declaredOffset + : directory.start; + const wide = count >= maxCount || size >= maxSize || offset >= maxOffset; + const wideBytes = wide ? ZIP64_END.fixedBytes + ZIP64_LOCATOR.bytes : 0; + const tail = new Uint8Array(size + wideBytes + ZIP_EOCD_BYTES); let at = 0; for (const record of directory.records) { tail.set(record, at); at += record.length; } - const end = new DataView(tail.buffer, size); + if (wide) { + const record = new DataView(tail.buffer, size, ZIP64_END.fixedBytes); + record.setUint32(0, ZIP64_END.signature, true); + record.setBigUint64( + ZIP64_END.sizeOffset, + BigInt(ZIP64_END.fixedBytes - ZIP64_END.sizePrefixBytes), + true, + ); + record.setUint16(12, 45, true); + record.setUint16(14, 45, true); + record.setBigUint64(ZIP64_END.entriesOnDiskOffset, BigInt(count), true); + record.setBigUint64(ZIP64_END.totalEntriesOffset, BigInt(count), true); + record.setBigUint64(ZIP64_END.directorySizeOffset, BigInt(size), true); + record.setBigUint64(ZIP64_END.directoryOffset, BigInt(offset), true); + const locator = new DataView( + tail.buffer, + size + ZIP64_END.fixedBytes, + ZIP64_LOCATOR.bytes, + ); + locator.setUint32(0, ZIP64_LOCATOR.signature, true); + locator.setBigUint64( + ZIP64_LOCATOR.recordOffset, + BigInt(offset) + BigInt(size), + true, + ); + locator.setUint32(16, 1, true); + } + const end = new DataView(tail.buffer, size + wideBytes, ZIP_EOCD_BYTES); end.setUint32(0, 0x06054b50, true); - end.setUint16(8, directory.records.length, true); - end.setUint16(10, directory.records.length, true); - end.setUint32(12, size, true); - // A declared offset past 32 bits cannot be right for a view-sized archive; the start stands in for it. - end.setUint32( - 16, - directory.declaredOffset <= 0xffffffff - ? directory.declaredOffset - : directory.start, - true, - ); + end.setUint16(8, Math.min(count, maxCount), true); + end.setUint16(10, Math.min(count, maxCount), true); + end.setUint32(12, Math.min(size, maxSize), true); + end.setUint32(16, Math.min(offset, maxOffset), true); return tail; }; diff --git a/src/domain/zip/zipExtraFields.ts b/src/domain/zip/zipExtraFields.ts new file mode 100644 index 0000000000..909864ae7b --- /dev/null +++ b/src/domain/zip/zipExtraFields.ts @@ -0,0 +1,34 @@ +import type { BinaryReader } from "../binaryContainer.js"; +import { err, ok, type Result } from "../result.js"; + +/** APPNOTE §4.5.1. The exported desktop reader uses the same field layout. */ +export const ZIP_EXTRA_FIELD = { + headerBytes: 4, + idOffset: 0, + sizeOffset: 2, + unicodePath: 0x7075, +} as const; + +/** The last matching payload, provided every framed field fits the extra block. */ +export const lastZipExtraField = ( + reader: BinaryReader, + start: number, + end: number, + kind: number, +): Result< + { readonly start: number; readonly end: number } | undefined, + "overrun" +> => { + let selected: { readonly start: number; readonly end: number } | undefined; + // Both desktop decoders have historically accepted a remainder shorter + // than a header. Only a complete header declares a payload span. + for (let at = start; at + ZIP_EXTRA_FIELD.headerBytes <= end;) { + const payload = at + ZIP_EXTRA_FIELD.headerBytes; + const next = payload + reader.u16(at + ZIP_EXTRA_FIELD.sizeOffset); + if (next > end) return err("overrun"); + if (reader.u16(at + ZIP_EXTRA_FIELD.idOffset) === kind) + selected = { start: payload, end: next }; + at = next; + } + return ok(selected); +}; diff --git a/src/domain/zipEndOfCentralDirectory.ts b/src/domain/zipEndOfCentralDirectory.ts index 8bf13ef80d..a377963b92 100644 --- a/src/domain/zipEndOfCentralDirectory.ts +++ b/src/domain/zipEndOfCentralDirectory.ts @@ -1,11 +1,88 @@ import { BinaryReader } from "./binaryContainer.js"; +import { err, ok, type Result } from "./result.js"; /** Classic ZIP EOCD fixed record and its format-defined comment limit. */ export const ZIP_EOCD_BYTES = 22; export const ZIP_MAX_COMMENT_BYTES = 0xffff; +/** Saturated classic directory fields defer to the Zip64 end record. */ +export const ZIP_CLASSIC_DIRECTORY_LIMITS = { + count: 0xffff, + size: 0xffffffff, + offset: 0xffffffff, +} as const; + +/** ZIP APPNOTE §§4.3.14–15. Shared fixed fields within the variable-size frame. */ +export const ZIP64_END = { + signature: 0x06064b50, + fixedBytes: 56, + sizePrefixBytes: 12, + sizeOffset: 4, + entriesOnDiskOffset: 24, + totalEntriesOffset: 32, + directorySizeOffset: 40, + directoryOffset: 48, +} as const; + +export const ZIP64_LOCATOR = { + signature: 0x07064b50, + bytes: 20, + recordOffset: 8, +} as const; + +type Zip64DirectoryRecord = { + readonly end: number; + readonly totalEntries: bigint; + readonly directorySize: bigint; + readonly directoryOffset: bigint; +}; + +/** Observe one complete Zip64 end record inside its declared carrier. */ +export const readZip64DirectoryRecord = ( + reader: BinaryReader, + offset: number, + limit: number, +): Result => { + const room = Math.min(limit, reader.bytes.length) - offset; + if (offset < 0 || room < ZIP64_END.fixedBytes) return err("fixed-fields"); + if (reader.u32(offset) !== ZIP64_END.signature) return err("signature"); + const bodyBytes = reader.word(offset + ZIP64_END.sizeOffset, 64); + if ( + bodyBytes < BigInt(ZIP64_END.fixedBytes - ZIP64_END.sizePrefixBytes) || + bodyBytes > BigInt(room - ZIP64_END.sizePrefixBytes) + ) + return err("size"); + return ok({ + end: offset + ZIP64_END.sizePrefixBytes + Number(bodyBytes), + totalEntries: reader.word(offset + ZIP64_END.totalEntriesOffset, 64), + directorySize: reader.word(offset + ZIP64_END.directorySizeOffset, 64), + directoryOffset: reader.word(offset + ZIP64_END.directoryOffset, 64), + }); +}; + const EOCD_SIGNATURE = Buffer.from([0x50, 0x4b, 0x05, 0x06]); +export const ZIP_CLASSIC_DISK_OFFSETS = { disk: 4, directoryDisk: 6 } as const; + +/** + * APPNOTE §§4.3.14–16 and §§4.4.19–20. The classic disk numbers can + * defer to Zip64 only after a Zip64 end record has been located. The two + * Zip64 layouts list 32-bit offsets and their required single-volume values. + * The exported desktop reader consumes the same rules in Python. + */ +export const ZIP_SINGLE_VOLUME = { + classic: [0], + zip64Classic: [0, 0xffff], + zip64Locator: [ + [4, 0], + [16, 1], + ], + zip64End: [ + [16, 0], + [20, 0], + ], +} as const; + export interface ZipEndOfCentralDirectory { /** Relative to the supplied byte window, which ends at the archive's end. */ readonly offset: number; @@ -18,6 +95,31 @@ export interface ZipEndOfCentralDirectory { readonly commentLength: number; } +/** Whether the observed classic end record admits the selected single volume. */ +export const zipEndUsesSingleVolume = ( + { + disk, + directoryDisk, + }: Pick, + layout: "classic" | "zip64Classic" = "classic", +): boolean => { + const allowed: readonly number[] = ZIP_SINGLE_VOLUME[layout]; + return allowed.includes(disk) && allowed.includes(directoryDisk); +}; + +/** Check the disk fields in the located Zip64 locator and end record. */ +export const zip64EndUsesSingleVolume = ( + reader: BinaryReader, + locator: number, + end: number, +): boolean => + ZIP_SINGLE_VOLUME.zip64Locator.every( + ([field, value]) => reader.u32(locator + field) === value, + ) && + ZIP_SINGLE_VOLUME.zip64End.every( + ([field, value]) => reader.u32(end + field) === value, + ); + /** * Observe complete classic records whose declared comment ends this window, * latest first. Candidate recognition does not establish directory validity, @@ -44,8 +146,10 @@ export function* zipEndOfCentralDirectoryCandidates( if (offset + ZIP_EOCD_BYTES + commentLength !== tail.length) continue; yield { offset: earliest + offset, - disk: reader.u16(offset + 4), - directoryDisk: reader.u16(offset + 6), + disk: reader.u16(offset + ZIP_CLASSIC_DISK_OFFSETS.disk), + directoryDisk: reader.u16( + offset + ZIP_CLASSIC_DISK_OFFSETS.directoryDisk, + ), entriesOnDisk: reader.u16(offset + 8), totalEntries: reader.u16(offset + 10), directorySize: reader.u32(offset + 12), diff --git a/src/domain/zipPackageFormat.ts b/src/domain/zipPackageFormat.ts index ce649c5212..43c39eb182 100644 --- a/src/domain/zipPackageFormat.ts +++ b/src/domain/zipPackageFormat.ts @@ -1,6 +1,7 @@ import { BinaryReader } from "./binaryContainer.js"; import { zipEndOfCentralDirectoryCandidates, + zipEndUsesSingleVolume, ZIP_EOCD_BYTES, ZIP_MAX_COMMENT_BYTES, } from "./zipEndOfCentralDirectory.js"; @@ -24,15 +25,24 @@ export const isZipPackageFormat = ( ): format is ZipPackageFormat => ZIP_PACKAGE_FORMATS.some((candidate) => candidate === format); -/** Recognize local-header, classic/Zip64 empty-archive, or split ZIP signatures, not validity. */ +/** Local-header, classic/Zip64 empty-archive, and split ZIP signatures. */ +export const ZIP_SIGNATURES = [ + [0x50, 0x4b, 0x03, 0x04], + [0x50, 0x4b, 0x05, 0x06], + [0x50, 0x4b, 0x06, 0x06], + [0x50, 0x4b, 0x07, 0x08], +] as const; + +/** A local file header can identify a ZIP before a complete end record is available. */ +export const hasZipLocalHeader = (bytes: Uint8Array): boolean => + ZIP_SIGNATURES[0].every((byte, index) => bytes[index] === byte); + +/** Recognize a ZIP signature, not directory validity or supported storage. */ export const hasZipSignature = (bytes: Uint8Array): boolean => bytes.length >= 4 && - bytes[0] === 0x50 && - bytes[1] === 0x4b && - ((bytes[2] === 0x03 && bytes[3] === 0x04) || - (bytes[2] === 0x05 && bytes[3] === 0x06) || - (bytes[2] === 0x06 && bytes[3] === 0x06) || - (bytes[2] === 0x07 && bytes[3] === 0x08)); + ZIP_SIGNATURES.some((signature) => + signature.every((byte, index) => bytes[index] === byte), + ); const CENTRAL_DIRECTORY_HEADER = 0x02014b50; const LOCAL_FILE_HEADER = 0x04034b50; @@ -68,8 +78,7 @@ export const locateTrailingZipArchive = async ( directoryOffset: declaredDirectory, } = candidate; if ( - candidate.disk !== 0 || - candidate.directoryDisk !== 0 || + !zipEndUsesSingleVolume(candidate) || entriesOnDisk === 0 || entriesOnDisk === 0xffff || entriesOnDisk !== candidate.totalEntries || diff --git a/tests/boundary/providers/desktopEmptyZip.test.ts b/tests/boundary/providers/desktopEmptyZip.test.ts new file mode 100644 index 0000000000..c767a3b8fb --- /dev/null +++ b/tests/boundary/providers/desktopEmptyZip.test.ts @@ -0,0 +1,153 @@ +import { execFileSync } from "node:child_process"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { gzipSync } from "node:zlib"; + +import { expect, it } from "vitest"; + +import { readZipView } from "../../../src/application/desktop/DesktopZipView.js"; +import { + PYTHON_HELPERS, + PYTHON_IMPORTS, +} from "../../../src/domain/desktop/appExportPython.js"; +import { PYTHON_TASK_EVALUATOR } from "../../../src/domain/desktop/appTaskPython.js"; +import { evaluateTaskChecks } from "../../../src/domain/desktop/appTaskChecks.js"; +import { detectStateFormat } from "../../../src/domain/desktop/stateFormats.js"; +import { + withZip64End, + zipArchive, +} from "../../../src/domain/desktop/zipArchive.fixture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const empty = zipArchive([]); +const commented = Buffer.concat([empty, Buffer.from("archive comment")]); +commented.writeUInt16LE(commented.length - 22, 20); +const archives = [ + ["classic", empty], + ["commented", commented], + ["Zip64 sentinels", withZip64End(empty)], + ["Zip64 ordinary counts", withZip64End(empty, false)], +] as const; + +const pythonViews = async (files: readonly Buffer[]) => { + const root = await createTestTempDirectory("desktop-empty-zip-"); + const paths = await Promise.all( + files.map(async (bytes, index) => { + const path = join(root, `${String(index)}.dat`); + await writeFile(path, bytes); + return path; + }), + ); + const imports = PYTHON_IMPORTS.split("\n") + .filter((line) => !line.includes("agentenv_protocol")) + .join("\n"); + const script = join(root, "read.py"); + await writeFile( + script, + `${imports}${PYTHON_HELPERS}${PYTHON_TASK_EVALUATOR} +check = {"id": "empty", "kind": "value_equals", "file": "archive.zip", "pointer": "/entry_count", "value": 0} +for path in json.load(sys.stdin): + view = _view(Path(path)) + print(json.dumps({"file": view, "passed": evaluate([check], {"archive.zip": view})[0]["result"]})) +`, + ); + return execFileSync( + process.platform === "win32" ? "python" : "python3", + ["-I", "-B", "-X", "utf8", script], + { input: JSON.stringify(paths), encoding: "utf8" }, + ) + .trim() + .split(/\r?\n/u) + .map( + (line) => + JSON.parse(line) as { + file: { format: string; view: unknown; error?: string }; + passed: boolean; + }, + ); +}; + +it.each(archives)( + "reads an empty %s archive without relying on its name", + async (_name, bytes) => { + const expected = { entries: [], entry_count: 0, truncated: false }; + expect(detectStateFormat("document.dat", bytes)).toBe("zip"); + expect(await readZipView(bytes)).toEqual({ ok: true, value: expected }); + expect((await pythonViews([bytes]))[0]).toEqual({ + file: { format: "zip", view: expected, truncated: false }, + passed: true, + }); + }, +); + +it("keeps empty nested archives viewless instead of recursing or calling them binary", async () => { + const outer = zipArchive( + archives.map(([name, data]) => ({ name: `${name}.zip`, data })), + ); + const native = await readZipView(outer); + if (!native.ok) throw new Error(native.error); + const python = (await pythonViews([outer]))[0]?.file; + expect(python?.format).toBe("zip"); + expect(python?.view).toEqual(native.value); + expect(native.value).toMatchObject({ + entries: archives.map(() => ({ + format: "zip", + view: null, + view_error: "an archive inside an archive is not read", + })), + entry_count: archives.length, + truncated: false, + }); +}); + +it("grades removing every entry as a change from a nonempty archive", async () => { + const files = [zipArchive([{ name: "old.txt", data: "old" }]), empty]; + const check = { + id: "empty", + kind: "value_equals" as const, + file: "archive.zip", + pointer: "/entry_count", + value: 0, + }; + const verdicts = await Promise.all( + files.map(async (bytes) => { + const read = await readZipView(bytes); + if (!read.ok) throw new Error(read.error); + return evaluateTaskChecks( + [check], + new Map([ + [ + check.file, + { format: detectStateFormat(check.file, bytes), view: read.value }, + ], + ]), + )[0]?.passed; + }), + ); + expect(verdicts).toEqual([false, true]); + expect((await pythonViews(files)).map((row) => row.passed)).toEqual(verdicts); +}); + +it("does not turn incomplete or damaged end records into empty views", async () => { + const incomplete = empty.subarray(0, 6); + expect(detectStateFormat("archive.zip", incomplete)).toBe("binary"); + const damaged = Buffer.from(empty); + damaged.writeUInt16LE(1, 20); + const zip64 = withZip64End(empty).subarray(0, 56); + for (const bytes of [damaged, zip64]) { + expect(detectStateFormat("archive.dat", bytes)).toBe("zip"); + expect((await readZipView(bytes)).ok).toBe(false); + const python = (await pythonViews([bytes]))[0]; + expect(python?.file).toMatchObject({ format: "zip", view: null }); + expect(python?.file.error).toBeTypeOf("string"); + expect(python?.passed).toBe(false); + } +}); + +it("reads the same empty archive when gzip wraps its bytes", async () => { + const [view] = await pythonViews([gzipSync(empty)]); + expect(view).toMatchObject({ + file: { format: "zip", compression: "gzip", view: { entry_count: 0 } }, + passed: true, + }); +}); diff --git a/tests/boundary/providers/desktopZipPython.test.ts b/tests/boundary/providers/desktopZipPython.test.ts index 0affdcabd4..ac5982a837 100644 --- a/tests/boundary/providers/desktopZipPython.test.ts +++ b/tests/boundary/providers/desktopZipPython.test.ts @@ -1,3 +1,4 @@ +import { execFileSync } from "node:child_process"; import { readFile } from "node:fs/promises"; import { join } from "node:path"; import { crc32 } from "node:zlib"; @@ -18,6 +19,11 @@ import { type ZipMember, } from "../../../src/domain/desktop/zipArchive.fixture.js"; import { zipEntryRefusal } from "../../../src/domain/desktop/zipView.js"; +import { + desktopViewPolicy, + pythonDesktopViewPolicy, +} from "../../../src/domain/desktop/desktopViewPolicy.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { expectSameCheckResults, expectSameFileViews, @@ -183,6 +189,33 @@ it("refuses invalid UTF-8 in the selected Unicode claim while preserving an unre }); }); +it("refuses a central extra field whose payload extends past the containing record's extra bytes", async () => { + const name = "report.txt"; + const opaque = Buffer.from([0xfe, 0xca, 2, 0, 1]); + const unicode = unicodePathExtra(name, "renamed.txt"); + unicode.writeUInt16LE(unicode.length - 3, 2); + const archives = [ + zipArchive([{ name, data: "observed", centralExtra: opaque }]), + zipArchive([{ name, data: "observed", centralExtra: unicode }]), + zipArchive([ + { name: "directory/", centralExtra: opaque }, + { name, data: "observed" }, + ]), + ]; + const fromPython = await pythonFileViews(saved(archives)); + for (const [index, archive] of archives.entries()) { + expect.soft(await readZipView(archive)).toEqual({ + ok: false, + error: "not a readable zip archive", + }); + expect.soft(fromPython[index]).toEqual({ + format: "zip", + view: null, + error: "not a readable zip archive", + }); + } +}); + it("excludes a final Unicode directory without consuming a retained file slot", async () => { const name = "settings.json"; const archive = zipArchive([ @@ -401,6 +434,153 @@ describe("zip directories in the exported module", () => { }); }); + it.each([false, true])( + "follows the Zip64 locator over a fixed-size signature in opaque extension bytes (%s)", + async (sentinels) => { + const extension = Buffer.alloc(6 + 56); + extension.writeUInt16LE(0xfffe, 0); + extension.writeUInt32LE(56, 2); + extension.writeUInt32LE(0x06064b50, 6); + const archive = withZip64End( + zipArchive([{ name: "notes.txt", data: "retained" }]), + sentinels, + extension, + ); + const result = await sameView(archive); + expect(result).toMatchObject({ + entry_count: 1, + entries: [{ name: "notes.txt", view: { lines: ["retained"] } }], + truncated: false, + }); + }, + ); + + it("retains the fixed Zip64 fallback when a prefix shifts its relative locator", async () => { + // Keep a recognized ZIP opening for desktop format detection. Only the + // real recorded local entry and central directory contribute to the view. + const prefix = Buffer.from("PK\x03\x04inert prefix"); + const archive = Buffer.concat([ + prefix, + withZip64End(zipArchive([{ name: "notes.txt", data: "retained" }])), + ]); + const result = await sameView(archive); + expect(result).toMatchObject({ + entry_count: 1, + entries: [{ name: "notes.txt", view: { lines: ["retained"] } }], + truncated: false, + }); + }); +}); + +describe("ZIP directory counts and volume admission in the exported module", () => { + it("retains the selected ZIP count at and beyond the classic sentinel", async () => { + const path = join( + await createTestTempDirectory("rea-zip64-count-"), + "many.zip", + ); + // An independent writer emits an original Zip64 archive, not a decoder-copy fixture. + execFileSync("python3", [ + "-I", + "-B", + "-c", + `import sys, zipfile +with zipfile.ZipFile(sys.argv[1], "w") as archive: + for index in range(65536): + archive.writestr(f"entry-{index:05d}.bin", b"ab") +with zipfile.ZipFile(sys.argv[1]) as archive: + assert len(archive.infolist()) == 65536`, + path, + ]); + const bytes = await readFile(path); + for (const keep of [65_535, 65_536]) { + const policy = desktopViewPolicy({ + zip_entries: keep, + zip_entry_bytes: 1, + }); + const result = await readZipView(bytes, undefined, policy); + const host = result.ok + ? (result.value as { + entries: { name: string }[]; + entry_count: number; + truncated: boolean; + }) + : null; + const python: unknown = JSON.parse( + exportedPython( + `${pythonDesktopViewPolicy(policy)} +result = _view(Path(json.load(sys.stdin))) +view = result.get("view") +print(json.dumps(None if view is None else { + "listed": len(view["entries"]), "count": view["entry_count"], + "truncated": view["truncated"], "last": view["entries"][-1]["name"] +}))`, + path, + ), + ); + expect([ + host === null + ? null + : { + listed: host.entries.length, + count: host.entry_count, + truncated: host.truncated, + last: host.entries.at(-1)?.name, + }, + python, + ]).toEqual( + Array.from({ length: 2 }, () => ({ + listed: keep, + count: 65_536, + truncated: keep < 65_536, + last: `entry-${String(keep - 1).padStart(5, "0")}.bin`, + })), + ); + } + }); + + it("refuses archives whose selected directory or Zip64 locator names another volume", async () => { + const classic = zipArchive([{ name: "settings.json", data: '{"zoom":2}' }]); + const zip64 = withZip64End(classic); + const variants = [ + ["classic disk", classic, 22, 4, 1], + ["classic directory disk", classic, 22, 6, 1], + ["classic disk in Zip64", zip64, 22, 4, 1], + ["Zip64 record disk", zip64, 98, 16, 1], + ["Zip64 directory disk", zip64, 98, 20, 1], + ["Zip64 locator disk", zip64, 42, 4, 1], + ["Zip64 locator disk count", zip64, 42, 16, 2], + ] as const; + const archives = variants.map(([, bytes, end, field, value]) => { + const copy = Buffer.from(bytes); + copy.writeUInt16LE(value, copy.length - end + field); + return copy; + }); + const python = await pythonFileViews(saved(archives)); + for (const [index, [description]] of variants.entries()) { + expect(await readZipView(archives[index]!), description).toEqual({ + ok: false, + error: "not a readable zip archive", + }); + expect(python[index], description).toEqual({ + format: "zip", + view: null, + error: "not a readable zip archive", + }); + } + }); + + it("uses the Zip64 record when the classic disk numbers defer to it", async () => { + const archive = withZip64End( + zipArchive([{ name: "settings.json", data: '{"zoom":2}' }]), + ); + archive.writeUInt16LE(0xffff, archive.length - 18); + archive.writeUInt16LE(0xffff, archive.length - 16); + expect(await sameView(archive)).toMatchObject({ + entries: [{ name: "settings.json", view: { zoom: 2 } }], + entry_count: 1, + }); + }); + it("keeps the classic values when bytes before the end record only look like a Zip64 locator", async () => { // The last name ends in a locator signature and 16 more bytes, which sit just before the end record. const name = `c-PK\u0006\u0007${"0".repeat(16)}`;