fix(zip): share desktop and APK framing and preserve bounded ZIP64 views (#12157)

* fix(desktop): reject split ZIP volume metadata

Refs #12137

* refactor(zip): read Zip64 disk fields through BinaryReader

* fix(desktop): validate ZIP extra fields before selection

Refs #12146

* fix(desktop): keep bounded ZIP pairing compatible with Python 3.9

* fix(desktop): retain exact counts in bounded Zip64 views

* fix(desktop): follow the declared Zip64 directory frame

* fix(desktop): recognize empty ZIP state views (#12163)

* fix(desktop): recognize empty ZIP state views

(cherry picked from commit 263b25bd34)

* fix(desktop): reuse empty ZIP admission for wrapped documents

* refactor(zip): share the local-header signature with desktop state

---------

Co-authored-by: wunianze666-netizen <wunianze666@gmail.com>

* test(zip): inherit the portable ZIP count deadline

---------

Co-authored-by: wunianze666-netizen <wunianze666@gmail.com>
This commit is contained in:
morlutoandwunianze666-netizen authored and GitHub committed 2026-11-19 03:33:31 +00:00
1 parent 27e04ea2ec
commit 3d313b56bc
17 files changed
+840 -224

No files matched your search

+4 -1
View File
@@ -562,7 +562,10 @@ as their entries, read with Python's `zipfile` from the file in place, with
REA's entry, size, compression ratio and archive limits and its wording for an
entry it does not read. The module reads the central directory itself, as REA
does, and shows `zipfile` only the records it lists, reading each entry in
bounded chunks no further than a byte past its declared size. A gzip file is
bounded chunks no further than a byte past its declared size. The reader
follows a Zip64 locator to the complete end-record span and skips its
extension bytes before passing the bounded directory to `zipfile`.
A gzip file is
the view of the document inside it, with `"compression": "gzip"` beside its
format, decompressed with Python's `zlib` within the same entry size and ratio
limits, its members joined as REA joins them. A JSON file
+16 -12
View File
@@ -29,6 +29,9 @@ A path absent from the capture is an input error.
`sqlite_tables`, `sqlite_rows`, `sqlite_view_bytes`, `text_lines`, `xml_nodes`, `xml_depth`,
`parsed_bytes`, `json_depth`, `zip_entries`, `zip_entry_bytes`, `zip_bytes`,
and `zip_ratio`. Unspecified controls keep the defaults described below.
A request that raises `zip_entries` past the classic ZIP count range still uses
the selected entry limit. The host and exported readers encode the retained
directory's exact count with Zip64 fields when the classic fields are full.
`xml_depth` admits at most 494 and `json_depth` at most 991: these bounds
follow the fixed Evidence JSON nesting limit, including the captured-version
and ZIP-member wrappers and XML's element/children representation. A request
@@ -138,16 +141,16 @@ Views hold the captured bytes as they are, the capture host's name and user
included; the export removes those from the bundle. A version records
whether the file was present, the SHA-256 of its retained bytes, and a view:
| Format | Detected by | View |
| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| `sqlite` | The `SQLite format 3` header | Every ordinary table's columns, primary key and up to 500 rows |
| `dconf` | The `GVariant` header | Groups whose path names the application, plus the names of the other groups |
| `zip` | A zip local file header, `PK\x03\x04`, whatever the name | Its file entries in archive order, each with its own format and view |
| `json` | A `.json` name or a leading `{` or `[` that parses | The parsed value, from files up to 4 MiB nested up to 500 levels |
| `xml` | An XML name, a declaration or a leading element that parses | An element tree (`name`, `attributes`, `children`); text kept exactly, DTDs never read |
| `keyfile` | Only `[group]`, `key=value` and comment lines, with at least one group | Groups of `key` → string value, from files up to 4 MiB |
| `text` | Other decoded text | Lines, up to 5,000 |
| `binary` | Content that cannot be read as text | None |
| Format | Detected by | View |
| --------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
| `sqlite` | The `SQLite format 3` header | Every ordinary table's columns, primary key and up to 500 rows |
| `dconf` | The `GVariant` header | Groups whose path names the application, plus the names of the other groups |
| `zip` | A ZIP local file header or an empty archive signature, whatever the name | Its file entries in archive order, each with its own format and view |
| `json` | A `.json` name or a leading `{` or `[` that parses | The parsed value, from files up to 4 MiB nested up to 500 levels |
| `xml` | An XML name, a declaration or a leading element that parses | An element tree (`name`, `attributes`, `children`); text kept exactly, DTDs never read |
| `keyfile` | Only `[group]`, `key=value` and comment lines, with at least one group | Groups of `key` → string value, from files up to 4 MiB |
| `text` | Other decoded text | Lines, up to 5,000 |
| `binary` | Content that cannot be read as text | None |
Text views decode UTF-8 or strictly valid UTF-16 with a leading byte-order
mark. UTF-16 containing a NUL character stays binary. These rules also apply
@@ -275,8 +278,9 @@ database, or an archive, inside an archive keeps its format and `sha256` and
has a `view_error` instead of a view: it is not read. The view lists at most
1,000 entries; `entry_count` counts them all, and `truncated` is true when some
are not listed. REA reads the central directory itself, a record at a time,
through the Zip64 end record whenever a Zip64 locator precedes the end record,
as Python's `zipfile` does, and hands the zip reader only the records it lists,
and follows a Zip64 locator to the complete end-record span, including any
extension bytes. The fixed-record fallback supports prepended archives
whose offsets omit the prefix. REA hands the zip reader only the records it lists,
so an archive of hundreds of thousands of entries costs no more to read
than one of a thousand. An archive whose central directory cannot be read has a
`view_error` of its own.
@@ -1,6 +1,7 @@
# Plan
1. Format and reader: `zip` detected by its local file header; the text, JSON,
1. Format and reader: `zip` detected by its local file header or an empty
classic/Zip64 archive signature; the text, JSON,
key file and XML readers shared as `textualView`, so a file and an entry
read alike; a domain module for the entry limits, the entry view and the
archive view, and an application reader that walks the verified bytes with
@@ -13,8 +13,9 @@ evaluators, REA's and the exported verifier's, agree.
## Requirements
- Format: a file starting with a zip local file header (`PK\x03\x04`) is
`zip`, whatever its name. Its view is
- Format: a file starting with a zip local file header (`PK\x03\x04`), or
a classic/Zip64 empty-archive signature with at least 22 bytes, is `zip`,
whatever its name. A signature is not proof of directory validity. Its view is
`{entries, entry_count, truncated}`: the file entries in archive order,
directories left out, each `{name, size, sha256, format, view, view_error}`.
- Reading: from the retained bytes already checked against their digest, in
@@ -40,7 +40,7 @@ generation reads the kinds of change.
only when it has no internal subset and its DTD never read
([XML text checks](../xml-text-checks/spec.md)); key files up to 4 MiB
whose every line parses; zip archives by
their local file header, as their entries with each entry's own format and
their local file header or empty archive signature, as their entries with each entry's own format and
view ([document views](../document-views/spec.md)); text; binary. A gzip
file, told by its magic bytes, is read as the document inside, its members
joined, within a zip entry's 16 MiB and 100-times limits, and its version
@@ -13,7 +13,7 @@
"src/artifacts/nodeSea/NodeSeaLocation.ts": 2,
"src/artifacts/pyinstaller/CArchiveLayout.ts": 8,
"src/domain/apkSignature/apkSignatureSchemes.ts": 1,
"src/domain/apkSignature/apkSigningBlock.ts": 7,
"src/domain/apkSignature/apkSigningBlock.ts": 2,
"src/domain/apkSignature/apkV1Signature.ts": 15,
"src/domain/apple/appleMachoFixups.ts": 22,
"src/domain/apple/appleMachoSelection.ts": 2,
@@ -32,7 +32,6 @@
"src/domain/binaryTarget.ts": 10,
"src/domain/desktop/elfDependencies.ts": 4,
"src/domain/desktop/openPgpKeys.ts": 2,
"src/domain/desktop/zipDirectory.ts": 14,
"src/domain/dosMz.ts": 10,
"src/domain/elfDynamicEntry.ts": 4,
"src/domain/elfFixedHeader.ts": 4,
+59 -49
View File
@@ -1,6 +1,12 @@
import { BinaryReader } from "../binaryContainer.js";
import {
readZip64DirectoryRecord,
zipEndOfCentralDirectoryCandidates,
zipEndUsesSingleVolume,
ZIP_CLASSIC_DIRECTORY_LIMITS,
ZIP_EOCD_BYTES,
ZIP64_END,
ZIP64_LOCATOR,
type ZipEndOfCentralDirectory,
} from "../zipEndOfCentralDirectory.js";
@@ -20,11 +26,6 @@ import {
*/
import { type Result, err, ok } from "../result.js";
const ZIP64_EOCD_LOCATOR_SIGNATURE = 0x07064b50;
const ZIP64_EOCD_SIGNATURE = 0x06064b50;
const ZIP64_SENTINEL = 0xffffffff;
/** The 16-bit EOCD entry count saturates here when Zip64 carries the real count. */
const ZIP64_ENTRIES_SENTINEL = 0xffff;
/** "APK Sig Block 42" — the 16-byte block magic (JAR/APK signing block spec). */
const BLOCK_MAGIC = "APK Sig Block 42";
/** A single pair is at least 12 bytes (uint64 length + uint32 id); cap the count. */
@@ -123,7 +124,7 @@ interface CentralDirectory {
const classicDirectory = (
eocd: ZipEndOfCentralDirectory,
): Result<CentralDirectory, ApkSigningBlockError> => {
if (eocd.disk !== 0 || eocd.directoryDisk !== 0)
if (!zipEndUsesSingleVolume(eocd))
return err({
kind: "multi-disk-unsupported",
detail: `classic EOCD names disk ${String(eocd.disk)} and central-directory disk ${String(eocd.directoryDisk)}`,
@@ -147,7 +148,6 @@ const centralDirectoryOffset = (
bytes: Uint8Array,
eocd: ZipEndOfCentralDirectory,
): Result<CentralDirectory, ApkSigningBlockError> => {
const dv = view(bytes);
const {
offset: eocdOffset,
totalEntries: entries,
@@ -155,19 +155,24 @@ const centralDirectoryOffset = (
directoryOffset: offset,
} = eocd;
const saturated =
offset === ZIP64_SENTINEL ||
size === ZIP64_SENTINEL ||
entries === ZIP64_ENTRIES_SENTINEL;
offset === ZIP_CLASSIC_DIRECTORY_LIMITS.offset ||
size === ZIP_CLASSIC_DIRECTORY_LIMITS.size ||
entries === ZIP_CLASSIC_DIRECTORY_LIMITS.count;
if (!saturated) return classicDirectory(eocd);
const reader = new BinaryReader(
Buffer.from(bytes.buffer, bytes.byteOffset, bytes.byteLength),
true,
"zip",
);
// Zip64: a locator precedes the EOCD, pointing at the Zip64 EOCD record.
const locator = eocdOffset - 20;
if (
locator < 0 ||
dv.getUint32(locator, true) !== ZIP64_EOCD_LOCATOR_SIGNATURE
) {
const locator = eocdOffset - ZIP64_LOCATOR.bytes;
if (locator < 0 || reader.u32(locator) !== ZIP64_LOCATOR.signature) {
// Exactly 65,535 entries saturates the count without Zip64; that archive
// is still described completely by its 32-bit EOCD fields.
if (offset !== ZIP64_SENTINEL && size !== ZIP64_SENTINEL)
if (
offset !== ZIP_CLASSIC_DIRECTORY_LIMITS.offset &&
size !== ZIP_CLASSIC_DIRECTORY_LIMITS.size
)
return classicDirectory(eocd);
return err({
kind: "malformed-zip",
@@ -177,8 +182,8 @@ const centralDirectoryOffset = (
}
// APPNOTE 6.3.10 §4.3.15 makes the record offset relative to its disk.
// A selected file cannot resolve another disk's offset as a local span.
const recordDisk = dv.getUint32(locator + 4, true);
const totalDisks = dv.getUint32(locator + 16, true);
const recordDisk = reader.u32(locator + 4);
const totalDisks = reader.u32(locator + 16);
if (totalDisks === 0)
return err({
kind: "malformed-zip",
@@ -196,46 +201,51 @@ const centralDirectoryOffset = (
kind: "zip64-unsupported",
detail: "multi-disk Zip64 archives are not supported",
});
const recordOffset = u64(dv, locator + 8);
if (recordOffset === null)
const offsetOnDisk = reader.word(locator + ZIP64_LOCATOR.recordOffset, 64);
const maximumExact = BigInt(Number.MAX_SAFE_INTEGER);
if (offsetOnDisk > maximumExact)
return err({
kind: "zip64-unsupported",
detail: "Zip64 EOCD offset exceeds safe range",
});
if (recordOffset + 56 > locator)
return err({
kind: "malformed-zip",
offset: locator + 8,
detail: "Zip64 EOCD record extends beyond its locator",
});
if (dv.getUint32(recordOffset, true) !== ZIP64_EOCD_SIGNATURE)
return err({
kind: "malformed-zip",
offset: recordOffset,
detail: "invalid Zip64 EOCD signature",
});
// The size excludes the signature and size field. Check the declared
// body before using its fixed fields to locate the central directory.
const recordSize = dv.getBigUint64(recordOffset + 4, true);
if (recordSize < 44n || recordSize > BigInt(locator - recordOffset - 12))
return err({
kind: "malformed-zip",
offset: recordOffset + 4,
detail: "invalid Zip64 EOCD record size",
});
const z64Offset = u64(dv, recordOffset + 48);
const z64Size = u64(dv, recordOffset + 40);
// The Zip64 record's total entry count; the 16-bit EOCD field saturates.
const z64Entries = u64(dv, recordOffset + 32);
if (z64Offset === null || z64Size === null || z64Entries === null)
const recordOffset = Number(offsetOnDisk);
const record = readZip64DirectoryRecord(reader, recordOffset, locator);
if (!record.ok) {
switch (record.error) {
case "fixed-fields":
return err({
kind: "malformed-zip",
offset: locator + ZIP64_LOCATOR.recordOffset,
detail: "Zip64 EOCD record extends beyond its locator",
});
case "signature":
return err({
kind: "malformed-zip",
offset: recordOffset,
detail: "invalid Zip64 EOCD signature",
});
case "size":
return err({
kind: "malformed-zip",
offset: recordOffset + ZIP64_END.sizeOffset,
detail: "invalid Zip64 EOCD record size",
});
}
}
const { directoryOffset, directorySize, totalEntries } = record.value;
if (
directoryOffset > maximumExact ||
directorySize > maximumExact ||
totalEntries > maximumExact
)
return err({
kind: "zip64-unsupported",
detail: "Zip64 sizes exceed safe range",
});
return ok({
offset: z64Offset,
size: z64Size,
entries: z64Entries,
offset: Number(directoryOffset),
size: Number(directorySize),
entries: Number(totalEntries),
end: recordOffset,
});
};
+11 -2
View File
@@ -1,3 +1,5 @@
import { ZIP_EOCD_BYTES } from "../zipEndOfCentralDirectory.js";
import { ZIP_SIGNATURES } from "../zipPackageFormat.js";
import { XML_ELEMENT_OPENING } from "../xmlDocument.js";
import {
DESKTOP_VIEW_DEFAULTS,
@@ -90,7 +92,9 @@ LINE_BREAK = re.compile(${JSON.stringify(DESKTOP_LINE_BREAK)})
ECMASCRIPT_TRIM_CHARACTERS = ${PYTHON_ECMASCRIPT_TRIM_CHARACTERS}
SQLITE_HEADER = b"SQLite format 3\x00"
GVDB_HEADER = b"GVariant"
ZIP_HEADER = b"PK\x03\x04"
ZIP_SIGNATURES = (${ZIP_SIGNATURES.map((signature) => `bytes(${JSON.stringify(signature)})`).join(", ")})
ZIP_HEADER = ZIP_SIGNATURES[0]
ZIP_EOCD_BYTES = ${String(ZIP_EOCD_BYTES)}
GZIP_HEADER = b"\x1f\x8b\x08"
XML_SPACE = "xml:space"
MAX_SAFE_INTEGER = 2**53 - 1
@@ -428,6 +432,11 @@ def _bytes_view(name: str, data: bytes) -> dict:
return _text_view(name, text, len(data))
def _is_zip(data: bytes) -> bool:
"""Recognize storage signatures before the directory reader checks validity."""
return data.startswith(ZIP_HEADER) or (len(data) >= ZIP_EOCD_BYTES and data.startswith(ZIP_SIGNATURES))
def _view(path: Path) -> dict:
"""A typed JSON view of one file in the shapes of REA's state view; truncated says the view is partial."""
too_large = {"format": None, "view": None, "error": VIEW_TOO_LARGE}
@@ -446,7 +455,7 @@ def _view(path: Path) -> dict:
return too_large
if data.startswith(SQLITE_HEADER):
return _sqlite(path)
if data.startswith(ZIP_HEADER):
if _is_zip(data):
with path.open("rb") as file:
return _zip(file, os.fstat(file.fileno()).st_size)
if data.startswith(GZIP_HEADER):
+112 -53
View File
@@ -1,3 +1,12 @@
import {
ZIP_CLASSIC_DIRECTORY_LIMITS,
ZIP_CLASSIC_DISK_OFFSETS,
ZIP_SINGLE_VOLUME,
ZIP64_END,
ZIP64_LOCATOR,
} from "../zipEndOfCentralDirectory.js";
import { ZIP_EXTRA_FIELD } from "../zip/zipExtraFields.js";
/**
* The exported module's zip and gzip readers: the central directory read a
* record at a time, entries read in bounded chunks, a gzip file's document
@@ -12,9 +21,14 @@ ZIP_END = b"PK\x05\x06"
ZIP_CENTRAL = b"PK\x01\x02"
ZIP64_LOCATOR = b"PK\x06\x07"
ZIP64_END = b"PK\x06\x06"
ZIP64_END_LAYOUT = ${JSON.stringify(ZIP64_END)}
ZIP64_LOCATOR_LAYOUT = ${JSON.stringify(ZIP64_LOCATOR)}
ZIP_CLASSIC_DIRECTORY_LIMITS = ${JSON.stringify(ZIP_CLASSIC_DIRECTORY_LIMITS)}
ZIP_CLASSIC_DISK_OFFSETS = ${JSON.stringify(Object.values(ZIP_CLASSIC_DISK_OFFSETS))}
ZIP_SINGLE_VOLUME = ${JSON.stringify(ZIP_SINGLE_VOLUME)}
ZIP_EXTRA_FIELD = ${JSON.stringify(ZIP_EXTRA_FIELD)}
#: Formats an archive entry may have whose reader needs a tool, a worker or another archive opened.
NOT_READ_INSIDE = ((SQLITE_HEADER, "sqlite", "a SQLite database"), (GVDB_HEADER, "dconf", "a dconf database"),
(ZIP_HEADER, "zip", "an archive"))
NOT_READ_INSIDE = ((SQLITE_HEADER, "sqlite", "a SQLite database"), (GVDB_HEADER, "dconf", "a dconf database"))
def _entry_view(name: str, data: bytes) -> dict:
@@ -23,6 +37,8 @@ def _entry_view(name: str, data: bytes) -> dict:
for header, form, inside in NOT_READ_INSIDE:
if data.startswith(header):
return {"format": form, "view": None, "error": f"{inside} inside an archive is not read"}
if _is_zip(data):
return {"format": "zip", "view": None, "error": "an archive inside an archive is not read"}
return _bytes_view(name, data)
@@ -74,11 +90,24 @@ def _zip_entry(archive, info, refusal, name):
"view": view.get("view"), "view_error": view.get("error")}
def _zip64_directory_record(file, record: int, locator: int):
"""Read the fixed fields only, after the entire declared record fits before the locator."""
fixed = ZIP64_END_LAYOUT["fixedBytes"]
prefix = ZIP64_END_LAYOUT["sizePrefixBytes"]
if record < 0 or record + fixed > locator:
return None
file.seek(record)
header = file.read(fixed)
if len(header) != fixed or header[:4] != ZIP64_END:
return None
body = struct.unpack_from("<Q", header, ZIP64_END_LAYOUT["sizeOffset"])[0]
if body < fixed - prefix or record + prefix + body != locator:
return None
return header
def _zip_end(file, size: int):
"""Where an archive's central directory ends, how long it is and the offset it declares, from the last end
record whose comment reaches the end of the file and, when a Zip64 locator precedes it whatever that record
holds, the Zip64 one just before the locator; None without an end record, or when sentinels defer to a
Zip64 record that is missing."""
"""Read the classic end and any declared Zip64 frame from the single supplied volume."""
window = min(size, 22 + 0xFFFF)
file.seek(size - window)
tail = file.read(window)
@@ -88,32 +117,65 @@ def _zip_end(file, size: int):
if at < 0:
return None
total, length, offset = struct.unpack_from("<HII", tail, at + 10)
classic_disks = [struct.unpack_from("<H", tail, at + field)[0] for field in ZIP_CLASSIC_DISK_OFFSETS]
end = size - window + at
record = end - 20 - 56
if record < 0:
return end, length, offset
file.seek(end - 20)
if file.read(4) != ZIP64_LOCATOR:
return end, length, offset
file.seek(record)
zip64 = file.read(56)
if len(zip64) == 56 and zip64[:4] == ZIP64_END:
length, offset = struct.unpack_from("<QQ", zip64, 40)
classic = (end, length, offset) if all(disk in ZIP_SINGLE_VOLUME["classic"] for disk in classic_disks) else None
locator_at = end - ZIP64_LOCATOR_LAYOUT["bytes"]
if locator_at < 0:
return classic
file.seek(locator_at)
locator = file.read(ZIP64_LOCATOR_LAYOUT["bytes"])
if len(locator) != ZIP64_LOCATOR_LAYOUT["bytes"] or locator[:4] != ZIP64_LOCATOR:
return classic
declared = struct.unpack_from("<Q", locator, ZIP64_LOCATOR_LAYOUT["recordOffset"])[0]
fallback = locator_at - ZIP64_END_LAYOUT["fixedBytes"]
# The fixed fallback preserves archives whose offsets omit prepended bytes.
for record in (declared, fallback):
zip64 = _zip64_directory_record(file, record, locator_at)
if zip64 is None:
continue
if (any(disk not in ZIP_SINGLE_VOLUME["zip64Classic"] for disk in classic_disks)
or any(struct.unpack_from("<I", locator, field)[0] != value
for field, value in ZIP_SINGLE_VOLUME["zip64Locator"])
or any(struct.unpack_from("<I", zip64, field)[0] != value
for field, value in ZIP_SINGLE_VOLUME["zip64End"])):
return None
length = struct.unpack_from("<Q", zip64, ZIP64_END_LAYOUT["directorySizeOffset"])[0]
offset = struct.unpack_from("<Q", zip64, ZIP64_END_LAYOUT["directoryOffset"])[0]
return record, length, offset
if total == 0xFFFF or 0xFFFFFFFF in (length, offset):
limits = ZIP_CLASSIC_DIRECTORY_LIMITS
if total == limits["count"] or length == limits["size"] or offset == limits["offset"]:
return None
return end, length, offset
return classic
def _zip_unicode_path(raw: bytes, flags: int, extra: bytes):
"""The last Info-ZIP Unicode Path, validated only after it replaces earlier claims, as in zip.js."""
at, data, length = 0, None, 0
while at + 4 <= len(extra):
kind, size = struct.unpack_from("<HH", extra, at)
if kind == 0x7075:
data, length = extra[at + 4:at + 4 + size], size
at += 4 + size
usable = (data is not None and length >= 5 and len(data) == length and data[0] == 1 and not flags & 0x800
def _zip_extras(extra: bytes):
"""Check all framed fields before selection and return the last Unicode Path and the other original bytes."""
at, copied, selected, kept = 0, 0, None, None
header = ZIP_EXTRA_FIELD["headerBytes"]
while at + header <= len(extra):
payload = at + header
size = struct.unpack_from("<H", extra, at + ZIP_EXTRA_FIELD["sizeOffset"])[0]
end = payload + size
if end > len(extra):
raise zipfile.BadZipFile("Corrupt extra field")
kind = struct.unpack_from("<H", extra, at + ZIP_EXTRA_FIELD["idOffset"])[0]
if kind == ZIP_EXTRA_FIELD["unicodePath"]:
selected = (payload, end)
if kept is None:
kept = bytearray()
kept += extra[copied:at]
copied = end
at = end
if selected is None:
return None, extra
kept += extra[copied:]
return extra[selected[0]:selected[1]], kept
def _zip_unicode_path(raw: bytes, flags: int, data):
"""Use the last Info-ZIP Unicode Path only when it binds to the stored name, as zip.js does."""
usable = (data is not None and len(data) >= 5 and data[0] == 1 and not flags & 0x800
and int.from_bytes(data[1:5], "little") == zlib.crc32(raw))
return data[5:] if usable else None
@@ -125,24 +187,13 @@ def _zip_name(raw: bytes, flags: int, unicode_path):
return raw.decode("cp437") if unicode_path is None else unicode_path.decode("utf-8")
def _zip_kept_record(fixed: bytes, rest: bytes, name_length: int, extra_length: int):
def _zip_kept_record(fixed: bytes, raw: bytes, extra, comment: bytes):
"""Remove already-interpreted Unicode claims from the bounded decoder copy. zipfile otherwise interprets each
occurrence, retaining an earlier valid name or rejecting invalid UTF-8 that zip.js already superseded.
All other fields, comments and the original artifact bytes remain unchanged."""
raw, extra = rest[:name_length], rest[name_length:name_length + extra_length]
kept, at = bytearray(), 0
while at + 4 <= len(extra):
kind, length = struct.unpack_from("<HH", extra, at)
end = at + 4 + length
if end > len(extra):
raise zipfile.BadZipFile("Corrupt extra field")
if kind != 0x7075:
kept += extra[at:end]
at = end
kept += extra[at:]
header = bytearray(fixed)
struct.pack_into("<H", header, 30, len(kept))
return header + raw + kept + rest[name_length + extra_length:]
struct.pack_into("<H", header, 30, len(extra))
return header + raw + extra + comment
def _zip_directory(file, size: int, keep: int):
@@ -169,12 +220,14 @@ def _zip_directory(file, size: int, keep: int):
at += 46 + name_length + extra_length + comment_length
if len(rest) < name_length + extra_length + comment_length or at > end:
return None
unicode_path = _zip_unicode_path(rest[:name_length], flags, rest[name_length:name_length + extra_length])
name = _zip_name(rest[:name_length], flags, unicode_path)
extra_end = name_length + extra_length
raw = rest[:name_length]
selected, extra = _zip_extras(rest[name_length:extra_end])
name = _zip_name(raw, flags, _zip_unicode_path(raw, flags, selected))
if not (name.endswith("/") or (external >> 16) & 0o170000 == 0o040000 or (made_by >> 8 == 0 and external & 0x10)):
count += 1
if len(records) < keep:
records.append(_zip_kept_record(fixed, rest, name_length, extra_length))
records.append(_zip_kept_record(fixed, raw, extra, rest[extra_end:]))
names.append(name)
return start, declared, records, count, names
@@ -222,16 +275,22 @@ def _zip(file, size: int) -> dict:
if directory is None:
raise zipfile.BadZipFile("no readable central directory")
start, declared, records, count, names = directory
kept = b"".join(records)
offset = declared if declared <= 0xFFFFFFFF else start
# zipfile follows a Zip64 locator signature wherever one sits before the end record, and the kept records
# may end in those bytes, so a real Zip64 end record and locator for them come first.
kept, retained = b"".join(records), len(records)
limit = ZIP_CLASSIC_DIRECTORY_LIMITS
offset = declared if declared <= limit["offset"] else start
classic_count = min(retained, limit["count"])
# zipfile follows a Zip64 locator signature wherever one sits before the end record, including in
# a kept record's comment. Give it real Zip64 records with the exact selected directory facts.
tail = (kept
+ struct.pack("<4sQ2H2I4Q", ZIP64_END, 44, 45, 45, 0, 0, len(records), len(records), len(kept), offset)
+ struct.pack("<4sQ2H2I4Q", ZIP64_END, 44, 45, 45, 0, 0, retained, retained, len(kept), offset)
+ struct.pack("<4sIQI", ZIP64_LOCATOR, 0, offset + len(kept), 1)
+ struct.pack("<4s4H2IH", ZIP_END, 0, 0, len(records), len(records), len(kept), offset, 0))
+ struct.pack("<4s4H2IH", ZIP_END, 0, 0, classic_count, classic_count,
min(len(kept), limit["size"]), min(offset, limit["offset"]), 0))
with zipfile.ZipFile(_BoundedArchive(file, start, tail)) as archive:
for info, name in zip(archive.infolist(), names, strict=True):
infos = archive.infolist()
if len(infos) != len(names):
raise zipfile.BadZipFile("retained member count differs from decoded member count")
for info, name in zip(infos, names):
refusal = _zip_refusal(info, spent)
spent += info.file_size if refusal is None else 0
entries.append(_zip_entry(archive, info, refusal, name))
@@ -279,8 +338,8 @@ def _gzip(name: str, data: bytes) -> dict:
if MAX_ZIP_RATIO * len(data) < MAX_ZIP_ENTRY_BYTES
else f"larger than the {_view_byte_limit(MAX_ZIP_ENTRY_BYTES)} a compressed document's view reads"}
for header, form, inside in NOT_READ_INSIDE:
if form != "zip" and document.startswith(header):
if document.startswith(header):
return {**compressed, "format": form, "error": f"{inside} inside a gzip file is not read"}
view = _zip(io.BytesIO(document), len(document)) if document.startswith(ZIP_HEADER) else _bytes_view(name, document)
view = _zip(io.BytesIO(document), len(document)) if _is_zip(document) else _bytes_view(name, document)
return {**view, "compression": "gzip"}
`;
+7 -3
View File
@@ -9,6 +9,8 @@ import {
} from "./desktopViewPolicy.js";
import { err, ok, type Result } from "../result.js";
import { safeParseJson } from "../safeJson.js";
import { hasZipLocalHeader, hasZipSignature } from "../zipPackageFormat.js";
import { ZIP_EOCD_BYTES } from "../zipEndOfCentralDirectory.js";
import {
declaredDocumentType,
isWellFormedXml,
@@ -42,8 +44,6 @@ export const SQLITE_SIDECARS = [
{ suffix: "-journal", label: "rollback journal" },
] as const;
const GVDB_HEADER = "GVariant";
/** A zip archive's local file header, which FreeCAD, ODF, OOXML, EPUB and JAR files begin with. */
const ZIP_HEADER = "PK\u0003\u0004";
const XML_EXTENSIONS = /\.(?:xml|xbel|plist|svg|ui)$/iu;
/** A DOCTYPE that only names its DTD reads as XML; one with an internal subset, which can declare entities, does not. */
export const XML_POLICY = {
@@ -306,7 +306,11 @@ export const detectStateFormat = (
const header = new TextDecoder("latin1").decode(bytes.subarray(0, 16));
if (header === SQLITE_HEADER) return "sqlite";
if (header.startsWith(GVDB_HEADER)) return "dconf";
if (header.startsWith(ZIP_HEADER)) return "zip";
if (
hasZipLocalHeader(bytes) ||
(bytes.length >= ZIP_EOCD_BYTES && hasZipSignature(bytes))
)
return "zip";
const text = decodeStateText(bytes);
if (text === undefined) return "binary";
const trimmed = text.trimStart();
+8 -3
View File
@@ -173,20 +173,25 @@ export const emptyEntriesArchive = (count: number): Buffer => {
* and a classic one that defers to them, or with `sentinels` false keeps its
* own counts, size and offset, as a writer may when they fit.
*/
export const withZip64End = (archive: Buffer, sentinels = true): Buffer => {
export const withZip64End = (
archive: Buffer,
sentinels = true,
extension: Uint8Array = new Uint8Array(),
): Buffer => {
const end = archive.length - 22;
const entries = archive.readUInt16LE(end + 10);
const size = archive.readUInt32LE(end + 12);
const offset = archive.readUInt32LE(end + 16);
const record = Buffer.alloc(56);
const record = Buffer.alloc(56 + extension.length);
record.writeUInt32LE(0x06064b50, 0);
record.writeBigUInt64LE(44n, 4);
record.writeBigUInt64LE(BigInt(44 + extension.length), 4);
record.writeUInt16LE(45, 12);
record.writeUInt16LE(45, 14);
record.writeBigUInt64LE(BigInt(entries), 24);
record.writeBigUInt64LE(BigInt(entries), 32);
record.writeBigUInt64LE(BigInt(size), 40);
record.writeBigUInt64LE(BigInt(offset), 48);
record.set(extension, 56);
const locator = Buffer.alloc(20);
locator.writeUInt32LE(0x07064b50, 0);
locator.writeBigUInt64LE(BigInt(end), 8);
+125 -84
View File
@@ -1,20 +1,22 @@
import { crc32 } from "node:zlib";
import { BinaryReader } from "../binaryContainer.js";
import {
ZIP_CLASSIC_DIRECTORY_LIMITS,
ZIP64_END,
ZIP64_LOCATOR,
ZIP_EOCD_BYTES,
ZIP_MAX_COMMENT_BYTES,
readZip64DirectoryRecord,
zip64EndUsesSingleVolume,
zipEndOfCentralDirectoryCandidates,
zipEndUsesSingleVolume,
} from "../zipEndOfCentralDirectory.js";
import { strictUtf8Decoder } from "../strictUtf8.js";
import { ZIP_EXTRA_FIELD, lastZipExtraField } from "../zip/zipExtraFields.js";
const CENTRAL_HEADER = 0x02014b50;
const CENTRAL_HEADER_BYTES = 46;
const ZIP64_LOCATOR = 0x07064b50;
const ZIP64_LOCATOR_BYTES = 20;
const ZIP64_END = 0x06064b50;
const ZIP64_END_BYTES = 56;
const UTF8_NAME = 0x800;
const UNICODE_PATH = 0x7075;
const UNIX_TYPE = 0o170000;
const UNIX_DIRECTORY = 0o040000;
const MSDOS_DIRECTORY = 0x10;
@@ -33,41 +35,45 @@ export interface ZipDirectory {
}
/** Where the directory ends, how long it is and the offset it declares, from the end record and any Zip64 one. */
const directoryEnd = (view: DataView) => {
const window = Math.max(
0,
view.byteLength - ZIP_EOCD_BYTES - ZIP_MAX_COMMENT_BYTES,
);
const found = zipEndOfCentralDirectoryCandidates(
new Uint8Array(
view.buffer,
view.byteOffset + window,
view.byteLength - window,
),
).next();
const directoryEnd = (reader: BinaryReader) => {
const found = zipEndOfCentralDirectoryCandidates(reader.bytes).next();
if (found.done === true) return undefined;
const end = window + found.value.offset;
const { totalEntries, directorySize, directoryOffset } = found.value;
const classic = { end, size: directorySize, declaredOffset: directoryOffset };
const record = end - ZIP64_LOCATOR_BYTES - ZIP64_END_BYTES;
// Like Python's zipfile, a Zip64 locator is followed whether or not the classic record defers to it.
if (
record < 0 ||
view.getUint32(end - ZIP64_LOCATOR_BYTES, true) !== ZIP64_LOCATOR
)
const eocd = found.value;
const end = eocd.offset;
const { totalEntries, directorySize, directoryOffset } = eocd;
const classic = zipEndUsesSingleVolume(eocd)
? { end, size: directorySize, declaredOffset: directoryOffset }
: undefined;
const locator = end - ZIP64_LOCATOR.bytes;
// Like Python's zipfile, follow a locator whether or not the classic record defers to it.
if (locator < 0 || reader.u32(locator) !== ZIP64_LOCATOR.signature)
return classic;
// Read just before its locator, where zipfile and zip.js look when the locator's own offset holds no Zip64 record.
if (view.getUint32(record, true) === ZIP64_END)
const declared = reader.word(locator + ZIP64_LOCATOR.recordOffset, 64);
const fallback = locator - ZIP64_END.fixedBytes;
// Keep the fixed-record fallback for archives with an unadjusted prepended offset.
for (const at of [
declared <= BigInt(fallback) ? Number(declared) : -1,
fallback,
]) {
const record = readZip64DirectoryRecord(reader, at, locator);
if (!record.ok || record.value.end !== locator) continue;
if (
!zipEndUsesSingleVolume(eocd, "zip64Classic") ||
!zip64EndUsesSingleVolume(reader, locator, at)
)
return undefined;
return {
end: record,
size: Number(view.getBigUint64(record + 40, true)),
declaredOffset: Number(view.getBigUint64(record + 48, true)),
end: at,
size: Number(record.value.directorySize),
declaredOffset: Number(record.value.directoryOffset),
};
// Without one the classic values stand, as zip.js reads them, unless they are sentinels deferring to it.
}
// Keep the classic values for a stray locator when the classic record is complete.
const limits = ZIP_CLASSIC_DIRECTORY_LIMITS;
const deferred =
totalEntries === 0xffff ||
directorySize === 0xffffffff ||
directoryOffset === 0xffffffff;
totalEntries === limits.count ||
directorySize === limits.size ||
directoryOffset === limits.offset;
return deferred ? undefined : classic;
};
@@ -79,45 +85,44 @@ interface RecordName {
readonly flags: number;
}
/** An Info-ZIP Unicode Path when zip.js uses it: version 1, no UTF-8 flag, and the CRC-32 of the stored name. */
/** The usable last Unicode Path, or null when the extra block is malformed. */
const unicodePath = (
bytes: Uint8Array,
view: DataView,
reader: BinaryReader,
{ name, extra, comment, flags }: RecordName,
): Uint8Array | undefined => {
): Uint8Array | undefined | null => {
// zip.js indexes extra fields by ID with Map.set: the last occurrence
// replaces earlier ones, even when that last claim fails validation.
let selected: { at: number; length: number } | undefined;
for (let at = extra; at + 4 <= comment;) {
const length = view.getUint16(at + 2, true);
if (view.getUint16(at, true) === UNICODE_PATH) selected = { at, length };
at += 4 + length;
}
if (selected === undefined) return undefined;
const { at, length } = selected;
return length >= 5 &&
at + 4 + length <= comment &&
bytes[at + 4] === 1 &&
const selected = lastZipExtraField(
reader,
extra,
comment,
ZIP_EXTRA_FIELD.unicodePath,
);
if (!selected.ok) return null;
if (selected.value === undefined) return undefined;
const { start, end } = selected.value;
const { bytes } = reader;
return end - start >= 5 &&
bytes[start] === 1 &&
(flags & UTF8_NAME) === 0 &&
view.getUint32(at + 5, true) === crc32(bytes.subarray(name, extra))
? bytes.subarray(at + 9, at + 4 + length)
reader.u32(start + 1) === crc32(bytes.subarray(name, extra))
? bytes.subarray(start + 5, end)
: undefined;
};
/**
* Whether an entry's name, as REA decodes it, ends in a slash; undefined when
* it must be UTF-8 and is not. A plain CP437 name is judged by its last byte,
* so most records are read without allocating.
* its extra block is malformed or its name must be UTF-8 and is not. A plain
* CP437 name is judged by its last byte, so most records are read without allocating.
*/
const endsInSlash = (
bytes: Uint8Array,
view: DataView,
reader: BinaryReader,
record: RecordName,
): boolean | undefined => {
const unicode =
record.comment > record.extra
? unicodePath(bytes, view, record)
: undefined;
record.comment > record.extra ? unicodePath(reader, record) : undefined;
if (unicode === null) return undefined;
const { bytes } = reader;
const utf8 =
unicode ??
((record.flags & UTF8_NAME) === 0
@@ -143,8 +148,12 @@ export const scanZipDirectory = (
bytes: Uint8Array,
keep: number,
): ZipDirectory | undefined => {
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
const located = directoryEnd(view);
const reader = new BinaryReader(
Buffer.from(bytes.buffer, bytes.byteOffset, bytes.byteLength),
true,
"zip",
);
const located = directoryEnd(reader);
if (located === undefined) return undefined;
const start = located.end - located.size;
if (start < 0) return undefined;
@@ -153,27 +162,26 @@ export const scanZipDirectory = (
for (let at = start; at < located.end;) {
if (
at + CENTRAL_HEADER_BYTES > located.end ||
view.getUint32(at, true) !== CENTRAL_HEADER
reader.u32(at) !== CENTRAL_HEADER
)
return undefined;
const name = at + CENTRAL_HEADER_BYTES;
const extra = name + view.getUint16(at + 28, true);
const comment = extra + view.getUint16(at + 30, true);
const next = comment + view.getUint16(at + 32, true);
const extra = name + reader.u16(at + 28);
const comment = extra + reader.u16(at + 30);
const next = comment + reader.u16(at + 32);
if (next > located.end) return undefined;
const slash = endsInSlash(bytes, view, {
const slash = endsInSlash(reader, {
name,
extra,
comment,
flags: view.getUint16(at + 8, true),
flags: reader.u16(at + 8),
});
if (slash === undefined) return undefined;
const external = view.getUint32(at + 38, true);
const external = reader.u32(at + 38);
const directory =
slash ||
((external >>> 16) & UNIX_TYPE) === UNIX_DIRECTORY ||
(view.getUint16(at + 4, true) >> 8 === 0 &&
(external & MSDOS_DIRECTORY) !== 0);
(reader.u16(at + 4) >> 8 === 0 && (external & MSDOS_DIRECTORY) !== 0);
if (!directory) {
count += 1;
if (records.length < keep) records.push(bytes.subarray(at, next));
@@ -189,24 +197,57 @@ export const boundedDirectoryTail = (directory: ZipDirectory): Uint8Array => {
(total, record) => total + record.length,
0,
);
const tail = new Uint8Array(size + ZIP_EOCD_BYTES);
const {
count: maxCount,
size: maxSize,
offset: maxOffset,
} = ZIP_CLASSIC_DIRECTORY_LIMITS;
const count = directory.records.length;
// A declared offset past 32 bits cannot be right for a view-sized archive.
const offset =
directory.declaredOffset <= maxOffset
? directory.declaredOffset
: directory.start;
const wide = count >= maxCount || size >= maxSize || offset >= maxOffset;
const wideBytes = wide ? ZIP64_END.fixedBytes + ZIP64_LOCATOR.bytes : 0;
const tail = new Uint8Array(size + wideBytes + ZIP_EOCD_BYTES);
let at = 0;
for (const record of directory.records) {
tail.set(record, at);
at += record.length;
}
const end = new DataView(tail.buffer, size);
if (wide) {
const record = new DataView(tail.buffer, size, ZIP64_END.fixedBytes);
record.setUint32(0, ZIP64_END.signature, true);
record.setBigUint64(
ZIP64_END.sizeOffset,
BigInt(ZIP64_END.fixedBytes - ZIP64_END.sizePrefixBytes),
true,
);
record.setUint16(12, 45, true);
record.setUint16(14, 45, true);
record.setBigUint64(ZIP64_END.entriesOnDiskOffset, BigInt(count), true);
record.setBigUint64(ZIP64_END.totalEntriesOffset, BigInt(count), true);
record.setBigUint64(ZIP64_END.directorySizeOffset, BigInt(size), true);
record.setBigUint64(ZIP64_END.directoryOffset, BigInt(offset), true);
const locator = new DataView(
tail.buffer,
size + ZIP64_END.fixedBytes,
ZIP64_LOCATOR.bytes,
);
locator.setUint32(0, ZIP64_LOCATOR.signature, true);
locator.setBigUint64(
ZIP64_LOCATOR.recordOffset,
BigInt(offset) + BigInt(size),
true,
);
locator.setUint32(16, 1, true);
}
const end = new DataView(tail.buffer, size + wideBytes, ZIP_EOCD_BYTES);
end.setUint32(0, 0x06054b50, true);
end.setUint16(8, directory.records.length, true);
end.setUint16(10, directory.records.length, true);
end.setUint32(12, size, true);
// A declared offset past 32 bits cannot be right for a view-sized archive; the start stands in for it.
end.setUint32(
16,
directory.declaredOffset <= 0xffffffff
? directory.declaredOffset
: directory.start,
true,
);
end.setUint16(8, Math.min(count, maxCount), true);
end.setUint16(10, Math.min(count, maxCount), true);
end.setUint32(12, Math.min(size, maxSize), true);
end.setUint32(16, Math.min(offset, maxOffset), true);
return tail;
};
+34
View File
@@ -0,0 +1,34 @@
import type { BinaryReader } from "../binaryContainer.js";
import { err, ok, type Result } from "../result.js";
/** APPNOTE §4.5.1. The exported desktop reader uses the same field layout. */
export const ZIP_EXTRA_FIELD = {
headerBytes: 4,
idOffset: 0,
sizeOffset: 2,
unicodePath: 0x7075,
} as const;
/** The last matching payload, provided every framed field fits the extra block. */
export const lastZipExtraField = (
reader: BinaryReader,
start: number,
end: number,
kind: number,
): Result<
{ readonly start: number; readonly end: number } | undefined,
"overrun"
> => {
let selected: { readonly start: number; readonly end: number } | undefined;
// Both desktop decoders have historically accepted a remainder shorter
// than a header. Only a complete header declares a payload span.
for (let at = start; at + ZIP_EXTRA_FIELD.headerBytes <= end;) {
const payload = at + ZIP_EXTRA_FIELD.headerBytes;
const next = payload + reader.u16(at + ZIP_EXTRA_FIELD.sizeOffset);
if (next > end) return err("overrun");
if (reader.u16(at + ZIP_EXTRA_FIELD.idOffset) === kind)
selected = { start: payload, end: next };
at = next;
}
return ok(selected);
};
+106 -2
View File
@@ -1,11 +1,88 @@
import { BinaryReader } from "./binaryContainer.js";
import { err, ok, type Result } from "./result.js";
/** Classic ZIP EOCD fixed record and its format-defined comment limit. */
export const ZIP_EOCD_BYTES = 22;
export const ZIP_MAX_COMMENT_BYTES = 0xffff;
/** Saturated classic directory fields defer to the Zip64 end record. */
export const ZIP_CLASSIC_DIRECTORY_LIMITS = {
count: 0xffff,
size: 0xffffffff,
offset: 0xffffffff,
} as const;
/** ZIP APPNOTE §§4.3.14–15. Shared fixed fields within the variable-size frame. */
export const ZIP64_END = {
signature: 0x06064b50,
fixedBytes: 56,
sizePrefixBytes: 12,
sizeOffset: 4,
entriesOnDiskOffset: 24,
totalEntriesOffset: 32,
directorySizeOffset: 40,
directoryOffset: 48,
} as const;
export const ZIP64_LOCATOR = {
signature: 0x07064b50,
bytes: 20,
recordOffset: 8,
} as const;
type Zip64DirectoryRecord = {
readonly end: number;
readonly totalEntries: bigint;
readonly directorySize: bigint;
readonly directoryOffset: bigint;
};
/** Observe one complete Zip64 end record inside its declared carrier. */
export const readZip64DirectoryRecord = (
reader: BinaryReader,
offset: number,
limit: number,
): Result<Zip64DirectoryRecord, "fixed-fields" | "signature" | "size"> => {
const room = Math.min(limit, reader.bytes.length) - offset;
if (offset < 0 || room < ZIP64_END.fixedBytes) return err("fixed-fields");
if (reader.u32(offset) !== ZIP64_END.signature) return err("signature");
const bodyBytes = reader.word(offset + ZIP64_END.sizeOffset, 64);
if (
bodyBytes < BigInt(ZIP64_END.fixedBytes - ZIP64_END.sizePrefixBytes) ||
bodyBytes > BigInt(room - ZIP64_END.sizePrefixBytes)
)
return err("size");
return ok({
end: offset + ZIP64_END.sizePrefixBytes + Number(bodyBytes),
totalEntries: reader.word(offset + ZIP64_END.totalEntriesOffset, 64),
directorySize: reader.word(offset + ZIP64_END.directorySizeOffset, 64),
directoryOffset: reader.word(offset + ZIP64_END.directoryOffset, 64),
});
};
const EOCD_SIGNATURE = Buffer.from([0x50, 0x4b, 0x05, 0x06]);
export const ZIP_CLASSIC_DISK_OFFSETS = { disk: 4, directoryDisk: 6 } as const;
/**
* APPNOTE §§4.3.14–16 and §§4.4.19–20. The classic disk numbers can
* defer to Zip64 only after a Zip64 end record has been located. The two
* Zip64 layouts list 32-bit offsets and their required single-volume values.
* The exported desktop reader consumes the same rules in Python.
*/
export const ZIP_SINGLE_VOLUME = {
classic: [0],
zip64Classic: [0, 0xffff],
zip64Locator: [
[4, 0],
[16, 1],
],
zip64End: [
[16, 0],
[20, 0],
],
} as const;
export interface ZipEndOfCentralDirectory {
/** Relative to the supplied byte window, which ends at the archive's end. */
readonly offset: number;
@@ -18,6 +95,31 @@ export interface ZipEndOfCentralDirectory {
readonly commentLength: number;
}
/** Whether the observed classic end record admits the selected single volume. */
export const zipEndUsesSingleVolume = (
{
disk,
directoryDisk,
}: Pick<ZipEndOfCentralDirectory, "disk" | "directoryDisk">,
layout: "classic" | "zip64Classic" = "classic",
): boolean => {
const allowed: readonly number[] = ZIP_SINGLE_VOLUME[layout];
return allowed.includes(disk) && allowed.includes(directoryDisk);
};
/** Check the disk fields in the located Zip64 locator and end record. */
export const zip64EndUsesSingleVolume = (
reader: BinaryReader,
locator: number,
end: number,
): boolean =>
ZIP_SINGLE_VOLUME.zip64Locator.every(
([field, value]) => reader.u32(locator + field) === value,
) &&
ZIP_SINGLE_VOLUME.zip64End.every(
([field, value]) => reader.u32(end + field) === value,
);
/**
* Observe complete classic records whose declared comment ends this window,
* latest first. Candidate recognition does not establish directory validity,
@@ -44,8 +146,10 @@ export function* zipEndOfCentralDirectoryCandidates(
if (offset + ZIP_EOCD_BYTES + commentLength !== tail.length) continue;
yield {
offset: earliest + offset,
disk: reader.u16(offset + 4),
directoryDisk: reader.u16(offset + 6),
disk: reader.u16(offset + ZIP_CLASSIC_DISK_OFFSETS.disk),
directoryDisk: reader.u16(
offset + ZIP_CLASSIC_DISK_OFFSETS.directoryDisk,
),
entriesOnDisk: reader.u16(offset + 8),
totalEntries: reader.u16(offset + 10),
directorySize: reader.u32(offset + 12),
+18 -9
View File
@@ -1,6 +1,7 @@
import { BinaryReader } from "./binaryContainer.js";
import {
zipEndOfCentralDirectoryCandidates,
zipEndUsesSingleVolume,
ZIP_EOCD_BYTES,
ZIP_MAX_COMMENT_BYTES,
} from "./zipEndOfCentralDirectory.js";
@@ -24,15 +25,24 @@ export const isZipPackageFormat = (
): format is ZipPackageFormat =>
ZIP_PACKAGE_FORMATS.some((candidate) => candidate === format);
/** Recognize local-header, classic/Zip64 empty-archive, or split ZIP signatures, not validity. */
/** Local-header, classic/Zip64 empty-archive, and split ZIP signatures. */
export const ZIP_SIGNATURES = [
[0x50, 0x4b, 0x03, 0x04],
[0x50, 0x4b, 0x05, 0x06],
[0x50, 0x4b, 0x06, 0x06],
[0x50, 0x4b, 0x07, 0x08],
] as const;
/** A local file header can identify a ZIP before a complete end record is available. */
export const hasZipLocalHeader = (bytes: Uint8Array): boolean =>
ZIP_SIGNATURES[0].every((byte, index) => bytes[index] === byte);
/** Recognize a ZIP signature, not directory validity or supported storage. */
export const hasZipSignature = (bytes: Uint8Array): boolean =>
bytes.length >= 4 &&
bytes[0] === 0x50 &&
bytes[1] === 0x4b &&
((bytes[2] === 0x03 && bytes[3] === 0x04) ||
(bytes[2] === 0x05 && bytes[3] === 0x06) ||
(bytes[2] === 0x06 && bytes[3] === 0x06) ||
(bytes[2] === 0x07 && bytes[3] === 0x08));
ZIP_SIGNATURES.some((signature) =>
signature.every((byte, index) => bytes[index] === byte),
);
const CENTRAL_DIRECTORY_HEADER = 0x02014b50;
const LOCAL_FILE_HEADER = 0x04034b50;
@@ -68,8 +78,7 @@ export const locateTrailingZipArchive = async (
directoryOffset: declaredDirectory,
} = candidate;
if (
candidate.disk !== 0 ||
candidate.directoryDisk !== 0 ||
!zipEndUsesSingleVolume(candidate) ||
entriesOnDisk === 0 ||
entriesOnDisk === 0xffff ||
entriesOnDisk !== candidate.totalEntries ||
@@ -0,0 +1,153 @@
import { execFileSync } from "node:child_process";
import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { gzipSync } from "node:zlib";
import { expect, it } from "vitest";
import { readZipView } from "../../../src/application/desktop/DesktopZipView.js";
import {
PYTHON_HELPERS,
PYTHON_IMPORTS,
} from "../../../src/domain/desktop/appExportPython.js";
import { PYTHON_TASK_EVALUATOR } from "../../../src/domain/desktop/appTaskPython.js";
import { evaluateTaskChecks } from "../../../src/domain/desktop/appTaskChecks.js";
import { detectStateFormat } from "../../../src/domain/desktop/stateFormats.js";
import {
withZip64End,
zipArchive,
} from "../../../src/domain/desktop/zipArchive.fixture.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
const empty = zipArchive([]);
const commented = Buffer.concat([empty, Buffer.from("archive comment")]);
commented.writeUInt16LE(commented.length - 22, 20);
const archives = [
["classic", empty],
["commented", commented],
["Zip64 sentinels", withZip64End(empty)],
["Zip64 ordinary counts", withZip64End(empty, false)],
] as const;
const pythonViews = async (files: readonly Buffer[]) => {
const root = await createTestTempDirectory("desktop-empty-zip-");
const paths = await Promise.all(
files.map(async (bytes, index) => {
const path = join(root, `${String(index)}.dat`);
await writeFile(path, bytes);
return path;
}),
);
const imports = PYTHON_IMPORTS.split("\n")
.filter((line) => !line.includes("agentenv_protocol"))
.join("\n");
const script = join(root, "read.py");
await writeFile(
script,
`${imports}${PYTHON_HELPERS}${PYTHON_TASK_EVALUATOR}
check = {"id": "empty", "kind": "value_equals", "file": "archive.zip", "pointer": "/entry_count", "value": 0}
for path in json.load(sys.stdin):
view = _view(Path(path))
print(json.dumps({"file": view, "passed": evaluate([check], {"archive.zip": view})[0]["result"]}))
`,
);
return execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", "-X", "utf8", script],
{ input: JSON.stringify(paths), encoding: "utf8" },
)
.trim()
.split(/\r?\n/u)
.map(
(line) =>
JSON.parse(line) as {
file: { format: string; view: unknown; error?: string };
passed: boolean;
},
);
};
it.each(archives)(
"reads an empty %s archive without relying on its name",
async (_name, bytes) => {
const expected = { entries: [], entry_count: 0, truncated: false };
expect(detectStateFormat("document.dat", bytes)).toBe("zip");
expect(await readZipView(bytes)).toEqual({ ok: true, value: expected });
expect((await pythonViews([bytes]))[0]).toEqual({
file: { format: "zip", view: expected, truncated: false },
passed: true,
});
},
);
it("keeps empty nested archives viewless instead of recursing or calling them binary", async () => {
const outer = zipArchive(
archives.map(([name, data]) => ({ name: `${name}.zip`, data })),
);
const native = await readZipView(outer);
if (!native.ok) throw new Error(native.error);
const python = (await pythonViews([outer]))[0]?.file;
expect(python?.format).toBe("zip");
expect(python?.view).toEqual(native.value);
expect(native.value).toMatchObject({
entries: archives.map(() => ({
format: "zip",
view: null,
view_error: "an archive inside an archive is not read",
})),
entry_count: archives.length,
truncated: false,
});
});
it("grades removing every entry as a change from a nonempty archive", async () => {
const files = [zipArchive([{ name: "old.txt", data: "old" }]), empty];
const check = {
id: "empty",
kind: "value_equals" as const,
file: "archive.zip",
pointer: "/entry_count",
value: 0,
};
const verdicts = await Promise.all(
files.map(async (bytes) => {
const read = await readZipView(bytes);
if (!read.ok) throw new Error(read.error);
return evaluateTaskChecks(
[check],
new Map([
[
check.file,
{ format: detectStateFormat(check.file, bytes), view: read.value },
],
]),
)[0]?.passed;
}),
);
expect(verdicts).toEqual([false, true]);
expect((await pythonViews(files)).map((row) => row.passed)).toEqual(verdicts);
});
it("does not turn incomplete or damaged end records into empty views", async () => {
const incomplete = empty.subarray(0, 6);
expect(detectStateFormat("archive.zip", incomplete)).toBe("binary");
const damaged = Buffer.from(empty);
damaged.writeUInt16LE(1, 20);
const zip64 = withZip64End(empty).subarray(0, 56);
for (const bytes of [damaged, zip64]) {
expect(detectStateFormat("archive.dat", bytes)).toBe("zip");
expect((await readZipView(bytes)).ok).toBe(false);
const python = (await pythonViews([bytes]))[0];
expect(python?.file).toMatchObject({ format: "zip", view: null });
expect(python?.file.error).toBeTypeOf("string");
expect(python?.passed).toBe(false);
}
});
it("reads the same empty archive when gzip wraps its bytes", async () => {
const [view] = await pythonViews([gzipSync(empty)]);
expect(view).toMatchObject({
file: { format: "zip", compression: "gzip", view: { entry_count: 0 } },
passed: true,
});
});
@@ -1,3 +1,4 @@
import { execFileSync } from "node:child_process";
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { crc32 } from "node:zlib";
@@ -18,6 +19,11 @@ import {
type ZipMember,
} from "../../../src/domain/desktop/zipArchive.fixture.js";
import { zipEntryRefusal } from "../../../src/domain/desktop/zipView.js";
import {
desktopViewPolicy,
pythonDesktopViewPolicy,
} from "../../../src/domain/desktop/desktopViewPolicy.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import {
expectSameCheckResults,
expectSameFileViews,
@@ -183,6 +189,33 @@ it("refuses invalid UTF-8 in the selected Unicode claim while preserving an unre
});
});
it("refuses a central extra field whose payload extends past the containing record's extra bytes", async () => {
const name = "report.txt";
const opaque = Buffer.from([0xfe, 0xca, 2, 0, 1]);
const unicode = unicodePathExtra(name, "renamed.txt");
unicode.writeUInt16LE(unicode.length - 3, 2);
const archives = [
zipArchive([{ name, data: "observed", centralExtra: opaque }]),
zipArchive([{ name, data: "observed", centralExtra: unicode }]),
zipArchive([
{ name: "directory/", centralExtra: opaque },
{ name, data: "observed" },
]),
];
const fromPython = await pythonFileViews(saved(archives));
for (const [index, archive] of archives.entries()) {
expect.soft(await readZipView(archive)).toEqual({
ok: false,
error: "not a readable zip archive",
});
expect.soft(fromPython[index]).toEqual({
format: "zip",
view: null,
error: "not a readable zip archive",
});
}
});
it("excludes a final Unicode directory without consuming a retained file slot", async () => {
const name = "settings.json";
const archive = zipArchive([
@@ -401,6 +434,153 @@ describe("zip directories in the exported module", () => {
});
});
it.each([false, true])(
"follows the Zip64 locator over a fixed-size signature in opaque extension bytes (%s)",
async (sentinels) => {
const extension = Buffer.alloc(6 + 56);
extension.writeUInt16LE(0xfffe, 0);
extension.writeUInt32LE(56, 2);
extension.writeUInt32LE(0x06064b50, 6);
const archive = withZip64End(
zipArchive([{ name: "notes.txt", data: "retained" }]),
sentinels,
extension,
);
const result = await sameView(archive);
expect(result).toMatchObject({
entry_count: 1,
entries: [{ name: "notes.txt", view: { lines: ["retained"] } }],
truncated: false,
});
},
);
it("retains the fixed Zip64 fallback when a prefix shifts its relative locator", async () => {
// Keep a recognized ZIP opening for desktop format detection. Only the
// real recorded local entry and central directory contribute to the view.
const prefix = Buffer.from("PK\x03\x04inert prefix");
const archive = Buffer.concat([
prefix,
withZip64End(zipArchive([{ name: "notes.txt", data: "retained" }])),
]);
const result = await sameView(archive);
expect(result).toMatchObject({
entry_count: 1,
entries: [{ name: "notes.txt", view: { lines: ["retained"] } }],
truncated: false,
});
});
});
describe("ZIP directory counts and volume admission in the exported module", () => {
it("retains the selected ZIP count at and beyond the classic sentinel", async () => {
const path = join(
await createTestTempDirectory("rea-zip64-count-"),
"many.zip",
);
// An independent writer emits an original Zip64 archive, not a decoder-copy fixture.
execFileSync("python3", [
"-I",
"-B",
"-c",
`import sys, zipfile
with zipfile.ZipFile(sys.argv[1], "w") as archive:
for index in range(65536):
archive.writestr(f"entry-{index:05d}.bin", b"ab")
with zipfile.ZipFile(sys.argv[1]) as archive:
assert len(archive.infolist()) == 65536`,
path,
]);
const bytes = await readFile(path);
for (const keep of [65_535, 65_536]) {
const policy = desktopViewPolicy({
zip_entries: keep,
zip_entry_bytes: 1,
});
const result = await readZipView(bytes, undefined, policy);
const host = result.ok
? (result.value as {
entries: { name: string }[];
entry_count: number;
truncated: boolean;
})
: null;
const python: unknown = JSON.parse(
exportedPython(
`${pythonDesktopViewPolicy(policy)}
result = _view(Path(json.load(sys.stdin)))
view = result.get("view")
print(json.dumps(None if view is None else {
"listed": len(view["entries"]), "count": view["entry_count"],
"truncated": view["truncated"], "last": view["entries"][-1]["name"]
}))`,
path,
),
);
expect([
host === null
? null
: {
listed: host.entries.length,
count: host.entry_count,
truncated: host.truncated,
last: host.entries.at(-1)?.name,
},
python,
]).toEqual(
Array.from({ length: 2 }, () => ({
listed: keep,
count: 65_536,
truncated: keep < 65_536,
last: `entry-${String(keep - 1).padStart(5, "0")}.bin`,
})),
);
}
});
it("refuses archives whose selected directory or Zip64 locator names another volume", async () => {
const classic = zipArchive([{ name: "settings.json", data: '{"zoom":2}' }]);
const zip64 = withZip64End(classic);
const variants = [
["classic disk", classic, 22, 4, 1],
["classic directory disk", classic, 22, 6, 1],
["classic disk in Zip64", zip64, 22, 4, 1],
["Zip64 record disk", zip64, 98, 16, 1],
["Zip64 directory disk", zip64, 98, 20, 1],
["Zip64 locator disk", zip64, 42, 4, 1],
["Zip64 locator disk count", zip64, 42, 16, 2],
] as const;
const archives = variants.map(([, bytes, end, field, value]) => {
const copy = Buffer.from(bytes);
copy.writeUInt16LE(value, copy.length - end + field);
return copy;
});
const python = await pythonFileViews(saved(archives));
for (const [index, [description]] of variants.entries()) {
expect(await readZipView(archives[index]!), description).toEqual({
ok: false,
error: "not a readable zip archive",
});
expect(python[index], description).toEqual({
format: "zip",
view: null,
error: "not a readable zip archive",
});
}
});
it("uses the Zip64 record when the classic disk numbers defer to it", async () => {
const archive = withZip64End(
zipArchive([{ name: "settings.json", data: '{"zoom":2}' }]),
);
archive.writeUInt16LE(0xffff, archive.length - 18);
archive.writeUInt16LE(0xffff, archive.length - 16);
expect(await sameView(archive)).toMatchObject({
entries: [{ name: "settings.json", view: { zoom: 2 } }],
entry_count: 1,
});
});
it("keeps the classic values when bytes before the end record only look like a Zip64 locator", async () => {
// The last name ends in a locator signature and 16 more bytes, which sit just before the end record.
const name = `c-PK\u0006\u0007${"0".repeat(16)}`;