mirror of
https://github.com/morluto/rea.git
synced 2026-11-19 13:12:26 +00:00
fix(zip): share desktop and APK framing and preserve bounded ZIP64 views (#12157)
* fix(desktop): reject split ZIP volume metadata
Refs #12137
* refactor(zip): read Zip64 disk fields through BinaryReader
* fix(desktop): validate ZIP extra fields before selection
Refs #12146
* fix(desktop): keep bounded ZIP pairing compatible with Python 3.9
* fix(desktop): retain exact counts in bounded Zip64 views
* fix(desktop): follow the declared Zip64 directory frame
* fix(desktop): recognize empty ZIP state views (#12163)
* fix(desktop): recognize empty ZIP state views
(cherry picked from commit 263b25bd34)
* fix(desktop): reuse empty ZIP admission for wrapped documents
* refactor(zip): share the local-header signature with desktop state
---------
Co-authored-by: wunianze666-netizen <wunianze666@gmail.com>
* test(zip): inherit the portable ZIP count deadline
---------
Co-authored-by: wunianze666-netizen <wunianze666@gmail.com>
This commit is contained in:
1 parent
27e04ea2ec
commit
3d313b56bc
17 files changed
+840
-224
No files matched your search
@@ -562,7 +562,10 @@ as their entries, read with Python's `zipfile` from the file in place, with
|
||||
REA's entry, size, compression ratio and archive limits and its wording for an
|
||||
entry it does not read. The module reads the central directory itself, as REA
|
||||
does, and shows `zipfile` only the records it lists, reading each entry in
|
||||
bounded chunks no further than a byte past its declared size. A gzip file is
|
||||
bounded chunks no further than a byte past its declared size. The reader
|
||||
follows a Zip64 locator to the complete end-record span and skips its
|
||||
extension bytes before passing the bounded directory to `zipfile`.
|
||||
A gzip file is
|
||||
the view of the document inside it, with `"compression": "gzip"` beside its
|
||||
format, decompressed with Python's `zlib` within the same entry size and ratio
|
||||
limits, its members joined as REA joins them. A JSON file
|
||||
|
||||
@@ -29,6 +29,9 @@ A path absent from the capture is an input error.
|
||||
`sqlite_tables`, `sqlite_rows`, `sqlite_view_bytes`, `text_lines`, `xml_nodes`, `xml_depth`,
|
||||
`parsed_bytes`, `json_depth`, `zip_entries`, `zip_entry_bytes`, `zip_bytes`,
|
||||
and `zip_ratio`. Unspecified controls keep the defaults described below.
|
||||
A request that raises `zip_entries` past the classic ZIP count range still uses
|
||||
the selected entry limit. The host and exported readers encode the retained
|
||||
directory's exact count with Zip64 fields when the classic fields are full.
|
||||
`xml_depth` admits at most 494 and `json_depth` at most 991: these bounds
|
||||
follow the fixed Evidence JSON nesting limit, including the captured-version
|
||||
and ZIP-member wrappers and XML's element/children representation. A request
|
||||
@@ -138,16 +141,16 @@ Views hold the captured bytes as they are, the capture host's name and user
|
||||
included; the export removes those from the bundle. A version records
|
||||
whether the file was present, the SHA-256 of its retained bytes, and a view:
|
||||
|
||||
| Format | Detected by | View |
|
||||
| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
|
||||
| `sqlite` | The `SQLite format 3` header | Every ordinary table's columns, primary key and up to 500 rows |
|
||||
| `dconf` | The `GVariant` header | Groups whose path names the application, plus the names of the other groups |
|
||||
| `zip` | A zip local file header, `PK\x03\x04`, whatever the name | Its file entries in archive order, each with its own format and view |
|
||||
| `json` | A `.json` name or a leading `{` or `[` that parses | The parsed value, from files up to 4 MiB nested up to 500 levels |
|
||||
| `xml` | An XML name, a declaration or a leading element that parses | An element tree (`name`, `attributes`, `children`); text kept exactly, DTDs never read |
|
||||
| `keyfile` | Only `[group]`, `key=value` and comment lines, with at least one group | Groups of `key` → string value, from files up to 4 MiB |
|
||||
| `text` | Other decoded text | Lines, up to 5,000 |
|
||||
| `binary` | Content that cannot be read as text | None |
|
||||
| Format | Detected by | View |
|
||||
| --------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
|
||||
| `sqlite` | The `SQLite format 3` header | Every ordinary table's columns, primary key and up to 500 rows |
|
||||
| `dconf` | The `GVariant` header | Groups whose path names the application, plus the names of the other groups |
|
||||
| `zip` | A ZIP local file header or an empty archive signature, whatever the name | Its file entries in archive order, each with its own format and view |
|
||||
| `json` | A `.json` name or a leading `{` or `[` that parses | The parsed value, from files up to 4 MiB nested up to 500 levels |
|
||||
| `xml` | An XML name, a declaration or a leading element that parses | An element tree (`name`, `attributes`, `children`); text kept exactly, DTDs never read |
|
||||
| `keyfile` | Only `[group]`, `key=value` and comment lines, with at least one group | Groups of `key` → string value, from files up to 4 MiB |
|
||||
| `text` | Other decoded text | Lines, up to 5,000 |
|
||||
| `binary` | Content that cannot be read as text | None |
|
||||
|
||||
Text views decode UTF-8 or strictly valid UTF-16 with a leading byte-order
|
||||
mark. UTF-16 containing a NUL character stays binary. These rules also apply
|
||||
@@ -275,8 +278,9 @@ database, or an archive, inside an archive keeps its format and `sha256` and
|
||||
has a `view_error` instead of a view: it is not read. The view lists at most
|
||||
1,000 entries; `entry_count` counts them all, and `truncated` is true when some
|
||||
are not listed. REA reads the central directory itself, a record at a time,
|
||||
through the Zip64 end record whenever a Zip64 locator precedes the end record,
|
||||
as Python's `zipfile` does, and hands the zip reader only the records it lists,
|
||||
and follows a Zip64 locator to the complete end-record span, including any
|
||||
extension bytes. The fixed-record fallback supports prepended archives
|
||||
whose offsets omit the prefix. REA hands the zip reader only the records it lists,
|
||||
so an archive of hundreds of thousands of entries costs no more to read
|
||||
than one of a thousand. An archive whose central directory cannot be read has a
|
||||
`view_error` of its own.
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Plan
|
||||
|
||||
1. Format and reader: `zip` detected by its local file header; the text, JSON,
|
||||
1. Format and reader: `zip` detected by its local file header or an empty
|
||||
classic/Zip64 archive signature; the text, JSON,
|
||||
key file and XML readers shared as `textualView`, so a file and an entry
|
||||
read alike; a domain module for the entry limits, the entry view and the
|
||||
archive view, and an application reader that walks the verified bytes with
|
||||
|
||||
@@ -13,8 +13,9 @@ evaluators, REA's and the exported verifier's, agree.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Format: a file starting with a zip local file header (`PK\x03\x04`) is
|
||||
`zip`, whatever its name. Its view is
|
||||
- Format: a file starting with a zip local file header (`PK\x03\x04`), or
|
||||
a classic/Zip64 empty-archive signature with at least 22 bytes, is `zip`,
|
||||
whatever its name. A signature is not proof of directory validity. Its view is
|
||||
`{entries, entry_count, truncated}`: the file entries in archive order,
|
||||
directories left out, each `{name, size, sha256, format, view, view_error}`.
|
||||
- Reading: from the retained bytes already checked against their digest, in
|
||||
|
||||
@@ -40,7 +40,7 @@ generation reads the kinds of change.
|
||||
only when it has no internal subset and its DTD never read
|
||||
([XML text checks](../xml-text-checks/spec.md)); key files up to 4 MiB
|
||||
whose every line parses; zip archives by
|
||||
their local file header, as their entries with each entry's own format and
|
||||
their local file header or empty archive signature, as their entries with each entry's own format and
|
||||
view ([document views](../document-views/spec.md)); text; binary. A gzip
|
||||
file, told by its magic bytes, is read as the document inside, its members
|
||||
joined, within a zip entry's 16 MiB and 100-times limits, and its version
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"src/artifacts/nodeSea/NodeSeaLocation.ts": 2,
|
||||
"src/artifacts/pyinstaller/CArchiveLayout.ts": 8,
|
||||
"src/domain/apkSignature/apkSignatureSchemes.ts": 1,
|
||||
"src/domain/apkSignature/apkSigningBlock.ts": 7,
|
||||
"src/domain/apkSignature/apkSigningBlock.ts": 2,
|
||||
"src/domain/apkSignature/apkV1Signature.ts": 15,
|
||||
"src/domain/apple/appleMachoFixups.ts": 22,
|
||||
"src/domain/apple/appleMachoSelection.ts": 2,
|
||||
@@ -32,7 +32,6 @@
|
||||
"src/domain/binaryTarget.ts": 10,
|
||||
"src/domain/desktop/elfDependencies.ts": 4,
|
||||
"src/domain/desktop/openPgpKeys.ts": 2,
|
||||
"src/domain/desktop/zipDirectory.ts": 14,
|
||||
"src/domain/dosMz.ts": 10,
|
||||
"src/domain/elfDynamicEntry.ts": 4,
|
||||
"src/domain/elfFixedHeader.ts": 4,
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
import { BinaryReader } from "../binaryContainer.js";
|
||||
import {
|
||||
readZip64DirectoryRecord,
|
||||
zipEndOfCentralDirectoryCandidates,
|
||||
zipEndUsesSingleVolume,
|
||||
ZIP_CLASSIC_DIRECTORY_LIMITS,
|
||||
ZIP_EOCD_BYTES,
|
||||
ZIP64_END,
|
||||
ZIP64_LOCATOR,
|
||||
type ZipEndOfCentralDirectory,
|
||||
} from "../zipEndOfCentralDirectory.js";
|
||||
|
||||
@@ -20,11 +26,6 @@ import {
|
||||
*/
|
||||
import { type Result, err, ok } from "../result.js";
|
||||
|
||||
const ZIP64_EOCD_LOCATOR_SIGNATURE = 0x07064b50;
|
||||
const ZIP64_EOCD_SIGNATURE = 0x06064b50;
|
||||
const ZIP64_SENTINEL = 0xffffffff;
|
||||
/** The 16-bit EOCD entry count saturates here when Zip64 carries the real count. */
|
||||
const ZIP64_ENTRIES_SENTINEL = 0xffff;
|
||||
/** "APK Sig Block 42" — the 16-byte block magic (JAR/APK signing block spec). */
|
||||
const BLOCK_MAGIC = "APK Sig Block 42";
|
||||
/** A single pair is at least 12 bytes (uint64 length + uint32 id); cap the count. */
|
||||
@@ -123,7 +124,7 @@ interface CentralDirectory {
|
||||
const classicDirectory = (
|
||||
eocd: ZipEndOfCentralDirectory,
|
||||
): Result<CentralDirectory, ApkSigningBlockError> => {
|
||||
if (eocd.disk !== 0 || eocd.directoryDisk !== 0)
|
||||
if (!zipEndUsesSingleVolume(eocd))
|
||||
return err({
|
||||
kind: "multi-disk-unsupported",
|
||||
detail: `classic EOCD names disk ${String(eocd.disk)} and central-directory disk ${String(eocd.directoryDisk)}`,
|
||||
@@ -147,7 +148,6 @@ const centralDirectoryOffset = (
|
||||
bytes: Uint8Array,
|
||||
eocd: ZipEndOfCentralDirectory,
|
||||
): Result<CentralDirectory, ApkSigningBlockError> => {
|
||||
const dv = view(bytes);
|
||||
const {
|
||||
offset: eocdOffset,
|
||||
totalEntries: entries,
|
||||
@@ -155,19 +155,24 @@ const centralDirectoryOffset = (
|
||||
directoryOffset: offset,
|
||||
} = eocd;
|
||||
const saturated =
|
||||
offset === ZIP64_SENTINEL ||
|
||||
size === ZIP64_SENTINEL ||
|
||||
entries === ZIP64_ENTRIES_SENTINEL;
|
||||
offset === ZIP_CLASSIC_DIRECTORY_LIMITS.offset ||
|
||||
size === ZIP_CLASSIC_DIRECTORY_LIMITS.size ||
|
||||
entries === ZIP_CLASSIC_DIRECTORY_LIMITS.count;
|
||||
if (!saturated) return classicDirectory(eocd);
|
||||
const reader = new BinaryReader(
|
||||
Buffer.from(bytes.buffer, bytes.byteOffset, bytes.byteLength),
|
||||
true,
|
||||
"zip",
|
||||
);
|
||||
// Zip64: a locator precedes the EOCD, pointing at the Zip64 EOCD record.
|
||||
const locator = eocdOffset - 20;
|
||||
if (
|
||||
locator < 0 ||
|
||||
dv.getUint32(locator, true) !== ZIP64_EOCD_LOCATOR_SIGNATURE
|
||||
) {
|
||||
const locator = eocdOffset - ZIP64_LOCATOR.bytes;
|
||||
if (locator < 0 || reader.u32(locator) !== ZIP64_LOCATOR.signature) {
|
||||
// Exactly 65,535 entries saturates the count without Zip64; that archive
|
||||
// is still described completely by its 32-bit EOCD fields.
|
||||
if (offset !== ZIP64_SENTINEL && size !== ZIP64_SENTINEL)
|
||||
if (
|
||||
offset !== ZIP_CLASSIC_DIRECTORY_LIMITS.offset &&
|
||||
size !== ZIP_CLASSIC_DIRECTORY_LIMITS.size
|
||||
)
|
||||
return classicDirectory(eocd);
|
||||
return err({
|
||||
kind: "malformed-zip",
|
||||
@@ -177,8 +182,8 @@ const centralDirectoryOffset = (
|
||||
}
|
||||
// APPNOTE 6.3.10 §4.3.15 makes the record offset relative to its disk.
|
||||
// A selected file cannot resolve another disk's offset as a local span.
|
||||
const recordDisk = dv.getUint32(locator + 4, true);
|
||||
const totalDisks = dv.getUint32(locator + 16, true);
|
||||
const recordDisk = reader.u32(locator + 4);
|
||||
const totalDisks = reader.u32(locator + 16);
|
||||
if (totalDisks === 0)
|
||||
return err({
|
||||
kind: "malformed-zip",
|
||||
@@ -196,46 +201,51 @@ const centralDirectoryOffset = (
|
||||
kind: "zip64-unsupported",
|
||||
detail: "multi-disk Zip64 archives are not supported",
|
||||
});
|
||||
const recordOffset = u64(dv, locator + 8);
|
||||
if (recordOffset === null)
|
||||
const offsetOnDisk = reader.word(locator + ZIP64_LOCATOR.recordOffset, 64);
|
||||
const maximumExact = BigInt(Number.MAX_SAFE_INTEGER);
|
||||
if (offsetOnDisk > maximumExact)
|
||||
return err({
|
||||
kind: "zip64-unsupported",
|
||||
detail: "Zip64 EOCD offset exceeds safe range",
|
||||
});
|
||||
if (recordOffset + 56 > locator)
|
||||
return err({
|
||||
kind: "malformed-zip",
|
||||
offset: locator + 8,
|
||||
detail: "Zip64 EOCD record extends beyond its locator",
|
||||
});
|
||||
if (dv.getUint32(recordOffset, true) !== ZIP64_EOCD_SIGNATURE)
|
||||
return err({
|
||||
kind: "malformed-zip",
|
||||
offset: recordOffset,
|
||||
detail: "invalid Zip64 EOCD signature",
|
||||
});
|
||||
// The size excludes the signature and size field. Check the declared
|
||||
// body before using its fixed fields to locate the central directory.
|
||||
const recordSize = dv.getBigUint64(recordOffset + 4, true);
|
||||
if (recordSize < 44n || recordSize > BigInt(locator - recordOffset - 12))
|
||||
return err({
|
||||
kind: "malformed-zip",
|
||||
offset: recordOffset + 4,
|
||||
detail: "invalid Zip64 EOCD record size",
|
||||
});
|
||||
const z64Offset = u64(dv, recordOffset + 48);
|
||||
const z64Size = u64(dv, recordOffset + 40);
|
||||
// The Zip64 record's total entry count; the 16-bit EOCD field saturates.
|
||||
const z64Entries = u64(dv, recordOffset + 32);
|
||||
if (z64Offset === null || z64Size === null || z64Entries === null)
|
||||
const recordOffset = Number(offsetOnDisk);
|
||||
const record = readZip64DirectoryRecord(reader, recordOffset, locator);
|
||||
if (!record.ok) {
|
||||
switch (record.error) {
|
||||
case "fixed-fields":
|
||||
return err({
|
||||
kind: "malformed-zip",
|
||||
offset: locator + ZIP64_LOCATOR.recordOffset,
|
||||
detail: "Zip64 EOCD record extends beyond its locator",
|
||||
});
|
||||
case "signature":
|
||||
return err({
|
||||
kind: "malformed-zip",
|
||||
offset: recordOffset,
|
||||
detail: "invalid Zip64 EOCD signature",
|
||||
});
|
||||
case "size":
|
||||
return err({
|
||||
kind: "malformed-zip",
|
||||
offset: recordOffset + ZIP64_END.sizeOffset,
|
||||
detail: "invalid Zip64 EOCD record size",
|
||||
});
|
||||
}
|
||||
}
|
||||
const { directoryOffset, directorySize, totalEntries } = record.value;
|
||||
if (
|
||||
directoryOffset > maximumExact ||
|
||||
directorySize > maximumExact ||
|
||||
totalEntries > maximumExact
|
||||
)
|
||||
return err({
|
||||
kind: "zip64-unsupported",
|
||||
detail: "Zip64 sizes exceed safe range",
|
||||
});
|
||||
return ok({
|
||||
offset: z64Offset,
|
||||
size: z64Size,
|
||||
entries: z64Entries,
|
||||
offset: Number(directoryOffset),
|
||||
size: Number(directorySize),
|
||||
entries: Number(totalEntries),
|
||||
end: recordOffset,
|
||||
});
|
||||
};
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { ZIP_EOCD_BYTES } from "../zipEndOfCentralDirectory.js";
|
||||
import { ZIP_SIGNATURES } from "../zipPackageFormat.js";
|
||||
import { XML_ELEMENT_OPENING } from "../xmlDocument.js";
|
||||
import {
|
||||
DESKTOP_VIEW_DEFAULTS,
|
||||
@@ -90,7 +92,9 @@ LINE_BREAK = re.compile(${JSON.stringify(DESKTOP_LINE_BREAK)})
|
||||
ECMASCRIPT_TRIM_CHARACTERS = ${PYTHON_ECMASCRIPT_TRIM_CHARACTERS}
|
||||
SQLITE_HEADER = b"SQLite format 3\x00"
|
||||
GVDB_HEADER = b"GVariant"
|
||||
ZIP_HEADER = b"PK\x03\x04"
|
||||
ZIP_SIGNATURES = (${ZIP_SIGNATURES.map((signature) => `bytes(${JSON.stringify(signature)})`).join(", ")})
|
||||
ZIP_HEADER = ZIP_SIGNATURES[0]
|
||||
ZIP_EOCD_BYTES = ${String(ZIP_EOCD_BYTES)}
|
||||
GZIP_HEADER = b"\x1f\x8b\x08"
|
||||
XML_SPACE = "xml:space"
|
||||
MAX_SAFE_INTEGER = 2**53 - 1
|
||||
@@ -428,6 +432,11 @@ def _bytes_view(name: str, data: bytes) -> dict:
|
||||
return _text_view(name, text, len(data))
|
||||
|
||||
|
||||
def _is_zip(data: bytes) -> bool:
|
||||
"""Recognize storage signatures before the directory reader checks validity."""
|
||||
return data.startswith(ZIP_HEADER) or (len(data) >= ZIP_EOCD_BYTES and data.startswith(ZIP_SIGNATURES))
|
||||
|
||||
|
||||
def _view(path: Path) -> dict:
|
||||
"""A typed JSON view of one file in the shapes of REA's state view; truncated says the view is partial."""
|
||||
too_large = {"format": None, "view": None, "error": VIEW_TOO_LARGE}
|
||||
@@ -446,7 +455,7 @@ def _view(path: Path) -> dict:
|
||||
return too_large
|
||||
if data.startswith(SQLITE_HEADER):
|
||||
return _sqlite(path)
|
||||
if data.startswith(ZIP_HEADER):
|
||||
if _is_zip(data):
|
||||
with path.open("rb") as file:
|
||||
return _zip(file, os.fstat(file.fileno()).st_size)
|
||||
if data.startswith(GZIP_HEADER):
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
import {
|
||||
ZIP_CLASSIC_DIRECTORY_LIMITS,
|
||||
ZIP_CLASSIC_DISK_OFFSETS,
|
||||
ZIP_SINGLE_VOLUME,
|
||||
ZIP64_END,
|
||||
ZIP64_LOCATOR,
|
||||
} from "../zipEndOfCentralDirectory.js";
|
||||
import { ZIP_EXTRA_FIELD } from "../zip/zipExtraFields.js";
|
||||
|
||||
/**
|
||||
* The exported module's zip and gzip readers: the central directory read a
|
||||
* record at a time, entries read in bounded chunks, a gzip file's document
|
||||
@@ -12,9 +21,14 @@ ZIP_END = b"PK\x05\x06"
|
||||
ZIP_CENTRAL = b"PK\x01\x02"
|
||||
ZIP64_LOCATOR = b"PK\x06\x07"
|
||||
ZIP64_END = b"PK\x06\x06"
|
||||
ZIP64_END_LAYOUT = ${JSON.stringify(ZIP64_END)}
|
||||
ZIP64_LOCATOR_LAYOUT = ${JSON.stringify(ZIP64_LOCATOR)}
|
||||
ZIP_CLASSIC_DIRECTORY_LIMITS = ${JSON.stringify(ZIP_CLASSIC_DIRECTORY_LIMITS)}
|
||||
ZIP_CLASSIC_DISK_OFFSETS = ${JSON.stringify(Object.values(ZIP_CLASSIC_DISK_OFFSETS))}
|
||||
ZIP_SINGLE_VOLUME = ${JSON.stringify(ZIP_SINGLE_VOLUME)}
|
||||
ZIP_EXTRA_FIELD = ${JSON.stringify(ZIP_EXTRA_FIELD)}
|
||||
#: Formats an archive entry may have whose reader needs a tool, a worker or another archive opened.
|
||||
NOT_READ_INSIDE = ((SQLITE_HEADER, "sqlite", "a SQLite database"), (GVDB_HEADER, "dconf", "a dconf database"),
|
||||
(ZIP_HEADER, "zip", "an archive"))
|
||||
NOT_READ_INSIDE = ((SQLITE_HEADER, "sqlite", "a SQLite database"), (GVDB_HEADER, "dconf", "a dconf database"))
|
||||
|
||||
|
||||
def _entry_view(name: str, data: bytes) -> dict:
|
||||
@@ -23,6 +37,8 @@ def _entry_view(name: str, data: bytes) -> dict:
|
||||
for header, form, inside in NOT_READ_INSIDE:
|
||||
if data.startswith(header):
|
||||
return {"format": form, "view": None, "error": f"{inside} inside an archive is not read"}
|
||||
if _is_zip(data):
|
||||
return {"format": "zip", "view": None, "error": "an archive inside an archive is not read"}
|
||||
return _bytes_view(name, data)
|
||||
|
||||
|
||||
@@ -74,11 +90,24 @@ def _zip_entry(archive, info, refusal, name):
|
||||
"view": view.get("view"), "view_error": view.get("error")}
|
||||
|
||||
|
||||
def _zip64_directory_record(file, record: int, locator: int):
|
||||
"""Read the fixed fields only, after the entire declared record fits before the locator."""
|
||||
fixed = ZIP64_END_LAYOUT["fixedBytes"]
|
||||
prefix = ZIP64_END_LAYOUT["sizePrefixBytes"]
|
||||
if record < 0 or record + fixed > locator:
|
||||
return None
|
||||
file.seek(record)
|
||||
header = file.read(fixed)
|
||||
if len(header) != fixed or header[:4] != ZIP64_END:
|
||||
return None
|
||||
body = struct.unpack_from("<Q", header, ZIP64_END_LAYOUT["sizeOffset"])[0]
|
||||
if body < fixed - prefix or record + prefix + body != locator:
|
||||
return None
|
||||
return header
|
||||
|
||||
|
||||
def _zip_end(file, size: int):
|
||||
"""Where an archive's central directory ends, how long it is and the offset it declares, from the last end
|
||||
record whose comment reaches the end of the file and, when a Zip64 locator precedes it whatever that record
|
||||
holds, the Zip64 one just before the locator; None without an end record, or when sentinels defer to a
|
||||
Zip64 record that is missing."""
|
||||
"""Read the classic end and any declared Zip64 frame from the single supplied volume."""
|
||||
window = min(size, 22 + 0xFFFF)
|
||||
file.seek(size - window)
|
||||
tail = file.read(window)
|
||||
@@ -88,32 +117,65 @@ def _zip_end(file, size: int):
|
||||
if at < 0:
|
||||
return None
|
||||
total, length, offset = struct.unpack_from("<HII", tail, at + 10)
|
||||
classic_disks = [struct.unpack_from("<H", tail, at + field)[0] for field in ZIP_CLASSIC_DISK_OFFSETS]
|
||||
end = size - window + at
|
||||
record = end - 20 - 56
|
||||
if record < 0:
|
||||
return end, length, offset
|
||||
file.seek(end - 20)
|
||||
if file.read(4) != ZIP64_LOCATOR:
|
||||
return end, length, offset
|
||||
file.seek(record)
|
||||
zip64 = file.read(56)
|
||||
if len(zip64) == 56 and zip64[:4] == ZIP64_END:
|
||||
length, offset = struct.unpack_from("<QQ", zip64, 40)
|
||||
classic = (end, length, offset) if all(disk in ZIP_SINGLE_VOLUME["classic"] for disk in classic_disks) else None
|
||||
locator_at = end - ZIP64_LOCATOR_LAYOUT["bytes"]
|
||||
if locator_at < 0:
|
||||
return classic
|
||||
file.seek(locator_at)
|
||||
locator = file.read(ZIP64_LOCATOR_LAYOUT["bytes"])
|
||||
if len(locator) != ZIP64_LOCATOR_LAYOUT["bytes"] or locator[:4] != ZIP64_LOCATOR:
|
||||
return classic
|
||||
declared = struct.unpack_from("<Q", locator, ZIP64_LOCATOR_LAYOUT["recordOffset"])[0]
|
||||
fallback = locator_at - ZIP64_END_LAYOUT["fixedBytes"]
|
||||
# The fixed fallback preserves archives whose offsets omit prepended bytes.
|
||||
for record in (declared, fallback):
|
||||
zip64 = _zip64_directory_record(file, record, locator_at)
|
||||
if zip64 is None:
|
||||
continue
|
||||
if (any(disk not in ZIP_SINGLE_VOLUME["zip64Classic"] for disk in classic_disks)
|
||||
or any(struct.unpack_from("<I", locator, field)[0] != value
|
||||
for field, value in ZIP_SINGLE_VOLUME["zip64Locator"])
|
||||
or any(struct.unpack_from("<I", zip64, field)[0] != value
|
||||
for field, value in ZIP_SINGLE_VOLUME["zip64End"])):
|
||||
return None
|
||||
length = struct.unpack_from("<Q", zip64, ZIP64_END_LAYOUT["directorySizeOffset"])[0]
|
||||
offset = struct.unpack_from("<Q", zip64, ZIP64_END_LAYOUT["directoryOffset"])[0]
|
||||
return record, length, offset
|
||||
if total == 0xFFFF or 0xFFFFFFFF in (length, offset):
|
||||
limits = ZIP_CLASSIC_DIRECTORY_LIMITS
|
||||
if total == limits["count"] or length == limits["size"] or offset == limits["offset"]:
|
||||
return None
|
||||
return end, length, offset
|
||||
return classic
|
||||
|
||||
|
||||
def _zip_unicode_path(raw: bytes, flags: int, extra: bytes):
|
||||
"""The last Info-ZIP Unicode Path, validated only after it replaces earlier claims, as in zip.js."""
|
||||
at, data, length = 0, None, 0
|
||||
while at + 4 <= len(extra):
|
||||
kind, size = struct.unpack_from("<HH", extra, at)
|
||||
if kind == 0x7075:
|
||||
data, length = extra[at + 4:at + 4 + size], size
|
||||
at += 4 + size
|
||||
usable = (data is not None and length >= 5 and len(data) == length and data[0] == 1 and not flags & 0x800
|
||||
def _zip_extras(extra: bytes):
|
||||
"""Check all framed fields before selection and return the last Unicode Path and the other original bytes."""
|
||||
at, copied, selected, kept = 0, 0, None, None
|
||||
header = ZIP_EXTRA_FIELD["headerBytes"]
|
||||
while at + header <= len(extra):
|
||||
payload = at + header
|
||||
size = struct.unpack_from("<H", extra, at + ZIP_EXTRA_FIELD["sizeOffset"])[0]
|
||||
end = payload + size
|
||||
if end > len(extra):
|
||||
raise zipfile.BadZipFile("Corrupt extra field")
|
||||
kind = struct.unpack_from("<H", extra, at + ZIP_EXTRA_FIELD["idOffset"])[0]
|
||||
if kind == ZIP_EXTRA_FIELD["unicodePath"]:
|
||||
selected = (payload, end)
|
||||
if kept is None:
|
||||
kept = bytearray()
|
||||
kept += extra[copied:at]
|
||||
copied = end
|
||||
at = end
|
||||
if selected is None:
|
||||
return None, extra
|
||||
kept += extra[copied:]
|
||||
return extra[selected[0]:selected[1]], kept
|
||||
|
||||
|
||||
def _zip_unicode_path(raw: bytes, flags: int, data):
|
||||
"""Use the last Info-ZIP Unicode Path only when it binds to the stored name, as zip.js does."""
|
||||
usable = (data is not None and len(data) >= 5 and data[0] == 1 and not flags & 0x800
|
||||
and int.from_bytes(data[1:5], "little") == zlib.crc32(raw))
|
||||
return data[5:] if usable else None
|
||||
|
||||
@@ -125,24 +187,13 @@ def _zip_name(raw: bytes, flags: int, unicode_path):
|
||||
return raw.decode("cp437") if unicode_path is None else unicode_path.decode("utf-8")
|
||||
|
||||
|
||||
def _zip_kept_record(fixed: bytes, rest: bytes, name_length: int, extra_length: int):
|
||||
def _zip_kept_record(fixed: bytes, raw: bytes, extra, comment: bytes):
|
||||
"""Remove already-interpreted Unicode claims from the bounded decoder copy. zipfile otherwise interprets each
|
||||
occurrence, retaining an earlier valid name or rejecting invalid UTF-8 that zip.js already superseded.
|
||||
All other fields, comments and the original artifact bytes remain unchanged."""
|
||||
raw, extra = rest[:name_length], rest[name_length:name_length + extra_length]
|
||||
kept, at = bytearray(), 0
|
||||
while at + 4 <= len(extra):
|
||||
kind, length = struct.unpack_from("<HH", extra, at)
|
||||
end = at + 4 + length
|
||||
if end > len(extra):
|
||||
raise zipfile.BadZipFile("Corrupt extra field")
|
||||
if kind != 0x7075:
|
||||
kept += extra[at:end]
|
||||
at = end
|
||||
kept += extra[at:]
|
||||
header = bytearray(fixed)
|
||||
struct.pack_into("<H", header, 30, len(kept))
|
||||
return header + raw + kept + rest[name_length + extra_length:]
|
||||
struct.pack_into("<H", header, 30, len(extra))
|
||||
return header + raw + extra + comment
|
||||
|
||||
|
||||
def _zip_directory(file, size: int, keep: int):
|
||||
@@ -169,12 +220,14 @@ def _zip_directory(file, size: int, keep: int):
|
||||
at += 46 + name_length + extra_length + comment_length
|
||||
if len(rest) < name_length + extra_length + comment_length or at > end:
|
||||
return None
|
||||
unicode_path = _zip_unicode_path(rest[:name_length], flags, rest[name_length:name_length + extra_length])
|
||||
name = _zip_name(rest[:name_length], flags, unicode_path)
|
||||
extra_end = name_length + extra_length
|
||||
raw = rest[:name_length]
|
||||
selected, extra = _zip_extras(rest[name_length:extra_end])
|
||||
name = _zip_name(raw, flags, _zip_unicode_path(raw, flags, selected))
|
||||
if not (name.endswith("/") or (external >> 16) & 0o170000 == 0o040000 or (made_by >> 8 == 0 and external & 0x10)):
|
||||
count += 1
|
||||
if len(records) < keep:
|
||||
records.append(_zip_kept_record(fixed, rest, name_length, extra_length))
|
||||
records.append(_zip_kept_record(fixed, raw, extra, rest[extra_end:]))
|
||||
names.append(name)
|
||||
return start, declared, records, count, names
|
||||
|
||||
@@ -222,16 +275,22 @@ def _zip(file, size: int) -> dict:
|
||||
if directory is None:
|
||||
raise zipfile.BadZipFile("no readable central directory")
|
||||
start, declared, records, count, names = directory
|
||||
kept = b"".join(records)
|
||||
offset = declared if declared <= 0xFFFFFFFF else start
|
||||
# zipfile follows a Zip64 locator signature wherever one sits before the end record, and the kept records
|
||||
# may end in those bytes, so a real Zip64 end record and locator for them come first.
|
||||
kept, retained = b"".join(records), len(records)
|
||||
limit = ZIP_CLASSIC_DIRECTORY_LIMITS
|
||||
offset = declared if declared <= limit["offset"] else start
|
||||
classic_count = min(retained, limit["count"])
|
||||
# zipfile follows a Zip64 locator signature wherever one sits before the end record, including in
|
||||
# a kept record's comment. Give it real Zip64 records with the exact selected directory facts.
|
||||
tail = (kept
|
||||
+ struct.pack("<4sQ2H2I4Q", ZIP64_END, 44, 45, 45, 0, 0, len(records), len(records), len(kept), offset)
|
||||
+ struct.pack("<4sQ2H2I4Q", ZIP64_END, 44, 45, 45, 0, 0, retained, retained, len(kept), offset)
|
||||
+ struct.pack("<4sIQI", ZIP64_LOCATOR, 0, offset + len(kept), 1)
|
||||
+ struct.pack("<4s4H2IH", ZIP_END, 0, 0, len(records), len(records), len(kept), offset, 0))
|
||||
+ struct.pack("<4s4H2IH", ZIP_END, 0, 0, classic_count, classic_count,
|
||||
min(len(kept), limit["size"]), min(offset, limit["offset"]), 0))
|
||||
with zipfile.ZipFile(_BoundedArchive(file, start, tail)) as archive:
|
||||
for info, name in zip(archive.infolist(), names, strict=True):
|
||||
infos = archive.infolist()
|
||||
if len(infos) != len(names):
|
||||
raise zipfile.BadZipFile("retained member count differs from decoded member count")
|
||||
for info, name in zip(infos, names):
|
||||
refusal = _zip_refusal(info, spent)
|
||||
spent += info.file_size if refusal is None else 0
|
||||
entries.append(_zip_entry(archive, info, refusal, name))
|
||||
@@ -279,8 +338,8 @@ def _gzip(name: str, data: bytes) -> dict:
|
||||
if MAX_ZIP_RATIO * len(data) < MAX_ZIP_ENTRY_BYTES
|
||||
else f"larger than the {_view_byte_limit(MAX_ZIP_ENTRY_BYTES)} a compressed document's view reads"}
|
||||
for header, form, inside in NOT_READ_INSIDE:
|
||||
if form != "zip" and document.startswith(header):
|
||||
if document.startswith(header):
|
||||
return {**compressed, "format": form, "error": f"{inside} inside a gzip file is not read"}
|
||||
view = _zip(io.BytesIO(document), len(document)) if document.startswith(ZIP_HEADER) else _bytes_view(name, document)
|
||||
view = _zip(io.BytesIO(document), len(document)) if _is_zip(document) else _bytes_view(name, document)
|
||||
return {**view, "compression": "gzip"}
|
||||
`;
|
||||
@@ -9,6 +9,8 @@ import {
|
||||
} from "./desktopViewPolicy.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import { safeParseJson } from "../safeJson.js";
|
||||
import { hasZipLocalHeader, hasZipSignature } from "../zipPackageFormat.js";
|
||||
import { ZIP_EOCD_BYTES } from "../zipEndOfCentralDirectory.js";
|
||||
import {
|
||||
declaredDocumentType,
|
||||
isWellFormedXml,
|
||||
@@ -42,8 +44,6 @@ export const SQLITE_SIDECARS = [
|
||||
{ suffix: "-journal", label: "rollback journal" },
|
||||
] as const;
|
||||
const GVDB_HEADER = "GVariant";
|
||||
/** A zip archive's local file header, which FreeCAD, ODF, OOXML, EPUB and JAR files begin with. */
|
||||
const ZIP_HEADER = "PK\u0003\u0004";
|
||||
const XML_EXTENSIONS = /\.(?:xml|xbel|plist|svg|ui)$/iu;
|
||||
/** A DOCTYPE that only names its DTD reads as XML; one with an internal subset, which can declare entities, does not. */
|
||||
export const XML_POLICY = {
|
||||
@@ -306,7 +306,11 @@ export const detectStateFormat = (
|
||||
const header = new TextDecoder("latin1").decode(bytes.subarray(0, 16));
|
||||
if (header === SQLITE_HEADER) return "sqlite";
|
||||
if (header.startsWith(GVDB_HEADER)) return "dconf";
|
||||
if (header.startsWith(ZIP_HEADER)) return "zip";
|
||||
if (
|
||||
hasZipLocalHeader(bytes) ||
|
||||
(bytes.length >= ZIP_EOCD_BYTES && hasZipSignature(bytes))
|
||||
)
|
||||
return "zip";
|
||||
const text = decodeStateText(bytes);
|
||||
if (text === undefined) return "binary";
|
||||
const trimmed = text.trimStart();
|
||||
|
||||
@@ -173,20 +173,25 @@ export const emptyEntriesArchive = (count: number): Buffer => {
|
||||
* and a classic one that defers to them, or with `sentinels` false keeps its
|
||||
* own counts, size and offset, as a writer may when they fit.
|
||||
*/
|
||||
export const withZip64End = (archive: Buffer, sentinels = true): Buffer => {
|
||||
export const withZip64End = (
|
||||
archive: Buffer,
|
||||
sentinels = true,
|
||||
extension: Uint8Array = new Uint8Array(),
|
||||
): Buffer => {
|
||||
const end = archive.length - 22;
|
||||
const entries = archive.readUInt16LE(end + 10);
|
||||
const size = archive.readUInt32LE(end + 12);
|
||||
const offset = archive.readUInt32LE(end + 16);
|
||||
const record = Buffer.alloc(56);
|
||||
const record = Buffer.alloc(56 + extension.length);
|
||||
record.writeUInt32LE(0x06064b50, 0);
|
||||
record.writeBigUInt64LE(44n, 4);
|
||||
record.writeBigUInt64LE(BigInt(44 + extension.length), 4);
|
||||
record.writeUInt16LE(45, 12);
|
||||
record.writeUInt16LE(45, 14);
|
||||
record.writeBigUInt64LE(BigInt(entries), 24);
|
||||
record.writeBigUInt64LE(BigInt(entries), 32);
|
||||
record.writeBigUInt64LE(BigInt(size), 40);
|
||||
record.writeBigUInt64LE(BigInt(offset), 48);
|
||||
record.set(extension, 56);
|
||||
const locator = Buffer.alloc(20);
|
||||
locator.writeUInt32LE(0x07064b50, 0);
|
||||
locator.writeBigUInt64LE(BigInt(end), 8);
|
||||
|
||||
@@ -1,20 +1,22 @@
|
||||
import { crc32 } from "node:zlib";
|
||||
|
||||
import { BinaryReader } from "../binaryContainer.js";
|
||||
import {
|
||||
ZIP_CLASSIC_DIRECTORY_LIMITS,
|
||||
ZIP64_END,
|
||||
ZIP64_LOCATOR,
|
||||
ZIP_EOCD_BYTES,
|
||||
ZIP_MAX_COMMENT_BYTES,
|
||||
readZip64DirectoryRecord,
|
||||
zip64EndUsesSingleVolume,
|
||||
zipEndOfCentralDirectoryCandidates,
|
||||
zipEndUsesSingleVolume,
|
||||
} from "../zipEndOfCentralDirectory.js";
|
||||
import { strictUtf8Decoder } from "../strictUtf8.js";
|
||||
import { ZIP_EXTRA_FIELD, lastZipExtraField } from "../zip/zipExtraFields.js";
|
||||
|
||||
const CENTRAL_HEADER = 0x02014b50;
|
||||
const CENTRAL_HEADER_BYTES = 46;
|
||||
const ZIP64_LOCATOR = 0x07064b50;
|
||||
const ZIP64_LOCATOR_BYTES = 20;
|
||||
const ZIP64_END = 0x06064b50;
|
||||
const ZIP64_END_BYTES = 56;
|
||||
const UTF8_NAME = 0x800;
|
||||
const UNICODE_PATH = 0x7075;
|
||||
const UNIX_TYPE = 0o170000;
|
||||
const UNIX_DIRECTORY = 0o040000;
|
||||
const MSDOS_DIRECTORY = 0x10;
|
||||
@@ -33,41 +35,45 @@ export interface ZipDirectory {
|
||||
}
|
||||
|
||||
/** Where the directory ends, how long it is and the offset it declares, from the end record and any Zip64 one. */
|
||||
const directoryEnd = (view: DataView) => {
|
||||
const window = Math.max(
|
||||
0,
|
||||
view.byteLength - ZIP_EOCD_BYTES - ZIP_MAX_COMMENT_BYTES,
|
||||
);
|
||||
const found = zipEndOfCentralDirectoryCandidates(
|
||||
new Uint8Array(
|
||||
view.buffer,
|
||||
view.byteOffset + window,
|
||||
view.byteLength - window,
|
||||
),
|
||||
).next();
|
||||
const directoryEnd = (reader: BinaryReader) => {
|
||||
const found = zipEndOfCentralDirectoryCandidates(reader.bytes).next();
|
||||
if (found.done === true) return undefined;
|
||||
const end = window + found.value.offset;
|
||||
const { totalEntries, directorySize, directoryOffset } = found.value;
|
||||
const classic = { end, size: directorySize, declaredOffset: directoryOffset };
|
||||
const record = end - ZIP64_LOCATOR_BYTES - ZIP64_END_BYTES;
|
||||
// Like Python's zipfile, a Zip64 locator is followed whether or not the classic record defers to it.
|
||||
if (
|
||||
record < 0 ||
|
||||
view.getUint32(end - ZIP64_LOCATOR_BYTES, true) !== ZIP64_LOCATOR
|
||||
)
|
||||
const eocd = found.value;
|
||||
const end = eocd.offset;
|
||||
const { totalEntries, directorySize, directoryOffset } = eocd;
|
||||
const classic = zipEndUsesSingleVolume(eocd)
|
||||
? { end, size: directorySize, declaredOffset: directoryOffset }
|
||||
: undefined;
|
||||
const locator = end - ZIP64_LOCATOR.bytes;
|
||||
// Like Python's zipfile, follow a locator whether or not the classic record defers to it.
|
||||
if (locator < 0 || reader.u32(locator) !== ZIP64_LOCATOR.signature)
|
||||
return classic;
|
||||
// Read just before its locator, where zipfile and zip.js look when the locator's own offset holds no Zip64 record.
|
||||
if (view.getUint32(record, true) === ZIP64_END)
|
||||
const declared = reader.word(locator + ZIP64_LOCATOR.recordOffset, 64);
|
||||
const fallback = locator - ZIP64_END.fixedBytes;
|
||||
// Keep the fixed-record fallback for archives with an unadjusted prepended offset.
|
||||
for (const at of [
|
||||
declared <= BigInt(fallback) ? Number(declared) : -1,
|
||||
fallback,
|
||||
]) {
|
||||
const record = readZip64DirectoryRecord(reader, at, locator);
|
||||
if (!record.ok || record.value.end !== locator) continue;
|
||||
if (
|
||||
!zipEndUsesSingleVolume(eocd, "zip64Classic") ||
|
||||
!zip64EndUsesSingleVolume(reader, locator, at)
|
||||
)
|
||||
return undefined;
|
||||
return {
|
||||
end: record,
|
||||
size: Number(view.getBigUint64(record + 40, true)),
|
||||
declaredOffset: Number(view.getBigUint64(record + 48, true)),
|
||||
end: at,
|
||||
size: Number(record.value.directorySize),
|
||||
declaredOffset: Number(record.value.directoryOffset),
|
||||
};
|
||||
// Without one the classic values stand, as zip.js reads them, unless they are sentinels deferring to it.
|
||||
}
|
||||
// Keep the classic values for a stray locator when the classic record is complete.
|
||||
const limits = ZIP_CLASSIC_DIRECTORY_LIMITS;
|
||||
const deferred =
|
||||
totalEntries === 0xffff ||
|
||||
directorySize === 0xffffffff ||
|
||||
directoryOffset === 0xffffffff;
|
||||
totalEntries === limits.count ||
|
||||
directorySize === limits.size ||
|
||||
directoryOffset === limits.offset;
|
||||
return deferred ? undefined : classic;
|
||||
};
|
||||
|
||||
@@ -79,45 +85,44 @@ interface RecordName {
|
||||
readonly flags: number;
|
||||
}
|
||||
|
||||
/** An Info-ZIP Unicode Path when zip.js uses it: version 1, no UTF-8 flag, and the CRC-32 of the stored name. */
|
||||
/** The usable last Unicode Path, or null when the extra block is malformed. */
|
||||
const unicodePath = (
|
||||
bytes: Uint8Array,
|
||||
view: DataView,
|
||||
reader: BinaryReader,
|
||||
{ name, extra, comment, flags }: RecordName,
|
||||
): Uint8Array | undefined => {
|
||||
): Uint8Array | undefined | null => {
|
||||
// zip.js indexes extra fields by ID with Map.set: the last occurrence
|
||||
// replaces earlier ones, even when that last claim fails validation.
|
||||
let selected: { at: number; length: number } | undefined;
|
||||
for (let at = extra; at + 4 <= comment;) {
|
||||
const length = view.getUint16(at + 2, true);
|
||||
if (view.getUint16(at, true) === UNICODE_PATH) selected = { at, length };
|
||||
at += 4 + length;
|
||||
}
|
||||
if (selected === undefined) return undefined;
|
||||
const { at, length } = selected;
|
||||
return length >= 5 &&
|
||||
at + 4 + length <= comment &&
|
||||
bytes[at + 4] === 1 &&
|
||||
const selected = lastZipExtraField(
|
||||
reader,
|
||||
extra,
|
||||
comment,
|
||||
ZIP_EXTRA_FIELD.unicodePath,
|
||||
);
|
||||
if (!selected.ok) return null;
|
||||
if (selected.value === undefined) return undefined;
|
||||
const { start, end } = selected.value;
|
||||
const { bytes } = reader;
|
||||
return end - start >= 5 &&
|
||||
bytes[start] === 1 &&
|
||||
(flags & UTF8_NAME) === 0 &&
|
||||
view.getUint32(at + 5, true) === crc32(bytes.subarray(name, extra))
|
||||
? bytes.subarray(at + 9, at + 4 + length)
|
||||
reader.u32(start + 1) === crc32(bytes.subarray(name, extra))
|
||||
? bytes.subarray(start + 5, end)
|
||||
: undefined;
|
||||
};
|
||||
|
||||
/**
|
||||
* Whether an entry's name, as REA decodes it, ends in a slash; undefined when
|
||||
* it must be UTF-8 and is not. A plain CP437 name is judged by its last byte,
|
||||
* so most records are read without allocating.
|
||||
* its extra block is malformed or its name must be UTF-8 and is not. A plain
|
||||
* CP437 name is judged by its last byte, so most records are read without allocating.
|
||||
*/
|
||||
const endsInSlash = (
|
||||
bytes: Uint8Array,
|
||||
view: DataView,
|
||||
reader: BinaryReader,
|
||||
record: RecordName,
|
||||
): boolean | undefined => {
|
||||
const unicode =
|
||||
record.comment > record.extra
|
||||
? unicodePath(bytes, view, record)
|
||||
: undefined;
|
||||
record.comment > record.extra ? unicodePath(reader, record) : undefined;
|
||||
if (unicode === null) return undefined;
|
||||
const { bytes } = reader;
|
||||
const utf8 =
|
||||
unicode ??
|
||||
((record.flags & UTF8_NAME) === 0
|
||||
@@ -143,8 +148,12 @@ export const scanZipDirectory = (
|
||||
bytes: Uint8Array,
|
||||
keep: number,
|
||||
): ZipDirectory | undefined => {
|
||||
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
||||
const located = directoryEnd(view);
|
||||
const reader = new BinaryReader(
|
||||
Buffer.from(bytes.buffer, bytes.byteOffset, bytes.byteLength),
|
||||
true,
|
||||
"zip",
|
||||
);
|
||||
const located = directoryEnd(reader);
|
||||
if (located === undefined) return undefined;
|
||||
const start = located.end - located.size;
|
||||
if (start < 0) return undefined;
|
||||
@@ -153,27 +162,26 @@ export const scanZipDirectory = (
|
||||
for (let at = start; at < located.end;) {
|
||||
if (
|
||||
at + CENTRAL_HEADER_BYTES > located.end ||
|
||||
view.getUint32(at, true) !== CENTRAL_HEADER
|
||||
reader.u32(at) !== CENTRAL_HEADER
|
||||
)
|
||||
return undefined;
|
||||
const name = at + CENTRAL_HEADER_BYTES;
|
||||
const extra = name + view.getUint16(at + 28, true);
|
||||
const comment = extra + view.getUint16(at + 30, true);
|
||||
const next = comment + view.getUint16(at + 32, true);
|
||||
const extra = name + reader.u16(at + 28);
|
||||
const comment = extra + reader.u16(at + 30);
|
||||
const next = comment + reader.u16(at + 32);
|
||||
if (next > located.end) return undefined;
|
||||
const slash = endsInSlash(bytes, view, {
|
||||
const slash = endsInSlash(reader, {
|
||||
name,
|
||||
extra,
|
||||
comment,
|
||||
flags: view.getUint16(at + 8, true),
|
||||
flags: reader.u16(at + 8),
|
||||
});
|
||||
if (slash === undefined) return undefined;
|
||||
const external = view.getUint32(at + 38, true);
|
||||
const external = reader.u32(at + 38);
|
||||
const directory =
|
||||
slash ||
|
||||
((external >>> 16) & UNIX_TYPE) === UNIX_DIRECTORY ||
|
||||
(view.getUint16(at + 4, true) >> 8 === 0 &&
|
||||
(external & MSDOS_DIRECTORY) !== 0);
|
||||
(reader.u16(at + 4) >> 8 === 0 && (external & MSDOS_DIRECTORY) !== 0);
|
||||
if (!directory) {
|
||||
count += 1;
|
||||
if (records.length < keep) records.push(bytes.subarray(at, next));
|
||||
@@ -189,24 +197,57 @@ export const boundedDirectoryTail = (directory: ZipDirectory): Uint8Array => {
|
||||
(total, record) => total + record.length,
|
||||
0,
|
||||
);
|
||||
const tail = new Uint8Array(size + ZIP_EOCD_BYTES);
|
||||
const {
|
||||
count: maxCount,
|
||||
size: maxSize,
|
||||
offset: maxOffset,
|
||||
} = ZIP_CLASSIC_DIRECTORY_LIMITS;
|
||||
const count = directory.records.length;
|
||||
// A declared offset past 32 bits cannot be right for a view-sized archive.
|
||||
const offset =
|
||||
directory.declaredOffset <= maxOffset
|
||||
? directory.declaredOffset
|
||||
: directory.start;
|
||||
const wide = count >= maxCount || size >= maxSize || offset >= maxOffset;
|
||||
const wideBytes = wide ? ZIP64_END.fixedBytes + ZIP64_LOCATOR.bytes : 0;
|
||||
const tail = new Uint8Array(size + wideBytes + ZIP_EOCD_BYTES);
|
||||
let at = 0;
|
||||
for (const record of directory.records) {
|
||||
tail.set(record, at);
|
||||
at += record.length;
|
||||
}
|
||||
const end = new DataView(tail.buffer, size);
|
||||
if (wide) {
|
||||
const record = new DataView(tail.buffer, size, ZIP64_END.fixedBytes);
|
||||
record.setUint32(0, ZIP64_END.signature, true);
|
||||
record.setBigUint64(
|
||||
ZIP64_END.sizeOffset,
|
||||
BigInt(ZIP64_END.fixedBytes - ZIP64_END.sizePrefixBytes),
|
||||
true,
|
||||
);
|
||||
record.setUint16(12, 45, true);
|
||||
record.setUint16(14, 45, true);
|
||||
record.setBigUint64(ZIP64_END.entriesOnDiskOffset, BigInt(count), true);
|
||||
record.setBigUint64(ZIP64_END.totalEntriesOffset, BigInt(count), true);
|
||||
record.setBigUint64(ZIP64_END.directorySizeOffset, BigInt(size), true);
|
||||
record.setBigUint64(ZIP64_END.directoryOffset, BigInt(offset), true);
|
||||
const locator = new DataView(
|
||||
tail.buffer,
|
||||
size + ZIP64_END.fixedBytes,
|
||||
ZIP64_LOCATOR.bytes,
|
||||
);
|
||||
locator.setUint32(0, ZIP64_LOCATOR.signature, true);
|
||||
locator.setBigUint64(
|
||||
ZIP64_LOCATOR.recordOffset,
|
||||
BigInt(offset) + BigInt(size),
|
||||
true,
|
||||
);
|
||||
locator.setUint32(16, 1, true);
|
||||
}
|
||||
const end = new DataView(tail.buffer, size + wideBytes, ZIP_EOCD_BYTES);
|
||||
end.setUint32(0, 0x06054b50, true);
|
||||
end.setUint16(8, directory.records.length, true);
|
||||
end.setUint16(10, directory.records.length, true);
|
||||
end.setUint32(12, size, true);
|
||||
// A declared offset past 32 bits cannot be right for a view-sized archive; the start stands in for it.
|
||||
end.setUint32(
|
||||
16,
|
||||
directory.declaredOffset <= 0xffffffff
|
||||
? directory.declaredOffset
|
||||
: directory.start,
|
||||
true,
|
||||
);
|
||||
end.setUint16(8, Math.min(count, maxCount), true);
|
||||
end.setUint16(10, Math.min(count, maxCount), true);
|
||||
end.setUint32(12, Math.min(size, maxSize), true);
|
||||
end.setUint32(16, Math.min(offset, maxOffset), true);
|
||||
return tail;
|
||||
};
|
||||
@@ -0,0 +1,34 @@
|
||||
import type { BinaryReader } from "../binaryContainer.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
|
||||
/** APPNOTE §4.5.1. The exported desktop reader uses the same field layout. */
|
||||
export const ZIP_EXTRA_FIELD = {
|
||||
headerBytes: 4,
|
||||
idOffset: 0,
|
||||
sizeOffset: 2,
|
||||
unicodePath: 0x7075,
|
||||
} as const;
|
||||
|
||||
/** The last matching payload, provided every framed field fits the extra block. */
|
||||
export const lastZipExtraField = (
|
||||
reader: BinaryReader,
|
||||
start: number,
|
||||
end: number,
|
||||
kind: number,
|
||||
): Result<
|
||||
{ readonly start: number; readonly end: number } | undefined,
|
||||
"overrun"
|
||||
> => {
|
||||
let selected: { readonly start: number; readonly end: number } | undefined;
|
||||
// Both desktop decoders have historically accepted a remainder shorter
|
||||
// than a header. Only a complete header declares a payload span.
|
||||
for (let at = start; at + ZIP_EXTRA_FIELD.headerBytes <= end;) {
|
||||
const payload = at + ZIP_EXTRA_FIELD.headerBytes;
|
||||
const next = payload + reader.u16(at + ZIP_EXTRA_FIELD.sizeOffset);
|
||||
if (next > end) return err("overrun");
|
||||
if (reader.u16(at + ZIP_EXTRA_FIELD.idOffset) === kind)
|
||||
selected = { start: payload, end: next };
|
||||
at = next;
|
||||
}
|
||||
return ok(selected);
|
||||
};
|
||||
@@ -1,11 +1,88 @@
|
||||
import { BinaryReader } from "./binaryContainer.js";
|
||||
import { err, ok, type Result } from "./result.js";
|
||||
|
||||
/** Classic ZIP EOCD fixed record and its format-defined comment limit. */
|
||||
export const ZIP_EOCD_BYTES = 22;
|
||||
export const ZIP_MAX_COMMENT_BYTES = 0xffff;
|
||||
|
||||
/** Saturated classic directory fields defer to the Zip64 end record. */
|
||||
export const ZIP_CLASSIC_DIRECTORY_LIMITS = {
|
||||
count: 0xffff,
|
||||
size: 0xffffffff,
|
||||
offset: 0xffffffff,
|
||||
} as const;
|
||||
|
||||
/** ZIP APPNOTE §§4.3.14–15. Shared fixed fields within the variable-size frame. */
|
||||
export const ZIP64_END = {
|
||||
signature: 0x06064b50,
|
||||
fixedBytes: 56,
|
||||
sizePrefixBytes: 12,
|
||||
sizeOffset: 4,
|
||||
entriesOnDiskOffset: 24,
|
||||
totalEntriesOffset: 32,
|
||||
directorySizeOffset: 40,
|
||||
directoryOffset: 48,
|
||||
} as const;
|
||||
|
||||
export const ZIP64_LOCATOR = {
|
||||
signature: 0x07064b50,
|
||||
bytes: 20,
|
||||
recordOffset: 8,
|
||||
} as const;
|
||||
|
||||
type Zip64DirectoryRecord = {
|
||||
readonly end: number;
|
||||
readonly totalEntries: bigint;
|
||||
readonly directorySize: bigint;
|
||||
readonly directoryOffset: bigint;
|
||||
};
|
||||
|
||||
/** Observe one complete Zip64 end record inside its declared carrier. */
|
||||
export const readZip64DirectoryRecord = (
|
||||
reader: BinaryReader,
|
||||
offset: number,
|
||||
limit: number,
|
||||
): Result<Zip64DirectoryRecord, "fixed-fields" | "signature" | "size"> => {
|
||||
const room = Math.min(limit, reader.bytes.length) - offset;
|
||||
if (offset < 0 || room < ZIP64_END.fixedBytes) return err("fixed-fields");
|
||||
if (reader.u32(offset) !== ZIP64_END.signature) return err("signature");
|
||||
const bodyBytes = reader.word(offset + ZIP64_END.sizeOffset, 64);
|
||||
if (
|
||||
bodyBytes < BigInt(ZIP64_END.fixedBytes - ZIP64_END.sizePrefixBytes) ||
|
||||
bodyBytes > BigInt(room - ZIP64_END.sizePrefixBytes)
|
||||
)
|
||||
return err("size");
|
||||
return ok({
|
||||
end: offset + ZIP64_END.sizePrefixBytes + Number(bodyBytes),
|
||||
totalEntries: reader.word(offset + ZIP64_END.totalEntriesOffset, 64),
|
||||
directorySize: reader.word(offset + ZIP64_END.directorySizeOffset, 64),
|
||||
directoryOffset: reader.word(offset + ZIP64_END.directoryOffset, 64),
|
||||
});
|
||||
};
|
||||
|
||||
const EOCD_SIGNATURE = Buffer.from([0x50, 0x4b, 0x05, 0x06]);
|
||||
|
||||
export const ZIP_CLASSIC_DISK_OFFSETS = { disk: 4, directoryDisk: 6 } as const;
|
||||
|
||||
/**
|
||||
* APPNOTE §§4.3.14–16 and §§4.4.19–20. The classic disk numbers can
|
||||
* defer to Zip64 only after a Zip64 end record has been located. The two
|
||||
* Zip64 layouts list 32-bit offsets and their required single-volume values.
|
||||
* The exported desktop reader consumes the same rules in Python.
|
||||
*/
|
||||
export const ZIP_SINGLE_VOLUME = {
|
||||
classic: [0],
|
||||
zip64Classic: [0, 0xffff],
|
||||
zip64Locator: [
|
||||
[4, 0],
|
||||
[16, 1],
|
||||
],
|
||||
zip64End: [
|
||||
[16, 0],
|
||||
[20, 0],
|
||||
],
|
||||
} as const;
|
||||
|
||||
export interface ZipEndOfCentralDirectory {
|
||||
/** Relative to the supplied byte window, which ends at the archive's end. */
|
||||
readonly offset: number;
|
||||
@@ -18,6 +95,31 @@ export interface ZipEndOfCentralDirectory {
|
||||
readonly commentLength: number;
|
||||
}
|
||||
|
||||
/** Whether the observed classic end record admits the selected single volume. */
|
||||
export const zipEndUsesSingleVolume = (
|
||||
{
|
||||
disk,
|
||||
directoryDisk,
|
||||
}: Pick<ZipEndOfCentralDirectory, "disk" | "directoryDisk">,
|
||||
layout: "classic" | "zip64Classic" = "classic",
|
||||
): boolean => {
|
||||
const allowed: readonly number[] = ZIP_SINGLE_VOLUME[layout];
|
||||
return allowed.includes(disk) && allowed.includes(directoryDisk);
|
||||
};
|
||||
|
||||
/** Check the disk fields in the located Zip64 locator and end record. */
|
||||
export const zip64EndUsesSingleVolume = (
|
||||
reader: BinaryReader,
|
||||
locator: number,
|
||||
end: number,
|
||||
): boolean =>
|
||||
ZIP_SINGLE_VOLUME.zip64Locator.every(
|
||||
([field, value]) => reader.u32(locator + field) === value,
|
||||
) &&
|
||||
ZIP_SINGLE_VOLUME.zip64End.every(
|
||||
([field, value]) => reader.u32(end + field) === value,
|
||||
);
|
||||
|
||||
/**
|
||||
* Observe complete classic records whose declared comment ends this window,
|
||||
* latest first. Candidate recognition does not establish directory validity,
|
||||
@@ -44,8 +146,10 @@ export function* zipEndOfCentralDirectoryCandidates(
|
||||
if (offset + ZIP_EOCD_BYTES + commentLength !== tail.length) continue;
|
||||
yield {
|
||||
offset: earliest + offset,
|
||||
disk: reader.u16(offset + 4),
|
||||
directoryDisk: reader.u16(offset + 6),
|
||||
disk: reader.u16(offset + ZIP_CLASSIC_DISK_OFFSETS.disk),
|
||||
directoryDisk: reader.u16(
|
||||
offset + ZIP_CLASSIC_DISK_OFFSETS.directoryDisk,
|
||||
),
|
||||
entriesOnDisk: reader.u16(offset + 8),
|
||||
totalEntries: reader.u16(offset + 10),
|
||||
directorySize: reader.u32(offset + 12),
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { BinaryReader } from "./binaryContainer.js";
|
||||
import {
|
||||
zipEndOfCentralDirectoryCandidates,
|
||||
zipEndUsesSingleVolume,
|
||||
ZIP_EOCD_BYTES,
|
||||
ZIP_MAX_COMMENT_BYTES,
|
||||
} from "./zipEndOfCentralDirectory.js";
|
||||
@@ -24,15 +25,24 @@ export const isZipPackageFormat = (
|
||||
): format is ZipPackageFormat =>
|
||||
ZIP_PACKAGE_FORMATS.some((candidate) => candidate === format);
|
||||
|
||||
/** Recognize local-header, classic/Zip64 empty-archive, or split ZIP signatures, not validity. */
|
||||
/** Local-header, classic/Zip64 empty-archive, and split ZIP signatures. */
|
||||
export const ZIP_SIGNATURES = [
|
||||
[0x50, 0x4b, 0x03, 0x04],
|
||||
[0x50, 0x4b, 0x05, 0x06],
|
||||
[0x50, 0x4b, 0x06, 0x06],
|
||||
[0x50, 0x4b, 0x07, 0x08],
|
||||
] as const;
|
||||
|
||||
/** A local file header can identify a ZIP before a complete end record is available. */
|
||||
export const hasZipLocalHeader = (bytes: Uint8Array): boolean =>
|
||||
ZIP_SIGNATURES[0].every((byte, index) => bytes[index] === byte);
|
||||
|
||||
/** Recognize a ZIP signature, not directory validity or supported storage. */
|
||||
export const hasZipSignature = (bytes: Uint8Array): boolean =>
|
||||
bytes.length >= 4 &&
|
||||
bytes[0] === 0x50 &&
|
||||
bytes[1] === 0x4b &&
|
||||
((bytes[2] === 0x03 && bytes[3] === 0x04) ||
|
||||
(bytes[2] === 0x05 && bytes[3] === 0x06) ||
|
||||
(bytes[2] === 0x06 && bytes[3] === 0x06) ||
|
||||
(bytes[2] === 0x07 && bytes[3] === 0x08));
|
||||
ZIP_SIGNATURES.some((signature) =>
|
||||
signature.every((byte, index) => bytes[index] === byte),
|
||||
);
|
||||
|
||||
const CENTRAL_DIRECTORY_HEADER = 0x02014b50;
|
||||
const LOCAL_FILE_HEADER = 0x04034b50;
|
||||
@@ -68,8 +78,7 @@ export const locateTrailingZipArchive = async (
|
||||
directoryOffset: declaredDirectory,
|
||||
} = candidate;
|
||||
if (
|
||||
candidate.disk !== 0 ||
|
||||
candidate.directoryDisk !== 0 ||
|
||||
!zipEndUsesSingleVolume(candidate) ||
|
||||
entriesOnDisk === 0 ||
|
||||
entriesOnDisk === 0xffff ||
|
||||
entriesOnDisk !== candidate.totalEntries ||
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { gzipSync } from "node:zlib";
|
||||
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
import { readZipView } from "../../../src/application/desktop/DesktopZipView.js";
|
||||
import {
|
||||
PYTHON_HELPERS,
|
||||
PYTHON_IMPORTS,
|
||||
} from "../../../src/domain/desktop/appExportPython.js";
|
||||
import { PYTHON_TASK_EVALUATOR } from "../../../src/domain/desktop/appTaskPython.js";
|
||||
import { evaluateTaskChecks } from "../../../src/domain/desktop/appTaskChecks.js";
|
||||
import { detectStateFormat } from "../../../src/domain/desktop/stateFormats.js";
|
||||
import {
|
||||
withZip64End,
|
||||
zipArchive,
|
||||
} from "../../../src/domain/desktop/zipArchive.fixture.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
|
||||
const empty = zipArchive([]);
|
||||
const commented = Buffer.concat([empty, Buffer.from("archive comment")]);
|
||||
commented.writeUInt16LE(commented.length - 22, 20);
|
||||
const archives = [
|
||||
["classic", empty],
|
||||
["commented", commented],
|
||||
["Zip64 sentinels", withZip64End(empty)],
|
||||
["Zip64 ordinary counts", withZip64End(empty, false)],
|
||||
] as const;
|
||||
|
||||
const pythonViews = async (files: readonly Buffer[]) => {
|
||||
const root = await createTestTempDirectory("desktop-empty-zip-");
|
||||
const paths = await Promise.all(
|
||||
files.map(async (bytes, index) => {
|
||||
const path = join(root, `${String(index)}.dat`);
|
||||
await writeFile(path, bytes);
|
||||
return path;
|
||||
}),
|
||||
);
|
||||
const imports = PYTHON_IMPORTS.split("\n")
|
||||
.filter((line) => !line.includes("agentenv_protocol"))
|
||||
.join("\n");
|
||||
const script = join(root, "read.py");
|
||||
await writeFile(
|
||||
script,
|
||||
`${imports}${PYTHON_HELPERS}${PYTHON_TASK_EVALUATOR}
|
||||
check = {"id": "empty", "kind": "value_equals", "file": "archive.zip", "pointer": "/entry_count", "value": 0}
|
||||
for path in json.load(sys.stdin):
|
||||
view = _view(Path(path))
|
||||
print(json.dumps({"file": view, "passed": evaluate([check], {"archive.zip": view})[0]["result"]}))
|
||||
`,
|
||||
);
|
||||
return execFileSync(
|
||||
process.platform === "win32" ? "python" : "python3",
|
||||
["-I", "-B", "-X", "utf8", script],
|
||||
{ input: JSON.stringify(paths), encoding: "utf8" },
|
||||
)
|
||||
.trim()
|
||||
.split(/\r?\n/u)
|
||||
.map(
|
||||
(line) =>
|
||||
JSON.parse(line) as {
|
||||
file: { format: string; view: unknown; error?: string };
|
||||
passed: boolean;
|
||||
},
|
||||
);
|
||||
};
|
||||
|
||||
it.each(archives)(
|
||||
"reads an empty %s archive without relying on its name",
|
||||
async (_name, bytes) => {
|
||||
const expected = { entries: [], entry_count: 0, truncated: false };
|
||||
expect(detectStateFormat("document.dat", bytes)).toBe("zip");
|
||||
expect(await readZipView(bytes)).toEqual({ ok: true, value: expected });
|
||||
expect((await pythonViews([bytes]))[0]).toEqual({
|
||||
file: { format: "zip", view: expected, truncated: false },
|
||||
passed: true,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps empty nested archives viewless instead of recursing or calling them binary", async () => {
|
||||
const outer = zipArchive(
|
||||
archives.map(([name, data]) => ({ name: `${name}.zip`, data })),
|
||||
);
|
||||
const native = await readZipView(outer);
|
||||
if (!native.ok) throw new Error(native.error);
|
||||
const python = (await pythonViews([outer]))[0]?.file;
|
||||
expect(python?.format).toBe("zip");
|
||||
expect(python?.view).toEqual(native.value);
|
||||
expect(native.value).toMatchObject({
|
||||
entries: archives.map(() => ({
|
||||
format: "zip",
|
||||
view: null,
|
||||
view_error: "an archive inside an archive is not read",
|
||||
})),
|
||||
entry_count: archives.length,
|
||||
truncated: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("grades removing every entry as a change from a nonempty archive", async () => {
|
||||
const files = [zipArchive([{ name: "old.txt", data: "old" }]), empty];
|
||||
const check = {
|
||||
id: "empty",
|
||||
kind: "value_equals" as const,
|
||||
file: "archive.zip",
|
||||
pointer: "/entry_count",
|
||||
value: 0,
|
||||
};
|
||||
const verdicts = await Promise.all(
|
||||
files.map(async (bytes) => {
|
||||
const read = await readZipView(bytes);
|
||||
if (!read.ok) throw new Error(read.error);
|
||||
return evaluateTaskChecks(
|
||||
[check],
|
||||
new Map([
|
||||
[
|
||||
check.file,
|
||||
{ format: detectStateFormat(check.file, bytes), view: read.value },
|
||||
],
|
||||
]),
|
||||
)[0]?.passed;
|
||||
}),
|
||||
);
|
||||
expect(verdicts).toEqual([false, true]);
|
||||
expect((await pythonViews(files)).map((row) => row.passed)).toEqual(verdicts);
|
||||
});
|
||||
|
||||
it("does not turn incomplete or damaged end records into empty views", async () => {
|
||||
const incomplete = empty.subarray(0, 6);
|
||||
expect(detectStateFormat("archive.zip", incomplete)).toBe("binary");
|
||||
const damaged = Buffer.from(empty);
|
||||
damaged.writeUInt16LE(1, 20);
|
||||
const zip64 = withZip64End(empty).subarray(0, 56);
|
||||
for (const bytes of [damaged, zip64]) {
|
||||
expect(detectStateFormat("archive.dat", bytes)).toBe("zip");
|
||||
expect((await readZipView(bytes)).ok).toBe(false);
|
||||
const python = (await pythonViews([bytes]))[0];
|
||||
expect(python?.file).toMatchObject({ format: "zip", view: null });
|
||||
expect(python?.file.error).toBeTypeOf("string");
|
||||
expect(python?.passed).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("reads the same empty archive when gzip wraps its bytes", async () => {
|
||||
const [view] = await pythonViews([gzipSync(empty)]);
|
||||
expect(view).toMatchObject({
|
||||
file: { format: "zip", compression: "gzip", view: { entry_count: 0 } },
|
||||
passed: true,
|
||||
});
|
||||
});
|
||||
@@ -1,3 +1,4 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { crc32 } from "node:zlib";
|
||||
@@ -18,6 +19,11 @@ import {
|
||||
type ZipMember,
|
||||
} from "../../../src/domain/desktop/zipArchive.fixture.js";
|
||||
import { zipEntryRefusal } from "../../../src/domain/desktop/zipView.js";
|
||||
import {
|
||||
desktopViewPolicy,
|
||||
pythonDesktopViewPolicy,
|
||||
} from "../../../src/domain/desktop/desktopViewPolicy.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import {
|
||||
expectSameCheckResults,
|
||||
expectSameFileViews,
|
||||
@@ -183,6 +189,33 @@ it("refuses invalid UTF-8 in the selected Unicode claim while preserving an unre
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses a central extra field whose payload extends past the containing record's extra bytes", async () => {
|
||||
const name = "report.txt";
|
||||
const opaque = Buffer.from([0xfe, 0xca, 2, 0, 1]);
|
||||
const unicode = unicodePathExtra(name, "renamed.txt");
|
||||
unicode.writeUInt16LE(unicode.length - 3, 2);
|
||||
const archives = [
|
||||
zipArchive([{ name, data: "observed", centralExtra: opaque }]),
|
||||
zipArchive([{ name, data: "observed", centralExtra: unicode }]),
|
||||
zipArchive([
|
||||
{ name: "directory/", centralExtra: opaque },
|
||||
{ name, data: "observed" },
|
||||
]),
|
||||
];
|
||||
const fromPython = await pythonFileViews(saved(archives));
|
||||
for (const [index, archive] of archives.entries()) {
|
||||
expect.soft(await readZipView(archive)).toEqual({
|
||||
ok: false,
|
||||
error: "not a readable zip archive",
|
||||
});
|
||||
expect.soft(fromPython[index]).toEqual({
|
||||
format: "zip",
|
||||
view: null,
|
||||
error: "not a readable zip archive",
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("excludes a final Unicode directory without consuming a retained file slot", async () => {
|
||||
const name = "settings.json";
|
||||
const archive = zipArchive([
|
||||
@@ -401,6 +434,153 @@ describe("zip directories in the exported module", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it.each([false, true])(
|
||||
"follows the Zip64 locator over a fixed-size signature in opaque extension bytes (%s)",
|
||||
async (sentinels) => {
|
||||
const extension = Buffer.alloc(6 + 56);
|
||||
extension.writeUInt16LE(0xfffe, 0);
|
||||
extension.writeUInt32LE(56, 2);
|
||||
extension.writeUInt32LE(0x06064b50, 6);
|
||||
const archive = withZip64End(
|
||||
zipArchive([{ name: "notes.txt", data: "retained" }]),
|
||||
sentinels,
|
||||
extension,
|
||||
);
|
||||
const result = await sameView(archive);
|
||||
expect(result).toMatchObject({
|
||||
entry_count: 1,
|
||||
entries: [{ name: "notes.txt", view: { lines: ["retained"] } }],
|
||||
truncated: false,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("retains the fixed Zip64 fallback when a prefix shifts its relative locator", async () => {
|
||||
// Keep a recognized ZIP opening for desktop format detection. Only the
|
||||
// real recorded local entry and central directory contribute to the view.
|
||||
const prefix = Buffer.from("PK\x03\x04inert prefix");
|
||||
const archive = Buffer.concat([
|
||||
prefix,
|
||||
withZip64End(zipArchive([{ name: "notes.txt", data: "retained" }])),
|
||||
]);
|
||||
const result = await sameView(archive);
|
||||
expect(result).toMatchObject({
|
||||
entry_count: 1,
|
||||
entries: [{ name: "notes.txt", view: { lines: ["retained"] } }],
|
||||
truncated: false,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("ZIP directory counts and volume admission in the exported module", () => {
|
||||
it("retains the selected ZIP count at and beyond the classic sentinel", async () => {
|
||||
const path = join(
|
||||
await createTestTempDirectory("rea-zip64-count-"),
|
||||
"many.zip",
|
||||
);
|
||||
// An independent writer emits an original Zip64 archive, not a decoder-copy fixture.
|
||||
execFileSync("python3", [
|
||||
"-I",
|
||||
"-B",
|
||||
"-c",
|
||||
`import sys, zipfile
|
||||
with zipfile.ZipFile(sys.argv[1], "w") as archive:
|
||||
for index in range(65536):
|
||||
archive.writestr(f"entry-{index:05d}.bin", b"ab")
|
||||
with zipfile.ZipFile(sys.argv[1]) as archive:
|
||||
assert len(archive.infolist()) == 65536`,
|
||||
path,
|
||||
]);
|
||||
const bytes = await readFile(path);
|
||||
for (const keep of [65_535, 65_536]) {
|
||||
const policy = desktopViewPolicy({
|
||||
zip_entries: keep,
|
||||
zip_entry_bytes: 1,
|
||||
});
|
||||
const result = await readZipView(bytes, undefined, policy);
|
||||
const host = result.ok
|
||||
? (result.value as {
|
||||
entries: { name: string }[];
|
||||
entry_count: number;
|
||||
truncated: boolean;
|
||||
})
|
||||
: null;
|
||||
const python: unknown = JSON.parse(
|
||||
exportedPython(
|
||||
`${pythonDesktopViewPolicy(policy)}
|
||||
result = _view(Path(json.load(sys.stdin)))
|
||||
view = result.get("view")
|
||||
print(json.dumps(None if view is None else {
|
||||
"listed": len(view["entries"]), "count": view["entry_count"],
|
||||
"truncated": view["truncated"], "last": view["entries"][-1]["name"]
|
||||
}))`,
|
||||
path,
|
||||
),
|
||||
);
|
||||
expect([
|
||||
host === null
|
||||
? null
|
||||
: {
|
||||
listed: host.entries.length,
|
||||
count: host.entry_count,
|
||||
truncated: host.truncated,
|
||||
last: host.entries.at(-1)?.name,
|
||||
},
|
||||
python,
|
||||
]).toEqual(
|
||||
Array.from({ length: 2 }, () => ({
|
||||
listed: keep,
|
||||
count: 65_536,
|
||||
truncated: keep < 65_536,
|
||||
last: `entry-${String(keep - 1).padStart(5, "0")}.bin`,
|
||||
})),
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("refuses archives whose selected directory or Zip64 locator names another volume", async () => {
|
||||
const classic = zipArchive([{ name: "settings.json", data: '{"zoom":2}' }]);
|
||||
const zip64 = withZip64End(classic);
|
||||
const variants = [
|
||||
["classic disk", classic, 22, 4, 1],
|
||||
["classic directory disk", classic, 22, 6, 1],
|
||||
["classic disk in Zip64", zip64, 22, 4, 1],
|
||||
["Zip64 record disk", zip64, 98, 16, 1],
|
||||
["Zip64 directory disk", zip64, 98, 20, 1],
|
||||
["Zip64 locator disk", zip64, 42, 4, 1],
|
||||
["Zip64 locator disk count", zip64, 42, 16, 2],
|
||||
] as const;
|
||||
const archives = variants.map(([, bytes, end, field, value]) => {
|
||||
const copy = Buffer.from(bytes);
|
||||
copy.writeUInt16LE(value, copy.length - end + field);
|
||||
return copy;
|
||||
});
|
||||
const python = await pythonFileViews(saved(archives));
|
||||
for (const [index, [description]] of variants.entries()) {
|
||||
expect(await readZipView(archives[index]!), description).toEqual({
|
||||
ok: false,
|
||||
error: "not a readable zip archive",
|
||||
});
|
||||
expect(python[index], description).toEqual({
|
||||
format: "zip",
|
||||
view: null,
|
||||
error: "not a readable zip archive",
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("uses the Zip64 record when the classic disk numbers defer to it", async () => {
|
||||
const archive = withZip64End(
|
||||
zipArchive([{ name: "settings.json", data: '{"zoom":2}' }]),
|
||||
);
|
||||
archive.writeUInt16LE(0xffff, archive.length - 18);
|
||||
archive.writeUInt16LE(0xffff, archive.length - 16);
|
||||
expect(await sameView(archive)).toMatchObject({
|
||||
entries: [{ name: "settings.json", view: { zoom: 2 } }],
|
||||
entry_count: 1,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps the classic values when bytes before the end record only look like a Zip64 locator", async () => {
|
||||
// The last name ends in a locator signature and 16 more bytes, which sit just before the end record.
|
||||
const name = `c-PK\u0006\u0007${"0".repeat(16)}`;
|
||||
|
||||
Reference in new issue
Block a user