From 2c3dd946d59c870388b03c3417eeb5218af96eeb Mon Sep 17 00:00:00 2001 From: Matt Van Horn Date: Thu, 22 Oct 2026 04:00:08 -0700 Subject: [PATCH] feat(python): inspect CPython pyc headers and marshal code objects (#4826) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(python): inspect CPython pyc headers and marshal code objects Add inspect_python_pyc / inspect-python-pyc to read one explicit local CPython 3.7–3.13 .pyc without executing bytecode or recovering source. * chore(python): drop unused PythonPycHeader export Knip reported the type as unused after inspect_python_pyc landed. * fix(python): bound pyc expansion and decode marshal like CPython - Count a nested code object reached through TYPE_REF once per copy the report carries, against max_objects and max_file_bytes, before counting or validating the tree - Charge long digits to max_string_bytes, reject out-of-range and unnormalized digits, build the value in one pass, and report integers above 256 bits by bit length only - Decode TYPE_UNICODE with surrogatepass semantics and reject other ill-formed UTF-8 - Map only 3.11 magics 3494-3495 to the py311 layout - Report non-finite float and complex parts as inf, -inf or nan - Allow an empty co_filename - Require bytes for co_linetable and co_exceptiontable - Bind cleanup-failure errors to their retained descriptor leases * fix(python): list pyc code objects once instead of a recursive tree Report each distinct code object once in code_objects, top-level first, with nested_code as indexes. A child shared through TYPE_REF is no longer copied per reference, and the MCP output schema is no longer recursive. Names repeated through TYPE_REF still count against max_file_bytes for each copy the report carries. * fix(python): bound shared pyc constants and reject malformed tables - Cache byte-string digests so a buffer shared through TYPE_REF is hashed once instead of on every reference. - Charge every reported co_consts entry, including tables shared through TYPE_REF, against max_objects before summaries are built. - Resolve the inspect-python-pyc path with absoluteCallerPath so link/../file.pyc reads the file the filesystem selects. - Reject TYPE_NULL inside marshal sequences, as CPython does. - Reject a co_localspluskinds length that does not match co_localsplusnames in the parser and the report schema. * fix(python): bound shared pyc names and read CPython marshal graphs Charge each reported code-constant name, keep integer bit lengths on the decoded value, and list code objects nested inside constant containers. Accept self-referential lists and dicts, marshal version 0 text floats, empty code names, the original string byte count, and Python set and dict lengths. Pass the CLI cancellation signal into command cleanup. Co-authored-by: morluto * fix(python): bound pyc key expansion and shared nested code Charge set and dictionary keys before a shared tuple doubles them, list each nested code object once per owner, and accept code objects as CPython keys. Co-authored-by: morluto * fix(python): correct pyc equality, counts, and nested walks Summarize code objects only after the marshal graph is complete, and collect nested code with an iterative walk that shares wrapper results instead of copying every descendant list. Keep tuple boundaries and the original string bytes in equality keys, and reject negative CPython counts. Co-authored-by: morluto * fix(python): bound shared pyc names and keep the first dict key Charge every copied name-table entry against the object budget before building the report array, including empty strings that add no bytes. When dictionary keys compare equal, keep the first key object and replace only its value, matching CPython. Co-authored-by: morluto * fix(python): walk each pyc dictionary child list once Keep a container's child array on the nested-code frame so a wide dictionary is not copied on every step of the synchronous marshal walk. Co-authored-by: morluto * fix(python): reuse stored pyc dictionary children when combining The nested walk keeps each container's child list on its frame. Combining nested code rebuilt that list, and the 8,000-key regression still accepted the old per-key copy. Use the stored list, and require that dictionary to finish well under the copy's runtime while still reporting a nested code object. Co-authored-by: morluto * fix(python): keep distinct pyc code objects and reject corrupt ones Encode each name table's length and entry boundaries in the equality key so a different name list cannot share a memo key. Walk tuple and frozenset keys on an explicit stack so a long TYPE_REF chain does not overflow. Treat code objects CPython rejects for an odd co_code or an undersized local table as malformed instead of complete coverage. Co-authored-by: morluto --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Cursor Agent Co-authored-by: morluto --- .../skills/reverse-engineer-anything/SKILL.md | 7 + .../references/native-and-artifacts.md | 10 + docs/.vitepress/config.ts | 1 + docs/cli.md | 11 + docs/index.md | 2 +- docs/python-pyc.md | 92 + scripts/lib/catalog-builders.mjs | 5 + scripts/lib/catalog-core.mjs | 2 + scripts/lib/module-boundaries.mjs | 1 + scripts/lib/test-projects.mjs | 2 +- src/application/CapabilityInventory.ts | 1 + src/application/python/PythonPycPort.ts | 20 + .../python/PythonPycService.test.ts | 123 ++ src/application/python/PythonPycService.ts | 102 ++ src/cli.ts | 2 + src/cli/pythonPycCommands.ts | 56 + src/cliCommandNames.ts | 1 + src/composition/pythonPyc.ts | 6 + .../python/pythonPycToolContracts.ts | 26 + src/contracts/toolContracts.ts | 2 + src/contracts/toolEffects.test.ts | 2 + src/contracts/toolEffects.ts | 1 + src/domain/python/pythonPyc.test.ts | 37 + src/domain/python/pythonPyc.ts | 298 +++ src/python/PythonPyc.errors.ts | 22 + src/python/PythonPyc.fixture.ts | 184 ++ src/python/PythonPycParser.test.ts | 1342 ++++++++++++++ src/python/PythonPycParser.ts | 1626 +++++++++++++++++ src/python/PythonPycProvider.ts | 174 ++ src/server/createServer.ts | 16 + src/server/registerPythonPycTool.ts | 31 + tests/acceptance/pythonPycCli.test.ts | 79 + .../tools/localToolContracts.test.ts | 1 + tests/boundary/filesystem/pythonPyc.test.ts | 60 + tests/boundary/mcp/pythonPycMcp.test.ts | 88 + .../stableArtifactConsumers.test.ts | 26 + 36 files changed, 4457 insertions(+), 2 deletions(-) create mode 100644 docs/python-pyc.md create mode 100644 src/application/python/PythonPycPort.ts create mode 100644 src/application/python/PythonPycService.test.ts create mode 100644 src/application/python/PythonPycService.ts create mode 100644 src/cli/pythonPycCommands.ts create mode 100644 src/composition/pythonPyc.ts create mode 100644 src/contracts/python/pythonPycToolContracts.ts create mode 100644 src/domain/python/pythonPyc.test.ts create mode 100644 src/domain/python/pythonPyc.ts create mode 100644 src/python/PythonPyc.errors.ts create mode 100644 src/python/PythonPyc.fixture.ts create mode 100644 src/python/PythonPycParser.test.ts create mode 100644 src/python/PythonPycParser.ts create mode 100644 src/python/PythonPycProvider.ts create mode 100644 src/server/registerPythonPycTool.ts create mode 100644 tests/acceptance/pythonPycCli.test.ts create mode 100644 tests/boundary/filesystem/pythonPyc.test.ts create mode 100644 tests/boundary/mcp/pythonPycMcp.test.ts diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md index 6a8caa4cbb..6e293a4669 100644 --- a/.agents/skills/reverse-engineer-anything/SKILL.md +++ b/.agents/skills/reverse-engineer-anything/SKILL.md @@ -233,6 +233,13 @@ explicit path or endpoint and do not need it. build-id or ABI-tag fields. A missing table is observed absence of those declarations, not proof the image never carried notes. See the [ELF notes guide](https://github.com/morluto/rea/blob/main/docs/elf-notes.md). +- CPython 3.7–3.13 `.pyc` header, nested marshal code objects and bytecode + identity: `inspect_python_pyc` with the explicit local file. This built-in + target-free inspection requires no Python installation. Preserve magic, + PEP 552 hash or mtime/size, nested names, and bytecode SHA-256. Bytecode is + not decompiled, and a decoded header does not prove the matching `.py` is + present. See the + [CPython pyc guide](https://github.com/morluto/rea/blob/main/docs/python-pyc.md). - Erlang BEAM IFF container: `inspect_erlang_beam` with the explicit local `.beam` file. This built-in target-free inspection requires no Erlang installation. Preserve chunk names, atom order, export/import/lambda tables, diff --git a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md index bbe49e7a76..4ab8155904 100644 --- a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md +++ b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md @@ -121,6 +121,16 @@ absence of those declarations, not proof the image never carried notes in another container. See [ELF note tables](https://github.com/morluto/rea/blob/main/docs/elf-notes.md). +## CPython pyc files + +Use `inspect_python_pyc` for an explicit local CPython 3.7–3.13 `.pyc` when the +question concerns the header magic, PEP 552 source hash or timestamp, nested +code-object names, or bytecode identity. No `open_binary`, Python installation, +or target execution is required. The CLI equivalent is `inspect-python-pyc PATH`. +Preserve artifact SHA-256, the 16-byte header fields, nested code names, and +bytecode SHA-256. Bytecode is not decompiled. A decoded header does not prove +the matching `.py` is present. See [CPython pyc inspection](https://github.com/morluto/rea/blob/main/docs/python-pyc.md). + ## Erlang BEAM files Use `inspect_erlang_beam` for an explicit local `FOR1`/`BEAM` file when the diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index a6ea20af62..14c911e8cd 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -34,6 +34,7 @@ export default defineConfig({ { text: "Rust dependencies", link: "/rust-binaries" }, { text: "PE certificates", link: "/pe-certificates" }, { text: "ELF notes", link: "/elf-notes" }, + { text: "CPython pyc files", link: "/python-pyc" }, { text: "Erlang BEAM", link: "/erlang-beam" }, { text: "JavaScript and Electron", diff --git a/docs/cli.md b/docs/cli.md index 4e21adf477..8aa5226722 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -177,6 +177,17 @@ The workflow reports note owners, types, descriptor hashes, and GNU build-id or ABI-tag fields when those descriptors match. A missing note table is observed absence of those declarations. See [ELF note tables](elf-notes.md). +Inspect a CPython 3.7–3.13 `.pyc` header and nested marshal code objects +without executing bytecode or installing Python: + +```bash +rea inspect-python-pyc /absolute/path/to/module.pyc --json +``` + +The workflow reports the pinned CPython version for the magic, PEP 552 hash or +mtime/size fields, nested function names, and bytecode SHA-256. Bytecode is +not decompiled. See [CPython pyc inspection](python-pyc.md). + Inspect an Erlang BEAM file without executing bytecode: ```bash diff --git a/docs/index.md b/docs/index.md index 9f7c9edce4..90309f161d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -50,7 +50,7 @@ For runtime instrumentation, see the [Frida guide](./frida-instrumentation.md). | Task | Guides | | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Identify a desktop installation | [Runtime candidates, evidence and follow-up files](desktop-applications.md) | -| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [PE certificates](pe-certificates.md), [ELF notes](elf-notes.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) | +| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [ELF notes](elf-notes.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) | | Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [Kuna](kuna-provider.md), [JEB](jeb-analysis.md) | | Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) | | Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) | diff --git a/docs/python-pyc.md b/docs/python-pyc.md new file mode 100644 index 0000000000..b68c74bf7a --- /dev/null +++ b/docs/python-pyc.md @@ -0,0 +1,92 @@ +# CPython pyc inspection + +`inspect_python_pyc` / `inspect-python-pyc` answers which CPython version wrote +one selected local `.pyc` file, how its 16-byte header records source identity, +and which nested code objects marshal stored. It reads the file without running +bytecode and requires no Python installation, active binary target, Hopper, +Ghidra, or IDA. + +```bash +rea inspect-python-pyc /absolute/path/to/module.pyc --json +``` + +The CLI resolves relative paths against the caller's working directory. MCP +requires an absolute filesystem path and preserves that exact selection: + +```json +{ + "name": "inspect_python_pyc", + "arguments": { "path": "/artifacts/module.pyc" } +} +``` + +## Observations + +The inline Evidence result contains the artifact path, SHA-256, and byte length. +`header` contains: + +- `magic`, the 16-bit little-endian field; +- `python_version` and `code_layout`, mapped from the pinned CPython 3.7–3.13 + table (`py37`, `py38`, or `py311`); 3.11 alpha magics before 3494 used + other code layouts and are unsupported; +- `flags`, `hash_based`, and `check_source` from PEP 552; +- `source_hash_hex` for hash-based pycs, or `source_mtime` and `source_size` + for timestamp pycs. + +`code_objects` lists each distinct marshal code object once, the top-level +module code first. `nested_code` holds the indexes of the distinct code objects +reachable from that object's constants, including code nested in tuples and +other containers and the owner itself when a constant container reaches it. +A function shared through a marshal reference is named by index once per owner, +even when that owner reaches it again through the same container. Relationships +are collected after the marshal graph is complete, so a container that a nested +code object also references is not summarized while its items are still missing. +Each object reports name, optional qualname, +filename, +argument counts, flags, names, locals (or `localsplusnames` on 3.11+), const +summaries, and bytecode length plus SHA-256. + +Bytecode is not decompiled. Const strings may be truncated in the summary while +the marshal walk still consumed the original bytes; a string summary's `bytes` +is that marshal length. Integer consts report `bits`; their decimal `value` is +`null` above 256 bits. Non-finite float and complex parts use CPython's `inf`, +`-inf`, and `nan` spellings. Dictionary and set lengths follow CPython: +unhashable keys and members fail the walk, equal set members count once, equal +dictionary keys keep the first key object and replace only its value, code +objects are valid keys and members, and a null dictionary value ends the dict +without that entry. + +## Limits and failures + +The selected file is limited to 16 MiB by default (64 MiB maximum). Marshal +object count, nesting depth, and per-string size are bounded; exceeding a +budget fails without partial success. Argument counts, positional-only counts, +keyword-only counts, local counts, and stack size are signed CPython fields; a +negative encoding is malformed. On the 3.11+ layout, `co_code` must cover a +whole number of code units. Every layout rejects argument counts that do not +fit the local table CPython checks (`co_varnames`, including `*args` / +`**kwargs`, or fast locals on 3.11+). Those files are malformed, so the +workflow does not report complete coverage. 3.7–3.10 still accept an odd +`co_code`, matching those constructors. Nested code collection walks reference +chains iteratively and keeps one shared descendant list for a container that +merely wraps another, instead of copying that list onto every wrapper. Set and +dictionary keys walk tuple and frozenset reference chains the same way, so a +long `TYPE_REF` chain stays within the call stack. A shared child stored twice +still counts its expanded key against the string budget. Code-object equality +keys record each name table's length and each entry's boundaries, so `()` and +`("",)` stay distinct, and `co_varnames`, `co_freevars`, and `co_cellvars` stay +separate tables. Names repeated through a marshal reference, including each +code-constant name the report copies, count against the file budget once per +copy. Empty strings add no bytes, so every copied +name-table entry (`co_names`, varnames, freevars, cellvars, and +`localsplusnames`) counts against the object budget before its array is built. +Copied constant tables and each reported nested code relationship count +against that same object budget. Integer digits and expanded +set or dictionary keys, including a shared tuple repeated at each level, count +against the string budget. Magics outside the pinned 3.7–3.13 table, +truncated headers, trailing marshal bytes, and a non-code top-level object are +typed failures. + +A decoded header does not prove the matching `.py` is present. `python_version` +is the magic table, not a running interpreter. This workflow does not recover +source. diff --git a/scripts/lib/catalog-builders.mjs b/scripts/lib/catalog-builders.mjs index 9ad4682393..c23f679096 100644 --- a/scripts/lib/catalog-builders.mjs +++ b/scripts/lib/catalog-builders.mjs @@ -38,6 +38,7 @@ export const toolFamilyCatalog = (sources) => { ...sources.il2cppContracts.IL2CPP_TOOL_CONTRACTS, ...sources.modelContracts.MODEL_TOOL_CONTRACTS, ...sources.rustContracts.RUST_TOOL_CONTRACTS, + ...sources.pythonPycContracts.PYTHON_PYC_TOOL_CONTRACTS, ...sources.erlangContracts.ERLANG_BEAM_TOOL_CONTRACTS, ...sources.wasmContracts.WASM_TOOL_CONTRACTS, ...sources.sqliteContracts.SQLITE_TOOL_CONTRACTS, @@ -317,6 +318,10 @@ export const providerCatalog = (sources) => { identity: sources.rustProvider.RUST_BINARY_PROVIDER_IDENTITY, contracts: sources.rustContracts.RUST_TOOL_CONTRACTS, }, + { + identity: sources.pythonPycProvider.PYTHON_PYC_PROVIDER_IDENTITY, + contracts: sources.pythonPycContracts.PYTHON_PYC_TOOL_CONTRACTS, + }, { identity: sources.erlangProvider.ERLANG_BEAM_PROVIDER_IDENTITY, contracts: sources.erlangContracts.ERLANG_BEAM_TOOL_CONTRACTS, diff --git a/scripts/lib/catalog-core.mjs b/scripts/lib/catalog-core.mjs index e0230f9f03..b700d1c621 100644 --- a/scripts/lib/catalog-core.mjs +++ b/scripts/lib/catalog-core.mjs @@ -77,6 +77,8 @@ const SOURCE_PATHS = { il2cppProvider: "dist/il2cpp/Il2cppMetadataProvider.js", rustContracts: "dist/contracts/rust/rustToolContracts.js", rustProvider: "dist/rust/RustBinaryProvider.js", + pythonPycContracts: "dist/contracts/python/pythonPycToolContracts.js", + pythonPycProvider: "dist/python/PythonPycProvider.js", erlangContracts: "dist/contracts/erlang/beamToolContracts.js", erlangProvider: "dist/erlang/BeamProvider.js", artifactContracts: "dist/contracts/artifactToolContracts.js", diff --git a/scripts/lib/module-boundaries.mjs b/scripts/lib/module-boundaries.mjs index 7577e953f5..315362873b 100644 --- a/scripts/lib/module-boundaries.mjs +++ b/scripts/lib/module-boundaries.mjs @@ -11,6 +11,7 @@ const MIGRATED_PROVIDER_ROOTS = new Set([ "il2cpp", "model", "gpu", + "python", "rust", "wasm", "captures", diff --git a/scripts/lib/test-projects.mjs b/scripts/lib/test-projects.mjs index 885df459c5..af9e567929 100644 --- a/scripts/lib/test-projects.mjs +++ b/scripts/lib/test-projects.mjs @@ -17,7 +17,7 @@ export const SOURCE_TEST_PROJECTS = [ group: "source", include: [ "src/*.test.ts", - "src/{android,apktool,jeb,adb,artifacts,browser,config,dotnet,electron,erlang,evm,firmware,flutter,pcap,ghidra,go,gpu,hopper,ida,inspector,javascript,native,process,reference,rust,server,sqlite,wasm,windows}/**/*.test.ts", + "src/{android,apktool,jeb,adb,artifacts,browser,config,dotnet,electron,erlang,evm,firmware,flutter,pcap,ghidra,go,gpu,hopper,ida,inspector,javascript,native,process,python,reference,rust,server,sqlite,wasm,windows}/**/*.test.ts", ], }, { diff --git a/src/application/CapabilityInventory.ts b/src/application/CapabilityInventory.ts index 6a337dcb7e..ba4c8f7d13 100644 --- a/src/application/CapabilityInventory.ts +++ b/src/application/CapabilityInventory.ts @@ -418,6 +418,7 @@ const workflowAvailabilityFor = ({ name === "inspect_model_artifact" || name === "inspect_gpu_code_objects" || name === "inspect_rust_binary" || + name === "inspect_python_pyc" || name === "inspect_erlang_beam" || name === "inspect_electron_fuses" ) diff --git a/src/application/python/PythonPycPort.ts b/src/application/python/PythonPycPort.ts new file mode 100644 index 0000000000..09a0964559 --- /dev/null +++ b/src/application/python/PythonPycPort.ts @@ -0,0 +1,20 @@ +import type { + ExecutionOptions, + ProviderIdentity, +} from "../AnalysisProvider.js"; +import type { AnalysisError } from "../../domain/analysisErrorBase.js"; +import type { + InspectPythonPycInput, + PythonPyc, +} from "../../domain/python/pythonPyc.js"; +import type { Result } from "../../domain/result.js"; + +/** Stable read and CPython pyc decoding, independent of a disassembler session. */ +export interface PythonPycPort { + readonly identity: ProviderIdentity; + close?(): Promise; + inspect( + input: InspectPythonPycInput, + options?: ExecutionOptions, + ): Promise>; +} diff --git a/src/application/python/PythonPycService.test.ts b/src/application/python/PythonPycService.test.ts new file mode 100644 index 0000000000..fb426512e7 --- /dev/null +++ b/src/application/python/PythonPycService.test.ts @@ -0,0 +1,123 @@ +import { expect, it } from "vitest"; +import { + AnalysisInputError, + AnalysisCancelledError, +} from "../../domain/analysisErrorCore.js"; +import type { PythonPyc } from "../../domain/python/pythonPyc.js"; +import { ok } from "../../domain/result.js"; +import { PythonPycService } from "./PythonPycService.js"; + +const identity = { + id: "fixture-python-pyc", + name: "Source-owned pyc reader seam", + version: "1", +}; + +const report: PythonPyc = { + artifact: { + path: "/artifacts/sample.pyc", + sha256: "a".repeat(64), + bytes: 64, + }, + header: { + magic: 3531, + python_version: "3.12", + code_layout: "py311", + flags: 0, + hash_based: false, + check_source: false, + source_hash_hex: null, + source_mtime: 1, + source_size: 10, + }, + code_objects: [ + { + name: "", + qualname: "", + filename: "sample.py", + firstlineno: 1, + argcount: 0, + posonlyargcount: 0, + kwonlyargcount: 0, + nlocals: null, + stacksize: 1, + flags: 0, + names: [], + varnames: null, + freevars: null, + cellvars: null, + localsplusnames: [], + localspluskinds_hex: "", + bytecode: { bytes: 2, sha256: "b".repeat(64) }, + consts: [{ kind: "none" }], + nested_code: [], + }, + ], + coverage: { + status: "complete", + objects: 4, + code_objects: 1, + marshal_bytes: 48, + marshal_depth: 1, + }, + limitations: [ + "Bytecode is retained as length and SHA-256 only; this workflow does not decompile, disassemble, or recover source.", + ], +}; + +it("rejects a relative path and extra arguments before reading", async () => { + let calls = 0; + const service = new PythonPycService({ + identity, + inspect: () => { + calls += 1; + return Promise.resolve(ok(report)); + }, + }); + const relative = await service.inspect({ path: "sample.pyc" }); + const extra = await service.inspect({ + path: report.artifact.path, + detail: "complete", + }); + expect(relative.ok).toBe(false); + expect(extra.ok).toBe(false); + if (!relative.ok) expect(relative.error).toBeInstanceOf(AnalysisInputError); + if (!extra.ok) expect(extra.error).toBeInstanceOf(AnalysisInputError); + expect(calls).toBe(0); +}); + +it("records the selected artifact and the header plus marshal ranges", async () => { + const result = await new PythonPycService({ + identity, + inspect: () => Promise.resolve(ok(report)), + }).inspect({ path: report.artifact.path }); + expect(result.ok).toBe(true); + if (!result.ok) throw result.error; + expect(result.value).toMatchObject({ + confidence: "observed", + subject: { format: "file" }, + normalized_result: report, + }); + expect(result.value.locations).toEqual([ + { kind: "artifact-path", path: report.artifact.path }, + { kind: "file-offset-range", start: 0, end: 16 }, + { kind: "file-offset-range", start: 16, end: report.artifact.bytes }, + ]); +}); + +it("discards an observation if the caller cancels while the provider completes", async () => { + const controller = new AbortController(); + const service = new PythonPycService({ + identity, + inspect: () => { + controller.abort(); + return Promise.resolve(ok(report)); + }, + }); + const result = await service.inspect( + { path: report.artifact.path }, + { signal: controller.signal }, + ); + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error).toBeInstanceOf(AnalysisCancelledError); +}); diff --git a/src/application/python/PythonPycService.ts b/src/application/python/PythonPycService.ts new file mode 100644 index 0000000000..464ce83d37 --- /dev/null +++ b/src/application/python/PythonPycService.ts @@ -0,0 +1,102 @@ +import { isAbsolute } from "node:path"; +import type { ExecutionOptions } from "../AnalysisProvider.js"; +import type { PythonPycPort } from "./PythonPycPort.js"; +import type { AnalysisError } from "../../domain/analysisErrorBase.js"; +import { + AnalysisCancelledError, + AnalysisInputError, + AnalysisOutputError, +} from "../../domain/analysisErrorCore.js"; +import { createEvidence, type Evidence } from "../../domain/evidence.js"; +import { + inspectPythonPycInputSchema, + PYTHON_PYC_HEADER_BYTES, + pythonPycSchema, +} from "../../domain/python/pythonPyc.js"; +import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js"; +import { err, ok, type Result } from "../../domain/result.js"; + +const OPERATION = "inspect_python_pyc"; + +/** Shared caller validation, output parsing and Evidence provenance for CLI and MCP. */ +export class PythonPycService { + constructor(readonly provider: PythonPycPort) {} + + close(): Promise { + return this.provider.close?.() ?? Promise.resolve(); + } + + async inspect( + rawInput: unknown, + options?: ExecutionOptions, + ): Promise> { + if (options?.signal?.aborted) + return err(new AnalysisCancelledError(OPERATION)); + const input = inspectPythonPycInputSchema.safeParse(rawInput); + if (!input.success) + return err( + analysisInputErrorFromIssues(OPERATION, input.error.issues, rawInput, { + cause: input.error, + }), + ); + if (!isAbsolute(input.data.path)) + return err( + new AnalysisInputError(OPERATION, undefined, [ + { + path: ["path"], + reason: "invalid_format", + message: "Expected an absolute filesystem path on this host.", + }, + ]), + ); + const result = await this.provider.inspect(input.data, options); + if (!result.ok) return result; + if (options?.signal?.aborted) + return err(new AnalysisCancelledError(OPERATION)); + const report = pythonPycSchema.safeParse(result.value); + if ( + !report.success || + report.data.artifact.path !== input.data.path || + report.data.artifact.bytes > input.data.max_file_bytes || + report.data.coverage.objects > input.data.max_objects || + report.data.coverage.marshal_depth > input.data.max_depth + ) + return err( + new AnalysisOutputError( + OPERATION, + "Python pyc provider returned malformed output or changed artifact identity.", + ), + ); + const value = report.data; + return ok( + createEvidence( + { + path: value.artifact.path, + format: "file", + sha256: value.artifact.sha256, + }, + this.provider.identity, + { + operation: OPERATION, + parameters: input.data, + result: value, + confidence: "observed", + limitations: value.limitations, + locations: [ + { kind: "artifact-path", path: value.artifact.path }, + { + kind: "file-offset-range", + start: 0, + end: PYTHON_PYC_HEADER_BYTES, + }, + { + kind: "file-offset-range", + start: PYTHON_PYC_HEADER_BYTES, + end: value.artifact.bytes, + }, + ], + }, + ), + ); + } +} diff --git a/src/cli.ts b/src/cli.ts index 0eabbc4664..758db31b1d 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -11,6 +11,7 @@ import { registerGoCommands } from "./cli/goCommands.js"; import { registerModelCommands } from "./cli/modelCommands.js"; import { registerIl2cppCommands } from "./cli/il2cppCommands.js"; import { registerRustCommands } from "./cli/rustCommands.js"; +import { registerPythonPycCommands } from "./cli/pythonPycCommands.js"; import { registerErlangCommands } from "./cli/erlangCommands.js"; import { registerSqliteCommands } from "./cli/sqliteCommands.js"; import { registerAnalysisViewCommands } from "./cli/analysisViewCommands.js"; @@ -140,6 +141,7 @@ export const createCli = ( registerModelCommands(cli, logger); registerGpuCommands(cli, logger); registerRustCommands(cli, logger); + registerPythonPycCommands(cli, logger); registerErlangCommands(cli, logger); registerSqliteCommands(cli, logger, environment); registerArtifactCommands(cli, logger, analysis.runProviderAnalysis); diff --git a/src/cli/pythonPycCommands.ts b/src/cli/pythonPycCommands.ts new file mode 100644 index 0000000000..cf79aaee4d --- /dev/null +++ b/src/cli/pythonPycCommands.ts @@ -0,0 +1,56 @@ +import { z } from "incur"; +import { absoluteCallerPath } from "../filesystem/callerPath.js"; +import { createPythonPycService } from "../composition/pythonPyc.js"; +import { CLI_COMMANDS } from "../cliCommandNames.js"; +import { logCliCommand } from "../cliLogging.js"; +import type { Logger } from "pino"; +import type { CliInstance } from "./types.js"; +import { withCommandCancellation } from "./commandCancellation.js"; +import { inspectPythonPycInputSchema } from "../domain/python/pythonPyc.js"; +import { finishAnalysisCommand } from "./finishAnalysisCommand.js"; + +/** Inspect operator-selected pyc files through the same shared workflow as MCP. */ +export const registerPythonPycCommands = ( + cli: CliInstance, + logger: Logger, +): void => { + cli.command(CLI_COMMANDS.inspectPythonPyc, { + description: + "Inspect a CPython 3.7–3.13 pyc header and marshal code objects without executing bytecode", + args: z.object({ + path: z.string().describe("Local CPython .pyc file"), + }), + options: z.object({ + maxFileBytes: inspectPythonPycInputSchema.shape.max_file_bytes, + maxObjects: inspectPythonPycInputSchema.shape.max_objects, + maxDepth: inspectPythonPycInputSchema.shape.max_depth, + maxStringBytes: inspectPythonPycInputSchema.shape.max_string_bytes, + }), + run: ({ args, options }) => + withCommandCancellation(async (signal) => { + const service = createPythonPycService(); + return await logCliCommand( + logger, + CLI_COMMANDS.inspectPythonPyc, + async () => + finishAnalysisCommand( + () => + service.inspect( + { + path: absoluteCallerPath(args.path), + max_file_bytes: options.maxFileBytes, + max_objects: options.maxObjects, + max_depth: options.maxDepth, + max_string_bytes: options.maxStringBytes, + }, + { signal }, + ), + service, + "rea-python-pyc", + "inspect_python_pyc", + signal, + ), + ); + }), + }); +}; diff --git a/src/cliCommandNames.ts b/src/cliCommandNames.ts index 7e3cb64d2c..de3faeb4de 100644 --- a/src/cliCommandNames.ts +++ b/src/cliCommandNames.ts @@ -65,6 +65,7 @@ export const CLI_COMMANDS = Object.freeze({ inspectModelArtifact: "inspect-model-artifact", inspectGpuCodeObjects: "inspect-gpu-code-objects", inspectRustBinary: "inspect-rust-binary", + inspectPythonPyc: "inspect-python-pyc", inspectErlangBeam: "inspect-erlang-beam", inspectPeResources: "inspect-pe-resources", inspectPeCertificates: "inspect-pe-certificates", diff --git a/src/composition/pythonPyc.ts b/src/composition/pythonPyc.ts new file mode 100644 index 0000000000..85262ee12d --- /dev/null +++ b/src/composition/pythonPyc.ts @@ -0,0 +1,6 @@ +import { PythonPycService } from "../application/python/PythonPycService.js"; +import { PythonPycProvider } from "../python/PythonPycProvider.js"; + +/** Compose read-only pyc inspection without invoking CPython or writing files. */ +export const createPythonPycService = (): PythonPycService => + new PythonPycService(new PythonPycProvider()); diff --git a/src/contracts/python/pythonPycToolContracts.ts b/src/contracts/python/pythonPycToolContracts.ts new file mode 100644 index 0000000000..7db11087d8 --- /dev/null +++ b/src/contracts/python/pythonPycToolContracts.ts @@ -0,0 +1,26 @@ +import { + inspectPythonPycInputSchema, + pythonPycSchema, +} from "../../domain/python/pythonPyc.js"; +import type { ToolContract } from "../toolContractTypes.js"; +import { toolContractMetadata } from "../toolEffects.js"; +import { evidenceResultOf } from "../toolOutputSchemaPrimitives.js"; + +/** Static CPython pyc inspection is independent of active native-analysis providers. */ +export const PYTHON_PYC_TOOL_CONTRACTS = [ + { + name: "inspect_python_pyc", + ...toolContractMetadata("inspect_python_pyc"), + kind: "artifact-provider", + description: + "Inspect one explicit local CPython 3.7–3.13 .pyc file without executing bytecode, invoking CPython or recovering source. Returns the 16-byte header (magic, pinned python_version, PEP 552 flags, source hash or mtime/size), the top-level code object with nested functions, names, const summaries, and bytecode length/SHA-256. Marshal is walked with object, depth and string budgets; unknown magics, truncated streams and exhausted budgets return no partial success. A decoded header is not proof the matching .py is present. Portable on supported Node hosts; no Python installation required.", + inputSchema: inspectPythonPycInputSchema, + outputSchema: evidenceResultOf(pythonPycSchema), + examples: [ + { + title: "Inspect a CPython 3.12 pyc header and nested code objects", + input: { path: "/artifacts/module.pyc" }, + }, + ], + }, +] as const satisfies readonly ToolContract[]; diff --git a/src/contracts/toolContracts.ts b/src/contracts/toolContracts.ts index a1426b7aa6..a6c6cda087 100644 --- a/src/contracts/toolContracts.ts +++ b/src/contracts/toolContracts.ts @@ -8,6 +8,7 @@ import { GO_TOOL_CONTRACTS } from "./go/goToolContracts.js"; import { MODEL_TOOL_CONTRACTS } from "./model/modelToolContracts.js"; import { IL2CPP_TOOL_CONTRACTS } from "./il2cpp/il2cppToolContracts.js"; import { RUST_TOOL_CONTRACTS } from "./rust/rustToolContracts.js"; +import { PYTHON_PYC_TOOL_CONTRACTS } from "./python/pythonPycToolContracts.js"; import { ERLANG_BEAM_TOOL_CONTRACTS } from "./erlang/beamToolContracts.js"; import { SQLITE_TOOL_CONTRACTS } from "./sqlite/sqliteToolContracts.js"; import { ANALYSIS_VIEW_TOOL_CONTRACTS } from "./analysisViewToolContracts.js"; @@ -55,6 +56,7 @@ export const TOOL_CONTRACTS = [ ...MODEL_TOOL_CONTRACTS, ...GPU_TOOL_CONTRACTS, ...RUST_TOOL_CONTRACTS, + ...PYTHON_PYC_TOOL_CONTRACTS, ...ERLANG_BEAM_TOOL_CONTRACTS, ...WASM_TOOL_CONTRACTS, ...SQLITE_TOOL_CONTRACTS, diff --git a/src/contracts/toolEffects.test.ts b/src/contracts/toolEffects.test.ts index 31f2c13c3f..b5509c4ccd 100644 --- a/src/contracts/toolEffects.test.ts +++ b/src/contracts/toolEffects.test.ts @@ -35,6 +35,8 @@ const EXECUTION_WORDING_EXCEPTIONS: Readonly> = { inspect_go_binary: "reads embedded metadata without executing the target", inspect_rust_binary: "reads an embedded dependency list without executing the target", + inspect_python_pyc: + "states that it does not execute bytecode or invoke CPython", inspect_erlang_beam: "states that it does not execute bytecode", inspect_sqlite_database: "uses a trusted inspector worker without executing application code or caller SQL", diff --git a/src/contracts/toolEffects.ts b/src/contracts/toolEffects.ts index 7dea6081da..3d7e7ed062 100644 --- a/src/contracts/toolEffects.ts +++ b/src/contracts/toolEffects.ts @@ -131,6 +131,7 @@ export const TOOL_EFFECTS: Readonly> = { inspect_model_artifact: evidence, inspect_gpu_code_objects: evidence, inspect_rust_binary: evidence, + inspect_python_pyc: evidence, inspect_erlang_beam: evidence, inspect_sqlite_database: effects({ mutatesSession: true, diff --git a/src/domain/python/pythonPyc.test.ts b/src/domain/python/pythonPyc.test.ts new file mode 100644 index 0000000000..ea98b5d25c --- /dev/null +++ b/src/domain/python/pythonPyc.test.ts @@ -0,0 +1,37 @@ +import { expect, it } from "vitest"; +import { + inspectPythonPycInputSchema, + pythonPycMagicEntry, + PYTHON_PYC_MAGIC_TABLE, +} from "./pythonPyc.js"; + +it("maps pinned CPython 3.7–3.13 magics and leaves gaps unknown", () => { + expect(pythonPycMagicEntry(3394)).toEqual({ + python_version: "3.7", + code_layout: "py37", + }); + expect(pythonPycMagicEntry(3413)).toEqual({ + python_version: "3.8", + code_layout: "py38", + }); + expect(pythonPycMagicEntry(3531)).toEqual({ + python_version: "3.12", + code_layout: "py311", + }); + expect(pythonPycMagicEntry(3571)).toEqual({ + python_version: "3.13", + code_layout: "py311", + }); + expect(pythonPycMagicEntry(3450)).toBeNull(); + expect(pythonPycMagicEntry(62171)).toBeNull(); + expect(PYTHON_PYC_MAGIC_TABLE.size).toBeGreaterThan(80); +}); + +it("rejects extra inspect arguments", () => { + expect( + inspectPythonPycInputSchema.safeParse({ + path: "/artifacts/module.pyc", + detail: "complete", + }).success, + ).toBe(false); +}); diff --git a/src/domain/python/pythonPyc.ts b/src/domain/python/pythonPyc.ts new file mode 100644 index 0000000000..5dc0e97c3d --- /dev/null +++ b/src/domain/python/pythonPyc.ts @@ -0,0 +1,298 @@ +import { z } from "zod"; +import { digestSchema } from "../digests.js"; +import { localPathStringSchema } from "../localPath.js"; + +export const PYTHON_PYC_HEADER_BYTES = 16; +export const PYTHON_PYC_HASH_BASED = 0x1; +export const PYTHON_PYC_CHECK_SOURCE = 0x2; + +const uint = z.number().int().min(0).max(0xffffffff); +const index = z.number().int().nonnegative(); +/** Non-negative CPython `int` counts. `0xffffffff` is -1 and is not evidence. */ +const signedCount = z.number().int().min(0).max(0x7fffffff); + +/** Marshal TYPE_CODE field order for CPython 3.7, 3.8–3.10, and 3.11+. */ +export const PYTHON_PYC_CODE_LAYOUTS = ["py37", "py38", "py311"] as const; +export type PythonPycCodeLayout = (typeof PYTHON_PYC_CODE_LAYOUTS)[number]; + +type MagicEntry = { + readonly python_version: string; + readonly code_layout: PythonPycCodeLayout; +}; + +const magics = ( + values: readonly number[], + python_version: string, + code_layout: PythonPycCodeLayout, +): ReadonlyArray => + values.map((magic) => [magic, { python_version, code_layout }]); + +const consecutive = ( + start: number, + end: number, + python_version: string, + code_layout: PythonPycCodeLayout, +): ReadonlyArray => + magics( + Array.from({ length: end - start + 1 }, (_, offset) => start + offset), + python_version, + code_layout, + ); + +/** + * Pinned CPython pyc magics for 3.7–3.13. Identity is the 16-bit little-endian + * field; `\r\n` is required separately. Layout follows marshal TYPE_CODE, not + * bytecode opcode tables. + */ +export const PYTHON_PYC_MAGIC_TABLE: ReadonlyMap = new Map([ + ...magics([3390, 3391, 3392, 3393, 3394], "3.7", "py37"), + ...magics([3400, 3401], "3.8", "py37"), + ...magics([3410, 3411, 3412, 3413], "3.8", "py38"), + ...magics([3420, 3421, 3422, 3423, 3424, 3425], "3.9", "py38"), + ...magics( + [3430, 3431, 3432, 3433, 3434, 3435, 3436, 3437, 3438, 3439], + "3.10", + "py38", + ), + // 3.11 alphas before 3494 marshal other code layouts: no co_qualname + // before 3460, and PEP 657 end-line and column tables until 3494. + ...consecutive(3494, 3495, "3.11", "py311"), + ...consecutive(3500, 3531, "3.12", "py311"), + ...consecutive(3550, 3571, "3.13", "py311"), +]); + +/** Map a pyc magic integer to its pinned CPython family, or null when unknown. */ +export const pythonPycMagicEntry = (magic: number): MagicEntry | null => + PYTHON_PYC_MAGIC_TABLE.get(magic) ?? null; + +export const inspectPythonPycInputSchema = z.strictObject({ + path: localPathStringSchema.describe( + "Absolute filesystem path to the selected CPython .pyc file", + ), + max_file_bytes: z + .number() + .int() + .min(16) + .max(64 * 1024 * 1024) + .default(16 * 1024 * 1024) + .describe( + "Maximum selected file size in bytes; exceeding the budget fails without partial success", + ), + max_objects: z + .number() + .int() + .min(1) + .max(65536) + .default(16384) + .describe( + "Maximum marshal objects to examine; exceeding the budget fails without partial success", + ), + max_depth: z + .number() + .int() + .min(1) + .max(256) + .default(64) + .describe( + "Maximum marshal nesting depth; exceeding the budget fails without partial success", + ), + max_string_bytes: z + .number() + .int() + .min(1) + .max(8 * 1024 * 1024) + .default(1024 * 1024) + .describe( + "Maximum bytes retained for one marshal string or byte string; exceeding the budget fails without partial success", + ), +}); + +const constPreviewLimit = 80; +/** Integers up to 256 bits fit the 80-character preview in decimal. */ +const intPreviewBits = 256; + +/** CPython float repr for values JSON numbers cannot carry. */ +const pythonFloatSchema = z.union([z.number(), z.enum(["inf", "-inf", "nan"])]); + +const pythonConstSchema = z.discriminatedUnion("kind", [ + z.strictObject({ kind: z.literal("none") }), + z.strictObject({ kind: z.literal("bool"), value: z.boolean() }), + z.strictObject({ + kind: z.literal("int"), + bits: uint, + value: z.string().min(1).max(constPreviewLimit).nullable(), + }), + z.strictObject({ kind: z.literal("float"), value: pythonFloatSchema }), + z.strictObject({ + kind: z.literal("str"), + bytes: uint, + truncated: z.boolean(), + value: z.string().max(constPreviewLimit), + }), + z.strictObject({ + kind: z.literal("bytes"), + bytes: uint, + sha256: digestSchema, + }), + z.strictObject({ + kind: z.literal("code"), + name: z.string(), + firstlineno: uint, + }), + z.strictObject({ kind: z.literal("tuple"), length: uint }), + z.strictObject({ kind: z.literal("list"), length: uint }), + z.strictObject({ kind: z.literal("set"), length: uint }), + z.strictObject({ kind: z.literal("frozenset"), length: uint }), + z.strictObject({ kind: z.literal("dict"), length: uint }), + z.strictObject({ kind: z.literal("ellipsis") }), + z.strictObject({ + kind: z.literal("complex"), + real: pythonFloatSchema, + imag: pythonFloatSchema, + }), + z.strictObject({ kind: z.literal("other"), type: z.string().min(1) }), +]); + +const pythonCodeFields = { + name: z.string(), + qualname: z.string().nullable(), + filename: z.string(), + firstlineno: uint, + argcount: signedCount, + posonlyargcount: signedCount.nullable(), + kwonlyargcount: signedCount, + nlocals: signedCount.nullable(), + stacksize: signedCount, + flags: uint, + names: z.array(z.string()), + varnames: z.array(z.string()).nullable(), + freevars: z.array(z.string()).nullable(), + cellvars: z.array(z.string()).nullable(), + localsplusnames: z.array(z.string()).nullable(), + localspluskinds_hex: z.string().nullable(), + bytecode: z.strictObject({ + bytes: uint, + sha256: digestSchema, + }), + consts: z.array(pythonConstSchema), +}; + +export type PythonPycConst = z.infer; + +/** + * One distinct marshal code object. `nested_code` holds indexes into the + * report's `code_objects`, so a child shared through TYPE_REF is listed once. + * The owner's own index is included when its constants reach that object. + */ +const pythonCodeObjectSchema = z.strictObject({ + ...pythonCodeFields, + nested_code: z.array(index), +}); +export type PythonPycCodeObject = z.infer; + +export const pythonPycHeaderSchema = z.strictObject({ + magic: z.number().int().min(0).max(0xffff), + python_version: z.string().min(1), + code_layout: z.enum(PYTHON_PYC_CODE_LAYOUTS), + flags: uint, + hash_based: z.boolean(), + check_source: z.boolean(), + source_hash_hex: z.string().nullable(), + source_mtime: uint.nullable(), + source_size: uint.nullable(), +}); + +/** Bounded CPython 3.7–3.13 pyc observation; marshal is walked, bytecode is not decompiled. */ +export const pythonPycSchema = z + .strictObject({ + artifact: z.strictObject({ + path: localPathStringSchema, + sha256: digestSchema, + bytes: index, + }), + header: pythonPycHeaderSchema, + /** The top-level module code object is first. */ + code_objects: z.array(pythonCodeObjectSchema).min(1), + coverage: z.strictObject({ + status: z.literal("complete"), + objects: uint, + code_objects: uint, + marshal_bytes: uint, + marshal_depth: uint, + }), + limitations: z.array(z.string().min(1)).min(1), + }) + .superRefine((report, context) => { + if (report.artifact.bytes < PYTHON_PYC_HEADER_BYTES) { + context.addIssue({ + code: "custom", + path: ["artifact", "bytes"], + message: "A pyc header does not fit in the selected artifact.", + }); + } + const marshalBytes = report.artifact.bytes - PYTHON_PYC_HEADER_BYTES; + if (report.coverage.marshal_bytes !== marshalBytes) { + context.addIssue({ + code: "custom", + path: ["coverage", "marshal_bytes"], + message: "Marshal byte count disagrees with the header-stripped file.", + }); + } + if (report.header.hash_based === (report.header.source_hash_hex === null)) { + context.addIssue({ + code: "custom", + path: ["header", "source_hash_hex"], + message: + "Hash-based pycs require an 8-byte source hash; others must omit it.", + }); + } + if (report.header.hash_based === (report.header.source_mtime !== null)) { + context.addIssue({ + code: "custom", + path: ["header", "source_mtime"], + message: "Timestamp pycs require mtime; hash-based pycs must omit it.", + }); + } + if (report.header.hash_based === (report.header.source_size !== null)) { + context.addIssue({ + code: "custom", + path: ["header", "source_size"], + message: + "Timestamp pycs require source_size; hash-based pycs must omit it.", + }); + } + if (report.coverage.code_objects !== report.code_objects.length) { + context.addIssue({ + code: "custom", + path: ["coverage", "code_objects"], + message: "Code-object coverage disagrees with the listed code objects.", + }); + } + report.code_objects.forEach((code, owner) => { + if ( + code.localspluskinds_hex !== null && + code.localspluskinds_hex.length !== + (code.localsplusnames?.length ?? 0) * 2 + ) + context.addIssue({ + code: "custom", + path: ["code_objects", owner, "localspluskinds_hex"], + message: + "co_localspluskinds length does not match co_localsplusnames.", + }); + code.nested_code.forEach((nested, position) => { + if (nested >= report.code_objects.length) + context.addIssue({ + code: "custom", + path: ["code_objects", owner, "nested_code", position], + message: "Nested code index does not name a listed code object.", + }); + }); + }); + }); + +export type PythonPyc = z.infer; +export type InspectPythonPycInput = z.infer; + +export const PYTHON_PYC_CONST_PREVIEW_LIMIT = constPreviewLimit; +export const PYTHON_PYC_INT_PREVIEW_BITS = intPreviewBits; diff --git a/src/python/PythonPyc.errors.ts b/src/python/PythonPyc.errors.ts new file mode 100644 index 0000000000..bf5bf1a3ef --- /dev/null +++ b/src/python/PythonPyc.errors.ts @@ -0,0 +1,22 @@ +/** Distinguish corrupt recognized pycs from magics outside parser coverage. */ +export class PythonPycFormatFailure extends Error { + constructor( + readonly kind: "malformed" | "unsupported", + message: string, + ) { + super(message); + this.name = "PythonPycFormatFailure"; + } +} + +/** Resource boundary exceeded while walking marshal objects. */ +export class PythonPycResourceFailure extends Error { + constructor( + readonly boundary: "objects" | "depth" | "string", + readonly reported_limits: Readonly>, + message: string, + ) { + super(message); + this.name = "PythonPycResourceFailure"; + } +} diff --git a/src/python/PythonPyc.fixture.ts b/src/python/PythonPyc.fixture.ts new file mode 100644 index 0000000000..550ba51b90 --- /dev/null +++ b/src/python/PythonPyc.fixture.ts @@ -0,0 +1,184 @@ +import type { PythonPycCodeLayout } from "../domain/python/pythonPyc.js"; + +const FLAG_REF = 0x80; + +const u8 = (value: number): Buffer => Buffer.from([value]); +const u32 = (value: number): Buffer => { + const bytes = Buffer.alloc(4); + bytes.writeUInt32LE(value); + return bytes; +}; +const concat = (...parts: Buffer[]): Buffer => Buffer.concat(parts); + +/** 16-byte CPython 3.7+ pyc header. Hash-based pycs store an 8-byte source hash. */ +export const pythonPycHeader = ( + magic: number, + fields: { + flags?: number; + mtime?: number; + sourceSize?: number; + sourceHash?: Buffer; + } = {}, +): Buffer => { + const header = Buffer.alloc(16); + header.writeUInt16LE(magic, 0); + header[2] = 0x0d; + header[3] = 0x0a; + const flags = fields.flags ?? 0; + header.writeUInt32LE(flags, 4); + if ((flags & 1) !== 0) { + const hash = fields.sourceHash ?? Buffer.alloc(8); + if (hash.length !== 8) + throw new Error("Hash-based pyc header requires 8 source-hash bytes."); + hash.copy(header, 8); + return header; + } + header.writeUInt32LE(fields.mtime ?? 0, 8); + header.writeUInt32LE(fields.sourceSize ?? 0, 12); + return header; +}; + +const marshalNone = (): Buffer => u8(0x4e); +const marshalBytes = (value: Buffer, flagged = false): Buffer => + concat(u8(0x73 | (flagged ? FLAG_REF : 0)), u32(value.length), value); +const marshalShortAscii = (value: string, interned = false): Buffer => + concat( + u8(interned ? 0x5a : 0x7a), + u8(Buffer.byteLength(value, "utf8")), + Buffer.from(value, "utf8"), + ); +const marshalSmallTuple = (items: Buffer[]): Buffer => + concat(u8(0x29), u8(items.length), ...items); + +export type FixtureCodeFields = { + argcount?: number; + posonlyargcount?: number; + kwonlyargcount?: number; + nlocals?: number; + stacksize?: number; + flags?: number; + bytecode?: Buffer; + consts?: Buffer[]; + names?: string[]; + varnames?: string[]; + freevars?: string[]; + cellvars?: string[]; + localsplusnames?: string[]; + localspluskinds?: Buffer; + /** Raw marshal object in place of the co_consts tuple. */ + constsObject?: Buffer; + filename?: string; + name?: string; + qualname?: string; + /** Raw marshal object in place of co_name. */ + nameObject?: Buffer; + /** Raw marshal object in place of co_qualname. */ + qualnameObject?: Buffer; + firstlineno?: number; + linetable?: Buffer; + exceptiontable?: Buffer; + /** Raw marshal object in place of the co_names tuple. */ + namesObject?: Buffer; + /** Raw marshal object in place of the co_exceptiontable bytes. */ + exceptiontableObject?: Buffer; + flagged?: boolean; +}; + +/** Marshal TYPE_CODE matching CPython 3.7, 3.8–3.10, or 3.11+ field order. */ +export const marshalCode = ( + layout: PythonPycCodeLayout, + fields: FixtureCodeFields = {}, +): Buffer => { + const parts: Buffer[] = [ + u8(0x63 | (fields.flagged === true ? FLAG_REF : 0)), + u32(fields.argcount ?? 0), + ]; + if (layout !== "py37") parts.push(u32(fields.posonlyargcount ?? 0)); + parts.push(u32(fields.kwonlyargcount ?? 0)); + if (layout !== "py311") parts.push(u32(fields.nlocals ?? 0)); + parts.push( + u32(fields.stacksize ?? 1), + u32(fields.flags ?? 0), + marshalBytes(fields.bytecode ?? Buffer.from([0x53, 0x00])), + fields.constsObject ?? marshalSmallTuple(fields.consts ?? [marshalNone()]), + fields.namesObject ?? + marshalSmallTuple( + (fields.names ?? []).map((name) => marshalShortAscii(name)), + ), + ); + if (layout === "py311") { + parts.push( + marshalSmallTuple( + (fields.localsplusnames ?? []).map((name) => marshalShortAscii(name)), + ), + marshalBytes(fields.localspluskinds ?? Buffer.alloc(0)), + ); + } else { + parts.push( + marshalSmallTuple( + (fields.varnames ?? []).map((name) => marshalShortAscii(name)), + ), + marshalSmallTuple( + (fields.freevars ?? []).map((name) => marshalShortAscii(name)), + ), + marshalSmallTuple( + (fields.cellvars ?? []).map((name) => marshalShortAscii(name)), + ), + ); + } + parts.push( + marshalShortAscii(fields.filename ?? "sample.py"), + fields.nameObject ?? marshalShortAscii(fields.name ?? ""), + ); + if (layout === "py311") + parts.push( + fields.qualnameObject ?? + marshalShortAscii(fields.qualname ?? fields.name ?? ""), + ); + parts.push( + u32(fields.firstlineno ?? 1), + marshalBytes(fields.linetable ?? Buffer.alloc(0)), + ); + if (layout === "py311") + parts.push( + fields.exceptiontableObject ?? + marshalBytes(fields.exceptiontable ?? Buffer.alloc(0)), + ); + return concat(...parts); +}; + +/** Complete pyc with a module code object and optional nested function. */ +export const pythonPycFixture = ( + layout: PythonPycCodeLayout = "py311", + options: { + magic?: number; + header?: Parameters[1]; + nested?: boolean; + flaggedCode?: boolean; + } = {}, +): Buffer => { + const magic = + options.magic ?? + (layout === "py37" ? 3394 : layout === "py38" ? 3439 : 3531); + const nested = + options.nested === true + ? marshalCode(layout, { + name: "inner", + qualname: "inner", + filename: "sample.py", + firstlineno: 2, + bytecode: Buffer.from([0x53, 0x00]), + }) + : undefined; + const module = marshalCode(layout, { + name: "", + qualname: "", + filename: "sample.py", + firstlineno: 1, + names: nested === undefined ? [] : ["inner"], + consts: nested === undefined ? [marshalNone()] : [nested, marshalNone()], + bytecode: Buffer.from([0x64, 0x00, 0x53, 0x00]), + ...(options.flaggedCode === true ? { flagged: true } : {}), + }); + return concat(pythonPycHeader(magic, options.header), module); +}; diff --git a/src/python/PythonPycParser.test.ts b/src/python/PythonPycParser.test.ts new file mode 100644 index 0000000000..a2fce3a085 --- /dev/null +++ b/src/python/PythonPycParser.test.ts @@ -0,0 +1,1342 @@ +import { createHash } from "node:crypto"; +import { expect, it } from "vitest"; +import { + inspectPythonPycInputSchema, + pythonPycSchema, + type PythonPyc, +} from "../domain/python/pythonPyc.js"; +import { + marshalCode, + pythonPycFixture, + pythonPycHeader, +} from "./PythonPyc.fixture.js"; +import { parsePythonPyc } from "./PythonPycParser.js"; +import { + PythonPycFormatFailure, + PythonPycResourceFailure, +} from "./PythonPyc.errors.js"; + +const defaults = inspectPythonPycInputSchema.parse({ + path: "/artifacts/sample.pyc", +}); + +const root = (report: PythonPyc) => { + const code = report.code_objects[0]; + if (code === undefined) throw new Error("Report lists no code object"); + return code; +}; + +const parse = (bytes: Buffer) => + parsePythonPyc( + bytes, + { + path: defaults.path, + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.length, + }, + defaults, + ); + +it("reads 3.7, 3.10 and 3.12 layouts including nested code", () => { + const py37 = parse(pythonPycFixture("py37")); + expect(py37.header).toMatchObject({ + magic: 3394, + python_version: "3.7", + code_layout: "py37", + hash_based: false, + }); + expect(root(py37).posonlyargcount).toBeNull(); + expect(root(py37).nlocals).toBe(0); + expect(root(py37).qualname).toBeNull(); + + const py38 = parse(pythonPycFixture("py38", { nested: true })); + expect(py38.header.python_version).toBe("3.10"); + expect(root(py38).posonlyargcount).toBe(0); + expect(py38.code_objects[1]?.name).toBe("inner"); + expect(py38.coverage.code_objects).toBe(2); + + const py311 = parse(pythonPycFixture("py311", { nested: true })); + expect(py311.header).toMatchObject({ + magic: 3531, + python_version: "3.12", + code_layout: "py311", + }); + expect(root(py311).nlocals).toBeNull(); + expect(root(py311).qualname).toBe(""); + expect(py311.code_objects[1]?.name).toBe("inner"); + expect(root(py311).consts).toEqual( + expect.arrayContaining([ + { kind: "code", name: "inner", firstlineno: 2 }, + { kind: "none" }, + ]), + ); +}); + +it("decodes a hash-based header and a FLAG_REF top-level code object", () => { + const bytes = pythonPycFixture("py311", { + flaggedCode: true, + header: { + flags: 0x1, + sourceHash: Buffer.from("0102030405060708", "hex"), + }, + }); + const report = parse(bytes); + expect(report.header).toMatchObject({ + hash_based: true, + check_source: false, + source_hash_hex: "0102030405060708", + source_mtime: null, + source_size: null, + }); + expect(root(report).name).toBe(""); +}); + +it("fails unknown magics and truncated marshal without partial success", () => { + const unknown = Buffer.concat([pythonPycHeader(1), marshalCode("py311")]); + expect(() => parse(unknown)).toThrow(PythonPycFormatFailure); + expect(() => parse(pythonPycFixture("py311").subarray(0, 20))).toThrow( + PythonPycFormatFailure, + ); +}); + +const u32 = (value: number): Buffer => { + const bytes = Buffer.alloc(4); + bytes.writeUInt32LE(value); + return bytes; +}; +const i32 = (value: number): Buffer => { + const bytes = Buffer.alloc(4); + bytes.writeInt32LE(value); + return bytes; +}; +const f64 = (value: number): Buffer => { + const bytes = Buffer.alloc(8); + bytes.writeDoubleLE(value); + return bytes; +}; +const marshalRef = (index: number): Buffer => + Buffer.concat([Buffer.from([0x72]), u32(index)]); +const marshalLong = (digits: readonly number[], negative = false): Buffer => + Buffer.concat([ + Buffer.from([0x6c]), + i32(negative ? -digits.length : digits.length), + ...digits.map((digit) => { + const bytes = Buffer.alloc(2); + bytes.writeUInt16LE(digit); + return bytes; + }), + ]); +const marshalUnicode = (bytes: Buffer): Buffer => + Buffer.concat([Buffer.from([0x75]), u32(bytes.length), bytes]); +const withConsts = (consts: Buffer[]): Buffer => + Buffer.concat([pythonPycHeader(3531), marshalCode("py311", { consts })]); +const CO_VARARGS = 0x04; + +/** Each level keeps one flagged child and two TYPE_REF copies of it. */ +const sharedChain = (levels: number): Buffer => { + const level = (index: number): Buffer => + index === levels + ? marshalCode("py311", { name: `leaf`, flagged: true }) + : marshalCode("py311", { + name: `level${index}`, + flagged: index > 0, + consts: [level(index + 1), marshalRef(index), marshalRef(index)], + }); + return Buffer.concat([pythonPycHeader(3531), level(0)]); +}; + +it("lists a code object shared through TYPE_REF once and refers to it by index", () => { + const small = parse(sharedChain(3)); + expect(small.code_objects.map((code) => code.name)).toEqual([ + "level0", + "level1", + "level2", + "leaf", + ]); + expect(small.code_objects.map((code) => code.nested_code)).toEqual([ + [1], + [2], + [3], + [], + ]); + expect(small.coverage.code_objects).toBe(4); + expect(pythonPycSchema.safeParse(small).success).toBe(true); + const started = Date.now(); + const deep = parse(sharedChain(25)); + expect(Date.now() - started).toBeLessThan(1000); + expect(deep.coverage.code_objects).toBe(26); + expect(JSON.stringify(deep).length).toBeLessThan(64 * 1024); +}); + +it("bounds names repeated through TYPE_REF by what the report carries", () => { + const name = Buffer.alloc(1000, 0x61); + const namesObject = Buffer.concat([ + Buffer.from([0x29, 101]), + Buffer.from([0x61 | 0x80]), + u32(name.length), + name, + ...Array.from({ length: 100 }, () => marshalRef(0)), + ]); + const bytes = Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { namesObject }), + ]); + expect(() => + parsePythonPyc( + bytes, + { + path: defaults.path, + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.length, + }, + { ...defaults, max_file_bytes: bytes.length }, + ), + ).toThrow(PythonPycResourceFailure); +}); + +it("charges shared empty name tables against the object budget", () => { + const width = 200; + const owners = 40; + const namesObject = Buffer.concat([ + Buffer.from([0xa8]), + u32(width), + Buffer.from([0xfa, 0x00]), + ...Array.from({ length: width - 1 }, () => marshalRef(2)), + ]); + const owner = (index: number) => + marshalCode("py311", { + name: `o${index}`, + qualname: `o${index}`, + filename: "n.py", + flagged: true, + namesObject: index === 0 ? namesObject : marshalRef(1), + }); + const bytes = Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { + consts: Array.from({ length: owners }, (_, index) => owner(index)), + }), + ]); + const report = parseWith(bytes, { + max_objects: 65_536, + max_file_bytes: bytes.length, + }); + const copiedNames = report.code_objects.reduce( + (sum, code) => sum + code.names.length, + 0, + ); + const shared = report.code_objects[1]?.names; + expect(copiedNames).toBe(owners * width); + expect(shared).toEqual(Array.from({ length: width }, () => "")); + expect(report.code_objects.at(-1)?.names).toEqual(shared); + expect(report.coverage.objects).toBeLessThan(copiedNames); + expect(report.coverage.status).toBe("complete"); + expect(() => + parseWith(bytes, { + max_objects: copiedNames - 1, + max_file_bytes: bytes.length, + }), + ).toThrow(/Reported names exceed max_objects/); +}); + +it("charges long digits to the string budget and keeps large previews bounded", () => { + expect(() => + parse(withConsts([Buffer.concat([Buffer.from([0x6c]), i32(600_000)])])), + ).toThrow(PythonPycResourceFailure); + const report = parse( + withConsts([ + marshalLong([1, 2]), + marshalLong(Array.from({ length: 20 }, (_, index) => index + 1)), + ]), + ); + expect(root(report).consts[0]).toEqual({ + kind: "int", + bits: 17, + value: String(1n + (2n << 15n)), + }); + expect(root(report).consts[1]).toMatchObject({ kind: "int", value: null }); + expect(pythonPycSchema.safeParse(report).success).toBe(true); + expect(() => parse(withConsts([marshalLong([1, 0])]))).toThrow( + PythonPycFormatFailure, + ); + expect(() => parse(withConsts([marshalLong([0x8000])]))).toThrow( + PythonPycFormatFailure, + ); +}); + +it("decodes marshal strings with CPython surrogatepass rules", () => { + const report = parse( + withConsts([marshalUnicode(Buffer.from([0x73, 0xed, 0xa0, 0x80]))]), + ); + expect(root(report).consts[0]).toMatchObject({ + kind: "str", + value: "s\ud800", + }); + expect(() => + parse(withConsts([marshalUnicode(Buffer.from([0xf0, 0x90, 0x80]))])), + ).toThrow(PythonPycFormatFailure); + expect(() => + parse(withConsts([marshalUnicode(Buffer.from([0xc0, 0x80]))])), + ).toThrow(PythonPycFormatFailure); +}); + +it("keeps non-finite float and complex constants JSON-safe", () => { + const report = parse( + withConsts([ + Buffer.concat([Buffer.from([0x67]), f64(Infinity)]), + Buffer.concat([Buffer.from([0x79]), f64(0), f64(-Infinity)]), + Buffer.concat([Buffer.from([0x67]), f64(Number.NaN)]), + ]), + ); + expect(root(report).consts).toEqual([ + { kind: "float", value: "inf" }, + { kind: "complex", real: 0, imag: "-inf" }, + { kind: "float", value: "nan" }, + ]); + expect(pythonPycSchema.safeParse(report).success).toBe(true); + expect(JSON.parse(JSON.stringify(report))).toEqual(report); +}); + +it("accepts an empty co_filename", () => { + const report = parse( + Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { filename: "" }), + ]), + ); + expect(root(report).filename).toBe(""); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +it("rejects a code object whose location tables are not bytes", () => { + expect(() => + parse( + Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { exceptiontableObject: Buffer.from([0x53]) }), + ]), + ), + ).toThrow(PythonPycFormatFailure); +}); + +it("hashes one shared byte string no matter how many times it is referenced", () => { + const payload = Buffer.alloc(1024 * 1024, 0x61); + const refs = 1200; + const bytes = Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { + constsObject: Buffer.concat([ + Buffer.from([0x28]), + u32(refs + 1), + Buffer.from([0xf3]), + u32(payload.length), + payload, + ...Array.from({ length: refs }, () => marshalRef(0)), + ]), + }), + ]); + const started = Date.now(); + const report = parse(bytes); + expect(Date.now() - started).toBeLessThan(1000); + expect(root(report).consts).toHaveLength(refs + 1); + expect(root(report).consts[refs]).toEqual(root(report).consts[0]); + expect(root(report).consts[0]).toMatchObject({ + kind: "bytes", + bytes: payload.length, + }); +}); + +it("rejects constant tables whose copies exceed max_objects", () => { + const shared = Buffer.concat([ + Buffer.from([0xa9, 20]), + Buffer.alloc(20, 0x4e), + ]); + const child = (constsObject: Buffer, name: string) => + marshalCode("py311", { constsObject, name }); + const bytes = Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { + consts: [ + child(shared, "a"), + ...Array.from({ length: 5 }, (_, index) => + child(marshalRef(0), `b${index}`), + ), + ], + }), + ]); + const artifact = { + path: defaults.path, + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.length, + }; + const allowed = parsePythonPyc(bytes, artifact, defaults); + const entries = allowed.code_objects.reduce( + (sum, code) => sum + code.consts.length, + 0, + ); + expect(entries).toBeGreaterThan(allowed.coverage.objects); + expect(() => + parsePythonPyc(bytes, artifact, { + ...defaults, + max_objects: entries - 1, + }), + ).toThrow(/Reported constants exceed max_objects/); +}); + +it("rejects NULL tuple items and mismatched local kinds", () => { + expect(() => parse(withConsts([Buffer.from([0x30])]))).toThrow( + PythonPycFormatFailure, + ); + expect( + root(parse(withConsts([Buffer.from([0x7b, 0x4e, 0x54, 0x30])]))).consts[0], + ).toEqual({ + kind: "dict", + length: 1, + }); + expect(() => + parse( + Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { + localsplusnames: ["v"], + localspluskinds: Buffer.alloc(0), + }), + ]), + ), + ).toThrow(PythonPycFormatFailure); + const matched = parse( + Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { + localsplusnames: ["v"], + localspluskinds: Buffer.from([0x20]), + }), + ]), + ); + expect(root(matched).localsplusnames).toEqual(["v"]); + expect(root(matched).localspluskinds_hex).toBe("20"); + const broken = { + ...matched, + code_objects: [ + { ...root(matched), localspluskinds_hex: "" }, + ...matched.code_objects.slice(1), + ], + }; + expect(pythonPycSchema.safeParse(broken).success).toBe(false); + expect(pythonPycSchema.safeParse(matched).success).toBe(true); +}); + +const textFloat = (text: string): Buffer => + Buffer.concat([ + Buffer.from([0x66, Buffer.byteLength(text)]), + Buffer.from(text), + ]); + +const textComplex = (real: string, imag: string): Buffer => + Buffer.concat([ + Buffer.from([0x78, Buffer.byteLength(real)]), + Buffer.from(real), + Buffer.from([Buffer.byteLength(imag)]), + Buffer.from(imag), + ]); + +const parseLimited = (bytes: Buffer, max_file_bytes: number) => + parsePythonPyc( + bytes, + { + path: defaults.path, + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.length, + }, + { ...defaults, max_file_bytes }, + ); + +it("charges every reported code-constant name against the file budget", () => { + const name = Buffer.alloc(2000, 0x6e); + const nameObject = Buffer.concat([ + Buffer.from([0x61]), + u32(name.length), + name, + ]); + const child = marshalCode("py311", { + nameObject, + qualnameObject: nameObject, + flagged: true, + }); + const file = (copies: number): Buffer => + Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { + constsObject: Buffer.concat([ + Buffer.from([0x28]), + u32(copies), + child, + ...Array.from({ length: copies - 1 }, () => marshalRef(0)), + ]), + }), + ]); + const once = parseLimited(file(1), 20_000); + expect(root(once).consts[0]).toMatchObject({ + kind: "code", + name: "n".repeat(2000), + }); + expect(pythonPycSchema.safeParse(once).success).toBe(true); + expect(() => parseLimited(file(30), 20_000)).toThrow( + PythonPycResourceFailure, + ); +}); + +it("reuses a shared integer's bit length instead of converting every reference", () => { + const digits = Array.from({ length: 20_000 }, () => 0x7fff); + const long = marshalLong(digits); + const flagged = Buffer.concat([ + Buffer.from([(long[0] ?? 0) | 0x80]), + long.subarray(1), + ]); + const refs = 400; + const bytes = Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { + constsObject: Buffer.concat([ + Buffer.from([0x28]), + u32(refs + 1), + flagged, + ...Array.from({ length: refs }, () => marshalRef(0)), + ]), + }), + ]); + const started = Date.now(); + const report = parse(bytes); + expect(Date.now() - started).toBeLessThan(1000); + expect(root(report).consts[0]).toEqual({ + kind: "int", + bits: digits.length * 15, + value: null, + }); + expect(root(report).consts[refs]).toEqual(root(report).consts[0]); +}); + +it("lists code objects nested inside constant containers", () => { + const hidden = marshalCode("py311", { + name: "hidden", + qualname: "hidden", + firstlineno: 4, + }); + const wrapped = Buffer.concat([Buffer.from([0x29, 1]), hidden]); + const report = parse(withConsts([wrapped])); + expect(report.code_objects.map((code) => code.name)).toEqual([ + "", + "hidden", + ]); + expect(root(report).nested_code).toEqual([1]); + expect(root(report).consts[0]).toEqual({ kind: "tuple", length: 1 }); + expect(report.coverage).toMatchObject({ + status: "complete", + code_objects: 2, + }); + expect(pythonPycSchema.safeParse(report).success).toBe(true); + + const listed = Buffer.concat([Buffer.from([0x5b]), u32(1), hidden]); + const deeper = parse( + withConsts([Buffer.concat([Buffer.from([0x29, 1]), listed])]), + ); + expect(deeper.code_objects.map((code) => code.name)).toEqual([ + "", + "hidden", + ]); + const keyed = Buffer.concat([ + Buffer.from([0x7b, 0x69]), + i32(1), + hidden, + Buffer.from([0x30]), + ]); + expect( + parse(withConsts([keyed])).code_objects.map((code) => code.name), + ).toEqual(["", "hidden"]); +}); + +it("accepts containers that contain themselves", () => { + const list = Buffer.concat([ + Buffer.from([0x5b | 0x80]), + u32(1), + marshalRef(0), + ]); + expect(root(parse(withConsts([list]))).consts[0]).toEqual({ + kind: "list", + length: 1, + }); + const dict = Buffer.concat([ + Buffer.from([0x7b | 0x80, 0x69 | 0x80]), + i32(1), + marshalRef(0), + Buffer.from([0x30]), + ]); + const report = parse(withConsts([dict])); + expect(root(report).consts[0]).toEqual({ kind: "dict", length: 1 }); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +it("reads marshal version 0 text floats inside a code object", () => { + const version0 = Buffer.from( + "cb0d0d0a0000000000000000000000006300000000000000000000000001000000000000007308000000970064005a00790128020000006603312e354e280100000075010000007828000000007300000000750900000073616d706c652e707975080000003c6d6f64756c653e75080000003c6d6f64756c653e01000000730a000000f003010101d8040781017300000000", + "hex", + ); + const report = parse(version0); + expect(report.header).toMatchObject({ + magic: 3531, + python_version: "3.12", + code_layout: "py311", + }); + expect(root(report).name).toBe(""); + expect(root(report).consts[0]).toEqual({ kind: "float", value: 1.5 }); + expect(pythonPycSchema.safeParse(report).success).toBe(true); + const specials = parse( + withConsts([ + textFloat("inf"), + textFloat("-inf"), + textFloat("nan"), + textComplex("0", "inf"), + textFloat("1e-2"), + ]), + ); + expect(root(specials).consts).toEqual([ + { kind: "float", value: "inf" }, + { kind: "float", value: "-inf" }, + { kind: "float", value: "nan" }, + { kind: "complex", real: 0, imag: "inf" }, + { kind: "float", value: 0.01 }, + ]); + for (const text of ["1.5x", "", "0x1.8p1", " 1.5"]) { + expect(() => parse(withConsts([textFloat(text)]))).toThrow( + PythonPycFormatFailure, + ); + } +}); + +it("accepts empty code names", () => { + const child = marshalCode("py311", { name: "", qualname: "" }); + const report = parse( + Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { + name: "", + qualname: "", + consts: [child], + }), + ]), + ); + expect(root(report).name).toBe(""); + expect(root(report).qualname).toBe(""); + expect(root(report).consts[0]).toEqual({ + kind: "code", + name: "", + firstlineno: 1, + }); + expect(report.code_objects[1]?.name).toBe(""); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +it("reports the marshal byte length of a string constant", () => { + const report = parse( + withConsts([ + marshalUnicode(Buffer.from([0xed, 0xa0, 0x80, 0xed, 0xb0, 0x80])), + ]), + ); + expect(root(report).consts[0]).toMatchObject({ kind: "str", bytes: 6 }); +}); + +it("applies Python container rules before reporting lengths", () => { + const setOf = (...members: Buffer[]): Buffer => + Buffer.concat([Buffer.from([0x3c]), u32(members.length), ...members]); + const int = (value: number): Buffer => + Buffer.concat([Buffer.from([0x69]), i32(value)]); + expect(root(parse(withConsts([setOf(int(1), int(1))]))).consts[0]).toEqual({ + kind: "set", + length: 1, + }); + expect( + root(parse(withConsts([setOf(Buffer.from([0x54]), int(1))]))).consts[0], + ).toEqual({ kind: "set", length: 1 }); + expect( + root( + parse( + withConsts([ + setOf(int(1), Buffer.concat([Buffer.from([0x67]), f64(1)])), + ]), + ), + ).consts[0], + ).toEqual({ kind: "set", length: 1 }); + expect( + root( + parse( + withConsts([ + setOf( + Buffer.concat([Buffer.from([0x67]), f64(Number.NaN)]), + Buffer.concat([Buffer.from([0x67]), f64(Number.NaN)]), + ), + ]), + ), + ).consts[0], + ).toEqual({ kind: "set", length: 2 }); + const sameNan = Buffer.concat([Buffer.from([0x67 | 0x80]), f64(Number.NaN)]); + expect( + root(parse(withConsts([setOf(sameNan, marshalRef(0))]))).consts[0], + ).toEqual({ kind: "set", length: 1 }); + expect( + root( + parse( + withConsts([ + Buffer.concat([Buffer.from([0x3e]), u32(2), int(1), int(1)]), + ]), + ), + ).consts[0], + ).toEqual({ kind: "frozenset", length: 1 }); + expect( + root(parse(withConsts([Buffer.from([0x7b, 0x4e, 0x30])]))).consts[0], + ).toEqual({ kind: "dict", length: 0 }); + const overwritten = Buffer.concat([ + Buffer.from([0x7b]), + int(1), + int(10), + int(1), + int(11), + Buffer.from([0x30]), + ]); + expect(root(parse(withConsts([overwritten]))).consts[0]).toEqual({ + kind: "dict", + length: 1, + }); + const complexOne = Buffer.concat([Buffer.from([0x79]), f64(1), f64(0)]); + expect( + root(parse(withConsts([setOf(int(1), complexOne)]))).consts[0], + ).toEqual({ kind: "set", length: 1 }); + expect(() => + parse( + withConsts([ + Buffer.concat([ + Buffer.from([0x7b, 0x5b]), + u32(0), + Buffer.from([0x4e, 0x30]), + ]), + ]), + ), + ).toThrow(PythonPycFormatFailure); + expect(() => + parse(withConsts([setOf(Buffer.concat([Buffer.from([0x5b]), u32(0)]))])), + ).toThrow(PythonPycFormatFailure); +}); + +it("rejects a shared tuple key before its expansion doubles without bound", () => { + const depth = 30; + let body = Buffer.concat([Buffer.from([0xe9]), i32(1)]); + for (let index = depth - 1; index >= 0; index -= 1) + body = Buffer.concat([ + Buffer.from([0x29 | 0x80, 2]), + body, + marshalRef(index + 1), + ]); + const bytes = withConsts([ + Buffer.concat([Buffer.from([0x3c]), u32(1), body]), + ]); + const started = Date.now(); + expect(() => parse(bytes)).toThrow(PythonPycResourceFailure); + expect(Date.now() - started).toBeLessThan(1000); + let shallow = Buffer.concat([Buffer.from([0xe9]), i32(1)]); + for (let index = 7; index >= 0; index -= 1) + shallow = Buffer.concat([ + Buffer.from([0x29 | 0x80, 2]), + shallow, + marshalRef(index + 1), + ]); + expect( + root( + parse( + withConsts([Buffer.concat([Buffer.from([0x3c]), u32(1), shallow])]), + ), + ).consts[0], + ).toEqual({ kind: "set", length: 1 }); +}); + +it("lists a shared container's code objects once per owner and charges each copy", () => { + const child = marshalCode("py311", { name: "child", flagged: true }); + const copies = 40; + const container = Buffer.concat([ + Buffer.from([0x28 | 0x80]), + u32(copies), + child, + ...Array.from({ length: copies - 1 }, () => marshalRef(1)), + ]); + const repeated = Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { + consts: [container, ...Array.from({ length: 7 }, () => marshalRef(0))], + }), + ]); + const started = Date.now(); + const shared = parse(repeated); + expect(Date.now() - started).toBeLessThan(1000); + expect(shared.code_objects.map((code) => code.name)).toEqual([ + "", + "child", + ]); + expect(root(shared).nested_code).toEqual([1]); + expect(JSON.stringify(shared).length).toBeLessThan(64 * 1024); + + const owners = 8; + const ownedContainer = Buffer.concat([ + Buffer.from([0x28 | 0x80]), + u32(copies), + child, + ...Array.from({ length: copies - 1 }, () => marshalRef(2)), + ]); + const first = marshalCode("py311", { + name: "owner0", + flagged: true, + constsObject: ownedContainer, + }); + const bytes = Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { + consts: [ + first, + ...Array.from({ length: owners - 1 }, (_, index) => + marshalCode("py311", { + name: `owner${index + 1}`, + constsObject: marshalRef(1), + }), + ), + ], + }), + ]); + const report = parse(bytes); + expect(report.code_objects.map((code) => code.name)).toEqual([ + "", + "owner0", + "child", + ...Array.from({ length: owners - 1 }, (_, index) => `owner${index + 1}`), + ]); + expect(report.code_objects[1]?.nested_code).toEqual([2]); + expect(report.code_objects[3]?.nested_code).toEqual([2]); + const relationships = report.code_objects.reduce( + (sum, code) => sum + code.nested_code.length, + 0, + ); + const constants = report.code_objects.reduce( + (sum, code) => sum + code.consts.length, + 0, + ); + expect(relationships).toBe(owners + owners); + expect(constants).toBeGreaterThan(report.coverage.objects); + const artifact = { + path: defaults.path, + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.length, + }; + expect(() => + parsePythonPyc(bytes, artifact, { ...defaults, max_objects: constants }), + ).toThrow(PythonPycResourceFailure); + expect( + parsePythonPyc(bytes, artifact, { + ...defaults, + max_objects: constants + relationships, + }).coverage.status, + ).toBe("complete"); +}); + +it("accepts code objects as set members and dictionary keys", () => { + const setOf = (...members: Buffer[]): Buffer => + Buffer.concat([Buffer.from([0x3c]), u32(members.length), ...members]); + const same = (filename: string): Buffer => + marshalCode("py311", { name: "f", filename, firstlineno: 3 }); + expect( + root(parse(withConsts([setOf(same("a.py"), same("b.py"))]))).consts[0], + ).toEqual({ kind: "set", length: 1 }); + expect( + root( + parse( + withConsts([ + setOf( + same("a.py"), + marshalCode("py311", { name: "g", firstlineno: 3 }), + ), + ]), + ), + ).consts[0], + ).toEqual({ kind: "set", length: 2 }); + const produced = Buffer.from( + "cb0d0d0a000000000000000000000000e30000000000000000000000000000000000000000f3040000009700790029033e02000000e30000000000000000000000000100000000000000f308000000970064005a0079012902e9010000004e2901da0178a900f300000000fa04692e7079fa083c6d6f64756c653e720900000001000000730a000000f003010101d8020381017207000000e30000000000000000000000000100000000000000f308000000970064005a007901290272040000004e2901da01797206000000720700000072080000007209000000720900000001000000730a000000f003010101d80203810172070000007b7202000000e902000000304e720600000072060000007207000000fa046f2e707972090000007209000000010000007308000000f003010101d800047207000000", + "hex", + ); + const report = parse(produced); + expect(root(report).consts[0]).toEqual({ kind: "frozenset", length: 2 }); + expect(root(report).consts[1]).toEqual({ kind: "dict", length: 1 }); + expect(report.code_objects.map((code) => code.name)).toEqual([ + "", + "", + "", + ]); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +it("keeps the first dictionary key when a later key compares equal", () => { + const key = (filename: string): Buffer => + marshalCode("py311", { + name: "f", + qualname: "f", + filename, + firstlineno: 3, + }); + const value = (name: string): Buffer => + marshalCode("py311", { + name, + qualname: name, + filename: `${name}.py`, + }); + const dict = Buffer.concat([ + Buffer.from([0x7b]), + key("first.py"), + value("old"), + key("last.py"), + value("new"), + Buffer.from([0x30]), + ]); + const report = parse(withConsts([dict])); + expect(root(report).consts[0]).toEqual({ kind: "dict", length: 1 }); + expect(report.code_objects.map((code) => [code.name, code.filename])).toEqual( + [ + ["", "sample.py"], + ["f", "first.py"], + ["new", "new.py"], + ], + ); + expect(root(report).nested_code).toEqual([1, 2]); + expect(report.coverage.status).toBe("complete"); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +it("walks a wide dictionary once per child instead of copying it each step", () => { + const keys = 8_000; + const nested = marshalCode("py311", { + name: "inside", + qualname: "inside", + filename: "inside.py", + }); + const pieces: Buffer[] = [Buffer.from([0x7b])]; + for (let index = 0; index < keys; index += 1) + pieces.push( + Buffer.from([0x69]), + i32(index), + index === 0 ? nested : Buffer.from([0x4e]), + ); + pieces.push(Buffer.from([0x30])); + const bytes = withConsts([Buffer.concat(pieces)]); + const started = Date.now(); + const report = parse(bytes); + // Rebuilding the child list on every key took about 700ms for 8,000 entries. + expect(Date.now() - started).toBeLessThan(250); + expect(root(report).consts[0]).toEqual({ kind: "dict", length: keys }); + expect(root(report).nested_code).toEqual([1]); + expect(report.code_objects[1]?.name).toBe("inside"); + expect(report.coverage.objects).toBeLessThanOrEqual(defaults.max_objects); + expect(report.coverage.status).toBe("complete"); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +const parseWith = (bytes: Buffer, limits: Partial = {}) => + parsePythonPyc( + bytes, + { + path: defaults.path, + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.length, + }, + { ...defaults, ...limits }, + ); + +it("keeps one descendant list when many wrappers share a container", () => { + const children = 1800; + const wrappers = 14_000; + const child = marshalCode("py311", { + name: "c", + qualname: "c", + filename: "a.py", + }); + const pieces: Buffer[] = [ + Buffer.from([0x28]), + u32(1 + wrappers), + Buffer.from([0x28 | 0x80]), + u32(children), + ]; + for (let index = 0; index < children; index += 1) pieces.push(child); + const shared = marshalRef(0); + for (let index = 0; index < wrappers; index += 1) + pieces.push(Buffer.from([0x29 | 0x80, 1]), shared); + const bytes = Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { constsObject: Buffer.concat(pieces) }), + ]); + const before = process.memoryUsage().heapUsed; + const started = Date.now(); + const report = parseWith(bytes, { max_objects: 65_536 }); + const retained = process.memoryUsage().heapUsed - before; + // 1,800 children and 14,000 wrappers: 49,612 reads, previously ~181 MiB. + expect(Date.now() - started).toBeLessThan(5000); + expect(retained).toBeLessThan(64 * 1024 * 1024); + expect(report.coverage.objects).toBe(49_612); + expect(report.code_objects).toHaveLength(children + 1); + expect(root(report).nested_code).toHaveLength(children); + expect(report.coverage.status).toBe("complete"); +}); + +it("keeps code objects whose consts differ only by tuple shape", () => { + const int = (value: number): Buffer => + Buffer.concat([Buffer.from([0x69]), i32(value)]); + const tuple = (items: Buffer[]): Buffer => + Buffer.concat([Buffer.from([0x29, items.length]), ...items]); + const code = (consts: Buffer): Buffer => + marshalCode("py311", { + name: "f", + qualname: "f", + filename: "f.py", + firstlineno: 1, + constsObject: consts, + }); + const frozen = Buffer.concat([ + Buffer.from([0x3e]), + u32(2), + code(tuple([tuple([int(1), int(2)])])), + code(tuple([tuple([int(1)]), int(2)])), + ]); + const report = parse(withConsts([frozen])); + expect(root(report).consts[0]).toEqual({ kind: "frozenset", length: 2 }); + expect(report.code_objects.map((code) => code.name)).toEqual([ + "", + "f", + "f", + ]); + expect(report.coverage).toMatchObject({ + status: "complete", + code_objects: 3, + }); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +it("reports code inside a container that the code also references", () => { + const list = Buffer.concat([ + Buffer.from([0x5b | 0x80]), + u32(2), + marshalCode("py311", { + name: "a", + qualname: "a", + constsObject: Buffer.concat([Buffer.from([0x29, 1]), marshalRef(0)]), + }), + marshalCode("py311", { name: "b", qualname: "b" }), + ]); + const report = parse(withConsts([list])); + expect(root(report).consts[0]).toEqual({ kind: "list", length: 2 }); + expect(report.code_objects.map((code) => code.name)).toEqual([ + "", + "a", + "b", + ]); + expect(root(report).nested_code).toEqual([1, 2]); + expect(report.code_objects[1]?.consts[0]).toEqual({ + kind: "list", + length: 2, + }); + expect(report.code_objects[1]?.nested_code).toEqual([1, 2]); + expect(report.code_objects[2]?.nested_code).toEqual([]); + expect(report.coverage.status).toBe("complete"); + expect(pythonPycSchema.safeParse(report).success).toBe(true); + + const cyclic = Buffer.concat([ + Buffer.from([0x5b | 0x80]), + u32(2), + marshalRef(0), + marshalCode("py311", { name: "inside", qualname: "inside" }), + ]); + const cycle = parse(withConsts([cyclic])); + expect(root(cycle).consts[0]).toEqual({ kind: "list", length: 2 }); + expect(cycle.code_objects.map((code) => code.name)).toEqual([ + "", + "inside", + ]); + expect(root(cycle).nested_code).toEqual([1]); +}); + +it("keeps distinct strings that JavaScript merges through surrogates", () => { + const unicode = (bytes: Buffer): Buffer => + Buffer.concat([Buffer.from([0x75]), u32(bytes.length), bytes]); + const first = Buffer.from([ + 0xf0, 0x90, 0x80, 0x80, 0xed, 0xa0, 0x80, 0xed, 0xb0, 0x80, + ]); + const second = Buffer.from([ + 0xed, 0xa0, 0x80, 0xed, 0xb0, 0x80, 0xf0, 0x90, 0x80, 0x80, + ]); + const dict = Buffer.concat([ + Buffer.from([0x7b]), + unicode(first), + marshalCode("py311", { name: "a", qualname: "a" }), + unicode(second), + marshalCode("py311", { name: "b", qualname: "b" }), + Buffer.from([0x30]), + ]); + const report = parse(withConsts([dict])); + expect(root(report).consts[0]).toEqual({ kind: "dict", length: 2 }); + expect(report.code_objects.map((code) => code.name)).toEqual([ + "", + "a", + "b", + ]); + expect(root(report).nested_code).toEqual([1, 2]); + expect(report.coverage.status).toBe("complete"); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +it("follows a long TYPE_REF chain without a recursive walk", () => { + const links = 6000; + const hidden = marshalCode("py311", { + name: "hidden", + qualname: "hidden", + }); + const later = marshalCode("py311", { + name: "later", + qualname: "later", + constsObject: Buffer.concat([ + Buffer.from([0x29, 1]), + marshalRef(links - 1), + ]), + }); + const pieces: Buffer[] = [Buffer.from([0x28]), u32(links + 1)]; + for (let index = 0; index < links; index += 1) { + pieces.push(Buffer.from([0x29 | 0x80, 1])); + pieces.push(index === 0 ? hidden : marshalRef(index - 1)); + } + pieces.push(later); + const bytes = Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { constsObject: Buffer.concat(pieces) }), + ]); + const started = Date.now(); + const report = parse(bytes); + expect(Date.now() - started).toBeLessThan(1000); + expect(report.code_objects.map((code) => code.name)).toEqual([ + "", + "hidden", + "later", + ]); + expect(report.code_objects[2]?.nested_code).toEqual([1]); + expect(root(report).nested_code).toEqual([1, 2]); + expect(report.coverage.status).toBe("complete"); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +it("rejects negative CPython code counts and keeps them out of the schema", () => { + const negative = ( + layout: "py37" | "py311", + fields: { + argcount?: number; + posonlyargcount?: number; + kwonlyargcount?: number; + nlocals?: number; + stacksize?: number; + }, + ) => + Buffer.concat([ + pythonPycHeader(layout === "py37" ? 3394 : 3531), + marshalCode(layout, fields), + ]); + for (const field of [ + "argcount", + "posonlyargcount", + "kwonlyargcount", + "stacksize", + ] as const) { + expect(() => parse(negative("py311", { [field]: 0xffffffff }))).toThrow( + PythonPycFormatFailure, + ); + } + expect(() => parse(negative("py37", { nlocals: 0xffffffff }))).toThrow( + /co_nlocals is negative/, + ); + expect(() => + parse(negative("py311", { argcount: 1, posonlyargcount: 2 })), + ).toThrow(/co_argcount is smaller than co_posonlyargcount/); + expect(() => parse(negative("py311", { argcount: 0x7fffffff }))).toThrow( + /co_varnames is too small/, + ); + const high = parse(negative("py311", {})); + expect(pythonPycSchema.safeParse(high).success).toBe(true); + expect( + pythonPycSchema.safeParse({ + ...high, + code_objects: [{ ...root(high), argcount: 0x7fffffff }], + }).success, + ).toBe(true); + expect( + pythonPycSchema.safeParse({ + ...high, + code_objects: [{ ...root(high), argcount: 0xffffffff }], + }).success, + ).toBe(false); + expect( + pythonPycSchema.safeParse({ + ...high, + code_objects: [{ ...root(high), stacksize: 0xffffffff }], + }).success, + ).toBe(false); +}); + +it("keeps code objects whose name tables differ only by boundaries", () => { + const frozen = (...members: Buffer[]): Buffer => + Buffer.concat([Buffer.from([0x3e]), u32(members.length), ...members]); + const code = ( + layout: "py311" | "py38", + fields: Parameters[1], + ): Buffer => + marshalCode(layout, { + name: "f", + qualname: "f", + filename: "f.py", + firstlineno: 1, + ...fields, + }); + const split = parse( + withConsts([ + frozen( + code("py311", { names: [] }), + code("py311", { names: [""] }), + code("py311", { names: ["a\0b"] }), + code("py311", { names: ["a", "b"] }), + code("py311", { names: ["a"] }), + code("py311", { names: ["a"], filename: "other.py" }), + ), + ]), + ); + expect(root(split).consts[0]).toEqual({ kind: "frozenset", length: 5 }); + expect(split.code_objects).toHaveLength(6); + expect(split.coverage).toMatchObject({ status: "complete", code_objects: 6 }); + expect(pythonPycSchema.safeParse(split).success).toBe(true); + + const tables = parse( + Buffer.concat([ + pythonPycHeader(3439), + marshalCode("py38", { + consts: [ + frozen( + code("py38", { varnames: ["a"], cellvars: ["b"] }), + code("py38", { varnames: ["a", "b"] }), + code("py38", { varnames: ["a"], freevars: ["b"] }), + ), + ], + }), + ]), + ); + expect(root(tables).consts[0]).toEqual({ kind: "frozenset", length: 3 }); + expect(tables.code_objects).toHaveLength(4); + expect(tables.coverage.status).toBe("complete"); +}); + +it("hashes a long TYPE_REF tuple chain without recursing", () => { + const links = 6000; + // Outer pair is (tuple chain, ref to the set) so the set length is visible. + // Small tuples are flagged in order; each one references the previous slot. + const pieces: Buffer[] = [ + Buffer.from([0x29, 2, 0x28 | 0x80]), + u32(links + 1), + Buffer.from([0x29 | 0x80, 1, 0x69]), + i32(1), + ]; + for (let index = 1; index < links; index += 1) + pieces.push(Buffer.from([0x29 | 0x80, 1]), marshalRef(index)); + pieces.push( + Buffer.from([0x3c | 0x80]), + u32(1), + marshalRef(links), + marshalRef(links + 1), + ); + const bytes = Buffer.concat([ + pythonPycHeader(3531), + marshalCode("py311", { constsObject: Buffer.concat(pieces) }), + ]); + expect(bytes.length).toBeLessThan(64_000); + const started = Date.now(); + const report = parse(bytes); + expect(Date.now() - started).toBeLessThan(1000); + expect(root(report).consts[0]).toEqual({ kind: "tuple", length: links + 1 }); + expect(root(report).consts[1]).toEqual({ kind: "set", length: 1 }); + expect(report.coverage.status).toBe("complete"); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +it("rejects code CPython treats as corrupt instead of complete coverage", () => { + const file = ( + layout: "py37" | "py38" | "py311", + magic: number, + fields: Parameters[1], + ): Buffer => + Buffer.concat([pythonPycHeader(magic), marshalCode(layout, fields)]); + expect(() => + parse(file("py311", 3531, { bytecode: Buffer.from([0x53]) })), + ).toThrow(/co_code is malformed/); + expect(() => parse(file("py311", 3531, { argcount: 100 }))).toThrow( + /co_varnames is too small/, + ); + expect(() => + parse( + file("py311", 3531, { + argcount: 1, + localsplusnames: ["a"], + localspluskinds: Buffer.from([0x00]), + }), + ), + ).toThrow(/co_varnames is too small/); + expect(() => parse(file("py311", 3531, { flags: CO_VARARGS }))).toThrow( + /co_varnames is too small/, + ); + expect(() => parse(file("py38", 3439, { argcount: 1 }))).toThrow( + /co_varnames is too small/, + ); + expect(() => + parse(file("py38", 3439, { argcount: 1, cellvars: ["a"] })), + ).toThrow(/co_varnames is too small/); + expect(() => parse(file("py37", 3394, { argcount: 1 }))).toThrow( + /co_varnames is too small/, + ); + + const valid = parse( + file("py311", 3531, { + argcount: 1, + kwonlyargcount: 1, + flags: CO_VARARGS, + localsplusnames: ["a", "b", "c"], + localspluskinds: Buffer.from([0x20, 0x20, 0x20]), + bytecode: Buffer.alloc(0), + }), + ); + expect(root(valid).argcount).toBe(1); + expect(valid.coverage.status).toBe("complete"); + const py37 = parse( + file("py37", 3394, { + argcount: 1, + varnames: ["a"], + bytecode: Buffer.from([0x53]), + }), + ); + expect(root(py37).argcount).toBe(1); + expect(root(py37).bytecode.bytes).toBe(1); + expect(py37.coverage.status).toBe("complete"); + expect(pythonPycSchema.safeParse(valid).success).toBe(true); +}); + +it("leaves 3.11 alpha magics with other code layouts unsupported", () => { + for (const magic of [3452, 3459, 3460, 3493]) { + const bytes = Buffer.concat([pythonPycHeader(magic), marshalCode("py311")]); + expect(() => parse(bytes)).toThrow( + expect.objectContaining({ kind: "unsupported" }), + ); + } + expect( + parse(pythonPycFixture("py311", { magic: 3495 })).header, + ).toMatchObject({ python_version: "3.11", code_layout: "py311" }); +}); diff --git a/src/python/PythonPycParser.ts b/src/python/PythonPycParser.ts new file mode 100644 index 0000000000..03b79bddab --- /dev/null +++ b/src/python/PythonPycParser.ts @@ -0,0 +1,1626 @@ +import { createHash } from "node:crypto"; +import { + PYTHON_PYC_CHECK_SOURCE, + PYTHON_PYC_CONST_PREVIEW_LIMIT, + PYTHON_PYC_HASH_BASED, + PYTHON_PYC_HEADER_BYTES, + PYTHON_PYC_INT_PREVIEW_BITS, + pythonPycMagicEntry, + type InspectPythonPycInput, + type PythonPyc, + type PythonPycCodeLayout, + type PythonPycCodeObject, + type PythonPycConst, +} from "../domain/python/pythonPyc.js"; +import { + PythonPycFormatFailure, + PythonPycResourceFailure, +} from "./PythonPyc.errors.js"; + +const FLAG_REF = 0x80; +const TYPE_NULL = 0x30; +const TYPE_NONE = 0x4e; +const TYPE_FALSE = 0x46; +const TYPE_TRUE = 0x54; +const TYPE_STOPITER = 0x53; +const TYPE_ELLIPSIS = 0x2e; +const TYPE_INT = 0x69; +const TYPE_FLOAT_TEXT = 0x66; +const TYPE_FLOAT = 0x67; +const TYPE_COMPLEX_TEXT = 0x78; +const TYPE_COMPLEX = 0x79; +const TYPE_LONG = 0x6c; +const TYPE_BYTES = 0x73; +const TYPE_INTERNED = 0x74; +const TYPE_REF = 0x72; +const TYPE_TUPLE = 0x28; +const TYPE_LIST = 0x5b; +const TYPE_DICT = 0x7b; +const TYPE_CODE = 0x63; +const TYPE_UNICODE = 0x75; +const TYPE_SET = 0x3c; +const TYPE_FROZENSET = 0x3e; +const TYPE_ASCII = 0x61; +const TYPE_ASCII_INTERNED = 0x41; +const TYPE_SMALL_TUPLE = 0x29; +const TYPE_SHORT_ASCII = 0x7a; +const TYPE_SHORT_ASCII_INTERNED = 0x5a; +/** CPython `CO_VARARGS` / `CO_VARKEYWORDS`. Each consumes one local name. */ +const CO_VARARGS = 0x04; +const CO_VARKEYWORDS = 0x08; +/** CPython `_PyLocals_Kind` bit counted by `_PyCode_Validate` as a local. */ +const CO_FAST_LOCAL = 0x20; + +type MarshalValue = + | { readonly kind: "null" } + | { readonly kind: "none" } + | { readonly kind: "bool"; readonly value: boolean } + | { readonly kind: "ellipsis" } + | { readonly kind: "stopiter" } + | { readonly kind: "int"; readonly value: bigint; readonly bits: number } + | { readonly kind: "float"; readonly value: number } + | { readonly kind: "complex"; readonly real: number; readonly imag: number } + | { readonly kind: "bytes"; readonly value: Buffer } + | { + readonly kind: "str"; + readonly value: string; + readonly bytes: number; + /** Marshal bytes. JavaScript strings merge lone surrogates with pairs. */ + readonly raw: Buffer; + } + | { readonly kind: "tuple"; readonly items: MarshalValue[] } + | { readonly kind: "list"; readonly items: MarshalValue[] } + | { readonly kind: "set"; readonly items: MarshalValue[] } + | { readonly kind: "frozenset"; readonly items: MarshalValue[] } + | { + readonly kind: "dict"; + readonly items: [MarshalValue, MarshalValue][]; + } + | { readonly kind: "code"; readonly value: CodeNode }; + +/** + * A decoded code object. `consts` and `nested` stay empty until the marshal + * graph is complete: a container can refer to itself before its items exist. + * TYPE_REF children are the same node, not copies. + */ +type CodeNode = Omit & { + consts: PythonPycConst[]; + nested: CodeNode[]; +}; + +const EMPTY_CODES: readonly CodeNode[] = []; + +type KeyMode = "hash" | "const"; + +/** + * Equality token plus the size of the expanded CPython-style key. + * Container tokens are interned so a reference chain does not copy every + * ancestor into the next string. `expanded` is still the size that string + * would have, so a repeated child cannot double without bound. + */ +type ResolvedKey = { + readonly token: string; + readonly expanded: number; +}; + +type KeyFrame = { + value: MarshalValue; + index: number; + parts: string[]; + expanded: number[]; + failed: boolean; +}; + +const isContainer = (value: MarshalValue): boolean => + value.kind === "tuple" || + value.kind === "list" || + value.kind === "set" || + value.kind === "frozenset" || + value.kind === "dict"; + +const containerChildren = (value: MarshalValue): readonly MarshalValue[] => { + if (value.kind === "dict") { + const children: MarshalValue[] = []; + for (const [key, mapped] of value.items) children.push(key, mapped); + return children; + } + if ( + value.kind === "tuple" || + value.kind === "list" || + value.kind === "set" || + value.kind === "frozenset" + ) + return value.items; + return []; +}; + +/** + * Fields CPython uses for code equality that the public report does not keep. + * `consts` stays the decoded graph so a code key can follow Python's rules + * without rebuilding marshal text. + */ +type CodeEquality = { + readonly consts: readonly MarshalValue[]; + readonly linetable: string; + readonly exceptiontable: string; +}; + +const malformed = (message: string): never => { + throw new PythonPycFormatFailure("malformed", message); +}; + +/** Locals CPython's 3.11+ validator counts toward `co_varnames`. */ +const fastLocalCount = (kinds: Buffer): number => { + let count = 0; + for (let index = 0; index < kinds.length; index += 1) { + const kind = kinds[index] ?? 0; + if ((kind & CO_FAST_LOCAL) !== 0) count += 1; + } + return count; +}; + +const unsupported = (message: string): never => { + throw new PythonPycFormatFailure("unsupported", message); +}; + +const digests = new WeakMap(); +const sha256 = (bytes: Buffer): string => { + const cached = digests.get(bytes); + if (cached !== undefined) return cached; + const digest = createHash("sha256").update(bytes).digest("hex"); + digests.set(bytes, digest); + return digest; +}; + +/** Decode one bounded CPython 3.7–3.13 pyc without executing bytecode. */ +export const parsePythonPyc = ( + bytes: Buffer, + artifact: { path: string; sha256: string; bytes: number }, + input: InspectPythonPycInput, +): PythonPyc => { + if (bytes.length !== artifact.bytes) + malformed("Selected artifact length disagrees with the retained bytes."); + if (bytes.length < PYTHON_PYC_HEADER_BYTES) + malformed( + "Selected file is shorter than a 16-byte CPython 3.7+ pyc header.", + ); + if (bytes[2] !== 0x0d || bytes[3] !== 0x0a) + malformed("Pyc magic is not a little-endian integer followed by CR LF."); + const magic = bytes.readUInt16LE(0); + const mapped = pythonPycMagicEntry(magic); + if (mapped === null) + return unsupported( + `Pyc magic ${magic} is outside the pinned CPython 3.7–3.13 table.`, + ); + const flags = bytes.readUInt32LE(4); + const hashBased = (flags & PYTHON_PYC_HASH_BASED) !== 0; + const header = { + magic, + python_version: mapped.python_version, + code_layout: mapped.code_layout, + flags, + hash_based: hashBased, + check_source: (flags & PYTHON_PYC_CHECK_SOURCE) !== 0, + source_hash_hex: hashBased ? bytes.subarray(8, 16).toString("hex") : null, + source_mtime: hashBased ? null : bytes.readUInt32LE(8), + source_size: hashBased ? null : bytes.readUInt32LE(12), + }; + const marshal = bytes.subarray(PYTHON_PYC_HEADER_BYTES); + const reader = new MarshalReader(marshal, mapped.code_layout, input); + const root = reader.readObject(0); + if (reader.offset !== marshal.length) + malformed("Marshal stream has trailing bytes after the top-level object."); + if (root.kind !== "code") + return malformed("Top-level marshal object is not a code object."); + reader.finishCodes(); + const codeObjects = flattenCode(root.value); + const unknownFlags = + flags & ~(PYTHON_PYC_HASH_BASED | PYTHON_PYC_CHECK_SOURCE); + return { + artifact: { + path: artifact.path, + sha256: artifact.sha256, + bytes: artifact.bytes, + }, + header, + code_objects: codeObjects, + coverage: { + status: "complete", + objects: reader.objects, + code_objects: codeObjects.length, + marshal_bytes: marshal.length, + marshal_depth: reader.depthReached, + }, + limitations: [ + "Bytecode is retained as length and SHA-256 only; this workflow does not decompile, disassemble, or recover source.", + "The selected file is not executed and CPython is not invoked.", + hashBased + ? "source_hash_hex is the PEP 552 header field; the original .py is not read." + : "source_mtime and source_size are header fields; they do not prove a matching .py is present.", + "python_version is the pinned CPython magic table for this header, not a runtime interpreter.", + ...(unknownFlags === 0 + ? [] + : [ + `Header flags ${flags} include unrecognized bits; only hash-based and check-source bits are decoded.`, + ]), + ], + }; +}; + +class MarshalReader { + offset = 0; + objects = 0; + depthReached = 0; + readonly #intern: (MarshalValue | undefined)[] = []; + /** Name and identifier bytes the report carries across all code objects. */ + #reportedStringBytes = 0; + /** Const summaries the report copies, including tables shared by TYPE_REF. */ + #reportedConsts = 0; + /** Bytes of hash-key descriptors materialized for set and dict rules. */ + #keyBytes = 0; + /** Identity keys for NaN values, which are not equal to themselves. */ + readonly #identities = new WeakMap(); + #identityNext = 0; + /** One Python equality key per object. Null means the value is unhashable. */ + readonly #keyCache = new WeakMap(); + /** Constant keys keep signed zeroes distinct and give cycles an identity. */ + readonly #constCache = new WeakMap(); + /** Interned descriptors so equal values share one token within this walk. */ + readonly #keyTokens = new Map(); + #keyTokenNext = 0; + /** Explicit key-walk steps. Stream `max_depth` does not bound reference chains. */ + #keySteps = 0; + /** One code-equality key per code object. Empty while that key is in progress. */ + readonly #codeKeys = new WeakMap(); + readonly #codeIdentities = new WeakMap(); + readonly #codeEquality = new WeakMap(); + /** Code objects whose const summaries wait until the graph is fully decoded. */ + readonly #codes: CodeNode[] = []; + /** + * Nested code objects reached from one container. Pure wrappers store the + * child's array; fresh arrays are counted so copies cannot grow without a + * bound. Absent while a container is still on the explicit walk stack. + */ + readonly #nestedCache = new WeakMap(); + #nestedSteps = 0; + #nestedRetained = 0; + + constructor( + readonly bytes: Buffer, + readonly layout: PythonPycCodeLayout, + readonly input: InspectPythonPycInput, + ) {} + + readObject(depth: number): MarshalValue { + this.objects += 1; + if (this.objects > this.input.max_objects) + throw new PythonPycResourceFailure( + "objects", + { max_objects: this.input.max_objects }, + `Marshal object count exceeds max_objects ${this.input.max_objects}.`, + ); + if (depth > this.input.max_depth) + throw new PythonPycResourceFailure( + "depth", + { max_depth: this.input.max_depth }, + `Marshal nesting exceeds max_depth ${this.input.max_depth}.`, + ); + this.depthReached = Math.max(this.depthReached, depth); + const raw = this.u8(); + const flagged = (raw & FLAG_REF) !== 0; + const type = raw & ~FLAG_REF; + if (type === TYPE_REF) { + if (flagged) malformed("TYPE_REF must not also set FLAG_REF."); + return this.readRef(); + } + const slot = flagged ? this.#intern.length : undefined; + if (slot !== undefined) this.#intern.push(undefined); + const value = this.readTyped(type, depth, slot); + if (slot !== undefined && this.#intern[slot] === undefined) + this.#intern[slot] = value; + return value; + } + + readTyped( + type: number, + depth: number, + slot: number | undefined, + ): MarshalValue { + switch (type) { + case TYPE_NULL: + return { kind: "null" }; + case TYPE_NONE: + return { kind: "none" }; + case TYPE_FALSE: + return { kind: "bool", value: false }; + case TYPE_TRUE: + return { kind: "bool", value: true }; + case TYPE_STOPITER: + return { kind: "stopiter" }; + case TYPE_ELLIPSIS: + return { kind: "ellipsis" }; + case TYPE_INT: { + const value = BigInt(this.i32()); + return { kind: "int", value, bits: smallBitLength(value) }; + } + case TYPE_FLOAT_TEXT: + return { kind: "float", value: this.textFloat() }; + case TYPE_FLOAT: + return { kind: "float", value: this.f64() }; + case TYPE_COMPLEX_TEXT: + return { + kind: "complex", + real: this.textFloat(), + imag: this.textFloat(), + }; + case TYPE_COMPLEX: + return { kind: "complex", real: this.f64(), imag: this.f64() }; + case TYPE_LONG: { + const decoded = this.long(); + return { kind: "int", value: decoded.value, bits: decoded.bits }; + } + case TYPE_BYTES: + return { kind: "bytes", value: this.blob(this.u32()) }; + case TYPE_INTERNED: + case TYPE_UNICODE: + return this.utf8(this.u32(), false); + case TYPE_ASCII: + case TYPE_ASCII_INTERNED: + return this.utf8(this.u32(), true); + case TYPE_SHORT_ASCII: + case TYPE_SHORT_ASCII_INTERNED: + return this.utf8(this.u8(), true); + case TYPE_TUPLE: + return this.sequenceValue("tuple", this.u32(), depth, slot); + case TYPE_SMALL_TUPLE: + return this.sequenceValue("tuple", this.u8(), depth, slot); + case TYPE_LIST: + return this.sequenceValue("list", this.u32(), depth, slot); + case TYPE_SET: + return this.setValue("set", this.u32(), depth, slot); + case TYPE_FROZENSET: + return this.setValue("frozenset", this.u32(), depth, slot); + case TYPE_DICT: + return this.dict(depth, slot); + case TYPE_CODE: + return { kind: "code", value: this.code(depth) }; + default: + return malformed(`Unsupported marshal type 0x${type.toString(16)}.`); + } + } + + code(depth: number): CodeNode { + const argcount = this.signedCount("co_argcount"); + const posonlyargcount = + this.layout === "py37" ? null : this.signedCount("co_posonlyargcount"); + if (posonlyargcount !== null && argcount < posonlyargcount) + malformed("co_argcount is smaller than co_posonlyargcount."); + const kwonlyargcount = this.signedCount("co_kwonlyargcount"); + const nlocals = + this.layout === "py311" ? null : this.signedCount("co_nlocals"); + const stacksize = this.signedCount("co_stacksize"); + const flags = this.u32(); + const bytecode = asBytes(this.readObject(depth + 1), "co_code"); + // 3.11+ `_PyCode_Validate` rejects a length that is not a whole code unit. + // 3.7–3.10 constructors do not, so an odd `co_code` stays readable there. + if (this.layout === "py311" && bytecode.length % 2 !== 0) + malformed("co_code is malformed."); + const consts = asTuple(this.readObject(depth + 1), "co_consts"); + this.#chargeConsts(consts.length); + const names = this.#nameTable( + asTuple(this.readObject(depth + 1), "co_names"), + ); + let varnames: string[] | null = null; + let freevars: string[] | null = null; + let cellvars: string[] | null = null; + let localsplusnames: string[] | null = null; + let localspluskinds_hex: string | null = null; + if (this.layout === "py311") { + localsplusnames = this.#nameTable( + asTuple(this.readObject(depth + 1), "co_localsplusnames"), + ); + const localspluskinds = asBytes( + this.readObject(depth + 1), + "co_localspluskinds", + ); + if (localspluskinds.length !== localsplusnames.length) + malformed( + "co_localspluskinds length does not match co_localsplusnames.", + ); + localspluskinds_hex = localspluskinds.toString("hex"); + this.#rejectUndersizedLocals( + argcount, + kwonlyargcount, + flags, + fastLocalCount(localspluskinds), + ); + } else { + varnames = this.#nameTable( + asTuple(this.readObject(depth + 1), "co_varnames"), + ); + freevars = this.#nameTable( + asTuple(this.readObject(depth + 1), "co_freevars"), + ); + cellvars = this.#nameTable( + asTuple(this.readObject(depth + 1), "co_cellvars"), + ); + this.#rejectUndersizedLocals( + argcount, + kwonlyargcount, + flags, + varnames.length, + ); + } + const filename = asString(this.readObject(depth + 1), "co_filename"); + const name = asString(this.readObject(depth + 1), "co_name"); + const qualname = + this.layout === "py311" + ? asString(this.readObject(depth + 1), "co_qualname") + : null; + const firstlineno = this.u32(); + const linetable = asBytes(this.readObject(depth + 1), "co_linetable"); + const exceptiontable = + this.layout === "py311" + ? asBytes(this.readObject(depth + 1), "co_exceptiontable") + : Buffer.alloc(0); + this.#reportStrings([ + name, + qualname ?? "", + filename, + ...names, + ...(varnames ?? []), + ...(freevars ?? []), + ...(cellvars ?? []), + ...(localsplusnames ?? []), + localspluskinds_hex ?? "", + ]); + const node: CodeNode = { + name, + qualname, + filename, + firstlineno, + argcount, + posonlyargcount, + kwonlyargcount, + nlocals, + stacksize, + flags, + names, + varnames, + freevars, + cellvars, + localsplusnames, + localspluskinds_hex, + bytecode: { bytes: bytecode.length, sha256: sha256(bytecode) }, + consts: [], + nested: [], + }; + this.#codeEquality.set(node, { + consts, + linetable: `${linetable.length}:${sha256(linetable)}`, + exceptiontable: `${exceptiontable.length}:${sha256(exceptiontable)}`, + }); + this.#codes.push(node); + return node; + } + + /** + * Const summaries and nested relationships follow Python's object graph. + * Building them while `code()` is still inside an ancestor container caches + * that container before its items exist. + */ + finishCodes(): void { + for (const node of this.#codes) { + const material = + this.#codeEquality.get(node) ?? + malformed("Code object is missing decoded constants."); + node.consts = this.#summarizeConsts(material.consts); + node.nested = this.#nestedCodeObjects(material.consts); + } + } + + /** + * A code-constant summary copies the child's name. Charge every copy the + * report will carry before that summary is built. + */ + #summarizeConsts(items: readonly MarshalValue[]): PythonPycConst[] { + const summaries: PythonPycConst[] = []; + for (const item of items) { + if (item.kind === "code") this.#reportStrings([item.value.name]); + summaries.push(summarizeConst(item)); + } + return summaries; + } + + /** + * Strings reached through TYPE_REF are one marshal object but a full copy + * in the report, so bound what the report carries, not the stream. + */ + #reportStrings(values: readonly string[]): void { + for (const value of values) + this.#reportedStringBytes += Buffer.byteLength(value, "utf8"); + if (this.#reportedStringBytes > this.input.max_file_bytes) + throw new PythonPycResourceFailure( + "string", + { max_file_bytes: this.input.max_file_bytes }, + `Reported code-object strings exceed max_file_bytes ${this.input.max_file_bytes}.`, + ); + } + + /** + * A consts tuple shared through TYPE_REF is one decode but a full copy in + * every code object. Stop before those copies are built. + */ + #chargeConsts(count: number): void { + this.#chargeReported(count, "constants"); + } + + /** + * A name table shared through TYPE_REF is one marshal object and a full + * copy in every code object. Empty strings add no report bytes, so charge + * each copied entry before `strings()` builds the array the report keeps. + */ + #nameTable(items: readonly MarshalValue[]): string[] { + this.#chargeReported(items.length, "names"); + return strings(items); + } + + /** + * Charge copies the report will carry. Nested relationships are separate + * from the immediate const tuple, which misses a shared container repeated + * by many owners. Name-table copies are separate from the byte budget, + * which misses an empty string repeated by many owners. + */ + #chargeReported( + count: number, + what: "constants" | "names" | "nested code relationships", + ): void { + this.#reportedConsts += count; + if (this.#reportedConsts > this.input.max_objects) + throw new PythonPycResourceFailure( + "objects", + { max_objects: this.input.max_objects }, + `Reported ${what} exceed max_objects ${this.input.max_objects}.`, + ); + } + + /** + * Distinct code objects reachable from this constant tuple. Containers are + * walked iteratively, and a wrapper keeps the child's array instead of + * copying every descendant. Charge the owner's reported relationships + * before the array the report keeps is built. + */ + #nestedCodeObjects(items: readonly MarshalValue[]): CodeNode[] { + const seen = new Set(); + const ordered: CodeNode[] = []; + const add = (code: CodeNode): void => { + if (seen.has(code)) return; + seen.add(code); + ordered.push(code); + }; + for (const item of items) { + if (item.kind === "code") { + add(item.value); + continue; + } + for (const code of this.#reachableCodes(item)) add(code); + } + this.#chargeReported(ordered.length, "nested code relationships"); + return ordered; + } + + #reachableCodes(value: MarshalValue): readonly CodeNode[] { + if (value.kind === "code") return [value.value]; + if (!isContainer(value)) return EMPTY_CODES; + const cached = this.#nestedCache.get(value); + if (cached !== undefined) return cached; + const added: { value: MarshalValue; retained: number }[] = []; + if (this.#memoizeNested(value, added)) { + for (const entry of added) { + this.#nestedCache.delete(entry.value); + this.#nestedRetained -= entry.retained; + } + const walked = this.#walkNested(value); + this.#nestedCache.set(value, walked); + return walked; + } + return this.#nestedCache.get(value) ?? EMPTY_CODES; + } + + /** + * Post-order walk on an explicit stack. Each frame keeps its child list, + * so a dictionary is not copied once per entry. A back edge returns before + * any cyclic container is cached; the caller then scans that component once. + */ + #memoizeNested( + root: MarshalValue, + added: { value: MarshalValue; retained: number }[], + ): boolean { + const gray = new Set(); + const frames: { + value: MarshalValue; + index: number; + /** Built once. A dictionary child list is not copied on every step. */ + children: readonly MarshalValue[]; + }[] = [{ value: root, index: 0, children: containerChildren(root) }]; + gray.add(root); + while (frames.length > 0) { + this.#takeNestedStep(); + const frame = frames[frames.length - 1]; + if (frame === undefined) break; + const children = frame.children; + if (frame.index < children.length) { + const child = children[frame.index]; + frame.index += 1; + if (child === undefined || !isContainer(child)) continue; + if (this.#nestedCache.has(child)) continue; + if (gray.has(child)) return true; + gray.add(child); + frames.push({ + value: child, + index: 0, + children: containerChildren(child), + }); + continue; + } + const retainedBefore = this.#nestedRetained; + const codes = this.#combineNested(frame.children); + this.#nestedCache.set(frame.value, codes); + added.push({ + value: frame.value, + retained: this.#nestedRetained - retainedBefore, + }); + gray.delete(frame.value); + frames.pop(); + } + return false; + } + + /** One scan of a container cycle. Cached containers outside the cycle are reused. */ + #walkNested(root: MarshalValue): readonly CodeNode[] { + const seen = new Set(); + const visited = new Set(); + const stack: MarshalValue[] = [root]; + while (stack.length > 0) { + this.#takeNestedStep(); + const current = stack.pop(); + if (current === undefined) break; + if (current.kind === "code") { + seen.add(current.value); + continue; + } + if (!isContainer(current) || visited.has(current)) continue; + const known = + current === root ? undefined : this.#nestedCache.get(current); + if (known !== undefined) { + for (const code of known) seen.add(code); + continue; + } + visited.add(current); + const children = containerChildren(current); + for (let index = children.length - 1; index >= 0; index -= 1) { + const child = children[index]; + if (child !== undefined) stack.push(child); + } + } + const codes = [...seen]; + this.#retainNested(codes); + return codes; + } + + /** + * Share a child's array when this container adds no further code object. + * Otherwise retain one new list and count its slots against the walk budget. + * `children` is the list the frame already built. + */ + #combineNested(children: readonly MarshalValue[]): readonly CodeNode[] { + let shared: readonly CodeNode[] | undefined; + let onlyShared = true; + const codes: CodeNode[] = []; + const seen = new Set(); + const add = (code: CodeNode): void => { + if (seen.has(code)) return; + seen.add(code); + codes.push(code); + }; + for (const child of children) { + if (child.kind === "code") { + if (shared !== undefined || codes.length > 0) onlyShared = false; + add(child.value); + continue; + } + if (!isContainer(child)) continue; + const part = this.#nestedCache.get(child); + if (part === undefined || part.length === 0) continue; + if (shared === undefined && codes.length === 0) shared = part; + else onlyShared = false; + for (const code of part) add(code); + } + if (codes.length === 0) return EMPTY_CODES; + if (onlyShared && shared !== undefined && shared.length === codes.length) + return shared; + this.#retainNested(codes); + return codes; + } + + #takeNestedStep(): void { + this.#nestedSteps += 1; + const limit = this.input.max_objects * this.input.max_depth; + if (this.#nestedSteps > limit) + throw new PythonPycResourceFailure( + "objects", + { + max_objects: this.input.max_objects, + max_depth: this.input.max_depth, + }, + `Nested code walk exceeds ${limit} steps (max_objects * max_depth).`, + ); + } + + #retainNested(codes: readonly CodeNode[]): void { + this.#nestedRetained += codes.length; + const limit = this.input.max_objects * this.input.max_depth; + if (this.#nestedRetained > limit) + throw new PythonPycResourceFailure( + "objects", + { + max_objects: this.input.max_objects, + max_depth: this.input.max_depth, + }, + `Nested code walk retains more than ${limit} code references.`, + ); + } + + readRef(): MarshalValue { + const index = this.u32(); + const value = this.#intern[index]; + if (value === undefined) + return malformed( + `TYPE_REF index ${index} is missing or still being constructed.`, + ); + return value; + } + + #publish(slot: number | undefined, value: MarshalValue): void { + if (slot !== undefined) this.#intern[slot] = value; + } + + /** + * CPython registers tuples and lists before reading items, so a container + * can refer to itself. The same object is what later TYPE_REF entries see. + */ + sequenceValue( + kind: "tuple" | "list", + count: number, + depth: number, + slot: number | undefined, + ): MarshalValue { + const items: MarshalValue[] = []; + const value: MarshalValue = + kind === "tuple" ? { kind: "tuple", items } : { kind: "list", items }; + this.#publish(slot, value); + for (let index = 0; index < count; index += 1) { + const item = this.readObject(depth + 1); + if (item.kind === "null") + malformed("NULL object in marshal data for sequence."); + items.push(item); + } + return value; + } + + /** + * Sets are registered before their members. Frozensets are registered after, + * matching CPython, so a reference inside a frozenset body is still invalid. + * Equal members count once; unhashable members fail the walk. + */ + setValue( + kind: "set" | "frozenset", + count: number, + depth: number, + slot: number | undefined, + ): MarshalValue { + const items: MarshalValue[] = []; + const value: MarshalValue = + kind === "set" ? { kind: "set", items } : { kind: "frozenset", items }; + if (kind === "set") this.#publish(slot, value); + for (let index = 0; index < count; index += 1) { + const item = this.readObject(depth + 1); + if (item.kind === "null") + malformed("NULL object in marshal data for set."); + items.push(item); + } + const unique = this.#unique(items, "set member"); + items.length = 0; + for (const item of unique) items.push(item); + return value; + } + + /** + * Dictionaries are registered before their pairs. A null key or a null value + * ends the dict; a null value does not become an entry. Equal keys keep the + * first key object and replace only its value. An unhashable key fails the + * walk. + */ + dict(depth: number, slot: number | undefined): MarshalValue { + const items: [MarshalValue, MarshalValue][] = []; + const value: MarshalValue = { kind: "dict", items }; + this.#publish(slot, value); + const indexes = new Map(); + for (;;) { + const key = this.readObject(depth + 1); + if (key.kind === "null") return value; + const mapped = this.readObject(depth + 1); + if (mapped.kind === "null") return value; + const hash = this.#hashKey(key, "dictionary key"); + const existing = indexes.get(hash); + if (existing === undefined) { + indexes.set(hash, items.length); + items.push([key, mapped]); + } else { + const kept = + items[existing] ?? malformed("Dictionary entry is missing."); + kept[1] = mapped; + } + } + } + + #unique(items: readonly MarshalValue[], role: string): MarshalValue[] { + const seen = new Map(); + for (const item of items) { + const key = this.#hashKey(item, role); + if (!seen.has(key)) seen.set(key, item); + } + return [...seen.values()]; + } + + /** + * CPython rejects code whose argument counts do not fit the local table. + * 3.11+ counts `CO_FAST_LOCAL` entries (`co_varnames is too small`). + * 3.7–3.10 count `co_varnames` the same way, including `*args` and `**kwargs`. + */ + #rejectUndersizedLocals( + argcount: number, + kwonlyargcount: number, + flags: number, + locals: number, + ): void { + let plain = locals - argcount - kwonlyargcount; + if ((flags & CO_VARARGS) !== 0) plain -= 1; + if ((flags & CO_VARKEYWORDS) !== 0) plain -= 1; + if (plain < 0) malformed("co_varnames is too small."); + } + + #hashKey(value: MarshalValue, role: string): string { + return this.#equalityKey(value, "hash") ?? malformed(`Unhashable ${role}.`); + } + + /** + * Python equality for hash-based containers and for code-constant identity. + * Tuple and frozenset chains use an explicit stack: a `TYPE_REF` chain is + * not stream nesting, and `max_depth` must not turn a loadable file into + * a stack overflow. Expanded size is checked per container so a shared + * child that is stored twice still cannot double without bound. + */ + #equalityKey(value: MarshalValue, mode: KeyMode): string | null { + const resolved = this.#resolveKey(value, mode); + return resolved === null ? null : resolved.token; + } + + #resolveKey(value: MarshalValue, mode: KeyMode): ResolvedKey | null { + const cached = this.#cachedKey(value, mode); + if (cached !== undefined) return cached; + const frames: KeyFrame[] = [ + { value, index: 0, parts: [], expanded: [], failed: false }, + ]; + const gray = new Set(); + gray.add(value); + while (frames.length > 0) { + this.#takeKeyStep(); + const frame = frames[frames.length - 1]; + if (frame === undefined) break; + const items = + frame.value.kind === "tuple" || frame.value.kind === "frozenset" + ? frame.value.items + : undefined; + if (items === undefined) { + const key = this.#leafKey(frame.value, mode); + this.#storeKey(frame.value, mode, key, gray); + frames.pop(); + this.#recordChild(frames, key); + continue; + } + if (frame.failed || frame.index >= items.length) { + const key = frame.failed + ? null + : this.#containerKey(frame.value, frame.parts, frame.expanded); + this.#storeKey(frame.value, mode, key, gray); + frames.pop(); + this.#recordChild(frames, key); + continue; + } + const child = items[frame.index]; + frame.index += 1; + if (child === undefined) continue; + const known = this.#cachedKey(child, mode); + if (known !== undefined) { + if (known === null) frame.failed = true; + else this.#pushPart(frame, known); + continue; + } + if (gray.has(child)) { + if (mode === "hash") frame.failed = true; + else this.#pushPart(frame, this.#identityKey(child)); + continue; + } + gray.add(child); + frames.push({ + value: child, + index: 0, + parts: [], + expanded: [], + failed: false, + }); + } + const resolved = this.#cachedKey(value, mode); + return resolved === undefined ? null : resolved; + } + + #cachedKey( + value: MarshalValue, + mode: KeyMode, + ): ResolvedKey | null | undefined { + return mode === "hash" + ? this.#keyCache.get(value) + : this.#constCache.get(value); + } + + #storeKey( + value: MarshalValue, + mode: KeyMode, + key: ResolvedKey | null, + gray: Set, + ): void { + if (mode === "hash") this.#keyCache.set(value, key); + else if (key !== null) this.#constCache.set(value, key); + gray.delete(value); + } + + #recordChild(frames: KeyFrame[], key: ResolvedKey | null): void { + const parent = frames[frames.length - 1]; + if (parent === undefined) return; + if (key === null) { + parent.failed = true; + return; + } + this.#pushPart(parent, key); + } + + #pushPart(frame: KeyFrame, key: ResolvedKey): void { + frame.parts.push(key.token); + frame.expanded.push(key.expanded); + } + + /** + * One step per explicit-stack iteration. The cap follows decoded objects, + * not `max_depth`, because a reference chain is shallow in the stream. + */ + #takeKeyStep(): void { + this.#keySteps += 1; + const limit = this.input.max_objects * 8; + if (this.#keySteps > limit) + throw new PythonPycResourceFailure( + "objects", + { max_objects: this.input.max_objects }, + `Marshal key walk exceeds ${limit} steps (max_objects * 8).`, + ); + } + + #leafKey(value: MarshalValue, mode: KeyMode): ResolvedKey | null { + const descriptor = this.#leafDescriptor(value, mode); + if (descriptor === null) return null; + return { + token: this.#internToken(descriptor), + expanded: descriptor.length, + }; + } + + #leafDescriptor(value: MarshalValue, mode: KeyMode): string | null { + switch (value.kind) { + case "none": + return "N"; + case "ellipsis": + return "E"; + case "stopiter": + return "S"; + case "bool": + if (mode === "hash") return value.value ? "i1" : "i0"; + return value.value ? "B1" : "B0"; + case "int": + if (value.bits > this.input.max_string_bytes * 3) + this.#keyTooLarge(value.bits); + return this.#boundedJoin("i", [value.value.toString()], ""); + case "float": + if (Number.isNaN(value.value)) return this.#identity(value); + if (mode === "const") return this.#constFloat(value.value); + return this.#numberKey(value.value); + case "complex": + if (Number.isNaN(value.real) || Number.isNaN(value.imag)) + return this.#identity(value); + if (mode === "hash" && value.imag === 0) + return this.#numberKey(value.real); + if (mode === "hash") + return this.#boundedJoin( + "c", + [this.#numberKey(value.real), this.#numberKey(value.imag)], + ":", + ); + return this.#boundedJoin( + "c", + [ + this.#signedZero(value.real), + this.#signedZero(value.imag), + this.#numberKey(value.real), + this.#numberKey(value.imag), + ], + ":", + ); + case "str": + return this.#stringKey(value.raw); + case "bytes": { + this.#reserveKey(value.value.length * 2 + 8); + const prefix = mode === "hash" ? "b" : "y"; + return `${prefix}${value.value.length}:${value.value.toString("hex")}`; + } + case "code": + return this.#codeKey(value.value); + default: + return mode === "const" ? this.#identity(value) : null; + } + } + + #constFloat(value: number): string { + if (Object.is(value, -0)) return "z-0"; + if (Object.is(value, 0)) return "z0"; + return `z${this.#numberKey(value)}`; + } + + #containerKey( + value: MarshalValue, + parts: readonly string[], + childExpanded: readonly number[], + ): ResolvedKey { + const expanded = this.#expandedContainerSize(childExpanded); + if (expanded > this.input.max_string_bytes) this.#keyTooLarge(expanded); + const ordered = value.kind === "frozenset" ? [...parts].sort() : parts; + const kind = value.kind === "tuple" ? "T" : "F"; + const descriptor = `${kind}:${ordered.join(",")}`; + // Charge the descriptor that is stored, not the unmaterialized expansion. + // A linear reference chain stays within the file budget; repeating one + // child still fails above when `expanded` passes max_string_bytes. + this.#reserveKey(descriptor.length); + return { token: this.#internToken(descriptor), expanded }; + } + + /** Size of the old length-prefixed container key, without building it. */ + #expandedContainerSize(childExpanded: readonly number[]): number { + let size = 1; + if (childExpanded.length > 0) size += childExpanded.length - 1; + for (const child of childExpanded) size += String(child).length + 1 + child; + return size; + } + + #internToken(descriptor: string): string { + const existing = this.#keyTokens.get(descriptor); + if (existing !== undefined) return existing; + this.#keyTokenNext += 1; + const token = `k${this.#keyTokenNext.toString(36)}`; + this.#keyTokens.set(descriptor, token); + return token; + } + + #identityKey(value: MarshalValue): ResolvedKey { + const token = this.#identity(value); + return { token, expanded: token.length }; + } + + /** + * CPython code equality ignores filename, qualname, and stack size. Consts + * use the constant-key rules, which keep -0 distinct from 0. + */ + #codeKey(node: CodeNode): string { + const existing = this.#codeKeys.get(node); + if (existing !== undefined && existing !== "") return existing; + if (existing === "") return `cycle${this.#codeIdentity(node)}`; + this.#codeKeys.set(node, ""); + const material = + this.#codeEquality.get(node) ?? + malformed("Code object is missing equality material."); + const nameTables = [ + this.#nameTableKey(node.names), + ...(node.localsplusnames !== null + ? [this.#nameTableKey(node.localsplusnames)] + : [ + this.#nameTableKey(node.varnames ?? []), + this.#nameTableKey(node.freevars ?? []), + this.#nameTableKey(node.cellvars ?? []), + ]), + ]; + const fields = [ + node.name, + String(node.argcount), + String(node.posonlyargcount ?? 0), + String(node.kwonlyargcount), + node.nlocals === null ? "" : String(node.nlocals), + String(node.flags), + String(node.firstlineno), + `${node.bytecode.bytes}:${node.bytecode.sha256}`, + this.#constKeyItems(material.consts), + ...nameTables, + material.linetable, + material.exceptiontable, + ]; + const key = this.#boundedJoin( + "", + fields.map((field) => `${field.length}:${field}`), + ",", + ); + this.#codeKeys.set(node, key); + return key; + } + + #codeIdentity(node: CodeNode): number { + const existing = this.#codeIdentities.get(node); + if (existing !== undefined) return existing; + this.#identityNext += 1; + this.#codeIdentities.set(node, this.#identityNext); + return this.#identityNext; + } + + /** + * One name table, with its length and each entry's boundary. Joining on a + * bare NUL makes `()` and `("",)` the same key, and the same join hides + * which of `co_varnames`, `co_freevars`, and `co_cellvars` held each name. + */ + #nameTableKey(items: readonly string[]): string { + const entries = items.map((item) => `${item.length}:${item}`).join(","); + return `${items.length}[${entries}]`; + } + + /** Constant key used when two code objects compare their co_consts. */ + #constKeyItems(items: readonly MarshalValue[]): string { + const parts: string[] = []; + for (const item of items) { + const key = this.#resolveKey(item, "const"); + const token = key === null ? this.#identity(item) : key.token; + parts.push(`${token.length}:${token}`); + } + return this.#boundedJoin("k", parts, ","); + } + + /** Python string identity is the marshal bytes, not the JavaScript string. */ + #stringKey(raw: Buffer): string { + this.#reserveKey(raw.length * 2 + 8); + return `s${raw.length}:${raw.toString("hex")}`; + } + + #signedZero(value: number): string { + if (Object.is(value, -0)) return "-"; + if (Object.is(value, 0)) return "+"; + return "n"; + } + + /** + * Join a key only after its size fits the string budget and the running + * key-byte budget. Caching the result does not make an oversized key safe. + */ + #boundedJoin( + prefix: string, + parts: readonly string[], + separator: string, + ): string { + let size = prefix.length; + if (parts.length > 0) size += (parts.length - 1) * separator.length; + for (const part of parts) size += part.length; + this.#reserveKey(size); + return prefix + parts.join(separator); + } + + #reserveKey(size: number): void { + if (size > this.input.max_string_bytes) this.#keyTooLarge(size); + this.#keyBytes += size; + if (this.#keyBytes > this.input.max_file_bytes) + throw new PythonPycResourceFailure( + "string", + { max_file_bytes: this.input.max_file_bytes }, + `Marshal key expansion exceeds max_file_bytes ${this.input.max_file_bytes}.`, + ); + } + + #keyTooLarge(size: number): never { + throw new PythonPycResourceFailure( + "string", + { max_string_bytes: this.input.max_string_bytes }, + `Marshal key of ${size} bytes exceeds max_string_bytes ${this.input.max_string_bytes}.`, + ); + } + + #numberKey(value: number): string { + if (value === 0) return "i0"; + if (Number.isFinite(value) && Number.isInteger(value)) + return `i${BigInt(value).toString()}`; + const bytes = Buffer.alloc(8); + bytes.writeDoubleLE(value); + return `d${bytes.toString("hex")}`; + } + + #identity(value: MarshalValue): string { + const existing = this.#identities.get(value); + if (existing !== undefined) return `u${existing}`; + this.#identityNext += 1; + this.#identities.set(value, this.#identityNext); + return `u${this.#identityNext}`; + } + + /** + * CPython stores a long as 15-bit digits, least significant first. Charge + * the digits against the string budget (they never use another one), reject + * the digits CPython rejects, and build the value in one linear pass. + */ + long(): { value: bigint; bits: number } { + const count = this.i32(); + const size = Math.abs(count); + if (size * 2 > this.input.max_string_bytes) + throw new PythonPycResourceFailure( + "string", + { max_string_bytes: this.input.max_string_bytes }, + `Marshal long of ${size * 2} digit bytes exceeds max_string_bytes ${this.input.max_string_bytes}.`, + ); + if (this.offset > this.bytes.length - size * 2) + malformed("Marshal long overruns the selected file."); + const nibbles: number[] = []; + let carry = 0; + let carryBits = 0; + let digit = 0; + for (let index = 0; index < size; index += 1) { + digit = this.u16(); + if (digit >= 0x8000) malformed("Marshal long digit is out of range."); + carry |= digit << carryBits; + carryBits += 15; + while (carryBits >= 4) { + nibbles.push(carry & 0xf); + carry >>>= 4; + carryBits -= 4; + } + } + if (size > 0 && digit === 0) malformed("Marshal long is not normalized."); + if (carryBits > 0) nibbles.push(carry); + let bits = 0; + if (size > 0) { + let top = digit; + while (top > 0) { + top >>>= 1; + bits += 1; + } + bits += (size - 1) * 15; + } + if (nibbles.length === 0) return { value: 0n, bits: 0 }; + const magnitude = BigInt( + `0x${nibbles + .reverse() + .map((nibble) => nibble.toString(16)) + .join("")}`, + ); + return { value: count < 0 ? -magnitude : magnitude, bits }; + } + + /** Marshal version 0 writes a one-byte length and a CPython float literal. */ + textFloat(): number { + const bytes = this.blob(this.u8()); + if (bytes.some((byte) => byte > 0x7f)) + malformed("Marshal float text is not ASCII."); + return ( + parseMarshalFloat(bytes.toString("latin1")) ?? + malformed("Marshal float text is not a CPython float.") + ); + } + + utf8(length: number, ascii: boolean): MarshalValue { + const bytes = this.blob(length); + if (ascii) { + if (bytes.some((byte) => byte > 0x7f)) + malformed("Marshal ASCII string contains a non-ASCII byte."); + return { + kind: "str", + value: bytes.toString("latin1"), + bytes: length, + raw: bytes, + }; + } + return { + kind: "str", + value: decodeSurrogatePass(bytes), + bytes: length, + raw: bytes, + }; + } + + blob(length: number): Buffer { + if (length > this.input.max_string_bytes) + throw new PythonPycResourceFailure( + "string", + { max_string_bytes: this.input.max_string_bytes }, + `Marshal string of ${length} bytes exceeds max_string_bytes ${this.input.max_string_bytes}.`, + ); + if (this.offset > this.bytes.length - length) + malformed("Marshal string overruns the selected file."); + const value = this.bytes.subarray(this.offset, this.offset + length); + this.offset += length; + return Buffer.from(value); + } + + u8(): number { + if (this.offset >= this.bytes.length) + malformed("Marshal stream ended inside an object."); + const value = this.bytes[this.offset]; + this.offset += 1; + if (value === undefined) + return malformed("Marshal stream ended inside an object."); + return value; + } + + u16(): number { + if (this.offset > this.bytes.length - 2) + malformed("Marshal stream ended inside a 16-bit field."); + const value = this.bytes.readUInt16LE(this.offset); + this.offset += 2; + return value; + } + + u32(): number { + if (this.offset > this.bytes.length - 4) + malformed("Marshal stream ended inside a 32-bit field."); + const value = this.bytes.readUInt32LE(this.offset); + this.offset += 4; + return value; + } + + i32(): number { + if (this.offset > this.bytes.length - 4) + malformed("Marshal stream ended inside a 32-bit field."); + const value = this.bytes.readInt32LE(this.offset); + this.offset += 4; + return value; + } + + /** CPython `r_long` count. Negative values are rejected before Evidence. */ + signedCount(field: string): number { + const value = this.i32(); + if (value < 0) malformed(`${field} is negative.`); + return value; + } + + f64(): number { + if (this.offset > this.bytes.length - 8) + malformed("Marshal stream ended inside a float."); + const value = this.bytes.readDoubleLE(this.offset); + this.offset += 8; + return value; + } +} + +/** + * Decode marshal UTF-8 the way CPython does (`surrogatepass`): encoded lone + * surrogates are kept, every other ill-formed sequence is rejected. + */ +const decodeSurrogatePass = (bytes: Buffer): string => { + const units: number[] = []; + let text = ""; + for (let index = 0; index < bytes.length;) { + const lead = bytes[index] ?? 0; + let codePoint: number; + let width: number; + if (lead < 0x80) { + codePoint = lead; + width = 1; + } else if (lead >= 0xc2 && lead <= 0xdf) { + codePoint = lead & 0x1f; + width = 2; + } else if (lead >= 0xe0 && lead <= 0xef) { + codePoint = lead & 0x0f; + width = 3; + } else if (lead >= 0xf0 && lead <= 0xf4) { + codePoint = lead & 0x07; + width = 4; + } else { + return malformed("Marshal string is not valid UTF-8."); + } + if (index + width > bytes.length) + return malformed("Marshal string is not valid UTF-8."); + for (let offset = 1; offset < width; offset += 1) { + const next = bytes[index + offset] ?? 0; + if ((next & 0xc0) !== 0x80) + return malformed("Marshal string is not valid UTF-8."); + codePoint = (codePoint << 6) | (next & 0x3f); + } + if ( + (width === 3 && codePoint < 0x800) || + (width === 4 && (codePoint < 0x10000 || codePoint > 0x10ffff)) + ) + return malformed("Marshal string is not valid UTF-8."); + units.push(codePoint); + index += width; + if (units.length === 4096) { + text += codePointsToString(units); + units.length = 0; + } + } + return text + codePointsToString(units); +}; + +const codePointsToString = (codePoints: readonly number[]): string => + codePoints + .map((codePoint) => + codePoint >= 0xd800 && codePoint <= 0xdfff + ? String.fromCharCode(codePoint) + : String.fromCodePoint(codePoint), + ) + .join(""); + +/** JSON has no non-finite numbers; keep CPython's repr for them. */ +const jsonFloat = (value: number): number | "inf" | "-inf" | "nan" => + Number.isNaN(value) + ? "nan" + : value === Infinity + ? "inf" + : value === -Infinity + ? "-inf" + : value; + +/** + * Emit each distinct code object once, the top-level object first. A child + * shared through TYPE_REF is listed once and referenced by index, so the + * report never repeats a subtree. + */ +const flattenCode = (root: CodeNode): PythonPycCodeObject[] => { + const indexes = new Map(); + const order: CodeNode[] = []; + const visit = (node: CodeNode): void => { + if (indexes.has(node)) return; + indexes.set(node, order.length); + order.push(node); + for (const child of node.nested) visit(child); + }; + visit(root); + return order.map(({ nested, ...code }) => ({ + ...code, + nested_code: nested.map( + (child) => + indexes.get(child) ?? malformed("Nested code object was not listed."), + ), + })); +}; + +const asBytes = (value: MarshalValue, field: string): Buffer => { + if (value.kind !== "bytes") + return malformed(`${field} is not marshal bytes.`); + return value.value; +}; + +const asString = (value: MarshalValue, field: string): string => { + if (value.kind !== "str") + return malformed(`${field} is not a marshal string.`); + return value.value; +}; + +const asTuple = ( + value: MarshalValue, + field: string, +): readonly MarshalValue[] => { + if (value.kind !== "tuple") + return malformed(`${field} is not a marshal tuple.`); + return value.items; +}; + +const strings = (items: readonly MarshalValue[]): string[] => + items.map((item, index) => { + if (item.kind !== "str") + return malformed(`Tuple item ${index} is not a marshal string.`); + return item.value; + }); + +const constSummaries = new WeakMap(); + +/** One summary per decoded object, so TYPE_REF does not repeat the work. */ +const summarizeConst = (value: MarshalValue): PythonPycConst => { + const cached = constSummaries.get(value); + if (cached !== undefined) return cached; + const summary = buildConstSummary(value); + constSummaries.set(value, summary); + return summary; +}; + +const buildConstSummary = (value: MarshalValue): PythonPycConst => { + switch (value.kind) { + case "none": + return { kind: "none" }; + case "bool": + return { kind: "bool", value: value.value }; + case "int": + return { + kind: "int", + bits: value.bits, + value: + value.bits <= PYTHON_PYC_INT_PREVIEW_BITS + ? value.value.toString() + : null, + }; + case "float": + return { kind: "float", value: jsonFloat(value.value) }; + case "str": { + const truncated = value.value.length > PYTHON_PYC_CONST_PREVIEW_LIMIT; + return { + kind: "str", + bytes: value.bytes, + truncated, + value: truncated + ? value.value.slice(0, PYTHON_PYC_CONST_PREVIEW_LIMIT) + : value.value, + }; + } + case "bytes": + return { + kind: "bytes", + bytes: value.value.length, + sha256: sha256(value.value), + }; + case "code": + return { + kind: "code", + name: value.value.name, + firstlineno: value.value.firstlineno, + }; + case "tuple": + return { kind: "tuple", length: value.items.length }; + case "list": + return { kind: "list", length: value.items.length }; + case "set": + return { kind: "set", length: value.items.length }; + case "frozenset": + return { kind: "frozenset", length: value.items.length }; + case "dict": + return { kind: "dict", length: value.items.length }; + case "ellipsis": + return { kind: "ellipsis" }; + case "complex": + return { + kind: "complex", + real: jsonFloat(value.real), + imag: jsonFloat(value.imag), + }; + default: + return { kind: "other", type: value.kind }; + } +}; + +const smallBitLength = (value: bigint): number => { + if (value === 0n) return 0; + const magnitude = value < 0n ? -value : value; + return magnitude.toString(2).length; +}; + +/** + * CPython's `PyOS_string_to_double` grammar for marshal version 0: an optional + * sign, then `inf`/`infinity`, `nan`, or a decimal literal. No hex floats and + * no surrounding whitespace. + */ +const MARSHAL_FLOAT = + /^([+-]?)(?:(inf(?:inity)?)|(nan)|((?:\d+\.\d*|\.\d+|\d+)(?:[eE][+-]?\d+)?))$/i; + +const parseMarshalFloat = (text: string): number | undefined => { + const match = MARSHAL_FLOAT.exec(text); + if (match === null) return undefined; + if (match[2] !== undefined) return match[1] === "-" ? -Infinity : Infinity; + if (match[3] !== undefined) return Number.NaN; + const value = Number(text); + return Number.isNaN(value) ? undefined : value; +}; diff --git a/src/python/PythonPycProvider.ts b/src/python/PythonPycProvider.ts new file mode 100644 index 0000000000..112087996c --- /dev/null +++ b/src/python/PythonPycProvider.ts @@ -0,0 +1,174 @@ +import { bindArtifactFileCleanup } from "../artifacts/ArtifactFileOperation.js"; +import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; +import { readStableArtifact } from "../artifacts/readStableArtifact.js"; +import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; +import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; +import type { PythonPycPort } from "../application/python/PythonPycPort.js"; +import type { ExecutionOptions } from "../application/AnalysisProvider.js"; +import { AnalysisError } from "../domain/analysisErrorBase.js"; +import { + AnalysisAccessDeniedError, + AnalysisArtifactChangedError, + AnalysisCancelledError, + AnalysisInputError, + AnalysisResourceConstraintError, + AnalysisUnsupportedTargetError, +} from "../domain/analysisErrorCore.js"; +import type { InspectPythonPycInput } from "../domain/python/pythonPyc.js"; +import { ProviderAdapterError } from "../domain/providerAdapterError.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import { err, ok } from "../domain/result.js"; +import { + PythonPycFormatFailure, + PythonPycResourceFailure, +} from "./PythonPyc.errors.js"; +import { parsePythonPyc } from "./PythonPycParser.js"; + +/** Bundled reader identity. Header python_version is a separate observed mapping. */ +export const PYTHON_PYC_PROVIDER_IDENTITY = { + id: "rea-python-pyc", + name: "REA CPython pyc parser", + version: "1", +} as const; + +const OPERATION = "inspect_python_pyc"; + +const failureForPrimary = ( + cause: unknown, + input: InspectPythonPycInput, + signal: AbortSignal | undefined, +) => { + if (cause instanceof AnalysisError) return cause; + if (cause instanceof ArtifactReaderFailure && cause.reason === "integrity") + return new AnalysisArtifactChangedError( + OPERATION, + input.path, + cause.message, + { cause }, + ); + if (cause instanceof ArtifactReaderFailure && cause.reason === "cancelled") + return new AnalysisCancelledError(OPERATION, { cause }); + if (signal?.aborted) return new AnalysisCancelledError(OPERATION, { cause }); + if (cause instanceof PythonPycResourceFailure) + return new AnalysisResourceConstraintError( + OPERATION, + cause.boundary === "depth" ? "capacity" : "memory", + cause.message, + cause.reported_limits, + { cause }, + ); + if (cause instanceof PythonPycFormatFailure) + return cause.kind === "unsupported" + ? new AnalysisUnsupportedTargetError( + OPERATION, + input.path, + cause.message, + { cause }, + ) + : new AnalysisInputError(OPERATION, { cause }, [ + { + path: ["path"], + reason: "invalid_format", + message: `Selected pyc ${input.path}: ${cause.message}`, + }, + ]); + if (cause instanceof ArtifactReaderFailure) { + if (cause.reason === "limit") + return new AnalysisResourceConstraintError( + OPERATION, + "file-size", + cause.message, + { max_file_bytes: input.max_file_bytes }, + { cause }, + ); + if (cause.reason === "path" || cause.reason === "format") + return new AnalysisInputError(OPERATION, { cause }, [ + { + path: ["path"], + reason: "invalid_format", + message: cause.message, + }, + ]); + } + if (cause instanceof Error && "code" in cause) { + if (cause.code === "EACCES" || cause.code === "EPERM") + return new AnalysisAccessDeniedError(OPERATION, input.path, cause.code, { + cause, + }); + if ( + ["ENOENT", "ENOTDIR", "ELOOP", "ENAMETOOLONG"].includes( + String(cause.code), + ) + ) + return new AnalysisInputError(OPERATION, { cause }, [ + { + path: ["path"], + reason: "invalid_value", + message: `Selected pyc could not be read (${String(cause.code)}): ${input.path}.`, + }, + ]); + } + return new ProviderAdapterError(PYTHON_PYC_PROVIDER_IDENTITY.id, OPERATION, { + cause, + diagnostics: { + path: input.path, + reason: cause instanceof Error ? cause.message : String(cause), + }, + }); +}; + +const failureFor = ( + cause: unknown, + input: InspectPythonPycInput, + signal: AbortSignal | undefined, +) => { + const primary = failureForPrimary(cause, input, signal); + if (!(cause instanceof ArtifactReaderFailure) || cause.cleanup === undefined) + return primary; + const cleanup = new ProviderCleanupError( + PYTHON_PYC_PROVIDER_IDENTITY.id, + cause.cleanup.resources, + { reason: cause.cleanup.reason }, + { operation: OPERATION, cause }, + ); + const error = analysisErrorWithCleanupFailure(primary, cleanup, OPERATION); + bindArtifactFileCleanup(cause, error); + return error; +}; + +/** Portable parser over one bounded, identity-checked regular file. */ +export class PythonPycProvider implements PythonPycPort { + readonly identity = PYTHON_PYC_PROVIDER_IDENTITY; + readonly #resources = new ArtifactResourceScope(); + + close(): Promise { + return this.#resources.close(); + } + + async inspect(input: InspectPythonPycInput, options?: ExecutionOptions) { + try { + const snapshot = await this.#resources.run(() => + readStableArtifact( + input.path, + input.max_file_bytes, + options?.signal, + undefined, + this.#resources, + ), + ); + return ok( + parsePythonPyc( + snapshot.bytes, + { + path: input.path, + sha256: snapshot.sha256, + bytes: snapshot.bytes.length, + }, + input, + ), + ); + } catch (cause) { + return err(failureFor(cause, input, options?.signal)); + } + } +} diff --git a/src/server/createServer.ts b/src/server/createServer.ts index 902fe56aa9..b76b367d89 100644 --- a/src/server/createServer.ts +++ b/src/server/createServer.ts @@ -52,6 +52,9 @@ import { registerIl2cppTools } from "./registerIl2cppTools.js"; import { createRustBinaryService } from "../composition/rust.js"; import type { RustBinaryService } from "../application/rust/RustBinaryService.js"; import { registerRustTools } from "./registerRustTools.js"; +import { createPythonPycService } from "../composition/pythonPyc.js"; +import type { PythonPycService } from "../application/python/PythonPycService.js"; +import { registerPythonPycTool } from "./registerPythonPycTool.js"; import { createBeamService } from "../composition/erlang.js"; import type { BeamService } from "../application/erlang/BeamService.js"; import { registerErlangTools } from "./registerErlangTools.js"; @@ -196,6 +199,7 @@ export interface CreateServerOptions { readonly goBinary?: GoBinaryService; readonly il2cppMetadata?: Il2cppMetadataService; readonly rustBinary?: RustBinaryService; + readonly pythonPyc?: PythonPycService; readonly erlangBeam?: BeamService; readonly electronFuses?: ElectronFuseService; readonly recordedCrash?: RecordedCrashService; @@ -370,6 +374,7 @@ export const createServer = ( const elfNotes = createElfNotesService(); const peCertificates = createPeCertificatesService(); const electronFuses = options.electronFuses ?? createElectronFuseService(); + const pythonPyc = options.pythonPyc ?? createPythonPycService(); const webNetworkCapture = options.webNetworkCapture ?? createWebNetworkCaptureService(environment); const javascriptRecovery = @@ -475,6 +480,7 @@ export const createServer = ( elfNotes.close(), peCertificates.close(), electronFuses.close(), + pythonPyc.close(), webNetworkCapture.close(), javascriptRecovery.close?.() ?? Promise.resolve(), ...(options.optionalObservationProviderOwner === "caller" @@ -542,6 +548,7 @@ export const createServer = ( elfNotes, peCertificates, electronFuses, + pythonPyc, webNetworkCapture, }; registerDesktopApplicationTool( @@ -776,6 +783,7 @@ const registerConfiguredAnalysisTools = ( nativeExports, elfNotes, peCertificates, + pythonPyc, webNetworkCapture, artifactResources, }: ServerToolContext, @@ -870,6 +878,13 @@ const registerConfiguredAnalysisTools = ( withAdmittedAnalysis, evidenceById, ); + registerPythonPycTool( + server, + pythonPyc, + toolLogger, + recordEvidence, + withAdmittedAnalysis, + ); registerErlangTools( server, erlangBeam, @@ -991,6 +1006,7 @@ interface ServerToolContext extends ReturnType { readonly elfNotes: ElfNotesService; readonly peCertificates: PeCertificatesService; readonly electronFuses: ElectronFuseService; + readonly pythonPyc: PythonPycService; readonly webNetworkCapture: WebNetworkCaptureService; readonly artifactResources: ArtifactResourceScope; readonly server: EvidenceMcpServer; diff --git a/src/server/registerPythonPycTool.ts b/src/server/registerPythonPycTool.ts new file mode 100644 index 0000000000..f16e017f77 --- /dev/null +++ b/src/server/registerPythonPycTool.ts @@ -0,0 +1,31 @@ +import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; +import type { PythonPycService } from "../application/python/PythonPycService.js"; +import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; +import { toolContract } from "../contracts/toolContracts.js"; +import type { Logger } from "pino"; +import { createEvidenceToolHandler } from "./contractToolHandler.js"; +import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; +import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; + +/** Register the explicit-file CPython pyc workflow and retain its complete Evidence. */ +export const registerPythonPycTool = ( + server: EvidenceMcpServer, + service: PythonPycService, + logger: Logger, + recordEvidence?: EvidenceWriter["recordEvidence"], + withAdmittedAnalysis?: WithAdmittedAnalysis, +): void => { + const contract = toolContract("inspect_python_pyc"); + const handler = createEvidenceToolHandler( + server, + (input, options) => service.inspect(input, options), + logger, + recordEvidence, + withAdmittedAnalysis, + ); + server.registerTool( + contract.name, + toolRegistrationOptions(contract), + handler(contract), + ); +}; diff --git a/tests/acceptance/pythonPycCli.test.ts b/tests/acceptance/pythonPycCli.test.ts new file mode 100644 index 0000000000..f10e78ce16 --- /dev/null +++ b/tests/acceptance/pythonPycCli.test.ts @@ -0,0 +1,79 @@ +import { mkdir, symlink, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; +import { toolContract } from "../../src/contracts/toolContracts.js"; +import { pythonPycFixture } from "../../src/python/PythonPyc.fixture.js"; +import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js"; +import { cliTest } from "../support/cli/cliFixture.js"; + +cliTest( + "reports nested CPython code objects through the public CLI", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-python-pyc-cli-"); + const path = join(root, "sample.pyc"); + await writeFile(path, pythonPycFixture("py311", { nested: true })); + const result = await cli.run({ + arguments: ["inspect-python-pyc", path, "--json"], + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = toolContract("inspect_python_pyc").outputSchema.parse( + result.json, + ); + expect(evidence.normalized_result.header.python_version).toBe("3.12"); + expect(evidence.normalized_result.code_objects[1]?.name).toBe("inner"); + }, +); + +cliTest( + "resolves a relative pyc path against the working directory", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-python-pyc-cli-rel-"); + await writeFile(join(root, "sample.pyc"), pythonPycFixture("py37")); + const result = await cli.run({ + arguments: ["inspect-python-pyc", "sample.pyc", "--json"], + cwd: root, + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = toolContract("inspect_python_pyc").outputSchema.parse( + result.json, + ); + expect(evidence.normalized_result.header.python_version).toBe("3.7"); + expect(evidence.subject?.local_path).toBe(join(root, "sample.pyc")); + }, +); + +cliTest.skipIf(process.platform === "win32")( + "reads the file the symlink path names, not its lexical sibling", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-python-pyc-cli-link-"); + const target = join(root, "target"); + await mkdir(join(target, "child"), { recursive: true }); + await writeFile(join(target, "sample.pyc"), pythonPycFixture("py37")); + await writeFile(join(root, "sample.pyc"), pythonPycFixture("py311")); + await symlink(join(target, "child"), join(root, "link")); + const result = await cli.run({ + arguments: ["inspect-python-pyc", "link/../sample.pyc", "--json"], + cwd: root, + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = toolContract("inspect_python_pyc").outputSchema.parse( + result.json, + ); + expect(evidence.normalized_result.header.python_version).toBe("3.7"); + expect(evidence.subject?.local_path).toBe(`${root}/link/../sample.pyc`); + }, +); + +cliTest("reports a truncated pyc as invalid input", async ({ cli }) => { + const root = await createTestTempDirectory("rea-python-pyc-cli-bad-"); + const path = join(root, "broken.pyc"); + await writeFile(path, Buffer.from("not a pyc")); + const result = await cli.run({ + arguments: ["inspect-python-pyc", path, "--json"], + }); + expect(result.exitCode).toBe(1); + expect(result.json).toMatchObject({ + code: "invalid_request", + details: { operation: "inspect_python_pyc" }, + }); +}); diff --git a/tests/acceptance/tools/localToolContracts.test.ts b/tests/acceptance/tools/localToolContracts.test.ts index 422a502125..6cbb5d9c93 100644 --- a/tests/acceptance/tools/localToolContracts.test.ts +++ b/tests/acceptance/tools/localToolContracts.test.ts @@ -221,6 +221,7 @@ const pathRequests: readonly [string, Record][] = [ ["inspect_pe_certificates", { path: nul }], ["inspect_native_exports", { path: nul }], ["inspect_elf_notes", { path: nul }], + ["inspect_python_pyc", { path: nul }], ["inspect_erlang_beam", { path: nul }], ["inspect_binary_layout", { path: nul }], ["inspect_recorded_crash", { path: nul }], diff --git a/tests/boundary/filesystem/pythonPyc.test.ts b/tests/boundary/filesystem/pythonPyc.test.ts new file mode 100644 index 0000000000..4937bb027c --- /dev/null +++ b/tests/boundary/filesystem/pythonPyc.test.ts @@ -0,0 +1,60 @@ +import { symlink, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createPythonPycService } from "../../../src/composition/pythonPyc.js"; +import { pythonPycFixture } from "../../../src/python/PythonPyc.fixture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +describe("Python pyc stable filesystem boundary", () => { + it("reports non-file and malformed pyc selections as caller input errors", async () => { + const root = await createTestTempDirectory("rea-python-pyc-files-"); + const path = join(root, "sample.pyc"); + const alias = join(root, "alias.pyc"); + const malformed = join(root, "malformed.pyc"); + await writeFile(path, pythonPycFixture()); + await writeFile(malformed, Buffer.from("not a pyc")); + await symlink(path, alias); + const service = createPythonPycService(); + for (const [candidate, reason] of [ + [root, "invalid_format"], + [alias, "invalid_format"], + [malformed, "invalid_format"], + [join(root, "missing.pyc"), "invalid_value"], + [join(path, "child.pyc"), "invalid_value"], + ] as const) { + const result = await service.inspect({ path: candidate }); + if (result.ok) throw new Error("Expected file acquisition failure"); + expect(result.error).toMatchObject({ + _tag: "AnalysisInputError", + operation: "inspect_python_pyc", + issues: [ + { + path: ["path"], + reason, + message: expect.any(String), + }, + ], + }); + } + }); + + it("returns typed resource and cancellation failures", async () => { + const root = await createTestTempDirectory("rea-python-pyc-budget-"); + const path = join(root, "sample.pyc"); + const bytes = pythonPycFixture("py311", { nested: true }); + await writeFile(path, bytes); + const service = createPythonPycService(); + const tooSmall = await service.inspect({ path, max_file_bytes: 16 }); + if (tooSmall.ok) throw new Error("Expected resource budget failure"); + expect(tooSmall.error._tag).toBe("AnalysisResourceConstraintError"); + const tooFewObjects = await service.inspect({ path, max_objects: 1 }); + if (tooFewObjects.ok) throw new Error("Expected object budget failure"); + expect(tooFewObjects.error._tag).toBe("AnalysisResourceConstraintError"); + const cancelled = await service.inspect( + { path }, + { signal: AbortSignal.abort() }, + ); + if (cancelled.ok) throw new Error("Expected cancellation"); + expect(cancelled.error._tag).toBe("AnalysisCancelledError"); + }); +}); diff --git a/tests/boundary/mcp/pythonPycMcp.test.ts b/tests/boundary/mcp/pythonPycMcp.test.ts new file mode 100644 index 0000000000..07d13da005 --- /dev/null +++ b/tests/boundary/mcp/pythonPycMcp.test.ts @@ -0,0 +1,88 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; +import { toolContract } from "../../../src/contracts/toolContracts.js"; +import { createServer } from "../../../src/server/createServer.js"; +import { createTestBinarySession } from "../../fixtures/binarySession.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; +import { pythonPycFixture } from "../../../src/python/PythonPyc.fixture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { mcpTest, type TestMcp } from "../../support/mcp/mcpFixture.js"; + +const connect = async (mcp: TestMcp) => { + const session = createTestBinarySession(() => { + throw new Error("Python pyc inspection must not start a deep provider"); + }); + const client = await mcp.connect(createServer({ kind: "session", session })); + return { client, session }; +}; + +mcpTest( + "advertises inspect_python_pyc and reads nested code objects", + async ({ mcp }) => { + const { client, session } = await connect(mcp); + try { + const advertised = (await client.listTools()).tools.find( + (tool) => tool.name === "inspect_python_pyc", + ); + expect(advertised?.annotations).toMatchObject({ + readOnlyHint: false, + idempotentHint: true, + }); + const root = await createTestTempDirectory("rea-python-pyc-mcp-"); + const path = join(root, "sample.pyc"); + await writeFile(path, pythonPycFixture("py311", { nested: true })); + const response = await client.callTool({ + name: "inspect_python_pyc", + arguments: { path }, + }); + expect(response.isError, JSON.stringify(response.content)).not.toBe(true); + const evidence = toolContract("inspect_python_pyc").outputSchema.parse( + response.structuredContent, + ); + expect(evidence.normalized_result.header.python_version).toBe("3.12"); + expect(evidence.normalized_result.code_objects[1]?.name).toBe("inner"); + } finally { + await session.close(); + } + }, +); + +mcpTest("requires an absolute path for inspect_python_pyc", async ({ mcp }) => { + const { client, session } = await connect(mcp); + try { + const response = await client.callTool({ + name: "inspect_python_pyc", + arguments: { path: "sample.pyc" }, + }); + expect(response.isError).toBe(true); + expect(parseMcpToolError(response).error).toMatchObject({ + code: "invalid_request", + details: { operation: "inspect_python_pyc" }, + }); + } finally { + await session.close(); + } +}); + +mcpTest("reports an unknown pyc magic as unsupported", async ({ mcp }) => { + const { client, session } = await connect(mcp); + try { + const root = await createTestTempDirectory("rea-python-pyc-mcp-magic-"); + const path = join(root, "unknown.pyc"); + const bytes = pythonPycFixture("py311"); + bytes.writeUInt16LE(1, 0); + await writeFile(path, bytes); + const response = await client.callTool({ + name: "inspect_python_pyc", + arguments: { path }, + }); + expect(response.isError).toBe(true); + expect(parseMcpToolError(response).error).toMatchObject({ + code: "unsupported_target", + details: { operation: "inspect_python_pyc" }, + }); + } finally { + await session.close(); + } +}); diff --git a/tests/process-global/stableArtifactConsumers.test.ts b/tests/process-global/stableArtifactConsumers.test.ts index 0f7b471f7d..7e442f00df 100644 --- a/tests/process-global/stableArtifactConsumers.test.ts +++ b/tests/process-global/stableArtifactConsumers.test.ts @@ -78,6 +78,8 @@ import { import { ArtifactResourceScope } from "../../src/artifacts/ArtifactResourceScope.js"; import { ArtifactReaderFailure } from "../../src/artifacts/ArtifactReader.js"; import { GoBinaryProvider } from "../../src/go/GoBinaryProvider.js"; +import { PythonPycProvider } from "../../src/python/PythonPycProvider.js"; +import { inspectPythonPycInputSchema } from "../../src/domain/python/pythonPyc.js"; import { PeResourcesProvider } from "../../src/native/pe/PeResourcesProvider.js"; import { Il2cppMetadataProvider } from "../../src/il2cpp/Il2cppMetadataProvider.js"; import { ElfNotesProvider } from "../../src/native/elf/ElfNotesProvider.js"; @@ -303,6 +305,30 @@ it.each(staticProviders)( } }, ); +it("clears the Python pyc cleanup flag once a retained descriptor closes", async () => { + const path = await selectedImage(); + const provider = new PythonPycProvider(); + const primary = new ArtifactReaderFailure( + "integrity", + "controlled source replacement", + ); + fault.readFailure = { cause: primary }; + try { + const result = await provider.inspect( + inspectPythonPycInputSchema.parse({ path }), + ); + if (result.ok) + throw new Error("Expected primary failure and retained descriptor"); + expect(result.error.cleanupIncomplete).toBe(true); + fault.blocked.clear(); + await provider.close(); + expect(result.error.cleanupIncomplete).toBe(false); + } finally { + fault.blocked.clear(); + await provider.close(); + } +}); + afterEach(async () => { fault.blockNew = false; fault.blocked.clear();