diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md index baea9f529b..53ce105a81 100644 --- a/.agents/skills/reverse-engineer-anything/SKILL.md +++ b/.agents/skills/reverse-engineer-anything/SKILL.md @@ -363,7 +363,7 @@ explicit path or endpoint and do not need it. `CONFIG_IKCONFIG`. Non-gzip payloads are unsupported; decompress externally, then inspect the result. See the [kernel build config guide](https://github.com/morluto/rea/blob/main/docs/guides/native/kernel-build-config.md). -- CPython 3.7–3.13 `.pyc` header, nested marshal code objects and bytecode +- CPython 3.7–3.15 `.pyc` header, nested marshal code objects and bytecode identity: `inspect_python_pyc` with the explicit local file. This built-in target-free inspection requires no Python installation. Preserve magic, PEP 552 hash or mtime/size, nested names, and bytecode SHA-256. Bytecode is diff --git a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md index 151f38ad1c..ab90dd1557 100644 --- a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md +++ b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md @@ -296,7 +296,7 @@ external utility, then call `inspect_kernel_build_config` on the result. See ## CPython pyc files -Use `inspect_python_pyc` for an explicit local CPython 3.7–3.13 `.pyc` when the +Use `inspect_python_pyc` for an explicit local CPython 3.7–3.15 `.pyc` when the question concerns the header magic, PEP 552 source hash or timestamp, nested code-object names, or bytecode identity. No `open_binary`, Python installation, or target execution is required. The CLI equivalent is `inspect-python-pyc PATH`. diff --git a/docs/getting-started/cli.md b/docs/getting-started/cli.md index 730dccc211..dedec3ea8b 100644 --- a/docs/getting-started/cli.md +++ b/docs/getting-started/cli.md @@ -531,8 +531,7 @@ is inflated and searched; other payload codecs are unsupported and name `external_decompression` as the next probe. See [Linux kernel build config](../guides/native/kernel-build-config.md). -Inspect a CPython 3.7–3.13 `.pyc` header and nested marshal code objects (3.14 and -3.15 headers are qualified but their marshal layout is not pinned) +Inspect a CPython 3.7–3.15 `.pyc` header and nested marshal code objects without executing bytecode or installing Python: ```bash diff --git a/docs/python-pyc.md b/docs/python-pyc.md index c5b69976ee..05fe465999 100644 --- a/docs/python-pyc.md +++ b/docs/python-pyc.md @@ -80,15 +80,15 @@ The inline Evidence result contains the artifact path, SHA-256, and byte length. `header` contains: - `magic`, the 16-bit little-endian field; -- `python_version` and `code_layout`, mapped from the pinned CPython 3.7–3.13 - marshal table (`py37`, `py38`, or `py311`); 3.11 alpha magics before 3494 used - other code layouts and are unsupported. The 3.7.0a1/a2 magics 3390/3391 - used 12-byte headers and are outside this reader's supported format. The - identity table also recognizes 3.14 (3627) and 3.15 (3666): those magics - carry the same packed 16-byte header, so the typed refusal names the observed - version and flags, and says that no marshal code layout is pinned for it. - Without a pinned layout the code objects cannot be decoded, so a recognized - version is still not a successful decode; +- `python_version` and `code_layout`, mapped from the pinned CPython marshal + table (`py37`, `py38`, or `py311`): 3.7–3.13, plus three 3.14 and 3.15 + magics: 3627 (3.14.4) and 3663 (3.15.0a8), both read back from real + interpreters, and 3666 (3.15.2), checked with real 3.15.2 release bytes + decoded by the 3.15.0a8 interpreter (3.15.2 itself was not run). Other 3.14 + and 3.15 pre-release magics are not pinned and are unsupported. 3.11 alpha + magics before 3494 used other code layouts and are unsupported. The + 3.7.0a1/a2 magics 3390/3391 used 12-byte headers and are outside this + reader's supported format; - `flags`, `hash_based`, and `check_source` from PEP 552; - `source_hash_hex` for hash-based pycs, or `source_mtime` and `source_size` for timestamp pycs. @@ -108,6 +108,12 @@ filename, the signed 32-bit `co_firstlineno` value, argument counts, flags, names, locals (or `localsplusnames` on 3.11+), const summaries, and bytecode length plus SHA-256. +CPython 3.14 marshals folded slice subscripts such as `a[1:10:2]` as slice +constants. They are summarized as `slice`, walked for nested code like other +containers, and compare as CPython slices do: through their start, stop, and +step, never equal to a tuple. A slice tag in a pyc older than magic 3608 is +malformed, as it is for CPython. + Code-object equality used by sets and dictionaries follows the selected CPython format: 3.7–3.10 derive `CO_NOFREE` from the free/cell tables for equality, while 3.11+ compare raw flags; raw `co_flags` remain in reported facts. The older @@ -187,9 +193,9 @@ against that same object budget. Integer digits and expanded set or dictionary keys, including a shared tuple repeated at each level, count against the string budget. Magics outside the pinned identity table, truncated headers, trailing marshal bytes, and a non-code top-level object are typed -failures. A recognized 3.14/3.15 magic is a typed unsupported failure too: the -header is qualified from its packed 16-byte fields, but no marshal code layout is -pinned, so no code object is reported and the result is never partial success. +failures. A recognized magic with no pinned marshal code layout is a typed +unsupported failure too: the header is qualified from its packed 16-byte +fields, but no code object is reported and the result is never partial success. A decoded header does not prove the matching `.py` is present. `python_version` is the magic table, not a running interpreter. This workflow does not recover diff --git a/src/cli/pythonPycCommands.ts b/src/cli/pythonPycCommands.ts index d9c4a7659a..0b7dbf23e1 100644 --- a/src/cli/pythonPycCommands.ts +++ b/src/cli/pythonPycCommands.ts @@ -24,7 +24,7 @@ export const registerPythonPycCommands = ( ): void => { cli.command(CLI_COMMANDS.inspectPythonPyc, { description: - "Inspect a supported CPython 3.7–3.13 pyc header and marshal code objects without executing bytecode", + "Inspect a supported CPython 3.7–3.15 pyc header and marshal code objects without executing bytecode", args: z.object({ path: localPathStringSchema.describe("Local CPython .pyc file"), }), diff --git a/src/contracts/python/pythonPycToolContracts.ts b/src/contracts/python/pythonPycToolContracts.ts index 3056ccb70d..71f85fec14 100644 --- a/src/contracts/python/pythonPycToolContracts.ts +++ b/src/contracts/python/pythonPycToolContracts.ts @@ -31,7 +31,7 @@ export const PYTHON_PYC_TOOL_CONTRACTS = [ kind: "artifact-provider", requiresActiveTarget: true, description: - "Inspect local CPython 3.7–3.13 .pyc headers and nested code objects without executing bytecode, invoking Python or recovering source. Returns pinned version/header fields, names, constant summaries and bytecode lengths/digests; nested_code links code constants, not runtime calls. Default complete returns full Evidence; detail summary retains it for inspect_analysis_view kind python-pyc, collection code_objects, with exact names/indices/fields and paging. source_index preserves same-name alternatives. Unsupported layouts, truncation and exhausted budgets fail without partial success. Recognized 3.14/3.15 headers receive a version/flags-qualified refusal, not decoded code objects. A header does not prove matching .py source exists.", + "Inspect local CPython 3.7–3.15 .pyc headers and nested code objects without executing bytecode, invoking Python or recovering source. Returns pinned version/header fields, names, constant summaries and bytecode lengths/digests; nested_code links code constants, not runtime calls. Default complete returns full Evidence; detail summary retains it for inspect_analysis_view kind python-pyc, collection code_objects, with exact names/indices/fields and paging. source_index preserves same-name alternatives. Unsupported magics/layouts, truncation and exhausted budgets fail without partial success. A header does not prove matching .py source exists.", inputSchema: inspectPythonPycRequestSchema, outputSchema: analysisDetailOutputSchemaOf( pythonPycSchema, diff --git a/src/domain/python/pythonBytecodeIdentity.ts b/src/domain/python/pythonBytecodeIdentity.ts index dd97920cb9..815953e10b 100644 --- a/src/domain/python/pythonBytecodeIdentity.ts +++ b/src/domain/python/pythonBytecodeIdentity.ts @@ -27,7 +27,8 @@ const PYTHON_BYTECODE_IDENTITIES: ReadonlyMap = ...identities(range(3550, 3571), "3.13"), ...identities([3627], "3.14"), // Reported against Python 3.15, the default `python3` on current Homebrew. - ...identities([3666], "3.15"), + // 3663 is the 3.15.0a8 magic, read back from a real interpreter. + ...identities([3663, 3666], "3.15"), ]); export const pythonBytecodeIdentity = ( diff --git a/src/domain/python/pythonPyc.test.ts b/src/domain/python/pythonPyc.test.ts index 1435d3ff97..d5e19ee62b 100644 --- a/src/domain/python/pythonPyc.test.ts +++ b/src/domain/python/pythonPyc.test.ts @@ -76,17 +76,30 @@ it("keeps pinned CPython identity separate from standalone marshal coverage", () code_layout: "py311", negative_code_fields: "rejected", }); - // Recognizing a newer CPython identity does not authorize its code layout. expect(pythonBytecodeIdentity(3627)).toEqual({ pythonVersion: "3.14", headerBytes: 16, }); + expect(pythonPycMarshalProfile(3627)).toEqual({ + code_layout: "py311", + negative_code_fields: "rejected", + }); expect(pythonBytecodeIdentity(3666)).toEqual({ pythonVersion: "3.15", headerBytes: 16, }); - expect(pythonPycMarshalProfile(3627)).toBeNull(); - expect(pythonPycMarshalProfile(3666)).toBeNull(); + expect(pythonBytecodeIdentity(3663)).toEqual({ + pythonVersion: "3.15", + headerBytes: 16, + }); + expect(pythonPycMarshalProfile(3663)?.code_layout).toBe("py311"); + expect(pythonPycMarshalProfile(3666)?.code_layout).toBe("py311"); + // Only magics checked against real CPython are pinned: the neighbouring + // 3.14 and 3.15 pre-release magics and the 3.16 magics stay unknown. + for (const magic of [3600, 3626, 3650, 3662, 3664, 3665, 3706]) { + expect(pythonBytecodeIdentity(magic), String(magic)).toBeNull(); + expect(pythonPycMarshalProfile(magic), String(magic)).toBeNull(); + } expect(pythonPycMarshalProfile(3450)).toBeNull(); expect(pythonPycMarshalProfile(62171)).toBeNull(); expect(PYTHON_PYC_MARSHAL_PROFILES.size).toBeGreaterThan(80); diff --git a/src/domain/python/pythonPyc.ts b/src/domain/python/pythonPyc.ts index f9ac4f3625..cc85636b01 100644 --- a/src/domain/python/pythonPyc.ts +++ b/src/domain/python/pythonPyc.ts @@ -8,6 +8,8 @@ import { export const PYTHON_PYC_HEADER_BYTES = 16; export const PYTHON_PYC_HASH_BASED = 0x1; export const PYTHON_PYC_CHECK_SOURCE = 0x2; +/** CPython 3.14a1 magic 3608 added TYPE_SLICE to marshal (gh-125063). */ +export const PYTHON_PYC_SLICE_MAGIC = 3608; const uint = z.number().int().min(0).max(0xffffffff); const index = z.number().int().nonnegative(); @@ -59,6 +61,13 @@ export const PYTHON_PYC_MARSHAL_PROFILES: ReadonlyMap< ...consecutiveProfiles(3494, 3495, "py311"), ...consecutiveProfiles(3500, 3531, "py311"), ...consecutiveProfiles(3550, 3571, "py311"), + // 3.14 and 3.15 keep the 3.11 code layout; 3.14a1 magic 3608 adds slice + // constants to marshal, which the reader gates by magic. Only magics checked + // against real CPython are pinned: 3627 (3.14.4) and 3663 (3.15.0a8) were + // read back from those interpreters; 3666 (3.15.2) was checked with real + // 3.15.2 release bytes decoded by the 3.15.0a8 interpreter. Other + // pre-release magics stay unpinned. + ...profiles([3627, 3663, 3666], "py311"), ]); export const pythonPycMarshalProfile = ( @@ -160,6 +169,7 @@ const pythonConstSchema = z.discriminatedUnion("kind", [ z.strictObject({ kind: z.literal("frozenset"), length: uint }), z.strictObject({ kind: z.literal("dict"), length: uint }), z.strictObject({ kind: z.literal("ellipsis") }), + z.strictObject({ kind: z.literal("slice") }), z.strictObject({ kind: z.literal("complex"), real: pythonFloatSchema, @@ -253,7 +263,7 @@ export const pythonPycHeaderSchema = z }); }); -/** Bounded CPython 3.7–3.13 pyc observation; marshal is walked, bytecode is not decompiled. */ +/** Bounded CPython 3.7–3.15 pyc observation; marshal is walked, bytecode is not decompiled. */ export const pythonPycSchema = z .strictObject({ artifact: z.strictObject({ diff --git a/src/python/PythonPycBytecodeEquality.test.ts b/src/python/PythonPycBytecodeEquality.test.ts new file mode 100644 index 0000000000..d5b6a393cb --- /dev/null +++ b/src/python/PythonPycBytecodeEquality.test.ts @@ -0,0 +1,21 @@ +import { createHash } from "node:crypto"; +import { expect, it } from "vitest"; + +import { pythonPycBytecodeIdentity } from "./PythonPycBytecodeEquality.js"; + +const raw = (bytes: Buffer): string => + createHash("sha256").update(bytes).digest("hex"); +const identity = (bytes: Buffer, magic: number): string => + pythonPycBytecodeIdentity(bytes, magic, raw(bytes)); + +it("skips inline-cache units only under the pinned 3.14 and 3.15 magics", () => { + // Opcode 80 carries nine cache code units in the 3627 table; only the cache + // bytes differ between the two bytecodes. + const first = Buffer.from([80, 0, ...Array(18).fill(1), 83, 0]); + const second = Buffer.from([80, 0, ...Array(18).fill(9), 83, 0]); + expect(identity(first, 3627)).toBe(identity(second, 3627)); + // A magic with no pinned table keeps the raw bytecode digest, so code + // objects that differ only in cache units stay distinct. + expect(identity(first, 3619)).toBe(`${String(first.length)}:${raw(first)}`); + expect(identity(first, 3619)).not.toBe(identity(second, 3619)); +}); diff --git a/src/python/PythonPycBytecodeEquality.ts b/src/python/PythonPycBytecodeEquality.ts index a2eae7b73f..8c370b2060 100644 --- a/src/python/PythonPycBytecodeEquality.ts +++ b/src/python/PythonPycBytecodeEquality.ts @@ -40,6 +40,16 @@ const inlineCacheProfiles = ( * profile only where their source changes do not alter cache widths or IDs. * Final 3.13 magic 3571 matches the opcode IDs and cache widths in CPython * v3.13.17's Lib/_opcode_metadata.py and Lib/opcode.py. + * The 3.14 and 3.15 tables are pinned only to magics checked against real + * CPython: 3627 (final 3.14, matches v3.14.4) and 3663 (3.15.0a8) were read + * back from those builds; 3666 (final 3.15, matches v3.15.0) was checked with + * real 3.15.2 release bytes, not by running 3.15.2. Each is the generated + * _PyOpcode_Caches and opmap from CPython's Include/internal/pycore_magic_number.h + * history at the last cache-changing bump commit at or before it: + * 60202609a2c2d0971aadfa4729ba30b50e89c6ea (3623, shared by 3624 to 3627), + * 879c85f6e4ca8424ff6b4fddfbc946889572bf13 (3663) and + * 600f4dbd54bb35f56ecf1430134d2072b0520aa3 (3665, shared by 3666). Other 3.14 + * and 3.15 pre-release magics have no table here and are not decoded. */ const INLINE_CACHE_ENTRIES_BY_MAGIC: ReadonlyMap< number, @@ -416,6 +426,83 @@ const INLINE_CACHE_ENTRIES_BY_MAGIC: ReadonlyMap< [117, 1], ], ), + ...inlineCacheProfiles( + [3627], + [ + [38, 1], + [39, 3], + [44, 5], + [52, 3], + [55, 3], + [56, 1], + [57, 1], + [70, 1], + [75, 1], + [80, 9], + [92, 4], + [96, 1], + [100, 1], + [101, 1], + [102, 1], + [103, 1], + [106, 1], + [110, 4], + [119, 1], + ], + ), + ...inlineCacheProfiles( + [3663], + [ + [4, 1], + [36, 1], + [37, 3], + [42, 5], + [50, 3], + [53, 3], + [54, 1], + [55, 1], + [68, 1], + [74, 1], + [79, 9], + [91, 4], + [95, 1], + [99, 1], + [100, 1], + [101, 1], + [102, 1], + [105, 1], + [109, 4], + [118, 1], + [128, 1], + ], + ), + ...inlineCacheProfiles( + [3666], + [ + [4, 1], + [36, 1], + [37, 3], + [42, 5], + [50, 3], + [53, 3], + [54, 1], + [55, 1], + [68, 1], + [70, 1], + [74, 1], + [79, 9], + [91, 4], + [95, 1], + [99, 1], + [100, 1], + [101, 1], + [102, 1], + [105, 1], + [109, 4], + [118, 1], + [128, 1], + ], + ), ]); /** Preserve CPython's bytecode-size check and omit only cache code units. */ diff --git a/src/python/PythonPycParser.test.ts b/src/python/PythonPycParser.test.ts index 2c37f97a28..e06ef1a274 100644 --- a/src/python/PythonPycParser.test.ts +++ b/src/python/PythonPycParser.test.ts @@ -139,6 +139,8 @@ it("rejects truncated marshal without partial success", () => { it.each([ { magic: 3571, layout: "py311", version: "3.13" }, + { magic: 3627, layout: "py311", version: "3.14" }, + { magic: 3666, layout: "py311", version: "3.15" }, { magic: 3401, layout: "py37", version: "3.8" }, ] as const)( "decodes supported standalone magic $magic with its own marshal layout", @@ -155,44 +157,14 @@ it.each([ }, ); -it.each([ - { magic: 3627, version: "3.14", flags: 0x0, mode: "timestamp-based" }, - { magic: 3666, version: "3.15", flags: 0x0, mode: "timestamp-based" }, - // A non-zero flag word and a PEP 552 hash-based header, so a refusal that hardcoded - // `flags 0x0, timestamp-based` could not pass. - { magic: 3666, version: "3.15", flags: 0x3, mode: "hash-based" }, - { magic: 3627, version: "3.14", flags: 0x1, mode: "hash-based" }, -] as const)( - "qualifies the recognized CPython $version header without a pinned code layout", - ({ magic, version, flags, mode }) => { - // The header is still a packed 16-byte 3.7+ header, so its identity and flags are - // reported even though no marshal layout is pinned for this version. - const header = pythonPycHeader(magic, { - flags, - ...(flags === 0 ? {} : { sourceHash: Buffer.alloc(8, 0xab) }), - }); - let failure: unknown; - try { - parse(header); - } catch (error) { - failure = error; - } - expect(failure).toBeInstanceOf(PythonPycFormatFailure); - expect(failure).toMatchObject({ kind: "unsupported" }); - if (!(failure instanceof PythonPycFormatFailure)) throw failure; - expect(failure.message).toContain( - `CPython ${version} header is recognized`, - ); - expect(failure.message).toContain(`magic ${magic}, 16-byte header`); - expect(failure.message).toContain(`flags 0x${flags.toString(16)}`); - expect(failure.message).toContain(mode); - expect(failure.message).not.toContain( - mode === "hash-based" ? "timestamp-based" : "hash-based", - ); - expect(failure.message).toMatch(/no marshal code layout.*pinned/iu); - expect(failure.message).toMatch(/marshal decoding.*CPython 3\.7/iu); - }, -); +it("keeps CPython 3.16 pre-release magics outside standalone decoder coverage", () => { + expect(() => parse(pythonPycHeader(3706))).toThrow( + expect.objectContaining({ + kind: "unsupported", + message: expect.stringMatching(/3706.*3\.7.*3\.15/), + }), + ); +}); it("does not qualify a magic outside the pinned identity table", () => { expect(() => parse(pythonPycHeader(1))).toThrow( @@ -203,6 +175,115 @@ it("does not qualify a magic outside the pinned identity table", () => { ); }); +const withSliceConsts = (magic: number, consts: Buffer[]): Buffer => + Buffer.concat([pythonPycHeader(magic), marshalCode("py311", { consts })]); +const marshalSlice = ( + start: Buffer, + stop: Buffer, + step: Buffer, + flagged = false, +): Buffer => + Buffer.concat([Buffer.from([flagged ? 0xba : 0x3a]), start, stop, step]); +const marshalInt = (value: number): Buffer => + Buffer.concat([Buffer.from([0x69]), i32(value)]); +const marshalFrozenset = (...members: Buffer[]): Buffer => + Buffer.concat([Buffer.from([0x3e]), u32(members.length), ...members]); +const MARSHAL_NONE = Buffer.from([0x4e]); + +it("decodes CPython 3.14 slice constants with Python slice equality", () => { + const report = parse( + withSliceConsts(3627, [ + marshalSlice(marshalInt(1), marshalInt(10), marshalInt(2)), + // 1 == True, so the first two slices are one member; the third differs + // in its step. + marshalFrozenset( + marshalSlice(marshalInt(1), marshalInt(2), MARSHAL_NONE), + marshalSlice(Buffer.from([0x54]), marshalInt(2), MARSHAL_NONE), + marshalSlice(marshalInt(1), marshalInt(2), marshalInt(1)), + ), + // A slice never equals the tuple of its fields. + marshalFrozenset( + marshalSlice(marshalInt(1), marshalInt(2), MARSHAL_NONE), + Buffer.concat([ + Buffer.from([0x29, 3]), + marshalInt(1), + marshalInt(2), + MARSHAL_NONE, + ]), + ), + ]), + ); + expect(report.header).toMatchObject({ magic: 3627, python_version: "3.14" }); + expect(root(report).consts).toEqual([ + { kind: "slice" }, + { kind: "frozenset", length: 2 }, + { kind: "frozenset", length: 2 }, + ]); + expect(pythonPycSchema.safeParse(report).success).toBe(true); +}); + +it("walks slice members for hashability and nested code", () => { + const emptyList = Buffer.concat([Buffer.from([0x5b]), u32(0)]); + expect(() => + parse( + withSliceConsts(3627, [ + marshalFrozenset(marshalSlice(emptyList, MARSHAL_NONE, MARSHAL_NONE)), + ]), + ), + ).toThrow(/Unhashable set member/); + const report = parse( + withSliceConsts(3627, [ + marshalSlice( + marshalCode("py311", { + name: "inner", + qualname: "inner", + firstlineno: 2, + }), + MARSHAL_NONE, + MARSHAL_NONE, + ), + ]), + ); + expect(root(report).nested_code).toEqual([1]); + expect(report.code_objects[1]?.name).toBe("inner"); +}); + +it("rejects slice tags before magic 3608 and follows CPython on null members", () => { + const sliced = marshalSlice(marshalInt(1), marshalInt(2), MARSHAL_NONE); + expect(() => parse(withSliceConsts(3571, [sliced]))).toThrow( + expect.objectContaining({ + kind: "malformed", + message: expect.stringMatching(/0x3a.*3608/), + }), + ); + expect(root(parse(withSliceConsts(3627, [sliced]))).consts).toEqual([ + { kind: "slice" }, + ]); + // A TYPE_NULL member leaves no slice object; the enclosing tuple reports it. + expect(() => + parse( + withSliceConsts(3627, [ + marshalSlice(marshalInt(1), Buffer.from([0x30]), MARSHAL_NONE), + ]), + ), + ).toThrow(/NULL object in marshal data for sequence/); + // A dictionary ends at that null key, and the slice's reserved slot stays + // invalid for a later reference, as in CPython. + expect(() => + parse( + withSliceConsts(3627, [Buffer.from([0x7b, 0xba, 0x30]), marshalRef(0)]), + ), + ).toThrow(/TYPE_REF index 0 is missing/); + // A reference back into a slice still being read is invalid too. + expect(() => + parse( + withSliceConsts(3627, [ + marshalSlice(marshalRef(0), MARSHAL_NONE, MARSHAL_NONE, true), + ]), + ), + ).toThrow(/TYPE_REF index 0 is missing/); +}); + it("reports a short unknown header before unsupported magic, and bad CR LF before either", () => { const unknown = pythonPycHeader(1).subarray(0, 4); expect(() => parse(unknown)).toThrow( diff --git a/src/python/PythonPycParser.ts b/src/python/PythonPycParser.ts index d170dc10e6..3f707f2c46 100644 --- a/src/python/PythonPycParser.ts +++ b/src/python/PythonPycParser.ts @@ -7,6 +7,7 @@ import { PYTHON_PYC_HASH_BASED, PYTHON_PYC_HEADER_BYTES, PYTHON_PYC_INT_PREVIEW_BITS, + PYTHON_PYC_SLICE_MAGIC, pythonPycMarshalProfile, type PythonPycLimits, type PythonPyc, @@ -49,6 +50,8 @@ const TYPE_ASCII_INTERNED = 0x41; const TYPE_SMALL_TUPLE = 0x29; const TYPE_SHORT_ASCII = 0x7a; const TYPE_SHORT_ASCII_INTERNED = 0x5a; +/** CPython 3.14 slice constants; `PYTHON_PYC_SLICE_MAGIC` gates the tag. */ +const TYPE_SLICE = 0x3a; const TYPES_WITHOUT_REFERENCE_SLOTS = new Set([ TYPE_NULL, TYPE_NONE, @@ -87,6 +90,8 @@ type MarshalValue = | { readonly kind: "list"; readonly items: MarshalValue[] } | { readonly kind: "set"; readonly items: MarshalValue[] } | { readonly kind: "frozenset"; readonly items: MarshalValue[] } + /** CPython 3.14+ slice constant: start, stop, and step. */ + | { readonly kind: "slice"; readonly items: MarshalValue[] } | { readonly kind: "dict"; readonly items: [MarshalValue, MarshalValue][]; @@ -131,7 +136,8 @@ const isContainer = (value: MarshalValue): boolean => value.kind === "list" || value.kind === "set" || value.kind === "frozenset" || - value.kind === "dict"; + value.kind === "dict" || + value.kind === "slice"; const containerChildren = (value: MarshalValue): readonly MarshalValue[] => { if (value.kind === "dict") { @@ -143,7 +149,8 @@ const containerChildren = (value: MarshalValue): readonly MarshalValue[] => { value.kind === "tuple" || value.kind === "list" || value.kind === "set" || - value.kind === "frozenset" + value.kind === "frozenset" || + value.kind === "slice" ) return value.items; return []; @@ -198,7 +205,7 @@ const sha256 = (bytes: Buffer): string => { return digest; }; -/** Decode one bounded CPython 3.7–3.13 pyc without executing bytecode. */ +/** Decode one bounded CPython 3.7–3.15 pyc without executing bytecode. */ export const parsePythonPyc = ( bytes: Buffer, artifact: { path: string; sha256: string; bytes: number }, @@ -234,8 +241,8 @@ export const parsePythonPyc = ( }`; return unsupported( identity === null - ? `Pyc magic ${magic} is not a pinned CPython identity, so its header cannot be qualified; standalone marshal decoding is pinned for CPython 3.7–3.13.` - : `CPython ${identity.pythonVersion} header is recognized (magic ${magic}, ${headerFacts}), but no marshal code layout is pinned for that version, so its code objects cannot be decoded. Standalone marshal decoding is pinned for CPython 3.7–3.13.`, + ? `Pyc magic ${magic} is not a pinned CPython identity, so its header cannot be qualified; standalone marshal decoding is pinned for CPython 3.7–3.13 and the 3.14 and 3.15 magics 3627, 3663 and 3666.` + : `CPython ${identity.pythonVersion} header is recognized (magic ${magic}, ${headerFacts}), but no marshal code layout is pinned for that version, so its code objects cannot be decoded. Standalone marshal decoding is pinned for CPython 3.7–3.13 and the 3.14 and 3.15 magics 3627, 3663 and 3666.`, ); } const flags = bytes.readUInt32LE(4); @@ -258,6 +265,7 @@ export const parsePythonPyc = ( magic, input, mapped.negative_code_fields === "rejected", + magic >= PYTHON_PYC_SLICE_MAGIC, ); const root = reader.readObject(0); if (reader.offset !== marshal.length) @@ -353,6 +361,8 @@ class MarshalReader { readonly magic: number, readonly input: PythonPycLimits, readonly rejectNegativeCodeFields: boolean, + /** Marshal slice constants exist from CPython 3.14a1 magic 3608. */ + readonly acceptSlices: boolean, ) {} readObject(depth: number): MarshalValue { @@ -382,7 +392,13 @@ class MarshalReader { : undefined; if (slot !== undefined) this.#intern.push(undefined); const value = this.readTyped(type, depth, slot); - if (slot !== undefined && this.#intern[slot] === undefined) + // A slice with a TYPE_NULL member yields no object, as in CPython, so + // its reserved slot stays invalid for later references. + if ( + slot !== undefined && + this.#intern[slot] === undefined && + value.kind !== "null" + ) this.#intern[slot] = value; return value; } @@ -483,6 +499,12 @@ class MarshalReader { return this.dict(depth, slot); case TYPE_CODE: return { kind: "code", value: this.code(depth) }; + case TYPE_SLICE: + if (!this.acceptSlices) + return malformed( + `Unsupported marshal type 0x3a before CPython magic ${PYTHON_PYC_SLICE_MAGIC}.`, + ); + return this.slice(depth); default: return malformed(`Unsupported marshal type 0x${type.toString(16)}.`); } @@ -920,6 +942,22 @@ class MarshalReader { if (slot !== undefined) this.#intern[slot] = value; } + /** + * CPython 3.14 reserves a slice's reference slot before reading its start, + * stop, and step, so a reference back into the slice is still invalid. A + * null member leaves the slice without a value instead of failing here; + * the enclosing container or top-level read reports it, as in CPython. + */ + slice(depth: number): MarshalValue { + const items: MarshalValue[] = []; + for (let index = 0; index < 3; index += 1) { + const item = this.readObject(depth + 1); + if (item.kind === "null") return item; + items.push(item); + } + return { kind: "slice", items }; + } + /** * CPython registers tuples and lists before reading items, so a container * can refer to itself. The same object is what later TYPE_REF entries see. @@ -1058,8 +1096,12 @@ class MarshalReader { // A tuple still being read has missing items; CPython cannot hash it. if (mode === "hash" && this.#readingTuples.has(frame.value)) frame.failed = true; + // Slices hash and compare through start, stop, and step since CPython + // 3.12, so they walk like a tuple but never equal one. const items = - frame.value.kind === "tuple" || frame.value.kind === "frozenset" + frame.value.kind === "tuple" || + frame.value.kind === "frozenset" || + frame.value.kind === "slice" ? frame.value.items : undefined; if (items === undefined) { @@ -1233,7 +1275,8 @@ class MarshalReader { const expanded = this.#expandedContainerSize(childExpanded); if (expanded > this.input.max_string_bytes) this.#keyTooLarge(expanded); const ordered = value.kind === "frozenset" ? [...parts].sort() : parts; - const kind = value.kind === "tuple" ? "T" : "F"; + const kind = + value.kind === "tuple" ? "T" : value.kind === "slice" ? "L" : "F"; const descriptor = `${kind}:${ordered.join(",")}`; // Charge the descriptor that is stored, not the unmaterialized expansion. // A linear reference chain stays within the file budget; repeating one @@ -1781,6 +1824,8 @@ const buildConstSummary = (value: MarshalValue): PythonPycConst => { return { kind: "dict", length: value.items.length }; case "ellipsis": return { kind: "ellipsis" }; + case "slice": + return { kind: "slice" }; case "complex": return { kind: "complex", diff --git a/tests/acceptance/pythonPycCli.test.ts b/tests/acceptance/pythonPycCli.test.ts index 3b2220fad4..d3b705c37b 100644 --- a/tests/acceptance/pythonPycCli.test.ts +++ b/tests/acceptance/pythonPycCli.test.ts @@ -143,6 +143,142 @@ cliTest( }, ); +cliTest.for([ + { + version: "3.14.4", + pythonVersion: "3.14", + magic: 3627, + encoded: + "Kw4NCgAAAAAAAAAAHwAAAOMAAAAAAAAAAAAAAAABAAAAAAAAAPMMAAAAgABSABcAdABSASMAKQNjAQAAAAAAAAAAAAAAAgAAAAMAAADzFAAAAIAAVgBSACwaAAAAAAAAAAAAACMAKQE66QEAAADpCgAAAOkCAAAAqQApAdoBYXMBAAAAJtoKZml4dHVyZS5wedoBZnIJAAAAAQAAAHMMAAAAgADYCwyIVo050AQU8wAAAABOPgIAAADjAgAAAAAAAAAAAAAAAgAAAAMAAADzHgAAAIAAVgBQAAAAAAAAAAAAAAAAAAAAAAAAAFYAMwIjAKkBTqkB2gF4qQJyBwAAANoBYnMCAAAAJiZyCAAAANoIPGxhbWJkYT5yEgAAAAEAAABzDAAAAIAAsAG3A7EDsFGpeHIKAAAA4wIAAAAAAAAAAAAAAAIAAAADAAAA8x4AAACAAFYAUAAAAAAAAAAAAAAAAAAAAAAAAABWATMCIwByDQAAAHIOAAAAchAAAABzAgAAACYmcggAAAByEgAAAHISAAAAAQAAAHMMAAAAgACQcZdzkXOYQZFocgoAAAApAXIJAAAAcgYAAAByCgAAAHIIAAAA2gg8bW9kdWxlPnIVAAAAAQAAAHMKAAAA8AMBAQH0AgEBFXIKAAAA", + }, + { + version: "3.15.0a8", + pythonVersion: "3.15", + magic: 3663, + encoded: + "Tw4NCgAAAAAAAAAAHwAAAOMAAAAAAAAAAAAAAAABAAAAAAAAAPMOAAAAgAAAAFEAFQBzAFEBIQApA2MBAAAAAAAAAAAAAAACAAAAAwAAAPMWAAAAgAAAAFUAUQAqGgAAAAAAAAAAAAAhACkBOukBAAAA6QoAAADpAgAAAKkAKQHaAWFzAQAAACbaCmZpeHR1cmUucHnaAWZyCQAAAAEAAABzDAAAAIEA2AsMiFaNOdAEFPMAAAAATj4CAAAA4wIAAAAAAAAAAAAAAAIAAAADAAAA8yAAAACAAAAAVQBPAAAAAAAAAAAAAAAAAAAAAAAAAFUAMQIhAKkBTqkB2gF4qQJyBwAAANoBYnMCAAAAJiZyCAAAANoIPGxhbWJkYT5yEgAAAAEAAABzDAAAAIEAsAG3A7EDsFGpeHIKAAAA4wIAAAAAAAAAAAAAAAIAAAADAAAA8yAAAACAAAAAVQBPAAAAAAAAAAAAAAAAAAAAAAAAAFUBMQIhAHINAAAAcg4AAAByEAAAAHMCAAAAJiZyCAAAAHISAAAAchIAAAABAAAAcwwAAACBAJBxl3ORc5hBkWhyCgAAACkBcgkAAAByBgAAAHIKAAAAcggAAADaCDxtb2R1bGU+chUAAAABAAAAcwoAAADxAwEBAfQCAQEVcgoAAAA=", + }, +])( + "reads CPython $version output with a slice constant and distinct lambdas", + async ({ version, pythonVersion, magic, encoded }, { cli }) => { + // CPython marshal output of `def f(a): return a[1:10:2]` plus a frozenset + // of two same-line lambdas, `(a.x, b)` and `(a.x, a)`, whose bytecode + // differs only in the LOAD_FAST_BORROW operand after LOAD_ATTR's caches. + const pyc = Buffer.from(encoded, "base64"); + const root = await createTestTempDirectory("rea-python-pyc-slice-"); + const path = join(root, `slice-${version}.pyc`); + await writeFile(path, pyc); + const result = await cli.run({ + arguments: ["inspect-python-pyc", path, "--json"], + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = completePycEvidenceSchema.parse(result.json); + expect(evidence.normalized_result.header).toMatchObject({ + magic, + python_version: pythonVersion, + code_layout: "py311", + }); + expect(evidence.normalized_result.code_objects).toHaveLength(4); + expect(evidence.normalized_result.code_objects[0]).toMatchObject({ + consts: [ + { kind: "code", name: "f", firstlineno: 1 }, + { kind: "none" }, + { kind: "frozenset", length: 2 }, + ], + nested_code: [1, 2, 3], + }); + expect(evidence.normalized_result.code_objects[1]).toMatchObject({ + name: "f", + consts: [{ kind: "slice" }], + }); + expect(evidence.normalized_result.code_objects[2]?.name).toBe(""); + expect(evidence.normalized_result.code_objects[3]?.name).toBe(""); + }, +); + +cliTest( + "reads the final CPython 3.15.2 magic 3666 from a real release pyc", + async ({ cli }) => { + // `encodings/__pycache__/utf_8.cpython-315.pyc`, byte for byte as shipped in + // python-build-standalone cpython-3.15.2+20261026-aarch64-apple-darwin + // (install_only tarball, SHA-256 285a6b92...e64a6): the header carries magic 3666. + // The expected rows are what a CPython 3.15.0a8 interpreter's own + // `marshal.loads` reports for these bytes (qualname, first line, SHA-256 of + // `co_code`), so they are not derived from this reader. a8 and the final + // release share the 3.11 code layout; 3.15.2 itself was not executed. + const pyc = Buffer.from( + "Ug4NCgAAAAAQXd9q7QMAAOMAAAAAAAAAAAAAAAAFAAAAAAAAAPPkAAAAgAAAAFEAcwBdAFAHSARzAVwBTwQAAAAAAAAAAAAAAAAAAAAAAABzA1ELUQEVAGsBcwQTAB8AUQIVAFEDXAFPCgAAAAAAAAAAAAAAAAAAAAAAADIDAAAAAAAAcwUTAB8AUQQVAFEFXAFPDAAAAAAAAAAAAAAAAAAAAAAAADIDAAAAAAAAcwcTAB8AUQYVAFEHXAFPEAAAAAAAAAAAAAAAAAAAAAAAADIDAAAAAAAAcwgTAB8AUQgVAFEJXAFPEgAAAAAAAAAAAAAAAAAAAAAAADIDAAAAAAAAcwlRChUAcwpQByEAKQx6gFB5dGhvbiAndXRmLTgnIENvZGVjCgoKV3JpdHRlbiBieSBNYXJjLUFuZHJlIExlbWJ1cmcgKG1hbEBsZW1idXJnLmNvbSkuCgooYykgQ29weXJpZ2h0IENOUkksIEFsbCBSaWdodHMgUmVzZXJ2ZWQuIE5PIFdBUlJBTlRZLgoKYwIAAAAAAAAAAAAAAAUAAAADAAAA8zIAAACAAAAAWwAAAAAAAAAAAE8CAAAAAAAAAAAAAAAAAAAAAAAAHwBWAVAJMgMAAAAAAAAhACkBVCkC2gZjb2RlY3PaDHV0Zl84X2RlY29kZSkC2gVpbnB1dNoGZXJyb3JzcwIAAAAmJtpNL3ByaXZhdGUvdG1wL3RtcGUzY3J1Y2ZiL291dC9weXRob24vaW5zdGFsbC9saWIvcHl0aG9uMy4xNS9lbmNvZGluZ3MvdXRmXzgucHnaBmRlY29kZXIIAAAADwAAAHMVAAAAgQDcCxHXCx7SCx6Ydahk0wsz0AQz8wAAAABjAAAAAAAAAAAAAAAAAgAAAAAAAADzLAAAAGAAgAAAAFwAcwFRAHMCXRJzAxQAbgBRA1EBFQBrAXMEUQJzBVUAcwZQByEAKQTaEkluY3JlbWVudGFsRW5jb2RlcmMDAAAAAAAAAAAAAAAEAAAAAwAACPNSAAAAgAAAAFsAAAAAAAAAAABPAgAAAAAAAAAAAAAAAAAAAAAAAB8AVhBPBAAAAAAAAAAAAAAAAAAAAAAAADICAAAAAAAAXQAqGgAAAAAAAAAAAAAhACkB6QAAAAApA3IDAAAA2gx1dGZfOF9lbmNvZGVyBgAAACkD2gRzZWxmcgUAAADaBWZpbmFscwMAAAAmJiZyBwAAANoGZW5jb2Rl2hlJbmNyZW1lbnRhbEVuY29kZXIuZW5jb2RlEwAAAHMcAAAAgQDcDxXXDyLSDyKgNa8rqSvTDzawcdUPOdAIOXIJAAAAqQApAUYpB9oIX19uYW1lX1/aCl9fbW9kdWxlX1/aDF9fcXVhbG5hbWVfX9oPX19maXJzdGxpbmVub19fchEAAADaFV9fc3RhdGljX2F0dHJpYnV0ZXNfX9oRX19jbGFzc2RpY3RjZWxsX18pAdoNX19jbGFzc2RpY3RfX3MBAAAAQHIHAAAAcgsAAAByCwAAABIAAABzDwAAAPiHAIEA9wIBBTryAAEFOnIJAAAAcgsAAABjAAAAAAAAAAAAAAAAAQAAAAAAAADzMAAAAIAAAABcAHMBUQBzAl0WcwNcBE8KAAAAAAAAAAAAAAAAAAAAAAAAcwZRAXMHUAchACkC2hJJbmNyZW1lbnRhbERlY29kZXJyEwAAACkIchQAAAByFQAAAHIWAAAAchcAAAByAwAAAHIEAAAA2g5fYnVmZmVyX2RlY29kZXIYAAAAchMAAAByCQAAAHIHAAAAchwAAAByHAAAABYAAABzDQAAAIcA2BUb1xUo0RUohE5yCQAAAHIcAAAAYwAAAAAAAAAAAAAAAAEAAAAAAAAA8zAAAACAAAAAXABzAVEAcwJdGXMDXARPCgAAAAAAAAAAAAAAAAAAAAAAAHMGUQFzB1AHIQApAtoMU3RyZWFtV3JpdGVychMAAAApCHIUAAAAchUAAAByFgAAAHIXAAAAcgMAAAByDgAAAHIRAAAAchgAAAByEwAAAHIJAAAAcgcAAAByHwAAAHIfAAAAGQAAAPMNAAAAhwDYDRPXDSDRDSCERnIJAAAAch8AAABjAAAAAAAAAAAAAAAAAQAAAAAAAADzMAAAAIAAAABcAHMBUQBzAl0ccwNcBE8KAAAAAAAAAAAAAAAAAAAAAAAAcwZRAXMHUAchACkC2gxTdHJlYW1SZWFkZXJyEwAAACkIchQAAAByFQAAAHIWAAAAchcAAAByAwAAAHIEAAAAcggAAAByGAAAAHITAAAAcgkAAAByBwAAAHIiAAAAciIAAAAcAAAAciAAAAByCQAAAHIiAAAAYwAAAAAAAAAAAAAAAAoAAAADAAAA824AAACAAAAAWwAAAAAAAAAAAE8CAAAAAAAAAAAAAAAAAAAAAAAAHwBRAFsEAAAAAAAAAABbBgAAAAAAAAAAWwgAAAAAAAAAAFsKAAAAAAAAAABbDAAAAAAAAAAAWw4AAAAAAAAAAFEBNQcAAAAAAAAhACkCegV1dGYtOCkH2gRuYW1lchEAAAByCAAAANoSaW5jcmVtZW50YWxlbmNvZGVy2hJpbmNyZW1lbnRhbGRlY29kZXLaDHN0cmVhbXJlYWRlctoMc3RyZWFtd3JpdGVyKQhyAwAAANoJQ29kZWNJbmZvchEAAAByCAAAAHILAAAAchwAAAByIgAAAHIfAAAAchMAAAByCQAAAHIHAAAA2gtnZXRyZWdlbnRyeXIqAAAAIQAAAHMqAAAAgQDcCxHXCxvSCxvYDRTcDxXcDxXcGy3cGy3cFSHcFSH0DwgMBvAACAUGcgkAAAApAdoGc3RyaWN0KQvaB19fZG9jX19yAwAAAHIOAAAAchEAAAByCAAAAHILAAAA2hpCdWZmZXJlZEluY3JlbWVudGFsRGVjb2RlcnIcAAAAch8AAAByIgAAAHIqAAAAchMAAAByCQAAAHIHAAAA2gg8bW9kdWxlPnIuAAAAAQAAAHNuAAAA8QMBAQHxAgcBBPMQAAEO8AgAChDXCRzRCRyABvQEAQE09AYCATqYFtcZMtEZMvQAAgE69AgBASmYFtcZOtEZOvQAAQEp9AYBASGQNtcTJtETJvQAAQEh9AYBASGQNtcTJtETJvQAAQEh9AoJAQZyCQAAAA==", + "base64", + ); + const root = await createTestTempDirectory("rea-python-pyc-final-315-"); + const path = join(root, "utf_8.cpython-315.pyc"); + await writeFile(path, pyc); + const result = await cli.run({ + arguments: ["inspect-python-pyc", path, "--json"], + }); + expect(result.exitCode, result.stderr).toBe(0); + const evidence = completePycEvidenceSchema.parse(result.json); + expect(evidence.normalized_result.header).toMatchObject({ + magic: 3666, + python_version: "3.15", + code_layout: "py311", + flags: 0, + hash_based: false, + }); + expect( + evidence.normalized_result.code_objects.map((code) => [ + code.qualname, + code.firstlineno, + code.bytecode.sha256, + ]), + ).toEqual([ + [ + "", + 1, + "dec009b8d182273295bb1af2572eaf510c5a2c9b79c9ed5b5867acd3ab3f3532", + ], + [ + "decode", + 15, + "fe1a50cda8d8510aee7c6c8a4300c4fb151a360ba9e830b7ac9bd797d4b522f6", + ], + [ + "IncrementalEncoder", + 18, + "32afcda5f0c08dccd01a0e42c4a2b5575247c8ed5723bc704db2dd995511c73b", + ], + [ + "IncrementalEncoder.encode", + 19, + "df5053d342e0600648ddbd48ec64a5f1c4336c6956cde360b8799c61da5d93fe", + ], + [ + "IncrementalDecoder", + 22, + "2d2cf7a62ece3c133ac82e9d59fee2ecfd21eb1a79c31b7d4a6e64a30fc2072d", + ], + [ + "StreamWriter", + 25, + "b77dff0cd7cf304af6ae747b34e03499353b914282cb7d782ec4d8e24e442c67", + ], + [ + "StreamReader", + 28, + "0b21e2f77e98d3f01c2698ed2b9cff5778331afb1a8bdc12b3eb80c389d36861", + ], + [ + "getregentry", + 33, + "98cecbb09af6594440e3caab695700c9ca4d5a1355a3e8a83e2172dd8a9fc6a5", + ], + ]); + expect(evidence.normalized_result.artifact.sha256).toBe( + createHash("sha256").update(pyc).digest("hex"), + ); + }, +); + cliTest( "uses magic-specific cache tables for synthetic pyc bytecode equality", async ({ cli }) => { @@ -156,6 +292,9 @@ cliTest( { magic: 3553, opcode: 25, cacheUnits: 1 }, { magic: 3564, opcode: 58, cacheUnits: 1 }, { magic: 3571, opcode: 82, cacheUnits: 9 }, + { magic: 3627, opcode: 80, cacheUnits: 9 }, + { magic: 3663, opcode: 79, cacheUnits: 9 }, + { magic: 3666, opcode: 128, cacheUnits: 1 }, ]; for (const { magic, opcode, cacheUnits } of cases) { const cache = Buffer.alloc(cacheUnits * 2);