feat(python): inspect CPython pyc headers and marshal code objects (#4826)

* feat(python): inspect CPython pyc headers and marshal code objects

Add inspect_python_pyc / inspect-python-pyc to read one explicit local
CPython 3.7–3.13 .pyc without executing bytecode or recovering source.

* chore(python): drop unused PythonPycHeader export

Knip reported the type as unused after inspect_python_pyc landed.

* fix(python): bound pyc expansion and decode marshal like CPython

- Count a nested code object reached through TYPE_REF once per copy the
  report carries, against max_objects and max_file_bytes, before counting
  or validating the tree
- Charge long digits to max_string_bytes, reject out-of-range and
  unnormalized digits, build the value in one pass, and report integers
  above 256 bits by bit length only
- Decode TYPE_UNICODE with surrogatepass semantics and reject other
  ill-formed UTF-8
- Map only 3.11 magics 3494-3495 to the py311 layout
- Report non-finite float and complex parts as inf, -inf or nan
- Allow an empty co_filename
- Require bytes for co_linetable and co_exceptiontable
- Bind cleanup-failure errors to their retained descriptor leases

* fix(python): list pyc code objects once instead of a recursive tree

Report each distinct code object once in code_objects, top-level first,
with nested_code as indexes. A child shared through TYPE_REF is no longer
copied per reference, and the MCP output schema is no longer recursive.
Names repeated through TYPE_REF still count against max_file_bytes for
each copy the report carries.

* fix(python): bound shared pyc constants and reject malformed tables

- Cache byte-string digests so a buffer shared through TYPE_REF is
  hashed once instead of on every reference.
- Charge every reported co_consts entry, including tables shared
  through TYPE_REF, against max_objects before summaries are built.
- Resolve the inspect-python-pyc path with absoluteCallerPath so
  link/../file.pyc reads the file the filesystem selects.
- Reject TYPE_NULL inside marshal sequences, as CPython does.
- Reject a co_localspluskinds length that does not match
  co_localsplusnames in the parser and the report schema.

* fix(python): bound shared pyc names and read CPython marshal graphs

Charge each reported code-constant name, keep integer bit lengths on the
decoded value, and list code objects nested inside constant containers.
Accept self-referential lists and dicts, marshal version 0 text floats,
empty code names, the original string byte count, and Python set and dict
lengths. Pass the CLI cancellation signal into command cleanup.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): bound pyc key expansion and shared nested code

Charge set and dictionary keys before a shared tuple doubles them, list each
nested code object once per owner, and accept code objects as CPython keys.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): correct pyc equality, counts, and nested walks

Summarize code objects only after the marshal graph is complete, and
collect nested code with an iterative walk that shares wrapper results
instead of copying every descendant list. Keep tuple boundaries and the
original string bytes in equality keys, and reject negative CPython counts.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): bound shared pyc names and keep the first dict key

Charge every copied name-table entry against the object budget before
building the report array, including empty strings that add no bytes.
When dictionary keys compare equal, keep the first key object and
replace only its value, matching CPython.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): walk each pyc dictionary child list once

Keep a container's child array on the nested-code frame so a wide
dictionary is not copied on every step of the synchronous marshal walk.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): reuse stored pyc dictionary children when combining

The nested walk keeps each container's child list on its frame.
Combining nested code rebuilt that list, and the 8,000-key regression
still accepted the old per-key copy. Use the stored list, and require
that dictionary to finish well under the copy's runtime while still
reporting a nested code object.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): keep distinct pyc code objects and reject corrupt ones

Encode each name table's length and entry boundaries in the equality
key so a different name list cannot share a memo key. Walk tuple and
frozenset keys on an explicit stack so a long TYPE_REF chain does not
overflow. Treat code objects CPython rejects for an odd co_code or an
undersized local table as malformed instead of complete coverage.

Co-authored-by: morluto <morluto@users.noreply.github.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-10-22 19:00:08 +08:00
1 parent 4043e71f6c
commit 2c3dd946d5
36 files changed
+4457 -2

No files matched your search

+298
View File
@@ -0,0 +1,298 @@
import { z } from "zod";
import { digestSchema } from "../digests.js";
import { localPathStringSchema } from "../localPath.js";
export const PYTHON_PYC_HEADER_BYTES = 16;
export const PYTHON_PYC_HASH_BASED = 0x1;
export const PYTHON_PYC_CHECK_SOURCE = 0x2;
const uint = z.number().int().min(0).max(0xffffffff);
const index = z.number().int().nonnegative();
/** Non-negative CPython `int` counts. `0xffffffff` is -1 and is not evidence. */
const signedCount = z.number().int().min(0).max(0x7fffffff);
/** Marshal TYPE_CODE field order for CPython 3.7, 3.8–3.10, and 3.11+. */
export const PYTHON_PYC_CODE_LAYOUTS = ["py37", "py38", "py311"] as const;
export type PythonPycCodeLayout = (typeof PYTHON_PYC_CODE_LAYOUTS)[number];
type MagicEntry = {
readonly python_version: string;
readonly code_layout: PythonPycCodeLayout;
};
const magics = (
values: readonly number[],
python_version: string,
code_layout: PythonPycCodeLayout,
): ReadonlyArray<readonly [number, MagicEntry]> =>
values.map((magic) => [magic, { python_version, code_layout }]);
const consecutive = (
start: number,
end: number,
python_version: string,
code_layout: PythonPycCodeLayout,
): ReadonlyArray<readonly [number, MagicEntry]> =>
magics(
Array.from({ length: end - start + 1 }, (_, offset) => start + offset),
python_version,
code_layout,
);
/**
* Pinned CPython pyc magics for 3.7–3.13. Identity is the 16-bit little-endian
* field; `\r\n` is required separately. Layout follows marshal TYPE_CODE, not
* bytecode opcode tables.
*/
export const PYTHON_PYC_MAGIC_TABLE: ReadonlyMap<number, MagicEntry> = new Map([
...magics([3390, 3391, 3392, 3393, 3394], "3.7", "py37"),
...magics([3400, 3401], "3.8", "py37"),
...magics([3410, 3411, 3412, 3413], "3.8", "py38"),
...magics([3420, 3421, 3422, 3423, 3424, 3425], "3.9", "py38"),
...magics(
[3430, 3431, 3432, 3433, 3434, 3435, 3436, 3437, 3438, 3439],
"3.10",
"py38",
),
// 3.11 alphas before 3494 marshal other code layouts: no co_qualname
// before 3460, and PEP 657 end-line and column tables until 3494.
...consecutive(3494, 3495, "3.11", "py311"),
...consecutive(3500, 3531, "3.12", "py311"),
...consecutive(3550, 3571, "3.13", "py311"),
]);
/** Map a pyc magic integer to its pinned CPython family, or null when unknown. */
export const pythonPycMagicEntry = (magic: number): MagicEntry | null =>
PYTHON_PYC_MAGIC_TABLE.get(magic) ?? null;
export const inspectPythonPycInputSchema = z.strictObject({
path: localPathStringSchema.describe(
"Absolute filesystem path to the selected CPython .pyc file",
),
max_file_bytes: z
.number()
.int()
.min(16)
.max(64 * 1024 * 1024)
.default(16 * 1024 * 1024)
.describe(
"Maximum selected file size in bytes; exceeding the budget fails without partial success",
),
max_objects: z
.number()
.int()
.min(1)
.max(65536)
.default(16384)
.describe(
"Maximum marshal objects to examine; exceeding the budget fails without partial success",
),
max_depth: z
.number()
.int()
.min(1)
.max(256)
.default(64)
.describe(
"Maximum marshal nesting depth; exceeding the budget fails without partial success",
),
max_string_bytes: z
.number()
.int()
.min(1)
.max(8 * 1024 * 1024)
.default(1024 * 1024)
.describe(
"Maximum bytes retained for one marshal string or byte string; exceeding the budget fails without partial success",
),
});
const constPreviewLimit = 80;
/** Integers up to 256 bits fit the 80-character preview in decimal. */
const intPreviewBits = 256;
/** CPython float repr for values JSON numbers cannot carry. */
const pythonFloatSchema = z.union([z.number(), z.enum(["inf", "-inf", "nan"])]);
const pythonConstSchema = z.discriminatedUnion("kind", [
z.strictObject({ kind: z.literal("none") }),
z.strictObject({ kind: z.literal("bool"), value: z.boolean() }),
z.strictObject({
kind: z.literal("int"),
bits: uint,
value: z.string().min(1).max(constPreviewLimit).nullable(),
}),
z.strictObject({ kind: z.literal("float"), value: pythonFloatSchema }),
z.strictObject({
kind: z.literal("str"),
bytes: uint,
truncated: z.boolean(),
value: z.string().max(constPreviewLimit),
}),
z.strictObject({
kind: z.literal("bytes"),
bytes: uint,
sha256: digestSchema,
}),
z.strictObject({
kind: z.literal("code"),
name: z.string(),
firstlineno: uint,
}),
z.strictObject({ kind: z.literal("tuple"), length: uint }),
z.strictObject({ kind: z.literal("list"), length: uint }),
z.strictObject({ kind: z.literal("set"), length: uint }),
z.strictObject({ kind: z.literal("frozenset"), length: uint }),
z.strictObject({ kind: z.literal("dict"), length: uint }),
z.strictObject({ kind: z.literal("ellipsis") }),
z.strictObject({
kind: z.literal("complex"),
real: pythonFloatSchema,
imag: pythonFloatSchema,
}),
z.strictObject({ kind: z.literal("other"), type: z.string().min(1) }),
]);
const pythonCodeFields = {
name: z.string(),
qualname: z.string().nullable(),
filename: z.string(),
firstlineno: uint,
argcount: signedCount,
posonlyargcount: signedCount.nullable(),
kwonlyargcount: signedCount,
nlocals: signedCount.nullable(),
stacksize: signedCount,
flags: uint,
names: z.array(z.string()),
varnames: z.array(z.string()).nullable(),
freevars: z.array(z.string()).nullable(),
cellvars: z.array(z.string()).nullable(),
localsplusnames: z.array(z.string()).nullable(),
localspluskinds_hex: z.string().nullable(),
bytecode: z.strictObject({
bytes: uint,
sha256: digestSchema,
}),
consts: z.array(pythonConstSchema),
};
export type PythonPycConst = z.infer<typeof pythonConstSchema>;
/**
* One distinct marshal code object. `nested_code` holds indexes into the
* report's `code_objects`, so a child shared through TYPE_REF is listed once.
* The owner's own index is included when its constants reach that object.
*/
const pythonCodeObjectSchema = z.strictObject({
...pythonCodeFields,
nested_code: z.array(index),
});
export type PythonPycCodeObject = z.infer<typeof pythonCodeObjectSchema>;
export const pythonPycHeaderSchema = z.strictObject({
magic: z.number().int().min(0).max(0xffff),
python_version: z.string().min(1),
code_layout: z.enum(PYTHON_PYC_CODE_LAYOUTS),
flags: uint,
hash_based: z.boolean(),
check_source: z.boolean(),
source_hash_hex: z.string().nullable(),
source_mtime: uint.nullable(),
source_size: uint.nullable(),
});
/** Bounded CPython 3.7–3.13 pyc observation; marshal is walked, bytecode is not decompiled. */
export const pythonPycSchema = z
.strictObject({
artifact: z.strictObject({
path: localPathStringSchema,
sha256: digestSchema,
bytes: index,
}),
header: pythonPycHeaderSchema,
/** The top-level module code object is first. */
code_objects: z.array(pythonCodeObjectSchema).min(1),
coverage: z.strictObject({
status: z.literal("complete"),
objects: uint,
code_objects: uint,
marshal_bytes: uint,
marshal_depth: uint,
}),
limitations: z.array(z.string().min(1)).min(1),
})
.superRefine((report, context) => {
if (report.artifact.bytes < PYTHON_PYC_HEADER_BYTES) {
context.addIssue({
code: "custom",
path: ["artifact", "bytes"],
message: "A pyc header does not fit in the selected artifact.",
});
}
const marshalBytes = report.artifact.bytes - PYTHON_PYC_HEADER_BYTES;
if (report.coverage.marshal_bytes !== marshalBytes) {
context.addIssue({
code: "custom",
path: ["coverage", "marshal_bytes"],
message: "Marshal byte count disagrees with the header-stripped file.",
});
}
if (report.header.hash_based === (report.header.source_hash_hex === null)) {
context.addIssue({
code: "custom",
path: ["header", "source_hash_hex"],
message:
"Hash-based pycs require an 8-byte source hash; others must omit it.",
});
}
if (report.header.hash_based === (report.header.source_mtime !== null)) {
context.addIssue({
code: "custom",
path: ["header", "source_mtime"],
message: "Timestamp pycs require mtime; hash-based pycs must omit it.",
});
}
if (report.header.hash_based === (report.header.source_size !== null)) {
context.addIssue({
code: "custom",
path: ["header", "source_size"],
message:
"Timestamp pycs require source_size; hash-based pycs must omit it.",
});
}
if (report.coverage.code_objects !== report.code_objects.length) {
context.addIssue({
code: "custom",
path: ["coverage", "code_objects"],
message: "Code-object coverage disagrees with the listed code objects.",
});
}
report.code_objects.forEach((code, owner) => {
if (
code.localspluskinds_hex !== null &&
code.localspluskinds_hex.length !==
(code.localsplusnames?.length ?? 0) * 2
)
context.addIssue({
code: "custom",
path: ["code_objects", owner, "localspluskinds_hex"],
message:
"co_localspluskinds length does not match co_localsplusnames.",
});
code.nested_code.forEach((nested, position) => {
if (nested >= report.code_objects.length)
context.addIssue({
code: "custom",
path: ["code_objects", owner, "nested_code", position],
message: "Nested code index does not name a listed code object.",
});
});
});
});
export type PythonPyc = z.infer<typeof pythonPycSchema>;
export type InspectPythonPycInput = z.infer<typeof inspectPythonPycInputSchema>;
export const PYTHON_PYC_CONST_PREVIEW_LIMIT = constPreviewLimit;
export const PYTHON_PYC_INT_PREVIEW_BITS = intPreviewBits;