feat(python): qualify a recognized 3.14/3.15 pyc header (#8927)

* feat(python): qualify a recognized 3.14/3.15 pyc header

Python 3.15 is the default `python3` on current Homebrew, but its pyc was rejected with
"Pyc magic 3666 is outside the pinned CPython 3.7–3.13 table" — a message that says
nothing about the file and reads as if the version were unknown. Issue #8206.

Two facts were wrong.

**3.15 was not in the identity table at all,** so `pythonBytecodeIdentity` reported it as
unknown even though the table already pins 3.14 (3627). 3666 is now pinned as 3.15.

**A known identity without a pinned code layout shared one refusal with an unknown
magic.** The two are different outcomes: every 3.7+ pyc carries the same packed 16-byte
header, so a recognized newer magic can still be qualified even when no marshal layout is
pinned for it. The parser now separates them:

    magic 3666 (3.15)  -> CPython 3.15 header is recognized (magic 3666, 16-byte header,
                          flags 0x0, timestamp-based), but no marshal code layout is
                          pinned for that version, so its code objects cannot be decoded.
                          Standalone marshal decoding is pinned for CPython 3.7–3.13.
    magic 1            -> Pyc magic 1 is not a pinned CPython identity, so its header
                          cannot be qualified; ...

The recognized case reports the observed version, the header shape, the PEP 552 flags and
whether the pyc is hash-based or timestamp-based, which is what an agent needs to decide
that a `.jsc`-style text-recovery attempt is pointless and that the file is a 3.15 pyc
rather than a corrupt one. It stays a typed `unsupported` failure: no pinned layout means
no decoded code object, so there is no partial success.

I did not guess a 3.14/3.15 marshal layout. The 3.11+ `TYPE_CODE` order is pinned through
3.13; asserting the same order holds for 3.14 or 3.15 without a real interpreter to
verify against would be an unsupported claim, which is why this change qualifies the
header instead of decoding it.

Docs and the tool description now say marshal decoding is pinned for 3.7–3.13 while the
identity table also recognizes 3.14 and 3.15.

Three regressions fail on the base commit.

* chore: retrigger title and readiness checks after the title was corrected

* test(python): pin the qualified header facts to the packed fields

Review finding. Every new case used `pythonPycHeader(magic)` with its default zero flags
and never asserted the reported flags or header mode, so a refusal that hardcoded
`flags 0x0, timestamp-based` would have passed all of them.

The cases now build the header from explicit fields and assert the facts that field
produces: `flags 0x0/timestamp-based` and `flags 0x1` or `0x3` with a PEP 552 source hash
for hash-based. A separate case parses the same magic in both modes and requires each
refusal to name its own mode and not the other.

Verified by hardcoding `flags 0x0` in the refusal: the two hash-based cases fail.
This commit is contained in:
吴佳浩Alben authored and GitHub committed 2026-11-06 15:36:22 +08:00
1 parent 99bcd3e4e0
commit fe9797b487
7 files changed
+123 -12

No files matched your search

+88 -3
View File
@@ -157,12 +157,97 @@ it.each([
},
);
it("keeps a known newer CPython identity outside standalone decoder coverage", () => {
expect(() => parse(pythonPycHeader(3627))).toThrow(
it.each([
{ magic: 3627, version: "3.14", flags: 0x0, mode: "timestamp-based" },
{ magic: 3666, version: "3.15", flags: 0x0, mode: "timestamp-based" },
// A non-zero flag word and a PEP 552 hash-based header, so a refusal that hardcoded
// `flags 0x0, timestamp-based` could not pass.
{ magic: 3666, version: "3.15", flags: 0x3, mode: "hash-based" },
{ magic: 3627, version: "3.14", flags: 0x1, mode: "hash-based" },
] as const)(
"qualifies the recognized CPython $version header without a pinned code layout",
({ magic, version, flags, mode }) => {
// The header is still a packed 16-byte 3.7+ header, so its identity and flags are
// reported even though no marshal layout is pinned for this version.
const header = pythonPycHeader(magic, {
flags,
...(flags === 0 ? {} : { sourceHash: Buffer.alloc(8, 0xab) }),
});
expect(() => parse(header)).toThrow(
expect.objectContaining({
kind: "unsupported",
message: expect.stringContaining(
`CPython ${version} header is recognized`,
),
}),
);
expect(() => parse(header)).toThrow(
expect.objectContaining({
message: expect.stringContaining(`magic ${magic}, 16-byte header`),
}),
);
// The reported facts are the ones packed into this header.
expect(() => parse(header)).toThrow(
expect.objectContaining({
message: expect.stringContaining(`flags 0x${flags.toString(16)}`),
}),
);
expect(() => parse(header)).toThrow(
expect.objectContaining({
message: expect.stringContaining(mode),
}),
);
expect(() => parse(header)).toThrow(
expect.objectContaining({
message: expect.stringContaining(
"no marshal code layout is pinned for that version",
),
}),
);
expect(() => parse(header)).toThrow(
expect.objectContaining({
message: expect.stringContaining(
"marshal decoding is pinned for CPython 3.7",
),
}),
);
},
);
it("reports timestamp-based for a pyc without the hash flag and hash-based with it", () => {
// Same magic, opposite header mode: the two refusals must differ only in that fact.
const timestamp = () =>
parse(pythonPycHeader(3666, { mtime: 7, sourceSize: 9 }));
const hash = () =>
parse(
pythonPycHeader(3666, { flags: 0x1, sourceHash: Buffer.alloc(8, 1) }),
);
expect(timestamp).toThrow(
expect.objectContaining({
message: expect.stringContaining("timestamp-based"),
}),
);
expect(timestamp).toThrow(
expect.not.objectContaining({
message: expect.stringContaining("hash-based"),
}),
);
expect(hash).toThrow(
expect.objectContaining({ message: expect.stringContaining("hash-based") }),
);
expect(hash).toThrow(
expect.not.objectContaining({
message: expect.stringContaining("timestamp-based"),
}),
);
});
it("does not qualify a magic outside the pinned identity table", () => {
expect(() => parse(pythonPycHeader(1))).toThrow(
expect.objectContaining({
kind: "unsupported",
message: expect.stringContaining(
"outside the pinned CPython 3.7–3.13 table",
"is not a pinned CPython identity, so its header cannot be qualified",
),
}),
);
+13 -2
View File
@@ -223,10 +223,21 @@ export const parsePythonPyc = (
malformed(
"Selected file is shorter than a 16-byte CPython 3.7+ pyc header.",
);
if (mapped === null || identity === null)
if (mapped === null || identity === null) {
// A known identity without a pinned code layout is a different outcome from an
// unrecognized magic: the header below is still qualified from the same packed
// fields every 3.7+ pyc uses, so report it instead of a bare rejection.
const flags = bytes.readUInt32LE(4);
const hashBased = (flags & PYTHON_PYC_HASH_BASED) !== 0;
const headerFacts = `16-byte header, flags 0x${flags.toString(16)}, ${
hashBased ? "hash-based" : "timestamp-based"
}`;
return unsupported(
`Pyc magic ${magic} is outside the pinned CPython 3.7–3.13 table.`,
identity === null
? `Pyc magic ${magic} is not a pinned CPython identity, so its header cannot be qualified; standalone marshal decoding is pinned for CPython 3.7–3.13.`
: `CPython ${identity.pythonVersion} header is recognized (magic ${magic}, ${headerFacts}), but no marshal code layout is pinned for that version, so its code objects cannot be decoded. Standalone marshal decoding is pinned for CPython 3.7–3.13.`,
);
}
const flags = bytes.readUInt32LE(4);
const hashBased = (flags & PYTHON_PYC_HASH_BASED) !== 0;
const header = {