Commit Graph
29 Commits
Author SHA1 Message Date
morluto 84551888ec refactor(python): report pyc parser limits from LimitRef (#12213) 2026-11-19 05:32:28 +00:00
morluto b51ed4c3ec refactor(kernel): migrate the remaining SHA-256 hex digests to sha256Hex and move isRecord into src/kernel (#11571)
* refactor(kernel): migrate the remaining one-shot SHA-256 hex digests to sha256Hex

* refactor(kernel): move isRecord into src/kernel and replace its inline copies (#11574)

* fix(scripts): import isRecord from src/kernel in the managed manifest parser
2026-11-15 20:53:36 -05:00
morluto a80a47f143 test(python): bound pyc parser work by counts instead of elapsed time (#11608)
PythonPycParser takes an optional work counter (digested bytes, constant
summaries, listed container children, nested and key walk steps, key bytes
and report flattening visits). The eight pyc tests that bounded wall-clock
time now assert those counts, so each regression they guard (repeated
hashing, per-reference summaries, per-step child copies, shared walks,
exponential key expansion) fails deterministically on any machine.

Refs #10513
2026-11-16 08:45:43 +08:00
morluto 264dcd5fee refactor(artifacts): break the artifacts-firmware and artifacts-gpu import cycles (#11331)
* refactor(domain): move BinaryContainer into src/domain

BinaryContainer imports only domain modules and is shared by artifacts, firmware, gpu, go, il2cpp, python, rust and native. Living under src/native made artifacts and firmware import native for its failure types and BinaryReader. Pure move: git mv plus import specifier updates.

* refactor(firmware): move the devicetree parser into src/domain/firmware

The FIT artifact reader in src/artifacts parses embedded devicetree blobs with the parser that lived under src/firmware, while firmware's providers import src/artifacts: the two units imported each other. The parser and its fixture are pure and now sit beside domain/firmware/devicetree.ts. Pure move: git mv plus import specifier and baseline key updates.

* refactor(artifacts): break the artifacts-gpu import cycle (#11335)

* refactor(artifacts): move the LZ4 block decoder into src/artifacts

Pure move: git mv plus import specifier updates. The next commit removes its GPU-specific budget.

* refactor(artifacts): take the LZ4 output budget from the caller

decodeLz4Block bounded every decode by the GPU object budget and threw through rejectGpu, so UnityFS decoding in src/artifacts imported src/gpu while gpu imports artifacts. Callers now pass their budget: GPU payloads keep GPU_OBJECT_BYTES; UnityFS passes its 64 MiB block bound, which its size checks already enforce before decoding. Malformed blocks still throw BinaryFormatFailure with the same messages.

* test(architecture): drop the artifacts-firmware and artifacts-gpu cycles from the layer baseline

* refactor(artifacts): move the PE resource reader into src/artifacts/pe (#11347)

refactor(artifacts): move the PE resource layout and parser into src/artifacts/pe

The Node SEA and Deno standalone readers in src/artifacts locate PE resources with the layout and parser that lived under src/native/pe, while those modules already read through src/artifacts (ArtifactReaderFailure, assertArtifactReadActive). Pure move: git mv plus import specifier, baseline key and spec path updates.
2026-11-16 03:44:53 +08:00
morluto 28332d43ca refactor(javascript): charge the worker's retained result bytes through an over-committing Budget (#11399)
* refactor(javascript): charge the worker's retained result bytes through an over-committing Budget

Static checkpoints may retain past the result budget up to the static capacity, and the worker reports that exhaustion itself. Budget.chargeOverCommitting records the excess (overCommitted) and reports no remainder until refunds bring use back under the maximum. The budget-arithmetic baseline is now empty.

* refactor(domain): charge hand-written allowances through Budget and count limit-checked fields (#11405)

* refactor(domain): charge hand-written allowances through Budget and count limit-checked fields

Pyc report and key-walk tallies, reverse-engineering capture admission, Playwright trace fact retention and semantic alias selections now charge a Budget. The budget-arithmetic guard also counts a field outside Budget classes that is adjusted by hand and compared with a limit-named value; the eleven remaining sites are a counted baseline.

* fix(verify): scope limit-checked fields to their class and match limit names in any case
2026-11-16 03:34:31 +08:00
morluto 64c119df47 refactor(domain): charge remaining fixed allowances through Budget (#11319)
* refactor(domain): charge the Interface Builder, NativeAOT, managed and runtime budgets through Budget

* test(browser): show a refused runtime retention charge leaves the room intact

* refactor(domain): charge the remaining fixed allowances through Budget (#11383)

* refactor(domain): charge CUDA metadata and SQLite reply budgets through Budget

CudaMetadataBudget (visits, section reads, temporary index) and the SQLite
OutputBudget kept hand-written allowances whose field names escaped the
budget-arithmetic guard; the SQLite reply budget also charged each part
before checking it. Both now charge Budget, which refuses an oversized
charge without changing the remainder.

The guard now also counts, inside any hand-written ...Budget class, a field
that is adjusted by hand and compared with < or >, whatever its name.
Unchecked tallies (a yield counter, a reservation ledger) stay uncounted.
Firmware arena cursors are write offsets, not allowances: they are renamed
#arenaWriteOffset and leave the guard's scope. Baseline 17 -> 15.

* refactor(domain): move cache, re-priced, checkpointed and ledger retention onto Budget

Budget gains chargeEvicting (evict the oldest cache entries until a charge
fits) and restore (refund everything charged since a used mark, never
re-charging). With them:

- the Bun metadata range cache evicts through chargeEvicting and charges
  physical reads through a second Budget;
- Playwright scenario events refund an event's old cost and try-charge its
  re-priced cost, and admission charges with tryCharge;
- WebNetworkProjectionAccumulator keeps one shared allowance plus retained
  and temporary shares whose checkpoints restore by refunding;
- EvidenceLedger commits its admitted delta (charge, or refund when a
  replacement shrinks) and clears with restore(0);
- the web bundle worker charges an admitted result, and on the completed-
  snapshot path retains facts only when they still fit;
- the Pyc nested code walk charges steps and retained references.

budget-arithmetic 15 -> 4 (JavaScriptAnalysisWorker only).

* build(verify): declare the budget-arithmetic guard's counter for typed tests
2026-11-16 03:08:28 +08:00
morluto 479abaf3b2 refactor(python): read pyc and PYZ marshal values through one shared reader (#10806)
* refactor(python): read pyc and PYZ marshal values through one shared reader

* refactor(python): read pyc 64-bit values and header fields through shared readers (#10834)
2026-11-15 16:50:58 +08:00
morluto b33e5c4867 refactor(python): report pyc format failures through ParseFailure (#10664) 2026-11-15 13:45:40 +08:00
gensokyo-devandgensokyo-dev 2caa2598d3 fix(python): walk slice members in the code hashability stack
The key resolver followed slice members, but the explicit-stack hashability
walk skipped them, so a code object reached through a slice constant started
a recursive hashability check. A 3,000-link slice-wrapped TYPE_REF code chain
overflowed the JavaScript stack. Push slice members onto the same explicit
stack as tuple and frozenset members.

Co-Authored-By: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
2026-11-14 13:37:55 +01:00
gensokyo-devandgensokyo-dev 9d3eb083dd fix(python): pin only the 3.14 and 3.15 magics checked locally
Keep magic 3627 (CPython 3.14.6) and 3666 (CPython 3.15.0b3), whose code
layout and inline-cache tables match those interpreters' opcode tables and
whose decoded code objects match marshal.loads across a stdlib sample. Leave
the 3.15.0a8 magic 3663 unpinned, since no 3.15.0a8 interpreter was available
to check it, and regenerate the 3.15 slice fixture with 3.15.0b3.

Co-Authored-By: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
2026-11-14 13:37:55 +01:00
Abhishek Sharma 1d08253ff2 feat(python): decode CPython 3.14 and 3.15 pyc code objects
Decode marshal slice constants from magic 3608 onward and pin the 3.11 code
layout and inline-cache tables for the 3.14 and 3.15 magics.

Refs #8206
2026-11-14 13:37:55 +01:00
morlutoandCursor Agent 1e80a96094 fix(python): handle one-byte marshal strings and deep code references (#9895)
* fix(python): decode marshal one-byte strings as Latin-1

* fix(python): hash referenced code objects iteratively

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-11-11 00:44:54 -05:00
morluto 25d3c7237f fix(paths): enforce selected path contracts at admission (#9465) 2026-11-09 03:14:48 +08:00
吴佳浩Alben fe9797b487 feat(python): qualify a recognized 3.14/3.15 pyc header (#8927)
* feat(python): qualify a recognized 3.14/3.15 pyc header

Python 3.15 is the default `python3` on current Homebrew, but its pyc was rejected with
"Pyc magic 3666 is outside the pinned CPython 3.7–3.13 table" — a message that says
nothing about the file and reads as if the version were unknown. Issue #8206.

Two facts were wrong.

**3.15 was not in the identity table at all,** so `pythonBytecodeIdentity` reported it as
unknown even though the table already pins 3.14 (3627). 3666 is now pinned as 3.15.

**A known identity without a pinned code layout shared one refusal with an unknown
magic.** The two are different outcomes: every 3.7+ pyc carries the same packed 16-byte
header, so a recognized newer magic can still be qualified even when no marshal layout is
pinned for it. The parser now separates them:

    magic 3666 (3.15)  -> CPython 3.15 header is recognized (magic 3666, 16-byte header,
                          flags 0x0, timestamp-based), but no marshal code layout is
                          pinned for that version, so its code objects cannot be decoded.
                          Standalone marshal decoding is pinned for CPython 3.7–3.13.
    magic 1            -> Pyc magic 1 is not a pinned CPython identity, so its header
                          cannot be qualified; ...

The recognized case reports the observed version, the header shape, the PEP 552 flags and
whether the pyc is hash-based or timestamp-based, which is what an agent needs to decide
that a `.jsc`-style text-recovery attempt is pointless and that the file is a 3.15 pyc
rather than a corrupt one. It stays a typed `unsupported` failure: no pinned layout means
no decoded code object, so there is no partial success.

I did not guess a 3.14/3.15 marshal layout. The 3.11+ `TYPE_CODE` order is pinned through
3.13; asserting the same order holds for 3.14 or 3.15 without a real interpreter to
verify against would be an unsupported claim, which is why this change qualifies the
header instead of decoding it.

Docs and the tool description now say marshal decoding is pinned for 3.7–3.13 while the
identity table also recognizes 3.14 and 3.15.

Three regressions fail on the base commit.

* chore: retrigger title and readiness checks after the title was corrected

* test(python): pin the qualified header facts to the packed fields

Review finding. Every new case used `pythonPycHeader(magic)` with its default zero flags
and never asserted the reported flags or header mode, so a refusal that hardcoded
`flags 0x0, timestamp-based` would have passed all of them.

The cases now build the header from explicit fields and assert the facts that field
produces: `flags 0x0/timestamp-based` and `flags 0x1` or `0x3` with a PEP 552 source hash
for hash-based. A separate case parses the same magic in both modes and requires each
refusal to name its own mode and not the other.

Verified by hardcoding `flags 0x0` in the refusal: the two hash-based cases fail.
2026-11-06 15:36:22 +08:00
morluto d06ca092fb refactor(python): consolidate bytecode identity across readers
Share CPython magic framing, version and header facts while keeping standalone marshal coverage and PyInstaller reconstruction eligibility distinct. Preserve legacy headers, opaque unknowns and source bytes.

Validated 272 focused tests across source, CLI and MCP workflows, a 38-test reader recheck, and check:fast.
2026-10-27 02:09:52 +08:00
xjh 1081008c9d fix(python): preserve source boundaries in pyc previews (#6492)
Preserve source code-point prefix endpoints in the shared marshal UTF-8 decoder, including separately encoded surrogate values. Keep native TextDecoder spans, full validation, byte identity and the existing 80-unit budget. Update the PyInstaller consumer without changing its text contract.

Validation: 207 relevant source tests;24 affected compiled CLI/MCP/filesystem CI tests; independent CPython3.11.2 compiler/marshal oracle and real argparse;18 equivalent performance tasks; build/check passed. Codex reviewed7c0f7abc6b without major findings.

CI bypass authorized for demonstrated baseline-only failures:101 source and179 built cases reproduced on actual tested main ba9c5bc68. Cancelled shard remainder is unverified. See PR body and discussion for scope and evidence.

Fixes #6476
2026-10-24 06:42:13 +08:00
morluto 06032237c6 fix(python): preserve marshal scalar identity and decode limits 2026-10-24 02:35:21 +08:00
morluto f8f52f205a fix(python): preserve marshal names and packaged bytecode
Use one marshal surrogatepass decoder for PYZ directories and Python bytecode.
Validate ordinary UTF-8 spans with the native fatal decoder, preserving BOMs,
lone surrogates, raw bytes and marshal reference slots.

Distinguish legacy full-pyc CArchive modules from modern marshal-only modules
before reconstructing headers. Share typed source/codec failures between stream
output and completion while retaining native-read drain and cleanup ownership.

Validated focused producer, reader, lifecycle, CLI and MCP regressions, plus
public inventory/extraction of CPython 3.14-generated marshal tables and code.
No packaged application was executed. Document original versus derived bytes.

Fixes #6800
Fixes #6789
2026-10-24 01:24:43 +08:00
morluto d0a8e3c60d fix(python): read legacy TYPE_INT64 marshal values
Fixes #6609
2026-10-23 19:07:45 +08:00
morluto 3784a8e547 fix(python): compare code objects without inline caches
Fixes #6597
2026-10-23 18:31:56 +08:00
morluto c6ab965062 fix(python): preserve historical signed code fields
Keep signed co_stacksize and co_flags for historical magic entries whose constructors accepted them. Preserve magic 3410 with a constructor-revision limitation; reject negative values from 3411 onward.

Fixes #6487
2026-10-23 18:31:02 +08:00
morluto f921cb943e fix(python): preserve code-name identity
Fixes #6593
2026-10-23 18:25:56 +08:00
Wadeandmorluto cc99d896be fix(python): reject code keys and set members with unhashable consts (#6424)
* fix(python): reject code keys and set members with unhashable consts

* fix(python): refuse to hash a tuple that is still being read

* fix(python): hash a code object's co_consts tuple, not its item list

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-23 18:05:58 +08:00
morluto aee249b1dd fix(python): normalize legacy code flags for equality
Preserve CPython 3.7–3.10 constructor flag semantics in code keys.
2026-10-23 17:43:07 +08:00
morluto 81de82a1a4 fix(python): classify historical headers and signed flags
Fixes #6568, #6487
2026-10-23 17:43:07 +08:00
morluto fa56001f18 fix(python): align pyc and PYZ marshal decoding
Fixes #6560
Fixes #6559
2026-10-23 17:16:28 +08:00
morluto 60981e0801 fix(python): honor version-specific code equality
Fixes #6547
2026-10-23 17:04:03 +08:00
morluto b58aa63796 fix(python): preserve signed code-object line numbers
Fixes #6490
2026-10-23 16:31:49 +08:00
2c3dd946d5 feat(python): inspect CPython pyc headers and marshal code objects (#4826)
* feat(python): inspect CPython pyc headers and marshal code objects

Add inspect_python_pyc / inspect-python-pyc to read one explicit local
CPython 3.7–3.13 .pyc without executing bytecode or recovering source.

* chore(python): drop unused PythonPycHeader export

Knip reported the type as unused after inspect_python_pyc landed.

* fix(python): bound pyc expansion and decode marshal like CPython

- Count a nested code object reached through TYPE_REF once per copy the
  report carries, against max_objects and max_file_bytes, before counting
  or validating the tree
- Charge long digits to max_string_bytes, reject out-of-range and
  unnormalized digits, build the value in one pass, and report integers
  above 256 bits by bit length only
- Decode TYPE_UNICODE with surrogatepass semantics and reject other
  ill-formed UTF-8
- Map only 3.11 magics 3494-3495 to the py311 layout
- Report non-finite float and complex parts as inf, -inf or nan
- Allow an empty co_filename
- Require bytes for co_linetable and co_exceptiontable
- Bind cleanup-failure errors to their retained descriptor leases

* fix(python): list pyc code objects once instead of a recursive tree

Report each distinct code object once in code_objects, top-level first,
with nested_code as indexes. A child shared through TYPE_REF is no longer
copied per reference, and the MCP output schema is no longer recursive.
Names repeated through TYPE_REF still count against max_file_bytes for
each copy the report carries.

* fix(python): bound shared pyc constants and reject malformed tables

- Cache byte-string digests so a buffer shared through TYPE_REF is
  hashed once instead of on every reference.
- Charge every reported co_consts entry, including tables shared
  through TYPE_REF, against max_objects before summaries are built.
- Resolve the inspect-python-pyc path with absoluteCallerPath so
  link/../file.pyc reads the file the filesystem selects.
- Reject TYPE_NULL inside marshal sequences, as CPython does.
- Reject a co_localspluskinds length that does not match
  co_localsplusnames in the parser and the report schema.

* fix(python): bound shared pyc names and read CPython marshal graphs

Charge each reported code-constant name, keep integer bit lengths on the
decoded value, and list code objects nested inside constant containers.
Accept self-referential lists and dicts, marshal version 0 text floats,
empty code names, the original string byte count, and Python set and dict
lengths. Pass the CLI cancellation signal into command cleanup.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): bound pyc key expansion and shared nested code

Charge set and dictionary keys before a shared tuple doubles them, list each
nested code object once per owner, and accept code objects as CPython keys.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): correct pyc equality, counts, and nested walks

Summarize code objects only after the marshal graph is complete, and
collect nested code with an iterative walk that shares wrapper results
instead of copying every descendant list. Keep tuple boundaries and the
original string bytes in equality keys, and reject negative CPython counts.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): bound shared pyc names and keep the first dict key

Charge every copied name-table entry against the object budget before
building the report array, including empty strings that add no bytes.
When dictionary keys compare equal, keep the first key object and
replace only its value, matching CPython.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): walk each pyc dictionary child list once

Keep a container's child array on the nested-code frame so a wide
dictionary is not copied on every step of the synchronous marshal walk.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): reuse stored pyc dictionary children when combining

The nested walk keeps each container's child list on its frame.
Combining nested code rebuilt that list, and the 8,000-key regression
still accepted the old per-key copy. Use the stored list, and require
that dictionary to finish well under the copy's runtime while still
reporting a nested code object.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* fix(python): keep distinct pyc code objects and reject corrupt ones

Encode each name table's length and entry boundaries in the equality
key so a different name list cannot share a memo key. Walk tuple and
frozenset keys on an explicit stack so a long TYPE_REF chain does not
overflow. Treat code objects CPython rejects for an odd co_code or an
undersized local table as malformed instead of complete coverage.

Co-authored-by: morluto <morluto@users.noreply.github.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>
2026-10-22 19:00:08 +08:00