* refactor(domain): charge seven remaining hand-written allowances through Budget
Projection work, Kuna run slots, HTTP dispatch slots, marshal object counts, kept deleted environment entities, MCP operation output and route-scan facts and work now charge Budget instead of adjusting a field and comparing it with a limit. The budget-arithmetic baseline drops from 11 to 4; the four left are a scan cursor, a token string and two output tallies that admit then trim, which are not allowances.
* fix(process): omit the verifier transport's operation byte limit instead of passing Infinity
* refactor(domain): give strict UTF-8 decoding one owner with an explicit BOM policy
* fix(domain): keep the leading-BOM behaviour every migrated UTF-8 decoder had
* refactor(domain): read the domain layer's strict UTF-8 through the BOM-policy owner (#10803)
Twelve domain decoders built their own fatal TextDecoder with ignoreBOM: true. They now use strictUtf8Decoder("preserve"), which keeps that behaviour: each reads bytes where a leading U+FEFF is content (binary-format names and fields, JSON and XML read byte-for-byte, recorded bodies, registry seeds whose kept bytes are re-emitted). The counted baseline drops from 74 to 62.
* refactor(artifacts): read artifact readers' strict UTF-8 through the BOM-policy owner (#10871)
Fourteen artifact reader modules (17 sites) built their own fatal TextDecoder. They now take it from strictUtf8Decoder with the policy each had: every one kept a leading BOM (ignoreBOM: true), so they use "preserve"; JavaScriptArtifactFiles keeps it for JavaScript and HTML entries and strips it otherwise, as before. The strict-utf8-decoder baseline drops those 17 sites.
* refactor(domain): drop TextDecoder imports left by the strict UTF-8 migration
* refactor(application): read MCP registration metadata through the strict UTF-8 owner
* refactor(formats): read binary-format readers' strict UTF-8 through the BOM-policy owner (#10903)
* refactor(formats): read binary-format readers' strict UTF-8 through the BOM-policy owner
* refactor(formats): drop the model readers' unused TextDecoder imports
* refactor(browser): read browser capture and fetch strict UTF-8 through the BOM-policy owner (#10935)
* refactor(browser): read browser capture and fetch strict UTF-8 through the BOM-policy owner
* refactor(native): read LLDB and Mach-O name strict UTF-8 through the BOM-policy owner (#10936)
* refactor(native): read LLDB and Mach-O name strict UTF-8 through the BOM-policy owner
* refactor(process): read MCP and NDJSON transport frames' strict UTF-8 through the BOM-policy owner (#10937)
* refactor(process): read MCP and NDJSON transport frames' strict UTF-8 through the BOM-policy owner
* refactor(application): read JSON and file inputs' strict UTF-8 through the BOM-policy owner (#10938)
* refactor(application): read JSON and file inputs' strict UTF-8 through the BOM-policy owner
* refactor(providers): read IDA, Electron, Ghidra and reference strict UTF-8 through the BOM-policy owner (#10939)
* refactor(providers): read IDA, Electron, Ghidra and reference strict UTF-8 through the BOM-policy owner
* refactor(evm): read the hex bytecode carrier's strict UTF-8 through the owner and refuse new local decoders (#10940)
* refactor(evm): read the hex bytecode carrier's strict UTF-8 through the BOM-policy owner
* chore(architecture): match local strict UTF-8 decoders in any option order
* fix(plist): rethrow internal invariant violations from plist consumers
KeyedArchiveReader reported any non-limit decode failure as a format
failure, InterfaceBuilderAnalysis listed it as an invalid archive,
ReleaseExposureReader fell back to scanning raw bytes, and the native
plist parser reported it as provider output. An InternalInvariantError
is a REA defect, not evidence about the input, so each consumer now
rethrows it. The native parser's binary path now shares its one cause
projection instead of a duplicate.
* fix(native): rethrow internal invariant violations from the native output projection
nativeOutputFailure wrapped any producer failure as AnalysisOutputError,
so a REA defect thrown while reading a plist or dispatching a command was
reported as native output parsing. It now rethrows InternalInvariantError,
which reaches the provider's adapter-failure path with the invariant as
its cause.
* fix(artifacts): rethrow internal invariant violations from the shared artifact failure mapper
translateArtifactFailure turned every unrecognised cause into an
ArtifactOperationError with reason io, so the keyed-archive and
Interface Builder rethrows still reached users as artifact read
failures. The shared mapper now rethrows InternalInvariantError, and the
deferred client reports it as a ProviderAdapterError. Provider-level
tests drive both operations through that client.
* refactor(dotnet): report managed reader defects through invariantViolation (#11030)
* refactor(dotnet): report managed reader defects through invariantViolation
The managed readers threw plain TypeError and RangeError for states their
callers rule out: an allowance or method page outside the input schema, a
missing snapshot after identity admission, and a missing CLI header after
the caller already returned for it. These are REA defects, not malformed
targets, so they throw invariantViolation. The two CLI re-checks are gone:
the callers pass the CLI header they already narrowed. The dotnet
parser-plain-throw baseline drops from 6 to 0.
* refactor(providers): refuse unusable tool paths and record marker mismatches without throwing to their own catch (#11031)
* refactor(firmware): refuse unusable tool paths without throwing to its own catch
resolveFirmwareCommand threw a plain TypeError for a configured engine or
limiter path that is not a regular file, only to catch it in the next line
and wrap it as an unavailable capability. A helper now returns why a path
cannot run, and each caller builds the one refusal directly. Refusal
reasons are unchanged; the directory tests now pin them exactly. The
firmware parser-plain-throw baseline drops from 2 to 0.
* refactor(pwndbg): refuse unusable debugger paths without throwing to its own catch
inspectPwndbgCore threw plain Errors for a relative or wrong-kind
configured path only to catch them and wrap them as an unavailable
capability. A helper now returns why a path cannot be used, with any host
error as its cause, and the loop builds the one refusal. Cancellation
between host checks still propagates. Refusal reasons are unchanged. The
PwndbgCoreContext parser-plain-throw baseline entry is removed.
* refactor(pwntools): record a mismatched failure marker without throwing to its own catch
readPwntoolsFailureEvidence threw a plain Error when the private bridge
marker disagreed with the exit status, only to catch it and record its
message as the marker failure. The mismatch is now recorded directly. The
recorded evidence is unchanged; a unit test pins both the matching and
the mismatched marker. The PwntoolsResourceLimits parser-plain-throw
baseline entry is removed.
* fix(android): report an unobserved Java probe as a REA defect, not a Java refusal (#11032)
* fix(android): report an unobserved Java probe as a REA defect, not a Java refusal
When runOwnedCommand resolved without observing the completed run, the
probe threw a plain Error that its catch then classified as a Java
runtime refusal, telling the caller to review their launcher. The state
is a broken internal assumption, so the probe throws invariantViolation
and projects it as a jadx adapter failure. The JadxJavaProbe
parser-plain-throw baseline entry is removed.
* refactor(native): report native provider defects through invariantViolation (#11033)
* refactor(native): report native provider defects through invariantViolation
Native providers threw plain Error and TypeError for states their own
earlier steps rule out: a client with no target, a measured slice with no
selection, a read longer than requested, a decoder command without its
runtime, an LLDB root released before its children, and an IPS integer
fact from another reader. These are REA defects, so they throw
invariantViolation. NativeUiInterruptionControl no longer needs a throw:
recording the deadline interruption returns the interruption its error is
built from. Six native parser-plain-throw baseline entries drop.
* refactor(native): report malformed lipo, swift-demangle and UI helper output through ParseFailure (#11034)
* refactor(native): report malformed lipo and swift-demangle output through ParseFailure
The lipo and swift-demangle output parsers threw plain TypeError for
output that breaks their framing, and the lipo result classifier caught
every TypeError as malformed output, so a host TypeError from a REA defect
was reported as lipo output. Both parsers now throw malformedInput with
their own format, and the lipo classifier accepts ParseFailureError
instead of TypeError. Messages and the consumers' output and format
failure categories are unchanged. Four native parser-plain-throw
baseline entries drop.
* refactor(native): report the UI helper's non-JSON output through ParseFailure
The native UI helper runtime threw a plain Error when the helper's
stdout was not JSON. It now throws malformedInput with its own format and
the JSON failure as cause. The observation boundary still reports it as a
native-macos adapter failure with the same reason. The
NativeUiHelperRuntime parser-plain-throw baseline entry is removed.
* refactor(verify): keep process-lifecycle owners out of the parser plain-throw guard (#11035)
* refactor(verify): keep process-lifecycle owners out of the parser plain-throw guard
JadxWorkspaceRoot and PwntoolsDecoder sit in parser folders but own a
private helper process; their remaining plain throws report the helper's
startup, handshake and cleanup outcomes, not parse faults. The guard now
names them as process-lifecycle owners and leaves them out, so their
five baseline entries drop without inventing parse errors for process
failures.
* refactor(parsers): report the remaining artifact, il2cpp, model and domain defects through invariantViolation (#11036)
* refactor(parsers): report the remaining artifact, il2cpp, model and domain defects through invariantViolation
Nine plain Error, TypeError and RangeError throws in parser folders
reported broken internal assumptions or caller contracts (an
uninitialized ZIP or DMG reader, a missing scan result, a non-byte
stream chunk, an invalid traversal or marshal limit, a header count or
linking fact the reader itself should have produced, a misordered
Safetensors budget, an append after fingerprinting). None is an input
fault, and no catch site depends on their class. They now throw
invariantViolation with the same messages, and their parser-plain-throw
baseline entries drop.
* refactor(evidence): refuse an earlier Evidence format through ParseFailure (#11037)
* refactor(evidence): refuse an earlier Evidence format through ParseFailure
UnsupportedEvidenceFormatError restated the ParseFailure algebra as its
own TypeError subclass. The format owner now throws unsupportedInput for
the evidence-bundle format at the evidence_format field and exports the
predicate its consumers use. The bundle reader's unsupported-format kind
and message are unchanged. The snapshot reader surfaced only TypeError
messages, so it now also surfaces the format refusal and keeps its
recreate guidance; a boundary test pins that.
* refactor(native): report Mach-O reader and inspection defects through invariantViolation (#11042)
The Mach-O inspection plan reads the lipo, nm and dyld_info captures it
just ran, the dispatch metadata reader is called only for Mach-O
executables, and the command-name reader takes limits its callers
derive. A missing capture, a non-Mach-O target or a malformed limit is a
REA defect, not an input fault, so these six sites throw
invariantViolation with their existing messages instead of TypeError.
No catch site depends on their class. Their parser-plain-throw baseline
entries drop.
* fix(artifacts): join the Interface Builder cleanup failure to a catalog defect
When catalog work failed while the concurrent Interface Builder reader
failed to close, analyzeAppleAssetCatalogs built the primary through
translateArtifactFailure before joining cleanup. That mapper now rethrows
an InternalInvariantError, so a defect skipped the join and the reader's
retained resources were lost. The defect now becomes the artifact graph
provider's adapter failure at that site and joins the cleanup failure as
before.
* refactor(python): read pyc and PYZ marshal values through one shared reader
* refactor(python): read pyc 64-bit values and header fields through shared readers (#10834)
Keep magic 3627 (CPython 3.14.6) and 3666 (CPython 3.15.0b3), whose code
layout and inline-cache tables match those interpreters' opcode tables and
whose decoded code objects match marshal.loads across a stdlib sample. Leave
the 3.15.0a8 magic 3663 unpinned, since no 3.15.0a8 interpreter was available
to check it, and regenerate the 3.15 slice fixture with 3.15.0b3.
Co-Authored-By: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
* feat(python): qualify a recognized 3.14/3.15 pyc header
Python 3.15 is the default `python3` on current Homebrew, but its pyc was rejected with
"Pyc magic 3666 is outside the pinned CPython 3.7–3.13 table" — a message that says
nothing about the file and reads as if the version were unknown. Issue #8206.
Two facts were wrong.
**3.15 was not in the identity table at all,** so `pythonBytecodeIdentity` reported it as
unknown even though the table already pins 3.14 (3627). 3666 is now pinned as 3.15.
**A known identity without a pinned code layout shared one refusal with an unknown
magic.** The two are different outcomes: every 3.7+ pyc carries the same packed 16-byte
header, so a recognized newer magic can still be qualified even when no marshal layout is
pinned for it. The parser now separates them:
magic 3666 (3.15) -> CPython 3.15 header is recognized (magic 3666, 16-byte header,
flags 0x0, timestamp-based), but no marshal code layout is
pinned for that version, so its code objects cannot be decoded.
Standalone marshal decoding is pinned for CPython 3.7–3.13.
magic 1 -> Pyc magic 1 is not a pinned CPython identity, so its header
cannot be qualified; ...
The recognized case reports the observed version, the header shape, the PEP 552 flags and
whether the pyc is hash-based or timestamp-based, which is what an agent needs to decide
that a `.jsc`-style text-recovery attempt is pointless and that the file is a 3.15 pyc
rather than a corrupt one. It stays a typed `unsupported` failure: no pinned layout means
no decoded code object, so there is no partial success.
I did not guess a 3.14/3.15 marshal layout. The 3.11+ `TYPE_CODE` order is pinned through
3.13; asserting the same order holds for 3.14 or 3.15 without a real interpreter to
verify against would be an unsupported claim, which is why this change qualifies the
header instead of decoding it.
Docs and the tool description now say marshal decoding is pinned for 3.7–3.13 while the
identity table also recognizes 3.14 and 3.15.
Three regressions fail on the base commit.
* chore: retrigger title and readiness checks after the title was corrected
* test(python): pin the qualified header facts to the packed fields
Review finding. Every new case used `pythonPycHeader(magic)` with its default zero flags
and never asserted the reported flags or header mode, so a refusal that hardcoded
`flags 0x0, timestamp-based` would have passed all of them.
The cases now build the header from explicit fields and assert the facts that field
produces: `flags 0x0/timestamp-based` and `flags 0x1` or `0x3` with a PEP 552 source hash
for hash-based. A separate case parses the same magic in both modes and requires each
refusal to name its own mode and not the other.
Verified by hardcoding `flags 0x0` in the refusal: the two hash-based cases fail.
Share CPython magic framing, version and header facts while keeping standalone marshal coverage and PyInstaller reconstruction eligibility distinct. Preserve legacy headers, opaque unknowns and source bytes.
Validated 272 focused tests across source, CLI and MCP workflows, a 38-test reader recheck, and check:fast.
Preserve source code-point prefix endpoints in the shared marshal UTF-8 decoder, including separately encoded surrogate values. Keep native TextDecoder spans, full validation, byte identity and the existing 80-unit budget. Update the PyInstaller consumer without changing its text contract.
Validation: 207 relevant source tests;24 affected compiled CLI/MCP/filesystem CI tests; independent CPython3.11.2 compiler/marshal oracle and real argparse;18 equivalent performance tasks; build/check passed. Codex reviewed7c0f7abc6b without major findings.
CI bypass authorized for demonstrated baseline-only failures:101 source and179 built cases reproduced on actual tested main ba9c5bc68. Cancelled shard remainder is unverified. See PR body and discussion for scope and evidence.
Fixes#6476
Use one marshal surrogatepass decoder for PYZ directories and Python bytecode.
Validate ordinary UTF-8 spans with the native fatal decoder, preserving BOMs,
lone surrogates, raw bytes and marshal reference slots.
Distinguish legacy full-pyc CArchive modules from modern marshal-only modules
before reconstructing headers. Share typed source/codec failures between stream
output and completion while retaining native-read drain and cleanup ownership.
Validated focused producer, reader, lifecycle, CLI and MCP regressions, plus
public inventory/extraction of CPython 3.14-generated marshal tables and code.
No packaged application was executed. Document original versus derived bytes.
Fixes#6800Fixes#6789
Keep signed co_stacksize and co_flags for historical magic entries whose constructors accepted them. Preserve magic 3410 with a constructor-revision limitation; reject negative values from 3411 onward.
Fixes#6487
* feat(python): inspect CPython pyc headers and marshal code objects
Add inspect_python_pyc / inspect-python-pyc to read one explicit local
CPython 3.7–3.13 .pyc without executing bytecode or recovering source.
* chore(python): drop unused PythonPycHeader export
Knip reported the type as unused after inspect_python_pyc landed.
* fix(python): bound pyc expansion and decode marshal like CPython
- Count a nested code object reached through TYPE_REF once per copy the
report carries, against max_objects and max_file_bytes, before counting
or validating the tree
- Charge long digits to max_string_bytes, reject out-of-range and
unnormalized digits, build the value in one pass, and report integers
above 256 bits by bit length only
- Decode TYPE_UNICODE with surrogatepass semantics and reject other
ill-formed UTF-8
- Map only 3.11 magics 3494-3495 to the py311 layout
- Report non-finite float and complex parts as inf, -inf or nan
- Allow an empty co_filename
- Require bytes for co_linetable and co_exceptiontable
- Bind cleanup-failure errors to their retained descriptor leases
* fix(python): list pyc code objects once instead of a recursive tree
Report each distinct code object once in code_objects, top-level first,
with nested_code as indexes. A child shared through TYPE_REF is no longer
copied per reference, and the MCP output schema is no longer recursive.
Names repeated through TYPE_REF still count against max_file_bytes for
each copy the report carries.
* fix(python): bound shared pyc constants and reject malformed tables
- Cache byte-string digests so a buffer shared through TYPE_REF is
hashed once instead of on every reference.
- Charge every reported co_consts entry, including tables shared
through TYPE_REF, against max_objects before summaries are built.
- Resolve the inspect-python-pyc path with absoluteCallerPath so
link/../file.pyc reads the file the filesystem selects.
- Reject TYPE_NULL inside marshal sequences, as CPython does.
- Reject a co_localspluskinds length that does not match
co_localsplusnames in the parser and the report schema.
* fix(python): bound shared pyc names and read CPython marshal graphs
Charge each reported code-constant name, keep integer bit lengths on the
decoded value, and list code objects nested inside constant containers.
Accept self-referential lists and dicts, marshal version 0 text floats,
empty code names, the original string byte count, and Python set and dict
lengths. Pass the CLI cancellation signal into command cleanup.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(python): bound pyc key expansion and shared nested code
Charge set and dictionary keys before a shared tuple doubles them, list each
nested code object once per owner, and accept code objects as CPython keys.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(python): correct pyc equality, counts, and nested walks
Summarize code objects only after the marshal graph is complete, and
collect nested code with an iterative walk that shares wrapper results
instead of copying every descendant list. Keep tuple boundaries and the
original string bytes in equality keys, and reject negative CPython counts.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(python): bound shared pyc names and keep the first dict key
Charge every copied name-table entry against the object budget before
building the report array, including empty strings that add no bytes.
When dictionary keys compare equal, keep the first key object and
replace only its value, matching CPython.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(python): walk each pyc dictionary child list once
Keep a container's child array on the nested-code frame so a wide
dictionary is not copied on every step of the synchronous marshal walk.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(python): reuse stored pyc dictionary children when combining
The nested walk keeps each container's child list on its frame.
Combining nested code rebuilt that list, and the 8,000-key regression
still accepted the old per-key copy. Use the stored list, and require
that dictionary to finish well under the copy's runtime while still
reporting a nested code object.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(python): keep distinct pyc code objects and reject corrupt ones
Encode each name table's length and entry boundaries in the equality
key so a different name list cannot share a memo key. Walk tuple and
frozenset keys on an explicit stack so a long TYPE_REF chain does not
overflow. Treat code objects CPython rejects for an odd co_code or an
undersized local table as malformed instead of complete coverage.
Co-authored-by: morluto <morluto@users.noreply.github.com>
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>