mirror of
https://github.com/morluto/rea.git
synced 2026-10-26 17:35:11 +00:00
feat(python): inspect CPython pyc headers and marshal code objects (#4826)
* feat(python): inspect CPython pyc headers and marshal code objects Add inspect_python_pyc / inspect-python-pyc to read one explicit local CPython 3.7–3.13 .pyc without executing bytecode or recovering source. * chore(python): drop unused PythonPycHeader export Knip reported the type as unused after inspect_python_pyc landed. * fix(python): bound pyc expansion and decode marshal like CPython - Count a nested code object reached through TYPE_REF once per copy the report carries, against max_objects and max_file_bytes, before counting or validating the tree - Charge long digits to max_string_bytes, reject out-of-range and unnormalized digits, build the value in one pass, and report integers above 256 bits by bit length only - Decode TYPE_UNICODE with surrogatepass semantics and reject other ill-formed UTF-8 - Map only 3.11 magics 3494-3495 to the py311 layout - Report non-finite float and complex parts as inf, -inf or nan - Allow an empty co_filename - Require bytes for co_linetable and co_exceptiontable - Bind cleanup-failure errors to their retained descriptor leases * fix(python): list pyc code objects once instead of a recursive tree Report each distinct code object once in code_objects, top-level first, with nested_code as indexes. A child shared through TYPE_REF is no longer copied per reference, and the MCP output schema is no longer recursive. Names repeated through TYPE_REF still count against max_file_bytes for each copy the report carries. * fix(python): bound shared pyc constants and reject malformed tables - Cache byte-string digests so a buffer shared through TYPE_REF is hashed once instead of on every reference. - Charge every reported co_consts entry, including tables shared through TYPE_REF, against max_objects before summaries are built. - Resolve the inspect-python-pyc path with absoluteCallerPath so link/../file.pyc reads the file the filesystem selects. - Reject TYPE_NULL inside marshal sequences, as CPython does. - Reject a co_localspluskinds length that does not match co_localsplusnames in the parser and the report schema. * fix(python): bound shared pyc names and read CPython marshal graphs Charge each reported code-constant name, keep integer bit lengths on the decoded value, and list code objects nested inside constant containers. Accept self-referential lists and dicts, marshal version 0 text floats, empty code names, the original string byte count, and Python set and dict lengths. Pass the CLI cancellation signal into command cleanup. Co-authored-by: morluto <morluto@users.noreply.github.com> * fix(python): bound pyc key expansion and shared nested code Charge set and dictionary keys before a shared tuple doubles them, list each nested code object once per owner, and accept code objects as CPython keys. Co-authored-by: morluto <morluto@users.noreply.github.com> * fix(python): correct pyc equality, counts, and nested walks Summarize code objects only after the marshal graph is complete, and collect nested code with an iterative walk that shares wrapper results instead of copying every descendant list. Keep tuple boundaries and the original string bytes in equality keys, and reject negative CPython counts. Co-authored-by: morluto <morluto@users.noreply.github.com> * fix(python): bound shared pyc names and keep the first dict key Charge every copied name-table entry against the object budget before building the report array, including empty strings that add no bytes. When dictionary keys compare equal, keep the first key object and replace only its value, matching CPython. Co-authored-by: morluto <morluto@users.noreply.github.com> * fix(python): walk each pyc dictionary child list once Keep a container's child array on the nested-code frame so a wide dictionary is not copied on every step of the synchronous marshal walk. Co-authored-by: morluto <morluto@users.noreply.github.com> * fix(python): reuse stored pyc dictionary children when combining The nested walk keeps each container's child list on its frame. Combining nested code rebuilt that list, and the 8,000-key regression still accepted the old per-key copy. Use the stored list, and require that dictionary to finish well under the copy's runtime while still reporting a nested code object. Co-authored-by: morluto <morluto@users.noreply.github.com> * fix(python): keep distinct pyc code objects and reject corrupt ones Encode each name table's length and entry boundaries in the equality key so a different name list cannot share a memo key. Walk tuple and frozenset keys on an explicit stack so a long TYPE_REF chain does not overflow. Treat code objects CPython rejects for an odd co_code or an undersized local table as malformed instead of complete coverage. Co-authored-by: morluto <morluto@users.noreply.github.com> --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: morluto <morluto@users.noreply.github.com>
This commit is contained in:
36 files changed
+4457
-2
No files matched your search
@@ -233,6 +233,13 @@ explicit path or endpoint and do not need it.
|
||||
build-id or ABI-tag fields. A missing table is observed absence of those
|
||||
declarations, not proof the image never carried notes. See the
|
||||
[ELF notes guide](https://github.com/morluto/rea/blob/main/docs/elf-notes.md).
|
||||
- CPython 3.7–3.13 `.pyc` header, nested marshal code objects and bytecode
|
||||
identity: `inspect_python_pyc` with the explicit local file. This built-in
|
||||
target-free inspection requires no Python installation. Preserve magic,
|
||||
PEP 552 hash or mtime/size, nested names, and bytecode SHA-256. Bytecode is
|
||||
not decompiled, and a decoded header does not prove the matching `.py` is
|
||||
present. See the
|
||||
[CPython pyc guide](https://github.com/morluto/rea/blob/main/docs/python-pyc.md).
|
||||
- Erlang BEAM IFF container: `inspect_erlang_beam` with the explicit local
|
||||
`.beam` file. This built-in target-free inspection requires no Erlang
|
||||
installation. Preserve chunk names, atom order, export/import/lambda tables,
|
||||
|
||||
@@ -121,6 +121,16 @@ absence of those declarations, not proof the image never carried notes in
|
||||
another container. See
|
||||
[ELF note tables](https://github.com/morluto/rea/blob/main/docs/elf-notes.md).
|
||||
|
||||
## CPython pyc files
|
||||
|
||||
Use `inspect_python_pyc` for an explicit local CPython 3.7–3.13 `.pyc` when the
|
||||
question concerns the header magic, PEP 552 source hash or timestamp, nested
|
||||
code-object names, or bytecode identity. No `open_binary`, Python installation,
|
||||
or target execution is required. The CLI equivalent is `inspect-python-pyc PATH`.
|
||||
Preserve artifact SHA-256, the 16-byte header fields, nested code names, and
|
||||
bytecode SHA-256. Bytecode is not decompiled. A decoded header does not prove
|
||||
the matching `.py` is present. See [CPython pyc inspection](https://github.com/morluto/rea/blob/main/docs/python-pyc.md).
|
||||
|
||||
## Erlang BEAM files
|
||||
|
||||
Use `inspect_erlang_beam` for an explicit local `FOR1`/`BEAM` file when the
|
||||
|
||||
@@ -34,6 +34,7 @@ export default defineConfig({
|
||||
{ text: "Rust dependencies", link: "/rust-binaries" },
|
||||
{ text: "PE certificates", link: "/pe-certificates" },
|
||||
{ text: "ELF notes", link: "/elf-notes" },
|
||||
{ text: "CPython pyc files", link: "/python-pyc" },
|
||||
{ text: "Erlang BEAM", link: "/erlang-beam" },
|
||||
{
|
||||
text: "JavaScript and Electron",
|
||||
|
||||
+11
@@ -177,6 +177,17 @@ The workflow reports note owners, types, descriptor hashes, and GNU build-id
|
||||
or ABI-tag fields when those descriptors match. A missing note table is
|
||||
observed absence of those declarations. See [ELF note tables](elf-notes.md).
|
||||
|
||||
Inspect a CPython 3.7–3.13 `.pyc` header and nested marshal code objects
|
||||
without executing bytecode or installing Python:
|
||||
|
||||
```bash
|
||||
rea inspect-python-pyc /absolute/path/to/module.pyc --json
|
||||
```
|
||||
|
||||
The workflow reports the pinned CPython version for the magic, PEP 552 hash or
|
||||
mtime/size fields, nested function names, and bytecode SHA-256. Bytecode is
|
||||
not decompiled. See [CPython pyc inspection](python-pyc.md).
|
||||
|
||||
Inspect an Erlang BEAM file without executing bytecode:
|
||||
|
||||
```bash
|
||||
|
||||
+1
-1
@@ -50,7 +50,7 @@ For runtime instrumentation, see the [Frida guide](./frida-instrumentation.md).
|
||||
| Task | Guides |
|
||||
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Identify a desktop installation | [Runtime candidates, evidence and follow-up files](desktop-applications.md) |
|
||||
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [PE certificates](pe-certificates.md), [ELF notes](elf-notes.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
|
||||
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [ELF notes](elf-notes.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
|
||||
| Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [Kuna](kuna-provider.md), [JEB](jeb-analysis.md) |
|
||||
| Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) |
|
||||
| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) |
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
# CPython pyc inspection
|
||||
|
||||
`inspect_python_pyc` / `inspect-python-pyc` answers which CPython version wrote
|
||||
one selected local `.pyc` file, how its 16-byte header records source identity,
|
||||
and which nested code objects marshal stored. It reads the file without running
|
||||
bytecode and requires no Python installation, active binary target, Hopper,
|
||||
Ghidra, or IDA.
|
||||
|
||||
```bash
|
||||
rea inspect-python-pyc /absolute/path/to/module.pyc --json
|
||||
```
|
||||
|
||||
The CLI resolves relative paths against the caller's working directory. MCP
|
||||
requires an absolute filesystem path and preserves that exact selection:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "inspect_python_pyc",
|
||||
"arguments": { "path": "/artifacts/module.pyc" }
|
||||
}
|
||||
```
|
||||
|
||||
## Observations
|
||||
|
||||
The inline Evidence result contains the artifact path, SHA-256, and byte length.
|
||||
`header` contains:
|
||||
|
||||
- `magic`, the 16-bit little-endian field;
|
||||
- `python_version` and `code_layout`, mapped from the pinned CPython 3.7–3.13
|
||||
table (`py37`, `py38`, or `py311`); 3.11 alpha magics before 3494 used
|
||||
other code layouts and are unsupported;
|
||||
- `flags`, `hash_based`, and `check_source` from PEP 552;
|
||||
- `source_hash_hex` for hash-based pycs, or `source_mtime` and `source_size`
|
||||
for timestamp pycs.
|
||||
|
||||
`code_objects` lists each distinct marshal code object once, the top-level
|
||||
module code first. `nested_code` holds the indexes of the distinct code objects
|
||||
reachable from that object's constants, including code nested in tuples and
|
||||
other containers and the owner itself when a constant container reaches it.
|
||||
A function shared through a marshal reference is named by index once per owner,
|
||||
even when that owner reaches it again through the same container. Relationships
|
||||
are collected after the marshal graph is complete, so a container that a nested
|
||||
code object also references is not summarized while its items are still missing.
|
||||
Each object reports name, optional qualname,
|
||||
filename,
|
||||
argument counts, flags, names, locals (or `localsplusnames` on 3.11+), const
|
||||
summaries, and bytecode length plus SHA-256.
|
||||
|
||||
Bytecode is not decompiled. Const strings may be truncated in the summary while
|
||||
the marshal walk still consumed the original bytes; a string summary's `bytes`
|
||||
is that marshal length. Integer consts report `bits`; their decimal `value` is
|
||||
`null` above 256 bits. Non-finite float and complex parts use CPython's `inf`,
|
||||
`-inf`, and `nan` spellings. Dictionary and set lengths follow CPython:
|
||||
unhashable keys and members fail the walk, equal set members count once, equal
|
||||
dictionary keys keep the first key object and replace only its value, code
|
||||
objects are valid keys and members, and a null dictionary value ends the dict
|
||||
without that entry.
|
||||
|
||||
## Limits and failures
|
||||
|
||||
The selected file is limited to 16 MiB by default (64 MiB maximum). Marshal
|
||||
object count, nesting depth, and per-string size are bounded; exceeding a
|
||||
budget fails without partial success. Argument counts, positional-only counts,
|
||||
keyword-only counts, local counts, and stack size are signed CPython fields; a
|
||||
negative encoding is malformed. On the 3.11+ layout, `co_code` must cover a
|
||||
whole number of code units. Every layout rejects argument counts that do not
|
||||
fit the local table CPython checks (`co_varnames`, including `*args` /
|
||||
`**kwargs`, or fast locals on 3.11+). Those files are malformed, so the
|
||||
workflow does not report complete coverage. 3.7–3.10 still accept an odd
|
||||
`co_code`, matching those constructors. Nested code collection walks reference
|
||||
chains iteratively and keeps one shared descendant list for a container that
|
||||
merely wraps another, instead of copying that list onto every wrapper. Set and
|
||||
dictionary keys walk tuple and frozenset reference chains the same way, so a
|
||||
long `TYPE_REF` chain stays within the call stack. A shared child stored twice
|
||||
still counts its expanded key against the string budget. Code-object equality
|
||||
keys record each name table's length and each entry's boundaries, so `()` and
|
||||
`("",)` stay distinct, and `co_varnames`, `co_freevars`, and `co_cellvars` stay
|
||||
separate tables. Names repeated through a marshal reference, including each
|
||||
code-constant name the report copies, count against the file budget once per
|
||||
copy. Empty strings add no bytes, so every copied
|
||||
name-table entry (`co_names`, varnames, freevars, cellvars, and
|
||||
`localsplusnames`) counts against the object budget before its array is built.
|
||||
Copied constant tables and each reported nested code relationship count
|
||||
against that same object budget. Integer digits and expanded
|
||||
set or dictionary keys, including a shared tuple repeated at each level, count
|
||||
against the string budget. Magics outside the pinned 3.7–3.13 table,
|
||||
truncated headers, trailing marshal bytes, and a non-code top-level object are
|
||||
typed failures.
|
||||
|
||||
A decoded header does not prove the matching `.py` is present. `python_version`
|
||||
is the magic table, not a running interpreter. This workflow does not recover
|
||||
source.
|
||||
@@ -38,6 +38,7 @@ export const toolFamilyCatalog = (sources) => {
|
||||
...sources.il2cppContracts.IL2CPP_TOOL_CONTRACTS,
|
||||
...sources.modelContracts.MODEL_TOOL_CONTRACTS,
|
||||
...sources.rustContracts.RUST_TOOL_CONTRACTS,
|
||||
...sources.pythonPycContracts.PYTHON_PYC_TOOL_CONTRACTS,
|
||||
...sources.erlangContracts.ERLANG_BEAM_TOOL_CONTRACTS,
|
||||
...sources.wasmContracts.WASM_TOOL_CONTRACTS,
|
||||
...sources.sqliteContracts.SQLITE_TOOL_CONTRACTS,
|
||||
@@ -317,6 +318,10 @@ export const providerCatalog = (sources) => {
|
||||
identity: sources.rustProvider.RUST_BINARY_PROVIDER_IDENTITY,
|
||||
contracts: sources.rustContracts.RUST_TOOL_CONTRACTS,
|
||||
},
|
||||
{
|
||||
identity: sources.pythonPycProvider.PYTHON_PYC_PROVIDER_IDENTITY,
|
||||
contracts: sources.pythonPycContracts.PYTHON_PYC_TOOL_CONTRACTS,
|
||||
},
|
||||
{
|
||||
identity: sources.erlangProvider.ERLANG_BEAM_PROVIDER_IDENTITY,
|
||||
contracts: sources.erlangContracts.ERLANG_BEAM_TOOL_CONTRACTS,
|
||||
|
||||
@@ -77,6 +77,8 @@ const SOURCE_PATHS = {
|
||||
il2cppProvider: "dist/il2cpp/Il2cppMetadataProvider.js",
|
||||
rustContracts: "dist/contracts/rust/rustToolContracts.js",
|
||||
rustProvider: "dist/rust/RustBinaryProvider.js",
|
||||
pythonPycContracts: "dist/contracts/python/pythonPycToolContracts.js",
|
||||
pythonPycProvider: "dist/python/PythonPycProvider.js",
|
||||
erlangContracts: "dist/contracts/erlang/beamToolContracts.js",
|
||||
erlangProvider: "dist/erlang/BeamProvider.js",
|
||||
artifactContracts: "dist/contracts/artifactToolContracts.js",
|
||||
|
||||
@@ -11,6 +11,7 @@ const MIGRATED_PROVIDER_ROOTS = new Set([
|
||||
"il2cpp",
|
||||
"model",
|
||||
"gpu",
|
||||
"python",
|
||||
"rust",
|
||||
"wasm",
|
||||
"captures",
|
||||
|
||||
@@ -17,7 +17,7 @@ export const SOURCE_TEST_PROJECTS = [
|
||||
group: "source",
|
||||
include: [
|
||||
"src/*.test.ts",
|
||||
"src/{android,apktool,jeb,adb,artifacts,browser,config,dotnet,electron,erlang,evm,firmware,flutter,pcap,ghidra,go,gpu,hopper,ida,inspector,javascript,native,process,reference,rust,server,sqlite,wasm,windows}/**/*.test.ts",
|
||||
"src/{android,apktool,jeb,adb,artifacts,browser,config,dotnet,electron,erlang,evm,firmware,flutter,pcap,ghidra,go,gpu,hopper,ida,inspector,javascript,native,process,python,reference,rust,server,sqlite,wasm,windows}/**/*.test.ts",
|
||||
],
|
||||
},
|
||||
{
|
||||
|
||||
@@ -418,6 +418,7 @@ const workflowAvailabilityFor = ({
|
||||
name === "inspect_model_artifact" ||
|
||||
name === "inspect_gpu_code_objects" ||
|
||||
name === "inspect_rust_binary" ||
|
||||
name === "inspect_python_pyc" ||
|
||||
name === "inspect_erlang_beam" ||
|
||||
name === "inspect_electron_fuses"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
import type {
|
||||
ExecutionOptions,
|
||||
ProviderIdentity,
|
||||
} from "../AnalysisProvider.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import type {
|
||||
InspectPythonPycInput,
|
||||
PythonPyc,
|
||||
} from "../../domain/python/pythonPyc.js";
|
||||
import type { Result } from "../../domain/result.js";
|
||||
|
||||
/** Stable read and CPython pyc decoding, independent of a disassembler session. */
|
||||
export interface PythonPycPort {
|
||||
readonly identity: ProviderIdentity;
|
||||
close?(): Promise<void>;
|
||||
inspect(
|
||||
input: InspectPythonPycInput,
|
||||
options?: ExecutionOptions,
|
||||
): Promise<Result<PythonPyc, AnalysisError>>;
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
import { expect, it } from "vitest";
|
||||
import {
|
||||
AnalysisInputError,
|
||||
AnalysisCancelledError,
|
||||
} from "../../domain/analysisErrorCore.js";
|
||||
import type { PythonPyc } from "../../domain/python/pythonPyc.js";
|
||||
import { ok } from "../../domain/result.js";
|
||||
import { PythonPycService } from "./PythonPycService.js";
|
||||
|
||||
const identity = {
|
||||
id: "fixture-python-pyc",
|
||||
name: "Source-owned pyc reader seam",
|
||||
version: "1",
|
||||
};
|
||||
|
||||
const report: PythonPyc = {
|
||||
artifact: {
|
||||
path: "/artifacts/sample.pyc",
|
||||
sha256: "a".repeat(64),
|
||||
bytes: 64,
|
||||
},
|
||||
header: {
|
||||
magic: 3531,
|
||||
python_version: "3.12",
|
||||
code_layout: "py311",
|
||||
flags: 0,
|
||||
hash_based: false,
|
||||
check_source: false,
|
||||
source_hash_hex: null,
|
||||
source_mtime: 1,
|
||||
source_size: 10,
|
||||
},
|
||||
code_objects: [
|
||||
{
|
||||
name: "<module>",
|
||||
qualname: "<module>",
|
||||
filename: "sample.py",
|
||||
firstlineno: 1,
|
||||
argcount: 0,
|
||||
posonlyargcount: 0,
|
||||
kwonlyargcount: 0,
|
||||
nlocals: null,
|
||||
stacksize: 1,
|
||||
flags: 0,
|
||||
names: [],
|
||||
varnames: null,
|
||||
freevars: null,
|
||||
cellvars: null,
|
||||
localsplusnames: [],
|
||||
localspluskinds_hex: "",
|
||||
bytecode: { bytes: 2, sha256: "b".repeat(64) },
|
||||
consts: [{ kind: "none" }],
|
||||
nested_code: [],
|
||||
},
|
||||
],
|
||||
coverage: {
|
||||
status: "complete",
|
||||
objects: 4,
|
||||
code_objects: 1,
|
||||
marshal_bytes: 48,
|
||||
marshal_depth: 1,
|
||||
},
|
||||
limitations: [
|
||||
"Bytecode is retained as length and SHA-256 only; this workflow does not decompile, disassemble, or recover source.",
|
||||
],
|
||||
};
|
||||
|
||||
it("rejects a relative path and extra arguments before reading", async () => {
|
||||
let calls = 0;
|
||||
const service = new PythonPycService({
|
||||
identity,
|
||||
inspect: () => {
|
||||
calls += 1;
|
||||
return Promise.resolve(ok(report));
|
||||
},
|
||||
});
|
||||
const relative = await service.inspect({ path: "sample.pyc" });
|
||||
const extra = await service.inspect({
|
||||
path: report.artifact.path,
|
||||
detail: "complete",
|
||||
});
|
||||
expect(relative.ok).toBe(false);
|
||||
expect(extra.ok).toBe(false);
|
||||
if (!relative.ok) expect(relative.error).toBeInstanceOf(AnalysisInputError);
|
||||
if (!extra.ok) expect(extra.error).toBeInstanceOf(AnalysisInputError);
|
||||
expect(calls).toBe(0);
|
||||
});
|
||||
|
||||
it("records the selected artifact and the header plus marshal ranges", async () => {
|
||||
const result = await new PythonPycService({
|
||||
identity,
|
||||
inspect: () => Promise.resolve(ok(report)),
|
||||
}).inspect({ path: report.artifact.path });
|
||||
expect(result.ok).toBe(true);
|
||||
if (!result.ok) throw result.error;
|
||||
expect(result.value).toMatchObject({
|
||||
confidence: "observed",
|
||||
subject: { format: "file" },
|
||||
normalized_result: report,
|
||||
});
|
||||
expect(result.value.locations).toEqual([
|
||||
{ kind: "artifact-path", path: report.artifact.path },
|
||||
{ kind: "file-offset-range", start: 0, end: 16 },
|
||||
{ kind: "file-offset-range", start: 16, end: report.artifact.bytes },
|
||||
]);
|
||||
});
|
||||
|
||||
it("discards an observation if the caller cancels while the provider completes", async () => {
|
||||
const controller = new AbortController();
|
||||
const service = new PythonPycService({
|
||||
identity,
|
||||
inspect: () => {
|
||||
controller.abort();
|
||||
return Promise.resolve(ok(report));
|
||||
},
|
||||
});
|
||||
const result = await service.inspect(
|
||||
{ path: report.artifact.path },
|
||||
{ signal: controller.signal },
|
||||
);
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok) expect(result.error).toBeInstanceOf(AnalysisCancelledError);
|
||||
});
|
||||
@@ -0,0 +1,102 @@
|
||||
import { isAbsolute } from "node:path";
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import type { PythonPycPort } from "./PythonPycPort.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import {
|
||||
AnalysisCancelledError,
|
||||
AnalysisInputError,
|
||||
AnalysisOutputError,
|
||||
} from "../../domain/analysisErrorCore.js";
|
||||
import { createEvidence, type Evidence } from "../../domain/evidence.js";
|
||||
import {
|
||||
inspectPythonPycInputSchema,
|
||||
PYTHON_PYC_HEADER_BYTES,
|
||||
pythonPycSchema,
|
||||
} from "../../domain/python/pythonPyc.js";
|
||||
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
|
||||
const OPERATION = "inspect_python_pyc";
|
||||
|
||||
/** Shared caller validation, output parsing and Evidence provenance for CLI and MCP. */
|
||||
export class PythonPycService {
|
||||
constructor(readonly provider: PythonPycPort) {}
|
||||
|
||||
close(): Promise<void> {
|
||||
return this.provider.close?.() ?? Promise.resolve();
|
||||
}
|
||||
|
||||
async inspect(
|
||||
rawInput: unknown,
|
||||
options?: ExecutionOptions,
|
||||
): Promise<Result<Evidence, AnalysisError>> {
|
||||
if (options?.signal?.aborted)
|
||||
return err(new AnalysisCancelledError(OPERATION));
|
||||
const input = inspectPythonPycInputSchema.safeParse(rawInput);
|
||||
if (!input.success)
|
||||
return err(
|
||||
analysisInputErrorFromIssues(OPERATION, input.error.issues, rawInput, {
|
||||
cause: input.error,
|
||||
}),
|
||||
);
|
||||
if (!isAbsolute(input.data.path))
|
||||
return err(
|
||||
new AnalysisInputError(OPERATION, undefined, [
|
||||
{
|
||||
path: ["path"],
|
||||
reason: "invalid_format",
|
||||
message: "Expected an absolute filesystem path on this host.",
|
||||
},
|
||||
]),
|
||||
);
|
||||
const result = await this.provider.inspect(input.data, options);
|
||||
if (!result.ok) return result;
|
||||
if (options?.signal?.aborted)
|
||||
return err(new AnalysisCancelledError(OPERATION));
|
||||
const report = pythonPycSchema.safeParse(result.value);
|
||||
if (
|
||||
!report.success ||
|
||||
report.data.artifact.path !== input.data.path ||
|
||||
report.data.artifact.bytes > input.data.max_file_bytes ||
|
||||
report.data.coverage.objects > input.data.max_objects ||
|
||||
report.data.coverage.marshal_depth > input.data.max_depth
|
||||
)
|
||||
return err(
|
||||
new AnalysisOutputError(
|
||||
OPERATION,
|
||||
"Python pyc provider returned malformed output or changed artifact identity.",
|
||||
),
|
||||
);
|
||||
const value = report.data;
|
||||
return ok(
|
||||
createEvidence(
|
||||
{
|
||||
path: value.artifact.path,
|
||||
format: "file",
|
||||
sha256: value.artifact.sha256,
|
||||
},
|
||||
this.provider.identity,
|
||||
{
|
||||
operation: OPERATION,
|
||||
parameters: input.data,
|
||||
result: value,
|
||||
confidence: "observed",
|
||||
limitations: value.limitations,
|
||||
locations: [
|
||||
{ kind: "artifact-path", path: value.artifact.path },
|
||||
{
|
||||
kind: "file-offset-range",
|
||||
start: 0,
|
||||
end: PYTHON_PYC_HEADER_BYTES,
|
||||
},
|
||||
{
|
||||
kind: "file-offset-range",
|
||||
start: PYTHON_PYC_HEADER_BYTES,
|
||||
end: value.artifact.bytes,
|
||||
},
|
||||
],
|
||||
},
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import { registerGoCommands } from "./cli/goCommands.js";
|
||||
import { registerModelCommands } from "./cli/modelCommands.js";
|
||||
import { registerIl2cppCommands } from "./cli/il2cppCommands.js";
|
||||
import { registerRustCommands } from "./cli/rustCommands.js";
|
||||
import { registerPythonPycCommands } from "./cli/pythonPycCommands.js";
|
||||
import { registerErlangCommands } from "./cli/erlangCommands.js";
|
||||
import { registerSqliteCommands } from "./cli/sqliteCommands.js";
|
||||
import { registerAnalysisViewCommands } from "./cli/analysisViewCommands.js";
|
||||
@@ -140,6 +141,7 @@ export const createCli = (
|
||||
registerModelCommands(cli, logger);
|
||||
registerGpuCommands(cli, logger);
|
||||
registerRustCommands(cli, logger);
|
||||
registerPythonPycCommands(cli, logger);
|
||||
registerErlangCommands(cli, logger);
|
||||
registerSqliteCommands(cli, logger, environment);
|
||||
registerArtifactCommands(cli, logger, analysis.runProviderAnalysis);
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
import { z } from "incur";
|
||||
import { absoluteCallerPath } from "../filesystem/callerPath.js";
|
||||
import { createPythonPycService } from "../composition/pythonPyc.js";
|
||||
import { CLI_COMMANDS } from "../cliCommandNames.js";
|
||||
import { logCliCommand } from "../cliLogging.js";
|
||||
import type { Logger } from "pino";
|
||||
import type { CliInstance } from "./types.js";
|
||||
import { withCommandCancellation } from "./commandCancellation.js";
|
||||
import { inspectPythonPycInputSchema } from "../domain/python/pythonPyc.js";
|
||||
import { finishAnalysisCommand } from "./finishAnalysisCommand.js";
|
||||
|
||||
/** Inspect operator-selected pyc files through the same shared workflow as MCP. */
|
||||
export const registerPythonPycCommands = (
|
||||
cli: CliInstance,
|
||||
logger: Logger,
|
||||
): void => {
|
||||
cli.command(CLI_COMMANDS.inspectPythonPyc, {
|
||||
description:
|
||||
"Inspect a CPython 3.7–3.13 pyc header and marshal code objects without executing bytecode",
|
||||
args: z.object({
|
||||
path: z.string().describe("Local CPython .pyc file"),
|
||||
}),
|
||||
options: z.object({
|
||||
maxFileBytes: inspectPythonPycInputSchema.shape.max_file_bytes,
|
||||
maxObjects: inspectPythonPycInputSchema.shape.max_objects,
|
||||
maxDepth: inspectPythonPycInputSchema.shape.max_depth,
|
||||
maxStringBytes: inspectPythonPycInputSchema.shape.max_string_bytes,
|
||||
}),
|
||||
run: ({ args, options }) =>
|
||||
withCommandCancellation(async (signal) => {
|
||||
const service = createPythonPycService();
|
||||
return await logCliCommand(
|
||||
logger,
|
||||
CLI_COMMANDS.inspectPythonPyc,
|
||||
async () =>
|
||||
finishAnalysisCommand(
|
||||
() =>
|
||||
service.inspect(
|
||||
{
|
||||
path: absoluteCallerPath(args.path),
|
||||
max_file_bytes: options.maxFileBytes,
|
||||
max_objects: options.maxObjects,
|
||||
max_depth: options.maxDepth,
|
||||
max_string_bytes: options.maxStringBytes,
|
||||
},
|
||||
{ signal },
|
||||
),
|
||||
service,
|
||||
"rea-python-pyc",
|
||||
"inspect_python_pyc",
|
||||
signal,
|
||||
),
|
||||
);
|
||||
}),
|
||||
});
|
||||
};
|
||||
@@ -65,6 +65,7 @@ export const CLI_COMMANDS = Object.freeze({
|
||||
inspectModelArtifact: "inspect-model-artifact",
|
||||
inspectGpuCodeObjects: "inspect-gpu-code-objects",
|
||||
inspectRustBinary: "inspect-rust-binary",
|
||||
inspectPythonPyc: "inspect-python-pyc",
|
||||
inspectErlangBeam: "inspect-erlang-beam",
|
||||
inspectPeResources: "inspect-pe-resources",
|
||||
inspectPeCertificates: "inspect-pe-certificates",
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
import { PythonPycService } from "../application/python/PythonPycService.js";
|
||||
import { PythonPycProvider } from "../python/PythonPycProvider.js";
|
||||
|
||||
/** Compose read-only pyc inspection without invoking CPython or writing files. */
|
||||
export const createPythonPycService = (): PythonPycService =>
|
||||
new PythonPycService(new PythonPycProvider());
|
||||
@@ -0,0 +1,26 @@
|
||||
import {
|
||||
inspectPythonPycInputSchema,
|
||||
pythonPycSchema,
|
||||
} from "../../domain/python/pythonPyc.js";
|
||||
import type { ToolContract } from "../toolContractTypes.js";
|
||||
import { toolContractMetadata } from "../toolEffects.js";
|
||||
import { evidenceResultOf } from "../toolOutputSchemaPrimitives.js";
|
||||
|
||||
/** Static CPython pyc inspection is independent of active native-analysis providers. */
|
||||
export const PYTHON_PYC_TOOL_CONTRACTS = [
|
||||
{
|
||||
name: "inspect_python_pyc",
|
||||
...toolContractMetadata("inspect_python_pyc"),
|
||||
kind: "artifact-provider",
|
||||
description:
|
||||
"Inspect one explicit local CPython 3.7–3.13 .pyc file without executing bytecode, invoking CPython or recovering source. Returns the 16-byte header (magic, pinned python_version, PEP 552 flags, source hash or mtime/size), the top-level code object with nested functions, names, const summaries, and bytecode length/SHA-256. Marshal is walked with object, depth and string budgets; unknown magics, truncated streams and exhausted budgets return no partial success. A decoded header is not proof the matching .py is present. Portable on supported Node hosts; no Python installation required.",
|
||||
inputSchema: inspectPythonPycInputSchema,
|
||||
outputSchema: evidenceResultOf(pythonPycSchema),
|
||||
examples: [
|
||||
{
|
||||
title: "Inspect a CPython 3.12 pyc header and nested code objects",
|
||||
input: { path: "/artifacts/module.pyc" },
|
||||
},
|
||||
],
|
||||
},
|
||||
] as const satisfies readonly ToolContract[];
|
||||
@@ -8,6 +8,7 @@ import { GO_TOOL_CONTRACTS } from "./go/goToolContracts.js";
|
||||
import { MODEL_TOOL_CONTRACTS } from "./model/modelToolContracts.js";
|
||||
import { IL2CPP_TOOL_CONTRACTS } from "./il2cpp/il2cppToolContracts.js";
|
||||
import { RUST_TOOL_CONTRACTS } from "./rust/rustToolContracts.js";
|
||||
import { PYTHON_PYC_TOOL_CONTRACTS } from "./python/pythonPycToolContracts.js";
|
||||
import { ERLANG_BEAM_TOOL_CONTRACTS } from "./erlang/beamToolContracts.js";
|
||||
import { SQLITE_TOOL_CONTRACTS } from "./sqlite/sqliteToolContracts.js";
|
||||
import { ANALYSIS_VIEW_TOOL_CONTRACTS } from "./analysisViewToolContracts.js";
|
||||
@@ -55,6 +56,7 @@ export const TOOL_CONTRACTS = [
|
||||
...MODEL_TOOL_CONTRACTS,
|
||||
...GPU_TOOL_CONTRACTS,
|
||||
...RUST_TOOL_CONTRACTS,
|
||||
...PYTHON_PYC_TOOL_CONTRACTS,
|
||||
...ERLANG_BEAM_TOOL_CONTRACTS,
|
||||
...WASM_TOOL_CONTRACTS,
|
||||
...SQLITE_TOOL_CONTRACTS,
|
||||
|
||||
@@ -35,6 +35,8 @@ const EXECUTION_WORDING_EXCEPTIONS: Readonly<Record<string, string>> = {
|
||||
inspect_go_binary: "reads embedded metadata without executing the target",
|
||||
inspect_rust_binary:
|
||||
"reads an embedded dependency list without executing the target",
|
||||
inspect_python_pyc:
|
||||
"states that it does not execute bytecode or invoke CPython",
|
||||
inspect_erlang_beam: "states that it does not execute bytecode",
|
||||
inspect_sqlite_database:
|
||||
"uses a trusted inspector worker without executing application code or caller SQL",
|
||||
|
||||
@@ -131,6 +131,7 @@ export const TOOL_EFFECTS: Readonly<Record<string, ToolEffects>> = {
|
||||
inspect_model_artifact: evidence,
|
||||
inspect_gpu_code_objects: evidence,
|
||||
inspect_rust_binary: evidence,
|
||||
inspect_python_pyc: evidence,
|
||||
inspect_erlang_beam: evidence,
|
||||
inspect_sqlite_database: effects({
|
||||
mutatesSession: true,
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
import { expect, it } from "vitest";
|
||||
import {
|
||||
inspectPythonPycInputSchema,
|
||||
pythonPycMagicEntry,
|
||||
PYTHON_PYC_MAGIC_TABLE,
|
||||
} from "./pythonPyc.js";
|
||||
|
||||
it("maps pinned CPython 3.7–3.13 magics and leaves gaps unknown", () => {
|
||||
expect(pythonPycMagicEntry(3394)).toEqual({
|
||||
python_version: "3.7",
|
||||
code_layout: "py37",
|
||||
});
|
||||
expect(pythonPycMagicEntry(3413)).toEqual({
|
||||
python_version: "3.8",
|
||||
code_layout: "py38",
|
||||
});
|
||||
expect(pythonPycMagicEntry(3531)).toEqual({
|
||||
python_version: "3.12",
|
||||
code_layout: "py311",
|
||||
});
|
||||
expect(pythonPycMagicEntry(3571)).toEqual({
|
||||
python_version: "3.13",
|
||||
code_layout: "py311",
|
||||
});
|
||||
expect(pythonPycMagicEntry(3450)).toBeNull();
|
||||
expect(pythonPycMagicEntry(62171)).toBeNull();
|
||||
expect(PYTHON_PYC_MAGIC_TABLE.size).toBeGreaterThan(80);
|
||||
});
|
||||
|
||||
it("rejects extra inspect arguments", () => {
|
||||
expect(
|
||||
inspectPythonPycInputSchema.safeParse({
|
||||
path: "/artifacts/module.pyc",
|
||||
detail: "complete",
|
||||
}).success,
|
||||
).toBe(false);
|
||||
});
|
||||
@@ -0,0 +1,298 @@
|
||||
import { z } from "zod";
|
||||
import { digestSchema } from "../digests.js";
|
||||
import { localPathStringSchema } from "../localPath.js";
|
||||
|
||||
export const PYTHON_PYC_HEADER_BYTES = 16;
|
||||
export const PYTHON_PYC_HASH_BASED = 0x1;
|
||||
export const PYTHON_PYC_CHECK_SOURCE = 0x2;
|
||||
|
||||
const uint = z.number().int().min(0).max(0xffffffff);
|
||||
const index = z.number().int().nonnegative();
|
||||
/** Non-negative CPython `int` counts. `0xffffffff` is -1 and is not evidence. */
|
||||
const signedCount = z.number().int().min(0).max(0x7fffffff);
|
||||
|
||||
/** Marshal TYPE_CODE field order for CPython 3.7, 3.8–3.10, and 3.11+. */
|
||||
export const PYTHON_PYC_CODE_LAYOUTS = ["py37", "py38", "py311"] as const;
|
||||
export type PythonPycCodeLayout = (typeof PYTHON_PYC_CODE_LAYOUTS)[number];
|
||||
|
||||
type MagicEntry = {
|
||||
readonly python_version: string;
|
||||
readonly code_layout: PythonPycCodeLayout;
|
||||
};
|
||||
|
||||
const magics = (
|
||||
values: readonly number[],
|
||||
python_version: string,
|
||||
code_layout: PythonPycCodeLayout,
|
||||
): ReadonlyArray<readonly [number, MagicEntry]> =>
|
||||
values.map((magic) => [magic, { python_version, code_layout }]);
|
||||
|
||||
const consecutive = (
|
||||
start: number,
|
||||
end: number,
|
||||
python_version: string,
|
||||
code_layout: PythonPycCodeLayout,
|
||||
): ReadonlyArray<readonly [number, MagicEntry]> =>
|
||||
magics(
|
||||
Array.from({ length: end - start + 1 }, (_, offset) => start + offset),
|
||||
python_version,
|
||||
code_layout,
|
||||
);
|
||||
|
||||
/**
|
||||
* Pinned CPython pyc magics for 3.7–3.13. Identity is the 16-bit little-endian
|
||||
* field; `\r\n` is required separately. Layout follows marshal TYPE_CODE, not
|
||||
* bytecode opcode tables.
|
||||
*/
|
||||
export const PYTHON_PYC_MAGIC_TABLE: ReadonlyMap<number, MagicEntry> = new Map([
|
||||
...magics([3390, 3391, 3392, 3393, 3394], "3.7", "py37"),
|
||||
...magics([3400, 3401], "3.8", "py37"),
|
||||
...magics([3410, 3411, 3412, 3413], "3.8", "py38"),
|
||||
...magics([3420, 3421, 3422, 3423, 3424, 3425], "3.9", "py38"),
|
||||
...magics(
|
||||
[3430, 3431, 3432, 3433, 3434, 3435, 3436, 3437, 3438, 3439],
|
||||
"3.10",
|
||||
"py38",
|
||||
),
|
||||
// 3.11 alphas before 3494 marshal other code layouts: no co_qualname
|
||||
// before 3460, and PEP 657 end-line and column tables until 3494.
|
||||
...consecutive(3494, 3495, "3.11", "py311"),
|
||||
...consecutive(3500, 3531, "3.12", "py311"),
|
||||
...consecutive(3550, 3571, "3.13", "py311"),
|
||||
]);
|
||||
|
||||
/** Map a pyc magic integer to its pinned CPython family, or null when unknown. */
|
||||
export const pythonPycMagicEntry = (magic: number): MagicEntry | null =>
|
||||
PYTHON_PYC_MAGIC_TABLE.get(magic) ?? null;
|
||||
|
||||
export const inspectPythonPycInputSchema = z.strictObject({
|
||||
path: localPathStringSchema.describe(
|
||||
"Absolute filesystem path to the selected CPython .pyc file",
|
||||
),
|
||||
max_file_bytes: z
|
||||
.number()
|
||||
.int()
|
||||
.min(16)
|
||||
.max(64 * 1024 * 1024)
|
||||
.default(16 * 1024 * 1024)
|
||||
.describe(
|
||||
"Maximum selected file size in bytes; exceeding the budget fails without partial success",
|
||||
),
|
||||
max_objects: z
|
||||
.number()
|
||||
.int()
|
||||
.min(1)
|
||||
.max(65536)
|
||||
.default(16384)
|
||||
.describe(
|
||||
"Maximum marshal objects to examine; exceeding the budget fails without partial success",
|
||||
),
|
||||
max_depth: z
|
||||
.number()
|
||||
.int()
|
||||
.min(1)
|
||||
.max(256)
|
||||
.default(64)
|
||||
.describe(
|
||||
"Maximum marshal nesting depth; exceeding the budget fails without partial success",
|
||||
),
|
||||
max_string_bytes: z
|
||||
.number()
|
||||
.int()
|
||||
.min(1)
|
||||
.max(8 * 1024 * 1024)
|
||||
.default(1024 * 1024)
|
||||
.describe(
|
||||
"Maximum bytes retained for one marshal string or byte string; exceeding the budget fails without partial success",
|
||||
),
|
||||
});
|
||||
|
||||
const constPreviewLimit = 80;
|
||||
/** Integers up to 256 bits fit the 80-character preview in decimal. */
|
||||
const intPreviewBits = 256;
|
||||
|
||||
/** CPython float repr for values JSON numbers cannot carry. */
|
||||
const pythonFloatSchema = z.union([z.number(), z.enum(["inf", "-inf", "nan"])]);
|
||||
|
||||
const pythonConstSchema = z.discriminatedUnion("kind", [
|
||||
z.strictObject({ kind: z.literal("none") }),
|
||||
z.strictObject({ kind: z.literal("bool"), value: z.boolean() }),
|
||||
z.strictObject({
|
||||
kind: z.literal("int"),
|
||||
bits: uint,
|
||||
value: z.string().min(1).max(constPreviewLimit).nullable(),
|
||||
}),
|
||||
z.strictObject({ kind: z.literal("float"), value: pythonFloatSchema }),
|
||||
z.strictObject({
|
||||
kind: z.literal("str"),
|
||||
bytes: uint,
|
||||
truncated: z.boolean(),
|
||||
value: z.string().max(constPreviewLimit),
|
||||
}),
|
||||
z.strictObject({
|
||||
kind: z.literal("bytes"),
|
||||
bytes: uint,
|
||||
sha256: digestSchema,
|
||||
}),
|
||||
z.strictObject({
|
||||
kind: z.literal("code"),
|
||||
name: z.string(),
|
||||
firstlineno: uint,
|
||||
}),
|
||||
z.strictObject({ kind: z.literal("tuple"), length: uint }),
|
||||
z.strictObject({ kind: z.literal("list"), length: uint }),
|
||||
z.strictObject({ kind: z.literal("set"), length: uint }),
|
||||
z.strictObject({ kind: z.literal("frozenset"), length: uint }),
|
||||
z.strictObject({ kind: z.literal("dict"), length: uint }),
|
||||
z.strictObject({ kind: z.literal("ellipsis") }),
|
||||
z.strictObject({
|
||||
kind: z.literal("complex"),
|
||||
real: pythonFloatSchema,
|
||||
imag: pythonFloatSchema,
|
||||
}),
|
||||
z.strictObject({ kind: z.literal("other"), type: z.string().min(1) }),
|
||||
]);
|
||||
|
||||
const pythonCodeFields = {
|
||||
name: z.string(),
|
||||
qualname: z.string().nullable(),
|
||||
filename: z.string(),
|
||||
firstlineno: uint,
|
||||
argcount: signedCount,
|
||||
posonlyargcount: signedCount.nullable(),
|
||||
kwonlyargcount: signedCount,
|
||||
nlocals: signedCount.nullable(),
|
||||
stacksize: signedCount,
|
||||
flags: uint,
|
||||
names: z.array(z.string()),
|
||||
varnames: z.array(z.string()).nullable(),
|
||||
freevars: z.array(z.string()).nullable(),
|
||||
cellvars: z.array(z.string()).nullable(),
|
||||
localsplusnames: z.array(z.string()).nullable(),
|
||||
localspluskinds_hex: z.string().nullable(),
|
||||
bytecode: z.strictObject({
|
||||
bytes: uint,
|
||||
sha256: digestSchema,
|
||||
}),
|
||||
consts: z.array(pythonConstSchema),
|
||||
};
|
||||
|
||||
export type PythonPycConst = z.infer<typeof pythonConstSchema>;
|
||||
|
||||
/**
|
||||
* One distinct marshal code object. `nested_code` holds indexes into the
|
||||
* report's `code_objects`, so a child shared through TYPE_REF is listed once.
|
||||
* The owner's own index is included when its constants reach that object.
|
||||
*/
|
||||
const pythonCodeObjectSchema = z.strictObject({
|
||||
...pythonCodeFields,
|
||||
nested_code: z.array(index),
|
||||
});
|
||||
export type PythonPycCodeObject = z.infer<typeof pythonCodeObjectSchema>;
|
||||
|
||||
export const pythonPycHeaderSchema = z.strictObject({
|
||||
magic: z.number().int().min(0).max(0xffff),
|
||||
python_version: z.string().min(1),
|
||||
code_layout: z.enum(PYTHON_PYC_CODE_LAYOUTS),
|
||||
flags: uint,
|
||||
hash_based: z.boolean(),
|
||||
check_source: z.boolean(),
|
||||
source_hash_hex: z.string().nullable(),
|
||||
source_mtime: uint.nullable(),
|
||||
source_size: uint.nullable(),
|
||||
});
|
||||
|
||||
/** Bounded CPython 3.7–3.13 pyc observation; marshal is walked, bytecode is not decompiled. */
|
||||
export const pythonPycSchema = z
|
||||
.strictObject({
|
||||
artifact: z.strictObject({
|
||||
path: localPathStringSchema,
|
||||
sha256: digestSchema,
|
||||
bytes: index,
|
||||
}),
|
||||
header: pythonPycHeaderSchema,
|
||||
/** The top-level module code object is first. */
|
||||
code_objects: z.array(pythonCodeObjectSchema).min(1),
|
||||
coverage: z.strictObject({
|
||||
status: z.literal("complete"),
|
||||
objects: uint,
|
||||
code_objects: uint,
|
||||
marshal_bytes: uint,
|
||||
marshal_depth: uint,
|
||||
}),
|
||||
limitations: z.array(z.string().min(1)).min(1),
|
||||
})
|
||||
.superRefine((report, context) => {
|
||||
if (report.artifact.bytes < PYTHON_PYC_HEADER_BYTES) {
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
path: ["artifact", "bytes"],
|
||||
message: "A pyc header does not fit in the selected artifact.",
|
||||
});
|
||||
}
|
||||
const marshalBytes = report.artifact.bytes - PYTHON_PYC_HEADER_BYTES;
|
||||
if (report.coverage.marshal_bytes !== marshalBytes) {
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
path: ["coverage", "marshal_bytes"],
|
||||
message: "Marshal byte count disagrees with the header-stripped file.",
|
||||
});
|
||||
}
|
||||
if (report.header.hash_based === (report.header.source_hash_hex === null)) {
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
path: ["header", "source_hash_hex"],
|
||||
message:
|
||||
"Hash-based pycs require an 8-byte source hash; others must omit it.",
|
||||
});
|
||||
}
|
||||
if (report.header.hash_based === (report.header.source_mtime !== null)) {
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
path: ["header", "source_mtime"],
|
||||
message: "Timestamp pycs require mtime; hash-based pycs must omit it.",
|
||||
});
|
||||
}
|
||||
if (report.header.hash_based === (report.header.source_size !== null)) {
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
path: ["header", "source_size"],
|
||||
message:
|
||||
"Timestamp pycs require source_size; hash-based pycs must omit it.",
|
||||
});
|
||||
}
|
||||
if (report.coverage.code_objects !== report.code_objects.length) {
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
path: ["coverage", "code_objects"],
|
||||
message: "Code-object coverage disagrees with the listed code objects.",
|
||||
});
|
||||
}
|
||||
report.code_objects.forEach((code, owner) => {
|
||||
if (
|
||||
code.localspluskinds_hex !== null &&
|
||||
code.localspluskinds_hex.length !==
|
||||
(code.localsplusnames?.length ?? 0) * 2
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
path: ["code_objects", owner, "localspluskinds_hex"],
|
||||
message:
|
||||
"co_localspluskinds length does not match co_localsplusnames.",
|
||||
});
|
||||
code.nested_code.forEach((nested, position) => {
|
||||
if (nested >= report.code_objects.length)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
path: ["code_objects", owner, "nested_code", position],
|
||||
message: "Nested code index does not name a listed code object.",
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
export type PythonPyc = z.infer<typeof pythonPycSchema>;
|
||||
export type InspectPythonPycInput = z.infer<typeof inspectPythonPycInputSchema>;
|
||||
|
||||
export const PYTHON_PYC_CONST_PREVIEW_LIMIT = constPreviewLimit;
|
||||
export const PYTHON_PYC_INT_PREVIEW_BITS = intPreviewBits;
|
||||
@@ -0,0 +1,22 @@
|
||||
/** Distinguish corrupt recognized pycs from magics outside parser coverage. */
|
||||
export class PythonPycFormatFailure extends Error {
|
||||
constructor(
|
||||
readonly kind: "malformed" | "unsupported",
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = "PythonPycFormatFailure";
|
||||
}
|
||||
}
|
||||
|
||||
/** Resource boundary exceeded while walking marshal objects. */
|
||||
export class PythonPycResourceFailure extends Error {
|
||||
constructor(
|
||||
readonly boundary: "objects" | "depth" | "string",
|
||||
readonly reported_limits: Readonly<Record<string, number>>,
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = "PythonPycResourceFailure";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
import type { PythonPycCodeLayout } from "../domain/python/pythonPyc.js";
|
||||
|
||||
const FLAG_REF = 0x80;
|
||||
|
||||
const u8 = (value: number): Buffer => Buffer.from([value]);
|
||||
const u32 = (value: number): Buffer => {
|
||||
const bytes = Buffer.alloc(4);
|
||||
bytes.writeUInt32LE(value);
|
||||
return bytes;
|
||||
};
|
||||
const concat = (...parts: Buffer[]): Buffer => Buffer.concat(parts);
|
||||
|
||||
/** 16-byte CPython 3.7+ pyc header. Hash-based pycs store an 8-byte source hash. */
|
||||
export const pythonPycHeader = (
|
||||
magic: number,
|
||||
fields: {
|
||||
flags?: number;
|
||||
mtime?: number;
|
||||
sourceSize?: number;
|
||||
sourceHash?: Buffer;
|
||||
} = {},
|
||||
): Buffer => {
|
||||
const header = Buffer.alloc(16);
|
||||
header.writeUInt16LE(magic, 0);
|
||||
header[2] = 0x0d;
|
||||
header[3] = 0x0a;
|
||||
const flags = fields.flags ?? 0;
|
||||
header.writeUInt32LE(flags, 4);
|
||||
if ((flags & 1) !== 0) {
|
||||
const hash = fields.sourceHash ?? Buffer.alloc(8);
|
||||
if (hash.length !== 8)
|
||||
throw new Error("Hash-based pyc header requires 8 source-hash bytes.");
|
||||
hash.copy(header, 8);
|
||||
return header;
|
||||
}
|
||||
header.writeUInt32LE(fields.mtime ?? 0, 8);
|
||||
header.writeUInt32LE(fields.sourceSize ?? 0, 12);
|
||||
return header;
|
||||
};
|
||||
|
||||
const marshalNone = (): Buffer => u8(0x4e);
|
||||
const marshalBytes = (value: Buffer, flagged = false): Buffer =>
|
||||
concat(u8(0x73 | (flagged ? FLAG_REF : 0)), u32(value.length), value);
|
||||
const marshalShortAscii = (value: string, interned = false): Buffer =>
|
||||
concat(
|
||||
u8(interned ? 0x5a : 0x7a),
|
||||
u8(Buffer.byteLength(value, "utf8")),
|
||||
Buffer.from(value, "utf8"),
|
||||
);
|
||||
const marshalSmallTuple = (items: Buffer[]): Buffer =>
|
||||
concat(u8(0x29), u8(items.length), ...items);
|
||||
|
||||
export type FixtureCodeFields = {
|
||||
argcount?: number;
|
||||
posonlyargcount?: number;
|
||||
kwonlyargcount?: number;
|
||||
nlocals?: number;
|
||||
stacksize?: number;
|
||||
flags?: number;
|
||||
bytecode?: Buffer;
|
||||
consts?: Buffer[];
|
||||
names?: string[];
|
||||
varnames?: string[];
|
||||
freevars?: string[];
|
||||
cellvars?: string[];
|
||||
localsplusnames?: string[];
|
||||
localspluskinds?: Buffer;
|
||||
/** Raw marshal object in place of the co_consts tuple. */
|
||||
constsObject?: Buffer;
|
||||
filename?: string;
|
||||
name?: string;
|
||||
qualname?: string;
|
||||
/** Raw marshal object in place of co_name. */
|
||||
nameObject?: Buffer;
|
||||
/** Raw marshal object in place of co_qualname. */
|
||||
qualnameObject?: Buffer;
|
||||
firstlineno?: number;
|
||||
linetable?: Buffer;
|
||||
exceptiontable?: Buffer;
|
||||
/** Raw marshal object in place of the co_names tuple. */
|
||||
namesObject?: Buffer;
|
||||
/** Raw marshal object in place of the co_exceptiontable bytes. */
|
||||
exceptiontableObject?: Buffer;
|
||||
flagged?: boolean;
|
||||
};
|
||||
|
||||
/** Marshal TYPE_CODE matching CPython 3.7, 3.8–3.10, or 3.11+ field order. */
|
||||
export const marshalCode = (
|
||||
layout: PythonPycCodeLayout,
|
||||
fields: FixtureCodeFields = {},
|
||||
): Buffer => {
|
||||
const parts: Buffer[] = [
|
||||
u8(0x63 | (fields.flagged === true ? FLAG_REF : 0)),
|
||||
u32(fields.argcount ?? 0),
|
||||
];
|
||||
if (layout !== "py37") parts.push(u32(fields.posonlyargcount ?? 0));
|
||||
parts.push(u32(fields.kwonlyargcount ?? 0));
|
||||
if (layout !== "py311") parts.push(u32(fields.nlocals ?? 0));
|
||||
parts.push(
|
||||
u32(fields.stacksize ?? 1),
|
||||
u32(fields.flags ?? 0),
|
||||
marshalBytes(fields.bytecode ?? Buffer.from([0x53, 0x00])),
|
||||
fields.constsObject ?? marshalSmallTuple(fields.consts ?? [marshalNone()]),
|
||||
fields.namesObject ??
|
||||
marshalSmallTuple(
|
||||
(fields.names ?? []).map((name) => marshalShortAscii(name)),
|
||||
),
|
||||
);
|
||||
if (layout === "py311") {
|
||||
parts.push(
|
||||
marshalSmallTuple(
|
||||
(fields.localsplusnames ?? []).map((name) => marshalShortAscii(name)),
|
||||
),
|
||||
marshalBytes(fields.localspluskinds ?? Buffer.alloc(0)),
|
||||
);
|
||||
} else {
|
||||
parts.push(
|
||||
marshalSmallTuple(
|
||||
(fields.varnames ?? []).map((name) => marshalShortAscii(name)),
|
||||
),
|
||||
marshalSmallTuple(
|
||||
(fields.freevars ?? []).map((name) => marshalShortAscii(name)),
|
||||
),
|
||||
marshalSmallTuple(
|
||||
(fields.cellvars ?? []).map((name) => marshalShortAscii(name)),
|
||||
),
|
||||
);
|
||||
}
|
||||
parts.push(
|
||||
marshalShortAscii(fields.filename ?? "sample.py"),
|
||||
fields.nameObject ?? marshalShortAscii(fields.name ?? "<module>"),
|
||||
);
|
||||
if (layout === "py311")
|
||||
parts.push(
|
||||
fields.qualnameObject ??
|
||||
marshalShortAscii(fields.qualname ?? fields.name ?? "<module>"),
|
||||
);
|
||||
parts.push(
|
||||
u32(fields.firstlineno ?? 1),
|
||||
marshalBytes(fields.linetable ?? Buffer.alloc(0)),
|
||||
);
|
||||
if (layout === "py311")
|
||||
parts.push(
|
||||
fields.exceptiontableObject ??
|
||||
marshalBytes(fields.exceptiontable ?? Buffer.alloc(0)),
|
||||
);
|
||||
return concat(...parts);
|
||||
};
|
||||
|
||||
/** Complete pyc with a module code object and optional nested function. */
|
||||
export const pythonPycFixture = (
|
||||
layout: PythonPycCodeLayout = "py311",
|
||||
options: {
|
||||
magic?: number;
|
||||
header?: Parameters<typeof pythonPycHeader>[1];
|
||||
nested?: boolean;
|
||||
flaggedCode?: boolean;
|
||||
} = {},
|
||||
): Buffer => {
|
||||
const magic =
|
||||
options.magic ??
|
||||
(layout === "py37" ? 3394 : layout === "py38" ? 3439 : 3531);
|
||||
const nested =
|
||||
options.nested === true
|
||||
? marshalCode(layout, {
|
||||
name: "inner",
|
||||
qualname: "inner",
|
||||
filename: "sample.py",
|
||||
firstlineno: 2,
|
||||
bytecode: Buffer.from([0x53, 0x00]),
|
||||
})
|
||||
: undefined;
|
||||
const module = marshalCode(layout, {
|
||||
name: "<module>",
|
||||
qualname: "<module>",
|
||||
filename: "sample.py",
|
||||
firstlineno: 1,
|
||||
names: nested === undefined ? [] : ["inner"],
|
||||
consts: nested === undefined ? [marshalNone()] : [nested, marshalNone()],
|
||||
bytecode: Buffer.from([0x64, 0x00, 0x53, 0x00]),
|
||||
...(options.flaggedCode === true ? { flagged: true } : {}),
|
||||
});
|
||||
return concat(pythonPycHeader(magic, options.header), module);
|
||||
};
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,174 @@
|
||||
import { bindArtifactFileCleanup } from "../artifacts/ArtifactFileOperation.js";
|
||||
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
|
||||
import { readStableArtifact } from "../artifacts/readStableArtifact.js";
|
||||
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
|
||||
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
|
||||
import type { PythonPycPort } from "../application/python/PythonPycPort.js";
|
||||
import type { ExecutionOptions } from "../application/AnalysisProvider.js";
|
||||
import { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import {
|
||||
AnalysisAccessDeniedError,
|
||||
AnalysisArtifactChangedError,
|
||||
AnalysisCancelledError,
|
||||
AnalysisInputError,
|
||||
AnalysisResourceConstraintError,
|
||||
AnalysisUnsupportedTargetError,
|
||||
} from "../domain/analysisErrorCore.js";
|
||||
import type { InspectPythonPycInput } from "../domain/python/pythonPyc.js";
|
||||
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
|
||||
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
|
||||
import { err, ok } from "../domain/result.js";
|
||||
import {
|
||||
PythonPycFormatFailure,
|
||||
PythonPycResourceFailure,
|
||||
} from "./PythonPyc.errors.js";
|
||||
import { parsePythonPyc } from "./PythonPycParser.js";
|
||||
|
||||
/** Bundled reader identity. Header python_version is a separate observed mapping. */
|
||||
export const PYTHON_PYC_PROVIDER_IDENTITY = {
|
||||
id: "rea-python-pyc",
|
||||
name: "REA CPython pyc parser",
|
||||
version: "1",
|
||||
} as const;
|
||||
|
||||
const OPERATION = "inspect_python_pyc";
|
||||
|
||||
const failureForPrimary = (
|
||||
cause: unknown,
|
||||
input: InspectPythonPycInput,
|
||||
signal: AbortSignal | undefined,
|
||||
) => {
|
||||
if (cause instanceof AnalysisError) return cause;
|
||||
if (cause instanceof ArtifactReaderFailure && cause.reason === "integrity")
|
||||
return new AnalysisArtifactChangedError(
|
||||
OPERATION,
|
||||
input.path,
|
||||
cause.message,
|
||||
{ cause },
|
||||
);
|
||||
if (cause instanceof ArtifactReaderFailure && cause.reason === "cancelled")
|
||||
return new AnalysisCancelledError(OPERATION, { cause });
|
||||
if (signal?.aborted) return new AnalysisCancelledError(OPERATION, { cause });
|
||||
if (cause instanceof PythonPycResourceFailure)
|
||||
return new AnalysisResourceConstraintError(
|
||||
OPERATION,
|
||||
cause.boundary === "depth" ? "capacity" : "memory",
|
||||
cause.message,
|
||||
cause.reported_limits,
|
||||
{ cause },
|
||||
);
|
||||
if (cause instanceof PythonPycFormatFailure)
|
||||
return cause.kind === "unsupported"
|
||||
? new AnalysisUnsupportedTargetError(
|
||||
OPERATION,
|
||||
input.path,
|
||||
cause.message,
|
||||
{ cause },
|
||||
)
|
||||
: new AnalysisInputError(OPERATION, { cause }, [
|
||||
{
|
||||
path: ["path"],
|
||||
reason: "invalid_format",
|
||||
message: `Selected pyc ${input.path}: ${cause.message}`,
|
||||
},
|
||||
]);
|
||||
if (cause instanceof ArtifactReaderFailure) {
|
||||
if (cause.reason === "limit")
|
||||
return new AnalysisResourceConstraintError(
|
||||
OPERATION,
|
||||
"file-size",
|
||||
cause.message,
|
||||
{ max_file_bytes: input.max_file_bytes },
|
||||
{ cause },
|
||||
);
|
||||
if (cause.reason === "path" || cause.reason === "format")
|
||||
return new AnalysisInputError(OPERATION, { cause }, [
|
||||
{
|
||||
path: ["path"],
|
||||
reason: "invalid_format",
|
||||
message: cause.message,
|
||||
},
|
||||
]);
|
||||
}
|
||||
if (cause instanceof Error && "code" in cause) {
|
||||
if (cause.code === "EACCES" || cause.code === "EPERM")
|
||||
return new AnalysisAccessDeniedError(OPERATION, input.path, cause.code, {
|
||||
cause,
|
||||
});
|
||||
if (
|
||||
["ENOENT", "ENOTDIR", "ELOOP", "ENAMETOOLONG"].includes(
|
||||
String(cause.code),
|
||||
)
|
||||
)
|
||||
return new AnalysisInputError(OPERATION, { cause }, [
|
||||
{
|
||||
path: ["path"],
|
||||
reason: "invalid_value",
|
||||
message: `Selected pyc could not be read (${String(cause.code)}): ${input.path}.`,
|
||||
},
|
||||
]);
|
||||
}
|
||||
return new ProviderAdapterError(PYTHON_PYC_PROVIDER_IDENTITY.id, OPERATION, {
|
||||
cause,
|
||||
diagnostics: {
|
||||
path: input.path,
|
||||
reason: cause instanceof Error ? cause.message : String(cause),
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
const failureFor = (
|
||||
cause: unknown,
|
||||
input: InspectPythonPycInput,
|
||||
signal: AbortSignal | undefined,
|
||||
) => {
|
||||
const primary = failureForPrimary(cause, input, signal);
|
||||
if (!(cause instanceof ArtifactReaderFailure) || cause.cleanup === undefined)
|
||||
return primary;
|
||||
const cleanup = new ProviderCleanupError(
|
||||
PYTHON_PYC_PROVIDER_IDENTITY.id,
|
||||
cause.cleanup.resources,
|
||||
{ reason: cause.cleanup.reason },
|
||||
{ operation: OPERATION, cause },
|
||||
);
|
||||
const error = analysisErrorWithCleanupFailure(primary, cleanup, OPERATION);
|
||||
bindArtifactFileCleanup(cause, error);
|
||||
return error;
|
||||
};
|
||||
|
||||
/** Portable parser over one bounded, identity-checked regular file. */
|
||||
export class PythonPycProvider implements PythonPycPort {
|
||||
readonly identity = PYTHON_PYC_PROVIDER_IDENTITY;
|
||||
readonly #resources = new ArtifactResourceScope();
|
||||
|
||||
close(): Promise<void> {
|
||||
return this.#resources.close();
|
||||
}
|
||||
|
||||
async inspect(input: InspectPythonPycInput, options?: ExecutionOptions) {
|
||||
try {
|
||||
const snapshot = await this.#resources.run(() =>
|
||||
readStableArtifact(
|
||||
input.path,
|
||||
input.max_file_bytes,
|
||||
options?.signal,
|
||||
undefined,
|
||||
this.#resources,
|
||||
),
|
||||
);
|
||||
return ok(
|
||||
parsePythonPyc(
|
||||
snapshot.bytes,
|
||||
{
|
||||
path: input.path,
|
||||
sha256: snapshot.sha256,
|
||||
bytes: snapshot.bytes.length,
|
||||
},
|
||||
input,
|
||||
),
|
||||
);
|
||||
} catch (cause) {
|
||||
return err(failureFor(cause, input, options?.signal));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -52,6 +52,9 @@ import { registerIl2cppTools } from "./registerIl2cppTools.js";
|
||||
import { createRustBinaryService } from "../composition/rust.js";
|
||||
import type { RustBinaryService } from "../application/rust/RustBinaryService.js";
|
||||
import { registerRustTools } from "./registerRustTools.js";
|
||||
import { createPythonPycService } from "../composition/pythonPyc.js";
|
||||
import type { PythonPycService } from "../application/python/PythonPycService.js";
|
||||
import { registerPythonPycTool } from "./registerPythonPycTool.js";
|
||||
import { createBeamService } from "../composition/erlang.js";
|
||||
import type { BeamService } from "../application/erlang/BeamService.js";
|
||||
import { registerErlangTools } from "./registerErlangTools.js";
|
||||
@@ -196,6 +199,7 @@ export interface CreateServerOptions {
|
||||
readonly goBinary?: GoBinaryService;
|
||||
readonly il2cppMetadata?: Il2cppMetadataService;
|
||||
readonly rustBinary?: RustBinaryService;
|
||||
readonly pythonPyc?: PythonPycService;
|
||||
readonly erlangBeam?: BeamService;
|
||||
readonly electronFuses?: ElectronFuseService;
|
||||
readonly recordedCrash?: RecordedCrashService;
|
||||
@@ -370,6 +374,7 @@ export const createServer = (
|
||||
const elfNotes = createElfNotesService();
|
||||
const peCertificates = createPeCertificatesService();
|
||||
const electronFuses = options.electronFuses ?? createElectronFuseService();
|
||||
const pythonPyc = options.pythonPyc ?? createPythonPycService();
|
||||
const webNetworkCapture =
|
||||
options.webNetworkCapture ?? createWebNetworkCaptureService(environment);
|
||||
const javascriptRecovery =
|
||||
@@ -475,6 +480,7 @@ export const createServer = (
|
||||
elfNotes.close(),
|
||||
peCertificates.close(),
|
||||
electronFuses.close(),
|
||||
pythonPyc.close(),
|
||||
webNetworkCapture.close(),
|
||||
javascriptRecovery.close?.() ?? Promise.resolve(),
|
||||
...(options.optionalObservationProviderOwner === "caller"
|
||||
@@ -542,6 +548,7 @@ export const createServer = (
|
||||
elfNotes,
|
||||
peCertificates,
|
||||
electronFuses,
|
||||
pythonPyc,
|
||||
webNetworkCapture,
|
||||
};
|
||||
registerDesktopApplicationTool(
|
||||
@@ -776,6 +783,7 @@ const registerConfiguredAnalysisTools = (
|
||||
nativeExports,
|
||||
elfNotes,
|
||||
peCertificates,
|
||||
pythonPyc,
|
||||
webNetworkCapture,
|
||||
artifactResources,
|
||||
}: ServerToolContext,
|
||||
@@ -870,6 +878,13 @@ const registerConfiguredAnalysisTools = (
|
||||
withAdmittedAnalysis,
|
||||
evidenceById,
|
||||
);
|
||||
registerPythonPycTool(
|
||||
server,
|
||||
pythonPyc,
|
||||
toolLogger,
|
||||
recordEvidence,
|
||||
withAdmittedAnalysis,
|
||||
);
|
||||
registerErlangTools(
|
||||
server,
|
||||
erlangBeam,
|
||||
@@ -991,6 +1006,7 @@ interface ServerToolContext extends ReturnType<typeof createSessionRecorders> {
|
||||
readonly elfNotes: ElfNotesService;
|
||||
readonly peCertificates: PeCertificatesService;
|
||||
readonly electronFuses: ElectronFuseService;
|
||||
readonly pythonPyc: PythonPycService;
|
||||
readonly webNetworkCapture: WebNetworkCaptureService;
|
||||
readonly artifactResources: ArtifactResourceScope;
|
||||
readonly server: EvidenceMcpServer;
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
import type { EvidenceMcpServer } from "./EvidenceMcpServer.js";
|
||||
import type { PythonPycService } from "../application/python/PythonPycService.js";
|
||||
import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js";
|
||||
import { toolContract } from "../contracts/toolContracts.js";
|
||||
import type { Logger } from "pino";
|
||||
import { createEvidenceToolHandler } from "./contractToolHandler.js";
|
||||
import type { WithAdmittedAnalysis } from "./analysisAdmission.js";
|
||||
import { toolRegistrationOptions } from "./toolRegistrationOptions.js";
|
||||
|
||||
/** Register the explicit-file CPython pyc workflow and retain its complete Evidence. */
|
||||
export const registerPythonPycTool = (
|
||||
server: EvidenceMcpServer,
|
||||
service: PythonPycService,
|
||||
logger: Logger,
|
||||
recordEvidence?: EvidenceWriter["recordEvidence"],
|
||||
withAdmittedAnalysis?: WithAdmittedAnalysis,
|
||||
): void => {
|
||||
const contract = toolContract("inspect_python_pyc");
|
||||
const handler = createEvidenceToolHandler(
|
||||
server,
|
||||
(input, options) => service.inspect(input, options),
|
||||
logger,
|
||||
recordEvidence,
|
||||
withAdmittedAnalysis,
|
||||
);
|
||||
server.registerTool(
|
||||
contract.name,
|
||||
toolRegistrationOptions(contract),
|
||||
handler(contract),
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,79 @@
|
||||
import { mkdir, symlink, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect } from "vitest";
|
||||
import { toolContract } from "../../src/contracts/toolContracts.js";
|
||||
import { pythonPycFixture } from "../../src/python/PythonPyc.fixture.js";
|
||||
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
|
||||
import { cliTest } from "../support/cli/cliFixture.js";
|
||||
|
||||
cliTest(
|
||||
"reports nested CPython code objects through the public CLI",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-python-pyc-cli-");
|
||||
const path = join(root, "sample.pyc");
|
||||
await writeFile(path, pythonPycFixture("py311", { nested: true }));
|
||||
const result = await cli.run({
|
||||
arguments: ["inspect-python-pyc", path, "--json"],
|
||||
});
|
||||
expect(result.exitCode, result.stderr).toBe(0);
|
||||
const evidence = toolContract("inspect_python_pyc").outputSchema.parse(
|
||||
result.json,
|
||||
);
|
||||
expect(evidence.normalized_result.header.python_version).toBe("3.12");
|
||||
expect(evidence.normalized_result.code_objects[1]?.name).toBe("inner");
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"resolves a relative pyc path against the working directory",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-python-pyc-cli-rel-");
|
||||
await writeFile(join(root, "sample.pyc"), pythonPycFixture("py37"));
|
||||
const result = await cli.run({
|
||||
arguments: ["inspect-python-pyc", "sample.pyc", "--json"],
|
||||
cwd: root,
|
||||
});
|
||||
expect(result.exitCode, result.stderr).toBe(0);
|
||||
const evidence = toolContract("inspect_python_pyc").outputSchema.parse(
|
||||
result.json,
|
||||
);
|
||||
expect(evidence.normalized_result.header.python_version).toBe("3.7");
|
||||
expect(evidence.subject?.local_path).toBe(join(root, "sample.pyc"));
|
||||
},
|
||||
);
|
||||
|
||||
cliTest.skipIf(process.platform === "win32")(
|
||||
"reads the file the symlink path names, not its lexical sibling",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-python-pyc-cli-link-");
|
||||
const target = join(root, "target");
|
||||
await mkdir(join(target, "child"), { recursive: true });
|
||||
await writeFile(join(target, "sample.pyc"), pythonPycFixture("py37"));
|
||||
await writeFile(join(root, "sample.pyc"), pythonPycFixture("py311"));
|
||||
await symlink(join(target, "child"), join(root, "link"));
|
||||
const result = await cli.run({
|
||||
arguments: ["inspect-python-pyc", "link/../sample.pyc", "--json"],
|
||||
cwd: root,
|
||||
});
|
||||
expect(result.exitCode, result.stderr).toBe(0);
|
||||
const evidence = toolContract("inspect_python_pyc").outputSchema.parse(
|
||||
result.json,
|
||||
);
|
||||
expect(evidence.normalized_result.header.python_version).toBe("3.7");
|
||||
expect(evidence.subject?.local_path).toBe(`${root}/link/../sample.pyc`);
|
||||
},
|
||||
);
|
||||
|
||||
cliTest("reports a truncated pyc as invalid input", async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-python-pyc-cli-bad-");
|
||||
const path = join(root, "broken.pyc");
|
||||
await writeFile(path, Buffer.from("not a pyc"));
|
||||
const result = await cli.run({
|
||||
arguments: ["inspect-python-pyc", path, "--json"],
|
||||
});
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.json).toMatchObject({
|
||||
code: "invalid_request",
|
||||
details: { operation: "inspect_python_pyc" },
|
||||
});
|
||||
});
|
||||
@@ -221,6 +221,7 @@ const pathRequests: readonly [string, Record<string, unknown>][] = [
|
||||
["inspect_pe_certificates", { path: nul }],
|
||||
["inspect_native_exports", { path: nul }],
|
||||
["inspect_elf_notes", { path: nul }],
|
||||
["inspect_python_pyc", { path: nul }],
|
||||
["inspect_erlang_beam", { path: nul }],
|
||||
["inspect_binary_layout", { path: nul }],
|
||||
["inspect_recorded_crash", { path: nul }],
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
import { symlink, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createPythonPycService } from "../../../src/composition/pythonPyc.js";
|
||||
import { pythonPycFixture } from "../../../src/python/PythonPyc.fixture.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
|
||||
describe("Python pyc stable filesystem boundary", () => {
|
||||
it("reports non-file and malformed pyc selections as caller input errors", async () => {
|
||||
const root = await createTestTempDirectory("rea-python-pyc-files-");
|
||||
const path = join(root, "sample.pyc");
|
||||
const alias = join(root, "alias.pyc");
|
||||
const malformed = join(root, "malformed.pyc");
|
||||
await writeFile(path, pythonPycFixture());
|
||||
await writeFile(malformed, Buffer.from("not a pyc"));
|
||||
await symlink(path, alias);
|
||||
const service = createPythonPycService();
|
||||
for (const [candidate, reason] of [
|
||||
[root, "invalid_format"],
|
||||
[alias, "invalid_format"],
|
||||
[malformed, "invalid_format"],
|
||||
[join(root, "missing.pyc"), "invalid_value"],
|
||||
[join(path, "child.pyc"), "invalid_value"],
|
||||
] as const) {
|
||||
const result = await service.inspect({ path: candidate });
|
||||
if (result.ok) throw new Error("Expected file acquisition failure");
|
||||
expect(result.error).toMatchObject({
|
||||
_tag: "AnalysisInputError",
|
||||
operation: "inspect_python_pyc",
|
||||
issues: [
|
||||
{
|
||||
path: ["path"],
|
||||
reason,
|
||||
message: expect.any(String),
|
||||
},
|
||||
],
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("returns typed resource and cancellation failures", async () => {
|
||||
const root = await createTestTempDirectory("rea-python-pyc-budget-");
|
||||
const path = join(root, "sample.pyc");
|
||||
const bytes = pythonPycFixture("py311", { nested: true });
|
||||
await writeFile(path, bytes);
|
||||
const service = createPythonPycService();
|
||||
const tooSmall = await service.inspect({ path, max_file_bytes: 16 });
|
||||
if (tooSmall.ok) throw new Error("Expected resource budget failure");
|
||||
expect(tooSmall.error._tag).toBe("AnalysisResourceConstraintError");
|
||||
const tooFewObjects = await service.inspect({ path, max_objects: 1 });
|
||||
if (tooFewObjects.ok) throw new Error("Expected object budget failure");
|
||||
expect(tooFewObjects.error._tag).toBe("AnalysisResourceConstraintError");
|
||||
const cancelled = await service.inspect(
|
||||
{ path },
|
||||
{ signal: AbortSignal.abort() },
|
||||
);
|
||||
if (cancelled.ok) throw new Error("Expected cancellation");
|
||||
expect(cancelled.error._tag).toBe("AnalysisCancelledError");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,88 @@
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect } from "vitest";
|
||||
import { toolContract } from "../../../src/contracts/toolContracts.js";
|
||||
import { createServer } from "../../../src/server/createServer.js";
|
||||
import { createTestBinarySession } from "../../fixtures/binarySession.js";
|
||||
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
|
||||
import { pythonPycFixture } from "../../../src/python/PythonPyc.fixture.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { mcpTest, type TestMcp } from "../../support/mcp/mcpFixture.js";
|
||||
|
||||
const connect = async (mcp: TestMcp) => {
|
||||
const session = createTestBinarySession(() => {
|
||||
throw new Error("Python pyc inspection must not start a deep provider");
|
||||
});
|
||||
const client = await mcp.connect(createServer({ kind: "session", session }));
|
||||
return { client, session };
|
||||
};
|
||||
|
||||
mcpTest(
|
||||
"advertises inspect_python_pyc and reads nested code objects",
|
||||
async ({ mcp }) => {
|
||||
const { client, session } = await connect(mcp);
|
||||
try {
|
||||
const advertised = (await client.listTools()).tools.find(
|
||||
(tool) => tool.name === "inspect_python_pyc",
|
||||
);
|
||||
expect(advertised?.annotations).toMatchObject({
|
||||
readOnlyHint: false,
|
||||
idempotentHint: true,
|
||||
});
|
||||
const root = await createTestTempDirectory("rea-python-pyc-mcp-");
|
||||
const path = join(root, "sample.pyc");
|
||||
await writeFile(path, pythonPycFixture("py311", { nested: true }));
|
||||
const response = await client.callTool({
|
||||
name: "inspect_python_pyc",
|
||||
arguments: { path },
|
||||
});
|
||||
expect(response.isError, JSON.stringify(response.content)).not.toBe(true);
|
||||
const evidence = toolContract("inspect_python_pyc").outputSchema.parse(
|
||||
response.structuredContent,
|
||||
);
|
||||
expect(evidence.normalized_result.header.python_version).toBe("3.12");
|
||||
expect(evidence.normalized_result.code_objects[1]?.name).toBe("inner");
|
||||
} finally {
|
||||
await session.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
mcpTest("requires an absolute path for inspect_python_pyc", async ({ mcp }) => {
|
||||
const { client, session } = await connect(mcp);
|
||||
try {
|
||||
const response = await client.callTool({
|
||||
name: "inspect_python_pyc",
|
||||
arguments: { path: "sample.pyc" },
|
||||
});
|
||||
expect(response.isError).toBe(true);
|
||||
expect(parseMcpToolError(response).error).toMatchObject({
|
||||
code: "invalid_request",
|
||||
details: { operation: "inspect_python_pyc" },
|
||||
});
|
||||
} finally {
|
||||
await session.close();
|
||||
}
|
||||
});
|
||||
|
||||
mcpTest("reports an unknown pyc magic as unsupported", async ({ mcp }) => {
|
||||
const { client, session } = await connect(mcp);
|
||||
try {
|
||||
const root = await createTestTempDirectory("rea-python-pyc-mcp-magic-");
|
||||
const path = join(root, "unknown.pyc");
|
||||
const bytes = pythonPycFixture("py311");
|
||||
bytes.writeUInt16LE(1, 0);
|
||||
await writeFile(path, bytes);
|
||||
const response = await client.callTool({
|
||||
name: "inspect_python_pyc",
|
||||
arguments: { path },
|
||||
});
|
||||
expect(response.isError).toBe(true);
|
||||
expect(parseMcpToolError(response).error).toMatchObject({
|
||||
code: "unsupported_target",
|
||||
details: { operation: "inspect_python_pyc" },
|
||||
});
|
||||
} finally {
|
||||
await session.close();
|
||||
}
|
||||
});
|
||||
@@ -78,6 +78,8 @@ import {
|
||||
import { ArtifactResourceScope } from "../../src/artifacts/ArtifactResourceScope.js";
|
||||
import { ArtifactReaderFailure } from "../../src/artifacts/ArtifactReader.js";
|
||||
import { GoBinaryProvider } from "../../src/go/GoBinaryProvider.js";
|
||||
import { PythonPycProvider } from "../../src/python/PythonPycProvider.js";
|
||||
import { inspectPythonPycInputSchema } from "../../src/domain/python/pythonPyc.js";
|
||||
import { PeResourcesProvider } from "../../src/native/pe/PeResourcesProvider.js";
|
||||
import { Il2cppMetadataProvider } from "../../src/il2cpp/Il2cppMetadataProvider.js";
|
||||
import { ElfNotesProvider } from "../../src/native/elf/ElfNotesProvider.js";
|
||||
@@ -303,6 +305,30 @@ it.each(staticProviders)(
|
||||
}
|
||||
},
|
||||
);
|
||||
it("clears the Python pyc cleanup flag once a retained descriptor closes", async () => {
|
||||
const path = await selectedImage();
|
||||
const provider = new PythonPycProvider();
|
||||
const primary = new ArtifactReaderFailure(
|
||||
"integrity",
|
||||
"controlled source replacement",
|
||||
);
|
||||
fault.readFailure = { cause: primary };
|
||||
try {
|
||||
const result = await provider.inspect(
|
||||
inspectPythonPycInputSchema.parse({ path }),
|
||||
);
|
||||
if (result.ok)
|
||||
throw new Error("Expected primary failure and retained descriptor");
|
||||
expect(result.error.cleanupIncomplete).toBe(true);
|
||||
fault.blocked.clear();
|
||||
await provider.close();
|
||||
expect(result.error.cleanupIncomplete).toBe(false);
|
||||
} finally {
|
||||
fault.blocked.clear();
|
||||
await provider.close();
|
||||
}
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
fault.blockNew = false;
|
||||
fault.blocked.clear();
|
||||
|
||||
Reference in new issue
Block a user