mirror of
https://github.com/morluto/rea.git
synced 2026-11-15 19:23:24 +00:00
feat(go): recover bounded Go string literals from (ptr, len) headers (#9206)
* feat(go): recover bounded Go string literals from (ptr, len) headers * fix(go): deduplicate header scan spans, bound rejected cache, and reject Cf * fix(go): allow U+200C ZWNJ and U+200D ZWJ in Go string literal recovery * fix(go): index Mach-O load regions, validate MCP schema, and add feature specs * test(go): cover Mach-O region-index cancellation and load-region boundaries
This commit is contained in:
1 parent
5553acfe50
commit
94182f331e
26 files changed
+2709
-44
No files matched your search
@@ -49,14 +49,16 @@ selected dependency versions without reopening the file:
|
||||
}
|
||||
```
|
||||
|
||||
Go views support `dependencies` (exact `paths`) and `settings` (exact `keys`),
|
||||
optional original `indices`, field selection, and pagination. Filters intersect;
|
||||
empty filter arrays select no rows and `fields: []` returns only row identity.
|
||||
Each row keeps its original `index`, scoped to the parent Evidence and collection;
|
||||
duplicate module paths and setting keys keep distinct indices. Page offsets
|
||||
count matching records. Exhausting retained rows does not establish complete
|
||||
runtime dependencies. Original bytes, unknown lines and all complete metadata
|
||||
remain in the parent and can be exported as an Evidence bundle.
|
||||
Go views support `dependencies` (exact `paths`), `settings` (exact `keys`) and,
|
||||
when collected with `include_string_literals: true`, `string_literals` (exact
|
||||
`values` and substring `contains`), plus optional original `indices`, field
|
||||
selection, and pagination. Filters intersect; empty filter arrays select no rows
|
||||
and `fields: []` returns only row identity. Each row keeps its original `index`,
|
||||
scoped to the parent Evidence and collection; duplicate module paths and setting
|
||||
keys keep distinct indices. Page offsets count matching records. Exhausting
|
||||
retained rows does not establish complete runtime dependencies or string
|
||||
coverage. Original bytes, unknown lines and all complete metadata remain in the
|
||||
parent and can be exported as an Evidence bundle.
|
||||
|
||||
One-shot CLI summaries save complete Evidence before returning the projection:
|
||||
|
||||
@@ -113,6 +115,22 @@ their empty dependency array does not establish the absence of dependencies.
|
||||
The result's limitations state this explicitly. A module-mode build with a main
|
||||
module record and no dependency records has an observed empty dependency list.
|
||||
|
||||
When `include_string_literals: true` (CLI `--include-string-literals`, optional
|
||||
`max_string_literals` / `--max-string-literals`, default `2000`, maximum `10000`)
|
||||
is selected, `string_literals` reports `evidence_kind: "inferred"` alongside
|
||||
deduplicated printable UTF-8 literals referenced by word-aligned two-word
|
||||
`(ptr, len)` string headers in non-executable initialized data. Because Go packs
|
||||
read-only string data contiguously without NUL terminators, slicing by `(ptr, len)`
|
||||
recovers exact literal boundaries instead of concatenated C-string runs. Each
|
||||
target slice (`1`–`4096` bytes) must stay within a single read-only mapped
|
||||
section or segment, decode as printable UTF-8 (`\t`, `\n`, `\r`, joiners
|
||||
`U+200C` / `U+200D` and printable Unicode scalar values, excluding control and
|
||||
other format characters), and fit within
|
||||
a 64 MiB header-scan budget and 512 KiB aggregate string budget
|
||||
(`truncated: true` when capped). Compiler-inlined or
|
||||
stack-constructed strings without static headers, unapplied relocation addends
|
||||
and Mach-O `LC_DYLD_CHAINED_FIXUPS` pointer encodings are noted in `limitations`.
|
||||
|
||||
## Supported boundary
|
||||
|
||||
The built-in TypeScript reader supports ELF, PE and thin Mach-O images, and one
|
||||
|
||||
@@ -10,3 +10,8 @@ for the specification lifecycle, roadmap granularity and producer-owned contract
|
||||
|
||||
- [Organization](documentation/organization/spec.md)
|
||||
([plan](documentation/organization/plan.md), [tasks](documentation/organization/tasks.md))
|
||||
|
||||
## Native
|
||||
|
||||
- [Go static string-header literal recovery](native/go-string-literals/spec.md)
|
||||
([plan](native/go-string-literals/plan.md), [tasks](native/go-string-literals/tasks.md))
|
||||
@@ -0,0 +1,40 @@
|
||||
# Implementation plan
|
||||
|
||||
## Boundary and contract
|
||||
|
||||
Extend `goBinarySchema` in `src/domain/go/goBinary.ts` with optional
|
||||
`string_literals` (`evidence_kind: "inferred"`, `items`, `discovered_count`,
|
||||
`aggregate_bytes`, `max_items`, `truncated`), plus `include_string_literals` and
|
||||
`max_string_literals` on `src/contracts/go/goToolContracts.ts` and
|
||||
`src/cli/goCommands.ts`. Add the `string_literals` view projection to
|
||||
`src/domain/analysisView/goBinaryView.ts` so callers can page recovered literals
|
||||
from retained Evidence without repeating file parsing.
|
||||
|
||||
Implement `recoverGoStringLiterals` in `src/go/GoStringLiterals.ts` on top of
|
||||
`BinaryContainer` and `createFileMappingIndex`. Pre-index readable `LoadRegion`
|
||||
intervals once in `O(R log R)` so Mach-O section eligibility runs in `O(log R)`
|
||||
with periodic `checkActive` cancellation checks. Normalize header-scan spans by
|
||||
pointer alignment phase, merge overlaps, and sort by ascending file offset so
|
||||
`reference_count` counts each physical header once and `first_header_offset` is
|
||||
minimal. Deduplicate target slices in memory with bounded candidate and rejection
|
||||
caches before sorting deterministically by `(location.offset, location.bytes)`.
|
||||
|
||||
## Incremental work and validation
|
||||
|
||||
1. Define domain/MCP/CLI schemas, limitation notes and `string_literals` analysis
|
||||
view behavior while preserving default omission when not requested.
|
||||
2. Implement `recoverGoStringLiterals` and wire `GoBinaryImage`,
|
||||
`GoBinaryProvider` and `GoBinaryService` with cooperative cancellation.
|
||||
3. Add unit fixtures across ELF/PE/Mach-O, Unicode format-control filtering,
|
||||
budget/cap truncation, overlapping ELF spans, bounded negative caching and
|
||||
25,000-section Mach-O indexing/cancellation.
|
||||
4. Extend `scripts/verify-go-binary.mjs` via `artifactMcpCall` so real compiled Go
|
||||
targets verify both CLI and advertised MCP `outputSchema` parity.
|
||||
5. Update the [Go binary guide](../../../guides/native/go-binaries.md), skill
|
||||
guidance, generated catalogs and documentation route checks.
|
||||
|
||||
## Effects and compatibility
|
||||
|
||||
Default `inspect_go_binary` calls omit `string_literals` and preserve historical
|
||||
Evidence compatibility. No external tool or target execution is introduced; only
|
||||
bounded offline byte slicing over the already read `64 MiB` image buffer runs.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Go static string-header literal recovery
|
||||
|
||||
## Problem and outcome
|
||||
|
||||
Go compilers pack static string contents contiguously in read-only sections
|
||||
(`.rodata`, `__TEXT,__rodata` / `__TEXT,__const`, `.rdata`) without NUL
|
||||
terminators. Generic C-string scans merge adjacent Go literals into multi-kilobyte
|
||||
runs. Extend `inspect_go_binary` / `inspect-go-binary` with opt-in, bounded
|
||||
`(ptr, len)` `reflect.StringHeader` recovery across ELF, PE and thin Mach-O
|
||||
images so static string slices are recovered at their exact boundaries.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Keep `include_string_literals` opt-in (`false` by default) so existing
|
||||
build-metadata output and payload size are unchanged unless requested.
|
||||
- Scan only pointer-aligned non-executable initialized data spans (with indexed
|
||||
`O(log R)` readable segment checks on Mach-O and sectionless/unnamed-ELF
|
||||
load-region fallbacks) in deduplicated ascending file-offset order.
|
||||
- Admit candidate `[ptr, ptr + len)` slices only inside single read-only
|
||||
non-executable mapped spans with `1 <= len <= 4096`, non-overlapping with the
|
||||
header itself, and valid printable UTF-8 (allowing `\t`, `\n`, `\r`, `U+200C`
|
||||
and `U+200D` while rejecting control, surrogate, noncharacter, private-use and
|
||||
invisible/bidi `\p{Cf}` code points).
|
||||
- Bound header-scan work (`64 MiB`), tracked candidates (`100,000`), negative
|
||||
slice cache (`4,096`), returned items (`max_string_literals`, `1..10,000`,
|
||||
default `2,000`), and aggregate returned UTF-8 bytes (`512 KiB`), reporting
|
||||
`evidence_kind: "inferred"`, `discovered_count`, `aggregate_bytes` and
|
||||
`truncated` honestly.
|
||||
- Record limitation notes when Mach-O `LC_DYLD_CHAINED_FIXUPS` encodes rebases in
|
||||
pointer slots or when truncation occurs, and expose a focused `string_literals`
|
||||
projection in `inspect_analysis_view`.
|
||||
|
||||
## Acceptance
|
||||
|
||||
Unit, boundary and CLI acceptance tests cover ELF/PE/Mach-O (32/64-bit, little
|
||||
and big endian), sectionless and unreadable-`.shstrtab` ELF images, overlapping
|
||||
spans, `U+200C`/`U+200D` versus disallowed `\p{Cf}` controls, negative-cache and
|
||||
output caps, and 25,000-region Mach-O indexing/cancellation. The real Go binary
|
||||
verifier validates CLI and advertised-schema MCP parity across compiled targets.
|
||||
|
||||
## Out of scope
|
||||
|
||||
Instruction-operand immediate string recovery (stack/register `MOV`/`LEA` pairs
|
||||
without static two-word headers), Mach-O chained-fixup rebase decoding, binary
|
||||
execution, and non-UTF-8 byte-slice guessing.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Tasks
|
||||
|
||||
- [x] Define single-feature scope, contracts and implementation plan (`spec.md`, `plan.md`).
|
||||
- [x] Extend `goBinarySchema`, MCP contracts, CLI flags and `inspect_analysis_view` (`string_literals`).
|
||||
- [x] Implement bounded `(ptr, len)` recovery in `src/go/GoStringLiterals.ts` with `O(R log R)` Mach-O region indexing and non-overlapping file-order header scanning.
|
||||
- [x] Add unit, provider, MCP view and CLI acceptance tests for multi-format recovery, Unicode `\p{Cf}` / ZWNJ / ZWJ filtering, caps, negative-cache bounds and 25,000-section Mach-O scaling.
|
||||
- [x] Verify CLI and `artifactMcpCall` advertised-schema MCP parity across compiled targets in `scripts/verify-go-binary.mjs`.
|
||||
- [x] Update the Go binary guide, skill reference and generated catalogs; pass documentation route and static checks.
|
||||
Reference in new issue
Block a user