feat(go): recover bounded Go string literals from (ptr, len) headers (#9206)

* feat(go): recover bounded Go string literals from (ptr, len) headers

* fix(go): deduplicate header scan spans, bound rejected cache, and reject Cf

* fix(go): allow U+200C ZWNJ and U+200D ZWJ in Go string literal recovery

* fix(go): index Mach-O load regions, validate MCP schema, and add feature specs

* test(go): cover Mach-O region-index cancellation and load-region boundaries
This commit is contained in:
Ryan S authored and GitHub committed 2026-11-08 22:50:32 +00:00
1 parent 5553acfe50
commit 94182f331e
26 files changed
+2709 -44

No files matched your search

+26 -8
View File
@@ -49,14 +49,16 @@ selected dependency versions without reopening the file:
}
```
Go views support `dependencies` (exact `paths`) and `settings` (exact `keys`),
optional original `indices`, field selection, and pagination. Filters intersect;
empty filter arrays select no rows and `fields: []` returns only row identity.
Each row keeps its original `index`, scoped to the parent Evidence and collection;
duplicate module paths and setting keys keep distinct indices. Page offsets
count matching records. Exhausting retained rows does not establish complete
runtime dependencies. Original bytes, unknown lines and all complete metadata
remain in the parent and can be exported as an Evidence bundle.
Go views support `dependencies` (exact `paths`), `settings` (exact `keys`) and,
when collected with `include_string_literals: true`, `string_literals` (exact
`values` and substring `contains`), plus optional original `indices`, field
selection, and pagination. Filters intersect; empty filter arrays select no rows
and `fields: []` returns only row identity. Each row keeps its original `index`,
scoped to the parent Evidence and collection; duplicate module paths and setting
keys keep distinct indices. Page offsets count matching records. Exhausting
retained rows does not establish complete runtime dependencies or string
coverage. Original bytes, unknown lines and all complete metadata remain in the
parent and can be exported as an Evidence bundle.
One-shot CLI summaries save complete Evidence before returning the projection:
@@ -113,6 +115,22 @@ their empty dependency array does not establish the absence of dependencies.
The result's limitations state this explicitly. A module-mode build with a main
module record and no dependency records has an observed empty dependency list.
When `include_string_literals: true` (CLI `--include-string-literals`, optional
`max_string_literals` / `--max-string-literals`, default `2000`, maximum `10000`)
is selected, `string_literals` reports `evidence_kind: "inferred"` alongside
deduplicated printable UTF-8 literals referenced by word-aligned two-word
`(ptr, len)` string headers in non-executable initialized data. Because Go packs
read-only string data contiguously without NUL terminators, slicing by `(ptr, len)`
recovers exact literal boundaries instead of concatenated C-string runs. Each
target slice (`1`–`4096` bytes) must stay within a single read-only mapped
section or segment, decode as printable UTF-8 (`\t`, `\n`, `\r`, joiners
`U+200C` / `U+200D` and printable Unicode scalar values, excluding control and
other format characters), and fit within
a 64 MiB header-scan budget and 512 KiB aggregate string budget
(`truncated: true` when capped). Compiler-inlined or
stack-constructed strings without static headers, unapplied relocation addends
and Mach-O `LC_DYLD_CHAINED_FIXUPS` pointer encodings are noted in `limitations`.
## Supported boundary
The built-in TypeScript reader supports ELF, PE and thin Mach-O images, and one
+5
View File
@@ -10,3 +10,8 @@ for the specification lifecycle, roadmap granularity and producer-owned contract
- [Organization](documentation/organization/spec.md)
([plan](documentation/organization/plan.md), [tasks](documentation/organization/tasks.md))
## Native
- [Go static string-header literal recovery](native/go-string-literals/spec.md)
([plan](native/go-string-literals/plan.md), [tasks](native/go-string-literals/tasks.md))
@@ -0,0 +1,40 @@
# Implementation plan
## Boundary and contract
Extend `goBinarySchema` in `src/domain/go/goBinary.ts` with optional
`string_literals` (`evidence_kind: "inferred"`, `items`, `discovered_count`,
`aggregate_bytes`, `max_items`, `truncated`), plus `include_string_literals` and
`max_string_literals` on `src/contracts/go/goToolContracts.ts` and
`src/cli/goCommands.ts`. Add the `string_literals` view projection to
`src/domain/analysisView/goBinaryView.ts` so callers can page recovered literals
from retained Evidence without repeating file parsing.
Implement `recoverGoStringLiterals` in `src/go/GoStringLiterals.ts` on top of
`BinaryContainer` and `createFileMappingIndex`. Pre-index readable `LoadRegion`
intervals once in `O(R log R)` so Mach-O section eligibility runs in `O(log R)`
with periodic `checkActive` cancellation checks. Normalize header-scan spans by
pointer alignment phase, merge overlaps, and sort by ascending file offset so
`reference_count` counts each physical header once and `first_header_offset` is
minimal. Deduplicate target slices in memory with bounded candidate and rejection
caches before sorting deterministically by `(location.offset, location.bytes)`.
## Incremental work and validation
1. Define domain/MCP/CLI schemas, limitation notes and `string_literals` analysis
view behavior while preserving default omission when not requested.
2. Implement `recoverGoStringLiterals` and wire `GoBinaryImage`,
`GoBinaryProvider` and `GoBinaryService` with cooperative cancellation.
3. Add unit fixtures across ELF/PE/Mach-O, Unicode format-control filtering,
budget/cap truncation, overlapping ELF spans, bounded negative caching and
25,000-section Mach-O indexing/cancellation.
4. Extend `scripts/verify-go-binary.mjs` via `artifactMcpCall` so real compiled Go
targets verify both CLI and advertised MCP `outputSchema` parity.
5. Update the [Go binary guide](../../../guides/native/go-binaries.md), skill
guidance, generated catalogs and documentation route checks.
## Effects and compatibility
Default `inspect_go_binary` calls omit `string_literals` and preserve historical
Evidence compatibility. No external tool or target execution is introduced; only
bounded offline byte slicing over the already read `64 MiB` image buffer runs.
@@ -0,0 +1,45 @@
# Go static string-header literal recovery
## Problem and outcome
Go compilers pack static string contents contiguously in read-only sections
(`.rodata`, `__TEXT,__rodata` / `__TEXT,__const`, `.rdata`) without NUL
terminators. Generic C-string scans merge adjacent Go literals into multi-kilobyte
runs. Extend `inspect_go_binary` / `inspect-go-binary` with opt-in, bounded
`(ptr, len)` `reflect.StringHeader` recovery across ELF, PE and thin Mach-O
images so static string slices are recovered at their exact boundaries.
## Requirements
- Keep `include_string_literals` opt-in (`false` by default) so existing
build-metadata output and payload size are unchanged unless requested.
- Scan only pointer-aligned non-executable initialized data spans (with indexed
`O(log R)` readable segment checks on Mach-O and sectionless/unnamed-ELF
load-region fallbacks) in deduplicated ascending file-offset order.
- Admit candidate `[ptr, ptr + len)` slices only inside single read-only
non-executable mapped spans with `1 <= len <= 4096`, non-overlapping with the
header itself, and valid printable UTF-8 (allowing `\t`, `\n`, `\r`, `U+200C`
and `U+200D` while rejecting control, surrogate, noncharacter, private-use and
invisible/bidi `\p{Cf}` code points).
- Bound header-scan work (`64 MiB`), tracked candidates (`100,000`), negative
slice cache (`4,096`), returned items (`max_string_literals`, `1..10,000`,
default `2,000`), and aggregate returned UTF-8 bytes (`512 KiB`), reporting
`evidence_kind: "inferred"`, `discovered_count`, `aggregate_bytes` and
`truncated` honestly.
- Record limitation notes when Mach-O `LC_DYLD_CHAINED_FIXUPS` encodes rebases in
pointer slots or when truncation occurs, and expose a focused `string_literals`
projection in `inspect_analysis_view`.
## Acceptance
Unit, boundary and CLI acceptance tests cover ELF/PE/Mach-O (32/64-bit, little
and big endian), sectionless and unreadable-`.shstrtab` ELF images, overlapping
spans, `U+200C`/`U+200D` versus disallowed `\p{Cf}` controls, negative-cache and
output caps, and 25,000-region Mach-O indexing/cancellation. The real Go binary
verifier validates CLI and advertised-schema MCP parity across compiled targets.
## Out of scope
Instruction-operand immediate string recovery (stack/register `MOV`/`LEA` pairs
without static two-word headers), Mach-O chained-fixup rebase decoding, binary
execution, and non-UTF-8 byte-slice guessing.
@@ -0,0 +1,8 @@
# Tasks
- [x] Define single-feature scope, contracts and implementation plan (`spec.md`, `plan.md`).
- [x] Extend `goBinarySchema`, MCP contracts, CLI flags and `inspect_analysis_view` (`string_literals`).
- [x] Implement bounded `(ptr, len)` recovery in `src/go/GoStringLiterals.ts` with `O(R log R)` Mach-O region indexing and non-overlapping file-order header scanning.
- [x] Add unit, provider, MCP view and CLI acceptance tests for multi-format recovery, Unicode `\p{Cf}` / ZWNJ / ZWJ filtering, caps, negative-cache bounds and 25,000-section Mach-O scaling.
- [x] Verify CLI and `artifactMcpCall` advertised-schema MCP parity across compiled targets in `scripts/verify-go-binary.mjs`.
- [x] Update the Go binary guide, skill reference and generated catalogs; pass documentation route and static checks.