mirror of
https://github.com/morluto/rea.git
synced 2026-11-02 21:06:04 +00:00
chore: merge main into the streamable HTTP transport branch
Picks up upstream's test-runner configuration and the command-finalization test fixes so the fork lane stops failing on unrelated flaky shards. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Edgar ArakelyanandClaude Fable 5.1 committed
commit
cfbf595437
135 files changed
+33233
-1087
No files matched your search
@@ -1,3 +1,3 @@
|
||||
{
|
||||
".": "6.5.0"
|
||||
".": "6.7.0"
|
||||
}
|
||||
@@ -1,5 +1,93 @@
|
||||
# Changelog
|
||||
|
||||
## [6.7.0](https://github.com/morluto/rea/compare/rea-agents-6.6.0...rea-agents-6.7.0) (2026-10-18)
|
||||
|
||||
### Highlights
|
||||
|
||||
- Client configuration edits follow the duplicate-member semantics used by MCP clients.
|
||||
|
||||
### Changes
|
||||
|
||||
- Includes the [6.6.0 changes and migration notes](https://github.com/morluto/rea/releases/tag/rea-agents-6.6.0). That version was tagged but was not published to npm.
|
||||
|
||||
### Fixes
|
||||
|
||||
- **Client configuration:** Setup edits the last repeated JSON/JSONC member, matching the value clients actually read. Setup and uninstall refuse a removal that would reactivate a shadowed duplicate, preserving the original configuration. ([#4957](https://github.com/morluto/rea/pull/4957), [morluto/rea#4952](https://github.com/morluto/rea/issues/4952)). Thanks @kaoru0822-kitauji, @xu91102.
|
||||
|
||||
Full contribution record: [4 commits and 1 merged PR](https://github.com/morluto/rea/blob/rea-agents-6.7.0/docs/releases/6.7.0.contributions.json).
|
||||
|
||||
## [6.6.0](https://github.com/morluto/rea/compare/rea-agents-6.5.0...rea-agents-6.6.0) (2026-10-18)
|
||||
|
||||
### Highlights
|
||||
|
||||
- Add Kuna as a caller-supplied static-analysis provider.
|
||||
- Query selected JavaScript facts and multiple APK locale tables without repeating the underlying analysis.
|
||||
- Extend Ghidra analysis to Windows ARM PE targets and explicit raw m68k images.
|
||||
- Preserve completed Evidence, cancellation reasons and retryable cleanup across provider and CLI finalization.
|
||||
|
||||
### Changes
|
||||
|
||||
- **Kuna analysis:** Use a caller-installed Kuna for procedure listings, pseudocode, assembly, call relationships, exact-address xrefs and cached string searches. Project exports retain source spans and digests; repeated requests reuse completed work. Recorded real-provider coverage uses Kuna 1.833 on Linux; Windows, PE and universal Mach-O are unsupported. See [the provider guide](docs/kuna-provider.md). ([#3832](https://github.com/morluto/rea/pull/3832), [#3854](https://github.com/morluto/rea/pull/3854), [#4071](https://github.com/morluto/rea/pull/4071), [#4095](https://github.com/morluto/rea/pull/4095)). Thanks @gensokyo-dev, @kaoru0822-kitauji, @N0zoM1z0.
|
||||
- **Retained JavaScript facts:** Query exact application paths, selected properties and filtered module references through `inspect_analysis_view`, with paging and complete row Evidence. These views reuse completed analysis and preserve source selection and unknown coverage. ([#4265](https://github.com/morluto/rea/pull/4265), [morluto/rea#4190](https://github.com/morluto/rea/issues/4190), [#4097](https://github.com/morluto/rea/pull/4097), [morluto/rea#4092](https://github.com/morluto/rea/issues/4092)). Thanks @kaoru0822-kitauji, @N0zoM1z0.
|
||||
- **APK locale tables:** Request several resource locales in one decode with repeatable CLI `--locales` or MCP `locales`. Each table reports its status and coverage; existing single-locale fields remain available. ([#4804](https://github.com/morluto/rea/pull/4804), [morluto/rea#4777](https://github.com/morluto/rea/issues/4777)). Thanks @abhay-2108, @N0zoM1z0.
|
||||
- **Windows ARM binaries:** Ghidra admits ARMNT/Thumb-2 and ARM64 PE applications and DLLs on the Windows x64 host lane. Recorded Ghidra 12.1.4 verification uses `WindowsResourceReference` disabled and raised startup budgets; default-analyzer completion, ARM64EC/ARM64X and Windows-on-Arm hosts remain unverified or unsupported. See [the exact scope](docs/windows-ghidra-p0.md#native-authority-and-target-identity). ([#3965](https://github.com/morluto/rea/pull/3965), [morluto/rea#2186](https://github.com/morluto/rea/issues/2186)). Thanks @N0zoM1z0, @WestLabsAI.
|
||||
- **Raw m68k images:** Analyze explicitly selected big-endian m68k raw binaries using Ghidra’s fixed 68000 profile and a caller-supplied aligned entry. Real CLI/MCP verification used Ghidra 12.1.4 on macOS arm64; Linux is unverified and Windows P0 does not admit m68k. See [mapping requirements](docs/raw-binary.md). ([#4018](https://github.com/morluto/rea/pull/4018)). Thanks @icedac.
|
||||
- **JADX deadlines:** Set `REA_JADX_OPERATION_TIMEOUT_MS` to extend APK loading and upstream operation deadlines together. Doctor reports the effective value and setup forwards an explicit selection. ([#4466](https://github.com/morluto/rea/pull/4466), [morluto/rea#3918](https://github.com/morluto/rea/issues/3918)). Thanks @jaychou813, @SandstromPL.
|
||||
- **Electron window options:** Resolve stable local constant `BrowserWindow` options while retaining source provenance. Relevant writes and escapes keep the result unknown. ([#4400](https://github.com/morluto/rea/pull/4400), [morluto/rea#4366](https://github.com/morluto/rea/issues/4366)). Thanks @kaoru0822-kitauji, @N0zoM1z0.
|
||||
- **Language editions:** Add the Greek website edition and Malayalam documentation, with Korean typography and translation corrections across the site. ([#3520](https://github.com/morluto/rea/pull/3520), [#4792](https://github.com/morluto/rea/pull/4792), [#3821](https://github.com/morluto/rea/pull/3821), [morluto/rea#3820](https://github.com/morluto/rea/issues/3820), [#3799](https://github.com/morluto/rea/pull/3799), [morluto/rea#3798](https://github.com/morluto/rea/issues/3798), [#4344](https://github.com/morluto/rea/pull/4344), [#3856](https://github.com/morluto/rea/pull/3856), [#4166](https://github.com/morluto/rea/pull/4166)). Thanks @gensokyo-dev, @hi2242, @kalayciburak, @kaoru0822-kitauji, @kraken-503, @kukinghan, @N0zoM1z0, @yeseong0412.
|
||||
|
||||
### Fixes
|
||||
|
||||
- **Cancellation and cleanup:** Keep completed observations, exact cancellation reasons and the original physical failure through finalization. Failed cleanup retains its process or file owner for retry; recovered resources disappear from pending-cleanup reports without replacing the original error. ([#4238](https://github.com/morluto/rea/pull/4238), [morluto/rea#4232](https://github.com/morluto/rea/issues/4232), [#4274](https://github.com/morluto/rea/pull/4274), [morluto/rea#3923](https://github.com/morluto/rea/issues/3923), [3e82366](https://github.com/morluto/rea/commit/3e8236639abffa39deb5d5e939ec8baf6aaf3ca3), [16924e2](https://github.com/morluto/rea/commit/16924e275e939d0543ba01493324959a9e1d731c), [bbfe6ed](https://github.com/morluto/rea/commit/bbfe6edd60290f9bfbec68c8a0bcf605d8bbf5b3), [1f23e97](https://github.com/morluto/rea/commit/1f23e977b50301e91f02a4a17c9f8d48bb7b60f3), [08fbeb8](https://github.com/morluto/rea/commit/08fbeb8bdab27118b6c57039aef7c0fa8b6463bc)). Thanks @coygeek.
|
||||
- **Provider startup:** Join prerequisite probes and in-flight readiness work before shutdown, preserve typed timeout and JVM-memory failures, and retain startup diagnostics through cleanup. Ghidra recognizes keg-only Homebrew JDKs and names the setting responsible for a startup timeout. ([376eeca](https://github.com/morluto/rea/commit/376eeca770820b31f7f8d4c5fcc397315621643e), [ceeff5a](https://github.com/morluto/rea/commit/ceeff5a41788e47e5aee96a4baf191a6ce7fde4b), [3ad1032](https://github.com/morluto/rea/commit/3ad1032b11edea2dcb054e78f951cc43159ae83d), [964decd](https://github.com/morluto/rea/commit/964decd989acfa84c4bfc3295362c4c46877fc20), [3fa972a](https://github.com/morluto/rea/commit/3fa972af7c7cb6cf014ffc426bdf7130dc6f2fd2), [#4557](https://github.com/morluto/rea/pull/4557), [morluto/rea#4556](https://github.com/morluto/rea/issues/4556), [#4590](https://github.com/morluto/rea/pull/4590), [morluto/rea#4241](https://github.com/morluto/rea/issues/4241)). Thanks @DanielCvz1, @J0UH, @Rinkia, @RishTheFish21, @vadlachary2002.
|
||||
- **Client configuration:** Preserve the selected launcher, destination file, caller timeout choices and unresolved configuration evidence. Setup/uninstall recovery stays bound to the exact published file, keeps current verified backups, and transfers failed cleanup ownership for retry. ([d0253c9](https://github.com/morluto/rea/commit/d0253c99d65afbbd12732a2fdaf711640574e17f), [7e4eed9](https://github.com/morluto/rea/commit/7e4eed9dd48485b6a7100300aa98401811c2deb9), [2fb5e20](https://github.com/morluto/rea/commit/2fb5e204a35723f6f843d56ca47b0db94d8fcc44), [8235210](https://github.com/morluto/rea/commit/823521085a28f825e7fee1929d852d420042d1f7), [d818c3d](https://github.com/morluto/rea/commit/d818c3d2639d3115edb8a5269a754101aea6c3bd), [19329e4](https://github.com/morluto/rea/commit/19329e43bb38ca9d64512d801b25abcb188d2512), [dadf909](https://github.com/morluto/rea/commit/dadf909dce2e8b3d055541de69b4118d21276316), [333e0ab](https://github.com/morluto/rea/commit/333e0ab0dfc126a9915907d5079ed6195190e38e), [c15c206](https://github.com/morluto/rea/commit/c15c2062ad815cc9a3222ccfaeb7ef53cacd8b72)). Thanks @Adolanium.
|
||||
- **Configuration formats:** Read signed 64-bit TOML integers without rounding, handle large TOML documents without argument-spread failures, preserve JSONC trivia and Hermes YAML scalar text, and reject NUL bytes in registration environment values. ([#4937](https://github.com/morluto/rea/pull/4937), [morluto/rea#4917](https://github.com/morluto/rea/issues/4917), [c43bc0b](https://github.com/morluto/rea/commit/c43bc0bf66a4e245a0cafa40def0d0207bcbd73c), [4fd28fc](https://github.com/morluto/rea/commit/4fd28fc9e0c4f8358147af38c72ab9e814f52d61), [#3568](https://github.com/morluto/rea/pull/3568), [#4335](https://github.com/morluto/rea/pull/4335)). Thanks @alfredo1491, @gensokyo-dev, @kaoru0822-kitauji, @N0zoM1z0, @ShuaixinHuang, @xu91102.
|
||||
- **Claude Code configuration:** Setup, doctor and uninstall now accept only the strict JSON format Claude Code reads, including a leading UTF-8 BOM. Comments and trailing commas are refused before backup or writing; clients that support JSONC retain their existing behavior. ([#4944](https://github.com/morluto/rea/pull/4944), [morluto/rea#4940](https://github.com/morluto/rea/issues/4940)). Thanks @kaoru0822-kitauji, @xu91102.
|
||||
- **CLI output and dispatch:** Accept `--flag=value` and spaced global output values, keep array positions in filtered JSON, validate integer tokens before coercion, and exit quietly when the output reader closes. Cancellation remains effective through command finalization; app bundles and static native inspections select the intended workflow. ([#3811](https://github.com/morluto/rea/pull/3811), [#4823](https://github.com/morluto/rea/pull/4823), [2ae4e16](https://github.com/morluto/rea/commit/2ae4e1608f290c95ff7cff28c4d16a79b301d03d), [8d82579](https://github.com/morluto/rea/commit/8d8257928bbe8d43fedffc4db5fca26c8108820f), [#3781](https://github.com/morluto/rea/pull/3781), [279c590](https://github.com/morluto/rea/commit/279c590ef6bb737d4a48ea696a0ae5eb30010b3d), [#3511](https://github.com/morluto/rea/pull/3511), [morluto/rea#3321](https://github.com/morluto/rea/issues/3321), [f31b231](https://github.com/morluto/rea/commit/f31b231a92a61be6bd1800b16be0b3210ae82944)). Thanks @arjunprabhulal, @coygeek, @kaoru0822-kitauji, @ShuaixinHuang, @Sriram-PR, @Tapraj-Pathak.
|
||||
- **Updates:** Route `--update` through the verified updater and bound retained package-manager diagnostics. ([4f7b5fb](https://github.com/morluto/rea/commit/4f7b5fb6023ce2ec449561782d09e3c7e5ac7c7b), [01cdbbd](https://github.com/morluto/rea/commit/01cdbbdc556262c47d04a5b3391dd9c17946d3fc)).
|
||||
- **MCP contracts:** Advertise self-contained tool inputs while preserving tuple, array, Boolean and nested-JSON constraints across schema profiles. Schema-reference handling no longer rewrites literal metadata; catalog identity includes the output-schema profile. ([5e30969](https://github.com/morluto/rea/commit/5e30969cc5f1255d44ec7a1f1ca8dbac0af5e7c4), [10aa74d](https://github.com/morluto/rea/commit/10aa74dfedc3ae776f50d1fa1eb83b64ae82d3d6), [204b9f3](https://github.com/morluto/rea/commit/204b9f3ac140a993c3e458bb0e117de31818ab29), [b3389a5](https://github.com/morluto/rea/commit/b3389a5b0834265fe7a40391be315d810a53a97e), [a2e3016](https://github.com/morluto/rea/commit/a2e30168a046f3b79319eb7fbaf807b802104199), [ba98130](https://github.com/morluto/rea/commit/ba98130049fa40ade66b03611ff70c7ddc011a1d), [c5fb86d](https://github.com/morluto/rea/commit/c5fb86d1948af0a287a581b29b2b1d554743c607)). Thanks @akushonkamen, @HarsH-1877, @kaoru0822-kitauji, @xingzihai.
|
||||
- **Session ordering:** Keep snapshot admission ahead of later calls on the same connection, retain shutdown signals until cleanup succeeds, and preserve inspection authority through investigation publication. ([#3661](https://github.com/morluto/rea/pull/3661), [morluto/rea#3334](https://github.com/morluto/rea/issues/3334), [8873b39](https://github.com/morluto/rea/commit/8873b39770e89eae22b7693bc492d7e423b22bea), [663d07f](https://github.com/morluto/rea/commit/663d07f02cd62c99d81c59391d2c6d745c875335)). Thanks @coygeek, @rudycelekli, @Swatantra-66.
|
||||
- **JavaScript exports and resolution:** Retain CommonJS exports created with `Object.defineProperty` and track export receiver lifetimes. Resolve scoped packages, package `main`, JSON requires and `.mts`/`.cts` imports using the owning loader; overwritten `require` bindings and erased type-only imports no longer create false runtime links. ([#4725](https://github.com/morluto/rea/pull/4725), [morluto/rea#4724](https://github.com/morluto/rea/issues/4724), [#3567](https://github.com/morluto/rea/pull/3567), [morluto/rea#3541](https://github.com/morluto/rea/issues/3541), [#4838](https://github.com/morluto/rea/pull/4838), [morluto/rea#4632](https://github.com/morluto/rea/issues/4632), [#3844](https://github.com/morluto/rea/pull/3844), [morluto/rea#3383](https://github.com/morluto/rea/issues/3383), [#4164](https://github.com/morluto/rea/pull/4164), [morluto/rea#3915](https://github.com/morluto/rea/issues/3915), [#4223](https://github.com/morluto/rea/pull/4223), [#3801](https://github.com/morluto/rea/pull/3801), [morluto/rea#3393](https://github.com/morluto/rea/issues/3393), [#4165](https://github.com/morluto/rea/pull/4165), [morluto/rea#3932](https://github.com/morluto/rea/issues/3932)). Thanks @Abishek0070, @coygeek, @JNTMTMTM, @kaoru0822-kitauji, @prasad2512, @rodricksz4h5, @xu91102.
|
||||
- **JavaScript value flow:** Preserve values captured before later assignments, guarded factory fallthrough and nested destructuring. Track exact property keys and setter writes; direct `eval`, compound assignments and incoming callbacks preserve unresolved effects instead of inventing certainty. ([#4196](https://github.com/morluto/rea/pull/4196), [morluto/rea#3881](https://github.com/morluto/rea/issues/3881), [#4341](https://github.com/morluto/rea/pull/4341), [morluto/rea#3963](https://github.com/morluto/rea/issues/3963), [bfb7ed0](https://github.com/morluto/rea/commit/bfb7ed0bd43edc605e00fc52c6e21e735b285904), [#3480](https://github.com/morluto/rea/pull/3480), [#3831](https://github.com/morluto/rea/pull/3831), [morluto/rea#3312](https://github.com/morluto/rea/issues/3312), [9ac76d3](https://github.com/morluto/rea/commit/9ac76d3d9486856d25395b413d7ac5ffa744cad0), [#4342](https://github.com/morluto/rea/pull/4342), [morluto/rea#4206](https://github.com/morluto/rea/issues/4206), [#4243](https://github.com/morluto/rea/pull/4243), [morluto/rea#3880](https://github.com/morluto/rea/issues/3880)). Thanks @coygeek, @kaoru0822-kitauji, @maxazure, @N0zoM1z0, @ShuaixinHuang.
|
||||
- **JavaScript relationships:** Distinguish `Map` reads from network endpoints, associate response endpoints only with writes that reach the read, and preserve child-process receiver ownership and effective stdio choices. Version comparisons keep relocation and unknown relationships distinct from application changes. ([fbfdb6a](https://github.com/morluto/rea/commit/fbfdb6abe9d663be0be146a15a489e33ccba0f68), [#4759](https://github.com/morluto/rea/pull/4759), [morluto/rea#4606](https://github.com/morluto/rea/issues/4606), [#3819](https://github.com/morluto/rea/pull/3819), [morluto/rea#3362](https://github.com/morluto/rea/issues/3362), [#4336](https://github.com/morluto/rea/pull/4336), [morluto/rea#3355](https://github.com/morluto/rea/issues/3355), [556504e](https://github.com/morluto/rea/commit/556504edcbb0f8cd9eaf81a91b18ff891b8f5457), [4d980d0](https://github.com/morluto/rea/commit/4d980d017622d3175d86aaf3b0dcfbdf4ca3d091)). Thanks @Abdooo2235, @coygeek, @kaoru0822-kitauji, @Xinglan233.
|
||||
- **JavaScript source admission:** Parse JavaScript with its own grammar, refuse FIFO recovery inputs without blocking, and report oversized or resource-skipped files with their actual capacity and coverage state. Malformed runtime events stay excluded rather than becoming policy-filtered facts. ([#3782](https://github.com/morluto/rea/pull/3782), [morluto/rea#3780](https://github.com/morluto/rea/issues/3780), [#3788](https://github.com/morluto/rea/pull/3788), [morluto/rea#3332](https://github.com/morluto/rea/issues/3332), [#4757](https://github.com/morluto/rea/pull/4757), [morluto/rea#4629](https://github.com/morluto/rea/issues/4629), [#3852](https://github.com/morluto/rea/pull/3852), [morluto/rea#3670](https://github.com/morluto/rea/issues/3670), [#4755](https://github.com/morluto/rea/pull/4755), [morluto/rea#4685](https://github.com/morluto/rea/issues/4685)). Thanks @andresaragon, @coygeek, @kaoru0822-kitauji, @sx4im.
|
||||
- **Browser capture:** Bound CDP handshakes and command responses, preserve malformed inventory uncertainty and partial cleanup diagnostics, and report cancellation after content reads and final cleanup. Accessibility comparisons require observed coverage; attached emulation teardown limits remain explicit. ([#3555](https://github.com/morluto/rea/pull/3555), [morluto/rea#3553](https://github.com/morluto/rea/issues/3553), [#3901](https://github.com/morluto/rea/pull/3901), [morluto/rea#3379](https://github.com/morluto/rea/issues/3379), [4cc044b](https://github.com/morluto/rea/commit/4cc044be68f813e96907e6cedcf422d0a3170de3), [#4406](https://github.com/morluto/rea/pull/4406), [morluto/rea#4403](https://github.com/morluto/rea/issues/4403), [#2893](https://github.com/morluto/rea/pull/2893), [0123d59](https://github.com/morluto/rea/commit/0123d5976828bfe621bd51daa4cf39fee48a1349), [2ff0867](https://github.com/morluto/rea/commit/2ff086796ad7b8f60397956750584e170dbe2910)). Thanks @coygeek, @dakjdakd, @kaoru0822-kitauji, @maxazure, @N0zoM1z0, @YunzhouLi-hub.
|
||||
- **Browser page facts:** Preserve URL path bytes when removing credentials, repeated document `Link` headers and complete COOP/COEP structured items. WebMCP discovery reads the actual schema properties and exact receiver path. ([#4731](https://github.com/morluto/rea/pull/4731), [df2fc25](https://github.com/morluto/rea/commit/df2fc251e318a1255309dba232f9f73c700777b6), [c360cfc](https://github.com/morluto/rea/commit/c360cfcd6d7901514b16dd6113eba2e0b0fdf422), [ebd5a82](https://github.com/morluto/rea/commit/ebd5a82c439f6deb3a06062225ce009eb03257cf), [38f92d1](https://github.com/morluto/rea/commit/38f92d108e3148cdf47e4ffbfe3de8a76e795d51)). Thanks @Abdooo2235, @geekwolf, @ShuaixinHuang.
|
||||
- **Electron runtime capture:** Keep native IPC listener and one-shot dispatch behavior, block synchronous shell effects during capture, refuse active scenarios when the inspect fuse is disabled, and preserve fuse/transport cleanup ownership. ([#4389](https://github.com/morluto/rea/pull/4389), [morluto/rea#4390](https://github.com/morluto/rea/issues/4390), [#4605](https://github.com/morluto/rea/pull/4605), [morluto/rea#4593](https://github.com/morluto/rea/issues/4593), [#4562](https://github.com/morluto/rea/pull/4562), [morluto/rea#4393](https://github.com/morluto/rea/issues/4393), [#3605](https://github.com/morluto/rea/pull/3605), [morluto/rea#3453](https://github.com/morluto/rea/issues/3453), [f9de413](https://github.com/morluto/rea/commit/f9de41371f7536e2e22ce6e43c9314fa2f238a30), [e7df00e](https://github.com/morluto/rea/commit/e7df00e80777757b329bbe8aba6d421d8892b9d5)). Thanks @Adolanium, @gensokyo-dev, @gkobayashi0, @HarsH-1877, @kaoru0822-kitauji, @keeponlight, @N0zoM1z0, @Nitin13-git, @xu91102.
|
||||
- **APK resource accuracy:** Retain completed tables and decode facts after interrupted or failed reads. Derive locale limits from resource configuration fields, disclose unselectable qualified tables, distinguish document failures from omitted entries, and bind Evidence to the decoded snapshot. ([34a88d6](https://github.com/morluto/rea/commit/34a88d6c1674710953e2c94bd0a17332cd69ca6f), [f5fcf52](https://github.com/morluto/rea/commit/f5fcf5216ffb3a266b8c00f6d9a0943051084ab5), [6385ef4](https://github.com/morluto/rea/commit/6385ef4355d7a794b3154ee38529cbddee583336), [98886ad](https://github.com/morluto/rea/commit/98886adddfcbd7b4f4236b3a9825f91878c3cb11), [dc2e811](https://github.com/morluto/rea/commit/dc2e8117b60c4481372ed89b077e6c1cf81ee653), [#4771](https://github.com/morluto/rea/pull/4771), [morluto/rea#4768](https://github.com/morluto/rea/issues/4768)). Thanks @abhay-2108, @gensokyo-dev, @Iyanuoluwa007, @kaoru0822-kitauji, @N0zoM1z0.
|
||||
- **Apktool guidance:** Retain the observed version and disclose the matching platform/release advisory windows without treating a version string as proof of decoder safety. Missing prerequisites and decode refusals identify the selected APK or launcher. ([d38f099](https://github.com/morluto/rea/commit/d38f099692394ec90c582f9f0a14b036b421b160), [63696d2](https://github.com/morluto/rea/commit/63696d277328f556c971d1fc9f81ae83128d468b), [cc038da](https://github.com/morluto/rea/commit/cc038da218cada68c87700477aba3db4739791ff)). Thanks @GovinduSasanka, @peter-research.
|
||||
- **Android method bodies:** Recover original bytecode bodies when Java source is omitted while preserving genuine native/abstract body absence and primary failures during retryable cleanup. ([63c65a2](https://github.com/morluto/rea/commit/63c65a22430a4bbc0c1807a0298edb867775001f), [742b6f5](https://github.com/morluto/rea/commit/742b6f587f3c4ef1b8212e7376d6f3a305b194dc), [9fa6d2b](https://github.com/morluto/rea/commit/9fa6d2b254cc390665974efb52463386a029e9fc)). Thanks @balkan16, @Chessing234.
|
||||
- **ADB observations and transfers:** Preserve display-scoped focus, distinguish missing declarations from absent focus, retain numeric physical density and mark unknown permission coverage partial. Transfers use consumed snapshot identity and refuse unknown destination existence; uninstall requests explicitly select user zero. ([7086768](https://github.com/morluto/rea/commit/7086768a27242a15daa93486aefbaacc3c891e78), [3d028db](https://github.com/morluto/rea/commit/3d028dbc74a465f6f8df1813a931fbb5362e7dc3), [#4074](https://github.com/morluto/rea/pull/4074), [morluto/rea#3942](https://github.com/morluto/rea/issues/3942), [#4663](https://github.com/morluto/rea/pull/4663), [#4665](https://github.com/morluto/rea/pull/4665), [1342127](https://github.com/morluto/rea/commit/1342127e91d4db8b4113a7a4e85be909bc96e1db), [61e6880](https://github.com/morluto/rea/commit/61e688070f48016b6d178cd04c02fcc2f33e19ad), [be0b2d1](https://github.com/morluto/rea/commit/be0b2d138c46bb951fc3c8ee57ef7b0d57fed312)). Thanks @Copilot, @coygeek, @gongfpp, @Tapraj-Pathak, @yankikucuk.
|
||||
- **Apple metadata:** Preserve exact plist/keyed-archive integers, negative zero, non-finite real identity and dictionary keys. Reject malformed XML before value decoding, retain undecodable dates as unknown and report unsupported OpenStep entries explicitly. ([#4324](https://github.com/morluto/rea/pull/4324), [morluto/rea#4292](https://github.com/morluto/rea/issues/4292), [2cf9c28](https://github.com/morluto/rea/commit/2cf9c28f27261b7e64af2ba7ecb4e23ea7714aee), [#4839](https://github.com/morluto/rea/pull/4839), [morluto/rea#4834](https://github.com/morluto/rea/issues/4834), [f2b7bbb](https://github.com/morluto/rea/commit/f2b7bbb31279a820f4cf32d439570f8e5381d010), [6c98708](https://github.com/morluto/rea/commit/6c98708793c23615de8053b2cd14d9f164b960c5), [78cf610](https://github.com/morluto/rea/commit/78cf6100f33ad2412c0a3d60b83f8a44a43fbbb5), [c7e9e09](https://github.com/morluto/rea/commit/c7e9e096d3e2444e4c251123b8e914e9c0941f4e), [#3793](https://github.com/morluto/rea/pull/3793), [#4105](https://github.com/morluto/rea/pull/4105), [morluto/rea#3892](https://github.com/morluto/rea/issues/3892)). Thanks @coygeek, @kaoru0822-kitauji, @mertkayacs, @pablodiazjorge, @pollychen-lab, @xu91102.
|
||||
- **Apple application structure:** Derive bundle components and launchd domains from their owning directories, retain unavailable components, and select only exact `Assets.car` names as asset catalogs. Interface Builder keeps unresolved classes and counts hierarchy records beyond the returned object page. ([149a0fc](https://github.com/morluto/rea/commit/149a0fc06c3c67ee45624e6da14dfb0754473758), [5d6029e](https://github.com/morluto/rea/commit/5d6029ea62a65ec34fd739ee40223c77c6fdb63b), [c192c58](https://github.com/morluto/rea/commit/c192c58815ddeaf5cf30159e7082c742c0bc36b7), [#4094](https://github.com/morluto/rea/pull/4094), [morluto/rea#3914](https://github.com/morluto/rea/issues/3914), [#4118](https://github.com/morluto/rea/pull/4118), [morluto/rea#3958](https://github.com/morluto/rea/issues/3958), [#4162](https://github.com/morluto/rea/pull/4162), [morluto/rea#3950](https://github.com/morluto/rea/issues/3950)). Thanks @coygeek, @kaoru0822-kitauji.
|
||||
- **Native names and layout:** Keep valid Objective-C records beside malformed neighbors, preserve leading U+FEFF and repeated load-command fields, and select exact arm64e.x1 slices. PE resource ownership and NativeAOT reads stay within declared image/header ranges. ([#3833](https://github.com/morluto/rea/pull/3833), [morluto/rea#3336](https://github.com/morluto/rea/issues/3336), [#4549](https://github.com/morluto/rea/pull/4549), [morluto/rea#4540](https://github.com/morluto/rea/issues/4540), [morluto/rea#4541](https://github.com/morluto/rea/issues/4541), [#4730](https://github.com/morluto/rea/pull/4730), [#4048](https://github.com/morluto/rea/pull/4048), [morluto/rea#4008](https://github.com/morluto/rea/issues/4008), [#3500](https://github.com/morluto/rea/pull/3500), [morluto/rea#3389](https://github.com/morluto/rea/issues/3389), [#4463](https://github.com/morluto/rea/pull/4463), [morluto/rea#4373](https://github.com/morluto/rea/issues/4373), [morluto/rea#4455](https://github.com/morluto/rea/issues/4455), [3fc0075](https://github.com/morluto/rea/commit/3fc007534bb2104d662b43cc712281f9d1d6a7d3), [4a076f3](https://github.com/morluto/rea/commit/4a076f368c62bdfb19415accad342e03a38d1005)). Thanks @coygeek, @Iyanuoluwa007, @jayinvers, @kaoru0822-kitauji, @kevinjosh10, @maxazure, @ShuaixinHuang.
|
||||
- **Native analysis providers:** Keep Ghidra byte matches within examined coverage, retain PE PDB/function-boundary limitations and preserve configured MIPS choices. Attached IDA refuses universal Mach-O rather than silently choosing a slice; request timeout identity and captured output survive adapter handling. ([#4531](https://github.com/morluto/rea/pull/4531), [#2933](https://github.com/morluto/rea/pull/2933), [morluto/rea#2259](https://github.com/morluto/rea/issues/2259), [01a1b3f](https://github.com/morluto/rea/commit/01a1b3fda6a7dacd24a0b10619d41d12153080d4), [50dd9fc](https://github.com/morluto/rea/commit/50dd9fcad1efb47cb404192c30bdf190dcdaca58), [#4765](https://github.com/morluto/rea/pull/4765), [morluto/rea#3871](https://github.com/morluto/rea/issues/3871), [c0cfccb](https://github.com/morluto/rea/commit/c0cfccb45fe43e7eeff18e56c64caa5b121988f9)). Thanks @BeLit-Development-2, @coygeek, @kaoru0822-kitauji, @N0zoM1z0, @ShuaixinHuang.
|
||||
- **Recorded crashes:** Prevent GDB from discovering an executable named by an untrusted core, reject core mapping paths that escape the isolated sysroot, and preserve pwntools worker ownership across launcher exec. Completed decode facts and cancellation reasons remain available through failed final cleanup. ([5bf4fff](https://github.com/morluto/rea/commit/5bf4fffc1b835130ddb16f96db9ce13b639591db), [b33c0df](https://github.com/morluto/rea/commit/b33c0df02e93fabca00a44f2de625b1a63637ee9), [#4435](https://github.com/morluto/rea/pull/4435), [morluto/rea#4011](https://github.com/morluto/rea/issues/4011)). Thanks @coygeek, @N0zoM1z0.
|
||||
- **Cutter and Rizin:** Read Cutter’s active document from core configuration and attribute its host version correctly. Keep interrupted Rizin frames and output whitespace before completion markers, with exact command and cleanup failure attribution. ([#4507](https://github.com/morluto/rea/pull/4507), [morluto/rea#4497](https://github.com/morluto/rea/issues/4497), [#4514](https://github.com/morluto/rea/pull/4514), [morluto/rea#4498](https://github.com/morluto/rea/issues/4498), [#4459](https://github.com/morluto/rea/pull/4459), [morluto/rea#4015](https://github.com/morluto/rea/issues/4015), [d5fc7e6](https://github.com/morluto/rea/commit/d5fc7e6ebf9af050081fc4c6e18b9079cfaa5cdc), [199396f](https://github.com/morluto/rea/commit/199396f61c8f61da326d059606a992304a760147)). Thanks @coygeek, @N0zoM1z0, @Xander-AJ.
|
||||
- **Windows cleanup:** Retain unpublished runtimes after failed removal, handle read-only attributes and sharing conflicts, preserve primary snapshot failures and honor caller environment overrides. ([f78d137](https://github.com/morluto/rea/commit/f78d137b42741a0ed5fb8ac161286f9a50220c4d), [d69acc3](https://github.com/morluto/rea/commit/d69acc3455c33f90a2fc4381db43d38b6af50622), [0679a79](https://github.com/morluto/rea/commit/0679a7972b4f2b02eee6a86a44f50f4c2967d3cb), [5f9ac2c](https://github.com/morluto/rea/commit/5f9ac2c98a6fe11988c41afb737df9b6d5c3a1b3), [1bbf4bd](https://github.com/morluto/rea/commit/1bbf4bd16ca134cdb2187eaf29291cae90d14a57)). Thanks @DeryFerd.
|
||||
- **Managed metadata:** Compare nominal signatures and declaring identities across builds, preserve empty names without display substitutions, verify ABI-prefixed Mach-O exports exactly, and charge cached metadata once. Metadata errors report artifact file offsets; mixed-mode assemblies retain their actual boundary classification. ([#3059](https://github.com/morluto/rea/pull/3059), [#3079](https://github.com/morluto/rea/pull/3079), [#3853](https://github.com/morluto/rea/pull/3853), [morluto/rea#3352](https://github.com/morluto/rea/issues/3352), [#3974](https://github.com/morluto/rea/pull/3974), [morluto/rea#3961](https://github.com/morluto/rea/issues/3961), [#3491](https://github.com/morluto/rea/pull/3491), [morluto/rea#3449](https://github.com/morluto/rea/issues/3449), [#4418](https://github.com/morluto/rea/pull/4418), [morluto/rea#4372](https://github.com/morluto/rea/issues/4372), [#3516](https://github.com/morluto/rea/pull/3516), [morluto/rea#3366](https://github.com/morluto/rea/issues/3366)). Thanks @8rulerstar, @coygeek, @duckhunt113, @Hahaa909, @JayOfTheKeyboard, @kaoru0822-kitauji, @keeponlight, @N0zoM1z0, @tomyrms.
|
||||
- **Archive and filesystem identity:** Reject colliding extraction paths before writing, classify ZIP offsets beyond EOF as malformed, normalize ZIP current-directory prefixes once, and preserve symlink and unreadable-path provenance. A directory named `*.map` is no longer mistaken for a source-map file. ([862a0b2](https://github.com/morluto/rea/commit/862a0b2631785492f8dc20c8acd396646c545fd1), [c209dcb](https://github.com/morluto/rea/commit/c209dcb209118069f123e789949eeba1ef78853e), [6b1c36e](https://github.com/morluto/rea/commit/6b1c36e336eeacbb7159b211e687b4b84f99b9f5), [25c2222](https://github.com/morluto/rea/commit/25c22222abe7a681491bdb4d7316250fdcf32973), [d6e4fe4](https://github.com/morluto/rea/commit/d6e4fe4a4dd8914ec3db50d2a7fc3d0802649262), [#4131](https://github.com/morluto/rea/pull/4131), [morluto/rea#3909](https://github.com/morluto/rea/issues/3909)). Thanks @Adkid-Zephyr, @coygeek, @jackie-cqz, @kaoru0822-kitauji, @maxazure, @XY-code1.
|
||||
- **Evidence and reconstruction:** Authenticate all retained inventory substeps, preserve source-attributed caveats in composed workflows, and require comparable cited Evidence before declaring reconstruction coverage or readiness. Missing Evidence and residual replay uncertainty stay explicit. ([9ac91e0](https://github.com/morluto/rea/commit/9ac91e0c1e13e91cd21341635ad5be89b0e8642d), [c4608a2](https://github.com/morluto/rea/commit/c4608a29620ea3e7bb8259b47357b942fe4e4709), [#4075](https://github.com/morluto/rea/pull/4075), [morluto/rea#3357](https://github.com/morluto/rea/issues/3357), [#3526](https://github.com/morluto/rea/pull/3526), [morluto/rea#3363](https://github.com/morluto/rea/issues/3363), [153017d](https://github.com/morluto/rea/commit/153017d750322188c1a112b2284c1b8065c67320)). Thanks @coygeek, @gensokyo-dev, @kaoru0822-kitauji, @N0zoM1z0, @rudycelekli.
|
||||
- **Offline data:** Preserve exact UTC packet-capture instants, SQLite column identity and snapshot admission failures, and cooked WASM directive literals. Cleanup cancellation retains completed EVM/WASM observations rather than losing them. ([796faa0](https://github.com/morluto/rea/commit/796faa0e4e6bf745bcdc33acef65af8ccfc3622d), [748ac7f](https://github.com/morluto/rea/commit/748ac7f0df340f0d0569eca2e803a59cd51ffdbd), [bc51d0e](https://github.com/morluto/rea/commit/bc51d0ea5df7dcee1fcc42dfae3ab0c62c9cca5e), [4f87f02](https://github.com/morluto/rea/commit/4f87f02bb19bf77349a683a5bcb2da2d1da427d0), [d694aef](https://github.com/morluto/rea/commit/d694aef227e90164e6654ea9e5ce6a854d099404), [321cacb](https://github.com/morluto/rea/commit/321cacb84a020f447e1110df5c58fa71d99b7bf1)). Thanks @himamaheswarreddy, @mertkayacs, @rollroyces, @viniciusdsandrade.
|
||||
- **Kuna recovery:** Share concurrent single-function requests and cache repeats, validate warm exports once, stream string inventories beyond the former fixed buffer ceiling, and keep unproven processes in their run slots until close confirms release. Failure messages distinguish final buffer refusals from configurable deadlines. ([#4045](https://github.com/morluto/rea/pull/4045), [#4169](https://github.com/morluto/rea/pull/4169), [#4049](https://github.com/morluto/rea/pull/4049), [#4318](https://github.com/morluto/rea/pull/4318), [#4052](https://github.com/morluto/rea/pull/4052), [#4098](https://github.com/morluto/rea/pull/4098), [#4343](https://github.com/morluto/rea/pull/4343)). Thanks @kaoru0822-kitauji.
|
||||
- **Flutter snapshots:** Check symbol-table ranges against loadable segments and retain completed inspection facts when cancellation occurs during snapshot cleanup. ([#3859](https://github.com/morluto/rea/pull/3859), [morluto/rea#3346](https://github.com/morluto/rea/issues/3346), [#4766](https://github.com/morluto/rea/pull/4766), [morluto/rea#4717](https://github.com/morluto/rea/issues/4717), [#4762](https://github.com/morluto/rea/pull/4762), [morluto/rea#4640](https://github.com/morluto/rea/issues/4640)). Thanks @coygeek, @kaoru0822-kitauji.
|
||||
- **Native metadata recovery:** Keep unknown Objective-C category classes, empty protocol names, undecoded Swift aliases and unfinished property types explicit instead of discarding neighboring valid records. Preserve unnamed segment identity and exclude a dylib’s own install name from dependencies. ([#4093](https://github.com/morluto/rea/pull/4093), [morluto/rea#4012](https://github.com/morluto/rea/issues/4012), [#4078](https://github.com/morluto/rea/pull/4078), [morluto/rea#3873](https://github.com/morluto/rea/issues/3873), [#4085](https://github.com/morluto/rea/pull/4085), [morluto/rea#3934](https://github.com/morluto/rea/issues/3934), [#3825](https://github.com/morluto/rea/pull/3825), [morluto/rea#3345](https://github.com/morluto/rea/issues/3345), [#4051](https://github.com/morluto/rea/pull/4051), [morluto/rea#3875](https://github.com/morluto/rea/issues/3875), [#4050](https://github.com/morluto/rea/pull/4050), [morluto/rea#3872](https://github.com/morluto/rea/issues/3872)). Thanks @coygeek, @kaoru0822-kitauji.
|
||||
- **Document selection:** Preserve explicitly selected plist paths and bundle defaults, and only use document fallback after confirming its applicability. Finalization keeps completed selection facts and cancellation outcomes. ([e6c4877](https://github.com/morluto/rea/commit/e6c487799bb64f559e66b69be1863e6ec5187548), [#3774](https://github.com/morluto/rea/pull/3774), [9df559b](https://github.com/morluto/rea/commit/9df559b7e712298a761ff26e9caae8d659b7e54b)). Thanks @kaoru0822-kitauji.
|
||||
- **Capture and comparison accuracy:** Admit schema-valid HAR numeric values while retaining exact-publication limits, preserve empty URL path segments, and keep unrelated process traces and unnamed reference endpoints distinct or unknown. ([#3159](https://github.com/morluto/rea/pull/3159), [morluto/rea#3145](https://github.com/morluto/rea/issues/3145), [#3792](https://github.com/morluto/rea/pull/3792), [morluto/rea#3307](https://github.com/morluto/rea/issues/3307), [#4178](https://github.com/morluto/rea/pull/4178), [morluto/rea#3931](https://github.com/morluto/rea/issues/3931), [#3809](https://github.com/morluto/rea/pull/3809), [morluto/rea#3388](https://github.com/morluto/rea/issues/3388)). Thanks @coygeek, @gensokyo-dev, @kaoru0822-kitauji, @N0zoM1z0, @pablodiazjorge, @xu91102.
|
||||
- **Evidence publication:** Recheck staged output identity after its final write, classify missing review Evidence and duplicate ledger entries as caller input, and preserve the selected freshness requirement in comparison parameters. ([#4764](https://github.com/morluto/rea/pull/4764), [morluto/rea#4613](https://github.com/morluto/rea/issues/4613), [#4452](https://github.com/morluto/rea/pull/4452), [morluto/rea#4444](https://github.com/morluto/rea/issues/4444), [#4065](https://github.com/morluto/rea/pull/4065), [morluto/rea#4000](https://github.com/morluto/rea/issues/4000)). Thanks @amyanger, @coygeek, @kaoru0822-kitauji, @Rinkia, @Swatantra-66.
|
||||
|
||||
### ⚠ BREAKING CHANGES
|
||||
|
||||
- **Saved Evidence:** Evidence and analysis snapshots now require explicit nullable `relationship_coverage` in their identity commitments. Artifact inventories require scoped coverage and explicit `integrity_contradictions`; obsolete `coverage_status` and prose-based coverage inference are removed. Regenerate old Evidence/snapshots and inventory observations with this release: saved files are not migrated and omitted facts are not filled with guessed defaults. ([db8128d](https://github.com/morluto/rea/commit/db8128dc0d1c4f731f3d19877fa00d25cf17f657), [0405c9d](https://github.com/morluto/rea/commit/0405c9d361de6d904dfa16cc0b6d8350407f2c9b), [a9588be](https://github.com/morluto/rea/commit/a9588be4ebdd20eeda975cb63ffcdb1cd868b340)).
|
||||
- **Browser capture results:** Consumers must retain resolved capture limits, event families and omission facts. Set desired budgets through the supported capture options; omitted caller inputs still select defaults, but stored observations must report the resolved values. Admitted content reads queue behind bounded concurrency. ([57a23f0](https://github.com/morluto/rea/commit/57a23f0a2da0fdc3835e203718ded8acea5a34e6)).
|
||||
- **Android version codes:** `inspect_adb_package.version_code` is a decimal string rather than a JSON number. Read it as a string, or use `BigInt` for numeric comparison. ([#3485](https://github.com/morluto/rea/pull/3485), [morluto/rea#3368](https://github.com/morluto/rea/issues/3368), [7666a48](https://github.com/morluto/rea/commit/7666a482b69dc7cabdbac3bbaf89a43e32300192)). Thanks @coygeek, @kaoru0822-kitauji.
|
||||
- **Native UI failures:** Failed or cancelled native UI scenarios now return typed errors with the collected report in `partial_observation`. Handle the error path and read that report instead of expecting a successful result containing failed steps. ([d909919](https://github.com/morluto/rea/commit/d9099190ab3a819bd3a9fd62a8714cef398016d5)).
|
||||
- **Provider status:** `binary_session` candidate availability can be unknown during owner finalization. Preserve that state rather than treating it as unavailable; internal consumers must await the asynchronous session status API. ([150e996](https://github.com/morluto/rea/commit/150e996a86f2bec4982ba87f37bf502a8d5e8214)).
|
||||
- **Swift symbol locations:** `swift_symbols` entries may have a null `address` when the provider supplies no location. Handle unknown addresses; optional `evidence.reported_address` retains the original provider value. ([8116dc0](https://github.com/morluto/rea/commit/8116dc06512b4609d8db63cebc9f2ff8cb964151)).
|
||||
- **Linux Hopper admission:** Older builds previously admitted by a fixed allowlist need retained publisher-signed package evidence or an approved update to the current public demo. Update explicit `HOPPER_LAUNCHER_PATH` selections when migrating; the live feed offers only its current release. The official RPM metadata mismatch remains unresolved and is refused. ([#3590](https://github.com/morluto/rea/pull/3590), [morluto/rea#3577](https://github.com/morluto/rea/issues/3577), [70cd4cb](https://github.com/morluto/rea/commit/70cd4cbfb849b61f05d97d2690372a9782831426)). Thanks @gensokyo-dev, @kaoru0822-kitauji, @N0zoM1z0.
|
||||
|
||||
Full contribution record: [571 commits and 236 merged PRs](https://github.com/morluto/rea/blob/rea-agents-6.6.0/docs/releases/6.6.0.contributions.json).
|
||||
|
||||
## [6.5.0](https://github.com/morluto/rea/compare/rea-agents-6.4.0...rea-agents-6.5.0) (2026-10-13)
|
||||
|
||||
Provider prerequisites and recorded verification scope are listed in [the testing guide](docs/testing.md).
|
||||
|
||||
+10
-20
@@ -431,10 +431,6 @@ def _procedure_map(document):
|
||||
return result
|
||||
|
||||
|
||||
def _strings(document):
|
||||
return {address: record["value"] for address, record in _string_inventory(document).items()}
|
||||
|
||||
|
||||
def _string_record(segment, address, display):
|
||||
"""Read one native typed object, retaining its shortened display separately."""
|
||||
length = segment.getObjectLength(address)
|
||||
@@ -521,7 +517,7 @@ def _search_inventory(document, kind):
|
||||
if kind == "procedure":
|
||||
values = _procedure_map(document)
|
||||
elif kind == "string":
|
||||
values = _strings(document)
|
||||
values = _string_inventory(document)
|
||||
elif kind == "name":
|
||||
values = {_hex(address): value for address, value in _name_map(document).items()}
|
||||
else:
|
||||
@@ -548,23 +544,17 @@ def _search_results(document, kind, params):
|
||||
else:
|
||||
raise CapabilityUnavailableError("Regex matching requires REA's supervised worker; the native bridge only executes literal searches")
|
||||
|
||||
selected = []
|
||||
for item in _search_inventory(document, kind):
|
||||
if not matches(item[1]):
|
||||
continue
|
||||
selected.append(item)
|
||||
if not selected:
|
||||
return []
|
||||
if kind == "string":
|
||||
records = _string_inventory(document)
|
||||
return [{"address": address, **records[address]} for address, _ in selected]
|
||||
return [
|
||||
{
|
||||
"address": address,
|
||||
"value": value,
|
||||
}
|
||||
for address, value in selected
|
||||
return [
|
||||
{"address": address, **record}
|
||||
for address, record in _search_inventory(document, kind)
|
||||
if matches(record["value"])
|
||||
]
|
||||
return [
|
||||
{"address": address, "value": value}
|
||||
for address, value in _search_inventory(document, kind)
|
||||
if matches(value)
|
||||
]
|
||||
|
||||
|
||||
def _unavailable(reason):
|
||||
|
||||
@@ -363,6 +363,14 @@ export REA_ANDROID_TEST_APK="$PWD/_reference/apk-integration/ApiDemos-debug.apk"
|
||||
npm run verify:android
|
||||
```
|
||||
|
||||
The main lane and Windows Java-selection checks validate the selected JDK and
|
||||
engine archive through `validateVerifierJadxConfiguration`. This shared verifier
|
||||
boundary supplies the production validator's resource owner and waits for
|
||||
retained cleanup before proceeding. Failed closes retain their exact owners for
|
||||
retry; prerequisite failures must not be mistaken for completed bridge tests.
|
||||
Recovery checks match the typed Java or archive failure and its applicable
|
||||
remediation.
|
||||
|
||||
The lane also uses the selected full JDK's `jlink` to create a disposable JRE
|
||||
without the compiler module. It checks real Java startup failures, missing
|
||||
selectors, recovery after failures, and CLI cancellation with owned cleanup.
|
||||
|
||||
@@ -8,7 +8,7 @@ installs Apktool or a Java runtime.
|
||||
|
||||
## Supply tools
|
||||
|
||||
Use Apktool 2.7.0 or newer (verified with 2.7.0) and a Java runtime the
|
||||
Use Apktool 2.7.0 or newer (verified with 2.7.0 and 3.0.3 on Linux) and a Java runtime the
|
||||
launcher can find. Select the launcher through the environment:
|
||||
|
||||
```sh
|
||||
@@ -63,6 +63,7 @@ rea inspect-apktool-client
|
||||
rea decode-android-resources /targets/Example.apk
|
||||
rea decode-android-resources /targets/Example.apk --locale de
|
||||
rea decode-android-resources /targets/Example.apk --locales de --locales pt-BR
|
||||
rea decode-android-resources /targets/Example.apk --locales de --locales pt-BR --string-tables-only
|
||||
rea decode-android-resources /targets/Example.apk --no-strings
|
||||
```
|
||||
|
||||
@@ -74,6 +75,15 @@ from the same single decode pass), and `include_strings` (default true). Regiona
|
||||
select Android's `values-pt-rBR`; Android qualifiers such as `pt-rBR` and
|
||||
`b+zh+Hans+CN` may also be supplied directly.
|
||||
|
||||
Set `string_tables_only: true` (CLI `--string-tables-only`) to return string
|
||||
entries only inside `string_tables`, including when selecting a single table.
|
||||
The top-level `strings` array is then empty by request; it does not establish
|
||||
that the primary table is empty. Table status, counts and coverage still describe
|
||||
the examined source. This avoids repeating primary entries in multi-locale
|
||||
responses and charges the aggregate output budget for one copy per table.
|
||||
Omitting the option preserves existing responses. `include_strings: false`
|
||||
still suppresses all string projection, including in this mode.
|
||||
|
||||
## Interpret the result
|
||||
|
||||
- **An owned APK snapshot is digested and decoded.** `target.sha256` and
|
||||
@@ -108,7 +118,8 @@ select Android's `values-pt-rBR`; Android qualifiers such as `pt-rBR` and
|
||||
each projected table with its locale, status (`present`, `missing`,
|
||||
`oversized`, `malformed`, or `skipped`), entry counts, and coverage,
|
||||
bounded at an aggregate budget of 10,000 strings and 32 MiB XML. When `locales`
|
||||
is omitted, legacy single-table payloads are preserved without adding `string_tables`.
|
||||
is omitted and `string_tables_only` is not true, legacy single-table payloads are preserved
|
||||
without adding `string_tables`.
|
||||
- **Smali is skipped by design** (`apktool d --no-src`): Java-source
|
||||
recovery belongs to the JADX family. Resource smali edits and rebuild are
|
||||
a follow-up lane with mutation authority.
|
||||
@@ -143,3 +154,11 @@ locales — launcher version, on-disk digest agreement, apktool.yml metadata
|
||||
(1.2.3, SDK 24–34), manifest package agreement, default and `de` string
|
||||
tables projected. Parser goldens in the test suite come from the same
|
||||
decode. See `docs/testing.md` for the lane summary.
|
||||
|
||||
Multi-locale and table-only projection were also verified through real CLI and
|
||||
stdio MCP workflows on Linux with Apktool 3.0.3/JDK 21 against NewPipe 0.29.1,
|
||||
ApiDemos 6.0.18 and Appium Settings 8.2.1. Independent XML projection checks,
|
||||
exact facts/Evidence parity and bounded task costs are recorded in the
|
||||
[shared-decode and table-only checkpoint](../experiments/analysis-efficiency/apktool-string-tables.md).
|
||||
This qualifies selected resource facts, not application execution or independent
|
||||
binary resource-decoder completeness.
|
||||
+17
-3
@@ -276,14 +276,14 @@ Register with the variable unset:
|
||||
<!-- x-release-please-start-version -->
|
||||
|
||||
```bash
|
||||
claude mcp add --scope local rea -- npx -y rea-agents@6.5.0 mcp
|
||||
claude mcp add --scope local rea -- npx -y rea-agents@6.7.0 mcp
|
||||
```
|
||||
|
||||
On native Windows, use the same project directory and wrap the package-runner
|
||||
shim so the stdio client can launch it:
|
||||
|
||||
```powershell
|
||||
claude mcp add --scope local rea -- cmd.exe /d /c npx -y rea-agents@6.5.0 mcp
|
||||
claude mcp add --scope local rea -- cmd.exe /d /c npx -y rea-agents@6.7.0 mcp
|
||||
```
|
||||
|
||||
<!-- x-release-please-end -->
|
||||
@@ -599,6 +599,20 @@ host mount and the rest of `/tmp` remain unchanged; this fallback never invokes
|
||||
`sudo`. `rea doctor --provider hopper --json` reports the selected
|
||||
strategy and both host and effective mount facts.
|
||||
|
||||
Hopper also embeds Python; the Python executable and standard library alone
|
||||
are insufficient on Ubuntu. Approved setup checks the system interpreter
|
||||
at `/usr/bin/python3` and, when necessary, installs its matching
|
||||
`libpython3.X` package, then verifies that its shared library can load.
|
||||
APT resolves the concrete package (for example, `libpython3.12t64` on Ubuntu
|
||||
24.04). Doctor includes these fresh observations in the `embedded_python`
|
||||
details of its `hopper-demo-runtime` check, alongside the private-display
|
||||
diagnostic. Failed inspections remain unknown rather than proving a library
|
||||
absent. Run `rea setup --yes --install-hopper` to repair a supported install.
|
||||
This prerequisite check does not launch the bridge or validate a custom Python
|
||||
interpreter configured inside Hopper; that interpreter needs its own matching
|
||||
shared runtime. Custom installation prefixes or shared-library ABIs require
|
||||
manual dependency repair.
|
||||
|
||||
When the package-manager step fails, setup reports the failing command and its
|
||||
captured output, so a package conflict is distinguishable from an authorization
|
||||
failure. On CachyOS, an unversioned compatibility package such as
|
||||
@@ -1187,7 +1201,7 @@ For a client that requires manual configuration, use:
|
||||
"mcpServers": {
|
||||
"rea": {
|
||||
"command": "npx",
|
||||
"args": ["-y", "rea-agents@6.5.0", "mcp"]
|
||||
"args": ["-y", "rea-agents@6.7.0", "mcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+14
-4
@@ -8,7 +8,16 @@ Start a JEB client that serves MCP and point REA at it:
|
||||
|
||||
- Default endpoint: `http://127.0.0.1:8425/mcp`. Override with `REA_JEB_MCP_URL` (absolute `http`/`https` URL with literal host `127.0.0.1` or `[::1]`; redirects, remote hosts, credentials, query strings, and fragments are rejected rather than silently dropped).
|
||||
- In the JEB GUI client, start the MCP server from the _File_ menu (or via the VIBRE assistant).
|
||||
- Headless, use JEB's documented programmatic route (`JebMcpServerInstance` from the JEB API) with the JEB client on your classpath; JEB 5.48's `-c` headless mode does not accept `--mcp` directly.
|
||||
- Headless, the official Linux CE build `5.48.0.202610071539` supports `-c --mcp`. From the JEB installation directory, select JDK 21 or newer through `JAVA_HOME`, choose your target and bind explicitly to loopback:
|
||||
|
||||
```sh
|
||||
"$JAVA_HOME/bin/java" -jar bin/app/jeb.jar -c --mcp --interface=127.0.0.1 \
|
||||
--port=8425 --infile=/absolute/path/to/target
|
||||
```
|
||||
|
||||
Complete the vendor's first-use EULA and license-key activation when prompted. The Linux audit used an isolated installation, a 1536 MiB Java heap and one active processor inside a separately bounded process family; JVM heap limits alone do not bound the whole family.
|
||||
|
||||
- JEB's documented programmatic route (`JebMcpServerInstance` from the JEB API) remains an alternative. The earlier licensed 5.48 headless verification used that route because its CLI did not accept `--mcp`. Check your installed build's `-c --help` and live tool catalog; the version number alone does not establish the launcher options or supported operations.
|
||||
|
||||
For setup registration, set `REA_JEB_MCP_URL` in the REA process environment
|
||||
before running `rea setup`. The approved setup plan preserves the exact selected
|
||||
@@ -31,7 +40,7 @@ or establish one end-to-end deadline across a whole CLI or MCP operation. A
|
||||
request that reaches its budget returns a typed timeout, while caller
|
||||
cancellation remains a cancellation.
|
||||
|
||||
Verified against JEB 5.48.0 (Linux x64, licensed build) serving `jeb-mcp-server` 1.3.0 over streamable HTTP.
|
||||
Earlier verification used JEB 5.48.0 (Linux x64, licensed build) serving `jeb-mcp-server` 1.3.0 over streamable HTTP. The official Linux x64 CE build `5.48.0.202610071539` was also verified through real CLI and production stdio MCP native-code workflows; see the [Linux CE audit](../experiments/analysis-efficiency/jeb-linux-native.md) for measurements and limits. CE verification does not establish licensed/Demo Android decompilation or GUI behavior.
|
||||
|
||||
## Operations
|
||||
|
||||
@@ -42,15 +51,16 @@ Verified against JEB 5.48.0 (Linux x64, licensed build) serving `jeb-mcp-server`
|
||||
| `list_jeb_units` / `list-jeb-units` | Page units by stable path and type (`filter`, `parent-unit-path`, `index`, `count`) |
|
||||
| `decompile_jeb_item` / `decompile-jeb-item` | Decompile one type or method to engine pseudo-code |
|
||||
|
||||
All four are read-only with respect to the target file; only `open_jeb_project` changes engine-side project state, and Evidence records the endpoint, the caller-selected parameters, and the engine-reported digests. Mutation workflows the engine also offers (renames, comments, `save_project`, `close_project`) are intentionally not exposed yet.
|
||||
All four preserve target-file bytes. `open_jeb_project` selects or creates an engine-side project; analysis queries can populate computed analysis within it. Evidence records the endpoint, caller-selected parameters, and available engine-reported digests. Mutation workflows the engine also offers (renames, comments, `save_project`, `close_project`) are intentionally not exposed yet.
|
||||
|
||||
**Engine coverage note (verified):** JEB 5.48.0 headless MCP instances do not advertise `open_project` — the target must be selected when the engine client starts (`--infile` on the programmatic launcher, or the GUI's file opening). REA preserves that refusal with the engine's reason instead of failing silently; `open_jeb_project` completes only against engine builds that advertise the tool, such as GUI clients.
|
||||
**Engine coverage note:** The earlier licensed 5.48.0 headless verification did not advertise `open_project`; it selected the target when the engine client started. For that build, use `--infile` on the programmatic launcher or open the file in the GUI. The audited CE headless catalog also omitted `open_project`; the verified CLI route above selects its input at startup. Use `binary_session` to inspect the current engine's advertised operations. REA preserves an unavailable tool's refusal; `open_jeb_project` requires an engine that advertises `open_project`. The CE audit does not establish GUI project-opening support.
|
||||
|
||||
## Semantics
|
||||
|
||||
- Paths in `open_jeb_project` are resolved by the JEB process, not by REA; REA does not require the artifact to exist locally.
|
||||
- The engine caps `list_units` pages at 100; a full page is reported as `coverage: "partial"` because more units may exist. An empty remainder is `complete` only when fewer units than requested were returned.
|
||||
- When `decompile_jeb_item` omits `unit_path`, the engine implicitly selects the project's first code unit; REA reports `unit_path: null` and a limitation instead of guessing the effective unit.
|
||||
- `item_address` is a JEB code-item reference, such as a value returned by the engine's `list_code_items`. Preserve that representation and select its code unit explicitly. For the audited CE SQLite unit, the exported method reference was `sqlite3_libversion_number`; using its raw ELF symbol address did not produce a successful decompilation. File offsets, ELF virtual addresses and engine item references have different semantics. REA currently exposes unit listing, while code-item discovery remains an upstream JEB operation.
|
||||
- Engine refusals (`success: false` envelopes) are preserved as unsupported-target errors carrying the engine's own message, for example a nonexistent item address.
|
||||
- Successfully parsed client information is reused for the lifetime of the MCP connection, including when the engine omits its version. Explicit client inspection and availability probes refresh it.
|
||||
- An unreachable endpoint is a provider-availability failure with recovery guidance; REA does not retry or spawn the engine.
|
||||
|
||||
@@ -337,6 +337,12 @@ complete observation.
|
||||
Object `kind` describes the decoded archive value: data, dates and numbers
|
||||
remain `scalar` when their JSON value uses a `$plist_type` wrapper. An actual
|
||||
dictionary with the same fields remains `dictionary`.
|
||||
XML must be well-formed. Data literals use
|
||||
[CoreFoundation's raw ASCII quartet rules](https://github.com/swiftlang/swift-corelibs-foundation/blob/384ff680be7c7445fee6ce0c55e9ae11a46e07c5/Sources/CoreFoundation/CFPropertyList.c#L1588):
|
||||
non-alphabet ASCII characters are ignored and an incomplete final quartet
|
||||
emits no bytes. Character references retain their source spelling for data
|
||||
decoding. Non-ASCII content, child markup and self-closing `<data/>` are format
|
||||
failures; explicit `<data></data>` represents empty data.
|
||||
UID 0 cannot distinguish conditional nil from ordinary nil. Limits are
|
||||
64 MiB of input, 200,000 objects/references and 128 nesting levels.
|
||||
XML dictionary entries decoded as `__proto__` are omitted with an explicit
|
||||
@@ -598,9 +604,11 @@ digests identify the original serialized bytes.
|
||||
|
||||
The archive reader observes XML number element types and binary number markers.
|
||||
XML values and their number facts are interpreted together from the validated
|
||||
DOM. Hexadecimal `<integer>` text, including an optional sign, retains its value;
|
||||
DOM and its original scalar source spans. Hexadecimal `<integer>` text, including
|
||||
an optional sign, retains its value;
|
||||
`<real>` text and CDATA segments are combined without inferring text from nested
|
||||
elements. The XML path does not serialize or reparse character references.
|
||||
elements. Real character references expand outside CDATA; integer and date
|
||||
literals use raw source text.
|
||||
Integers beyond the exact range of a JSON number are reported with the original
|
||||
decimal text as `{ "$plist_type": "integer", "decimal": "<exact digits>" }` when
|
||||
that source value can be associated unambiguously. Integral real values remain
|
||||
@@ -609,9 +617,9 @@ markers remain decoded with an explicit precision limitation. The limitation
|
||||
also counts observed unsafe integer literals, including observations outside the
|
||||
selected object page. Incomplete supplemental metadata produces an explicit
|
||||
precision note without rejecting an archive accepted by the byte decoder.
|
||||
Mixed XML number content (comments, CDATA, processing instructions or child
|
||||
elements) and integer text the supplemental reader cannot classify disable exact
|
||||
association and retain decoded numbers with that precision note. Nonnumeric
|
||||
Malformed XML scalar content becomes an explicit unknown rather than a prefix
|
||||
number, guessed timestamp, or tag-only boolean. These unknowns do not disable
|
||||
exact association for unrelated observed numbers. Nonnumeric
|
||||
`UID` and `CF$UID` dictionary values remain ordinary data and are projected
|
||||
recursively; numeric reference markers retain their original representation.
|
||||
|
||||
@@ -632,11 +640,15 @@ decode budget reserves estimated room for number metadata and attribute
|
||||
projection; these estimates are not a hard process-memory bound. Native
|
||||
NIBArchive decoding is unchanged by this plist-specific representation.
|
||||
|
||||
Dates are reported as `{ "$plist_type": "date", "iso8601": "<timestamp>" }`. A
|
||||
date the decoder cannot represent, such as an XML date with a year beyond 9999
|
||||
or a negative year, is reported as `{ "$plist_type": "date", "iso8601": null }`
|
||||
with a counted limitation; the rest of the plist or archive is still decoded.
|
||||
`inspect_plist` and Interface Builder decoding use the same projection.
|
||||
Dates are reported as `{ "$plist_type": "date", "iso8601": "<timestamp>" }`.
|
||||
XML dates require a complete UTC spelling with fixed two-digit month, day and
|
||||
time fields; short, negative and extended years are interpreted as UTC year
|
||||
fields within JavaScript's date range. Unsupported calendar fields (including
|
||||
leap seconds) and out-of-range dates remain explicit unknowns. REA does not
|
||||
emulate CoreFoundation's normalization of overflowing calendar fields. Invalid
|
||||
dates returned by other decoders retain `{ "$plist_type": "date", "iso8601": null }`
|
||||
with a counted limitation. `inspect_plist` and Interface Builder decoding use
|
||||
the same projection.
|
||||
|
||||
A negative-zero real is reported as `{ "$plist_type": "real", "value": "-0" }`
|
||||
because a JSON number cannot carry the sign of zero. `inspect_plist` and
|
||||
@@ -653,14 +665,20 @@ for those details. XML CFPropertyList literals (`nan`, `+infinity`, `-infinity`,
|
||||
`infinity`, `inf`, `+inf`, `-inf`, case-insensitively) decode to the same known
|
||||
values. Case and aliases normalize; original archive bytes retain source text.
|
||||
|
||||
XML numeric text whose identity the decoder cannot establish, including
|
||||
unrecognized non-finite text and finite literals that overflow the numeric
|
||||
representation, remains `{ "$plist_type": "real", "value": null }` with a
|
||||
counted limitation. The XML decoder marks these values explicitly before shared
|
||||
projection, so they cannot be confused with known binary IEEE values.
|
||||
Incomplete numeric markup retains its existing compatibility behavior described
|
||||
above. Known non-finite and unresolved numeric values have separate counts.
|
||||
Native `inspect_plist` uses these same labels for plutil's XML conversion.
|
||||
XML scalars whose identity cannot be established use
|
||||
`{ "$plist_type": "unknown", "element": "integer" | "real" | "date" | "true" | "false", "literal": "<original content>", "reason": "invalid-literal" | "unsupported-markup" | "unrepresentable" }`.
|
||||
A counted limitation accompanies these wrappers. Literal content retains its
|
||||
original entity spelling, markup and line endings. Whole XML documents must
|
||||
still pass strict well-formedness checks. Empty integer, real and date elements
|
||||
are unknown scalars; boolean elements must be empty. Complete decimal/exponent
|
||||
reals and signed decimal/hexadecimal integers accept their respective native
|
||||
leading whitespace but reject trailing text. Arbitrary-precision integers
|
||||
remain an intentional REA extension beyond CoreFoundation's integer range.
|
||||
Finite real spellings that overflow the numeric representation remain unknown,
|
||||
rather than becoming observed infinities. Known non-finite and unresolved scalar
|
||||
values have separate counts. This replaces the former prefix-number and
|
||||
incomplete-metadata fallback for malformed XML numeric content. Native
|
||||
`inspect_plist` uses the same projection for plutil's XML conversion.
|
||||
|
||||
## Locate a byte signature
|
||||
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,112 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"repository": "morluto/rea",
|
||||
"version": "6.7.0",
|
||||
"base": "2c4338db1fdb7773e05376301c558cff2a0b4b80",
|
||||
"target": "b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf",
|
||||
"commits": [
|
||||
{
|
||||
"sha": "8cbaf6c7021b97dd4de6b5771c35fbed06b240e9",
|
||||
"subject": "fix(tests): isolate command finalization module mocks",
|
||||
"body": "fix(tests): isolate command finalization module mocks\n\nMove the five registered WASM finalization cases into the existing per-file isolated project. Preserve all result, exit-code, observation and cleanup assertions. A preloaded shared CLI graph reproduces three failures before the move; the isolated covered packet passes.\n\nFixes #4959\n",
|
||||
"parents": ["2c4338db1fdb7773e05376301c558cff2a0b4b80"],
|
||||
"authors": [
|
||||
{
|
||||
"name": "morluto",
|
||||
"email": "76467478+morluto@users.noreply.github.com",
|
||||
"user": {
|
||||
"login": "morluto",
|
||||
"__typename": "User"
|
||||
}
|
||||
}
|
||||
],
|
||||
"pullRequests": []
|
||||
},
|
||||
{
|
||||
"sha": "6b180783ed04ebc87548c8385c80e05725804c47",
|
||||
"subject": "fix(tests): budget CLI coverage from parsed options",
|
||||
"body": "fix(tests): budget CLI coverage from parsed options\n\nUse the pinned Vitest parser for coverage and shard flags. Keep ordinary CLI workers at 384 MiB and covered workers at the tested 512 MiB with unchanged concurrency. Retain complete coverage and add actual-config regressions for false, provider-only and spaced-shard options.\n\nValidation: 27 focused cases; 16 covered cases with CLI preloaded; 5 docs checks; typecheck/lint; exact discovery of 1578 files. Measured covered control: 22.18s elapsed, 46.52s CPU, about 2.33 GiB sampled summed process-family RSS on macOS arm64. Full Linux CI pending. No runtime/device verification claimed.\n\nFixes #4960\n",
|
||||
"parents": ["8cbaf6c7021b97dd4de6b5771c35fbed06b240e9"],
|
||||
"authors": [
|
||||
{
|
||||
"name": "morluto",
|
||||
"email": "76467478+morluto@users.noreply.github.com",
|
||||
"user": {
|
||||
"login": "morluto",
|
||||
"__typename": "User"
|
||||
}
|
||||
}
|
||||
],
|
||||
"pullRequests": []
|
||||
},
|
||||
{
|
||||
"sha": "bb212ffecb47e129cc569b9867b2b6e7c9b65685",
|
||||
"subject": "fix(tests): inherit server identity deadline policy",
|
||||
"body": "fix(tests): inherit server identity deadline policy\n\nRemove per-case ten-second overrides from MCP catalog and identity checks. Keep the suite-owned bounded ordinary and coverage deadlines, all assertions, and session cleanup.\n\nFixes #4967\n",
|
||||
"parents": ["6b180783ed04ebc87548c8385c80e05725804c47"],
|
||||
"authors": [
|
||||
{
|
||||
"name": "morluto",
|
||||
"email": "76467478+morluto@users.noreply.github.com",
|
||||
"user": {
|
||||
"login": "morluto",
|
||||
"__typename": "User"
|
||||
}
|
||||
}
|
||||
],
|
||||
"pullRequests": []
|
||||
},
|
||||
{
|
||||
"sha": "b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf",
|
||||
"subject": "fix(setup): edit the duplicate JSON member that clients read (#4957)",
|
||||
"body": "fix(setup): edit the duplicate JSON member that clients read (#4957)\n\nJSON.parse, JSONC parse, and native Claude Code keep the last duplicate\nobject member, and parseClientConfiguration therefore validates the last\none. The shared setup/uninstall serializer instead edited text through\njsonc-parser's modify and findNodeAtLocation, which select the first\nmatching member, so setup and uninstall on configurations with repeated\nserver tables or rea members wrote into a shadowed member and failed\nreadback.\n\nLocate every JSON text edit by the effective (last) member in one place,\nreplacing the first-member modify path. Removing an effective member\nthat shadows an earlier duplicate would activate that duplicate, whatever\nits ownership, so the serializer now returns a tagged shadowed_member\nfailure and setup and uninstall refuse before writing a backup or the\nconfiguration.\n\nFixes #4952\n\nCo-authored-by: Claude Opus 5.5 <noreply@anthropic.com>",
|
||||
"parents": ["bb212ffecb47e129cc569b9867b2b6e7c9b65685"],
|
||||
"authors": [
|
||||
{
|
||||
"name": "Kaoru0822",
|
||||
"email": "kanadekitauji@proton.me",
|
||||
"user": {
|
||||
"login": "kaoru0822-kitauji",
|
||||
"__typename": "User"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Claude Opus 5.5",
|
||||
"email": "noreply@anthropic.com",
|
||||
"user": {
|
||||
"login": "claude",
|
||||
"__typename": "User"
|
||||
}
|
||||
}
|
||||
],
|
||||
"pullRequests": [4957]
|
||||
}
|
||||
],
|
||||
"pullRequests": [
|
||||
{
|
||||
"number": 4957,
|
||||
"title": "fix(setup): edit the duplicate JSON member that clients read",
|
||||
"url": "https://github.com/morluto/rea/pull/4957",
|
||||
"author": {
|
||||
"login": "kaoru0822-kitauji",
|
||||
"__typename": "User"
|
||||
},
|
||||
"mergeCommit": {
|
||||
"oid": "b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf"
|
||||
},
|
||||
"issues": [
|
||||
{
|
||||
"number": 4952,
|
||||
"url": "https://github.com/morluto/rea/issues/4952",
|
||||
"author": {
|
||||
"login": "xu91102",
|
||||
"__typename": "User"
|
||||
},
|
||||
"repository": {
|
||||
"nameWithOwner": "morluto/rea"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
+23
-7
@@ -110,9 +110,14 @@ rendered declarations. Keep canonical rejection checks separate from the
|
||||
intentionally reduced compact advertisement, and probe actual client renderers
|
||||
when changing schema representations.
|
||||
|
||||
The reused CLI boundary workers cap V8 old space at 384 MiB, based on their
|
||||
profiled allocation and cleanup behavior. This encourages collection between
|
||||
files; spawned REA processes retain their own heap settings.
|
||||
The reused CLI boundary workers cap V8 old space at 384 MiB for ordinary runs
|
||||
and 512 MiB with coverage enabled. Covered ADB fixtures exhaust the ordinary
|
||||
384 MiB budget; those fixtures also complete with the shared CLI graph preloaded
|
||||
at 512 MiB. This is a measured capacity control, not a suite-wide speedup claim.
|
||||
Both profiles keep the
|
||||
same bounded worker count, and spawned REA processes retain their own heap
|
||||
settings. Explicitly disabling coverage retains the ordinary profile; coverage
|
||||
provider options alone do not enable instrumentation.
|
||||
|
||||
Vitest worker limits do not bound subprocesses launched within a test. Bound
|
||||
those batches separately so repeated CLI validation cannot oversubscribe the
|
||||
@@ -350,6 +355,11 @@ The process-global Vitest configuration contract rejects new direct temporary-ro
|
||||
creation outside the workspace seam and its narrowly documented boundary/package
|
||||
exceptions.
|
||||
|
||||
The shared temporary-directory and workspace helpers preserve caller labels
|
||||
under an owned `rea-` prefix when needed. `npm run verify:test-temp-hygiene`
|
||||
uses the same ownership rule to find retained fixtures under a private temporary
|
||||
root. Third-party caches remain outside that ownership rule.
|
||||
|
||||
Real Hopper, Ghidra, IDA, browser, package, and managed-code claims belong to their
|
||||
explicit `npm run verify:*` lanes. The reconstruction-readiness lane also
|
||||
checks deterministic rerun, tamper, and stale-input handling; those checks do
|
||||
@@ -741,11 +751,12 @@ See [Android analysis](android-analysis.md) for boundaries and resource budgets.
|
||||
|
||||
`verify:jeb` requires a caller-started JEB client serving MCP at
|
||||
`REA_JEB_MCP_URL` (default `http://127.0.0.1:8425/mcp`) with a project
|
||||
already open, and verifies real CLI client inspection, unit listing coverage,
|
||||
already open, and exercises shared application/provider client inspection, unit listing coverage,
|
||||
and method decompilation. The script records the engine's exact response to
|
||||
`open_jeb_project`; JEB 5.48.0 headless instances do not advertise that tool.
|
||||
`open_jeb_project`; the earlier licensed 5.48.0 headless verification did not
|
||||
advertise that tool.
|
||||
REA does not install or launch JEB; see [JEB analysis](jeb-analysis.md) for
|
||||
the bring-your-own boundary. Verified against JEB 5.48.0 serving
|
||||
the bring-your-own boundary. The earlier licensed build was verified against JEB 5.48.0 serving
|
||||
`jeb-mcp-server` 1.3.0 at four scales: a compiled Java class fixture, a
|
||||
locally built signed probe APK with a launcher activity (manifest, v1/v2/v3
|
||||
certificates, dex bytecode, filtered and paginated listing), the published
|
||||
@@ -756,7 +767,12 @@ decompilation through both CLI and MCP, and the official Flutter Gallery
|
||||
analyzed as native code (75k methods on arm64) and decompiled through both
|
||||
CLI and MCP with explicit unit selection, plus the thin Java wrapper and
|
||||
decoded manifest. The dedicated Dart snapshot processor and `run_script`
|
||||
remain GUI-only surfaces in JEB 5.48.0 headless.
|
||||
remained GUI-only surfaces in that headless build.
|
||||
The separate [Linux CE native audit](../experiments/analysis-efficiency/jeb-linux-native.md)
|
||||
uses the official CE `5.48.0.202610071539` direct CLI launcher and real
|
||||
CLI/stdio MCP ELF workflows. That audit does not qualify CE Java/Android
|
||||
decompilation or GUI project opening, and it does not replace the licensed
|
||||
verification above.
|
||||
Authenticated IPA and macOS application inventory projection is documented in
|
||||
[Apple application analysis](apple-application-analysis.md).
|
||||
|
||||
|
||||
@@ -1,5 +1,28 @@
|
||||
# Analysis efficiency measurements
|
||||
|
||||
For Hopper literal string queries, see the [decoded-record checkpoint](hopper-string-records.md).
|
||||
It measures record reuse through real Linux CLI/MCP workflows, preserving
|
||||
native recognition gaps, guard-stopped attempts and negative whole-task costs.
|
||||
|
||||
The [Linux embedding-runtime follow-up](hopper-linux-python-runtime.md) checks
|
||||
actual clean setup, dependency repair and normal-user analysis, retaining
|
||||
failed attempts and distinguishing prerequisite correctness from speedups.
|
||||
|
||||
For JEB on Linux, see the [CE native audit](jeb-linux-native.md). It qualifies
|
||||
the direct headless launch route and measures selected function reads through
|
||||
real CLI and production stdio MCP, preserving timeout attempts and the limits
|
||||
of wrapper-cache and token claims.
|
||||
|
||||
For Apktool locale queries, see the
|
||||
[shared-decode and table-only checkpoint](apktool-string-tables.md). It separates
|
||||
the existing decode reuse from duplicate string delivery, with three real APKs,
|
||||
CLI/MCP parity, whole-task estimates and negative timing results.
|
||||
|
||||
For immutable imported native/JavaScript parents, see the
|
||||
[validation-reuse checkpoint](immutable-parent-validation.md). It measures the
|
||||
owning format change with the existing retained-query driver, including first
|
||||
view costs, small cases, unchanged token estimates, and noisy whole-task results.
|
||||
|
||||
For authentic old Evidence and mixed investigations, see the
|
||||
[retained-query checkpoint](retained-investigation-workflows.md).
|
||||
`retained-investigation.mjs` compares existing complete delivery with selected
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,171 @@
|
||||
# Apktool shared decoding and table-only projection
|
||||
|
||||
The measured question is the same in every comparison: inspect an APK's
|
||||
metadata/manifest and its default, German and Brazilian Portuguese string tables.
|
||||
One existing batch removes repeated decoding. An optional table-only projection
|
||||
then removes repeated primary strings without changing the selected facts.
|
||||
|
||||
This is a static resource-query measurement, not an agent evaluation or a claim
|
||||
about reconstructing an application's behavior. The machine-readable measured
|
||||
summary is [apktool-string-tables.json](apktool-string-tables.json).
|
||||
|
||||
## Revisions, inputs and execution
|
||||
|
||||
- Baseline: `8f7d12e30d63f3d3b4cfe2a90336362841438f56`, latest main at issue start.
|
||||
- Projection source: `406af90023a545fb280601bf51f445e122c36378`.
|
||||
- Existing shared decode: [#4777](https://github.com/morluto/rea/issues/4777),
|
||||
integrated by [#4804](https://github.com/morluto/rea/pull/4804).
|
||||
- Projection follow-up: [#4982](https://github.com/morluto/rea/issues/4982).
|
||||
- Linux x86-64, kernel 6.1.0-53-cloud-amd64, Node 24.18.0, npm 11.16.0,
|
||||
Temurin 21.0.12.1, Apktool 3.0.3. The selected JAR's closeout digest is
|
||||
`dbf930b076c6b9be08d57c449cacefc3bdd6b71ebd59b3066fc0e1f5b14f9423`;
|
||||
decoder contents were not independently hashed for every request.
|
||||
|
||||
- [newpipe](https://github.com/TeamNewPipe/NewPipe/releases/tag/v0.29.1): 11,507,122 bytes; SHA-256 `18447bfb1e06d113edc88df93f471827280de06f6f3d4dc42f56f28b9c1bab79`. Valid projected counts: 1126/833/779; unsupported entries: 1/1/1.
|
||||
- [apidemos](https://github.com/appium/android-apidemos/releases/tag/v6.0.18): 6,521,217 bytes; SHA-256 `a9eecf37b26cd084855c530db81c2bb1b91f4c1b095a04f47aa7c20e2791f686`. Valid projected counts: 967/29/29; unsupported entries: 8/0/0.
|
||||
- [settings](https://github.com/appium/io.appium.settings/releases/tag/v8.2.1): 3,099,589 bytes; SHA-256 `e4024b3ee348929252bcaf5eb422bcab9e4ace96703499a84f3332281c03375f`. Valid projected counts: 25/19/19; unsupported entries: 0/0/0.
|
||||
|
||||
The counts are default/de/pt-BR respectively. Source paths, XML digests, exact
|
||||
caller paths, runtime hashes and per-run identities/costs are in the JSON summary.
|
||||
The APKs were never installed or executed. Input digests were checked before and
|
||||
after each completed task; provider-owned workspaces were confirmed absent.
|
||||
|
||||
Each task uses a fresh production stdio MCP runtime, complete paginated tool
|
||||
discovery, client inspection, resource call(s), `close_binary`, transport close
|
||||
and owned cleanup. OS file caches were not flushed. Every decode has a fresh JVM,
|
||||
APK snapshot, framework directory and workspace: these are not warm retained
|
||||
engine/database queries, nor cold-disk measurements.
|
||||
|
||||
Tasks run serially on one logical CPU, with an 8 GiB family hard limit, 7 GiB
|
||||
early stop and no swap. Admission requires 16 GiB host availability; running
|
||||
tasks stop below 8 GiB host availability. CPU and kernel `memory.peak` cover
|
||||
the complete owned cgroup, including subprocesses, observer and file cache.
|
||||
Counters are captured at payload exit while the observer is alive; disappeared
|
||||
final whole-scope counters remain unknown. All completed payload receipts show
|
||||
successful owned cleanup and zero observed OOM/limit events at that checkpoint.
|
||||
|
||||
## Correctness and owning change
|
||||
|
||||
Separate native decodes were retained for independent Python Expat validation.
|
||||
It checks direct string entries, source entity spelling, unsupported markup
|
||||
counts, ordering and emitted prefixes against the XML bytes. Manifest text and
|
||||
package identity are checked separately. All six serial/batch pairs agree on
|
||||
selected strings, manifest, metadata and target identity. All nine projection
|
||||
pairs additionally preserve table counts and coverage; batch/table-only counts
|
||||
and coverage match the XML oracle. This qualifies REA's projection of these
|
||||
native outputs, not independent binary ARSC decoder completeness.
|
||||
|
||||
The existing default batch repeats its primary table in top-level `strings` and
|
||||
`string_tables[0]`. On NewPipe this made the full resource replies larger than
|
||||
three serial replies despite reducing decode calls: 616,059 → 680,188 bytes.
|
||||
ApiDemos' larger common manifest produces the opposite tradeoff. The existing
|
||||
batch remains the compatibility default.
|
||||
|
||||
`string_tables_only: true` (CLI `--string-tables-only`) keeps the complete
|
||||
requested table facts inline in `string_tables` and leaves `strings` empty by
|
||||
request. An unrequested field does not establish that a source table is empty.
|
||||
This also supports one table without `locales`. `include_strings: false` still
|
||||
suppresses string projection. Omitting the new option preserves old requests:
|
||||
three real legacy workflows produce identical complete MCP replies, including
|
||||
Evidence identity, at the new source revision.
|
||||
|
||||
The resource projection owner computes the output choice and charges aggregate
|
||||
entries for the copies actually emitted. The same 10,000-entry policy can thus
|
||||
return five distinct 2,000-entry tables rather than consuming capacity on a
|
||||
duplicate primary copy. Per-table limits, XML budgets, physical alias reuse,
|
||||
coverage, partial observations and cancellation remain at the existing owner.
|
||||
CLI and MCP share this implementation; there is no new tool or cross-call cache.
|
||||
|
||||
## Existing shared decode: three serial calls versus one batch
|
||||
|
||||
Three alternating fresh-session pairs per APK, all at the same main revision.
|
||||
The table reports marginal medians; paired speedups are reported separately.
|
||||
Elapsed/CPU include discovery, client/Java startup, calls, transcript/hash checks
|
||||
and cleanup. Native oracle creation, tokenization and verification gates are
|
||||
separate costs in the JSON, not hidden inside these task timings.
|
||||
|
||||
| Target | Resource calls, s | Whole elapsed, s | Whole CPU, s | Family peak, MiB |
|
||||
| -------- | ----------------: | ---------------: | ------------: | ---------------: |
|
||||
| newpipe | 23.19 → 7.40 | 35.32 → 19.86 | 34.95 → 19.65 | 1341.7 → 1309.5 |
|
||||
| apidemos | 17.43 → 6.22 | 30.32 → 18.85 | 30.05 → 18.63 | 1187.1 → 1202.7 |
|
||||
|
||||
Paired resource-call speedup medians: NewPipe **3.23×** (2.90–3.23), ApiDemos
|
||||
**2.80×** (2.66–3.09). Instrumentation confirms three versus one actual decoder
|
||||
invocations. These call timings include protocol/projection costs; they do not
|
||||
isolate the Java engine phase. NewPipe's whole raw token estimate increases
|
||||
951,332 → 962,330 with the default batch; ApiDemos decreases
|
||||
1,054,893 → 948,591. Fewer calls alone do not establish less context.
|
||||
|
||||
## New selector: default batch versus table-only batch
|
||||
|
||||
Three alternating pairs per APK; NewPipe repetitions 1/4/3, others 1/2/3.
|
||||
Repetition 4 replaces a refused start, described below. Both variants still
|
||||
perform one decode and return the same required facts.
|
||||
|
||||
| Target | Resource call, s | Whole elapsed, s | Whole CPU, s | Family peak, MiB |
|
||||
| -------- | ---------------: | ---------------: | ------------: | ---------------: |
|
||||
| newpipe | 9.18 → 9.35 | 23.95 → 24.85 | 23.65 → 23.82 | 1327.5 → 1323.6 |
|
||||
| apidemos | 6.46 → 5.81 | 23.28 → 19.79 | 22.90 → 19.47 | 1247.8 → 1197.0 |
|
||||
| settings | 4.21 → 4.06 | 17.66 → 19.39 | 17.42 → 19.04 | 1191.6 → 1122.4 |
|
||||
|
||||
There is no consistent latency or whole-family memory improvement established
|
||||
for the selector. NewPipe and Settings have higher whole-task elapsed and CPU
|
||||
medians. Elapsed ranges overlap: NewPipe 23.53–34.46 → 21.59–42.80 s;
|
||||
ApiDemos 18.55–27.07 → 19.25–25.96 s; Settings 16.55–23.39 → 16.14–21.72 s.
|
||||
The cause of this variation was not established; a passing or faster run does
|
||||
not explain a slower one. The selector's supported benefit is reduced duplicate
|
||||
delivery, with its full task costs retained here.
|
||||
|
||||
Full resource reply bytes fall 680,188 → 506,481 for NewPipe,
|
||||
616,841 → 463,588 for ApiDemos and 38,725 → 34,199 for Settings.
|
||||
|
||||
Offline `tiktoken` 0.12.0 / `cl100k_base` estimates count one complete catalog
|
||||
page-array plus every call name/argument/reply, including inspection and close.
|
||||
Full replies include both textual and structured copies. Text-only replies are
|
||||
also counted separately. These are serialization estimates, not actual model
|
||||
usage; initialization/framing, prompts, client filtering/rendering/replay and
|
||||
continuations are unmeasured.
|
||||
|
||||
| Target | Text reply tokens | Whole raw transport tokens | Whole estimated reduction |
|
||||
| -------- | ----------------: | -------------------------: | ------------------------: |
|
||||
| newpipe | 73,661 → 55,797 | 962,330 → 924,352 | 3.95% |
|
||||
| apidemos | 65,693 → 49,162 | 948,591 → 913,693 | 3.68% |
|
||||
| settings | 4,387 → 3,942 | 814,859 → 814,029 | 0.10% |
|
||||
|
||||
Discovery still has 207 tools; only `decode_android_resources` changes.
|
||||
The added schema costs 557 catalog bytes / 113 estimated catalog tokens
|
||||
(805,217 → 805,330). Raw discovery dominates this transport estimate; it does
|
||||
not establish what a client supplies to its model. Smaller replies and these
|
||||
estimates do not prove an actual agent context or investigation advantage.
|
||||
|
||||
## Verification, attempts and limits
|
||||
|
||||
- 94 source tests: provider/resource projection/service, including alias reads,
|
||||
aggregate output exhaustion, malformed/physical failures, partial facts,
|
||||
cancellation and cleanup. The table-only capacity, single-table provider
|
||||
and CLI cases fail on the old implementation (three selected failures,
|
||||
82 unrelated tests skipped).
|
||||
- Nine compiled CLI/MCP tests: schema conversion/advertisement, option forwarding,
|
||||
Evidence parameters, legacy locale grammar and existing boolean choices.
|
||||
- 36 completed real MCP tasks and three real CLI tasks. Nine exact projection
|
||||
oracle pairs; three legacy full-reply comparisons; alias/missing, single-table
|
||||
and omitted-string cases pass. CLI normalized data, subjects and Evidence IDs
|
||||
match MCP exactly on all three APKs.
|
||||
- `check:fast`, `build:cached` and `verify:test-discovery` passed; the last
|
||||
discovered all 1,579 test files exactly once. `npm run check` and `npm run docs:check` also passed.
|
||||
- An authored test group initially exceeded the 150-line lint limit (153).
|
||||
That gate failed after 189.31 s / 186.19 CPU s, peak 4,487,798,784 bytes;
|
||||
build/compiled checks did not run in that attempt. The capacity test was moved
|
||||
out of that group, and the complete corrected gate passed after 343.97 s /
|
||||
338.67 CPU s, peak 4,311,293,952 bytes. No rule was disabled.
|
||||
- `projection-newpipe-after-2` was refused before scope/payload creation because
|
||||
host availability was below 16 GiB; a following poll showed about 10.5 GiB.
|
||||
It has no payload timing/counters and is not a completed measurement. The
|
||||
refusal stays recorded; a fresh whole repetition-4 pair ran after recovery.
|
||||
No existing receipts were overwritten and resource thresholds were unchanged.
|
||||
|
||||
Raw transcripts, native outputs, intermediate failures and per-request workspace
|
||||
arguments remain in ignored local audit storage; the committed JSON is a curated
|
||||
summary. Small Settings resources and negative time results are retained.
|
||||
Actual models, APK execution, other providers/platforms and general complex
|
||||
software effectiveness were not verified by this checkpoint.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,203 @@
|
||||
# Linux Hopper embedding-runtime qualification
|
||||
|
||||
This follow-up to the [string-record audit](hopper-string-records.md) resolves
|
||||
[#5015](https://github.com/morluto/rea/issues/5015): a successful package install
|
||||
could leave Hopper unable to start its Python bridge. This is a prerequisite
|
||||
correctness fix. It does not establish faster equivalent investigations.
|
||||
|
||||
## Reproduction and owning correction
|
||||
|
||||
Baseline REA `093ce695f0a7c3407c7fde520ed8b20c0981b408` ran the actual public
|
||||
`rea setup --yes --install-hopper --format json` in a fresh Ubuntu Base root,
|
||||
with full Linux UID/GID support, successful strict APT metadata refresh,
|
||||
default APT recommendations and unchanged vendor post-install scripts. Setup
|
||||
reported `ready` and `installed_hopper`; `dpkg --audit` was empty. Python 3.12
|
||||
was configured, but its reported `/usr/lib/x86_64-linux-gnu/libpython3.12.so.1.0`
|
||||
was absent. The signed Hopper package did not declare this dependency.
|
||||
|
||||
An actual public MCP control-ELF workflow then timed out at its selected
|
||||
30-second startup budget. Installing only `libpython3.12` resolved to
|
||||
`libpython3.12t64` version `3.12.3-1ubuntu0.17`: one new package, no upgrades or
|
||||
removals. The unchanged REA runtime and target then completed all 13 calls.
|
||||
The independent earlier bootstrap dialog and controlled package intervention
|
||||
support the embedding-dependency diagnosis; timing alone does not.
|
||||
|
||||
The shared Linux installer now asks `/usr/bin/python3` for its version and
|
||||
shared-library location after the original package transaction. It attempts an
|
||||
actual load. For a missing standard Ubuntu ABI/layout it installs the matching
|
||||
versioned library and verifies loadability again. APT success alone cannot
|
||||
establish readiness. Unknown metadata, alternate prefixes and alternate SONAMEs
|
||||
retain an explicit unknown diagnosis instead of guessing a package.
|
||||
|
||||
Doctor consumes the same fresh, bounded producer, retaining both private-display
|
||||
and Python observations. Its Python facts concern the system interpreter;
|
||||
the interpreter actually selected inside Hopper remains unknown. The query
|
||||
uses isolated Python mode, disables site hooks and bytecode writes, has a
|
||||
10-second work deadline and 64-KiB output bound, and is not cached: package/interpreter
|
||||
changes must be visible immediately. The approval plan discloses the conditional
|
||||
package change without returning the probe program. Existing MCP analysis
|
||||
contracts and Evidence semantics are unchanged.
|
||||
|
||||
Cancellation prevents starting another install stage and preserves the original
|
||||
reason. A running package transaction is allowed to settle, avoiding forced
|
||||
interruption of dpkg. This qualifies the install API's signal behavior through
|
||||
boundary tests; it does not claim new CLI interrupt plumbing or actual
|
||||
interruption of a real package transaction.
|
||||
|
||||
A review follow-up preserves cancellation through doctor as well. A cancelled
|
||||
Python probe now raises the existing domain cancellation error with its original
|
||||
reason and completed private-display observations; the owned doctor workflow
|
||||
performs cleanup and attaches any cleanup failure. It no longer becomes an
|
||||
ordinary unknown diagnostic. The probe also opts into the existing
|
||||
stop-and-wait supervisor: cancellation and timeout wait for Python to exit,
|
||||
with the shared five-second grace period and forced-kill fallback. A real
|
||||
Python fixture holds its SIGTERM handler until the test releases it, checking
|
||||
that the probe retains ownership until exit. The success-path native measurements below precede
|
||||
this error-branch correction. Its boundary tests cover both successful cleanup
|
||||
and cleanup refusal; final-head validation is tracked on the PR. A pre-fix red
|
||||
execution and a final-head native rerun are not claimed.
|
||||
|
||||
## Pinned environment and artifacts
|
||||
|
||||
Native success measurements pin the initial fix at
|
||||
`0c70be0030f6bb821c4292c4f5b7f776b365c8ec`, based on main
|
||||
`23f984b1a884baefc3c44e4ab697f6eff61dffb6`.
|
||||
The measured runtime and bridge manifest covers 1,733 regular files, 26,537,393
|
||||
bytes. Its SHA-256 is
|
||||
`8800edc8e70965ae2132e57ca6d999ba276503f27f1045b33305c83cd0e2ff2a`
|
||||
(sorted paths, compact sorted-key JSON records containing path, bytes and
|
||||
SHA-256). The tiny MCP loader alone is not a whole-runtime identity.
|
||||
Earlier prototype runs have a separately recorded manifest; they are not
|
||||
presented as executions of that same artifact.
|
||||
|
||||
- Node 24.18.0; npm 11.16.0; Debian 12 x86-64 host.
|
||||
- Official [Hopper 6.6.0 Linux demo](https://www.hopperapp.com:443/downloader/public/Hopper-6.6.0-Linux-demo.deb),
|
||||
35,748,722 bytes, SHA-256
|
||||
`59705e50f7ebf9104abe1e72ac3949bed1a8e43ec67a8c9510a1c5056c200d27`.
|
||||
REA's real publisher verifier authenticated signing fingerprint
|
||||
`7C2D73BAFAF5B9A710A69DA1A26DF815099B1BCD`; launcher SHA-256
|
||||
`b2271008fb82bf14cfae5c3644cdde2b11fc6804bd7c99d67032ec1b1942d53c`.
|
||||
- Private [Ubuntu Base 24.04.5 root](https://cdimage.ubuntu.com/ubuntu-base/releases/24.04/release/ubuntu-base-24.04.5-base-amd64.tar.gz),
|
||||
SHA-256 `e77b6f10c2590cef872b33ee9f635a0e3fd1f57fb074c0e52b5c7f56147a0c86`,
|
||||
matched its official HTTPS checksum; its publisher signature was not
|
||||
independently verified. The host glibc 2.36 cannot run this Hopper package;
|
||||
the private root supplies glibc 2.39. Related packages were installed only
|
||||
there. Strict refresh used `APT::Update::Error-Mode=any`.
|
||||
- Small control ELF: 17,360 bytes, SHA-256
|
||||
`1a6f55303712e096ff5196c7fd6e8acf75473fbf0f7467d7b315f96f50b3f841`.
|
||||
GNU objdump independently maps the arithmetic at `0x401126` to
|
||||
`arg0 * 7 + 13`; the original build flags were not recovered. The known
|
||||
`rea_jeb_native_probe` literal has one native string hit.
|
||||
- Shipped Debian SQLite 3.40.1-2+deb12u2, `libsqlite3.so.0.8.6`, 1,437,848
|
||||
bytes, SHA-256
|
||||
`2e6eef9a727f081f0d453b4e5e6cbd8b9ef8b6f86cbf7681cbad444d3b0b55c8`.
|
||||
Independent objdump identifies `sqlite3_libversion_number` at `0xa1d50`,
|
||||
returning `0x2e6301` (`3040001`). This is a shipped library qualification,
|
||||
not a claim about current SQLite versions or whole-program correctness.
|
||||
|
||||
## Actual workflows and measurements
|
||||
|
||||
The accompanying [receipt](hopper-linux-python-runtime.json) preserves native
|
||||
stage outcomes, elapsed time, CPU time, peak memory, cleanup and token-estimate
|
||||
scope, including failed and stopped attempts. Installation uses root inside the
|
||||
owned fixture. Final analysis also uses a normal UID 1004 with a separate home
|
||||
and the authenticated public publisher evidence copied unchanged into that
|
||||
user's cache; production verification still checks it. No target was executed
|
||||
or modified. Non-root package installation through real pkexec authorization,
|
||||
custom Hopper Python configuration, native Debian installation and other
|
||||
operating systems remain unverified.
|
||||
|
||||
Fresh final setup reported `ready` / `installed_hopper`, installed the matching
|
||||
shared library, left `dpkg --audit` empty and passed actual public doctor.
|
||||
Final normal-user sessions completed the positive, repeated, negative and
|
||||
decompilation checks: 19 `sqlite_master` matches in 1,437 recognized strings,
|
||||
74 broader `sqlite` matches and the expected version constant. Both CLI runs
|
||||
matched their MCP facts and Evidence identity; both target files stayed unchanged.
|
||||
|
||||
| Stage / outcome | Elapsed (s) | Native CPU (s) | Native peak (MiB) |
|
||||
| ---------------------------------------------- | ----------: | -------------: | ----------------: |
|
||||
| Baseline setup: ready, library absent | 75.353 | 39.002 | 1395.4 |
|
||||
| Baseline control MCP: startup timeout | 49.541 | 24.158 | 1282.5 |
|
||||
| Install only matching libPython | 5.012 | 1.925 | 174.8 |
|
||||
| Unchanged runtime control MCP: passed | 24.791 | 23.608 | 1228.4 |
|
||||
| Fixed fresh setup: ready | 80.739 | 40.404 | 1382.6 |
|
||||
| Fixed public doctor: healthy | 6.035 | 5.160 | 560.9 |
|
||||
| Fixed normal-user control MCP: 13 calls passed | 24.434 | 23.305 | 1195.7 |
|
||||
| Fixed normal-user SQLite MCP: 16 calls passed | 92.504 | 90.138 | 1507.2 |
|
||||
| Fixed normal-user control CLI: parity passed | 8.981 | 7.831 | 613.5 |
|
||||
| Fixed normal-user SQLite CLI: parity passed | 70.207 | 68.704 | 781.6 |
|
||||
|
||||
Each control MCP session includes discovery and 13 calls: open, status, first
|
||||
literal query, six repeats, absent query, procedure listing, decompilation and
|
||||
close. SQLite uses 16 calls: a known exported address avoids a separate name
|
||||
lookup, and full native string inventory, broader query, regex, case and absence
|
||||
checks are included. Applicable returned string bytes are checked against the
|
||||
file. Warm queries must preserve facts and Evidence ID. CLI searches compare
|
||||
normalized facts and Evidence ID with their corresponding MCP result. Both
|
||||
target digests are checked after the task.
|
||||
|
||||
“Fresh” means a new installation state or Hopper process, as indicated; OS
|
||||
caches were not flushed. Installation fixtures preload 247 previously fetched
|
||||
APT archives (138,023,324 bytes) without installing packages. These are not
|
||||
cold-network benchmarks. The baseline failure and successful repair are
|
||||
different outcomes, not a paired speedup. This change adds no analysis cache
|
||||
and makes no token reduction or complex-investigation advantage claim.
|
||||
|
||||
Native stages have a 6-GiB container cap, 5-GiB early stop, no swap, 128 tasks,
|
||||
one CPU and explicit affinity. Their separate controller has a 512-MiB cap,
|
||||
no swap and 64 tasks. Native plus controller caps total 6.5 GiB. Heavy tasks
|
||||
run serially; admission waits for at least 16 GiB host availability and aborts
|
||||
below 8 GiB. Host build/check stages have an 8-GiB cap and 7-GiB early stop.
|
||||
No other user's processes were signalled.
|
||||
|
||||
Native CPU and kernel memory peaks include the container guardian and observer.
|
||||
Payload-exit observations verify no analysis children remain before stopping
|
||||
the container. Controller counters are separate partial snapshots; adding
|
||||
separate peaks is only an upper bound, not a simultaneous measurement. Shared
|
||||
Docker daemon, containerd and system-manager costs and final counters after
|
||||
scope removal are unknown. Stopped attempts without payload-exit observations
|
||||
retain unknown cleanup/counter fields. Observed zero OOM counters at payload
|
||||
exit are not a claim about the whole host.
|
||||
|
||||
## Failed attempts and limits
|
||||
|
||||
- A single-ID namespace could not apply the real vendor/package UID/GID
|
||||
ownership changes. It did not qualify a successful installation.
|
||||
- An inherited container-loopback proxy prevented publisher downloads; the
|
||||
fixture was corrected to forward the same configured proxy through the
|
||||
owned bridge gateway. No global proxy configuration was changed.
|
||||
- An early APT refresh exited zero with failed-index warnings. It was not
|
||||
counted as a successful refresh; the strict clean run establishes the result.
|
||||
- One earlier setup was stopped by the host-memory guard during unpacking.
|
||||
Its interrupted package state was retained separately; clean fixtures were
|
||||
used for the qualified baseline and final installations.
|
||||
- SQLite exceeded a caller-selected 30-second startup budget. With the normal
|
||||
120-second budget the bridge started, but an initial `libc` positive oracle
|
||||
failed: those `.dynstr` bytes were not a recognized Hopper string. The
|
||||
corrected native positive is `sqlite_master`. These attempts remain in the
|
||||
receipt. Empty recognized-string results do not prove physical absence.
|
||||
- Early candidate compilation found a TypeScript signal-narrowing error; the
|
||||
fix reads live signal state through a helper. Lint found oversized new test
|
||||
groups; they were split by behavior. Failed checks are retained as failures.
|
||||
- Full local fast checks were stopped twice when other host workloads reduced
|
||||
available memory. Those attempts did not pass; complete CI and later executed
|
||||
checks must be reported separately.
|
||||
- Local package verification completed installation, update, module loading,
|
||||
discovery and several CLI stages, then the same host-memory guard stopped it
|
||||
during setup/uninstall after 336.196 seconds (sampled peak 1,742,426,112
|
||||
bytes). This is an incomplete package gate, despite the separately completed
|
||||
real Hopper setup and analysis workflows. Its payload-exit counters and
|
||||
cleanup observation are unknown. CI must complete this gate before merging.
|
||||
|
||||
Offline `cl100k_base` estimates include each discovered full catalog, named
|
||||
arguments and raw replies once per recorded session, plus selected CLI arguments
|
||||
and stdout. Raw MCP replies can contain both text and structured copies.
|
||||
The final control and SQLite MCP validation transcripts estimate 845,126 and
|
||||
1,064,574 tokens respectively, including 805,330 catalog tokens each. The CLI
|
||||
rows estimate 695 and 2,905 tokens for one selected search each. Those CLI rows
|
||||
cover different tasks from the multi-call MCP qualification sessions, which
|
||||
also include full-inventory validation; they do not form an interface-efficiency
|
||||
comparison. Failed attempts and retries have separate rows in the receipt.
|
||||
Actual agent/model usage, host schema filtering, protocol framing, prompts,
|
||||
rendering/replay, independent oracle context and unrecorded in-flight failures
|
||||
are unmeasured. This is provider qualification, not an agent benchmark.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,236 @@
|
||||
# Hopper literal search: reuse decoded records
|
||||
|
||||
For [#5010](https://github.com/morluto/rea/issues/5010), the bridge now keeps
|
||||
the existing decoded records in its sorted search inventory and projects only
|
||||
matches. Previously, every positive literal search called `_string_inventory`
|
||||
again, sorting all native object addresses and building another complete map
|
||||
before selecting records. The value-only `_strings` representation and the
|
||||
intermediate match list have been removed.
|
||||
|
||||
This removes repeated work at its owner. Three paired SQLite workflows found
|
||||
shorter warm queries, but **did not demonstrate an overall task speedup**:
|
||||
whole elapsed and CPU were worse. Token estimates were effectively unchanged; no memory
|
||||
improvement was established. The [curated receipt](hopper-string-records.json)
|
||||
keeps individual measurements, equality checks and failed attempts.
|
||||
|
||||
## Representation and reuse boundary
|
||||
|
||||
The existing `(id(document), kind)` search cache now holds references to full
|
||||
records rather than just their decoded values. The existing string-record
|
||||
owner still performs byte decoding, encoding inference and unknown handling.
|
||||
Matching reads `record.value`; output copies the selected record with its
|
||||
address, retaining native display, encoding/extent metadata and decoding
|
||||
failures. The original document lifetime and annotation invalidation apply.
|
||||
This does not add another cache or duplicate decoded payloads. Both CLI and
|
||||
MCP use the same bridge through the existing application workflow. Regex keeps
|
||||
its supervised worker implementation.
|
||||
|
||||
The lookup remains a linear scan of recognized native strings. A complete
|
||||
inventory is still decoded for the first exhaustive search, and returned hits
|
||||
still consume output and retained Evidence. This change does not introduce
|
||||
pagination, broader binary string discovery or a new context contract.
|
||||
|
||||
## Pinned real-provider scope
|
||||
|
||||
Baseline: REA 6.7.0 at `c12d9eda87aa5a022ee7e18ea96d8d03bfd12e96`, Node
|
||||
24.18.0, Debian 12 x86-64, Linux `6.1.0-53-cloud-amd64`. All paired tasks used
|
||||
the same compiled runtime SHA-256
|
||||
`0809b5004f214efa4b79775d5fda796eaecf4ad5756c55ee3b9d7cf82d91b750`.
|
||||
Baseline and candidate bridge digests are recorded separately in the receipt;
|
||||
the baseline file was mounted read-only over the bridge for baseline runs.
|
||||
|
||||
The official [Hopper 6.6.0 Linux demo package](https://www.hopperapp.com:443/downloader/public/Hopper-6.6.0-Linux-demo.deb)
|
||||
was authenticated through REA's production publisher verifier: 35,748,722
|
||||
bytes, SHA-256
|
||||
`59705e50f7ebf9104abe1e72ac3949bed1a8e43ec67a8c9510a1c5056c200d27`,
|
||||
primary signing fingerprint `7C2D73BAFAF5B9A710A69DA1A26DF815099B1BCD`.
|
||||
The extracted launcher digest was
|
||||
`b2271008fb82bf14cfae5c3644cdde2b11fc6804bd7c99d67032ec1b1942d53c`.
|
||||
Normal demo activation used an owned private Xvfb display.
|
||||
|
||||
The host's glibc 2.36 could not satisfy this package's GLIBC_2.38 requirement.
|
||||
Analysis therefore ran in a user-mapped private Ubuntu Base 24.04.5 namespace
|
||||
with glibc 2.39, Qt6/X11 and embedded Python 3.12. The
|
||||
[rootfs archive](https://cdimage.ubuntu.com/ubuntu-base/releases/24.04/release/ubuntu-base-24.04.5-base-amd64.tar.gz)
|
||||
matched its official HTTPS checksum, SHA-256
|
||||
`e77b6f10c2590cef872b33ee9f635a0e3fd1f57fb074c0e52b5c7f56147a0c86`;
|
||||
its publisher signature was not independently verified. Related libraries were
|
||||
installed only into that private image. Actual `rea setup` and vendor package
|
||||
post-install scripts were not executed. Native Debian installation and other
|
||||
operating systems remain unqualified.
|
||||
|
||||
| Artifact | Bytes | Independent oracle |
|
||||
| ------------------------------------ | --------: | -------------------------------------------------------- |
|
||||
| Small authored C ELF | 17,360 | Source `input * 7 + 13` and ELF symbol |
|
||||
| Source-built conformance ELF | 22,584 | Fixture source, known literal bytes and symbol addresses |
|
||||
| Shipped Debian `libsqlite3.so.0.8.6` | 1,437,848 | GNU objdump: version function returns `3040001` |
|
||||
|
||||
Artifact digests, compiler and flags are in the receipt. No target was executed.
|
||||
The small ELF was a baseline setup/decompilation smoke test; the conformance
|
||||
ELF had baseline/candidate semantic and CLI/MCP checks. SQLite supplied the
|
||||
repeated performance cohort. A constant-return oracle does not establish
|
||||
whole-program correctness, ABI inference or complex-investigation effectiveness.
|
||||
|
||||
## Completed task and correctness
|
||||
|
||||
Each of three baseline/candidate pairs used a fresh Hopper process and the
|
||||
same public stdio MCP workflow, including catalog discovery and 17 tool calls:
|
||||
|
||||
1. Open the SQLite copy and inspect `binary_session`.
|
||||
2. Search `sqlite_master` in case-sensitive literal mode, then repeat six times.
|
||||
3. Read `list_strings` as a full-inventory validation oracle; compare all
|
||||
matching records, including their metadata, with the selected result.
|
||||
4. Check broader `sqlite` search, regex, case-insensitive and absent-pattern
|
||||
queries against their expected native inventory scope.
|
||||
5. Resolve `sqlite3_libversion_number`, decompile it, check its constant against
|
||||
independent objdump, and close the binary and MCP transport.
|
||||
6. Verify unchanged target bytes and cleanup of the owned process family.
|
||||
|
||||
There were 1,437 recognized string objects, 19 `sqlite_master` matches and 74
|
||||
broader `sqlite` matches. Applicable decoded hit bytes were also checked
|
||||
against file contents. All 42 Evidence payloads across the three pairs matched
|
||||
on normalized and raw facts, identity, subject, provider/profile, authority,
|
||||
confidence, limitations, relationship coverage, locations and evidence links.
|
||||
Live environment observations were excluded from cross-session equality.
|
||||
|
||||
The real conformance workflows each completed 70 public calls, including six
|
||||
expected errors. They exercised Unicode symbol identity, literal metacharacters,
|
||||
case/regex modes, annotation cache invalidation, typed string fragments,
|
||||
display escaping and source-byte checks using the existing real-provider
|
||||
verifier. CLI `fragment_tail` search returned the same normalized facts and
|
||||
Evidence ID as MCP on both revisions (2,293 stdout bytes each).
|
||||
|
||||
Native recognition remains qualified: the full 4,120-byte source literal and
|
||||
some known UTF-8, ellipsis and Latin-1 literals were not recognized. Their file
|
||||
bytes were verified, while decoded native-object coverage remains unverified.
|
||||
The fragmented 549-byte source literal was recognized as separate objects;
|
||||
matching does not pretend to join those fragments. Escaped text and literal
|
||||
backslash-n were distinguished. Real UTF-16 provider decoding was not qualified;
|
||||
the focused boundary fixture covers that representation separately. Empty
|
||||
search results establish absence only within the native typed-object inventory.
|
||||
|
||||
## Measurements
|
||||
|
||||
Runs were serial and ordered B1/C1, C2/B2, B3/C3. C2 first stopped under the
|
||||
host memory guard; its separate completed retry supplies the C2 row. OS caches
|
||||
were not flushed. “Cold” means the first string query in a fresh provider;
|
||||
“warm” means six repeats within that provider.
|
||||
|
||||
Medians across the three completed sessions per revision:
|
||||
|
||||
| Cost | Baseline | Candidate |
|
||||
| ---------------------------------------- | ----------: | ----------: |
|
||||
| Per-run median warm query | 14.849 ms | 12.628 ms |
|
||||
| Six warm queries, total | 100.202 ms | 75.552 ms |
|
||||
| Whole SDK workflow | 86.583 s | 90.786 s |
|
||||
| Whole guarded workflow | 89.384 s | 93.630 s |
|
||||
| Family CPU at payload exit | 88.333 s | 92.590 s |
|
||||
| Kernel family peak, including file cache | 1,438.4 MiB | 1,440.0 MiB |
|
||||
|
||||
Warm-query medians were lower in all three pairs, although one baseline run
|
||||
was particularly variable. Whole guarded elapsed was about 4.7% worse and CPU
|
||||
about 4.8% worse. The first query accounted for 65.2–71.2 seconds on baseline
|
||||
and 72.3–77.5 seconds on candidate; that interval includes native preparation.
|
||||
Do not interpret this cohort as end-to-end acceleration or dismiss the
|
||||
unfavorable rows.
|
||||
|
||||
A separate pair moved the **existing** `procedure_address` call before the
|
||||
first string search, keeping the same 17 calls and equivalent facts/Evidence.
|
||||
Most preparation cost then occurred in that unchanged lookup: 67.949 seconds
|
||||
baseline versus 94.358 seconds candidate, before the changed search path ran.
|
||||
The subsequent first string query took 329.254 versus 337.533 ms; total guarded
|
||||
elapsed was 89.096 versus 113.917 seconds. This locates substantial cost outside
|
||||
literal projection and preserves the negative whole-task result. It does not
|
||||
prove its cause, establish a cold-query improvement or justify pooling the
|
||||
reordered pair with the original cohort.
|
||||
|
||||
## Resource and context accounting
|
||||
|
||||
Every heavy task used an owned cgroup v2 family: 8 GiB hard limit, 7 GiB early
|
||||
stop, zero swap, one CPU quota and affinity, 192 tasks, 3,072 MiB Node heap,
|
||||
and one heavy job at a time. Host admission required 16 GiB available; tasks
|
||||
stopped below 8 GiB. Later long runs added admission at 24 GiB sustained for
|
||||
20 seconds without relaxing the family guard.
|
||||
|
||||
Family counters include the namespace, native engine, MCP client/runtime,
|
||||
validation and cleanup, including charged file cache. SDK wall time includes
|
||||
catalog discovery and correctness assertions. Admission wait is outside the
|
||||
guarded time and recorded separately where available; older waits and outer
|
||||
guard/systemd costs were not measured. Payload-exit CPU, kernel memory peak,
|
||||
limit counters and owned cleanup were verified while an observer remained.
|
||||
Final counters after the scope disappeared remain unknown. Completed paired
|
||||
payloads observed zero memory-limit, OOM/OOM-kill and task-limit events.
|
||||
|
||||
The unchanged catalog advertised 207 tools, 3,158,927 serialized bytes,
|
||||
SHA-256 `a1851aa1bfe3d72d6be6bec130519dbdfc1ee49288320cfac0144ced77755c61`.
|
||||
Offline tiktoken 0.12.0 `cl100k_base` estimates for every completed SQLite
|
||||
comparison were effectively unchanged (medians across three runs):
|
||||
|
||||
| Serialization counted once | Baseline estimate | Candidate estimate |
|
||||
| -------------------------------------------------- | ----------------: | -----------------: |
|
||||
| Full catalog page array | 805,330 | 805,330 |
|
||||
| Named tool requests | 361 | 361 |
|
||||
| Raw replies | 259,809 | 259,811 |
|
||||
| Total above | 1,065,500 | 1,065,502 |
|
||||
| Text reply blocks alone, overlapping the raw count | 126,832 | 126,833 |
|
||||
|
||||
Whole raw estimates ranged from 1,065,496–1,065,500 on baseline and
|
||||
1,065,500–1,065,506 on candidate. The small variations include live environment
|
||||
observations; there is no context reduction.
|
||||
|
||||
The full inventory and extra controls are validation overhead included in
|
||||
these totals. Raw replies may contain text and structured copies. Actual model
|
||||
usage, prompts, initialization/protocol framing, client filtering/rendering or
|
||||
replay and independent-oracle context were not measured. The receipt separates
|
||||
other tasks and the interrupted transcript's persisted lower bound; it does
|
||||
not count an unrecorded in-flight request or reply as observed. No model-token
|
||||
saving or agent-reasoning advantage is claimed.
|
||||
|
||||
## Failed attempts and verification
|
||||
|
||||
The receipt preserves setup, diagnostic, failed and completed costs separately:
|
||||
|
||||
- Initial private-rootfs configuration failed on an unmapped package GID, then
|
||||
on the harness PATH; ordinary pending configuration completed successfully.
|
||||
- One low-host-memory probe was refused before creating a payload.
|
||||
- The first MCP harness used a relative path, correctly refused by the public
|
||||
input boundary. A later custom-runtime launch lacked libXtst6, which the
|
||||
production dependency list already includes. Both were harness/setup errors.
|
||||
- Two unchanged-baseline launches timed out at bridge startup. An owned-display
|
||||
observation showed completed native analysis but a Python bootstrap failure.
|
||||
Adding only the missing embedded Python library made the baseline pass;
|
||||
[#5015](https://github.com/morluto/rea/issues/5015) records the separate
|
||||
installer dependency hypothesis and its unverified actual-installer scope.
|
||||
- C2 stopped after 82.607 seconds when host available memory crossed the
|
||||
unchanged stop threshold. Its sampled family peak was 1,517,019,136 bytes
|
||||
and last observed CPU 78.255 seconds. Payload-exit/final counters are unknown;
|
||||
it is incomplete and excluded from completed-run statistics.
|
||||
- An initial `test:local` invocation rejected built boundary projects before
|
||||
running tests. The documented `test:focused` invocation then passed all
|
||||
35 tests in five files, covering string search, regex, bridge facade and
|
||||
real-provider verifier behavior. Those local tests do not themselves qualify
|
||||
a native engine; the public native workflows above provide that evidence.
|
||||
|
||||
`npm run check:fast` also passed (typecheck and lint both cached). Build and
|
||||
metadata preparation passed before the native workflows. `npm run docs:check`
|
||||
passed its cached metadata checks and all five README consistency tests.
|
||||
|
||||
Reproduction uses `node scripts/rea.mjs --mcp` with an explicitly configured
|
||||
Hopper launcher and private display. Drive the named calls above with absolute
|
||||
target paths, retain full replies and compare independently verified bytes,
|
||||
facts and Evidence. For the source fixture, compile
|
||||
`tests/conformance/c/fixture.c` with
|
||||
`gcc -O0 -g -fno-inline -fno-omit-frame-pointer -fno-pie -no-pie`.
|
||||
Debug build paths can change rebuilt artifact digests. The shipped SQLite
|
||||
digest pins the measured artifact rather than every build of its package.
|
||||
|
||||
The focused local gate was:
|
||||
|
||||
```sh
|
||||
npm run test:focused -- \
|
||||
tests/boundary/providers/hopper/hopperStringSearch.test.ts \
|
||||
src/hopper/HopperRegexSearch.test.ts \
|
||||
tests/boundary/providers/hopper/hopperBridgeFacade.test.ts \
|
||||
tests/conformance/providers/hopper/realHopperSearch.test.ts \
|
||||
tests/conformance/providers/hopper/realHopperExhaustiveSearch.test.ts
|
||||
```
|
||||
@@ -0,0 +1,654 @@
|
||||
{
|
||||
"issue": "https://github.com/morluto/rea/issues/4810",
|
||||
"baseline": "2c4338db1fdb7773e05376301c558cff2a0b4b80",
|
||||
"candidate": "3930a640fb8387cbd3079eee13fb90f82d1881dc",
|
||||
"units": {
|
||||
"latency": "milliseconds",
|
||||
"elapsed_cpu": "seconds",
|
||||
"memory": "kernel cgroup memory.peak bytes, including charged file cache and descendants"
|
||||
},
|
||||
"scope": "Three alternating before/after pairs per authentic retained native/JavaScript/small workflow; fresh stdio servers, three traversals of three two-row pages; catalog/import/oracles/export/close included in whole cost. OS caches uncontrolled. No actual model trial.",
|
||||
"resource_bounds": {
|
||||
"cpu_count": 1,
|
||||
"hard_memory_bytes": 8589934592,
|
||||
"early_stop_bytes": 7516192768,
|
||||
"swap_bytes": 0,
|
||||
"tasks_max": 192,
|
||||
"host_start_available_bytes_min": 17179869184,
|
||||
"host_stop_available_bytes_min": 8589934592,
|
||||
"observer_sha256": "6f089e645459eba80c5d73e39c21606c78173a44daf49fabc3b88c4631ce51c6",
|
||||
"guard_sha256s": [
|
||||
"760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992",
|
||||
"d2047721e50dd244900eb514d07025386e60908f77bb546bee5684baa1da2f19"
|
||||
],
|
||||
"guard_observability_note": "The newer private guard only records existing host-available-memory readings and its own cgroup path. Resource thresholds, CPU affinity, observer and stop/cleanup logic are unchanged."
|
||||
},
|
||||
"resource_accounting": "Final complete corpus tasks exit0/cleanup verified/oom0 at payload exit while observer remains alive. Resource-stopped and earlier failed attempts are recorded separately; terminal whole-scope counters are unavailable. Weak cache metadata/private heap not measured independently.",
|
||||
"catalog": {
|
||||
"tools": 207,
|
||||
"bytes": 3158368,
|
||||
"sha256": "1d6c9a679b7c935318f5ea59d5362ef0a06945b3f4a4f210c86997ec15ba4f4c",
|
||||
"equal_across_all_18_runs": true
|
||||
},
|
||||
"facts": {
|
||||
"native": {
|
||||
"answer_sha256": "4e28c2c2a25620576d27abe846a05835ec49386b889779e33410c31971523b54",
|
||||
"calls": 25,
|
||||
"query_structured_content_equal": true,
|
||||
"original_records_preserved": true,
|
||||
"closed_session_cleared": true
|
||||
},
|
||||
"javascript": {
|
||||
"answer_sha256": "0fe3ad249eb1a7233a4966a56652c173667c48fdaf395b6c08fc6901ef3b46f7",
|
||||
"calls": 28,
|
||||
"query_structured_content_equal": true,
|
||||
"original_records_preserved": true,
|
||||
"closed_session_cleared": true
|
||||
},
|
||||
"small": {
|
||||
"answer_sha256": "9ca5874ce7bb610e39632157304fe4f52b09401d5fbd97ec981ad0588b8da485",
|
||||
"calls": 19,
|
||||
"query_structured_content_equal": true,
|
||||
"original_records_preserved": true,
|
||||
"closed_session_cleared": true
|
||||
}
|
||||
},
|
||||
"runs": [
|
||||
{
|
||||
"name": "topology-paired-after-javascript-1",
|
||||
"first_page_ms": 2720.542831999999,
|
||||
"eight_later_pages_ms": 82.69408900000417,
|
||||
"elapsed_seconds": 20.713,
|
||||
"family_cpu_seconds": 20.50093,
|
||||
"family_peak_bytes": 1311358976,
|
||||
"import_ms": 10970.411619,
|
||||
"catalog_ms": 1194.7045099999996,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-after-javascript-2",
|
||||
"first_page_ms": 1978.732818999999,
|
||||
"eight_later_pages_ms": 84.12934999999197,
|
||||
"elapsed_seconds": 20.011,
|
||||
"family_cpu_seconds": 19.733879,
|
||||
"family_peak_bytes": 1349349376,
|
||||
"import_ms": 10382.461203,
|
||||
"catalog_ms": 1436.8567430000003,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-after-javascript-4",
|
||||
"first_page_ms": 1864.8931889999985,
|
||||
"eight_later_pages_ms": 75.18833300000188,
|
||||
"elapsed_seconds": 23.023,
|
||||
"family_cpu_seconds": 22.22643,
|
||||
"family_peak_bytes": 1410211840,
|
||||
"import_ms": 10857.059245,
|
||||
"catalog_ms": 2247.621502,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-after-native-1",
|
||||
"first_page_ms": 216.87685000000056,
|
||||
"eight_later_pages_ms": 85.03580100000181,
|
||||
"elapsed_seconds": 17.253,
|
||||
"family_cpu_seconds": 17.002728,
|
||||
"family_peak_bytes": 1003585536,
|
||||
"import_ms": 10361.605091,
|
||||
"catalog_ms": 1289.5255869999992,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-after-native-2",
|
||||
"first_page_ms": 209.0475239999996,
|
||||
"eight_later_pages_ms": 89.01302899999973,
|
||||
"elapsed_seconds": 14.383,
|
||||
"family_cpu_seconds": 14.091764,
|
||||
"family_peak_bytes": 1031806976,
|
||||
"import_ms": 7696.751557,
|
||||
"catalog_ms": 1175.233292,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-after-native-3",
|
||||
"first_page_ms": 203.10824999999932,
|
||||
"eight_later_pages_ms": 65.1013469999998,
|
||||
"elapsed_seconds": 14.061,
|
||||
"family_cpu_seconds": 13.773228,
|
||||
"family_peak_bytes": 1098002432,
|
||||
"import_ms": 7967.397724,
|
||||
"catalog_ms": 1246.0458209999997,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-before-javascript-1",
|
||||
"first_page_ms": 1681.2564180000008,
|
||||
"eight_later_pages_ms": 10378.131188999992,
|
||||
"elapsed_seconds": 31.213,
|
||||
"family_cpu_seconds": 30.872607,
|
||||
"family_peak_bytes": 1618685952,
|
||||
"import_ms": 11051.493961999999,
|
||||
"catalog_ms": 1508.7439510000004,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-before-javascript-2",
|
||||
"first_page_ms": 2441.075147000003,
|
||||
"eight_later_pages_ms": 15534.410540000004,
|
||||
"elapsed_seconds": 40.856,
|
||||
"family_cpu_seconds": 40.485251,
|
||||
"family_peak_bytes": 1364238336,
|
||||
"import_ms": 15032.366213,
|
||||
"catalog_ms": 1319.6895839999997,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-before-javascript-4",
|
||||
"first_page_ms": 2479.6895360000017,
|
||||
"eight_later_pages_ms": 16864.908822999998,
|
||||
"elapsed_seconds": 39.629,
|
||||
"family_cpu_seconds": 39.267656,
|
||||
"family_peak_bytes": 1403023360,
|
||||
"import_ms": 12437.073842999998,
|
||||
"catalog_ms": 1212.8024130000003,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-before-native-1",
|
||||
"first_page_ms": 170.1303939999998,
|
||||
"eight_later_pages_ms": 469.8933869999928,
|
||||
"elapsed_seconds": 19.996,
|
||||
"family_cpu_seconds": 19.55106,
|
||||
"family_peak_bytes": 1063051264,
|
||||
"import_ms": 11242.171146,
|
||||
"catalog_ms": 1922.7818150000003,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-before-native-2",
|
||||
"first_page_ms": 183.56196199999977,
|
||||
"eight_later_pages_ms": 393.4744759999994,
|
||||
"elapsed_seconds": 15.26,
|
||||
"family_cpu_seconds": 15.008304,
|
||||
"family_peak_bytes": 1058492416,
|
||||
"import_ms": 8713.099498,
|
||||
"catalog_ms": 1272.265849,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-paired-before-native-3",
|
||||
"first_page_ms": 244.26586400000087,
|
||||
"eight_later_pages_ms": 517.9172599999911,
|
||||
"elapsed_seconds": 19.083,
|
||||
"family_cpu_seconds": 18.752553,
|
||||
"family_peak_bytes": 1109987328,
|
||||
"import_ms": 9948.476274999999,
|
||||
"catalog_ms": 1879.787958,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-after-small-1",
|
||||
"first_page_ms": 226.6032009999999,
|
||||
"eight_later_pages_ms": 57.69190299999718,
|
||||
"elapsed_seconds": 12.312,
|
||||
"family_cpu_seconds": 12.043895,
|
||||
"family_peak_bytes": 1056452608,
|
||||
"import_ms": 6572.925846000001,
|
||||
"catalog_ms": 1173.0129960000004,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-after-small-2",
|
||||
"first_page_ms": 556.2178050000002,
|
||||
"eight_later_pages_ms": 96.23635600000307,
|
||||
"elapsed_seconds": 16.246,
|
||||
"family_cpu_seconds": 16.068419,
|
||||
"family_peak_bytes": 1002749952,
|
||||
"import_ms": 9666.640497,
|
||||
"catalog_ms": 1821.1315130000003,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-after-small-4",
|
||||
"first_page_ms": 111.79372199999852,
|
||||
"eight_later_pages_ms": 62.58284999999887,
|
||||
"elapsed_seconds": 11.901,
|
||||
"family_cpu_seconds": 11.707784,
|
||||
"family_peak_bytes": 976646144,
|
||||
"import_ms": 6473.91882,
|
||||
"catalog_ms": 1203.5203539999998,
|
||||
"guard_sha256": "d2047721e50dd244900eb514d07025386e60908f77bb546bee5684baa1da2f19"
|
||||
},
|
||||
{
|
||||
"name": "topology-before-small-1",
|
||||
"first_page_ms": 242.57812199999898,
|
||||
"eight_later_pages_ms": 299.467095,
|
||||
"elapsed_seconds": 17.585,
|
||||
"family_cpu_seconds": 17.312434,
|
||||
"family_peak_bytes": 1066864640,
|
||||
"import_ms": 9102.074585,
|
||||
"catalog_ms": 1829.971928,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-before-small-2",
|
||||
"first_page_ms": 133.3346359999996,
|
||||
"eight_later_pages_ms": 122.92684500000178,
|
||||
"elapsed_seconds": 14.302,
|
||||
"family_cpu_seconds": 13.936122,
|
||||
"family_peak_bytes": 1062400000,
|
||||
"import_ms": 7254.973107,
|
||||
"catalog_ms": 1170.888121,
|
||||
"guard_sha256": "760d541c60f4c0538f9c015964196e3be7bba12316f6085903244354e11aa992"
|
||||
},
|
||||
{
|
||||
"name": "topology-before-small-4",
|
||||
"first_page_ms": 100.81227200000103,
|
||||
"eight_later_pages_ms": 76.37957900000038,
|
||||
"elapsed_seconds": 12.0,
|
||||
"family_cpu_seconds": 11.765305,
|
||||
"family_peak_bytes": 983314432,
|
||||
"import_ms": 6614.461618999999,
|
||||
"catalog_ms": 1138.6469730000003,
|
||||
"guard_sha256": "d2047721e50dd244900eb514d07025386e60908f77bb546bee5684baa1da2f19"
|
||||
}
|
||||
],
|
||||
"token_estimates": {
|
||||
"tokenizer": "tiktoken",
|
||||
"version": "0.12.0",
|
||||
"encoding": "cl100k_base",
|
||||
"kind": "offline raw transport serialization estimate; one complete catalog plus each call name/arguments/reply exactly once; excludes initialization/framing, prompts, client rendering/discovery/replay, continuations and actual model usage; not an agent-token measurement",
|
||||
"runs": [
|
||||
{
|
||||
"run": "topology-paired-before-native-1",
|
||||
"catalog_tokens_estimated": 805215,
|
||||
"named_call_arguments_tokens_estimated": 1200,
|
||||
"reply_tokens_estimated": 43321,
|
||||
"total_tokens_estimated": 849736,
|
||||
"calls": 25
|
||||
},
|
||||
{
|
||||
"run": "topology-paired-after-native-1",
|
||||
"catalog_tokens_estimated": 805215,
|
||||
"named_call_arguments_tokens_estimated": 1200,
|
||||
"reply_tokens_estimated": 43321,
|
||||
"total_tokens_estimated": 849736,
|
||||
"calls": 25
|
||||
},
|
||||
{
|
||||
"run": "topology-paired-before-javascript-1",
|
||||
"catalog_tokens_estimated": 805215,
|
||||
"named_call_arguments_tokens_estimated": 1354,
|
||||
"reply_tokens_estimated": 49815,
|
||||
"total_tokens_estimated": 856384,
|
||||
"calls": 28
|
||||
},
|
||||
{
|
||||
"run": "topology-paired-after-javascript-1",
|
||||
"catalog_tokens_estimated": 805215,
|
||||
"named_call_arguments_tokens_estimated": 1354,
|
||||
"reply_tokens_estimated": 49815,
|
||||
"total_tokens_estimated": 856384,
|
||||
"calls": 28
|
||||
},
|
||||
{
|
||||
"run": "topology-before-small-1",
|
||||
"catalog_tokens_estimated": 805215,
|
||||
"named_call_arguments_tokens_estimated": 1118,
|
||||
"reply_tokens_estimated": 33970,
|
||||
"total_tokens_estimated": 840303,
|
||||
"calls": 19
|
||||
},
|
||||
{
|
||||
"run": "topology-after-small-1",
|
||||
"catalog_tokens_estimated": 805215,
|
||||
"named_call_arguments_tokens_estimated": 1118,
|
||||
"reply_tokens_estimated": 33970,
|
||||
"total_tokens_estimated": 840303,
|
||||
"calls": 19
|
||||
}
|
||||
]
|
||||
},
|
||||
"mixed_verification": [
|
||||
{
|
||||
"scenario": "native",
|
||||
"heads_checked": 64,
|
||||
"pages": 32,
|
||||
"revised_unknown_id": "unk_057c8f2dbf30ec98851a66ee054366bc6e2e168874581e6f99cb80673d0ff9b8",
|
||||
"revision_checked": 2,
|
||||
"stale_update_rejected": true,
|
||||
"tampered_import_rejected_atomically": true,
|
||||
"unknowns_after_close": 0,
|
||||
"old_reference_refused": true
|
||||
},
|
||||
{
|
||||
"scenario": "javascript",
|
||||
"heads_checked": 64,
|
||||
"pages": 32,
|
||||
"revised_unknown_id": "unk_057c8f2dbf30ec98851a66ee054366bc6e2e168874581e6f99cb80673d0ff9b8",
|
||||
"revision_checked": 2,
|
||||
"stale_update_rejected": true,
|
||||
"tampered_import_rejected_atomically": true,
|
||||
"unknowns_after_close": 0,
|
||||
"old_reference_refused": true
|
||||
}
|
||||
],
|
||||
"cli_parity": [
|
||||
{
|
||||
"scenario": "native",
|
||||
"exact_json_equal": true,
|
||||
"exact_bytes_equal": true,
|
||||
"bytes": 7316,
|
||||
"sha256_before": "eada9068db3b499da332b2c5226fdd3a48909ebcaa290fd72166325e2f664fca",
|
||||
"sha256_after": "eada9068db3b499da332b2c5226fdd3a48909ebcaa290fd72166325e2f664fca",
|
||||
"evidence_id": "ev_521a58e81427bb4b2c01b6a291bdfd7698068ee0eba371696daa9cc5ab32f3f0",
|
||||
"evidence_links": [
|
||||
"ev_14b4582084e425b597bea6246f0df94ab4856ac3ab9502c96551147210ca9c8b"
|
||||
]
|
||||
},
|
||||
{
|
||||
"scenario": "javascript",
|
||||
"exact_json_equal": true,
|
||||
"exact_bytes_equal": true,
|
||||
"bytes": 8876,
|
||||
"sha256_before": "1dc023e5ece8f9c8cea736cb0db4d4f60c9418f1573e1e46ff0ac55a4273cb5a",
|
||||
"sha256_after": "1dc023e5ece8f9c8cea736cb0db4d4f60c9418f1573e1e46ff0ac55a4273cb5a",
|
||||
"evidence_id": "ev_97cac46d5139f6c48c3142798087814dcf36a37a9c99f7253668576bd3ae282a",
|
||||
"evidence_links": [
|
||||
"ev_dd2ffe5ab2faa6e07e422886f64a87f99e35a0a1dce1acb5351b7a81cb2c9a11"
|
||||
]
|
||||
},
|
||||
{
|
||||
"scenario": "small",
|
||||
"exact_json_equal": true,
|
||||
"exact_bytes_equal": true,
|
||||
"bytes": 7181,
|
||||
"sha256_before": "00dd070690e7b113e0cad73f9f7bcda6f42c515f8fdcc5173e8815bd3748b084",
|
||||
"sha256_after": "00dd070690e7b113e0cad73f9f7bcda6f42c515f8fdcc5173e8815bd3748b084",
|
||||
"evidence_id": "ev_3dcbcc5b2002f1e6caf7df005e3fcfd06a3afe14ccd72fff924c8e76d8498914",
|
||||
"evidence_links": [
|
||||
"ev_775d6cc6d0de96402b6e57a4258a0b78c337cfb1b7760aa22c39c3ee450c7dff"
|
||||
]
|
||||
}
|
||||
],
|
||||
"stopped_source_validation": {
|
||||
"reason": "Started during initial before-javascript-3; stopped only the new owned validation scope, then retried serially. Baseline repetition3 retained but excluded from uncontended exploratory comparisons; all final cohort runs retained.",
|
||||
"status": "incomplete, shell reported exit241; no payload-completion receipt",
|
||||
"elapsed_seconds": 17.272,
|
||||
"family_cpu_seconds_last_observed": 12.204013,
|
||||
"peak_cgroup_bytes_sampled": 589766656,
|
||||
"final_scope_counters": null
|
||||
},
|
||||
"exploratory_costs": {
|
||||
"completed_tasks": 21,
|
||||
"elapsed_seconds_total": 649.262,
|
||||
"family_cpu_seconds_total": 599.762599,
|
||||
"family_peak_bytes_max": 1741795328,
|
||||
"comparison_status": "Earlier candidate and exploratory inputs; excluded from final comparison, raw individual reports retained in private audit cache.",
|
||||
"contended_baseline_task": {
|
||||
"name": "before-javascript-3",
|
||||
"revision": "2c4338db1fdb7773e05376301c558cff2a0b4b80",
|
||||
"status": "passed",
|
||||
"elapsed_seconds": 100.722,
|
||||
"family_cpu_seconds": 82.656181,
|
||||
"family_peak_bytes": 1290514432,
|
||||
"excluded_from_final_comparison": true
|
||||
}
|
||||
},
|
||||
"runtime_entry_sha256": {
|
||||
"before": "1586cea18f4d4702c949502cfc143a7dfd2c28414779e190ce8a42fedf90bd74",
|
||||
"after": "e05c70bed21c680445a5c6e314a16cce6f22f3909f93388bf602338062e63d1a"
|
||||
},
|
||||
"inputs": {
|
||||
"native": {
|
||||
"bundle_sha256": "180c6fe6d3706d9272c8bb33503c2dca6f145d5061e102e3457dd5f2d754b5c9",
|
||||
"bundle_bytes": 1745224,
|
||||
"source_evidence_id": "ev_14b4582084e425b597bea6246f0df94ab4856ac3ab9502c96551147210ca9c8b"
|
||||
},
|
||||
"javascript": {
|
||||
"bundle_sha256": "747ff602093a24d4876e3eb47da3f7ef68716a10ac7c09938f43c3be5c33e31d",
|
||||
"bundle_bytes": 21058430,
|
||||
"source_evidence_id": "ev_dd2ffe5ab2faa6e07e422886f64a87f99e35a0a1dce1acb5351b7a81cb2c9a11"
|
||||
},
|
||||
"small": {
|
||||
"bundle_sha256": "44ca561c2455706897d2ae8f51f45693e7276badd5e8454ec31d8c1b0cfb77eb",
|
||||
"bundle_bytes": 73051,
|
||||
"source_evidence_id": "ev_775d6cc6d0de96402b6e57a4258a0b78c337cfb1b7760aa22c39c3ee450c7dff"
|
||||
}
|
||||
},
|
||||
"validation": {
|
||||
"source": {
|
||||
"executed": "130 source tests across5files; check:fast; build:cached",
|
||||
"elapsed_seconds": 334.264,
|
||||
"family_cpu_seconds": 326.441674,
|
||||
"family_peak_bytes": 4487696384,
|
||||
"payload_exit_code": 0,
|
||||
"payload_cleanup_verified": true,
|
||||
"oom_events_at_payload_exit": 0
|
||||
},
|
||||
"compiled_boundary": {
|
||||
"executed": "36 CLI/MCP tests across4files",
|
||||
"elapsed_seconds": 93.343,
|
||||
"family_cpu_seconds": 91.926502,
|
||||
"family_peak_bytes": 2190270464,
|
||||
"payload_exit_code": 0,
|
||||
"payload_cleanup_verified": true,
|
||||
"oom_events_at_payload_exit": 0
|
||||
},
|
||||
"full_checks": {
|
||||
"source": "3930a640fb8387cbd3079eee13fb90f82d1881dc",
|
||||
"executed": "npm run check; npm run docs:check (five README consistency tests)",
|
||||
"elapsed_seconds": 45.552,
|
||||
"family_cpu_seconds": 43.309041,
|
||||
"family_peak_bytes": 1127768064,
|
||||
"payload_exit_code": 0,
|
||||
"payload_cleanup_verified": true,
|
||||
"oom_events_at_payload_exit": 0
|
||||
},
|
||||
"integration_current": {
|
||||
"source": "3930a640fb8387cbd3079eee13fb90f82d1881dc",
|
||||
"executed": "npm run test:focused -- three latest-main integration files; default test deadlines; 27 passed",
|
||||
"elapsed_seconds": 62.437,
|
||||
"family_cpu_seconds": 61.506209,
|
||||
"family_peak_bytes": 1846538240,
|
||||
"payload_exit_code": 0,
|
||||
"payload_cleanup_verified": true,
|
||||
"oom_events_at_payload_exit": 0
|
||||
}
|
||||
},
|
||||
"comparison_probe": {
|
||||
"scope": "Candidate-only internal JSON-owner regression, not baseline public-provider reproduction. Reflect.ownKeys instrumentation aborts at10000 inspections rather than consuming exponential CPU.",
|
||||
"objects_per_graph": 27,
|
||||
"before_candidate": "e30a94a4de6a3c4ec7c577c1551d2b448d82081e",
|
||||
"before_inspections": 10001,
|
||||
"before_status": "probe inspection budget exhausted",
|
||||
"after_candidate": "646312dad75e2cb2becb0dd4139b99a1c04d6bcc",
|
||||
"after_inspections": 54,
|
||||
"after_borrowed": true
|
||||
},
|
||||
"main_integration": {
|
||||
"main_commit": "bb212ffecb47e129cc569b9867b2b6e7c9b65685",
|
||||
"merge_commit": "71e0b2739a9cf9903e1cfa63dc5a5b576040fea7",
|
||||
"production_files_unchanged_from_measured_candidate": true,
|
||||
"scope": "Historical integration at71e0b2739 with production source646312dad; all attempts retained. Current source-specific checks are listed separately.",
|
||||
"attempts": [
|
||||
{
|
||||
"case": "initial_feature",
|
||||
"entrypoint": "npm run test:focused",
|
||||
"test_timeout_override_ms": null,
|
||||
"tests_passed": 26,
|
||||
"tests_failed": 1,
|
||||
"tests_skipped": 0,
|
||||
"payload_exit_code": 1,
|
||||
"elapsed_seconds": 139.805,
|
||||
"family_cpu_seconds": 125.252141,
|
||||
"family_peak_bytes": 1994223616,
|
||||
"payload_cleanup_verified": true,
|
||||
"oom_events_at_payload_exit": 0
|
||||
},
|
||||
{
|
||||
"case": "isolated_main",
|
||||
"entrypoint": "vitest run; one named test",
|
||||
"test_timeout_override_ms": null,
|
||||
"tests_passed": 1,
|
||||
"tests_failed": 0,
|
||||
"tests_skipped": 10,
|
||||
"payload_exit_code": 0,
|
||||
"elapsed_seconds": 25.96,
|
||||
"family_cpu_seconds": 25.502365,
|
||||
"family_peak_bytes": 1221394432,
|
||||
"payload_cleanup_verified": true,
|
||||
"oom_events_at_payload_exit": 0
|
||||
},
|
||||
{
|
||||
"case": "direct_main",
|
||||
"entrypoint": "vitest run; three files",
|
||||
"test_timeout_override_ms": null,
|
||||
"tests_passed": 27,
|
||||
"tests_failed": 0,
|
||||
"tests_skipped": 0,
|
||||
"payload_exit_code": 0,
|
||||
"elapsed_seconds": 68.868,
|
||||
"family_cpu_seconds": 68.020337,
|
||||
"family_peak_bytes": 1605603328,
|
||||
"payload_cleanup_verified": true,
|
||||
"oom_events_at_payload_exit": 0
|
||||
},
|
||||
{
|
||||
"case": "extended_feature",
|
||||
"entrypoint": "vitest run; three files",
|
||||
"test_timeout_override_ms": 120000,
|
||||
"tests_passed": 27,
|
||||
"tests_failed": 0,
|
||||
"tests_skipped": 0,
|
||||
"payload_exit_code": 0,
|
||||
"elapsed_seconds": 75.866,
|
||||
"family_cpu_seconds": 75.193788,
|
||||
"family_peak_bytes": 1937674240,
|
||||
"payload_cleanup_verified": true,
|
||||
"oom_events_at_payload_exit": 0
|
||||
},
|
||||
{
|
||||
"case": "exact_main",
|
||||
"entrypoint": "npm run test:focused",
|
||||
"test_timeout_override_ms": null,
|
||||
"tests_passed": 27,
|
||||
"tests_failed": 0,
|
||||
"tests_skipped": 0,
|
||||
"payload_exit_code": 0,
|
||||
"elapsed_seconds": 66.443,
|
||||
"family_cpu_seconds": 65.304325,
|
||||
"family_peak_bytes": 2055409664,
|
||||
"payload_cleanup_verified": true,
|
||||
"oom_events_at_payload_exit": 0
|
||||
},
|
||||
{
|
||||
"case": "exact_feature",
|
||||
"entrypoint": "npm run test:focused",
|
||||
"test_timeout_override_ms": null,
|
||||
"tests_passed": 27,
|
||||
"tests_failed": 0,
|
||||
"tests_skipped": 0,
|
||||
"payload_exit_code": 0,
|
||||
"elapsed_seconds": 66.878,
|
||||
"family_cpu_seconds": 66.012242,
|
||||
"family_peak_bytes": 1907380224,
|
||||
"payload_cleanup_verified": true,
|
||||
"oom_events_at_payload_exit": 0
|
||||
}
|
||||
],
|
||||
"measured_production_commit": "646312dad75e2cb2becb0dd4139b99a1c04d6bcc"
|
||||
},
|
||||
"small_cache_diagnostic": {
|
||||
"scope": "Additional real public MCP run with return-preserving WeakSet method instrumentation. Timings excluded from before/after comparison. Owner identified from captured add stack; raw trace retained privately.",
|
||||
"format_owner": "safeParseOwnedBinaryLayout",
|
||||
"source_objects": 1,
|
||||
"successful_admissions": 1,
|
||||
"misses": 1,
|
||||
"hits": 8,
|
||||
"queries": 9,
|
||||
"original_records_preserved": true,
|
||||
"closed_session_cleared": true,
|
||||
"small_workflow_latency_regression_retained": true,
|
||||
"measured_production_commit": "646312dad75e2cb2becb0dd4139b99a1c04d6bcc"
|
||||
},
|
||||
"previous_checkpoints": [
|
||||
{
|
||||
"production_commit": "e30a94a4de6a3c4ec7c577c1551d2b448d82081e",
|
||||
"report_commit": "3f271efdef8d71020bdcfd7b2d3d9817bf0a6ad8",
|
||||
"report_url": "https://github.com/morluto/rea/blob/3f271efdef8d71020bdcfd7b2d3d9817bf0a6ad8/experiments/analysis-efficiency/immutable-parent-validation.json",
|
||||
"completed_paired_tasks": 18,
|
||||
"elapsed_seconds_total": 498.234,
|
||||
"family_cpu_seconds_total": 477.513448,
|
||||
"family_peak_bytes_max": 1519980544,
|
||||
"scope": "Historical candidate before shared-node comparison memoization; every result retained, including slow npm repetition3. Not the final comparison."
|
||||
},
|
||||
{
|
||||
"production_commit": "646312dad75e2cb2becb0dd4139b99a1c04d6bcc",
|
||||
"report_commit": "fc55be1e05555c6b295209ef5f354617f663b8f8",
|
||||
"report_url": "https://github.com/morluto/rea/blob/fc55be1e05555c6b295209ef5f354617f663b8f8/experiments/analysis-efficiency/immutable-parent-validation.json",
|
||||
"completed_paired_tasks": 18,
|
||||
"elapsed_seconds_total": 483.511,
|
||||
"family_cpu_seconds_total": 458.990461,
|
||||
"family_peak_bytes_max": 1598119936,
|
||||
"scope": "Historical candidate before shared-node comparison memoization; every result retained, including slow npm repetition3. Not the final comparison."
|
||||
}
|
||||
],
|
||||
"review_normalization_fix": {
|
||||
"comment": "https://github.com/morluto/rea/pull/4962#discussion_r4305886491",
|
||||
"before_source": "646312dad75e2cb2becb0dd4139b99a1c04d6bcc",
|
||||
"before_pr_head": "fc55be1e05555c6b295209ef5f354617f663b8f8",
|
||||
"expected_failing_regressions": 4,
|
||||
"after_source": "3930a640fb8387cbd3079eee13fb90f82d1881dc",
|
||||
"source_tests_passed": 130,
|
||||
"structural_fix": "Two-way node identity mapping visits preserved descendants; already-owned JavaScript graph path keeps parsed metadata and avoids whole-graph comparison. Only the expensive imported fallback admits the full-result cache.",
|
||||
"red_probe_cost": {
|
||||
"elapsed_seconds": 5.332,
|
||||
"family_cpu_seconds": 4.412336,
|
||||
"family_peak_bytes": 438231040,
|
||||
"payload_exit_code": 1,
|
||||
"payload_cleanup_verified": true,
|
||||
"expected_failures": 4
|
||||
},
|
||||
"current_dag_probe": {
|
||||
"source": "3930a640fb8387cbd3079eee13fb90f82d1881dc",
|
||||
"objects_per_graph": 27,
|
||||
"inspections": 54,
|
||||
"borrowed": true
|
||||
}
|
||||
},
|
||||
"resource_stops": [
|
||||
{
|
||||
"task": "topology-paired-after-javascript-3",
|
||||
"status": "incomplete",
|
||||
"reason": "host_available_memory_early_stop",
|
||||
"elapsed_seconds": 15.048,
|
||||
"family_cpu_seconds_last_observed": 14.048702,
|
||||
"peak_cgroup_bytes_sampled": 947695616,
|
||||
"payload_completion_counters": null,
|
||||
"owned_scope_after_stop": "inactive/dead",
|
||||
"replacement": "fresh JavaScript pair4, after then before; final pairs1,2,4"
|
||||
},
|
||||
{
|
||||
"task": "topology-before-small-3",
|
||||
"status": "incomplete",
|
||||
"reason": "host_available_memory_early_stop",
|
||||
"elapsed_seconds": 11.83,
|
||||
"family_cpu_seconds_last_observed": 11.535779,
|
||||
"peak_cgroup_bytes_sampled": 947818496,
|
||||
"payload_completion_counters": null,
|
||||
"replacement": "fresh small pair4 after memory recovered; final small pairs1,2,4"
|
||||
}
|
||||
],
|
||||
"unpaired_completed_attempts": [
|
||||
{
|
||||
"task": "topology-paired-before-javascript-3",
|
||||
"status": "passed",
|
||||
"original_records_preserved": true,
|
||||
"elapsed_seconds": 38.862,
|
||||
"family_cpu_seconds": 38.570479,
|
||||
"family_peak_bytes": 1390338048,
|
||||
"reason_not_in_paired_summary": "Its paired candidate was resource-stopped; kept separately, with both sides rerun as pair4."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
# Reuse successful immutable parent validation
|
||||
|
||||
This checkpoint addresses [#4810](https://github.com/morluto/rea/issues/4810),
|
||||
found in the [retained-investigation audit](retained-investigation-workflows.md).
|
||||
It changes the shared format interpretation used by CLI and MCP views, without
|
||||
adding a tool, changing a schema, projecting away source data, or rerunning a
|
||||
provider.
|
||||
|
||||
## Question, ownership, and compatibility
|
||||
|
||||
The task resumes authentic exported Evidence in a fresh production stdio MCP
|
||||
server, discovers its source, reads six original symbols or modules in two-row
|
||||
pages, repeats that traversal three times, queries the existing Unknowns,
|
||||
exports the investigation, and closes it. The original source records must
|
||||
survive the round trip exactly. Pages are compared to original retained facts.
|
||||
Independent ELF byte decoding checks symbol names, values, and ranges; JavaScript
|
||||
checks cover selected source bytes and the `lib/cli.js` module identity, rather
|
||||
than independently proving the entire application inventory or semantic graph.
|
||||
|
||||
Imported JavaScript graphs lack the producer's realm-private format proofs.
|
||||
Previously each view repeated full schema, graph/hash, and cross-graph binding
|
||||
validation. Native layout views likewise parsed the entire layout for every
|
||||
page. An immutable Evidence root alone was insufficient: its selected normalized
|
||||
result did not carry the JSON owner's private sealing proof.
|
||||
|
||||
The JSON owner now transfers that proof to a selected own member without another
|
||||
walk or per-node registration. Native layouts and imported JavaScript results
|
||||
record a **successful full
|
||||
validation of the exact immutable source** in their owning WeakSets. Already
|
||||
owned JavaScript graphs keep the existing metadata normalization and binding
|
||||
checks, returning parsed metadata with their original graph references. That
|
||||
path avoids rewalking a large owned graph just to borrow its wrapper. There is
|
||||
no second
|
||||
retained parsed graph, strong cache, cross-process cache, or live-state cache.
|
||||
The schema implementation is fixed for the process; a different source object
|
||||
must validate independently. Weak entries do not keep Evidence alive, and session
|
||||
lookup continues to gate access after close.
|
||||
|
||||
Cache admission follows full validation and requires that parsing preserved the
|
||||
source representation. A shared iterative JSON comparison checks own data keys,
|
||||
scalar values, prototypes, and enumerability without calling accessors. Object
|
||||
member order is immaterial; array order remains exact. Two-way identity maps
|
||||
visit preserved descendants once and reject source-node splits or merges. Both
|
||||
comparison maps are temporary and discarded on return. Parser defaults,
|
||||
hidden-field removal, or other normalization retain the parser's
|
||||
result and do not cache the source. Writable/configurable descriptor differences
|
||||
are immaterial: the source is sealed while the temporary parse is mutable.
|
||||
The JSON sealing boundary rejects proxies before reflection, so arbitrary
|
||||
mutable proxy behavior cannot acquire an immutable proof. Mutable, merely
|
||||
Object.freeze'd, failed, and abandoned inputs do not qualify for successful
|
||||
format reuse.
|
||||
|
||||
No Evidence identity, source linkage, coverage, unknown, omission, error, or
|
||||
caller-selected page changes are intended. Complete source Evidence remains
|
||||
accessible through the existing contracts. The first imported view still pays for full
|
||||
validation and the representation comparison; a one-off CLI invocation cannot
|
||||
benefit from a cache in a previous process.
|
||||
|
||||
## Inputs and measurement scope
|
||||
|
||||
Baseline: `2c4338db1fdb7773e05376301c558cff2a0b4b80` (REA 6.6.0).
|
||||
Node 24.18.0, npm 11.16.0, Linux x86-64 (AMD EPYC 7B12, kernel 6.1.0-53-cloud).
|
||||
The ELF producer reports pwntools 4.15.0, pyelftools 0.33, and Unicorn 2.1.2.
|
||||
The baseline's built runtime was preserved
|
||||
before production changes. Candidate and baseline use the same locked packages.
|
||||
|
||||
- SQLite `/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6`: SHA-256
|
||||
`2e6eef9a727f081f0d453b4e5e6cbd8b9ef8b6f86cbf7681cbad444d3b0b55c8`;
|
||||
26 sections, 1,479 symbols, 2,963 relocations. Original history: 1,745,224 bytes.
|
||||
- Shipped npm 11.16.0: inventory digest
|
||||
`f8e576a7d162542c616168b25a95a5f62b9accf29467cc37e149905ad4b8e603`;
|
||||
five explicitly selected paths (`package.json`, `bin/npm-cli.js`, `lib/cli.js`,
|
||||
`lib/cli/entry.js`, `lib/commands/install.js`), 2,822 application nodes,
|
||||
1,796 semantic nodes, 13,682 semantic unknowns. The 1,416 unselected files stay
|
||||
unknown. Original history: 21,058,430 bytes.
|
||||
- Small case `/usr/bin/true`: SHA-256
|
||||
`c79bf44242829108e323378531f4ac839513ca1fba45efd6583643526e1e9fd2`;
|
||||
31 sections, 53 symbols, 66 relocations. Produced through the real production
|
||||
`inspect_binary_layout` MCP operation. An independent raw ELF reader checked
|
||||
every symbol name, value, and source range plus section/relocation counts.
|
||||
|
||||
The two larger histories were produced and independently checked in the linked
|
||||
previous audit. They are historical Evidence, not a claim of fresh provider
|
||||
analysis during a resumed view. All tasks here are static; no target executable
|
||||
is run.
|
||||
|
||||
Each completed comparison includes startup, the complete 207-tool catalog,
|
||||
import, queries, oracle checks, export, close, receipt writing, and SDK cleanup.
|
||||
RPC timings exclude subsequent client-side oracle/receipt work. Fresh servers
|
||||
separate first-view validation from later pages; OS file caches are uncontrolled.
|
||||
Runs alternate baseline/candidate order. One CPU, 8 GiB hard family memory,
|
||||
7 GiB early stop, no swap, and a 192-task ceiling bound each serial task.
|
||||
Resource counters are observed at payload exit while the observer remains alive;
|
||||
final whole-scope counters are unavailable. Kernel cgroup peaks include charged
|
||||
file cache and descendants, rather than just the server's JS heap.
|
||||
|
||||
## Results and executed validation
|
||||
|
||||
Candidate production code: `3930a640fb8387cbd3079eee13fb90f82d1881dc`.
|
||||
The [curated receipt](immutable-parent-validation.json) retains every final run,
|
||||
offline token estimates, compatibility checks, and costs of previous attempts.
|
||||
Values below are medians with min–max ranges across three fresh alternating pairs.
|
||||
|
||||
| Source | First page, before → after (ms) | Eight later pages, before → after (ms) |
|
||||
| ---------- | ----------------------------------------------------- | -------------------------------------------------- |
|
||||
| SQLite ELF | 183.56 [170.13–244.27] → 209.05 [203.11–216.88] | 469.89 [393.47–517.92] → 85.04 [65.10–89.01] |
|
||||
| npm | 2441.08 [1681.26–2479.69] → 1978.73 [1864.89–2720.54] | 15534.41 [10378.13–16864.91] → 82.69 [75.19–84.13] |
|
||||
| Small ELF | 133.33 [100.81–242.58] → 226.60 [111.79–556.22] | 122.93 [76.38–299.47] → 62.58 [57.69–96.24] |
|
||||
|
||||
| Source / runtime | Whole elapsed (s) | Family CPU (s) | Kernel family peak (MiB) |
|
||||
| ------------------- | ------------------- | ------------------- | ------------------------ |
|
||||
| SQLite ELF / before | 19.08 [15.26–20.00] | 18.75 [15.01–19.55] | 1013.8 [1009.5–1058.6] |
|
||||
| SQLite ELF / after | 14.38 [14.06–17.25] | 14.09 [13.77–17.00] | 984.0 [957.1–1047.1] |
|
||||
| npm / before | 39.63 [31.21–40.86] | 39.27 [30.87–40.49] | 1338.0 [1301.0–1543.7] |
|
||||
| npm / after | 20.71 [20.01–23.02] | 20.50 [19.73–22.23] | 1286.8 [1250.6–1344.9] |
|
||||
| Small ELF / before | 14.30 [12.00–17.59] | 13.94 [11.77–17.31] | 1013.2 [937.8–1017.4] |
|
||||
| Small ELF / after | 12.31 [11.90–16.25] | 12.04 [11.71–16.07] | 956.3 [931.4–1007.5] |
|
||||
|
||||
| Source | Paired warm-read ratio, before / after | Paired first-view change, after − before (ms) |
|
||||
| ---------- | -------------------------------------- | --------------------------------------------- |
|
||||
| SQLite ELF | 5.53× [4.42–7.96] | +25.49 [-41.16–+46.75] |
|
||||
| npm | 184.65× [125.50–224.30] | -462.34 [-614.80–+1039.29] |
|
||||
| Small ELF | 1.28× [1.22–5.19] | +10.98 [-15.97–+422.88] |
|
||||
|
||||
Each pair matches the before/after runs with the same repetition number.
|
||||
The paired effect can differ from the ratio of marginal medians when the host
|
||||
speed changes. Positive first-view changes mean higher latency.
|
||||
|
||||
The repeated-read benefit is measured per source and pair above. First-view
|
||||
validation
|
||||
also performs the representation comparison and can be more expensive. Whole
|
||||
task, CPU and memory figures above are measured SDK workflow costs, not a
|
||||
general provider-analysis or agent-investigation speedup. The previous e30
|
||||
checkpoint had a slow npm repetition on both sides, including a 65-second
|
||||
candidate task; its cause was not profiled. The original checkpoint and every
|
||||
fresh run remain retained. The previous `646312dad` checkpoint also had slower
|
||||
small-case warm reads in two of three pairs (135.54 → 373.14 ms marginal
|
||||
medians). Those negative results remain available in its linked historical
|
||||
receipt; the current cohort improves all three small pairs, without establishing
|
||||
a general small-input or whole-task benefit. No model-token saving, general memory improvement,
|
||||
or complex-software investigation advantage is established.
|
||||
|
||||
All 18 fresh workflow runs preserve original records and canonical answer
|
||||
digests. Every corresponding query's structured result and complete catalog
|
||||
is equal. The three public CLI source pairs are also byte-for-byte identical.
|
||||
The two additional mixed-history tasks preserve all 97 original records, check
|
||||
64 follow-up heads over 32 pages and revision two, reject stale updates and
|
||||
tampered imports atomically, clear Unknowns on close, and refuse old references.
|
||||
|
||||
Offline `tiktoken` 0.12.0 / `cl100k_base` estimates for one pass over the complete
|
||||
catalog, named-call arguments and replies are **unchanged** within each pair:
|
||||
849,736 native, 856,384 JavaScript, and 840,303 small-case tokens. The catalog
|
||||
contributes 805,215 tokens alone.
|
||||
These are raw transport serialization estimates, not actual model context or
|
||||
usage; initialization/framing, prompts, client discovery, rendering/replay,
|
||||
and continuations are unmeasured. Metadata paths differ from the historical
|
||||
checkpoint; compare equivalent runs within each checkpoint.
|
||||
|
||||
Executed checks at the final production code:
|
||||
|
||||
- `npm run check` and `npm run docs:check`: pass, including five README
|
||||
consistency tests.
|
||||
- Three latest-main integration files through `npm run test:focused`, using
|
||||
default deadlines: 27 tests pass at the current production source.
|
||||
- 130 source tests across immutable JSON, native/JavaScript format validation,
|
||||
and the shared application view service; `npm run check:fast` and
|
||||
`npm run build:cached` pass.
|
||||
- Four compiled CLI/MCP files (`evidenceBundleInline`, `serverUnknownsMcp`,
|
||||
`analysisViewMcp`, and `analysisViewCli`): 36 tests pass.
|
||||
- Real public stdio MCP: 18 fresh paired tasks and two additional mixed tasks;
|
||||
public CLI: six invocations, with three exact output pairs. These corpus
|
||||
workflows and four-file boundary checks have exit zero, verified payload
|
||||
cleanup and zero observed payload-exit OOM events.
|
||||
|
||||
Internal and automated review reproduced **candidate-only** defects before integration.
|
||||
Generic deep equality ignored hidden fields removed by parsing; the JSON owner
|
||||
comparison now preserves that normalization. Review also reproduced aliases crossing
|
||||
preserved subtrees and cloned metadata; four regressions failed against the prior
|
||||
source. The comparer now visits preserved descendants with two-way identity mapping.
|
||||
Already-owned JavaScript graphs keep their parsed-metadata fast path, so the generic
|
||||
comparison and full-result cache only apply to the expensive imported fallback. Its
|
||||
initial iterative traversal
|
||||
also repeatedly expanded shared nodes: a valid 27-object graph exhausted an
|
||||
instrumented 10,000-inspection budget. Tracking source-to-parsed node pairs
|
||||
reduces that same compiled probe to 54 inspections and successful borrowing.
|
||||
A parser that splits or merges source nodes keeps its parsed result rather than
|
||||
expanding repeated paths. Deterministic regressions enforce the inspection
|
||||
budget instead of relying on timing or letting a regression consume exponential
|
||||
CPU. These are internal JSON-owner probes, not claimed baseline public-provider
|
||||
reproductions.
|
||||
|
||||
The initial exploratory cohort and historical e30 paired checkpoint are not
|
||||
the final comparison. One earlier source-check attempt was mistakenly started
|
||||
during an initial baseline run; the newly started owned scope was stopped after
|
||||
17.27 s and that check remains incomplete (exit 241). Observed counters, the
|
||||
contended baseline, subsequent successful retries, and prior checkpoints remain
|
||||
retained; no OOM or terminal-counter claim is inferred from that stopped task.
|
||||
Two later tasks stopped because the host-available-memory guard reported its
|
||||
early-stop condition: candidate JavaScript repetition three and baseline small
|
||||
repetition three. Both incomplete attempts and the unmatched completed
|
||||
JavaScript baseline remain separate. After memory recovered, both sides were
|
||||
rerun as fresh pair four; final JavaScript and small samples use pairs 1, 2, 4.
|
||||
Neither interrupted task is a completed comparison. The guard later gained
|
||||
read-only host-memory/cgroup receipt fields; all resource thresholds and its
|
||||
observer/stop logic remained unchanged.
|
||||
|
||||
Real Ghidra, Hopper, JEB, fresh JavaScript producer performance, and actual
|
||||
agent trials are outside this checkpoint. Private WeakSet metadata/heap retention
|
||||
was not measured independently; the absence of a strong retained parsed graph
|
||||
is source-derived. The comparison maps exist only during the first
|
||||
interpretation; measured family
|
||||
peaks include their temporary cost.
|
||||
|
||||
## Latest-main integration and an unexplained timeout
|
||||
|
||||
Merge `71e0b2739` brings in main `bb212ffec`'s three test/config/guide fixes.
|
||||
The current source passes all 27 tests in those three integration files through
|
||||
`npm run test:focused` with the default deadlines. The earlier `646312dad`
|
||||
checkpoint initially passed 26 and timed out the live identity/catalog test at
|
||||
30 seconds. Matching official-command controls subsequently passed all 27 on
|
||||
both unmodified main and that candidate. Direct/extended-deadline diagnostics
|
||||
are recorded separately. Their passing results do not establish the cause of
|
||||
the original timeout, which remains preserved and unexplained. No test-policy
|
||||
or product file was changed to obtain a pass.
|
||||
|
||||
## Reproduction and retained evidence
|
||||
|
||||
Use `retained-investigation.mjs` from this directory with an authentic exported
|
||||
bundle and its original source Evidence ID. Build both selected runtime revisions
|
||||
with the pinned toolchain, preserving a separate baseline. For each runtime run:
|
||||
|
||||
```sh
|
||||
python3 experiments/analysis-efficiency/resource-run.py --observe-payload-exit -- \
|
||||
node experiments/analysis-efficiency/retained-investigation.mjs \
|
||||
--runtime-root /absolute/runtime --revision REVISION \
|
||||
--bundle /absolute/history.json --evidence-id ORIGINAL_EVIDENCE_ID \
|
||||
--scenario native --delivery focused --output /absolute/new-receipt-directory
|
||||
```
|
||||
|
||||
Use `--scenario javascript` for the npm history, and `--verify-mixed` with an
|
||||
actual mixed investigation. Reuse the same bytes and source identity on both
|
||||
sides. Do not regenerate parent identities or synthesize bundle records.
|
||||
The checked-in wrapper defaults to a 4 GiB hard / 3 GiB early stop. This
|
||||
checkpoint used a private resource-only copy with 8 GiB / 7 GiB thresholds,
|
||||
a 16 GiB available-memory start floor and 8 GiB host early stop. Its source
|
||||
and unchanged observer hashes are recorded in the curated receipt. This local
|
||||
resource policy does not change the public workflow or global wrapper defaults.
|
||||
Raw histories, transcripts, oracle outputs, and resource logs stay in the private
|
||||
local audit cache; only curated receipts are committed.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,238 @@
|
||||
# JEB Linux CE native workflow checkpoint
|
||||
|
||||
This checkpoint qualifies Linux CE setup and selected native reads for
|
||||
[#4996](https://github.com/morluto/rea/issues/4996). It changes documentation;
|
||||
it does not introduce a provider cache or claim a before/after speedup.
|
||||
The existing shared CLI/MCP implementation returned consistent facts and
|
||||
Evidence, with one metadata request per persistent MCP task. Further wrapper
|
||||
caching is not justified by these measurements.
|
||||
|
||||
## Pinned inputs and scope
|
||||
|
||||
Baseline: REA 6.6.0 at `7c458f04b9155707c8ba406f8b455b10016f1b99`, Node 24.18.0,
|
||||
npm 11.16.0, Temurin 21.0.12.1, Linux x86-64
|
||||
`6.1.0-53-cloud-amd64`. All six final sessions used the same compiled MCP
|
||||
runtime digest, recorded in the [curated receipt](jeb-linux-native.json).
|
||||
|
||||
The official CE archive was obtained through
|
||||
<https://www.pnfsoftware.com/dl?jeb4ce>: 697,549,250 bytes, observed SHA-256
|
||||
`709e7fe8e88bb0e9e2bc03b028f8b9fc78cec0289d04abe54fc26cda6369e175`.
|
||||
HTTPS download and ZIP CRC checks completed; a publisher signature was not
|
||||
independently verified. This distribution identifies as
|
||||
`5.48.0.202610071539`, serving MCP 1.3.0. The installation and first-use
|
||||
activation remained private; no engine binaries, keys or raw decompilation
|
||||
are included here.
|
||||
|
||||
| Artifact | Bytes | Independent check |
|
||||
| -------------------------------------------------------- | --------: | ----------------------------------------------------------------------------- |
|
||||
| Authored C ELF, ET_EXEC | 17,360 | Source arithmetic and GNU `nm` symbol |
|
||||
| Debian `libsqlite3.so.0.8.6`, package `3.40.1-2+deb12u2` | 1,437,848 | GNU `nm` and `objdump`: exported version function returns immediate `3040001` |
|
||||
|
||||
Target SHA-256 digests:
|
||||
|
||||
- Fixture: `1a6f55303712e096ff5196c7fd6e8acf75473fbf0f7467d7b315f96f50b3f841`.
|
||||
- SQLite: `2e6eef9a727f081f0d453b4e5e6cbd8b9ef8b6f86cbf7681cbad444d3b0b55c8`.
|
||||
|
||||
Neither target was executed. The fixture's expected arithmetic was `input * 7 + 13`.
|
||||
That check does not establish inferred ABI, signedness or equivalence for every
|
||||
input. SQLite's constant-return oracle is also narrower than complete-program
|
||||
correctness.
|
||||
|
||||
## Setup and item identity
|
||||
|
||||
The current official CE build successfully accepted `-c --mcp`, using
|
||||
`--interface=127.0.0.1`, a dedicated port and `--infile` for the selected ELF.
|
||||
This agrees with the [current upstream MCP manual](https://www.pnfsoftware.com/jeb/manual/mcp/).
|
||||
The earlier documented licensed-build result used a programmatic launcher;
|
||||
its CLI behavior should not be generalized to all 5.48 distributions.
|
||||
|
||||
Code-item references also need their owning representation. The CE native
|
||||
`list_code_items` tool returned `sqlite3_libversion_number` for a filtered
|
||||
method query. That exact reference succeeded through REA. Earlier raw ELF
|
||||
address attempts did not complete successfully. A numeric reference did work
|
||||
for the authored ET_EXEC pilot, so neither observation establishes a universal
|
||||
numeric-address rule. Preserve the engine's reference and explicit unit path;
|
||||
keep file offsets and independently obtained ELF addresses separate.
|
||||
|
||||
The native CE catalog advertised 27 tools. REA currently exposes four JEB
|
||||
operations; native code-item discovery is not among them. The final tasks
|
||||
therefore start with a known reference. This is a selected-read qualification,
|
||||
not a complete method-discovery or provider-parity comparison.
|
||||
|
||||
## Completed workflow and accounting
|
||||
|
||||
Three alternating fresh-engine sessions per artifact completed the same task:
|
||||
|
||||
1. Start the private headless engine; prove that its child PID owns the
|
||||
selected IPv4 or mapped-IPv6 loopback listener before connecting.
|
||||
2. Connect the production stdio MCP runtime and discover its full catalog.
|
||||
3. Inspect JEB, list all three units, then concatenate four one-item pages
|
||||
including the empty remainder and compare with the original listing.
|
||||
4. Decompile the selected method, check the independent oracle and complete
|
||||
inline Evidence, then repeat it six times with identical facts and Evidence IDs.
|
||||
5. Check a missing item returns an error; make one real CLI query and compare
|
||||
its normalized facts, raw text and Evidence ID with MCP; close the client.
|
||||
6. Verify target bytes are unchanged and the borrowed engine survived REA
|
||||
cleanup, then stop the harness-owned engine.
|
||||
|
||||
Each task made 15 public MCP calls and one CLI parity query. Request tracing
|
||||
observed one upstream catalog walk, one `get_client_information`, five
|
||||
`list_units` and eight `decompile_code_item` calls. Explicit metadata refresh
|
||||
and reconnect behavior were not part of this repeat-read measurement.
|
||||
Tracing covered the production MCP child; the separate CLI query's upstream
|
||||
requests were not instrumented.
|
||||
Evidence retained `subject: null` and the artifact-identity limitation;
|
||||
an independent native project-digest observation was not substituted for
|
||||
identity atomically bound to each read.
|
||||
|
||||
All heavy work ran serially inside a cgroup v2 family with an 8 GiB hard cap,
|
||||
7 GiB early stop, one CPU quota and affinity, and zero swap. Startup required
|
||||
16 GiB host availability; a running task would stop below 8 GiB. The JVM used
|
||||
`-Xmx1536m -XX:ActiveProcessorCount=1`; the production MCP child used a
|
||||
2048 MiB Node heap, and the harness/CLI inherited a 3072 MiB Node heap limit.
|
||||
These are audit settings, not general engine-size recommendations.
|
||||
|
||||
OS and vendor file caches were not flushed. “First” means the first selected
|
||||
function read in a fresh JVM; “warm” means repeated reads in that same engine.
|
||||
The private fetch observer recorded request names and byte counts without
|
||||
cloning response bodies; its overhead is included.
|
||||
|
||||
Whole-task counters include the JVM, MCP client, production runtime, CLI
|
||||
parity, native oracle and cleanup, including cgroup-charged file cache.
|
||||
Host-admission wait is recorded separately. Outer guard/systemd-manager costs
|
||||
were not measured. Payload-exit counters and cleanup were verified while an
|
||||
observer remained alive; disappeared final whole-scope counters remain unknown.
|
||||
|
||||
## Measured costs
|
||||
|
||||
Medians with observed minimum–maximum across three completed sessions:
|
||||
|
||||
| Cost | Authored ELF | Shipped SQLite |
|
||||
| ---------------------------------------- | -------------------------: | -------------------------: |
|
||||
| First selected MCP read | 4.134 s (3.845–5.037) | 24.508 s (22.823–49.977) |
|
||||
| Six warm MCP reads, total | 253.2 ms (113.4–842.9) | 190.0 ms (160.9–416.9) |
|
||||
| Whole guarded task elapsed | 27.638 s (26.749–32.622) | 48.296 s (45.625–82.039) |
|
||||
| Whole family CPU | 27.291 s (26.399–32.155) | 47.753 s (45.370–78.877) |
|
||||
| Kernel family peak, including file cache | 1540.1 MiB (1531.5–1614.9) | 1799.6 MiB (1714.7–1865.8) |
|
||||
|
||||
The slower third sessions remain in the results. Their extra elapsed and CPU
|
||||
costs are unexplained; no favorable resampling, stable latency improvement or
|
||||
memory reduction is claimed. Warm reads are already short relative to first
|
||||
reads. Adding another live-project cache would introduce identity and
|
||||
invalidation obligations without a demonstrated bottleneck here.
|
||||
|
||||
All six completed payloads recorded zero memory-limit, task-limit, OOM and
|
||||
OOM-kill events. This is a payload-exit observation, not a claim that missing
|
||||
final counters equal zero.
|
||||
|
||||
## Context estimates
|
||||
|
||||
The unchanged production catalog contained 207 tools, 3,158,927 serialized
|
||||
bytes, SHA-256
|
||||
`a1851aa1bfe3d72d6be6bec130519dbdfc1ee49288320cfac0144ced77755c61`.
|
||||
An offline tiktoken 0.12.0 `cl100k_base` count assigned 805,330 tokens to one
|
||||
raw catalog page-array in every session.
|
||||
|
||||
| Estimated serialization | Authored ELF median | SQLite median |
|
||||
| ---------------------------------------------------- | ------------------: | ------------: |
|
||||
| Catalog + named MCP arguments + raw replies | 814,268 | 814,878 |
|
||||
| Above, plus selected CLI arguments and parity stdout | 814,610 | 815,249 |
|
||||
| MCP text reply blocks alone | 3,957 | 4,203 |
|
||||
|
||||
Raw replies contain both text and structured copies. These counts describe
|
||||
the validation exchange, including its extra pagination, repeated and negative
|
||||
queries. Actual model usage, prompts, protocol/initialization framing, client
|
||||
filtering/rendering/replay, shell framing and context for independent oracles
|
||||
were not measured. No model-token savings or complex-investigation advantage
|
||||
is established. Catalog delivery is a shared concern; existing
|
||||
[PR #3026](https://github.com/morluto/rea/pull/3026) already explores family
|
||||
selection and is outside this checkpoint.
|
||||
|
||||
## Preserved attempts and verification limits
|
||||
|
||||
The JSON receipt retains costs and outcomes for all earlier attempts:
|
||||
|
||||
- Three host-memory refusals happened before scope/payload creation.
|
||||
- First-use EULA and missing-key attempts exited before analysis. Although
|
||||
JEB exited zero, the harness correctly reported failure.
|
||||
- The 30-second raw-address SQLite attempt returned typed `provider_timeout`;
|
||||
the 60-second raw-address attempt returned an upstream refusal. Neither is a
|
||||
completed benchmark or proof of a REA defect.
|
||||
- A producer-reference SQLite read succeeded, but the initial test oracle
|
||||
rejected its valid C `L` integer suffix. The oracle was corrected before
|
||||
the final cohort; that incomplete workflow and its cost remain recorded.
|
||||
- Setup, diagnostic help, successful pilots, native discovery checks and
|
||||
offline tokenizer work are recorded separately from the final cohort.
|
||||
|
||||
This audit does not qualify CE Java/Android decompilation, GUI project opening,
|
||||
Windows/macOS, more than 100 units, real caller cancellation, or recovery after
|
||||
engine restart. Existing reconnect/invalidation work is
|
||||
[PR #3754](https://github.com/morluto/rea/pull/3754). The selected function and
|
||||
constant checks do not establish broader software-analysis effectiveness.
|
||||
|
||||
## Release 6.7.0 integration control
|
||||
|
||||
Main advanced to `a25eb5fbb424197cb8e699e15a82a6ff869658aa` during review.
|
||||
The branch rebased onto that release checkpoint, rebuilt, and passed
|
||||
`docs:check` plus uncached typecheck/lint. Two additional fresh-engine public
|
||||
workflows passed against `837718b1968b5fd90762c4c54895b11e9f5478fd`: one
|
||||
authored ELF and one shipped SQLite, adding 30 MCP calls and two CLI parity
|
||||
queries. They verify the rebuilt 6.7.0 runtime and remain separate from the
|
||||
six-session performance cohort above. Their runtime/catalog digests and
|
||||
process-family costs are recorded in the JSON receipt.
|
||||
|
||||
The first additional SQLite attempt was stopped after 28.244 seconds when
|
||||
host available memory fell below the unchanged 8 GiB stop threshold. Final
|
||||
scope limit/OOM counters were observed as zero; payload-exit counters were
|
||||
unavailable, so that attempt is incomplete. Its sampled peak is not a kernel
|
||||
peak claim. Target bytes remained unchanged. The successful retry used a
|
||||
stricter 24 GiB admission wait with the original 8 GiB family cap, 7 GiB early
|
||||
stop, one CPU and zero swap. No failed attempt was replaced or counted passed.
|
||||
|
||||
## Reproduction entrypoints
|
||||
|
||||
The small source fixture was compiled with `gcc -O1 -g -fno-pie -no-pie`:
|
||||
|
||||
```c
|
||||
#include <stdio.h>
|
||||
__attribute__((noinline)) int rea_probe_value(int input) { return input * 7 + 13; }
|
||||
int main(void) { puts("rea_jeb_native_probe"); return rea_probe_value(4) == 41 ? 0 : 1; }
|
||||
```
|
||||
|
||||
Its recorded digest pins the tested build; debug/source paths can change a
|
||||
rebuilt binary's digest. The source expression and independently resolved
|
||||
symbol remain the fixture oracle.
|
||||
|
||||
From the caller's JEB directory, after completing its official first-use setup
|
||||
and selecting JDK 21 through `JAVA_HOME`:
|
||||
|
||||
```sh
|
||||
"$JAVA_HOME/bin/java" -Xmx1536m -XX:ActiveProcessorCount=1 -jar bin/app/jeb.jar -c --mcp \
|
||||
--interface=127.0.0.1 --port=18425 \
|
||||
--infile=/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
|
||||
```
|
||||
|
||||
In the pinned REA checkout, with the pinned Node toolchain and built runtime:
|
||||
|
||||
```sh
|
||||
export REA_JEB_MCP_URL=http://127.0.0.1:18425/mcp
|
||||
export REA_JEB_MCP_TIMEOUT_MS=60000
|
||||
node scripts/rea.mjs inspect-jeb-client --json
|
||||
node scripts/rea.mjs list-jeb-units --count 100 --json
|
||||
node scripts/rea.mjs decompile-jeb-item sqlite3_libversion_number method \
|
||||
--unit-path libsqlite3.so.0.8.6/libsqlite3.so.0.8.6/x86_64-image --json
|
||||
nm -D --defined-only /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
|
||||
objdump -d --disassemble=sqlite3_libversion_number \
|
||||
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
|
||||
```
|
||||
|
||||
The listed commands reproduce individual reads and the file oracle; the
|
||||
full measurement also used the MCP sequence and process-family observer
|
||||
described above. JVM heap flags alone do not impose its family/host guards.
|
||||
Private raw transcripts and engine documents are excluded from the repository.
|
||||
|
||||
The baseline gate passed `npm run docs:check`, `npm run check:fast` and
|
||||
`git diff --check`; its typecheck/lint were cache hits. The release integration
|
||||
gate above also passed with uncached typecheck/lint. The first gate attempt was refused
|
||||
before scope creation; the admitted gate passed under the unchanged bounds.
|
||||
The runtime results above were executed against the pinned baseline.
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "rea-agents",
|
||||
"version": "6.5.0",
|
||||
"version": "6.7.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "rea-agents",
|
||||
"version": "6.5.0",
|
||||
"version": "6.7.0",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/parser": "8.0.4",
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "rea-agents",
|
||||
"mcpName": "io.github.morluto/rea",
|
||||
"version": "6.5.0",
|
||||
"version": "6.7.0",
|
||||
"description": "Reverse engineer anything from your terminal or agent with one CLI and MCP server.",
|
||||
"license": "MIT",
|
||||
"repository": {
|
||||
|
||||
@@ -1,28 +1,9 @@
|
||||
import { ArtifactResourceScope } from "../../dist/artifacts/ArtifactResourceScope.js";
|
||||
import { AnalysisCleanupLease } from "../../dist/artifacts/AnalysisCleanupLease.js";
|
||||
import { inspectGhidraInstallation } from "../../dist/ghidra/GhidraInstallation.js";
|
||||
import { closeVerifierResources } from "./verifier-cleanup.mjs";
|
||||
import { withVerifierResourceOwner } from "./verifier-file-resources.mjs";
|
||||
|
||||
/** Complete one prerequisite inspection and release its exact retained owners. */
|
||||
export async function inspectVerifierGhidraInstallation(options, host) {
|
||||
const resources = new ArtifactResourceScope();
|
||||
const resourceOwner = {
|
||||
retain(owner, resource) {
|
||||
resources.retain({
|
||||
kind: "lease",
|
||||
resource,
|
||||
lease: AnalysisCleanupLease.forCloseOwner(owner, resource),
|
||||
});
|
||||
},
|
||||
run: (operation) => resources.run(operation),
|
||||
};
|
||||
let primaryError;
|
||||
try {
|
||||
return await inspectGhidraInstallation({ ...options, resourceOwner }, host);
|
||||
} catch (cause) {
|
||||
primaryError = cause;
|
||||
throw cause;
|
||||
} finally {
|
||||
await closeVerifierResources([() => resources.close()], primaryError);
|
||||
}
|
||||
export function inspectVerifierGhidraInstallation(options, host) {
|
||||
return withVerifierResourceOwner((resourceOwner) =>
|
||||
inspectGhidraInstallation({ ...options, resourceOwner }, host),
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
import type { validateJadxConfiguration } from "../../src/android/JadxConfiguration.js";
|
||||
|
||||
type ConfigurationArguments = Parameters<typeof validateJadxConfiguration>;
|
||||
|
||||
/** Validate prerequisites with a real owner, joining cleanup before completion. */
|
||||
export function validateVerifierJadxConfiguration(
|
||||
configuration: ConfigurationArguments[0],
|
||||
environment: ConfigurationArguments[1],
|
||||
operation: ConfigurationArguments[2],
|
||||
options?: Omit<ConfigurationArguments[3], "resourceOwner" | "operation">,
|
||||
): ReturnType<typeof validateJadxConfiguration>;
|
||||
@@ -0,0 +1,17 @@
|
||||
import { validateJadxConfiguration } from "../../dist/android/JadxConfiguration.js";
|
||||
import { withVerifierResourceOwner } from "./verifier-file-resources.mjs";
|
||||
|
||||
/** Validate Java and the engine archive before beginning real verification. */
|
||||
export function validateVerifierJadxConfiguration(
|
||||
configuration,
|
||||
environment,
|
||||
operation,
|
||||
options = {},
|
||||
) {
|
||||
return withVerifierResourceOwner((resourceOwner) =>
|
||||
validateJadxConfiguration(configuration, environment, operation, {
|
||||
...options,
|
||||
resourceOwner,
|
||||
}),
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
export function isOwnedTestTemporaryName(name: string): boolean;
|
||||
export function normalizeTestTemporaryPrefix(prefix: string): string;
|
||||
@@ -0,0 +1,18 @@
|
||||
const OWNED_ENTRY = /^(?:rea-|bb-session-)/u;
|
||||
const SAFE_PREFIX = /^[A-Za-z0-9][A-Za-z0-9._-]*-$/u;
|
||||
|
||||
/** Recognize test resources independently of their readiness or contents. */
|
||||
export const isOwnedTestTemporaryName = (name) => OWNED_ENTRY.test(name);
|
||||
|
||||
/** Preserve caller labels while making every shared fixture root recognizable. */
|
||||
export function normalizeTestTemporaryPrefix(prefix) {
|
||||
if (
|
||||
typeof prefix !== "string" ||
|
||||
!SAFE_PREFIX.test(prefix) ||
|
||||
prefix.length > 80
|
||||
)
|
||||
throw new TypeError(
|
||||
"Test temporary directory prefixes must be safe basenames ending in '-'",
|
||||
);
|
||||
return isOwnedTestTemporaryName(prefix) ? prefix : `rea-${prefix}`;
|
||||
}
|
||||
@@ -1,4 +1,10 @@
|
||||
import type { ArtifactResourceScope } from "../../src/artifacts/ArtifactResourceScope.js";
|
||||
import type { FileCloseResourceOwner } from "../../src/application/AnalysisProvider.js";
|
||||
|
||||
/** Complete a prerequisite operation, including retained cleanup on failure. */
|
||||
export function withVerifierResourceOwner<T>(
|
||||
operation: (owner: FileCloseResourceOwner) => Promise<T>,
|
||||
): Promise<T>;
|
||||
|
||||
/** Hash one verifier input and transfer failed-close ownership to its scope. */
|
||||
export function hashVerifierFile(
|
||||
|
||||
@@ -1,4 +1,31 @@
|
||||
import { hashStableFile } from "../../dist/filesystem/StableFileHash.js";
|
||||
import { ArtifactResourceScope } from "../../dist/artifacts/ArtifactResourceScope.js";
|
||||
import { AnalysisCleanupLease } from "../../dist/artifacts/AnalysisCleanupLease.js";
|
||||
import { closeVerifierResources } from "./verifier-cleanup.mjs";
|
||||
|
||||
/** Finish a prerequisite probe and release its exact retained cleanup owners. */
|
||||
export async function withVerifierResourceOwner(operation) {
|
||||
const resources = new ArtifactResourceScope();
|
||||
const resourceOwner = {
|
||||
retain(owner, resource) {
|
||||
resources.retain({
|
||||
kind: "lease",
|
||||
resource,
|
||||
lease: AnalysisCleanupLease.forCloseOwner(owner, resource),
|
||||
});
|
||||
},
|
||||
run: (work) => resources.run(work),
|
||||
};
|
||||
let primaryError;
|
||||
try {
|
||||
return await operation(resourceOwner);
|
||||
} catch (cause) {
|
||||
primaryError = cause;
|
||||
throw cause;
|
||||
} finally {
|
||||
await closeVerifierResources([() => resources.close()], primaryError);
|
||||
}
|
||||
}
|
||||
|
||||
/** Hash one verifier input while retaining any descriptor whose close fails. */
|
||||
export function hashVerifierFile(path, resources) {
|
||||
|
||||
@@ -4,15 +4,16 @@ import { dirname, join, relative, resolve } from "node:path";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import { isOwnedTestTemporaryName } from "./lib/test-temporary-ownership.mjs";
|
||||
|
||||
const repositoryRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const vitestPath = resolve(repositoryRoot, "node_modules/vitest/vitest.mjs");
|
||||
const OWNED_ENTRY = /^(?:rea-|bb-session-)/u;
|
||||
|
||||
const findOwnedEntries = async (root, directory = root) => {
|
||||
const owned = [];
|
||||
for (const entry of await readdir(directory, { withFileTypes: true })) {
|
||||
const path = join(directory, entry.name);
|
||||
if (OWNED_ENTRY.test(entry.name)) {
|
||||
if (isOwnedTestTemporaryName(entry.name)) {
|
||||
owned.push(relative(root, path));
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -14,10 +14,8 @@ import { verifyAndroidCliCancellation } from "./cli-cancellation.mjs";
|
||||
import { verifyAndroidMetadata } from "./metadata.mjs";
|
||||
import { verifyAndroidClassSearch } from "./class-search.mjs";
|
||||
import { verifyAndroidFailureRecovery } from "./failure-recovery.mjs";
|
||||
import {
|
||||
resolveJadxConfigurationInputs,
|
||||
validateJadxConfiguration,
|
||||
} from "../../../dist/android/JadxConfiguration.js";
|
||||
import { resolveJadxConfigurationInputs } from "../../../dist/android/JadxConfiguration.js";
|
||||
import { validateVerifierJadxConfiguration } from "../../lib/jadx-configuration-inspection.mjs";
|
||||
import { verifyWindowsAndroidLifecycle } from "./windows-lifecycle.mjs";
|
||||
|
||||
const repository = resolve(dirname(fileURLToPath(import.meta.url)), "../../..");
|
||||
@@ -43,7 +41,7 @@ const configuration = await resolveJadxConfigurationInputs(
|
||||
process.env,
|
||||
"inspect_android_package",
|
||||
);
|
||||
await validateJadxConfiguration(
|
||||
await validateVerifierJadxConfiguration(
|
||||
configuration,
|
||||
process.env,
|
||||
"inspect_android_package",
|
||||
|
||||
@@ -17,6 +17,7 @@ import { delimiter, dirname, isAbsolute, join } from "node:path";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { TOOL_CONTRACTS } from "../../../dist/contracts/toolContracts.js";
|
||||
import { analysisErrorProjectionSchema } from "../../../dist/contracts/errorSchemas.js";
|
||||
|
||||
/** Real unusable Java runtimes and selectors must lead to applicable recovery. */
|
||||
export async function verifyAndroidFailureRecovery({
|
||||
@@ -103,18 +104,46 @@ export async function verifyAndroidFailureRecovery({
|
||||
label: "JRE without compiler",
|
||||
env: { ...environment, JAVA_HOME: jre },
|
||||
observation: "jdk.compiler",
|
||||
diagnostics: {
|
||||
failure_kind: "java_runtime",
|
||||
failure_code: "runtime_missing",
|
||||
jdk_compiler: false,
|
||||
},
|
||||
message: /full JDK/u,
|
||||
remediation: /java --list-modules/u,
|
||||
},
|
||||
{
|
||||
label: "missing java",
|
||||
env: { ...withoutJavaHome, PATH: empty },
|
||||
observation:
|
||||
process.platform === "win32" ? "Cannot find java.exe" : "ENOENT",
|
||||
diagnostics: {
|
||||
failure_kind: "java_runtime",
|
||||
failure_code: "runtime_missing",
|
||||
...(process.platform === "win32"
|
||||
? {}
|
||||
: { process_error_code: "ENOENT" }),
|
||||
},
|
||||
message:
|
||||
process.platform === "win32"
|
||||
? /Cannot find java\.exe/u
|
||||
: /selected Java runtime/u,
|
||||
remediation:
|
||||
process.platform === "win32"
|
||||
? /JAVA_HOME|PATH/u
|
||||
: /selected launcher/u,
|
||||
},
|
||||
{
|
||||
label: "invalid engine archive with a working JDK",
|
||||
env: { ...environment, REA_JADX_MCP_JAR: invalidJar },
|
||||
observation: "Cannot inspect REA_JADX_MCP_JAR",
|
||||
jarFailure: true,
|
||||
diagnostics: {
|
||||
failure_kind: "jar",
|
||||
failure_code: "version_unresolved",
|
||||
},
|
||||
message: /REA_JADX_MCP_JAR/u,
|
||||
remediation: /jadx-headless-mcp/u,
|
||||
forbiddenGuidance: /Select a full JDK/u,
|
||||
},
|
||||
];
|
||||
if (process.platform === "darwin") {
|
||||
@@ -126,6 +155,12 @@ export async function verifyAndroidFailureRecovery({
|
||||
label: "macOS Java stub",
|
||||
env: { ...withoutJavaHome, PATH: "/usr/bin:/bin" },
|
||||
observation: "Unable to locate a Java Runtime",
|
||||
diagnostics: {
|
||||
failure_kind: "java_probe",
|
||||
failure_code: "runtime_unavailable",
|
||||
},
|
||||
message: /selected Java runtime/u,
|
||||
remediation: /selected launcher/u,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -145,23 +180,20 @@ export async function verifyAndroidFailureRecovery({
|
||||
assert.fail(`${scenario.label} unexpectedly succeeded`);
|
||||
} catch (cause) {
|
||||
assert.equal(cause.code, 1);
|
||||
failure = JSON.parse(cause.stdout);
|
||||
failure = analysisErrorProjectionSchema.parse(JSON.parse(cause.stdout));
|
||||
}
|
||||
assert.equal(failure.code, "capability_unavailable");
|
||||
assert.ok(
|
||||
JSON.stringify(failure).includes(scenario.observation),
|
||||
JSON.stringify(failure),
|
||||
);
|
||||
assert.match(
|
||||
failure.message,
|
||||
scenario.jarFailure ? /REA_JADX_MCP_JAR/u : /full JDK/u,
|
||||
);
|
||||
assert.match(
|
||||
failure.remediation.action,
|
||||
scenario.jarFailure ? /jadx-headless-mcp/u : /java --list-modules/u,
|
||||
);
|
||||
if (scenario.jarFailure)
|
||||
assert.doesNotMatch(failure.message, /Select a full JDK/u);
|
||||
assert.ok(failure.details.diagnostics);
|
||||
for (const [key, value] of Object.entries(scenario.diagnostics))
|
||||
assert.deepEqual(failure.details.diagnostics[key], value);
|
||||
assert.match(failure.message, scenario.message);
|
||||
assert.match(failure.remediation.action, scenario.remediation);
|
||||
if (scenario.forbiddenGuidance)
|
||||
assert.doesNotMatch(failure.message, scenario.forbiddenGuidance);
|
||||
assert.doesNotMatch(JSON.stringify(failure), /rea doctor/u);
|
||||
await verifyAndroidReadiness({
|
||||
entrypoint,
|
||||
|
||||
@@ -13,10 +13,8 @@ import { dirname, join } from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import {
|
||||
resolveJadxConfigurationInputs,
|
||||
validateJadxConfiguration,
|
||||
} from "../../../dist/android/JadxConfiguration.js";
|
||||
import { resolveJadxConfigurationInputs } from "../../../dist/android/JadxConfiguration.js";
|
||||
import { validateVerifierJadxConfiguration } from "../../lib/jadx-configuration-inspection.mjs";
|
||||
import { verifyWindowsGuardianIsolation } from "./windows-guardian-isolation.mjs";
|
||||
|
||||
/** Verify real Windows cancellation, disconnect and abrupt CLI owner exit. */
|
||||
@@ -45,7 +43,7 @@ async function verifyJavaSelection(environment) {
|
||||
environment,
|
||||
"inspect_android_package",
|
||||
);
|
||||
await validateJadxConfiguration(
|
||||
await validateVerifierJadxConfiguration(
|
||||
selected,
|
||||
environment,
|
||||
"inspect_android_package",
|
||||
@@ -65,7 +63,7 @@ async function verifyJavaSelection(environment) {
|
||||
pathEnvironment,
|
||||
"inspect_android_package",
|
||||
);
|
||||
await validateJadxConfiguration(
|
||||
await validateVerifierJadxConfiguration(
|
||||
onPath,
|
||||
pathEnvironment,
|
||||
"inspect_android_package",
|
||||
|
||||
+2
-2
@@ -8,13 +8,13 @@
|
||||
"url": "https://github.com/morluto/rea",
|
||||
"source": "github"
|
||||
},
|
||||
"version": "6.5.0",
|
||||
"version": "6.7.0",
|
||||
"packages": [
|
||||
{
|
||||
"registryType": "npm",
|
||||
"registryBaseUrl": "https://registry.npmjs.org",
|
||||
"identifier": "rea-agents",
|
||||
"version": "6.5.0",
|
||||
"version": "6.7.0",
|
||||
"runtimeHint": "npx",
|
||||
"transport": {
|
||||
"type": "stdio"
|
||||
|
||||
@@ -23,6 +23,37 @@ const probe = (signal?: AbortSignal) =>
|
||||
signal,
|
||||
);
|
||||
|
||||
it("classifies the complete diagnostic before shortening its displayed reason", async () => {
|
||||
const localPath = `/host/${"directory/".repeat(60)}payload.bin`;
|
||||
const stderr = `adb: error: cannot create '${localPath}': Permission denied`;
|
||||
vi.mocked(execFileBytes).mockRejectedValueOnce(
|
||||
Object.assign(new Error("local destination refused"), {
|
||||
code: 1,
|
||||
stdout: "",
|
||||
stderr,
|
||||
signal: null,
|
||||
killed: false,
|
||||
}),
|
||||
);
|
||||
await expect(
|
||||
runAdbCapture(
|
||||
"pull_adb_file",
|
||||
{ device: "device", path: "/remote/file", localPath },
|
||||
"/host/adb",
|
||||
[],
|
||||
{ timeoutMs: 100, maxOutputBytes: 2048 },
|
||||
undefined,
|
||||
),
|
||||
).resolves.toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
_tag: "AnalysisUnsupportedTargetError",
|
||||
path: localPath,
|
||||
reason: stderr.slice(0, 512),
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("recognizes only the completed empty false result as absence", async () => {
|
||||
vi.mocked(execFileBytes).mockRejectedValueOnce(
|
||||
Object.assign(new Error("test false"), {
|
||||
@@ -161,7 +192,7 @@ it("keeps a physical failure even when the later abort reason is that same error
|
||||
});
|
||||
const result = await runAdbCapture(
|
||||
"inspect_adb_device",
|
||||
"device",
|
||||
{ device: "device" },
|
||||
"/host/adb",
|
||||
[],
|
||||
{ timeoutMs: 100, maxOutputBytes: 1024 },
|
||||
|
||||
+72
-17
@@ -17,8 +17,26 @@ import {
|
||||
type ExecFileOutputFailure,
|
||||
} from "../process/ExecFileOutput.js";
|
||||
import { packageChangeFailureReason } from "./AdbShellOutput.js";
|
||||
import { isSuccessfulDaemonStartNotice } from "./AdbDeviceOutput.js";
|
||||
|
||||
const STDERR_EXCERPT_BYTES = 512;
|
||||
const STDERR_EXCERPT_CHARACTERS = 512;
|
||||
|
||||
// Match the known host operand in adb's local-file diagnostics. The suffix is
|
||||
// part of the format: opening/reading identify local failures explicitly.
|
||||
const LOCAL_FILE_DIAGNOSTICS = [
|
||||
["cannot create", ":"],
|
||||
["cannot create file/directory", ":"],
|
||||
["cannot stat", ":"],
|
||||
["cannot open", ":"],
|
||||
["cannot write", ":"],
|
||||
["failed to stat local file", ":"],
|
||||
["failed to read all of", ":"],
|
||||
["failed to access", ":"],
|
||||
["failed to stat", ":"],
|
||||
["failed to open", ":"],
|
||||
["opening", " locally failed:"],
|
||||
["reading", " locally failed:"],
|
||||
] as const;
|
||||
|
||||
/** Project parsed command-owner facts without reinterpreting the rejection. */
|
||||
export const adbCommandCapturedOutput = (
|
||||
@@ -70,10 +88,19 @@ export interface AdbRunLimits {
|
||||
readonly maxOutputBytes: number;
|
||||
}
|
||||
|
||||
/** Keep the selected device separate from the file addressed by the command. */
|
||||
export interface AdbCommandTarget {
|
||||
readonly device: string;
|
||||
/** Caller-addressed file, such as a remote path or the original APK input. */
|
||||
readonly path?: string;
|
||||
/** Exact host operand adb consumes, including an owned input snapshot. */
|
||||
readonly localPath?: string;
|
||||
}
|
||||
|
||||
/** Run one bounded adb invocation and keep its exact failure shape. */
|
||||
export const runAdbCapture = async (
|
||||
operation: string,
|
||||
targetPath: string,
|
||||
target: AdbCommandTarget,
|
||||
binary: string,
|
||||
arguments_: readonly string[],
|
||||
limits: AdbRunLimits,
|
||||
@@ -115,33 +142,61 @@ export const runAdbCapture = async (
|
||||
{ cause },
|
||||
),
|
||||
);
|
||||
const excerpt = failure.stderr.trim().slice(0, STDERR_EXCERPT_BYTES);
|
||||
if (/unauthorized/iu.test(excerpt))
|
||||
return err(
|
||||
new AnalysisUnsupportedTargetError(operation, targetPath, excerpt, {
|
||||
cause,
|
||||
}),
|
||||
const stderr = failure.stderr.trim();
|
||||
const excerpt = stderr.slice(0, STDERR_EXCERPT_CHARACTERS);
|
||||
// Device-state words inside a reported file name are not device verdicts.
|
||||
// Only leading notices need line parsing. Do not expand a capture-sized
|
||||
// stderr into an array whose entries can exceed the byte capture budget.
|
||||
let diagnosticStart = 0;
|
||||
while (diagnosticStart < stderr.length) {
|
||||
const newline = stderr.indexOf("\n", diagnosticStart);
|
||||
const line = stderr
|
||||
.slice(diagnosticStart, newline === -1 ? stderr.length : newline)
|
||||
.trimEnd();
|
||||
if (
|
||||
!isSuccessfulDaemonStartNotice(line) &&
|
||||
!/^\* daemon not running(?:; starting now at [^\r\n]*|\. starting it now on port \d+ \*)$/u.test(
|
||||
line,
|
||||
)
|
||||
)
|
||||
break;
|
||||
diagnosticStart = newline === -1 ? stderr.length : newline + 1;
|
||||
}
|
||||
const diagnostic = stderr
|
||||
.slice(diagnosticStart)
|
||||
.replace(
|
||||
/^(?:adb:\s*)?(?:error:\s*)?(?:(?:failed to get feature set|connect failed):\s*)?/iu,
|
||||
"",
|
||||
);
|
||||
if (
|
||||
/device[^\n]*not found|device offline|still authorizing|more than one device/iu.test(
|
||||
excerpt,
|
||||
/^(?:unauthorized\b|device(?: '[^'\n]*')? (?:unauthorized\b|offline\b|not found\b|still authorizing\b)|still authorizing\b|more than one device\b)/iu.test(
|
||||
diagnostic,
|
||||
)
|
||||
)
|
||||
return err(
|
||||
new AnalysisUnsupportedTargetError(operation, targetPath, excerpt, {
|
||||
new AnalysisUnsupportedTargetError(operation, target.device, excerpt, {
|
||||
cause,
|
||||
}),
|
||||
);
|
||||
if (
|
||||
/No such file or directory|Permission denied|read: Permission denied/iu.test(
|
||||
excerpt,
|
||||
diagnostic,
|
||||
)
|
||||
)
|
||||
) {
|
||||
const localPath = target.localPath;
|
||||
const path =
|
||||
localPath !== undefined &&
|
||||
LOCAL_FILE_DIAGNOSTICS.some(([verb, suffix]) =>
|
||||
[localPath, `'${localPath}'`, `"${localPath}"`].some((operand) =>
|
||||
diagnostic.startsWith(`${verb} ${operand}${suffix}`),
|
||||
),
|
||||
)
|
||||
? localPath
|
||||
: (target.path ?? target.device);
|
||||
return err(
|
||||
new AnalysisUnsupportedTargetError(operation, targetPath, excerpt, {
|
||||
cause,
|
||||
}),
|
||||
new AnalysisUnsupportedTargetError(operation, path, excerpt, { cause }),
|
||||
);
|
||||
}
|
||||
if (failure.killed)
|
||||
return err(
|
||||
new AnalysisTimeoutError(operation, limits.timeoutMs, { cause }),
|
||||
@@ -210,7 +265,7 @@ export const probeAdbPathExistence = async (
|
||||
): Promise<Result<boolean, AnalysisError>> => {
|
||||
const probed = await runAdbCapture(
|
||||
operation,
|
||||
path,
|
||||
{ device: serial, path },
|
||||
binary,
|
||||
adbArguments(serial, ["shell", "test", "-e", path]),
|
||||
limits,
|
||||
|
||||
@@ -32,7 +32,7 @@ const DEVICE_LINE = /^(\S+)\s+(.*)$/u;
|
||||
|
||||
const normalizeAdbOutputLine = (rawLine: string): string => rawLine.trimEnd();
|
||||
|
||||
const isSuccessfulDaemonStartNotice = (line: string): boolean =>
|
||||
export const isSuccessfulDaemonStartNotice = (line: string): boolean =>
|
||||
line === "* daemon started successfully" ||
|
||||
line === "* daemon started successfully *";
|
||||
|
||||
|
||||
+136
-2
@@ -568,7 +568,7 @@ it.each([
|
||||
async (_, stderr, excerpt) => {
|
||||
const { binary } = await writeAdbStub(
|
||||
() => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi
|
||||
if [ "$1" = "-s" ]; then printf '%s\\n' "${stderr.replace(/'/gu, "'\\''")}" >&2; exit 1; fi`,
|
||||
if [ "$1" = "-s" ]; then printf '%s\\n' "${stderr}" >&2; exit 1; fi`,
|
||||
);
|
||||
const projected = await refusedBy(providerOver(binary), {
|
||||
operation: "inspect_adb_device",
|
||||
@@ -873,6 +873,137 @@ exit 1`,
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
"pull_adb_file",
|
||||
"adb: error: failed to stat remote object /sdcard/unauthorized.bin: No such file or directory",
|
||||
"/sdcard/unauthorized.bin",
|
||||
],
|
||||
[
|
||||
"pull_adb_file",
|
||||
"adb: error: failed to open /sdcard/unauthorized.bin: Permission denied",
|
||||
"/sdcard/unauthorized.bin",
|
||||
],
|
||||
[
|
||||
"pull_adb_file",
|
||||
"* daemon not running; starting now at tcp:5037\n* daemon started successfully\nadb: error: device offline",
|
||||
"emulator-5554",
|
||||
],
|
||||
["pull_adb_file", "error: device unauthorized", "emulator-5554"],
|
||||
[
|
||||
"pull_adb_file",
|
||||
"* daemon not running. starting it now on port 5037 *\r\n* daemon started successfully *\r\nerror: device offline",
|
||||
"emulator-5554",
|
||||
],
|
||||
[
|
||||
"pull_adb_file",
|
||||
"* daemon started successfully *\r\nerror: device 'emulator-5554' not found",
|
||||
"emulator-5554",
|
||||
],
|
||||
[
|
||||
"push_adb_file",
|
||||
"adb: error: connect failed: device offline",
|
||||
"emulator-5554",
|
||||
],
|
||||
["push_adb_file", "error: device unauthorized", "emulator-5554"],
|
||||
["push_adb_file", "Permission denied", "/sdcard/unauthorized.bin"],
|
||||
] as const)(
|
||||
"attributes %s refusal %s to %s",
|
||||
async (operation, reason, path) => {
|
||||
const { binary, root } = await writeAdbStub(
|
||||
() => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi
|
||||
if [ "$1" != "-s" ] || [ "$2" != "emulator-5554" ]; then exit 91; fi
|
||||
if [ "$3" = "shell" ] && [ "$4" = "test" ] && [ "$6" = "/sdcard/unauthorized.bin" ]; then exit 1; fi
|
||||
if [ "$3:$4" = "pull:/sdcard/unauthorized.bin" ] || [ "$3:$5" = "push:/sdcard/unauthorized.bin" ]; then
|
||||
printf '%s\\n' '${reason.replaceAll("'", "'\\''")}' >&2
|
||||
exit 1
|
||||
fi
|
||||
exit 91`,
|
||||
);
|
||||
const localPath = join(root, "payload.bin");
|
||||
await writeFile(localPath, "payload");
|
||||
const input = {
|
||||
serial: "emulator-5554",
|
||||
device_path: "/sdcard/unauthorized.bin",
|
||||
};
|
||||
const projected = await refusedBy(
|
||||
providerOver(binary),
|
||||
operation === "pull_adb_file"
|
||||
? {
|
||||
operation,
|
||||
input: { ...input, output_directory: join(root, "pulls") },
|
||||
}
|
||||
: {
|
||||
operation,
|
||||
input: { ...input, local_path: localPath, overwrite: false },
|
||||
},
|
||||
);
|
||||
expect(projected).toMatchObject({
|
||||
code: "unsupported_target",
|
||||
details: { operation, path, reason },
|
||||
message: `Unsupported target for ${operation} at ${path}: ${reason}`,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
["pull_adb_file", "cannot create '%s': Permission denied"],
|
||||
["pull_adb_file", "cannot create file/directory '%s': Permission denied"],
|
||||
["pull_adb_file", "cannot write '%s': Permission denied"],
|
||||
["push_adb_file", "cannot stat '%s': No such file or directory"],
|
||||
["push_adb_file", "opening '%s' locally failed: Permission denied"],
|
||||
["push_adb_file", "reading '%s' locally failed: Permission denied"],
|
||||
["push_adb_file", "failed to read all of '%s': Permission denied"],
|
||||
] as const)(
|
||||
"keeps the actual host file identity for %s refusal %s",
|
||||
async (operation, diagnostic) => {
|
||||
const { binary, root } = await writeAdbStub(
|
||||
(stubRoot) => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi
|
||||
if [ "$1" != "-s" ] || [ "$2" != "emulator-5554" ]; then exit 91; fi
|
||||
if [ "$3" = "shell" ] && [ "$4" = "test" ]; then exit 1; fi
|
||||
if [ "$3" = "pull" ]; then
|
||||
printf '%s' "$5" > '${stubRoot}/host-path'
|
||||
printf "adb: error: ${diagnostic}\\n" "$5" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ "$3" = "push" ]; then
|
||||
printf '%s' "$4" > '${stubRoot}/host-path'
|
||||
printf "adb: error: ${diagnostic}\\n" "$4" >&2
|
||||
exit 1
|
||||
fi
|
||||
exit 91`,
|
||||
);
|
||||
const localPath = join(root, "payload.bin");
|
||||
await writeFile(localPath, "payload");
|
||||
const input = {
|
||||
serial: "emulator-5554",
|
||||
device_path: "/sdcard/sample.bin",
|
||||
};
|
||||
const projected = await refusedBy(
|
||||
providerOver(binary),
|
||||
operation === "pull_adb_file"
|
||||
? {
|
||||
operation,
|
||||
input: { ...input, output_directory: join(root, "pulls") },
|
||||
}
|
||||
: {
|
||||
operation,
|
||||
input: { ...input, local_path: localPath, overwrite: false },
|
||||
},
|
||||
);
|
||||
const hostPath = await readFile(join(root, "host-path"), "utf8");
|
||||
expect(projected).toMatchObject({
|
||||
code: "unsupported_target",
|
||||
details: {
|
||||
operation,
|
||||
path: hostPath,
|
||||
reason: expect.stringContaining(hostPath),
|
||||
},
|
||||
});
|
||||
expect(projected.message).toContain(`at ${hostPath}:`);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
["/data/local/tmp/..", "dot_segment"],
|
||||
["/data/local/tmp/", "empty_file_name"],
|
||||
@@ -1113,7 +1244,10 @@ if [ "$1" = "-s" ]; then printf '%s\\n' "$3" >> "${stubRoot}/calls"; printf '%s\
|
||||
},
|
||||
});
|
||||
// Existence is unknown, so nothing is claimed about the path and no push runs.
|
||||
expect(projected).toMatchObject({ code: "unsupported_target" });
|
||||
expect(projected).toMatchObject({
|
||||
code: "unsupported_target",
|
||||
details: { path: "emulator-5554", reason: stderr },
|
||||
});
|
||||
expect(projected.message).toContain(state);
|
||||
expect(projected.message).not.toContain("already exists");
|
||||
expect(await readFile(join(root, "calls"), "utf8")).toBe("shell\n");
|
||||
|
||||
+30
-16
@@ -1318,7 +1318,9 @@ export class AdbProvider {
|
||||
const shell = (arguments_: readonly string[], limits: AdbRunLimits) =>
|
||||
runAdb(
|
||||
operation,
|
||||
"serial" in request.input ? request.input.serial : operation,
|
||||
{
|
||||
device: "serial" in request.input ? request.input.serial : operation,
|
||||
},
|
||||
selection.binary,
|
||||
adbArguments(
|
||||
"serial" in request.input ? request.input.serial : undefined,
|
||||
@@ -1338,7 +1340,7 @@ export class AdbProvider {
|
||||
case "list_adb_devices": {
|
||||
const ran = await runAdb(
|
||||
operation,
|
||||
"adb-devices",
|
||||
{ device: "adb-devices" },
|
||||
selection.binary,
|
||||
["devices", "-l"],
|
||||
{
|
||||
@@ -1374,7 +1376,7 @@ export class AdbProvider {
|
||||
case "inspect_adb_device": {
|
||||
const ran = await runAdb(
|
||||
operation,
|
||||
request.input.serial,
|
||||
{ device: request.input.serial },
|
||||
selection.binary,
|
||||
adbArguments(request.input.serial, ["shell", "getprop"]),
|
||||
{
|
||||
@@ -1615,7 +1617,7 @@ export class AdbProvider {
|
||||
case "list_adb_directory": {
|
||||
const ran = await runAdbCapture(
|
||||
operation,
|
||||
request.input.serial,
|
||||
{ device: request.input.serial, path: request.input.device_path },
|
||||
selection.binary,
|
||||
adbArguments(request.input.serial, [
|
||||
"shell",
|
||||
@@ -1909,7 +1911,7 @@ export class AdbProvider {
|
||||
const { serial, package: packageName } = input;
|
||||
const paths = await runAdb(
|
||||
operation,
|
||||
serial,
|
||||
{ device: serial },
|
||||
selection.binary,
|
||||
adbArguments(serial, ["shell", "pm", "path", packageName]),
|
||||
{
|
||||
@@ -2129,7 +2131,11 @@ export class AdbProvider {
|
||||
const localPath = join(packageDirectory, entry.file_name);
|
||||
const pulled = await runAdb(
|
||||
operation,
|
||||
serial,
|
||||
{
|
||||
device: serial,
|
||||
path: entry.device_path,
|
||||
localPath,
|
||||
},
|
||||
selection.binary,
|
||||
adbArguments(serial, ["pull", entry.device_path, localPath]),
|
||||
{ timeoutMs: PULL_TIMEOUT_MS, maxOutputBytes: PULL_MAX_OUTPUT_BYTES },
|
||||
@@ -2302,7 +2308,11 @@ export class AdbProvider {
|
||||
if (existing !== undefined) return err(existing);
|
||||
const pulled = await runAdb(
|
||||
operation,
|
||||
input.serial,
|
||||
{
|
||||
device: input.serial,
|
||||
path: input.device_path,
|
||||
localPath,
|
||||
},
|
||||
selection.binary,
|
||||
adbArguments(input.serial, ["pull", input.device_path, localPath]),
|
||||
{ timeoutMs: PULL_TIMEOUT_MS, maxOutputBytes: PULL_MAX_OUTPUT_BYTES },
|
||||
@@ -2423,7 +2433,11 @@ export class AdbProvider {
|
||||
);
|
||||
const pushed = await runAdb(
|
||||
operation,
|
||||
input.device_path,
|
||||
{
|
||||
device: input.serial,
|
||||
path: input.device_path,
|
||||
localPath: digest.path,
|
||||
},
|
||||
selection.binary,
|
||||
adbArguments(input.serial, ["push", digest.path, input.device_path]),
|
||||
{ timeoutMs: PULL_TIMEOUT_MS, maxOutputBytes: PULL_MAX_OUTPUT_BYTES },
|
||||
@@ -2457,7 +2471,7 @@ export class AdbProvider {
|
||||
);
|
||||
const deviceDigest = await runAdb(
|
||||
operation,
|
||||
input.device_path,
|
||||
{ device: input.serial, path: input.device_path },
|
||||
selection.binary,
|
||||
adbArguments(input.serial, ["shell", "sha256sum", input.device_path]),
|
||||
{ timeoutMs: SHELL_TIMEOUT_MS, maxOutputBytes: 1024 },
|
||||
@@ -2563,7 +2577,7 @@ export class AdbProvider {
|
||||
);
|
||||
const installed = await runAdb(
|
||||
operation,
|
||||
input.serial,
|
||||
{ device: input.serial, path: apkPath, localPath: digest.path },
|
||||
selection.binary,
|
||||
adbArguments(input.serial, [
|
||||
"install",
|
||||
@@ -2621,7 +2635,7 @@ export class AdbProvider {
|
||||
): Promise<Result<AnalysisExecution, AnalysisError>> {
|
||||
const uninstalled = await runAdb(
|
||||
operation,
|
||||
input.serial,
|
||||
{ device: input.serial },
|
||||
selection.binary,
|
||||
adbArguments(input.serial, [
|
||||
"shell",
|
||||
@@ -2678,7 +2692,7 @@ export class AdbProvider {
|
||||
): Promise<Result<AnalysisExecution, AnalysisError>> {
|
||||
const resolved = await runAdb(
|
||||
operation,
|
||||
input.serial,
|
||||
{ device: input.serial },
|
||||
selection.binary,
|
||||
adbArguments(input.serial, [
|
||||
"shell",
|
||||
@@ -2816,7 +2830,7 @@ export class AdbProvider {
|
||||
): Promise<Result<AnalysisExecution, AnalysisError>> {
|
||||
const started = await runAdb(
|
||||
operation,
|
||||
serial,
|
||||
{ device: serial },
|
||||
selection.binary,
|
||||
adbArguments(serial, ["shell", "am", "start", "-W", ...arguments_]),
|
||||
{ timeoutMs: SHELL_TIMEOUT_MS, maxOutputBytes: PM_PATH_MAX_OUTPUT_BYTES },
|
||||
@@ -2885,7 +2899,7 @@ export class AdbProvider {
|
||||
): Promise<Result<AnalysisExecution, AnalysisError>> {
|
||||
const stopped = await runAdb(
|
||||
operation,
|
||||
input.serial,
|
||||
{ device: input.serial },
|
||||
selection.binary,
|
||||
adbArguments(input.serial, ["shell", "am", "force-stop", input.package]),
|
||||
{ timeoutMs: SHELL_TIMEOUT_MS, maxOutputBytes: PM_PATH_MAX_OUTPUT_BYTES },
|
||||
@@ -2927,7 +2941,7 @@ export class AdbProvider {
|
||||
if (existing !== undefined) return err(existing);
|
||||
const captured = await runAdbCapture(
|
||||
operation,
|
||||
input.serial,
|
||||
{ device: input.serial },
|
||||
selection.binary,
|
||||
adbArguments(input.serial, ["exec-out", "screencap", "-p"]),
|
||||
{ timeoutMs: SCREEN_TIMEOUT_MS, maxOutputBytes: SCREEN_MAX_OUTPUT_BYTES },
|
||||
@@ -3077,7 +3091,7 @@ export class AdbProvider {
|
||||
};
|
||||
const collected = await runAdb(
|
||||
operation,
|
||||
input.serial,
|
||||
{ device: input.serial },
|
||||
selection.binary,
|
||||
adbArguments(input.serial, ["bugreport", stagingDirectory]),
|
||||
{
|
||||
|
||||
@@ -169,6 +169,38 @@ const apkFixture = async (
|
||||
};
|
||||
};
|
||||
|
||||
it("returns table-only facts for a single selected primary table", async () => {
|
||||
const { command, root } = await writeApktoolStub(decodeStub());
|
||||
const apk = await apkFixture(root);
|
||||
const provider = providerOver(command);
|
||||
onTestFinished(() => provider.close());
|
||||
const outcome = await new ApktoolResourceAnalysisService(provider).execute(
|
||||
"decode_android_resources",
|
||||
{ path: apk.path, string_tables_only: true },
|
||||
);
|
||||
if (!outcome.ok) throw outcome.error;
|
||||
expect(outcome.value).toMatchObject({
|
||||
parameters: { path: apk.path, string_tables_only: true },
|
||||
subject: { digest: { sha256: apk.sha256 } },
|
||||
normalized_result: {
|
||||
strings: [],
|
||||
string_tables: [
|
||||
{
|
||||
locale: null,
|
||||
status: "present",
|
||||
strings: [
|
||||
{ name: "api_endpoint", value: "https://api.example.com/v1" },
|
||||
{ name: "app_name", value: "ApktoolProbe" },
|
||||
],
|
||||
total_count: 2,
|
||||
unparsed_count: 0,
|
||||
coverage: "complete",
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
[false, false],
|
||||
[true, false],
|
||||
|
||||
@@ -660,7 +660,7 @@ export class ApktoolProvider {
|
||||
: "complete";
|
||||
|
||||
completedDecode.strings = [...projectedStrings.strings];
|
||||
if (input.locales !== undefined)
|
||||
if (input.locales !== undefined || input.string_tables_only === true)
|
||||
completedDecode.string_tables = [...projectedStrings.tables];
|
||||
completedDecode.coverage = coverage;
|
||||
if (!projection.ok) throw projection.error.cause;
|
||||
|
||||
@@ -18,30 +18,66 @@ const textRead = (text: string) => ({
|
||||
sourceBytes: Buffer.byteLength(text, "utf8"),
|
||||
});
|
||||
|
||||
describe("physical Apktool string facts", () => {
|
||||
it("reads one physical table for two caller-selected aliases", async () => {
|
||||
const paths: string[] = [];
|
||||
const outcome = await projectApktoolStringTables(
|
||||
{
|
||||
async readText(path) {
|
||||
paths.push(path);
|
||||
return textRead(xml(1));
|
||||
},
|
||||
it("charges table-only output without reserving a duplicate primary copy", async () => {
|
||||
const paths: string[] = [];
|
||||
const outcome = await projectApktoolStringTables(
|
||||
{
|
||||
async readText(path) {
|
||||
paths.push(path);
|
||||
return textRead(xml(MAX_STRINGS));
|
||||
},
|
||||
"decoded",
|
||||
{ include_strings: true, locale: "pt-BR", locales: ["pt-rBR"] },
|
||||
);
|
||||
if (!outcome.ok) throw outcome.error.cause;
|
||||
const result = outcome.value;
|
||||
expect(paths).toEqual([
|
||||
join("decoded", "res", "values-pt-rBR", "strings.xml"),
|
||||
]);
|
||||
expect(result.tables.map((table) => table.locale)).toEqual([
|
||||
"pt-BR",
|
||||
"pt-rBR",
|
||||
]);
|
||||
expect(result.tables.map((table) => table.total_count)).toEqual([1, 1]);
|
||||
});
|
||||
},
|
||||
"decoded",
|
||||
{
|
||||
include_strings: true,
|
||||
locales: ["de", "es", "it", "fr"],
|
||||
string_tables_only: true,
|
||||
},
|
||||
);
|
||||
if (!outcome.ok) throw outcome.error.cause;
|
||||
expect(paths).toHaveLength(5);
|
||||
expect(outcome.value.strings).toEqual([]);
|
||||
expect(outcome.value.tables.map((table) => table.status)).toEqual(
|
||||
Array.from({ length: 5 }, () => "present"),
|
||||
);
|
||||
expect(
|
||||
outcome.value.tables.reduce((sum, table) => sum + table.strings.length, 0),
|
||||
).toBe(MAX_AGGREGATE_STRINGS);
|
||||
expect(outcome.value.coverage).toBe("complete");
|
||||
});
|
||||
|
||||
describe("physical Apktool string facts", () => {
|
||||
it.each([false, true])(
|
||||
"reads one physical table for two caller-selected aliases (table-only: %s)",
|
||||
async (tablesOnly) => {
|
||||
const paths: string[] = [];
|
||||
const outcome = await projectApktoolStringTables(
|
||||
{
|
||||
async readText(path) {
|
||||
paths.push(path);
|
||||
return textRead(xml(1));
|
||||
},
|
||||
},
|
||||
"decoded",
|
||||
{
|
||||
include_strings: true,
|
||||
locale: "pt-BR",
|
||||
locales: ["pt-rBR"],
|
||||
string_tables_only: tablesOnly,
|
||||
},
|
||||
);
|
||||
if (!outcome.ok) throw outcome.error.cause;
|
||||
const result = outcome.value;
|
||||
expect(paths).toEqual([
|
||||
join("decoded", "res", "values-pt-rBR", "strings.xml"),
|
||||
]);
|
||||
expect(result.tables.map((table) => table.locale)).toEqual([
|
||||
"pt-BR",
|
||||
"pt-rBR",
|
||||
]);
|
||||
expect(result.tables.map((table) => table.total_count)).toEqual([1, 1]);
|
||||
},
|
||||
);
|
||||
|
||||
it("counts complete source entries beyond the retained table prefix", async () => {
|
||||
const outcome = await projectApktoolStringTables(
|
||||
@@ -138,50 +174,71 @@ describe("physical Apktool string facts", () => {
|
||||
});
|
||||
|
||||
describe("Apktool string projection admission", () => {
|
||||
it("retains completed tables when cancellation arrives before the next selection", async () => {
|
||||
const controller = new AbortController();
|
||||
const reason = { selected: "caller" };
|
||||
const paths: string[] = [];
|
||||
const result = await projectApktoolStringTables(
|
||||
{
|
||||
async readText(path) {
|
||||
paths.push(path);
|
||||
controller.abort(reason);
|
||||
return textRead(xml(1));
|
||||
it.each([false, true])(
|
||||
"retains completed tables when cancellation arrives before the next selection (table-only: %s)",
|
||||
async (tablesOnly) => {
|
||||
const controller = new AbortController();
|
||||
const reason = { selected: "caller" };
|
||||
const paths: string[] = [];
|
||||
const result = await projectApktoolStringTables(
|
||||
{
|
||||
async readText(path) {
|
||||
paths.push(path);
|
||||
controller.abort(reason);
|
||||
return textRead(xml(1));
|
||||
},
|
||||
},
|
||||
},
|
||||
"decoded",
|
||||
{ include_strings: true, locales: ["de", "fr"] },
|
||||
controller.signal,
|
||||
);
|
||||
if (result.ok) throw new Error("Expected cancellation");
|
||||
expect(result.error.cause).toBeInstanceOf(AnalysisCancelledError);
|
||||
expect(result.error.cause).toHaveProperty("cause", reason);
|
||||
expect(paths).toHaveLength(1);
|
||||
expect(result.error.partial).toMatchObject({
|
||||
coverage: "partial",
|
||||
tables: [{ locale: null, total_count: 1 }],
|
||||
});
|
||||
expect(result.error.partial.limitations).toContain(
|
||||
'String projection stopped at locale "de"; this selection and 1 remaining selections were not projected',
|
||||
);
|
||||
});
|
||||
"decoded",
|
||||
{
|
||||
include_strings: true,
|
||||
locales: ["de", "fr"],
|
||||
string_tables_only: tablesOnly,
|
||||
},
|
||||
controller.signal,
|
||||
);
|
||||
if (result.ok) throw new Error("Expected cancellation");
|
||||
expect(result.error.cause).toBeInstanceOf(AnalysisCancelledError);
|
||||
expect(result.error.cause).toHaveProperty("cause", reason);
|
||||
expect(paths).toHaveLength(1);
|
||||
expect(result.error.partial).toMatchObject({
|
||||
coverage: "partial",
|
||||
tables: [{ locale: null, total_count: 1 }],
|
||||
});
|
||||
expect(result.error.partial.strings).toHaveLength(tablesOnly ? 0 : 1);
|
||||
expect(result.error.partial.limitations).toContain(
|
||||
'String projection stopped at locale "de"; this selection and 1 remaining selections were not projected',
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it("honors omitted strings without reading any selected table", async () => {
|
||||
const outcome = await projectApktoolStringTables(
|
||||
{
|
||||
async readText() {
|
||||
throw new Error("Unexpected read");
|
||||
it.each([false, true])(
|
||||
"honors omitted strings without reading any selected table (table-only: %s)",
|
||||
async (tablesOnly) => {
|
||||
const outcome = await projectApktoolStringTables(
|
||||
{
|
||||
async readText() {
|
||||
throw new Error("Unexpected read");
|
||||
},
|
||||
},
|
||||
},
|
||||
"decoded",
|
||||
{ include_strings: false, locale: "de", locales: ["fr"] },
|
||||
);
|
||||
expect(outcome).toEqual({
|
||||
ok: true,
|
||||
value: { strings: [], tables: [], limitations: [], coverage: "complete" },
|
||||
});
|
||||
});
|
||||
"decoded",
|
||||
{
|
||||
include_strings: false,
|
||||
locale: "de",
|
||||
locales: ["fr"],
|
||||
string_tables_only: tablesOnly,
|
||||
},
|
||||
);
|
||||
expect(outcome).toEqual({
|
||||
ok: true,
|
||||
value: {
|
||||
strings: [],
|
||||
tables: [],
|
||||
limitations: [],
|
||||
coverage: "complete",
|
||||
},
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("preserves the exact physical read failure", async () => {
|
||||
const cause = new Error("Physical read and close failed");
|
||||
|
||||
@@ -19,7 +19,7 @@ import type { ApktoolWorkspace } from "./ApktoolWorkspace.js";
|
||||
const STRINGS_XML_MAX_BYTES = 8 * 1024 * 1024;
|
||||
type Selection = Pick<
|
||||
Extract<ApktoolRequest, { operation: "decode_android_resources" }>["input"],
|
||||
"include_strings" | "locale" | "locales"
|
||||
"include_strings" | "locale" | "locales" | "string_tables_only"
|
||||
>;
|
||||
type PresentTable = {
|
||||
readonly status: "present";
|
||||
@@ -152,7 +152,8 @@ export const projectApktoolStringTables = async (
|
||||
"Aggregate string table projection stopped due to resource limits",
|
||||
);
|
||||
return {
|
||||
strings: tables[0]?.strings ?? [],
|
||||
strings:
|
||||
input.string_tables_only === true ? [] : (tables[0]?.strings ?? []),
|
||||
tables,
|
||||
limitations: [...limitations],
|
||||
coverage: tables.some((table) => table.coverage === "partial")
|
||||
@@ -223,9 +224,14 @@ export const projectApktoolStringTables = async (
|
||||
);
|
||||
continue;
|
||||
}
|
||||
// Batch output includes the primary strings twice, in the legacy field and
|
||||
// in string_tables. Charge that representation before selecting entries.
|
||||
const copies = index === 0 && input.locales !== undefined ? 2 : 1;
|
||||
// The default batch shape repeats primary entries in strings and tables;
|
||||
// table-only output charges one copy. Account before selecting entries.
|
||||
const copies =
|
||||
index === 0 &&
|
||||
input.locales !== undefined &&
|
||||
input.string_tables_only !== true
|
||||
? 2
|
||||
: 1;
|
||||
const remaining = Math.max(0, MAX_AGGREGATE_STRINGS - outputEntries);
|
||||
const selected = fact.entries.slice(0, Math.floor(remaining / copies));
|
||||
outputEntries += selected.length * copies;
|
||||
|
||||
@@ -4,8 +4,35 @@ import { join } from "node:path";
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
|
||||
import { resolveAppBundleExecutable } from "./AppBundleExecutable.js";
|
||||
|
||||
it.each([
|
||||
["macOS", ["Contents"], ["Contents", "MacOS"]],
|
||||
["iOS", [], []],
|
||||
] as const)(
|
||||
"resolves the %s bundle layout when Info.plist contains processing instructions",
|
||||
async (_kind, plistDirectory, programDirectory) => {
|
||||
const root = await createTestTempDirectory("rea-app-plist-pi-");
|
||||
const bundle = join(root, "Fixture.app");
|
||||
const plistPath = join(bundle, ...plistDirectory, "Info.plist");
|
||||
const executable = join(bundle, ...programDirectory, "Fixture");
|
||||
await mkdir(join(bundle, ...plistDirectory), { recursive: true });
|
||||
await mkdir(join(bundle, ...programDirectory), { recursive: true });
|
||||
await writeFile(
|
||||
plistPath,
|
||||
'<plist version="1.0"><?rea ignored?><dict><?rea ignored?><key>CFBundleExecutable</key><?rea ignored?><string>Fixture</string><?rea ignored?></dict><?rea ignored?></plist>',
|
||||
);
|
||||
await writeFile(executable, "inert fixture; never executed");
|
||||
|
||||
const result = await resolveAppBundleExecutable(bundle);
|
||||
expect(result).toEqual({
|
||||
ok: true,
|
||||
value: { executable, infoPlist: plistPath },
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
describe("app bundle Info.plist magic", () => {
|
||||
it("does not decode a high-bit fold of bplist as a binary Info.plist", async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), "rea-bplist-fold-"));
|
||||
|
||||
@@ -8,6 +8,14 @@ import {
|
||||
|
||||
const registration = { command: "rea", args: ["mcp"] };
|
||||
|
||||
const serializedText = (
|
||||
...input: Parameters<typeof serializeClientConfiguration>
|
||||
): string => {
|
||||
const serialized = serializeClientConfiguration(...input);
|
||||
if (!serialized.ok) throw new Error("TOML serialization was refused");
|
||||
return serialized.value;
|
||||
};
|
||||
|
||||
describe("TOML configuration source capacity", () => {
|
||||
it.each(["\n", "\r\n"])(
|
||||
"retains adjacent assignments and multiline values with %j breaks",
|
||||
@@ -29,18 +37,14 @@ describe("TOML configuration source capacity", () => {
|
||||
...parsed.servers,
|
||||
rea: registration,
|
||||
});
|
||||
const configured = serializeClientConfiguration(
|
||||
desired,
|
||||
"toml",
|
||||
original,
|
||||
);
|
||||
const configured = serializedText(desired, "toml", original);
|
||||
expect(
|
||||
configured.startsWith("\uFEFF" + prefix.replaceAll("\n", newline)),
|
||||
).toBe(true);
|
||||
expect(parseClientConfiguration(configured, "toml").document).toEqual(
|
||||
desired,
|
||||
);
|
||||
const removed = serializeClientConfiguration(
|
||||
const removed = serializedText(
|
||||
withClientServers(parsed, {}),
|
||||
"toml",
|
||||
configured,
|
||||
@@ -61,16 +65,12 @@ describe("TOML configuration source capacity", () => {
|
||||
...parsed.servers,
|
||||
rea: registration,
|
||||
});
|
||||
const configured = serializeClientConfiguration(desired, "toml", original);
|
||||
const configured = serializedText(desired, "toml", original);
|
||||
expect(configured).toContain(`NOTE="""${note}"""`);
|
||||
expect(parseClientConfiguration(configured, "toml").document).toEqual(
|
||||
desired,
|
||||
);
|
||||
const removed = serializeClientConfiguration(
|
||||
parsed.document,
|
||||
"toml",
|
||||
configured,
|
||||
);
|
||||
const removed = serializedText(parsed.document, "toml", configured);
|
||||
expect(removed).toContain(`NOTE="""${note}"""`);
|
||||
expect(parseClientConfiguration(removed, "toml").document).toEqual(
|
||||
parsed.document,
|
||||
@@ -87,7 +87,7 @@ describe("TOML configuration source capacity", () => {
|
||||
...parsed.servers,
|
||||
rea: registration,
|
||||
});
|
||||
const configured = serializeClientConfiguration(desired, "grok", original);
|
||||
const configured = serializedText(desired, "grok", original);
|
||||
const result = parseClientConfiguration(configured, "grok");
|
||||
expect(result.document.disabled_mcp_servers).toEqual(
|
||||
Array<string>(500_000).fill("other"),
|
||||
|
||||
@@ -15,16 +15,18 @@ import {
|
||||
import {
|
||||
applyEdits,
|
||||
createScanner,
|
||||
findNodeAtLocation,
|
||||
modify,
|
||||
format as formatJson,
|
||||
parse as parseJsonc,
|
||||
parseTree,
|
||||
printParseErrorCode,
|
||||
type Edit,
|
||||
type Node,
|
||||
type ParseError,
|
||||
type SyntaxKind,
|
||||
} from "jsonc-parser";
|
||||
import { z } from "zod";
|
||||
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { PRODUCT_IDENTITY } from "../identity.js";
|
||||
import type { SetupClient } from "./SupportedClients.js";
|
||||
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
|
||||
@@ -1312,6 +1314,37 @@ const nextJsoncToken = (
|
||||
return { kind, offset: scanner.getTokenOffset() };
|
||||
};
|
||||
|
||||
const isJsonMember = (property: Node, key: string): boolean =>
|
||||
property.children?.length === 2 && property.children[0]?.value === key;
|
||||
|
||||
/**
|
||||
* The property a client reads for `key`. `JSON.parse` and JSONC `parse` keep
|
||||
* the last duplicate member, while jsonc-parser's `findNodeAtLocation` and
|
||||
* `modify` select the first, so JSON text edits locate members only here.
|
||||
*/
|
||||
const effectiveJsonProperty = (object: Node, key: string): Node | undefined =>
|
||||
object.children?.findLast((property) => isJsonMember(property, key));
|
||||
|
||||
/** Apply one edit and format the lines it touched, as `modify` does. */
|
||||
const applyFormattedJsonEdit = (text: string, edit: Edit): string => {
|
||||
const edited = applyEdits(text, [edit]);
|
||||
let begin = edit.offset;
|
||||
let end = edit.offset + edit.content.length;
|
||||
if (edit.length === 0) {
|
||||
while (begin > 0 && !"\r\n".includes(edited.charAt(begin - 1))) begin -= 1;
|
||||
while (end < edited.length && !"\r\n".includes(edited.charAt(end)))
|
||||
end += 1;
|
||||
}
|
||||
return applyEdits(
|
||||
edited,
|
||||
formatJson(
|
||||
edited,
|
||||
{ offset: begin, length: end - begin },
|
||||
{ insertSpaces: true, tabSize: 2, eol: "\n", keepLines: false },
|
||||
),
|
||||
);
|
||||
};
|
||||
|
||||
/**
|
||||
* Delete a property by its JSONC AST span and the separator needed to keep its
|
||||
* parent object valid. All whitespace and comments outside those tokens stay
|
||||
@@ -1319,80 +1352,136 @@ const nextJsoncToken = (
|
||||
*/
|
||||
const deleteJsonProperty = (
|
||||
text: string,
|
||||
path: readonly string[],
|
||||
): string | undefined => {
|
||||
const root = parseTree(text, [], { allowTrailingComma: true });
|
||||
const value =
|
||||
root === undefined ? undefined : findNodeAtLocation(root, [...path]);
|
||||
const property = value?.parent;
|
||||
const parent = property?.parent;
|
||||
object: Node,
|
||||
property: Node,
|
||||
): string => {
|
||||
const siblings = object.children ?? [];
|
||||
const propertyIndex = siblings.indexOf(property);
|
||||
const previous = siblings[propertyIndex - 1];
|
||||
const next = siblings[propertyIndex + 1];
|
||||
const comma = nextJsoncToken(
|
||||
text,
|
||||
previous === undefined
|
||||
? property.offset + property.length
|
||||
: previous.offset + previous.length,
|
||||
);
|
||||
if (
|
||||
property === undefined ||
|
||||
parent?.type !== "object" ||
|
||||
property.type !== "property" ||
|
||||
parent.children === undefined
|
||||
propertyIndex < 0 ||
|
||||
(comma.kind !== JSON_COMMA &&
|
||||
(previous !== undefined || next !== undefined))
|
||||
)
|
||||
return undefined;
|
||||
const propertyIndex = parent.children.indexOf(property);
|
||||
if (propertyIndex < 0) return undefined;
|
||||
const previous = parent.children[propertyIndex - 1];
|
||||
const next = parent.children[propertyIndex + 1];
|
||||
let commaOffset: number | undefined;
|
||||
if (previous !== undefined) {
|
||||
const comma = nextJsoncToken(text, previous.offset + previous.length);
|
||||
if (comma.kind !== JSON_COMMA) return undefined;
|
||||
commaOffset = comma.offset;
|
||||
} else {
|
||||
const token = nextJsoncToken(text, property.offset + property.length);
|
||||
if (token.kind === JSON_COMMA) commaOffset = token.offset;
|
||||
else if (next !== undefined) return undefined;
|
||||
}
|
||||
throw new TypeError("JSON property separator was not found");
|
||||
const edits = [
|
||||
{ offset: property.offset, length: property.length, content: "" },
|
||||
...(commaOffset === undefined
|
||||
? []
|
||||
: [{ offset: commaOffset, length: 1, content: "" }]),
|
||||
...(comma.kind === JSON_COMMA
|
||||
? [{ offset: comma.offset, length: 1, content: "" }]
|
||||
: []),
|
||||
];
|
||||
return applyEdits(text, edits);
|
||||
};
|
||||
|
||||
/** Serialize a validated client document, retaining JSONC and TOML comments. */
|
||||
/** A JSON edit refused because it would activate a shadowed duplicate member. */
|
||||
export interface ClientConfigurationShadowedMember {
|
||||
readonly kind: "shadowed_member";
|
||||
/** Key path of the effective member whose removal was refused. */
|
||||
readonly path: readonly string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Set or delete the member a client reads at `path`, keeping every byte
|
||||
* outside that member. Shadowed duplicates stay in place, and deleting a
|
||||
* member that shadows one is refused so the earlier member never becomes
|
||||
* active, whatever its ownership.
|
||||
*/
|
||||
const editJsonMember = (
|
||||
text: string,
|
||||
path: readonly string[],
|
||||
value: unknown,
|
||||
): Result<string, ClientConfigurationShadowedMember> => {
|
||||
let object = parseTree(text, [], { allowTrailingComma: true });
|
||||
for (const [index, key] of path.entries()) {
|
||||
if (object?.type !== "object")
|
||||
throw new TypeError("JSON edit parent is not an object");
|
||||
const property = effectiveJsonProperty(object, key);
|
||||
const member = property?.children?.[1];
|
||||
if (property === undefined || member === undefined) {
|
||||
if (value === undefined) return ok(text);
|
||||
const content = `${JSON.stringify(key)}: ${JSON.stringify(
|
||||
path
|
||||
.slice(index + 1)
|
||||
.reduceRight<unknown>((nested, name) => ({ [name]: nested }), value),
|
||||
)}`;
|
||||
const previous = object.children?.at(-1);
|
||||
return ok(
|
||||
applyFormattedJsonEdit(
|
||||
text,
|
||||
previous === undefined
|
||||
? { offset: object.offset + 1, length: 0, content }
|
||||
: {
|
||||
offset: previous.offset + previous.length,
|
||||
length: 0,
|
||||
content: `,${content}`,
|
||||
},
|
||||
),
|
||||
);
|
||||
}
|
||||
if (index < path.length - 1) {
|
||||
object = member;
|
||||
continue;
|
||||
}
|
||||
if (value !== undefined)
|
||||
return ok(
|
||||
applyFormattedJsonEdit(text, {
|
||||
offset: member.offset,
|
||||
length: member.length,
|
||||
content: JSON.stringify(value),
|
||||
}),
|
||||
);
|
||||
if (
|
||||
object.children?.some(
|
||||
(sibling) => sibling !== property && isJsonMember(sibling, key),
|
||||
)
|
||||
)
|
||||
return err({ kind: "shadowed_member", path });
|
||||
return ok(deleteJsonProperty(text, object, property));
|
||||
}
|
||||
throw new TypeError("JSON edit path is empty");
|
||||
};
|
||||
|
||||
/**
|
||||
* Serialize a validated client document, retaining JSONC and TOML comments.
|
||||
* JSON edits fail only when they would activate a shadowed duplicate member.
|
||||
*/
|
||||
export const serializeClientConfiguration = (
|
||||
document: Record<string, unknown>,
|
||||
format: ClientConfigurationFormat | undefined,
|
||||
originalText?: string,
|
||||
editedPaths?: readonly (readonly string[])[],
|
||||
): string => {
|
||||
): Result<string, ClientConfigurationShadowedMember> => {
|
||||
if (format === undefined || format === "unsupported")
|
||||
throw new TypeError("client does not have a supported MCP config format");
|
||||
if (format === "toml")
|
||||
return serializeCodexConfiguration(document, originalText);
|
||||
return ok(serializeCodexConfiguration(document, originalText));
|
||||
if (format === "grok")
|
||||
return serializeGrokConfiguration(document, originalText);
|
||||
return ok(serializeGrokConfiguration(document, originalText));
|
||||
if (format === "hermes")
|
||||
return serializeHermesConfiguration(document, originalText);
|
||||
if (format === "dsh") return serializeDshDocument(document);
|
||||
return ok(serializeHermesConfiguration(document, originalText));
|
||||
if (format === "dsh") return ok(serializeDshDocument(document));
|
||||
// An empty original holds no comments or settings to preserve. Writing a
|
||||
// fresh document also keeps a lone BOM from shifting past the closing brace.
|
||||
if (
|
||||
originalText !== undefined &&
|
||||
!isEmptyClientConfigurationText(originalText) &&
|
||||
editedPaths !== undefined
|
||||
originalText === undefined ||
|
||||
isEmptyClientConfigurationText(originalText) ||
|
||||
editedPaths === undefined
|
||||
)
|
||||
return editedPaths.reduce((text, path) => {
|
||||
const value = valueAt(document, path);
|
||||
if (value === undefined) {
|
||||
const preserved = deleteJsonProperty(text, path);
|
||||
if (preserved !== undefined) return preserved;
|
||||
}
|
||||
return applyEdits(
|
||||
text,
|
||||
modify(text, [...path], value, {
|
||||
formattingOptions: { insertSpaces: true, tabSize: 2, eol: "\n" },
|
||||
}),
|
||||
);
|
||||
}, originalText);
|
||||
return `${JSON.stringify(document, null, 2)}\n`;
|
||||
return ok(`${JSON.stringify(document, null, 2)}\n`);
|
||||
let text = originalText;
|
||||
for (const path of editedPaths) {
|
||||
const edited = editJsonMember(text, path, valueAt(document, path));
|
||||
if (!edited.ok) return edited;
|
||||
text = edited.value;
|
||||
}
|
||||
return ok(text);
|
||||
};
|
||||
|
||||
/** Key path of a named registration in the server table. */
|
||||
|
||||
@@ -28,6 +28,7 @@ import {
|
||||
import type { VerifiedLinuxHopperPackage } from "./LinuxHopperPublisher.js";
|
||||
import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
|
||||
import * as nativeCommands from "../process/ExecFileOutput.js";
|
||||
import * as pythonRuntime from "./LinuxHopperPython.js";
|
||||
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
|
||||
|
||||
describe("Linux package-manager command display", () => {
|
||||
@@ -123,6 +124,7 @@ class RecordingLinuxHost implements LinuxHopperInstallHost {
|
||||
installPackage = (
|
||||
family: LinuxPackageFamily,
|
||||
archive: string,
|
||||
_options?: { readonly signal?: AbortSignal },
|
||||
): Promise<LinuxPackageInstallOutcome> => {
|
||||
if (!this.verifiedBeforeInstall)
|
||||
throw new Error("Package manager ran before publisher verification");
|
||||
@@ -260,6 +262,24 @@ describe("Linux Hopper installation", () => {
|
||||
});
|
||||
|
||||
describe("Linux Hopper package installation", () => {
|
||||
it("keeps runtime inspection failure distinct from package-manager failure and cleans the archive", async () => {
|
||||
const host = new RecordingLinuxHost();
|
||||
const failure = new pythonRuntime.LinuxHopperPythonError(
|
||||
"configuration_unknown",
|
||||
"Fixture unknown runtime",
|
||||
);
|
||||
host.installPackage = async () => ({
|
||||
status: "runtime_unavailable",
|
||||
failure,
|
||||
});
|
||||
expect(await installLinuxHopper(host)).toMatchObject({
|
||||
status: "failed",
|
||||
reason: "runtime_dependencies",
|
||||
pythonRuntimeFailure: failure,
|
||||
});
|
||||
expect(host.cleaned).toEqual(["/tmp/rea test"]);
|
||||
});
|
||||
|
||||
it.each(["deb", "rpm", "arch"] as const)(
|
||||
"downloads, verifies, installs, reads back, and cleans a %s package",
|
||||
async (family) => {
|
||||
@@ -426,10 +446,30 @@ describe("Linux Hopper cancellation", () => {
|
||||
).toEqual({
|
||||
status: "failed",
|
||||
reason: "cancelled",
|
||||
cancellationReason: controller.signal.reason,
|
||||
});
|
||||
expect(host.downloads).toEqual([]);
|
||||
expect(host.installed).toEqual([]);
|
||||
});
|
||||
|
||||
it("forwards cancellation to package stages and retains its original cause through cleanup", async () => {
|
||||
const host = new RecordingLinuxHost();
|
||||
const controller = new AbortController();
|
||||
const cause = new Error("caller cancelled between package transactions");
|
||||
host.installPackage = async (_family, _archive, options) => {
|
||||
expect(options?.signal).toBe(controller.signal);
|
||||
expect(host.verifiedBeforeInstall).toBe(true);
|
||||
controller.abort(cause);
|
||||
return { status: "cancelled", cause };
|
||||
};
|
||||
const result = await installLinuxHopper(host, {
|
||||
signal: controller.signal,
|
||||
});
|
||||
expect(result).toMatchObject({ status: "failed", reason: "cancelled" });
|
||||
if (result.status === "failed")
|
||||
expect(result.cancellationReason).toBe(cause);
|
||||
expect(host.cleaned).toEqual(["/tmp/rea test"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Linux Hopper temporary cleanup ownership", () => {
|
||||
|
||||
+178
-41
@@ -1,11 +1,16 @@
|
||||
import { access, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { join, posix } from "node:path";
|
||||
|
||||
import { execFileBoundedOutput } from "../process/ExecFileBoundedOutput.js";
|
||||
import { execFileOutput } from "../process/ExecFileOutput.js";
|
||||
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
|
||||
import { downloadHopperPackage } from "./HopperDownload.js";
|
||||
import {
|
||||
inspectLinuxHopperPythonRuntime,
|
||||
linuxHopperPythonUnavailable,
|
||||
LinuxHopperPythonError,
|
||||
} from "./LinuxHopperPython.js";
|
||||
import { RegularFileCleanupFailure } from "../filesystem/OwnedRegularFile.js";
|
||||
import { err, type Result } from "../domain/result.js";
|
||||
import {
|
||||
@@ -29,7 +34,7 @@ export type { LinuxPackageFamily } from "./HopperReleaseFeed.js";
|
||||
/** Package-manager output retained per stream without stopping the command. */
|
||||
const PACKAGE_MANAGER_OUTPUT_BYTES = 64 * 1024;
|
||||
|
||||
/** Exact stable external effects disclosed before a Linux Hopper installation. */
|
||||
/** Package mutations disclosed before a Linux Hopper installation. */
|
||||
export interface LinuxHopperInstallDisclosure {
|
||||
readonly networkOrigins: readonly string[];
|
||||
readonly integrity: string;
|
||||
@@ -48,11 +53,21 @@ export const linuxHopperInstallDisclosure = (
|
||||
HOPPER_DOWNLOAD_PREFIX,
|
||||
],
|
||||
integrity: `Detached PGP signature · primary ${HOPPER_LINUX_PRIMARY_FINGERPRINT} · package byte identity and installed launcher SHA-256 from the official feed`,
|
||||
commands: linuxPackageManagerCommands(
|
||||
family,
|
||||
`<verified-hopper.${family}>`,
|
||||
isRoot,
|
||||
).map(({ executable, args }) => formatCommandForDisplay(executable, args)),
|
||||
commands: [
|
||||
...linuxPackageManagerCommands(
|
||||
family,
|
||||
`<verified-hopper.${family}>`,
|
||||
isRoot,
|
||||
),
|
||||
...(family === "deb"
|
||||
? [
|
||||
debianPythonLibraryCommand(
|
||||
"<libpython matching /usr/bin/python3, if not loadable>",
|
||||
isRoot,
|
||||
),
|
||||
]
|
||||
: []),
|
||||
].map(({ executable, args }) => formatCommandForDisplay(executable, args)),
|
||||
};
|
||||
};
|
||||
|
||||
@@ -108,13 +123,21 @@ export interface LinuxPackageInstallFailure {
|
||||
}
|
||||
|
||||
/** Whether the package-manager step installed the verified package. */
|
||||
export type LinuxPackageInstallOutcome =
|
||||
type LinuxPackageCommandOutcome =
|
||||
| { readonly status: "installed" }
|
||||
| {
|
||||
readonly status: "failed";
|
||||
readonly failure: LinuxPackageInstallFailure;
|
||||
};
|
||||
|
||||
export type LinuxPackageInstallOutcome =
|
||||
| LinuxPackageCommandOutcome
|
||||
| { readonly status: "cancelled"; readonly cause: unknown }
|
||||
| {
|
||||
readonly status: "runtime_unavailable";
|
||||
readonly failure: LinuxHopperPythonError;
|
||||
};
|
||||
|
||||
/** External effects required to install the official Linux package. */
|
||||
export interface LinuxHopperInstallHost {
|
||||
distribution(): Promise<LinuxDistribution | undefined>;
|
||||
@@ -131,6 +154,7 @@ export interface LinuxHopperInstallHost {
|
||||
installPackage(
|
||||
family: LinuxPackageFamily,
|
||||
archive: string,
|
||||
options?: { readonly signal?: AbortSignal },
|
||||
): Promise<LinuxPackageInstallOutcome>;
|
||||
verifyArchive(
|
||||
family: LinuxPackageFamily,
|
||||
@@ -179,6 +203,10 @@ export type LinuxHopperInstallResult =
|
||||
readonly packageFailure?: LinuxPackageInstallFailure;
|
||||
/** Publisher-owned failure, retained for the public setup diagnostic. */
|
||||
readonly verificationFailure?: HopperReleaseError;
|
||||
/** System-interpreter inspection, kept separate from package-manager output. */
|
||||
readonly pythonRuntimeFailure?: LinuxHopperPythonError;
|
||||
/** Original caller cancellation, without flattening its value into text. */
|
||||
readonly cancellationReason?: unknown;
|
||||
} & { readonly cleanupFailure?: LinuxHopperTemporaryCleanupFailure });
|
||||
|
||||
/** Failed removal of an owned temporary install directory. */
|
||||
@@ -282,8 +310,8 @@ export const installLinuxHopper = async (
|
||||
readonly signal?: AbortSignal;
|
||||
} = {},
|
||||
): Promise<LinuxHopperInstallResult> => {
|
||||
if (options.signal?.aborted === true)
|
||||
return { status: "failed", reason: "cancelled" };
|
||||
if (isAborted(options.signal))
|
||||
return cancelledInstall(options.signal?.reason);
|
||||
const distribution = await host.distribution();
|
||||
if (distribution?.hopperSetupSupported !== true)
|
||||
return { status: "failed", reason: "unsupported_host" };
|
||||
@@ -339,18 +367,27 @@ export const installLinuxHopper = async (
|
||||
? "verification_unavailable"
|
||||
: retained.error.reason,
|
||||
};
|
||||
if (options.signal?.aborted === true)
|
||||
return { status: "failed", reason: "cancelled" };
|
||||
if (isAborted(options.signal))
|
||||
return cancelledInstall(options.signal?.reason);
|
||||
const install = await host.installPackage(
|
||||
distribution.packageFamily,
|
||||
archive,
|
||||
options,
|
||||
);
|
||||
if (install.status === "cancelled")
|
||||
return cancelledInstall(install.cause);
|
||||
if (install.status === "failed")
|
||||
return {
|
||||
status: "failed",
|
||||
reason: "authorization_or_package_manager",
|
||||
packageFailure: install.failure,
|
||||
};
|
||||
if (install.status === "runtime_unavailable")
|
||||
return {
|
||||
status: "failed",
|
||||
reason: "runtime_dependencies",
|
||||
pythonRuntimeFailure: install.failure,
|
||||
};
|
||||
const launcherPath = "/opt/hopper/bin/Hopper";
|
||||
const launcherStatus = await host.launcherStatus(
|
||||
launcherPath,
|
||||
@@ -367,9 +404,18 @@ export const installLinuxHopper = async (
|
||||
} catch (cause: unknown) {
|
||||
result = {
|
||||
status: "failed",
|
||||
reason: isAbortCause(cause) ? "cancelled" : "download",
|
||||
reason:
|
||||
isAborted(options.signal) || isAbortCause(cause)
|
||||
? "cancelled"
|
||||
: "download",
|
||||
};
|
||||
}
|
||||
if (
|
||||
result.status === "failed" &&
|
||||
result.reason === "cancelled" &&
|
||||
isAborted(options.signal)
|
||||
)
|
||||
result = { ...result, cancellationReason: options.signal?.reason };
|
||||
if (temporary === undefined) return result;
|
||||
try {
|
||||
await host.cleanup(temporary);
|
||||
@@ -418,8 +464,8 @@ export const systemLinuxHopperInstallHost = (
|
||||
download: downloadHopperPackage,
|
||||
createTemporaryDirectory: () => mkdtemp(join(tmpdir(), "rea-hopper-")),
|
||||
writeArchive: (path, bytes) => writeFile(path, bytes, { mode: 0o600 }),
|
||||
installPackage: (family, archive) =>
|
||||
installSystemPackage(family, archive, environment),
|
||||
installPackage: (family, archive, options) =>
|
||||
installSystemPackage(family, archive, environment, options?.signal),
|
||||
verifyArchive: (family, release, archive, signal) =>
|
||||
publisher.verifyArchive(family, release, archive, signal),
|
||||
retainVerifiedArchive: (verified, archive, signal) =>
|
||||
@@ -503,36 +549,130 @@ const installSystemPackage = async (
|
||||
family: LinuxPackageFamily,
|
||||
archive: string,
|
||||
environment: Readonly<NodeJS.ProcessEnv>,
|
||||
signal?: AbortSignal,
|
||||
): Promise<LinuxPackageInstallOutcome> => {
|
||||
for (const command of linuxPackageManagerCommands(
|
||||
family,
|
||||
archive,
|
||||
process.getuid?.() === 0,
|
||||
)) {
|
||||
const output = await execFileBoundedOutput(
|
||||
command.executable,
|
||||
command.args,
|
||||
{
|
||||
env: environment,
|
||||
limit: PACKAGE_MANAGER_OUTPUT_BYTES,
|
||||
},
|
||||
);
|
||||
if (output.exitCode !== 0) {
|
||||
return {
|
||||
const cancelled = (): LinuxPackageInstallOutcome => ({
|
||||
status: "cancelled",
|
||||
cause: signal?.reason,
|
||||
});
|
||||
const isRoot = process.getuid?.() === 0;
|
||||
for (const command of linuxPackageManagerCommands(family, archive, isRoot)) {
|
||||
if (isAborted(signal)) return cancelled();
|
||||
// Let a running package transaction settle; do not start another after
|
||||
// the caller cancels. Interrupting dpkg can leave configured state partial.
|
||||
const result = await runPackageManagerCommand(command, environment);
|
||||
if (result.status === "failed") return result;
|
||||
}
|
||||
if (isAborted(signal)) return cancelled();
|
||||
if (family !== "deb") return { status: "installed" };
|
||||
|
||||
// Ubuntu splits the executable/stdlib from the embedding library. Resolve
|
||||
// the installed distribution interpreter only after python3 is configured.
|
||||
const python = await inspectLinuxHopperPythonRuntime(environment, signal);
|
||||
if (isAborted(signal)) return cancelled();
|
||||
if (!python.ok) return pythonInspectionFailure(python.error);
|
||||
if (python.value.library.status === "loadable")
|
||||
return { status: "installed" };
|
||||
if (python.value.library.status === "unknown")
|
||||
return {
|
||||
status: "runtime_unavailable",
|
||||
failure: linuxHopperPythonUnavailable(python.value.library),
|
||||
};
|
||||
const packageName = "libpython" + python.value.version.join(".");
|
||||
const libraryPath = python.value.library.path;
|
||||
// Only the standard Ubuntu shared ABI/layout establishes this APT mapping.
|
||||
// Preserve alternate prefixes/SONAMEs as unknown instead of guessing a package.
|
||||
if (
|
||||
posix.basename(libraryPath) !== packageName + ".so.1.0" ||
|
||||
!/^\/usr\/lib\/[a-z0-9_]+-linux-gnu[a-z0-9_]*$/u.test(
|
||||
posix.dirname(libraryPath),
|
||||
)
|
||||
)
|
||||
return {
|
||||
status: "runtime_unavailable",
|
||||
failure: new LinuxHopperPythonError(
|
||||
"configuration_unknown",
|
||||
"Cannot determine the matching APT runtime package for " +
|
||||
libraryPath +
|
||||
". Repair this interpreter's shared runtime before retrying.",
|
||||
),
|
||||
};
|
||||
if (isAborted(signal)) return cancelled();
|
||||
const installed = await runPackageManagerCommand(
|
||||
debianPythonLibraryCommand(packageName, isRoot),
|
||||
environment,
|
||||
);
|
||||
if (installed.status === "failed") return installed;
|
||||
if (isAborted(signal)) return cancelled();
|
||||
// A successful APT exit is not proof that the selected runtime can load.
|
||||
const verified = await inspectLinuxHopperPythonRuntime(environment, signal);
|
||||
if (isAborted(signal)) return cancelled();
|
||||
if (!verified.ok) return pythonInspectionFailure(verified.error);
|
||||
if (verified.value.library.status !== "loadable")
|
||||
return {
|
||||
status: "runtime_unavailable",
|
||||
failure: linuxHopperPythonUnavailable(verified.value.library),
|
||||
};
|
||||
return { status: "installed" };
|
||||
};
|
||||
|
||||
const isAborted = (signal: AbortSignal | undefined): boolean =>
|
||||
signal?.aborted === true;
|
||||
|
||||
const cancelledInstall = (cause: unknown): LinuxHopperInstallResult => ({
|
||||
status: "failed",
|
||||
reason: "cancelled",
|
||||
cancellationReason: cause,
|
||||
});
|
||||
|
||||
const pythonInspectionFailure = (
|
||||
failure: LinuxHopperPythonError,
|
||||
): LinuxPackageInstallOutcome =>
|
||||
failure.reason === "cancelled"
|
||||
? { status: "cancelled", cause: failure.cause }
|
||||
: { status: "runtime_unavailable", failure };
|
||||
|
||||
interface PackageCommand {
|
||||
readonly executable: string;
|
||||
readonly args: readonly string[];
|
||||
}
|
||||
|
||||
const withPackagePrivilege = (
|
||||
executable: string,
|
||||
args: readonly string[],
|
||||
isRoot: boolean,
|
||||
): PackageCommand =>
|
||||
isRoot
|
||||
? { executable, args }
|
||||
: { executable: "pkexec", args: [executable, ...args] };
|
||||
|
||||
const debianPythonLibraryCommand = (
|
||||
packageName: string,
|
||||
isRoot: boolean,
|
||||
): PackageCommand =>
|
||||
withPackagePrivilege("apt-get", ["install", "-y", packageName], isRoot);
|
||||
|
||||
const runPackageManagerCommand = async (
|
||||
command: PackageCommand,
|
||||
environment: Readonly<NodeJS.ProcessEnv>,
|
||||
): Promise<LinuxPackageCommandOutcome> => {
|
||||
const output = await execFileBoundedOutput(command.executable, command.args, {
|
||||
env: environment,
|
||||
limit: PACKAGE_MANAGER_OUTPUT_BYTES,
|
||||
});
|
||||
return output.exitCode === 0
|
||||
? { status: "installed" }
|
||||
: {
|
||||
status: "failed",
|
||||
failure: {
|
||||
command: formatCommandForDisplay(command.executable, command.args),
|
||||
exitCode: output.exitCode,
|
||||
// The streams stay separate; their original interleaved order
|
||||
// is not recoverable after capture.
|
||||
// Preserve the separately captured streams and truncation marker.
|
||||
stdout: output.stdout,
|
||||
stderr: output.stderr,
|
||||
outputTruncated: output.outputTruncated,
|
||||
},
|
||||
};
|
||||
}
|
||||
}
|
||||
return { status: "installed" };
|
||||
};
|
||||
|
||||
/** Preserve argument boundaries in a command shown to the user. */
|
||||
@@ -551,10 +691,7 @@ export const linuxPackageManagerCommands = (
|
||||
family: LinuxPackageFamily,
|
||||
archive: string,
|
||||
isRoot: boolean,
|
||||
): readonly {
|
||||
readonly executable: string;
|
||||
readonly args: readonly string[];
|
||||
}[] => {
|
||||
): readonly PackageCommand[] => {
|
||||
const runtimeDependencies =
|
||||
family === "deb"
|
||||
? ["xvfb", "xauth", "python3", "libx11-6", "libxtst6"]
|
||||
@@ -577,9 +714,9 @@ export const linuxPackageManagerCommands = (
|
||||
["pacman", "-U", "--noconfirm", archive],
|
||||
];
|
||||
return commands.map((command) => {
|
||||
const executable = isRoot ? command[0] : "pkexec";
|
||||
const executable = command[0];
|
||||
if (executable === undefined) throw new Error("package command is empty");
|
||||
return { executable, args: isRoot ? command.slice(1) : command };
|
||||
return withPackagePrivilege(executable, command.slice(1), isRoot);
|
||||
});
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
|
||||
import * as boundedCommands from "../process/ExecFileBoundedOutput.js";
|
||||
import { err, ok } from "../domain/result.js";
|
||||
import * as pythonRuntime from "./LinuxHopperPython.js";
|
||||
import {
|
||||
linuxHopperInstallDisclosure,
|
||||
systemLinuxHopperInstallHost,
|
||||
} from "./LinuxHopper.js";
|
||||
|
||||
const missingPython = {
|
||||
interpreter: "/usr/bin/python3",
|
||||
version: [3, 14],
|
||||
library: {
|
||||
status: "unavailable",
|
||||
path: "/usr/lib/x86_64-linux-gnu/libpython3.14.so.1.0",
|
||||
reason: "No such file",
|
||||
},
|
||||
} satisfies pythonRuntime.LinuxHopperPythonRuntime;
|
||||
const readyPython = {
|
||||
...missingPython,
|
||||
library: {
|
||||
status: "loadable",
|
||||
path: "/usr/lib/x86_64-linux-gnu/libpython3.14.so.1.0",
|
||||
},
|
||||
} satisfies pythonRuntime.LinuxHopperPythonRuntime;
|
||||
const commandSuccess = {
|
||||
exitCode: 0,
|
||||
stdout: "",
|
||||
stderr: "",
|
||||
outputTruncated: false,
|
||||
};
|
||||
|
||||
describe("Linux Hopper package runtime", () => {
|
||||
it.each([
|
||||
{
|
||||
name: "installs and verifies the matching future Python library",
|
||||
inspections: [ok(missingPython), ok(readyPython)],
|
||||
packageCalls: 2,
|
||||
status: "installed",
|
||||
},
|
||||
{
|
||||
name: "skips the extra APT command for an already loadable library",
|
||||
inspections: [ok(readyPython)],
|
||||
packageCalls: 1,
|
||||
status: "installed",
|
||||
},
|
||||
{
|
||||
name: "does not guess a package for an unknown library location",
|
||||
inspections: [
|
||||
ok({
|
||||
...missingPython,
|
||||
library: { status: "unknown", reason: "No INSTSONAME" },
|
||||
} satisfies pythonRuntime.LinuxHopperPythonRuntime),
|
||||
],
|
||||
packageCalls: 1,
|
||||
status: "runtime_unavailable",
|
||||
},
|
||||
{
|
||||
name: "rejects APT success when the library still cannot load",
|
||||
inspections: [ok(missingPython), ok(missingPython)],
|
||||
packageCalls: 2,
|
||||
status: "runtime_unavailable",
|
||||
},
|
||||
{
|
||||
name: "preserves an interpreter inspection failure",
|
||||
inspections: [
|
||||
err(
|
||||
new pythonRuntime.LinuxHopperPythonError(
|
||||
"query_failed",
|
||||
"Fixture interpreter unavailable",
|
||||
),
|
||||
),
|
||||
],
|
||||
packageCalls: 1,
|
||||
status: "runtime_unavailable",
|
||||
},
|
||||
])("$name", async ({ inspections, packageCalls, status }) => {
|
||||
const directory = await createTestTempDirectory(
|
||||
"rea-linux-python-install-",
|
||||
);
|
||||
const host = systemLinuxHopperInstallHost({ XDG_CACHE_HOME: directory });
|
||||
const command = vi
|
||||
.spyOn(boundedCommands, "execFileBoundedOutput")
|
||||
.mockResolvedValue(commandSuccess);
|
||||
const inspect = vi
|
||||
.spyOn(pythonRuntime, "inspectLinuxHopperPythonRuntime")
|
||||
.mockRejectedValue(new Error("Unexpected extra Python inspection"));
|
||||
for (const result of inspections) inspect.mockResolvedValueOnce(result);
|
||||
try {
|
||||
expect(
|
||||
await host.installPackage("deb", "/tmp/verified-hopper.deb"),
|
||||
).toMatchObject({ status });
|
||||
expect(command).toHaveBeenCalledTimes(packageCalls);
|
||||
expect(inspect).toHaveBeenCalledTimes(inspections.length);
|
||||
if (packageCalls === 2) {
|
||||
const isRoot = process.getuid?.() === 0;
|
||||
expect(command.mock.calls[1]?.slice(0, 2)).toEqual([
|
||||
isRoot ? "apt-get" : "pkexec",
|
||||
[...(isRoot ? [] : ["apt-get"]), "install", "-y", "libpython3.14"],
|
||||
]);
|
||||
}
|
||||
} finally {
|
||||
try {
|
||||
await host.close?.();
|
||||
} finally {
|
||||
inspect.mockRestore();
|
||||
command.mockRestore();
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("preserves a failed runtime APT command and does not verify it as installed", async () => {
|
||||
const directory = await createTestTempDirectory(
|
||||
"rea-linux-python-failure-",
|
||||
);
|
||||
const host = systemLinuxHopperInstallHost({ XDG_CACHE_HOME: directory });
|
||||
const failed = {
|
||||
exitCode: 67,
|
||||
stdout: "original package output",
|
||||
stderr: "original package error",
|
||||
outputTruncated: true,
|
||||
};
|
||||
const command = vi
|
||||
.spyOn(boundedCommands, "execFileBoundedOutput")
|
||||
.mockResolvedValueOnce(commandSuccess)
|
||||
.mockResolvedValueOnce(failed);
|
||||
const inspect = vi
|
||||
.spyOn(pythonRuntime, "inspectLinuxHopperPythonRuntime")
|
||||
.mockResolvedValue(ok(missingPython));
|
||||
try {
|
||||
expect(
|
||||
await host.installPackage("deb", "/tmp/verified-hopper.deb"),
|
||||
).toMatchObject({ status: "failed", failure: failed });
|
||||
expect(command).toHaveBeenCalledTimes(2);
|
||||
expect(inspect).toHaveBeenCalledTimes(1);
|
||||
} finally {
|
||||
try {
|
||||
await host.close?.();
|
||||
} finally {
|
||||
inspect.mockRestore();
|
||||
command.mockRestore();
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("Linux Hopper package mapping", () => {
|
||||
it.each([
|
||||
"/custom/libpython3.14.so.1.0",
|
||||
"/usr/lib/x86_64-linux-gnu/libpython3.14d.so.1.0",
|
||||
"/usr/lib/x86_64-linux-gnu/libpython3.13.so.1.0",
|
||||
])("keeps an unproven APT mapping unknown: %s", async (path) => {
|
||||
const directory = await createTestTempDirectory("rea-python-package-map-");
|
||||
const host = systemLinuxHopperInstallHost({ XDG_CACHE_HOME: directory });
|
||||
const command = vi
|
||||
.spyOn(boundedCommands, "execFileBoundedOutput")
|
||||
.mockResolvedValue(commandSuccess);
|
||||
const inspect = vi
|
||||
.spyOn(pythonRuntime, "inspectLinuxHopperPythonRuntime")
|
||||
.mockResolvedValue(
|
||||
ok({
|
||||
...missingPython,
|
||||
library: { ...missingPython.library, path },
|
||||
}),
|
||||
);
|
||||
try {
|
||||
expect(
|
||||
await host.installPackage("deb", "/tmp/verified-hopper.deb"),
|
||||
).toMatchObject({
|
||||
status: "runtime_unavailable",
|
||||
failure: {
|
||||
reason: "configuration_unknown",
|
||||
message: expect.stringContaining(path),
|
||||
},
|
||||
});
|
||||
expect(command).toHaveBeenCalledTimes(1);
|
||||
expect(inspect).toHaveBeenCalledTimes(1);
|
||||
} finally {
|
||||
try {
|
||||
await host.close?.();
|
||||
} finally {
|
||||
inspect.mockRestore();
|
||||
command.mockRestore();
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("Linux Hopper package cancellation and disclosure", () => {
|
||||
it.each([
|
||||
{ phase: "before", commands: 0, inspections: 0 },
|
||||
{ phase: "first transaction", commands: 1, inspections: 0 },
|
||||
{ phase: "query", commands: 1, inspections: 1 },
|
||||
{ phase: "runtime transaction", commands: 2, inspections: 1 },
|
||||
{ phase: "verification", commands: 2, inspections: 2 },
|
||||
])(
|
||||
"honors cancellation at $phase without starting the next stage",
|
||||
async ({ phase, commands, inspections }) => {
|
||||
const directory = await createTestTempDirectory(
|
||||
"rea-python-install-cancel-",
|
||||
);
|
||||
const host = systemLinuxHopperInstallHost({ XDG_CACHE_HOME: directory });
|
||||
const controller = new AbortController();
|
||||
const cause = new Error("caller cancelled at " + phase);
|
||||
let commandCount = 0;
|
||||
let inspectionCount = 0;
|
||||
const command = vi
|
||||
.spyOn(boundedCommands, "execFileBoundedOutput")
|
||||
.mockImplementation(async () => {
|
||||
commandCount++;
|
||||
if (
|
||||
(phase === "first transaction" && commandCount === 1) ||
|
||||
(phase === "runtime transaction" && commandCount === 2)
|
||||
)
|
||||
controller.abort(cause);
|
||||
return commandSuccess;
|
||||
});
|
||||
const inspect = vi
|
||||
.spyOn(pythonRuntime, "inspectLinuxHopperPythonRuntime")
|
||||
.mockImplementation(async (_environment, signal) => {
|
||||
expect(signal).toBe(controller.signal);
|
||||
inspectionCount++;
|
||||
if (
|
||||
phase === "query" ||
|
||||
(phase === "verification" && inspectionCount === 2)
|
||||
)
|
||||
controller.abort(cause);
|
||||
return ok(inspectionCount === 1 ? missingPython : readyPython);
|
||||
});
|
||||
if (phase === "before") controller.abort(cause);
|
||||
try {
|
||||
const result = await host.installPackage(
|
||||
"deb",
|
||||
"/tmp/verified-hopper.deb",
|
||||
{
|
||||
signal: controller.signal,
|
||||
},
|
||||
);
|
||||
expect(result).toEqual({ status: "cancelled", cause });
|
||||
if (result.status === "cancelled") expect(result.cause).toBe(cause);
|
||||
expect(command).toHaveBeenCalledTimes(commands);
|
||||
expect(inspect).toHaveBeenCalledTimes(inspections);
|
||||
} finally {
|
||||
try {
|
||||
await host.close?.();
|
||||
} finally {
|
||||
inspect.mockRestore();
|
||||
command.mockRestore();
|
||||
}
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("discloses the conditional privileged library install without returning probe code", () => {
|
||||
const disclosure = linuxHopperInstallDisclosure("deb", false);
|
||||
expect(disclosure.commands).toHaveLength(2);
|
||||
expect(disclosure.commands[1]).toBe(
|
||||
'pkexec apt-get install -y "<libpython matching /usr/bin/python3, if not loadable>"',
|
||||
);
|
||||
expect(linuxHopperInstallDisclosure("rpm", false).commands).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,257 @@
|
||||
import { existsSync } from "node:fs";
|
||||
import { access, readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
|
||||
import { execFileOutput } from "../process/ExecFileOutput.js";
|
||||
import { inspectLinuxHopperPythonRuntime } from "./LinuxHopperPython.js";
|
||||
|
||||
const response = (value: unknown) => {
|
||||
const stdout = JSON.stringify(value);
|
||||
return {
|
||||
stdout,
|
||||
stderr: "",
|
||||
stdoutBytes: Buffer.byteLength(stdout),
|
||||
stderrBytes: 0,
|
||||
};
|
||||
};
|
||||
const available = {
|
||||
version: [3, 14],
|
||||
library: { status: "loadable", path: "/usr/lib/libpython3.14.so.1.0" },
|
||||
};
|
||||
|
||||
describe("distribution Python runtime inspection", () => {
|
||||
it("preserves version and library identity without assuming Python 3.12", async () => {
|
||||
const execute = vi
|
||||
.fn<typeof execFileOutput>()
|
||||
.mockResolvedValue(response(available));
|
||||
const environment = {
|
||||
PYTHONHOME: "/caller/python",
|
||||
PYTHONPATH: "/caller/modules",
|
||||
};
|
||||
const result = await inspectLinuxHopperPythonRuntime(
|
||||
environment,
|
||||
undefined,
|
||||
execute,
|
||||
);
|
||||
expect(result).toEqual({
|
||||
ok: true,
|
||||
value: { ...available, interpreter: "/usr/bin/python3" },
|
||||
});
|
||||
expect(execute).toHaveBeenCalledWith(
|
||||
"/usr/bin/python3",
|
||||
expect.arrayContaining(["-I", "-S", "-B"]),
|
||||
expect.objectContaining({
|
||||
env: environment,
|
||||
timeout: 10_000,
|
||||
maxBuffer: 64 * 1024,
|
||||
stopSignal: "SIGTERM",
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ version: [2, 7], library: available.library },
|
||||
{ version: [3, "14"], library: available.library },
|
||||
{
|
||||
version: [3, 14],
|
||||
library: { status: "loadable", path: "relative/libpython.so" },
|
||||
},
|
||||
{
|
||||
version: [3, 14],
|
||||
library: { status: "loadable", path: "C:\\not-a-linux-path\\python.dll" },
|
||||
},
|
||||
{
|
||||
version: [3, 14],
|
||||
library: { status: "loadable", path: "/usr/lib/python\0.so" },
|
||||
},
|
||||
{
|
||||
version: [3, 14],
|
||||
library: { status: "unavailable", path: "/lib/python.so" },
|
||||
},
|
||||
null,
|
||||
])(
|
||||
"refuses malformed facts without reporting library absence: %j",
|
||||
async (value) => {
|
||||
const execute = vi
|
||||
.fn<typeof execFileOutput>()
|
||||
.mockResolvedValue(response(value));
|
||||
const result = await inspectLinuxHopperPythonRuntime(
|
||||
{},
|
||||
undefined,
|
||||
execute,
|
||||
);
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: { reason: "invalid_response" },
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("retains unknown configuration and observes a later repaired library afresh", async () => {
|
||||
const unknown = {
|
||||
version: [3, 14],
|
||||
library: { status: "unknown", reason: "No INSTSONAME" },
|
||||
};
|
||||
const execute = vi
|
||||
.fn<typeof execFileOutput>()
|
||||
.mockResolvedValueOnce(response(unknown))
|
||||
.mockResolvedValueOnce(response(available));
|
||||
expect(
|
||||
await inspectLinuxHopperPythonRuntime({}, undefined, execute),
|
||||
).toMatchObject({ ok: true, value: unknown });
|
||||
expect(
|
||||
await inspectLinuxHopperPythonRuntime({}, undefined, execute),
|
||||
).toMatchObject({ ok: true, value: available });
|
||||
expect(execute).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("distinguishes execution and JSON failures from an unavailable library", async () => {
|
||||
const execute = vi
|
||||
.fn<typeof execFileOutput>()
|
||||
.mockRejectedValueOnce(new Error("fixture execution failure"))
|
||||
.mockResolvedValueOnce({
|
||||
stdout: "{",
|
||||
stderr: "",
|
||||
stdoutBytes: 1,
|
||||
stderrBytes: 0,
|
||||
});
|
||||
expect(
|
||||
await inspectLinuxHopperPythonRuntime({}, undefined, execute),
|
||||
).toMatchObject({ ok: false, error: { reason: "query_failed" } });
|
||||
expect(
|
||||
await inspectLinuxHopperPythonRuntime({}, undefined, execute),
|
||||
).toMatchObject({ ok: false, error: { reason: "invalid_response" } });
|
||||
});
|
||||
});
|
||||
|
||||
describe("distribution Python process cleanup", () => {
|
||||
it.skipIf(process.platform !== "linux" || !existsSync("/usr/bin/python3"))(
|
||||
"waits for a delayed Python exit before completing cancellation",
|
||||
async () => {
|
||||
const directory = await createTestTempDirectory("rea-python-stop-");
|
||||
const ready = join(directory, "ready");
|
||||
const stopping = join(directory, "stopping");
|
||||
const release = join(directory, "release");
|
||||
const stopped = join(directory, "stopped");
|
||||
const program = `from pathlib import Path
|
||||
import signal, sys, time
|
||||
def stop(signum, frame):
|
||||
Path(${JSON.stringify(stopping)}).write_text(str(signum))
|
||||
while not Path(${JSON.stringify(release)}).exists():
|
||||
time.sleep(0.01)
|
||||
Path(${JSON.stringify(stopped)}).write_text(str(signum))
|
||||
sys.exit(0)
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
Path(${JSON.stringify(ready)}).write_text("ready")
|
||||
while True:
|
||||
time.sleep(0.1)
|
||||
`;
|
||||
const controller = new AbortController();
|
||||
const reason = { request: "stop owned Python" };
|
||||
let settled = false;
|
||||
const execution = inspectLinuxHopperPythonRuntime(
|
||||
process.env,
|
||||
controller.signal,
|
||||
(command, _arguments, options) =>
|
||||
execFileOutput(command, ["-I", "-S", "-B", "-c", program], options),
|
||||
).then((result) => {
|
||||
settled = true;
|
||||
return result;
|
||||
});
|
||||
try {
|
||||
await vi.waitFor(() => access(ready), { timeout: 5_000, interval: 10 });
|
||||
controller.abort(reason);
|
||||
await vi.waitFor(() => access(stopping), {
|
||||
timeout: 3_000,
|
||||
interval: 10,
|
||||
});
|
||||
// The child cannot exit until released; early settlement loses ownership.
|
||||
expect(settled).toBe(false);
|
||||
} finally {
|
||||
try {
|
||||
await writeFile(release, "release");
|
||||
} finally {
|
||||
controller.abort(reason);
|
||||
await execution;
|
||||
}
|
||||
await vi.waitFor(() => access(stopped), {
|
||||
timeout: 5_000,
|
||||
interval: 10,
|
||||
});
|
||||
}
|
||||
expect(await readFile(stopped, "utf8")).toBe("15");
|
||||
const result = await execution;
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: { reason: "cancelled" },
|
||||
});
|
||||
if (!result.ok) expect(result.error.cause).toBe(reason);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe("distribution Python isolation and cancellation", () => {
|
||||
it("preserves cancellation and starts no interpreter for an already cancelled query", async () => {
|
||||
const controller = new AbortController();
|
||||
const reason = new Error("caller cancellation");
|
||||
controller.abort(reason);
|
||||
const execute = vi.fn<typeof execFileOutput>();
|
||||
const result = await inspectLinuxHopperPythonRuntime(
|
||||
{},
|
||||
controller.signal,
|
||||
execute,
|
||||
);
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: { reason: "cancelled", cause: reason },
|
||||
});
|
||||
expect(execute).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("observes cancellation that arrives during interpreter execution", async () => {
|
||||
const controller = new AbortController();
|
||||
const reason = new Error("cancelled during query");
|
||||
const execute = vi
|
||||
.fn<typeof execFileOutput>()
|
||||
.mockImplementation(async () => {
|
||||
controller.abort(reason);
|
||||
return response(available);
|
||||
});
|
||||
const result = await inspectLinuxHopperPythonRuntime(
|
||||
{},
|
||||
controller.signal,
|
||||
execute,
|
||||
);
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: { reason: "cancelled", cause: reason },
|
||||
});
|
||||
});
|
||||
|
||||
it.skipIf(process.platform !== "linux" || !existsSync("/usr/bin/python3"))(
|
||||
"runs the real producer without loading caller Python modules or PYTHONHOME",
|
||||
async () => {
|
||||
const directory = await createTestTempDirectory("rea-system-python-");
|
||||
for (const name of ["ctypes.py", "sysconfig.py", "sitecustomize.py"])
|
||||
await writeFile(
|
||||
join(directory, name),
|
||||
"raise RuntimeError('caller module was loaded')\n",
|
||||
);
|
||||
const result = await inspectLinuxHopperPythonRuntime({
|
||||
...process.env,
|
||||
PYTHONPATH: directory,
|
||||
PYTHONHOME: join(directory, "missing-python-home"),
|
||||
});
|
||||
expect(result).toMatchObject({
|
||||
ok: true,
|
||||
value: {
|
||||
interpreter: "/usr/bin/python3",
|
||||
version: [3, expect.any(Number)],
|
||||
},
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,157 @@
|
||||
import { posix } from "node:path";
|
||||
|
||||
import { z } from "zod";
|
||||
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { execFileOutput } from "../process/ExecFileOutput.js";
|
||||
|
||||
const libraryPath = z
|
||||
.string()
|
||||
.refine((value) => posix.isAbsolute(value) && !value.includes("\0"));
|
||||
const runtimeSchema = z.object({
|
||||
version: z.tuple([z.literal(3), z.number().int().nonnegative()]),
|
||||
library: z.discriminatedUnion("status", [
|
||||
z.object({ status: z.literal("loadable"), path: libraryPath }),
|
||||
z.object({
|
||||
status: z.literal("unavailable"),
|
||||
path: libraryPath,
|
||||
reason: z.string().min(1),
|
||||
}),
|
||||
z.object({ status: z.literal("unknown"), reason: z.string().min(1) }),
|
||||
]),
|
||||
});
|
||||
|
||||
/** Facts reported by the isolated system interpreter, without a GUI launch. */
|
||||
export type LinuxHopperPythonRuntime = z.infer<typeof runtimeSchema> & {
|
||||
readonly interpreter: "/usr/bin/python3";
|
||||
};
|
||||
|
||||
/** A failed inspection does not establish that a shared library is absent. */
|
||||
export class LinuxHopperPythonError extends Error {
|
||||
readonly _tag = "LinuxHopperPythonError";
|
||||
|
||||
constructor(
|
||||
readonly reason:
|
||||
| "query_failed"
|
||||
| "invalid_response"
|
||||
| "configuration_unknown"
|
||||
| "runtime_unavailable"
|
||||
| "cancelled",
|
||||
message: string,
|
||||
cause?: unknown,
|
||||
) {
|
||||
super(message, { cause });
|
||||
}
|
||||
}
|
||||
|
||||
// Isolate caller Python paths/site hooks and disable bytecode writes. Inspect
|
||||
// the distribution interpreter that the package manager installs, not PATH.
|
||||
const query = `import ctypes, json, os, sys, sysconfig
|
||||
directory = sysconfig.get_config_var("LIBDIR")
|
||||
name = sysconfig.get_config_var("INSTSONAME")
|
||||
if not isinstance(directory, str) or not os.path.isabs(directory) or not isinstance(name, str) or not name:
|
||||
library = {"status": "unknown", "reason": "Python did not report an absolute shared-library location"}
|
||||
else:
|
||||
path = os.path.join(directory, name)
|
||||
try:
|
||||
ctypes.CDLL(path)
|
||||
library = {"status": "loadable", "path": path}
|
||||
except OSError as error:
|
||||
library = {"status": "unavailable", "path": path, "reason": str(error)}
|
||||
print(json.dumps({"version": list(sys.version_info[:2]), "library": library}))
|
||||
`;
|
||||
|
||||
const runtimeCommand = {
|
||||
executable: "/usr/bin/python3",
|
||||
args: ["-I", "-S", "-B", "-c", query],
|
||||
} as const;
|
||||
|
||||
// AbortSignal changes while an awaited process runs, despite its readonly type.
|
||||
const isAborted = (signal: AbortSignal | undefined): boolean =>
|
||||
signal?.aborted === true;
|
||||
|
||||
/** Fresh, bounded inspection; package or interpreter changes must be observable. */
|
||||
export const inspectLinuxHopperPythonRuntime = async (
|
||||
environment: Readonly<NodeJS.ProcessEnv>,
|
||||
signal?: AbortSignal,
|
||||
execute: typeof execFileOutput = execFileOutput,
|
||||
): Promise<Result<LinuxHopperPythonRuntime, LinuxHopperPythonError>> => {
|
||||
const cancelled = () =>
|
||||
err(
|
||||
new LinuxHopperPythonError(
|
||||
"cancelled",
|
||||
"Python runtime inspection was cancelled.",
|
||||
signal?.reason,
|
||||
),
|
||||
);
|
||||
if (isAborted(signal)) return cancelled();
|
||||
let output: string;
|
||||
try {
|
||||
const result = await execute(
|
||||
runtimeCommand.executable,
|
||||
runtimeCommand.args,
|
||||
{
|
||||
env: environment,
|
||||
signal,
|
||||
stopSignal: "SIGTERM",
|
||||
timeout: 10_000,
|
||||
maxBuffer: 64 * 1024,
|
||||
},
|
||||
);
|
||||
output = result.stdout;
|
||||
} catch (cause: unknown) {
|
||||
if (isAborted(signal)) return cancelled();
|
||||
return err(
|
||||
new LinuxHopperPythonError(
|
||||
"query_failed",
|
||||
"Could not inspect the shared runtime of /usr/bin/python3.",
|
||||
cause,
|
||||
),
|
||||
);
|
||||
}
|
||||
if (isAborted(signal)) return cancelled();
|
||||
let value: unknown;
|
||||
try {
|
||||
value = JSON.parse(output);
|
||||
} catch (cause: unknown) {
|
||||
return err(
|
||||
new LinuxHopperPythonError(
|
||||
"invalid_response",
|
||||
"System Python returned an invalid shared-runtime description.",
|
||||
cause,
|
||||
),
|
||||
);
|
||||
}
|
||||
const parsed = runtimeSchema.safeParse(value);
|
||||
if (!parsed.success)
|
||||
return err(
|
||||
new LinuxHopperPythonError(
|
||||
"invalid_response",
|
||||
"System Python returned an invalid shared-runtime description.",
|
||||
parsed.error,
|
||||
),
|
||||
);
|
||||
return ok({
|
||||
...parsed.data,
|
||||
interpreter: runtimeCommand.executable,
|
||||
});
|
||||
};
|
||||
|
||||
/** Explain a known failed load separately from an unknown configuration. */
|
||||
export const linuxHopperPythonUnavailable = (
|
||||
library: Exclude<
|
||||
LinuxHopperPythonRuntime["library"],
|
||||
{ readonly status: "loadable" }
|
||||
>,
|
||||
): LinuxHopperPythonError =>
|
||||
new LinuxHopperPythonError(
|
||||
library.status === "unknown"
|
||||
? "configuration_unknown"
|
||||
: "runtime_unavailable",
|
||||
library.status === "unavailable"
|
||||
? "The system Python shared runtime cannot load " +
|
||||
library.path +
|
||||
": " +
|
||||
library.reason
|
||||
: "System Python did not identify its shared runtime: " + library.reason,
|
||||
);
|
||||
@@ -250,16 +250,6 @@ const persistClientConfiguration = async ({
|
||||
};
|
||||
if (target.value) return { status: "failed", reason: "target_changed" };
|
||||
}
|
||||
let recoveryBackupPath: string | undefined;
|
||||
if (original !== undefined) {
|
||||
const preserved = await preserveCurrentRecoveryBackup(
|
||||
`${client.configPath}.rea.backup`,
|
||||
transactionPath,
|
||||
original,
|
||||
);
|
||||
if (!preserved.ok) return { status: "failed", reason: "backup" };
|
||||
recoveryBackupPath = preserved.value;
|
||||
}
|
||||
// A native OpenCode V2 entry replaces REA's legacy V1 entry, which would
|
||||
// otherwise conflict with it.
|
||||
const legacyPath = legacyClientServerPath(
|
||||
@@ -292,19 +282,43 @@ const persistClientConfiguration = async ({
|
||||
),
|
||||
}
|
||||
: registered;
|
||||
let serialized: ReturnType<typeof serializeClientConfiguration>;
|
||||
try {
|
||||
serialized = serializeClientConfiguration(document, format, original, [
|
||||
...Object.keys(fallbackServers)
|
||||
.filter((name) => name !== PRODUCT_IDENTITY.mcpServerKey)
|
||||
.map((name) => clientServerPath(parsed, name)),
|
||||
clientServerPath(parsed, PRODUCT_IDENTITY.mcpServerKey),
|
||||
...(legacyPath === undefined ? [] : [legacyPath]),
|
||||
...(enableOmp ? [[OMP_DISABLED_SERVERS_KEY]] : []),
|
||||
]);
|
||||
} catch (cause: unknown) {
|
||||
// An unserializable edit is reported before any file is written.
|
||||
void cause;
|
||||
return { status: "failed", reason: "write" };
|
||||
}
|
||||
if (!serialized.ok)
|
||||
return {
|
||||
status: "failed",
|
||||
reason: "shadowed_member",
|
||||
path: serialized.error.path,
|
||||
};
|
||||
let recoveryBackupPath: string | undefined;
|
||||
if (original !== undefined) {
|
||||
const preserved = await preserveCurrentRecoveryBackup(
|
||||
`${client.configPath}.rea.backup`,
|
||||
transactionPath,
|
||||
original,
|
||||
);
|
||||
if (!preserved.ok) return { status: "failed", reason: "backup" };
|
||||
recoveryBackupPath = preserved.value;
|
||||
}
|
||||
let publication: ClientConfigurationPublication;
|
||||
try {
|
||||
await mkdir(dirname(client.configPath), { recursive: true });
|
||||
publication = await ClientConfigurationPublication.write(
|
||||
transactionPath,
|
||||
serializeClientConfiguration(document, format, original, [
|
||||
...Object.keys(fallbackServers)
|
||||
.filter((name) => name !== PRODUCT_IDENTITY.mcpServerKey)
|
||||
.map((name) => clientServerPath(parsed, name)),
|
||||
clientServerPath(parsed, PRODUCT_IDENTITY.mcpServerKey),
|
||||
...(legacyPath === undefined ? [] : [legacyPath]),
|
||||
...(enableOmp ? [[OMP_DISABLED_SERVERS_KEY]] : []),
|
||||
]),
|
||||
serialized.value,
|
||||
original,
|
||||
0o600,
|
||||
);
|
||||
|
||||
@@ -20,6 +20,8 @@ export const failedConfigurationMessage = (
|
||||
return "Agent configuration could not be backed up, so no change was made. Check file permissions, then rerun setup.";
|
||||
case "write":
|
||||
return "Agent configuration could not be updated. Check file permissions, then rerun setup.";
|
||||
case "shadowed_member":
|
||||
return `Agent configuration repeats the member at ${JSON.stringify(result.path)}; removing the member the client reads would activate an earlier duplicate, so no change was made. Remove the duplicate members, then rerun setup.`;
|
||||
case "rollback":
|
||||
return result.backupPath === undefined
|
||||
? `Agent configuration rollback failed at ${result.failure.path}: ${result.failure.detail}; the failed first-run file may remain. Remove or repair it before rerunning setup.`
|
||||
|
||||
@@ -222,10 +222,13 @@ export const systemSetupHost = (
|
||||
result.packageFailure,
|
||||
"linux",
|
||||
);
|
||||
return result.verificationFailure !== undefined
|
||||
const detail =
|
||||
result.verificationFailure?.message ??
|
||||
result.pythonRuntimeFailure?.message;
|
||||
return detail !== undefined
|
||||
? {
|
||||
...failure,
|
||||
remediation: `${result.verificationFailure.message} ${failure.remediation}`,
|
||||
remediation: `${detail} ${failure.remediation}`,
|
||||
}
|
||||
: failure;
|
||||
}
|
||||
|
||||
@@ -136,7 +136,7 @@ export const setupInstallFailure = (
|
||||
status: "failed",
|
||||
code: "runtime_dependency_unavailable",
|
||||
remediation:
|
||||
"Hopper was installed but required shared libraries are missing. Rerun rea setup, then apply the hopper-demo-runtime remediation from rea doctor.",
|
||||
"Hopper's required runtime could not be verified. Inspect rea doctor --provider hopper; on a supported system use rea setup --install-hopper to repair its dependencies.",
|
||||
};
|
||||
case "unsupported_hopper_build":
|
||||
return {
|
||||
@@ -150,7 +150,7 @@ export const setupInstallFailure = (
|
||||
status: "failed",
|
||||
code: "setup_cancelled",
|
||||
remediation:
|
||||
"Setup was cancelled before Hopper was installed. Rerun rea setup when ready.",
|
||||
"Setup was cancelled. Run rea doctor to inspect the current state before retrying.",
|
||||
};
|
||||
case "destination_exists":
|
||||
return {
|
||||
|
||||
@@ -301,7 +301,7 @@ const setupPlan = (input: {
|
||||
: "system package manager",
|
||||
detail:
|
||||
input.platform === "linux"
|
||||
? `Download the current official Hopper demo, verify its PGP publisher and launcher bytes, and retain archive, public-key.asc and release.json under ${input.hopperCacheDirectory ?? `${input.homeDirectory}/.cache/rea/hopper-releases`}/<launcher-sha256> for offline verification. Preserve replaced damaged evidence beside it as <launcher-sha256>.backup-<id>. Install Hopper plus its Xvfb demo-session dependencies. REA uses a private display and selects Hopper's offered demo mode for each analysis session.`
|
||||
? `Download the current official Hopper demo, verify its PGP publisher and launcher bytes, and retain archive, public-key.asc and release.json under ${input.hopperCacheDirectory ?? `${input.homeDirectory}/.cache/rea/hopper-releases`}/<launcher-sha256> for offline verification. Preserve replaced damaged evidence beside it as <launcher-sha256>.backup-<id>. Install Hopper plus its Xvfb demo-session dependencies${input.linuxPackageFamily === "deb" ? ", including the shared library matching the distribution Python interpreter" : ""}. REA uses a private display and selects Hopper's offered demo mode for each analysis session.`
|
||||
: "Download the official Hopper package, verify it, and install it. Hopper may show its demo or license prompt when first opened.",
|
||||
external: true,
|
||||
operation: "install" as const,
|
||||
|
||||
@@ -37,6 +37,12 @@ export type ClientConfigurationResult =
|
||||
readonly status: "failed";
|
||||
readonly reason: "path" | "target_changed" | "backup" | "write";
|
||||
}
|
||||
| {
|
||||
readonly status: "failed";
|
||||
/** Removing the member a client reads would activate an earlier duplicate. */
|
||||
readonly reason: "shadowed_member";
|
||||
readonly path: readonly string[];
|
||||
}
|
||||
| {
|
||||
readonly status: "failed";
|
||||
readonly reason: "readback";
|
||||
|
||||
@@ -551,6 +551,27 @@ const removeClient = async (
|
||||
...(removeNative ? [clientServerPath(parsed, name)] : []),
|
||||
...(legacyPath === undefined ? [] : [legacyPath]),
|
||||
];
|
||||
let serialized: ReturnType<typeof serializeClientConfiguration>;
|
||||
try {
|
||||
serialized = serializeClientConfiguration(
|
||||
document,
|
||||
client.format,
|
||||
original,
|
||||
editedPaths,
|
||||
);
|
||||
} catch {
|
||||
return item(
|
||||
client.name,
|
||||
"failed",
|
||||
"Configuration could not be updated, so no change was made. Repair the configuration, then rerun uninstall.",
|
||||
);
|
||||
}
|
||||
if (!serialized.ok)
|
||||
return item(
|
||||
client.name,
|
||||
"failed",
|
||||
`Configuration repeats the member at ${JSON.stringify(serialized.error.path)}; removing the registration the client reads would activate an earlier duplicate, so no change was made. Remove the duplicate members manually, then rerun uninstall.`,
|
||||
);
|
||||
const backupPath = await backupClientConfiguration(
|
||||
client,
|
||||
transactionPath,
|
||||
@@ -573,12 +594,7 @@ const removeClient = async (
|
||||
try {
|
||||
publication = await fileSystem.publishConfiguration(
|
||||
transactionPath,
|
||||
serializeClientConfiguration(
|
||||
document,
|
||||
client.format,
|
||||
original,
|
||||
editedPaths,
|
||||
),
|
||||
serialized.value,
|
||||
original,
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
|
||||
@@ -25,6 +25,36 @@ import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
|
||||
import { err, ok } from "../../domain/result.js";
|
||||
import { EvidenceLedger } from "../investigation/EvidenceLedger.js";
|
||||
import { WEB_NETWORK_CAPTURE_LIMITS } from "../../domain/webNetworkCapture.js";
|
||||
import { isImmutableJsonSnapshot } from "../../domain/immutableJson.js";
|
||||
|
||||
it.each(["layout", "javascript"])(
|
||||
"carries the retained %s source snapshot into repeatable shared views",
|
||||
(kind) => {
|
||||
const parent =
|
||||
kind === "layout"
|
||||
? analysisViewLayoutEvidence()
|
||||
: analysisViewJavaScriptEvidence();
|
||||
const ledger = new EvidenceLedger();
|
||||
const recorded = ledger.record(parent);
|
||||
if (!recorded.ok) throw recorded.error;
|
||||
const borrowed = ledger.forAnalysis(parent.evidence_id)!;
|
||||
const original = JSON.stringify(borrowed);
|
||||
const input = {
|
||||
source: { kind: "retained-evidence", evidence_id: parent.evidence_id },
|
||||
view: { kind: "summary" },
|
||||
} as const;
|
||||
const first = inspectAnalysisView(input, (id) => ledger.forAnalysis(id));
|
||||
if (!first.ok) throw first.error;
|
||||
const repeated = inspectAnalysisView(input, (id) => ledger.forAnalysis(id));
|
||||
if (!repeated.ok) throw repeated.error;
|
||||
expect(repeated.value.normalized_result).toEqual(
|
||||
first.value.normalized_result,
|
||||
);
|
||||
expect(isImmutableJsonSnapshot(borrowed.normalized_result)).toBe(true);
|
||||
expect(JSON.stringify(borrowed)).toBe(original);
|
||||
expect(ledger.forAnalysis(parent.evidence_id)).toBe(borrowed);
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["/fixtures/unknown.exe", ""])(
|
||||
"preserves recorded empty native identity fields: path=%j",
|
||||
|
||||
@@ -21,7 +21,10 @@ import {
|
||||
type InspectAnalysisViewInput,
|
||||
} from "../../domain/analysisView/analysisView.js";
|
||||
import { jsonValueSchema } from "../../domain/jsonValue.js";
|
||||
import { isImmutableJsonSnapshot } from "../../domain/immutableJson.js";
|
||||
import {
|
||||
immutableJsonSnapshotMember,
|
||||
isImmutableJsonSnapshot,
|
||||
} from "../../domain/immutableJson.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import { WebNetworkApiSurfaceAccumulator } from "../../domain/analysisView/WebNetworkApiSurfaceAccumulator.js";
|
||||
import { admitAnalysisViewData } from "../../domain/analysisView/analysisViewDataAdmission.js";
|
||||
@@ -70,7 +73,9 @@ export const inspectAnalysisViewValidated = (
|
||||
{
|
||||
evidenceId: parent.evidence_id,
|
||||
operation: parent.operation,
|
||||
normalizedResult: parent.normalized_result,
|
||||
normalizedResult:
|
||||
immutableJsonSnapshotMember(parent, "normalized_result") ??
|
||||
parent.normalized_result,
|
||||
// Sealing an Evidence root deep-freezes it; only the root is registered.
|
||||
sealedResult: isImmutableJsonSnapshot(parent),
|
||||
subject: parent.subject,
|
||||
|
||||
@@ -32,6 +32,7 @@ const refusedPort: ApktoolResourceAnalysisPort = {
|
||||
|
||||
it.each([
|
||||
[{ path: "/targets/app.apk", include_strings: "yes" }],
|
||||
[{ path: "/targets/app.apk", string_tables_only: "yes" }],
|
||||
[{ path: "/targets/app.apk", locale: "not a locale" }],
|
||||
[{ path: "/targets/app.apk", locales: ["not a locale"] }],
|
||||
[{ path: "/targets/app.apk", locales: [] }],
|
||||
|
||||
@@ -10,6 +10,7 @@ import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js";
|
||||
import {
|
||||
projectPlistValue,
|
||||
unknownDateLimitation,
|
||||
unknownScalarLimitation,
|
||||
} from "../../domain/apple/plistValue.js";
|
||||
import {
|
||||
omittedPrototypeKeysLimitation,
|
||||
@@ -67,6 +68,7 @@ export const analyzeInterfaceBuilderBundle = async (input: {
|
||||
const unresolvedConnections = new Map<string, number>();
|
||||
const prototypeKeyOmissions = new Map<string, number | null>();
|
||||
const unknownDates = new Map<string, number>();
|
||||
const unknownScalars = new Map<string, number>();
|
||||
let omitted = 0;
|
||||
let attempted = 0;
|
||||
let aggregateInputBytes = 0;
|
||||
@@ -148,6 +150,7 @@ export const analyzeInterfaceBuilderBundle = async (input: {
|
||||
value: raw,
|
||||
omittedPrototypeKeys,
|
||||
unknownDateCount,
|
||||
unknownScalarCount,
|
||||
numberLiterals,
|
||||
} = nib === null
|
||||
? decodePlist(bytes, xmlText, binaryPrototypeKeyObserved)
|
||||
@@ -155,6 +158,7 @@ export const analyzeInterfaceBuilderBundle = async (input: {
|
||||
value: nib.raw,
|
||||
omittedPrototypeKeys: 0,
|
||||
unknownDateCount: 0,
|
||||
unknownScalarCount: 0,
|
||||
numberLiterals: undefined,
|
||||
};
|
||||
if (nib !== null && nib.omitted > 0)
|
||||
@@ -165,6 +169,8 @@ export const analyzeInterfaceBuilderBundle = async (input: {
|
||||
prototypeKeyOmissions.set(entry.path, omittedPrototypeKeys);
|
||||
if (unknownDateCount !== 0)
|
||||
unknownDates.set(entry.path, unknownDateCount);
|
||||
if (unknownScalarCount !== 0)
|
||||
unknownScalars.set(entry.path, unknownScalarCount);
|
||||
const documentHash = createHash("sha256").update(bytes).digest("hex");
|
||||
documents.push({
|
||||
relativePath: entry.path,
|
||||
@@ -207,6 +213,7 @@ export const analyzeInterfaceBuilderBundle = async (input: {
|
||||
unresolvedConnections,
|
||||
prototypeKeyOmissions,
|
||||
unknownDates,
|
||||
unknownScalars,
|
||||
invalid,
|
||||
omittedArchives,
|
||||
unreportedOmittedArchives,
|
||||
@@ -276,6 +283,7 @@ const finalizeHierarchyCoverage = (
|
||||
unresolvedConnections: ReadonlyMap<string, number>;
|
||||
prototypeKeyOmissions: ReadonlyMap<string, number | null>;
|
||||
unknownDates: ReadonlyMap<string, number>;
|
||||
unknownScalars: ReadonlyMap<string, number>;
|
||||
invalid: readonly string[];
|
||||
omittedArchives: readonly { path: string; reason: string }[];
|
||||
unreportedOmittedArchives: number;
|
||||
@@ -288,6 +296,7 @@ const finalizeHierarchyCoverage = (
|
||||
unresolvedConnections,
|
||||
prototypeKeyOmissions,
|
||||
unknownDates,
|
||||
unknownScalars,
|
||||
invalid,
|
||||
omittedArchives,
|
||||
unreportedOmittedArchives,
|
||||
@@ -376,6 +385,9 @@ const finalizeHierarchyCoverage = (
|
||||
...[...unknownDates].map(
|
||||
([path, count]) => `${path}: ${unknownDateLimitation(count)}`,
|
||||
),
|
||||
...[...unknownScalars].map(
|
||||
([path, count]) => `${path}: ${unknownScalarLimitation(count)}`,
|
||||
),
|
||||
...omittedArchives.map(
|
||||
({ path, reason }) =>
|
||||
`${path}: omitted because ${reason.replaceAll("_", " ")}.`,
|
||||
@@ -746,6 +758,7 @@ const decodePlist = (
|
||||
readonly value: JsonValue;
|
||||
readonly omittedPrototypeKeys: number | null;
|
||||
readonly unknownDateCount: number;
|
||||
readonly unknownScalarCount: number;
|
||||
readonly numberLiterals: PlistNumberLiterals;
|
||||
} => {
|
||||
const binary = hasBinaryPlistSignature(bytes);
|
||||
@@ -766,6 +779,7 @@ const decodePlist = (
|
||||
value: projected.value,
|
||||
omittedPrototypeKeys,
|
||||
unknownDateCount: projected.unknownDateCount,
|
||||
unknownScalarCount: projected.unknownScalarCount,
|
||||
numberLiterals,
|
||||
};
|
||||
};
|
||||
@@ -10,7 +10,7 @@ import {
|
||||
negativeZeroRealLimitation,
|
||||
nonFiniteRealLimitation,
|
||||
unknownDateLimitation,
|
||||
unknownRealLimitation,
|
||||
unknownScalarLimitation,
|
||||
} from "../../domain/apple/plistValue.js";
|
||||
import { createPlistNumberProjection } from "../../domain/apple/plistNumbers.js";
|
||||
import {
|
||||
@@ -89,9 +89,9 @@ export const decodeKeyedArchiveBytes = (
|
||||
: parseXmlPropertyList(xmlText ?? "");
|
||||
const {
|
||||
nonFiniteRealCount,
|
||||
unknownRealCount,
|
||||
negativeZeroRealCount,
|
||||
unknownDateCount,
|
||||
unknownScalarCount,
|
||||
...graph
|
||||
} = projectKeyedArchive(parsed.value, selection);
|
||||
const numbers = createPlistNumberProjection(parsed.numberLiterals);
|
||||
@@ -121,15 +121,15 @@ export const decodeKeyedArchiveBytes = (
|
||||
...(nonFiniteRealCount === 0
|
||||
? []
|
||||
: [nonFiniteRealLimitation(nonFiniteRealCount)]),
|
||||
...(unknownRealCount === 0
|
||||
? []
|
||||
: [unknownRealLimitation(unknownRealCount)]),
|
||||
...(negativeZeroRealCount === 0
|
||||
? []
|
||||
: [negativeZeroRealLimitation(negativeZeroRealCount)]),
|
||||
...(unknownDateCount === 0
|
||||
? []
|
||||
: [unknownDateLimitation(unknownDateCount)]),
|
||||
...(unknownScalarCount === 0
|
||||
? []
|
||||
: [unknownScalarLimitation(unknownScalarCount)]),
|
||||
],
|
||||
};
|
||||
};
|
||||
|
||||
@@ -55,6 +55,8 @@ export const registerApktoolCommands = (
|
||||
// `b+zh+Hans+CN` form at the CLI boundary.
|
||||
options: z.object({
|
||||
locale: apktoolInputSchemas.decode_android_resources.shape.locale,
|
||||
"string-tables-only":
|
||||
apktoolInputSchemas.decode_android_resources.shape.string_tables_only,
|
||||
locales:
|
||||
apktoolInputSchemas.decode_android_resources.shape.locales.describe(
|
||||
"Repeat to project multiple localized string tables, e.g. --locales de --locales pt-BR",
|
||||
@@ -74,6 +76,9 @@ export const registerApktoolCommands = (
|
||||
execute(CLI_COMMANDS.decodeAndroidResources, "decode_android_resources", {
|
||||
path: args.path,
|
||||
include_strings: options.strings,
|
||||
...(options["string-tables-only"] === undefined
|
||||
? {}
|
||||
: { string_tables_only: options["string-tables-only"] }),
|
||||
...(options.locale === undefined ? {} : { locale: options.locale }),
|
||||
...(options.locales === undefined ? {} : { locales: options.locales }),
|
||||
}),
|
||||
|
||||
@@ -39,7 +39,11 @@ export const APKTOOL_TOOL_CONTRACTS = [
|
||||
},
|
||||
{
|
||||
title: "Decode multiple locale string tables in one pass",
|
||||
input: { path: "/targets/Example.apk", locales: ["de", "fr"] },
|
||||
input: {
|
||||
path: "/targets/Example.apk",
|
||||
locales: ["de", "fr"],
|
||||
string_tables_only: true,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
|
||||
+190
-1
@@ -1,13 +1,202 @@
|
||||
import { chmod, mkdir, realpath, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { expect, it as test } from "vitest";
|
||||
import { expect, it as test, vi } from "vitest";
|
||||
|
||||
import { createSystemDoctorHost } from "./doctorRuntime.js";
|
||||
import { DOCTOR_PROVIDER_IDS } from "./application/DoctorProviderIds.js";
|
||||
import { providerRegistrationEnvironment } from "./application/SetupRegistrationEnvironment.js";
|
||||
import { createTestTempDirectory } from "../tests/fixtures/temporaryDirectory.js";
|
||||
import { writeJadxJarInventory } from "../tests/fixtures/android/jadxJar.js";
|
||||
import * as privateDisplay from "./hopper/LinuxPrivateDisplayProbe.js";
|
||||
import * as pythonRuntime from "./application/LinuxHopperPython.js";
|
||||
import { err, ok } from "./domain/result.js";
|
||||
import { AnalysisError } from "./domain/analysisErrorBase.js";
|
||||
import { AnalysisCancelledError } from "./domain/analysisErrorCore.js";
|
||||
import { createDoctorHostFixture } from "./application/Doctor.fixture.js";
|
||||
import { runDoctorOwned } from "./application/Doctor.js";
|
||||
|
||||
const readyDisplay = {
|
||||
ok: true,
|
||||
strategy: "direct",
|
||||
diagnostic: {
|
||||
component: "hopper_private_display",
|
||||
operation: "probe",
|
||||
reason: "ready",
|
||||
socket_directory: "/tmp/.X11-unix",
|
||||
socket_directory_mode: "1777",
|
||||
mount_read_only: false,
|
||||
effective_socket_directory_mode: "1777",
|
||||
effective_mount_read_only: false,
|
||||
wsl: false,
|
||||
strategy: "direct",
|
||||
fallback_reason: null,
|
||||
xvfb_stderr_bytes: 0,
|
||||
status: "ready",
|
||||
failure_code: null,
|
||||
},
|
||||
} as const satisfies privateDisplay.LinuxPrivateDisplaySelection;
|
||||
|
||||
test.each([undefined, "controlled close refusal"])(
|
||||
"propagates runtime-probe cancellation through owned doctor cleanup: %s",
|
||||
async (cleanupFailure) => {
|
||||
const controller = new AbortController();
|
||||
const reason = { request: "cancel Python runtime inspection" };
|
||||
const display = vi
|
||||
.spyOn(privateDisplay, "selectLinuxPrivateDisplayStrategy")
|
||||
.mockResolvedValue(readyDisplay);
|
||||
const python = vi
|
||||
.spyOn(pythonRuntime, "inspectLinuxHopperPythonRuntime")
|
||||
.mockImplementation(async (_environment, signal) => {
|
||||
expect(signal).toBe(controller.signal);
|
||||
controller.abort(reason);
|
||||
return err(
|
||||
new pythonRuntime.LinuxHopperPythonError(
|
||||
"cancelled",
|
||||
"Controlled runtime cancellation",
|
||||
reason,
|
||||
),
|
||||
);
|
||||
});
|
||||
const systemHost = createSystemDoctorHost({}, controller.signal);
|
||||
const close = vi.fn(async () => {
|
||||
await systemHost.close?.();
|
||||
return cleanupFailure;
|
||||
});
|
||||
try {
|
||||
const failure = await runDoctorOwned(
|
||||
undefined,
|
||||
createDoctorHostFixture({
|
||||
platform: "linux",
|
||||
configuredHopperPath: "/fixture/Hopper",
|
||||
signal: controller.signal,
|
||||
linuxDemoRuntimeCheck: systemHost.linuxDemoRuntimeCheck,
|
||||
close,
|
||||
}),
|
||||
).then(
|
||||
() => undefined,
|
||||
(cause: unknown) => cause,
|
||||
);
|
||||
if (!(failure instanceof AnalysisError))
|
||||
throw new Error("Expected a doctor cancellation failure");
|
||||
const primary = failure.cleanupPrimary ?? failure;
|
||||
expect(primary).toBeInstanceOf(AnalysisCancelledError);
|
||||
expect(primary.cause).toBe(reason);
|
||||
expect(primary.diagnostics).toMatchObject({
|
||||
hopper_demo_runtime: {
|
||||
private_display: readyDisplay.diagnostic,
|
||||
embedded_python: { status: "unknown", reason: "cancelled" },
|
||||
},
|
||||
});
|
||||
expect(close).toHaveBeenCalledTimes(1);
|
||||
if (cleanupFailure === undefined) expect(failure).toBe(primary);
|
||||
else expect(failure.cleanupIncomplete).toBe(true);
|
||||
} finally {
|
||||
try {
|
||||
await systemHost.close?.();
|
||||
} finally {
|
||||
python.mockRestore();
|
||||
display.mockRestore();
|
||||
}
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
test.each(["unavailable", "unknown"] as const)(
|
||||
"does not report a healthy Hopper runtime when Python is %s",
|
||||
async (state) => {
|
||||
const display = vi
|
||||
.spyOn(privateDisplay, "selectLinuxPrivateDisplayStrategy")
|
||||
.mockResolvedValue(readyDisplay);
|
||||
const python = vi
|
||||
.spyOn(pythonRuntime, "inspectLinuxHopperPythonRuntime")
|
||||
.mockResolvedValue(
|
||||
state === "unavailable"
|
||||
? ok({
|
||||
interpreter: "/usr/bin/python3",
|
||||
version: [3, 14],
|
||||
library: {
|
||||
status: "unavailable",
|
||||
path: "/usr/lib/libpython3.14.so.1.0",
|
||||
reason: "No such file",
|
||||
},
|
||||
})
|
||||
: err(
|
||||
new pythonRuntime.LinuxHopperPythonError(
|
||||
"query_failed",
|
||||
"Fixture query failed",
|
||||
),
|
||||
),
|
||||
);
|
||||
try {
|
||||
const check = await createSystemDoctorHost({}).linuxDemoRuntimeCheck?.();
|
||||
expect(check).toMatchObject({
|
||||
name: "hopper-demo-runtime",
|
||||
ok: false,
|
||||
classification:
|
||||
state === "unavailable" ? "missing_dependency" : "unknown",
|
||||
details: {
|
||||
private_display: readyDisplay.diagnostic,
|
||||
embedded_python: {
|
||||
scope: "system_interpreter",
|
||||
hopper_interpreter: "unknown",
|
||||
...(state === "unavailable"
|
||||
? { status: "observed", library: { status: "unavailable" } }
|
||||
: { status: "unknown", reason: "query_failed" }),
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(check?.remediation).toContain(
|
||||
state === "unavailable"
|
||||
? "rea setup --install-hopper"
|
||||
: "Retry rea doctor",
|
||||
);
|
||||
} finally {
|
||||
python.mockRestore();
|
||||
display.mockRestore();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
test("preserves both X11 and Python failures and their independent remediation", async () => {
|
||||
const display = vi
|
||||
.spyOn(privateDisplay, "selectLinuxPrivateDisplayStrategy")
|
||||
.mockResolvedValue({
|
||||
ok: false,
|
||||
strategy: "unavailable",
|
||||
exitCode: 80,
|
||||
diagnostic: {
|
||||
...readyDisplay.diagnostic,
|
||||
status: "error",
|
||||
failure_code: "x11_socket_directory_unusable",
|
||||
},
|
||||
});
|
||||
const python = vi
|
||||
.spyOn(pythonRuntime, "inspectLinuxHopperPythonRuntime")
|
||||
.mockResolvedValue(
|
||||
ok({
|
||||
interpreter: "/usr/bin/python3",
|
||||
version: [3, 14],
|
||||
library: { status: "unknown", reason: "No INSTSONAME" },
|
||||
}),
|
||||
);
|
||||
try {
|
||||
const check = await createSystemDoctorHost({}).linuxDemoRuntimeCheck?.();
|
||||
expect(check).toMatchObject({
|
||||
ok: false,
|
||||
classification: "config_drift",
|
||||
details: {
|
||||
private_display: { failure_code: "x11_socket_directory_unusable" },
|
||||
embedded_python: { status: "observed", library: { status: "unknown" } },
|
||||
},
|
||||
});
|
||||
expect(check?.remediation).toContain("user and mount namespaces");
|
||||
expect(check?.remediation).toContain("shared-runtime configuration");
|
||||
} finally {
|
||||
python.mockRestore();
|
||||
display.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
const it = test.skipIf(process.platform === "win32");
|
||||
|
||||
|
||||
+68
-13
@@ -12,6 +12,11 @@ import { readIdaConfiguration } from "./ida/IdaConfiguration.js";
|
||||
import { parseJebMcpEndpoint } from "./jeb/JebMcpEndpoint.js";
|
||||
import { selectLinuxPrivateDisplayStrategy } from "./hopper/LinuxPrivateDisplayProbe.js";
|
||||
import { ok } from "./domain/result.js";
|
||||
import { AnalysisCancelledError } from "./domain/analysisErrorCore.js";
|
||||
import {
|
||||
inspectLinuxHopperPythonRuntime,
|
||||
linuxHopperPythonUnavailable,
|
||||
} from "./application/LinuxHopperPython.js";
|
||||
|
||||
/** Compose provider diagnostics at the outer CLI adapter boundary. */
|
||||
export const createSystemDoctorHost = (
|
||||
@@ -60,7 +65,8 @@ export const createSystemDoctorHost = (
|
||||
fileCloseResources,
|
||||
}),
|
||||
],
|
||||
linuxDemoRuntimeCheck: () => inspectLinuxPrivateDisplay(environment),
|
||||
linuxDemoRuntimeCheck: () =>
|
||||
inspectLinuxHopperDemoRuntime(environment, signal),
|
||||
});
|
||||
};
|
||||
|
||||
@@ -68,13 +74,31 @@ const HOPPER_DEMO_HELPER_PATH = fileURLToPath(
|
||||
new URL("../scripts/hopper-demo-x11.py", import.meta.url),
|
||||
);
|
||||
|
||||
const inspectLinuxPrivateDisplay = async (
|
||||
const inspectLinuxHopperDemoRuntime = async (
|
||||
environment: Readonly<NodeJS.ProcessEnv>,
|
||||
signal?: AbortSignal,
|
||||
): Promise<DoctorCheck> => {
|
||||
const selection = await selectLinuxPrivateDisplayStrategy({
|
||||
helperPath: HOPPER_DEMO_HELPER_PATH,
|
||||
environment,
|
||||
});
|
||||
const python = await inspectLinuxHopperPythonRuntime(environment, signal);
|
||||
const pythonFailure = !python.ok
|
||||
? python.error
|
||||
: python.value.library.status !== "loadable"
|
||||
? linuxHopperPythonUnavailable(python.value.library)
|
||||
: undefined;
|
||||
const pythonDiagnostic = {
|
||||
scope: "system_interpreter",
|
||||
hopper_interpreter: "unknown",
|
||||
...(python.ok
|
||||
? { status: "observed", ...python.value }
|
||||
: {
|
||||
status: "unknown",
|
||||
reason: python.error.reason,
|
||||
message: python.error.message,
|
||||
}),
|
||||
};
|
||||
const value = selection.diagnostic;
|
||||
const detail = [
|
||||
`strategy=${selection.strategy}`,
|
||||
@@ -84,29 +108,60 @@ const inspectLinuxPrivateDisplay = async (
|
||||
`effective_mode=${value.effective_socket_directory_mode ?? "unknown"}`,
|
||||
`effective_mount_read_only=${String(value.effective_mount_read_only)}`,
|
||||
`wsl=${String(value.wsl)}`,
|
||||
`system_python=${python.ok ? python.value.library.status : "unknown"}`,
|
||||
].join("; ");
|
||||
if (selection.ok)
|
||||
const details = {
|
||||
private_display: { ...value },
|
||||
embedded_python: pythonDiagnostic,
|
||||
};
|
||||
if (!python.ok && python.error.reason === "cancelled")
|
||||
throw new AnalysisCancelledError("doctor", {
|
||||
cause: python.error.cause,
|
||||
diagnostics: { hopper_demo_runtime: details },
|
||||
});
|
||||
if (selection.ok && pythonFailure === undefined)
|
||||
return {
|
||||
name: "hopper-demo-runtime",
|
||||
ok: true,
|
||||
classification: "healthy",
|
||||
detail,
|
||||
details: { private_display: { ...value } },
|
||||
details,
|
||||
};
|
||||
const missingDependency =
|
||||
value.failure_code === "runtime_dependency_unavailable" ||
|
||||
value.failure_code === "x11_authorization_failed";
|
||||
value.failure_code === "x11_authorization_failed" ||
|
||||
(python.ok && python.value.library.status === "unavailable");
|
||||
const pythonRemediation =
|
||||
pythonFailure === undefined
|
||||
? undefined
|
||||
: pythonFailure.reason === "runtime_unavailable"
|
||||
? pythonFailure.message +
|
||||
" On a supported system run rea setup --install-hopper, then rerun rea doctor --provider hopper. This checks the system interpreter; a custom Hopper Python interpreter needs its own matching shared runtime."
|
||||
: pythonFailure.reason === "configuration_unknown"
|
||||
? pythonFailure.message +
|
||||
" Inspect this interpreter's shared-runtime configuration, then rerun rea doctor --provider hopper."
|
||||
: pythonFailure.message +
|
||||
" Retry rea doctor --provider hopper and inspect /usr/bin/python3 if its runtime remains unknown.";
|
||||
const displayRemediation = selection.ok
|
||||
? undefined
|
||||
: value.failure_code === "x11_socket_directory_unusable"
|
||||
? "Enable unprivileged user and mount namespaces, or provide a writable mode-1777 /tmp/.X11-unix directory. Do not mutate the current host mount with sudo; then rerun rea doctor --provider hopper."
|
||||
: value.failure_code === "runtime_dependency_unavailable" ||
|
||||
value.failure_code === "x11_authorization_failed"
|
||||
? "Install Xvfb, xauth, Python 3, libX11, and libXtst, then rerun rea doctor --provider hopper."
|
||||
: "Inspect the private-display diagnostic, correct the reported local condition, and rerun rea doctor --provider hopper.";
|
||||
return {
|
||||
name: "hopper-demo-runtime",
|
||||
ok: false,
|
||||
classification: missingDependency ? "missing_dependency" : "config_drift",
|
||||
classification: missingDependency
|
||||
? "missing_dependency"
|
||||
: selection.ok
|
||||
? "unknown"
|
||||
: "config_drift",
|
||||
detail,
|
||||
details: { private_display: { ...value } },
|
||||
remediation:
|
||||
value.failure_code === "x11_socket_directory_unusable"
|
||||
? "Enable unprivileged user and mount namespaces, or provide a writable mode-1777 /tmp/.X11-unix directory. Do not mutate the current host mount with sudo; then rerun rea doctor --provider hopper."
|
||||
: missingDependency
|
||||
? "Install Xvfb, xauth, Python 3, libX11, and libXtst, then rerun rea doctor --provider hopper."
|
||||
: "Inspect the private-display diagnostic, correct the reported local condition, and rerun rea doctor --provider hopper.",
|
||||
details,
|
||||
remediation: [pythonRemediation, displayRemediation]
|
||||
.filter((message) => message !== undefined)
|
||||
.join(" "),
|
||||
};
|
||||
};
|
||||
@@ -79,6 +79,7 @@ export abstract class AnalysisError extends Error {
|
||||
cleanupIncomplete: boolean;
|
||||
#ownedCleanupResources: readonly string[];
|
||||
#cleanupResourceOwner: AnalysisCleanupResourceOwner | undefined;
|
||||
#secondaryFailures: AnalysisError[] = [];
|
||||
constructor(message: string, options?: AnalysisErrorOptions) {
|
||||
super(message, options);
|
||||
this.#cleanupPrimary = options?.cleanupPrimary;
|
||||
@@ -104,6 +105,10 @@ export abstract class AnalysisError extends Error {
|
||||
get cleanupPrimary(): AnalysisError | undefined {
|
||||
return this.#cleanupPrimary;
|
||||
}
|
||||
/** Later typed failures retained without changing the primary error cause. */
|
||||
get secondaryFailures(): readonly AnalysisError[] {
|
||||
return this.#secondaryFailures;
|
||||
}
|
||||
readonly cleanupReport: ProcessCaptureCleanupReport | undefined = undefined;
|
||||
|
||||
/** Live dependencies own current resource attribution; cleanup keeps history. */
|
||||
@@ -142,12 +147,8 @@ export abstract class AnalysisError extends Error {
|
||||
// the primary error's live cleanup status and original output.
|
||||
Object.assign(this, {
|
||||
diagnostics: structuredClone({ ...this.diagnostics, ...diagnostics }),
|
||||
cause: new AggregateError(
|
||||
this.cause === undefined ? [failure] : [this.cause, failure],
|
||||
"Analysis failed and a later lifecycle operation also failed.",
|
||||
{ cause: this.cause },
|
||||
),
|
||||
});
|
||||
this.#secondaryFailures.push(failure);
|
||||
return this;
|
||||
}
|
||||
|
||||
|
||||
@@ -839,7 +839,9 @@ describe("analysis error projection: evidence files", () => {
|
||||
});
|
||||
|
||||
it("retains the primary failure and both cleanup reports after a secondary failure", () => {
|
||||
const callerReason = { caller: "stop" };
|
||||
const primary = new AnalysisCancelledError("execute_rizin_command", {
|
||||
cause: callerReason,
|
||||
capturedOutput: {
|
||||
stdout: "partial",
|
||||
stderr: "diagnostic",
|
||||
@@ -855,6 +857,8 @@ it("retains the primary failure and both cleanup reports after a secondary failu
|
||||
});
|
||||
expect(enriched).toBe(primary);
|
||||
expect(enriched._tag).toBe("AnalysisCancelledError");
|
||||
expect(enriched.cause).toBe(callerReason);
|
||||
expect(enriched.secondaryFailures).toEqual([secondary]);
|
||||
expect(projectAnalysisError(enriched)).toMatchObject({
|
||||
code: "cleanup_incomplete",
|
||||
details: {
|
||||
@@ -867,6 +871,16 @@ it("retains the primary failure and both cleanup reports after a secondary failu
|
||||
});
|
||||
});
|
||||
|
||||
it("retains secondary failures when the primary error has no cause", () => {
|
||||
const primary = new AnalysisCancelledError("execute_rizin_command");
|
||||
const secondary = new AnalysisOutputError("open_binary", "cannot rebind");
|
||||
|
||||
primary.retainSecondaryFailure(secondary, {});
|
||||
|
||||
expect(primary.cause).toBeUndefined();
|
||||
expect(primary.secondaryFailures).toEqual([secondary]);
|
||||
});
|
||||
|
||||
it.each([false, true])(
|
||||
"preserves producer capability recovery with retryable=%s",
|
||||
(retryable) => {
|
||||
|
||||
@@ -10,10 +10,7 @@ import {
|
||||
} from "./javascriptFactsView.js";
|
||||
import { javaScriptSourceSelectionSchema } from "../javascript/javascriptSourceSelection.js";
|
||||
|
||||
import {
|
||||
AnalysisInputError,
|
||||
AnalysisUnsupportedTargetError,
|
||||
} from "../analysisErrorCore.js";
|
||||
import { AnalysisUnsupportedTargetError } from "../analysisErrorCore.js";
|
||||
import {
|
||||
INSPECT_ANALYSIS_VIEW_OPERATION,
|
||||
analysisViewInputError,
|
||||
@@ -24,7 +21,7 @@ import { evidenceSchema, type Evidence } from "../evidence.js";
|
||||
import { jsonObjectSchema, jsonValueSchema } from "../jsonValue.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { binaryLayoutSchema } from "../native/binaryLayout.js";
|
||||
import { safeParseOwnedBinaryLayout } from "../native/binaryLayout.js";
|
||||
import { functionDossierSchema } from "../hopperValues.js";
|
||||
import {
|
||||
artifactGraphManifestSchema,
|
||||
@@ -741,8 +738,8 @@ const parseJavaScriptParent = (
|
||||
input: unknown,
|
||||
): Result<JavaScriptApplicationAnalysisResult, AnalysisError> => {
|
||||
try {
|
||||
// Only the existing realm-private graph proofs permit reuse. Imported or
|
||||
// merely frozen graphs still receive full canonical validation.
|
||||
// The format owner authenticates imported results completely before reuse;
|
||||
// mutable or merely frozen inputs still receive canonical validation.
|
||||
const steps = parseOwnedJavaScriptApplicationAnalysisSteps(input);
|
||||
let step = steps.next();
|
||||
while (!step.done) step = steps.next();
|
||||
@@ -841,7 +838,7 @@ export const projectAnalysisView = (
|
||||
return projected.ok ? ok(sealAnalysisView(projected.value)) : projected;
|
||||
}
|
||||
if (parent.operation === "inspect_binary_layout") {
|
||||
const layout = binaryLayoutSchema.safeParse(parent.normalizedResult);
|
||||
const layout = safeParseOwnedBinaryLayout(parent.normalizedResult);
|
||||
if (!layout.success)
|
||||
return err(
|
||||
analysisViewInputError([
|
||||
|
||||
@@ -69,6 +69,12 @@ export const apktoolInputSchemas = {
|
||||
.describe(
|
||||
`Project res/values/strings.xml entries (bounded to ${String(MAX_STRINGS)} entries)`,
|
||||
),
|
||||
string_tables_only: z
|
||||
.boolean()
|
||||
.optional()
|
||||
.describe(
|
||||
"Return selected strings only in string_tables, leaving top-level strings empty. Omit to preserve the primary strings field.",
|
||||
),
|
||||
locale: androidLocaleQualifierSchema
|
||||
.optional()
|
||||
.describe(
|
||||
@@ -178,15 +184,24 @@ export const apktoolResultSchemas = {
|
||||
package_name: z.string().nullable(),
|
||||
}),
|
||||
manifest: z.string().nullable(),
|
||||
strings: z.array(
|
||||
z.strictObject({
|
||||
name: z.string().min(1),
|
||||
value: z.string(),
|
||||
}),
|
||||
),
|
||||
strings: z
|
||||
.array(
|
||||
z.strictObject({
|
||||
name: z.string().min(1),
|
||||
value: z.string(),
|
||||
}),
|
||||
)
|
||||
.describe(
|
||||
"Primary string entries; empty when include_strings is false or string_tables_only is true. An unrequested field does not establish an empty table.",
|
||||
),
|
||||
locale: z.string().nullable(),
|
||||
locales: z.array(z.string().min(1)),
|
||||
string_tables: z.array(stringTableSchema).optional(),
|
||||
string_tables: z
|
||||
.array(stringTableSchema)
|
||||
.optional()
|
||||
.describe(
|
||||
"Selected table facts, included when locales is supplied or string_tables_only is true",
|
||||
),
|
||||
decoded_file_count: z.number().int().nonnegative(),
|
||||
coverage: z.enum(["complete", "partial"]),
|
||||
}),
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { MAX_PLIST_DEPTH } from "./plistValue.js";
|
||||
import { projectKeyedArchive } from "./keyedArchive.js";
|
||||
import { UnknownPlistScalar } from "./unknownPlistScalar.js";
|
||||
|
||||
const deepValue = (depth: number): unknown => {
|
||||
let value: unknown = 1;
|
||||
@@ -86,3 +87,43 @@ describe("projectKeyedArchive root reference projection", () => {
|
||||
).toThrow(RangeError);
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps generated unknown scalar wrappers scalar and lookalike dictionaries ordinary", () => {
|
||||
const result = projectKeyedArchive(
|
||||
{
|
||||
$archiver: "NSKeyedArchiver",
|
||||
$objects: [
|
||||
"$null",
|
||||
new UnknownPlistScalar(
|
||||
"false",
|
||||
"false<!-- value -->",
|
||||
"unsupported-markup",
|
||||
),
|
||||
{
|
||||
$plist_type: "unknown",
|
||||
element: "false",
|
||||
literal: "false<!-- value -->",
|
||||
reason: "unsupported-markup",
|
||||
},
|
||||
],
|
||||
$top: { root: { UID: 1 } },
|
||||
},
|
||||
{ offset: 0, limit: 3 },
|
||||
);
|
||||
expect(result.unknownScalarCount).toBe(1);
|
||||
expect(result.objects[1]).toMatchObject({
|
||||
kind: "scalar",
|
||||
status: "observed",
|
||||
value: {
|
||||
$plist_type: "unknown",
|
||||
element: "false",
|
||||
literal: "false<!-- value -->",
|
||||
reason: "unsupported-markup",
|
||||
},
|
||||
});
|
||||
expect(result.objects[2]).toMatchObject({
|
||||
kind: "dictionary",
|
||||
status: "observed",
|
||||
value: { $plist_type: "unknown" },
|
||||
});
|
||||
});
|
||||
@@ -248,9 +248,9 @@ export const projectKeyedArchive = (
|
||||
const ids = new Set(page.map(({ id }) => id));
|
||||
return {
|
||||
nonFiniteRealCount: projected.nonFiniteRealCount,
|
||||
unknownRealCount: projected.unknownRealCount,
|
||||
negativeZeroRealCount: projected.negativeZeroRealCount,
|
||||
unknownDateCount: projected.unknownDateCount,
|
||||
unknownScalarCount: projected.unknownScalarCount,
|
||||
roots,
|
||||
objects: page,
|
||||
references: references.filter(
|
||||
|
||||
@@ -1,18 +1,5 @@
|
||||
import type { JsonValue } from "../jsonValue.js";
|
||||
|
||||
/** XML numeric decoding could not establish a representable value identity. */
|
||||
export class UnknownPlistReal {}
|
||||
|
||||
/** Exact decimal or signed hexadecimal XML integer text. */
|
||||
export const parsePlistIntegerLiteral = (text: string): bigint | undefined => {
|
||||
const match = /^([+-]?)(?:0[xX]([\da-fA-F]+)|(\d+))$/u.exec(text.trim());
|
||||
if (match === null) return undefined;
|
||||
const magnitude = BigInt(
|
||||
match[2] === undefined ? match[3]! : `0x${match[2]}`,
|
||||
);
|
||||
return match[1] === "-" ? -magnitude : magnitude;
|
||||
};
|
||||
|
||||
/** Source element types needed to distinguish large integers from integral reals. */
|
||||
export interface PlistNumberLiterals {
|
||||
readonly integers: ReadonlyMap<number, ReadonlySet<string>>;
|
||||
@@ -31,7 +18,6 @@ export const createPlistNumberCollector = () => {
|
||||
const integers = new Map<number, Set<string>>();
|
||||
const reals = new Set<number>();
|
||||
let unsafeIntegerLiterals = 0;
|
||||
let incompleteMetadata = false;
|
||||
return {
|
||||
integer: (exact: bigint) => {
|
||||
if (
|
||||
@@ -49,14 +35,11 @@ export const createPlistNumberCollector = () => {
|
||||
real: (value: number) => {
|
||||
reals.add(value);
|
||||
},
|
||||
markIncomplete: () => {
|
||||
incompleteMetadata = true;
|
||||
},
|
||||
result: (incomplete = false): PlistNumberLiterals => ({
|
||||
integers,
|
||||
reals,
|
||||
unsafeIntegerLiterals,
|
||||
incomplete: incomplete || incompleteMetadata,
|
||||
incomplete,
|
||||
}),
|
||||
};
|
||||
};
|
||||
@@ -136,30 +119,3 @@ export const createPlistNumberProjection = (literals: PlistNumberLiterals) => {
|
||||
],
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* Read a whole `<real>` text as one of CFPropertyList's non-finite spellings.
|
||||
* Its parser compares the complete text case-insensitively and also accepts
|
||||
* `infinity`, `inf`, `+inf` and `-inf`; its writer emits `nan`, `+infinity`
|
||||
* and `-infinity`. Any other text, including surrounding whitespace, is not
|
||||
* a spelled non-finite real.
|
||||
*/
|
||||
export const parsePlistNonFiniteRealLiteral = (
|
||||
text: string,
|
||||
): number | undefined => {
|
||||
if (text.length < 3 || text.length > 9) return undefined;
|
||||
switch (text.toLowerCase()) {
|
||||
case "nan":
|
||||
return Number.NaN;
|
||||
case "+infinity":
|
||||
case "infinity":
|
||||
case "+inf":
|
||||
case "inf":
|
||||
return Number.POSITIVE_INFINITY;
|
||||
case "-infinity":
|
||||
case "-inf":
|
||||
return Number.NEGATIVE_INFINITY;
|
||||
default:
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
@@ -2,7 +2,7 @@ import { expect, it } from "vitest";
|
||||
import { parseBinary } from "plist";
|
||||
|
||||
import { projectPlistValue } from "./plistValue.js";
|
||||
import { UnknownPlistReal } from "./plistNumbers.js";
|
||||
import { UnknownPlistScalar } from "./unknownPlistScalar.js";
|
||||
|
||||
// Binary decoders build dates from seconds; beyond the Date range they are invalid.
|
||||
it("projects an unrepresentable date as unknown and counts it", () => {
|
||||
@@ -14,9 +14,9 @@ it("projects an unrepresentable date as unknown and counts it", () => {
|
||||
near: { $plist_type: "date", iso8601: "1970-01-01T00:00:00.000Z" },
|
||||
},
|
||||
nonFiniteRealCount: 0,
|
||||
unknownRealCount: 0,
|
||||
negativeZeroRealCount: 0,
|
||||
unknownDateCount: 1,
|
||||
unknownScalarCount: 0,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -27,7 +27,6 @@ it("keeps a negative-zero real distinct from positive zero and counts it", () =>
|
||||
(wrapper) => observed.push(wrapper),
|
||||
);
|
||||
expect(projected.negativeZeroRealCount).toBe(1);
|
||||
expect(projected.unknownRealCount).toBe(0);
|
||||
expect(Array.isArray(projected.value)).toBe(true);
|
||||
if (!Array.isArray(projected.value)) throw new Error("Expected array");
|
||||
expect(projected.value[0]).toEqual({ $plist_type: "real", value: "-0" });
|
||||
@@ -120,7 +119,6 @@ it("keeps retained integer scalars distinct from lookalike dictionaries", () =>
|
||||
[BigInt(decimal), { $plist_type: "integer", decimal }],
|
||||
(value) => observed.push(value),
|
||||
);
|
||||
expect(projected.unknownRealCount).toBe(0);
|
||||
expect(projected.unknownDateCount).toBe(0);
|
||||
expect(Array.isArray(projected.value)).toBe(true);
|
||||
if (!Array.isArray(projected.value)) throw new Error("Expected array");
|
||||
@@ -130,30 +128,41 @@ it("keeps retained integer scalars distinct from lookalike dictionaries", () =>
|
||||
expect(observed[0]).not.toBe(projected.value[1]);
|
||||
});
|
||||
|
||||
it("projects known non-finite numbers while retaining explicit XML unknowns", () => {
|
||||
it("projects known non-finite and unknown XML scalars without wrapping lookalikes", () => {
|
||||
const observed: object[] = [];
|
||||
const projected = projectPlistValue(
|
||||
[
|
||||
Infinity,
|
||||
-Infinity,
|
||||
NaN,
|
||||
new UnknownPlistReal(),
|
||||
new UnknownPlistReal(),
|
||||
new UnknownPlistScalar("real", "1e999", "unrepresentable"),
|
||||
new UnknownPlistScalar(
|
||||
"integer",
|
||||
"12<!-- gap -->3",
|
||||
"unsupported-markup",
|
||||
),
|
||||
{ $plist_type: "real", value: "nan" },
|
||||
{
|
||||
$plist_type: "unknown",
|
||||
element: "integer",
|
||||
literal: "12<!-- gap -->3",
|
||||
reason: "unsupported-markup",
|
||||
},
|
||||
],
|
||||
(wrapper) => observed.push(wrapper),
|
||||
);
|
||||
expect(projected.nonFiniteRealCount).toBe(3);
|
||||
expect(projected.unknownRealCount).toBe(2);
|
||||
expect(projected.unknownScalarCount).toBe(2);
|
||||
expect(projected.negativeZeroRealCount).toBe(0);
|
||||
expect(JSON.stringify(projected.value)).toBe(
|
||||
'[{"$plist_type":"real","value":"+infinity"},{"$plist_type":"real","value":"-infinity"},{"$plist_type":"real","value":"nan"},{"$plist_type":"real","value":null},{"$plist_type":"real","value":null},{"$plist_type":"real","value":"nan"}]',
|
||||
'[{"$plist_type":"real","value":"+infinity"},{"$plist_type":"real","value":"-infinity"},{"$plist_type":"real","value":"nan"},{"$plist_type":"unknown","element":"real","literal":"1e999","reason":"unrepresentable"},{"$plist_type":"unknown","element":"integer","literal":"12<!-- gap -->3","reason":"unsupported-markup"},{"$plist_type":"real","value":"nan"},{"$plist_type":"unknown","element":"integer","literal":"12<!-- gap -->3","reason":"unsupported-markup"}]',
|
||||
);
|
||||
expect(Array.isArray(projected.value)).toBe(true);
|
||||
if (!Array.isArray(projected.value)) throw new Error("Expected array");
|
||||
// Only generated wrappers are reported; the lookalike stays a dictionary.
|
||||
expect(observed).toEqual(projected.value.slice(0, 5));
|
||||
expect(observed).not.toContain(projected.value[5]);
|
||||
expect(observed).not.toContain(projected.value[6]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
@@ -226,7 +235,6 @@ it.each([
|
||||
]);
|
||||
expect(projected.nonFiniteRealCount).toBe(4);
|
||||
expect(projected.negativeZeroRealCount).toBe(1);
|
||||
expect(projected.unknownRealCount).toBe(0);
|
||||
expect(observed).toHaveLength(5);
|
||||
expect(bytes).toEqual(original);
|
||||
});
|
||||
@@ -2,7 +2,7 @@ import { types } from "node:util";
|
||||
import { z } from "zod";
|
||||
|
||||
import type { JsonValue } from "../jsonValue.js";
|
||||
import { UnknownPlistReal } from "./plistNumbers.js";
|
||||
import { UnknownPlistScalar } from "./unknownPlistScalar.js";
|
||||
|
||||
/**
|
||||
* Maximum plist value nesting the JSON projection supports. The XML decode
|
||||
@@ -16,26 +16,26 @@ export interface ProjectedPlistValue {
|
||||
readonly value: JsonValue;
|
||||
/** Known non-finite values, rendered with canonical numeric labels. */
|
||||
readonly nonFiniteRealCount: number;
|
||||
/** XML numeric values whose identity could not be established. */
|
||||
readonly unknownRealCount: number;
|
||||
/** Negative-zero reals, which a JSON number would report as positive zero. */
|
||||
readonly negativeZeroRealCount: number;
|
||||
/** Dates the decoder could not represent, reported without a timestamp. */
|
||||
readonly unknownDateCount: number;
|
||||
/** XML scalar literals retained explicitly because their values are unknown. */
|
||||
readonly unknownScalarCount: number;
|
||||
}
|
||||
|
||||
/** Limitation for results whose plist held dates the decoder could not represent. */
|
||||
export const unknownDateLimitation = (count: number): string =>
|
||||
`${String(count)} date value(s) are reported as { "$plist_type": "date", "iso8601": null } because the decoder could not represent them, such as XML dates with years beyond 9999 or negative years, or dates outside the JavaScript date range.`;
|
||||
`${String(count)} date value(s) are reported as { "$plist_type": "date", "iso8601": null } because the decoder could not represent their timestamps.`;
|
||||
|
||||
/** Limitation for XML scalar literals whose values were not established. */
|
||||
export const unknownScalarLimitation = (count: number): string =>
|
||||
`${String(count)} XML scalar value(s) could not be established; their exact literals and reasons are retained in { "$plist_type": "unknown", "element": ..., "literal": ..., "reason": ... } wrappers.`;
|
||||
|
||||
/** Limitation for known non-finite values; the labels do not imply XML provenance. */
|
||||
export const nonFiniteRealLimitation = (count: number): string =>
|
||||
`${String(count)} non-finite real value(s) are reported as { "$plist_type": "real", "value": "<spelling>" } using canonical numeric labels nan, +infinity or -infinity because a JSON number cannot carry them.`;
|
||||
|
||||
/** Limitation for XML numeric values the decoder could not establish. */
|
||||
export const unknownRealLimitation = (count: number): string =>
|
||||
`${String(count)} numeric value(s) are reported as { "$plist_type": "real", "value": null } because XML numeric decoding could not establish their value identity.`;
|
||||
|
||||
/** The typed scalar that keeps a negative-zero real distinct from positive zero. */
|
||||
export const negativeZeroReal = (): Record<string, JsonValue> => ({
|
||||
$plist_type: "real",
|
||||
@@ -58,9 +58,9 @@ export const projectPlistValue = (
|
||||
onWrappedScalar?: (value: Record<string, JsonValue>) => void,
|
||||
): ProjectedPlistValue => {
|
||||
let nonFiniteRealCount = 0;
|
||||
let unknownRealCount = 0;
|
||||
let negativeZeroRealCount = 0;
|
||||
let unknownDateCount = 0;
|
||||
let unknownScalarCount = 0;
|
||||
const scalar = (value: Record<string, JsonValue>): JsonValue => {
|
||||
onWrappedScalar?.(value);
|
||||
return value;
|
||||
@@ -77,9 +77,14 @@ export const projectPlistValue = (
|
||||
$plist_type: "data",
|
||||
base64: Buffer.from(item).toString("base64"),
|
||||
});
|
||||
if (item instanceof UnknownPlistReal) {
|
||||
unknownRealCount += 1;
|
||||
return scalar({ $plist_type: "real", value: null });
|
||||
if (item instanceof UnknownPlistScalar) {
|
||||
unknownScalarCount += 1;
|
||||
return scalar({
|
||||
$plist_type: "unknown",
|
||||
element: item.element,
|
||||
literal: item.rawLiteral,
|
||||
reason: item.reason,
|
||||
});
|
||||
}
|
||||
if (item instanceof Date) {
|
||||
if (!Number.isNaN(Date.prototype.getTime.call(item)))
|
||||
@@ -137,8 +142,8 @@ export const projectPlistValue = (
|
||||
return {
|
||||
value: projected,
|
||||
nonFiniteRealCount,
|
||||
unknownRealCount,
|
||||
negativeZeroRealCount,
|
||||
unknownDateCount,
|
||||
unknownScalarCount,
|
||||
};
|
||||
};
|
||||
@@ -0,0 +1,11 @@
|
||||
/** An XML plist scalar whose value cannot be established without guessing. */
|
||||
export class UnknownPlistScalar {
|
||||
constructor(
|
||||
readonly element: "integer" | "real" | "date" | "true" | "false",
|
||||
readonly rawLiteral: string,
|
||||
readonly reason:
|
||||
| "invalid-literal"
|
||||
| "unsupported-markup"
|
||||
| "unrepresentable",
|
||||
) {}
|
||||
}
|
||||
@@ -1,12 +1,55 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import type { JsonValue } from "./jsonValue.js";
|
||||
import {
|
||||
freezeJsonSnapshot,
|
||||
canBorrowParsedJsonSnapshot,
|
||||
immutableJsonSnapshotMember,
|
||||
isImmutableJsonSnapshot,
|
||||
} from "./immutableJson.js";
|
||||
|
||||
describe("owned immutable JSON snapshots", () => {
|
||||
it("borrows only the selected own child of a completed sealed root", () => {
|
||||
const value = freezeJsonSnapshot({
|
||||
selected: { values: [1, 2] },
|
||||
other: { value: 3 },
|
||||
absent: null,
|
||||
});
|
||||
expect(isImmutableJsonSnapshot(value.selected)).toBe(false);
|
||||
expect(immutableJsonSnapshotMember(value, "selected")).toBe(value.selected);
|
||||
expect(isImmutableJsonSnapshot(value.selected)).toBe(true);
|
||||
expect(isImmutableJsonSnapshot(value.other)).toBe(false);
|
||||
expect(immutableJsonSnapshotMember(value.selected, "values")).toBe(
|
||||
value.selected.values,
|
||||
);
|
||||
expect(isImmutableJsonSnapshot(value.selected.values)).toBe(true);
|
||||
expect(immutableJsonSnapshotMember(value, "absent")).toBeNull();
|
||||
expect(immutableJsonSnapshotMember(value, "missing")).toBeUndefined();
|
||||
expect(immutableJsonSnapshotMember(value, "toString")).toBeUndefined();
|
||||
expect(Reflect.set(value.selected.values, 0, 9)).toBe(false);
|
||||
});
|
||||
|
||||
it("does not borrow children from unowned roots or invoke their accessors", () => {
|
||||
let accessed = false;
|
||||
const child = Object.freeze({ value: 1 });
|
||||
const root = Object.freeze({ child });
|
||||
expect(immutableJsonSnapshotMember(root, "child")).toBeUndefined();
|
||||
expect(isImmutableJsonSnapshot(child)).toBe(false);
|
||||
expect(
|
||||
immutableJsonSnapshotMember(
|
||||
{
|
||||
get child() {
|
||||
accessed = true;
|
||||
return child;
|
||||
},
|
||||
},
|
||||
"child",
|
||||
),
|
||||
).toBeUndefined();
|
||||
expect(accessed).toBe(false);
|
||||
expect(immutableJsonSnapshotMember(null, "child")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("seals reachable data and preserves repeated references", () => {
|
||||
const shared = { values: [1, "observed"] };
|
||||
const value = { left: shared, right: shared };
|
||||
@@ -61,4 +104,166 @@ describe("owned immutable JSON snapshots", () => {
|
||||
"JSON values",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects proxy snapshots before reflecting on their mutable behavior", () => {
|
||||
let reflected = false;
|
||||
const proxy = new Proxy(
|
||||
{ value: 1 },
|
||||
{
|
||||
getPrototypeOf() {
|
||||
reflected = true;
|
||||
throw new Error("must not reflect on a proxy");
|
||||
},
|
||||
},
|
||||
);
|
||||
const root = { child: proxy };
|
||||
for (const input of [proxy, root]) {
|
||||
expect(() => freezeJsonSnapshot(input)).toThrow("proxies");
|
||||
expect(isImmutableJsonSnapshot(input)).toBe(false);
|
||||
}
|
||||
expect(Object.isFrozen(root)).toBe(false);
|
||||
expect(reflected).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("borrowing a parsed JSON snapshot", () => {
|
||||
it("accepts reordered object fields but preserves arrays and exact scalars", () => {
|
||||
const source = freezeJsonSnapshot({ second: [true, null], first: -0 });
|
||||
expect(
|
||||
canBorrowParsedJsonSnapshot(source, { first: -0, second: [true, null] }),
|
||||
).toBe(true);
|
||||
expect(
|
||||
canBorrowParsedJsonSnapshot(source, { first: 0, second: [true, null] }),
|
||||
).toBe(false);
|
||||
expect(
|
||||
canBorrowParsedJsonSnapshot(source, { first: -0, second: [null, true] }),
|
||||
).toBe(false);
|
||||
expect(
|
||||
canBorrowParsedJsonSnapshot(Object.freeze({ value: 1 }), { value: 1 }),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("preserves parser defaults, hidden-field removal and enumerability changes", () => {
|
||||
const source = freezeJsonSnapshot({ observed: 1 });
|
||||
expect(
|
||||
canBorrowParsedJsonSnapshot(source, { observed: 1, defaulted: 0 }),
|
||||
).toBe(false);
|
||||
const hidden = { observed: 1 };
|
||||
Object.defineProperty(hidden, "unmodeled", { value: 2, enumerable: false });
|
||||
freezeJsonSnapshot(hidden);
|
||||
expect(canBorrowParsedJsonSnapshot(hidden, { observed: 1 })).toBe(false);
|
||||
const nonenumerable = { observed: 1 };
|
||||
Object.defineProperty(nonenumerable, "observed", { enumerable: false });
|
||||
freezeJsonSnapshot(nonenumerable);
|
||||
expect(canBorrowParsedJsonSnapshot(nonenumerable, { observed: 1 })).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
it("does not evaluate accessors introduced by a parsed representation", () => {
|
||||
let accessed = false;
|
||||
const source = freezeJsonSnapshot({ observed: 1 });
|
||||
expect(
|
||||
canBorrowParsedJsonSnapshot(source, {
|
||||
get observed() {
|
||||
accessed = true;
|
||||
return 1;
|
||||
},
|
||||
}),
|
||||
).toBe(false);
|
||||
expect(accessed).toBe(false);
|
||||
});
|
||||
|
||||
it("declines parser-introduced proxies without reflecting on them", () => {
|
||||
let reflected = false;
|
||||
const parsed = new Proxy(
|
||||
{ value: 1 },
|
||||
{
|
||||
getPrototypeOf() {
|
||||
reflected = true;
|
||||
throw new Error("must not reflect on a proxy");
|
||||
},
|
||||
},
|
||||
);
|
||||
expect(
|
||||
canBorrowParsedJsonSnapshot(freezeJsonSnapshot({ value: 1 }), parsed),
|
||||
).toBe(false);
|
||||
expect(reflected).toBe(false);
|
||||
});
|
||||
|
||||
it("compares deeply nested snapshots without depending on the call stack", () => {
|
||||
let source: JsonValue = 1;
|
||||
let parsed: JsonValue = 1;
|
||||
for (let depth = 0; depth < 6_000; depth += 1) {
|
||||
source = { child: source };
|
||||
parsed = { child: parsed };
|
||||
}
|
||||
freezeJsonSnapshot(source);
|
||||
expect(canBorrowParsedJsonSnapshot(source, parsed)).toBe(true);
|
||||
});
|
||||
|
||||
it("compares shared snapshot graphs without repeatedly expanding their paths", () => {
|
||||
let source: JsonValue = { value: 1 };
|
||||
for (let depth = 0; depth < 26; depth += 1)
|
||||
source = { left: source, right: source };
|
||||
const parsed = structuredClone(source);
|
||||
freezeJsonSnapshot(source);
|
||||
const ownKeys = Reflect.ownKeys;
|
||||
let inspections = 0;
|
||||
const observed = vi
|
||||
.spyOn(Reflect, "ownKeys")
|
||||
.mockImplementation((value) => {
|
||||
// Fail deterministically instead of consuming exponential CPU on regressions.
|
||||
if (++inspections > 10_000)
|
||||
throw new Error("Shared JSON comparison exceeded inspection budget");
|
||||
return ownKeys(value);
|
||||
});
|
||||
let borrowed: boolean;
|
||||
try {
|
||||
borrowed = canBorrowParsedJsonSnapshot(source, parsed);
|
||||
} finally {
|
||||
observed.mockRestore();
|
||||
}
|
||||
expect(borrowed).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps the parsed result when a parser splits shared source nodes", () => {
|
||||
const shared = { value: 1 };
|
||||
const source = freezeJsonSnapshot({ left: shared, right: shared });
|
||||
expect(
|
||||
canBorrowParsedJsonSnapshot(source, {
|
||||
left: { value: 1 },
|
||||
right: { value: 1 },
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it.each(["direct", "nested"])(
|
||||
"detects shared-node splits across a %s preserved subtree",
|
||||
(kind) => {
|
||||
const shared = { value: 1 };
|
||||
const source = freezeJsonSnapshot({
|
||||
preserved: kind === "direct" ? shared : { nested: shared },
|
||||
cloned: shared,
|
||||
});
|
||||
expect(
|
||||
canBorrowParsedJsonSnapshot(source, {
|
||||
preserved: source.preserved,
|
||||
cloned: { value: 1 },
|
||||
}),
|
||||
).toBe(false);
|
||||
expect(canBorrowParsedJsonSnapshot(source, source)).toBe(true);
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps the parsed result when a parser merges distinct source nodes", () => {
|
||||
const source = freezeJsonSnapshot({
|
||||
left: { value: 1 },
|
||||
right: { value: 1 },
|
||||
});
|
||||
const shared = { value: 1 };
|
||||
expect(
|
||||
canBorrowParsedJsonSnapshot(source, { left: shared, right: shared }),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -1,3 +1,5 @@
|
||||
import { isProxy } from "node:util/types";
|
||||
|
||||
import type { JsonValue } from "./jsonValue.js";
|
||||
|
||||
const immutableSnapshots = new WeakSet<object>();
|
||||
@@ -14,6 +16,87 @@ export const isImmutableJsonSnapshot = (value: unknown): value is JsonValue =>
|
||||
(typeof value === "number" && Number.isFinite(value)) ||
|
||||
(typeof value === "object" && immutableSnapshots.has(value));
|
||||
|
||||
/** Borrow one own member of a completed snapshot, carrying its sealing proof. */
|
||||
export const immutableJsonSnapshotMember = (
|
||||
snapshot: unknown,
|
||||
key: string,
|
||||
): JsonValue | undefined => {
|
||||
if (
|
||||
typeof snapshot !== "object" ||
|
||||
snapshot === null ||
|
||||
!immutableSnapshots.has(snapshot)
|
||||
)
|
||||
return undefined;
|
||||
const descriptor = Object.getOwnPropertyDescriptor(snapshot, key);
|
||||
if (descriptor === undefined || !Object.hasOwn(descriptor, "value"))
|
||||
return undefined;
|
||||
// Completed sealing checked and froze every reachable member. Register only
|
||||
// the selected child; do not traverse it again or add per-node weak entries.
|
||||
const member = descriptor.value as JsonValue;
|
||||
if (typeof member === "object" && member !== null)
|
||||
immutableSnapshots.add(member);
|
||||
return member;
|
||||
};
|
||||
|
||||
/** Match a completed snapshot to its parsed representation without recursion. */
|
||||
export const canBorrowParsedJsonSnapshot = <Parsed extends JsonValue>(
|
||||
source: unknown,
|
||||
parsed: Parsed,
|
||||
): source is Parsed => {
|
||||
if (!isImmutableJsonSnapshot(source)) return false;
|
||||
if (Object.is(source, parsed)) return true;
|
||||
const pending: unknown[] = [source, parsed];
|
||||
const compared = new Map<object, object>();
|
||||
const sources = new Map<object, object>();
|
||||
while (pending.length > 0) {
|
||||
const right = pending.pop();
|
||||
const left = pending.pop();
|
||||
if (
|
||||
typeof left !== "object" ||
|
||||
left === null ||
|
||||
typeof right !== "object" ||
|
||||
right === null
|
||||
) {
|
||||
if (!Object.is(left, right)) return false;
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
isProxy(right) ||
|
||||
Object.getPrototypeOf(left) !== Object.getPrototypeOf(right)
|
||||
)
|
||||
return false;
|
||||
const previous = compared.get(left);
|
||||
if (previous !== undefined) {
|
||||
if (previous !== right) return false;
|
||||
continue;
|
||||
}
|
||||
if (sources.has(right)) return false;
|
||||
// Preserve sharing in both directions, including descendants of an
|
||||
// identical subtree that may also appear in a separately parsed field.
|
||||
compared.set(left, right);
|
||||
sources.set(right, left);
|
||||
const keys = Reflect.ownKeys(left);
|
||||
if (keys.length !== Reflect.ownKeys(right).length) return false;
|
||||
for (const key of keys) {
|
||||
// A parser may drop hidden fields or normalize enumerability. Generic
|
||||
// deep equality omits these differences and can borrow unparsed data.
|
||||
if (typeof key !== "string") return false;
|
||||
const a = Object.getOwnPropertyDescriptor(left, key);
|
||||
const b = Object.getOwnPropertyDescriptor(right, key);
|
||||
if (
|
||||
a === undefined ||
|
||||
b === undefined ||
|
||||
!Object.hasOwn(a, "value") ||
|
||||
!Object.hasOwn(b, "value") ||
|
||||
a.enumerable !== b.enumerable
|
||||
)
|
||||
return false;
|
||||
pending.push(a.value as unknown, b.value as unknown);
|
||||
}
|
||||
}
|
||||
return true;
|
||||
};
|
||||
|
||||
/** Seal an already parsed, owned JSON snapshot for immutable zero-copy reuse. */
|
||||
export const freezeJsonSnapshot = <T extends JsonValue>(value: T): T => {
|
||||
const pending: SnapshotEntry[] = [{ value, exiting: false }];
|
||||
@@ -104,6 +187,8 @@ export function* freezeOwnedJsonSnapshotSteps<T extends JsonValue>(
|
||||
}
|
||||
|
||||
const pushSnapshotChildren = (item: object, pending: SnapshotEntry[]): void => {
|
||||
if (isProxy(item))
|
||||
throw new TypeError("Immutable JSON snapshots cannot contain proxies");
|
||||
const prototype: unknown = Object.getPrototypeOf(item);
|
||||
if (
|
||||
!Array.isArray(item) &&
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
freezeOwnedJsonSnapshotSteps,
|
||||
immutableJsonSnapshotMember,
|
||||
isImmutableJsonSnapshot,
|
||||
} from "./immutableJson.js";
|
||||
import { jsonValueValidationIssue, type JsonValue } from "./jsonValue.js";
|
||||
@@ -23,6 +24,7 @@ describe("cooperative owned snapshot sealing", () => {
|
||||
expect(steps.next().done).toBe(false);
|
||||
expect(Object.isFrozen(value)).toBe(true);
|
||||
expect(isImmutableJsonSnapshot(value)).toBe(false);
|
||||
expect(immutableJsonSnapshotMember(value, "deferred")).toBeUndefined();
|
||||
const introduced = { value: 2 };
|
||||
deferred.child = introduced;
|
||||
expect(complete(steps)).toBe(value);
|
||||
@@ -49,6 +51,7 @@ describe("cooperative owned snapshot sealing", () => {
|
||||
expect(interrupted.next().done).toBe(false);
|
||||
interrupted.return(cancelled);
|
||||
expect(isImmutableJsonSnapshot(cancelled)).toBe(false);
|
||||
expect(immutableJsonSnapshotMember(cancelled, "values")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("preserves aliases and rejects accessors, sparse values and cycles", () => {
|
||||
@@ -80,4 +83,26 @@ describe("cooperative owned snapshot sealing", () => {
|
||||
value.child.value = Number.NaN;
|
||||
expect(jsonValueValidationIssue(value)).toBe("JSON numbers must be finite");
|
||||
});
|
||||
|
||||
it("never authenticates roots containing proxies or invokes their traps", () => {
|
||||
let reflected = false;
|
||||
const proxy = new Proxy(
|
||||
{ value: 1 },
|
||||
{
|
||||
getPrototypeOf() {
|
||||
reflected = true;
|
||||
throw new Error("must not reflect on a proxy");
|
||||
},
|
||||
},
|
||||
);
|
||||
const root = { child: proxy };
|
||||
for (const input of [proxy, root]) {
|
||||
expect(() => complete(freezeOwnedJsonSnapshotSteps(input))).toThrow(
|
||||
"proxies",
|
||||
);
|
||||
expect(isImmutableJsonSnapshot(input)).toBe(false);
|
||||
expect(immutableJsonSnapshotMember(input, "child")).toBeUndefined();
|
||||
}
|
||||
expect(reflected).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -1,4 +1,5 @@
|
||||
import { z } from "zod";
|
||||
import { canBorrowParsedJsonSnapshot } from "../immutableJson.js";
|
||||
|
||||
import {
|
||||
javascriptApplicationGraphSchema,
|
||||
@@ -172,21 +173,43 @@ const ownedAnalysisResultShape = applicationAnalysisResultShape.extend({
|
||||
),
|
||||
});
|
||||
|
||||
// One successful interpretation per exact immutable source. Weak keys retain
|
||||
// neither a second graph nor a source beyond its Evidence lifetime.
|
||||
const validatedImmutableResults = new WeakSet<object>();
|
||||
const isValidatedImmutableResult = (
|
||||
input: unknown,
|
||||
): input is JavaScriptApplicationAnalysisResult =>
|
||||
typeof input === "object" &&
|
||||
input !== null &&
|
||||
validatedImmutableResults.has(input);
|
||||
|
||||
/** Reuse exact owned graphs while validating metadata and cross-graph bindings. */
|
||||
export function* parseOwnedJavaScriptApplicationAnalysisSteps(
|
||||
input: unknown,
|
||||
): Generator<void, JavaScriptApplicationAnalysisResult> {
|
||||
if (isValidatedImmutableResult(input)) return input;
|
||||
const prepared = ownedAnalysisResultShape.safeParse(input);
|
||||
if (!prepared.success)
|
||||
return javascriptApplicationAnalysisResultSchema.parse(input);
|
||||
const issues: BindingIssue[] = [];
|
||||
yield* checkApplicationBindingsSteps(prepared.data, {
|
||||
addIssue: (issue) => {
|
||||
issues.push(issue);
|
||||
},
|
||||
});
|
||||
if (issues.length > 0) throw new z.ZodError(issues);
|
||||
return prepared.data;
|
||||
if (prepared.success) {
|
||||
const issues: BindingIssue[] = [];
|
||||
yield* checkApplicationBindingsSteps(prepared.data, {
|
||||
addIssue: (issue) => {
|
||||
issues.push(issue);
|
||||
},
|
||||
});
|
||||
if (issues.length > 0) throw new z.ZodError(issues);
|
||||
// Keep this existing fast path and its metadata normalization. Rewalking
|
||||
// already owned graphs just to borrow the wrapper would add full-graph work.
|
||||
return prepared.data;
|
||||
}
|
||||
const result = javascriptApplicationAnalysisResultSchema.parse(input);
|
||||
// Borrow only if parsing preserved the exact representation. A future schema
|
||||
// default/normalization must still return its parsed value, without caching.
|
||||
// Admission follows full validation, including completion of yielding checks.
|
||||
if (canBorrowParsedJsonSnapshot(input, result)) {
|
||||
validatedImmutableResults.add(input);
|
||||
return input;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Parsed public JavaScript application analysis request. */
|
||||
|
||||
@@ -56,6 +56,101 @@ const issuesFrom = (action: () => unknown) => {
|
||||
throw new Error("Expected complete result validation to reject the input");
|
||||
};
|
||||
|
||||
describe("immutable JavaScript result reuse", () => {
|
||||
it("preserves parser copies shared across owned graphs and result metadata", () => {
|
||||
const owned = ownedExample();
|
||||
const input = freezeJsonSnapshot({
|
||||
...owned,
|
||||
limitations: owned.graph.limitations,
|
||||
});
|
||||
for (let repeat = 0; repeat < 2; repeat += 1) {
|
||||
const parsed = complete(
|
||||
parseOwnedJavaScriptApplicationAnalysisSteps(input),
|
||||
);
|
||||
expect(parsed).not.toBe(input);
|
||||
expect(parsed.graph).toBe(input.graph);
|
||||
expect(parsed.limitations).toEqual(input.graph.limitations);
|
||||
expect(parsed.limitations).not.toBe(parsed.graph.limitations);
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["hidden-extra", "changed-enumerability"])(
|
||||
"preserves parsed results when the source has %s normalization",
|
||||
(kind) => {
|
||||
const input = example();
|
||||
if (kind === "hidden-extra")
|
||||
Object.defineProperty(input, "not_in_parsed_result", {
|
||||
value: { unvalidated: true },
|
||||
enumerable: false,
|
||||
});
|
||||
else Object.defineProperty(input, "input_path", { enumerable: false });
|
||||
freezeJsonSnapshot(input);
|
||||
const expected = javascriptApplicationAnalysisResultSchema.parse(input);
|
||||
for (let repeat = 0; repeat < 2; repeat += 1) {
|
||||
const parsed = complete(
|
||||
parseOwnedJavaScriptApplicationAnalysisSteps(input),
|
||||
);
|
||||
expect(parsed).not.toBe(input);
|
||||
expect(Object.hasOwn(parsed, "not_in_parsed_result")).toBe(false);
|
||||
expect(
|
||||
Object.getOwnPropertyDescriptor(parsed, "input_path")?.enumerable,
|
||||
).toBe(
|
||||
Object.getOwnPropertyDescriptor(expected, "input_path")?.enumerable,
|
||||
);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("borrows imported immutable results only after complete format validation", () => {
|
||||
const input = example();
|
||||
delete input.statistics.skipped_javascript_files;
|
||||
freezeJsonSnapshot(input);
|
||||
const result = complete(
|
||||
parseOwnedJavaScriptApplicationAnalysisSteps(input),
|
||||
);
|
||||
expect(result).toBe(input);
|
||||
expect(result).toEqual(
|
||||
javascriptApplicationAnalysisResultSchema.parse(input),
|
||||
);
|
||||
expect(result.statistics).not.toHaveProperty("skipped_javascript_files");
|
||||
expect(complete(parseOwnedJavaScriptApplicationAnalysisSteps(input))).toBe(
|
||||
input,
|
||||
);
|
||||
expect(Reflect.set(input.statistics, "findings", -1)).toBe(false);
|
||||
});
|
||||
|
||||
it("preserves a normalized representation instead of borrowing changed input", () => {
|
||||
const input = example();
|
||||
Object.setPrototypeOf(input, null);
|
||||
freezeJsonSnapshot(input);
|
||||
for (let repeat = 0; repeat < 2; repeat += 1) {
|
||||
const result = complete(
|
||||
parseOwnedJavaScriptApplicationAnalysisSteps(input),
|
||||
);
|
||||
expect(result).not.toBe(input);
|
||||
expect(Object.getPrototypeOf(result)).toBe(Object.prototype);
|
||||
expect(result).toEqual(
|
||||
javascriptApplicationAnalysisResultSchema.parse(input),
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["mutable", "shallow-frozen"])(
|
||||
"rechecks changed %s metadata despite authenticated nested graphs",
|
||||
(kind) => {
|
||||
const input = ownedExample();
|
||||
if (kind === "shallow-frozen") Object.freeze(input);
|
||||
expect(
|
||||
complete(parseOwnedJavaScriptApplicationAnalysisSteps(input)),
|
||||
).not.toBe(input);
|
||||
input.statistics.findings = -1;
|
||||
expect(() =>
|
||||
complete(parseOwnedJavaScriptApplicationAnalysisSteps(input)),
|
||||
).toThrow(z.ZodError);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe("owned JavaScript graph validation", () => {
|
||||
it("reuses exact sealed graphs and preserves all public result fields", () => {
|
||||
const input = ownedExample();
|
||||
@@ -239,6 +334,7 @@ describe("owned JavaScript result bindings", () => {
|
||||
input.semantic_graph = complete(
|
||||
createImmutableJavaScriptSemanticGraphSteps(semantic),
|
||||
);
|
||||
freezeJsonSnapshot(input);
|
||||
const steps = parseOwnedJavaScriptApplicationAnalysisSteps(input);
|
||||
expect(steps.next().done).toBe(false);
|
||||
steps.return(input);
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { binaryLayoutSchema, type BinaryLayout } from "./binaryLayout.js";
|
||||
import {
|
||||
binaryLayoutSchema,
|
||||
safeParseOwnedBinaryLayout,
|
||||
type BinaryLayout,
|
||||
} from "./binaryLayout.js";
|
||||
import { freezeJsonSnapshot } from "../immutableJson.js";
|
||||
|
||||
const report = () => ({
|
||||
artifact: { path: "/fixture.elf", sha256: "a".repeat(64), bytes: 128 },
|
||||
@@ -60,6 +65,55 @@ const report = () => ({
|
||||
limitations: [],
|
||||
});
|
||||
|
||||
it("reuses an exact validated immutable layout without a second retained copy", () => {
|
||||
const value = freezeJsonSnapshot(report());
|
||||
const parsed = safeParseOwnedBinaryLayout(value);
|
||||
expect(parsed.success).toBe(true);
|
||||
if (!parsed.success) throw parsed.error;
|
||||
expect(parsed.data).toBe(value);
|
||||
expect(safeParseOwnedBinaryLayout(value)).toEqual(parsed);
|
||||
expect(parsed.data).toEqual(binaryLayoutSchema.parse(value));
|
||||
expect(Reflect.set(value.sections[0]!, "index", 5)).toBe(false);
|
||||
});
|
||||
|
||||
it.each(["mutable", "shallow-frozen"])(
|
||||
"revalidates a changed %s layout instead of caching its former validity",
|
||||
(kind) => {
|
||||
const value = report();
|
||||
if (kind === "shallow-frozen") Object.freeze(value);
|
||||
const first = safeParseOwnedBinaryLayout(value);
|
||||
if (!first.success) throw first.error;
|
||||
expect(first.data).not.toBe(value);
|
||||
value.sections[0]!.header_location.offset = "0x80";
|
||||
const invalid = safeParseOwnedBinaryLayout(value);
|
||||
expect(invalid.success).toBe(false);
|
||||
expect(invalid).toEqual(binaryLayoutSchema.safeParse(value));
|
||||
},
|
||||
);
|
||||
|
||||
it("never treats sealing as format validity or caches a refused layout", () => {
|
||||
const value = report();
|
||||
value.sections[0]!.index = 10;
|
||||
freezeJsonSnapshot(value);
|
||||
expect(safeParseOwnedBinaryLayout(value).success).toBe(false);
|
||||
expect(safeParseOwnedBinaryLayout(value).success).toBe(false);
|
||||
});
|
||||
|
||||
it("does not borrow hidden fields that the format parser removed", () => {
|
||||
const value = report();
|
||||
Object.defineProperty(value, "not_in_parsed_result", {
|
||||
value: { unvalidated: true },
|
||||
enumerable: false,
|
||||
});
|
||||
freezeJsonSnapshot(value);
|
||||
for (let repeat = 0; repeat < 2; repeat += 1) {
|
||||
const parsed = safeParseOwnedBinaryLayout(value);
|
||||
if (!parsed.success) throw parsed.error;
|
||||
expect(parsed.data).not.toBe(value);
|
||||
expect(Object.hasOwn(parsed.data, "not_in_parsed_result")).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("retains high linked values and NOBITS without inventing file backing", () => {
|
||||
const value = binaryLayoutSchema.parse(report());
|
||||
expect(value.entry_point.reported_value).toBe("0x20000000000001");
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { z } from "zod";
|
||||
import { canBorrowParsedJsonSnapshot } from "../immutableJson.js";
|
||||
|
||||
import { localPathStringSchema } from "../localPath.js";
|
||||
import { CANONICAL_BASE64_PATTERN } from "../stringPatterns.js";
|
||||
@@ -316,6 +317,28 @@ export type InspectBinaryLayoutInput = z.infer<
|
||||
>;
|
||||
export type BinaryLayout = z.infer<typeof binaryLayoutSchema>;
|
||||
|
||||
const validatedImmutableLayouts = new WeakSet<object>();
|
||||
|
||||
/** Validate once per exact immutable source; mutable inputs always revalidate. */
|
||||
export const safeParseOwnedBinaryLayout = (
|
||||
input: unknown,
|
||||
): z.ZodSafeParseResult<BinaryLayout> => {
|
||||
if (
|
||||
typeof input === "object" &&
|
||||
input !== null &&
|
||||
validatedImmutableLayouts.has(input)
|
||||
)
|
||||
return { success: true, data: input as BinaryLayout };
|
||||
const parsed = binaryLayoutSchema.safeParse(input);
|
||||
if (parsed.success && canBorrowParsedJsonSnapshot(input, parsed.data)) {
|
||||
// Preserve normalization if the schema changes: only unchanged, completely
|
||||
// validated data can be borrowed without retaining a duplicate layout.
|
||||
validatedImmutableLayouts.add(input);
|
||||
return { success: true, data: input };
|
||||
}
|
||||
return parsed;
|
||||
};
|
||||
|
||||
const validateRelocationReferences = (
|
||||
value: z.output<typeof binaryLayoutObjectSchema>,
|
||||
context: z.RefinementCtx,
|
||||
|
||||
@@ -2,20 +2,10 @@ import { describe, expect, it } from "vitest";
|
||||
import { parse } from "plist";
|
||||
|
||||
import {
|
||||
decodeXmlText,
|
||||
omitPrototypeKeys,
|
||||
parseXmlPropertyList as parseWithNumbers,
|
||||
} from "./propertyListKeys.js";
|
||||
|
||||
it("preserves unknown named entities that collide with object prototype properties", () => {
|
||||
expect(
|
||||
decodeXmlText("&constructor; &toString; &valueOf; &hasOwnProperty;"),
|
||||
).toBe("&constructor; &toString; &valueOf; &hasOwnProperty;");
|
||||
expect(decodeXmlText("&<>"'AB")).toBe(
|
||||
"&<>\"'AB",
|
||||
);
|
||||
});
|
||||
|
||||
// Existing key/omission controls assert those facts exactly; numeric facts
|
||||
// have separate source-type and ambiguity controls below.
|
||||
const parseXmlPropertyList = (text: string) => {
|
||||
|
||||
@@ -1,15 +1,13 @@
|
||||
import { normalizeLineEndings, type Document, type Node } from "@xmldom/xmldom";
|
||||
import { parseOpenStep } from "plist";
|
||||
|
||||
import { decodeXmlPropertyList } from "./propertyListXml.js";
|
||||
import {
|
||||
UnknownPlistReal,
|
||||
createPlistNumberCollector,
|
||||
type PlistNumberLiterals,
|
||||
} from "./apple/plistNumbers.js";
|
||||
import { UnknownPlistScalar } from "./apple/unknownPlistScalar.js";
|
||||
|
||||
import { MAX_PLIST_DEPTH } from "./apple/plistValue.js";
|
||||
import { parseXmlDocument } from "./xmlDocument.js";
|
||||
|
||||
/** The dictionary key REA results cannot carry: validation strips it. */
|
||||
const PROTOTYPE_KEY = "__proto__";
|
||||
@@ -43,7 +41,7 @@ export const omitPrototypeKeys = (
|
||||
typeof item !== "object" ||
|
||||
item instanceof Date ||
|
||||
item instanceof Uint8Array ||
|
||||
item instanceof UnknownPlistReal
|
||||
item instanceof UnknownPlistScalar
|
||||
)
|
||||
return item;
|
||||
return Object.fromEntries(
|
||||
@@ -58,37 +56,6 @@ export const omitPrototypeKeys = (
|
||||
return { value: stripped, omittedPrototypeKeys: omitted };
|
||||
};
|
||||
|
||||
const XML_ENTITIES: Readonly<Record<string, string>> = {
|
||||
amp: "&",
|
||||
apos: "'",
|
||||
gt: ">",
|
||||
lt: "<",
|
||||
quot: '"',
|
||||
};
|
||||
|
||||
/**
|
||||
* Expand XML character and predefined entity references. A reference beyond
|
||||
* Unicode stays as written: comments and CDATA may hold such text literally,
|
||||
* and the XML decoder rejects it anywhere else.
|
||||
*/
|
||||
export const decodeXmlText = (text: string): string =>
|
||||
text.replace(
|
||||
/&(?:#x([\da-f]+)|#(\d+)|([a-z]+));/giu,
|
||||
(entity, hex?: string, decimal?: string, name?: string) => {
|
||||
const codePoint =
|
||||
hex !== undefined
|
||||
? parseInt(hex, 16)
|
||||
: decimal !== undefined
|
||||
? parseInt(decimal, 10)
|
||||
: undefined;
|
||||
if (codePoint !== undefined)
|
||||
return codePoint <= 0x10ffff ? String.fromCodePoint(codePoint) : entity;
|
||||
return name !== undefined && Object.hasOwn(XML_ENTITIES, name)
|
||||
? XML_ENTITIES[name]!
|
||||
: entity;
|
||||
},
|
||||
);
|
||||
|
||||
/**
|
||||
* Decode one validated XML DOM into values and numeric source facts.
|
||||
* Entries keyed `__proto__` are omitted and counted at the same boundary.
|
||||
@@ -108,20 +75,7 @@ export const parseXmlPropertyList = (text: string): ParsedPropertyList => {
|
||||
omittedPrototypeKeys: prototypeKeys,
|
||||
};
|
||||
}
|
||||
// Normalize only the literal line endings the pinned decoder normalizes.
|
||||
// Decode this DOM directly so character references are never reserialized
|
||||
// into literal line endings and normalized a second time.
|
||||
const normalized = normalizeLineEndings(text);
|
||||
// Strict syntax owns well-formedness before value interpretation. Reuse the
|
||||
// same DOM for depth, keys, values and number facts.
|
||||
const parsed = parseXmlDocument(normalized, {
|
||||
doctype: "allow",
|
||||
lineEndings: "preserve",
|
||||
});
|
||||
if (!parsed.ok) throw parsed.error;
|
||||
const document = parsed.value;
|
||||
assertXmlPlistDepthWithinLimit(document);
|
||||
const decoded = decodeXmlPropertyList(document);
|
||||
const decoded = decodeXmlPropertyList(text);
|
||||
return {
|
||||
...omitPrototypeKeys(decoded.value),
|
||||
numberLiterals: decoded.numberLiterals,
|
||||
@@ -288,43 +242,3 @@ const skipOpenStepScalar = (text: string, start: number): number => {
|
||||
}
|
||||
return cursor;
|
||||
};
|
||||
|
||||
/** Elements whose children the plist value decoder visits recursively. */
|
||||
const PLIST_CONTAINER_ELEMENTS = new Set(["plist", "dict", "array"]);
|
||||
|
||||
/**
|
||||
* Reject XML plists nested deeper than the projection supports before any
|
||||
* recursive value pass runs. Value decoding and prototype omission recurse once
|
||||
* per nesting level, so a deeply nested document exhausts the call stack
|
||||
* before the projection limit can report the resource bound. The failure
|
||||
* matches `projectPlistValue` so callers classify it as a limit failure.
|
||||
*
|
||||
* Depth counts values the way the projection does: the root value is level 0
|
||||
* and every container member is one level deeper. A dictionary key names a
|
||||
* member instead of nesting one, and a self-closing element is the value at
|
||||
* its own level. The outer plist envelope is unwrapped by the decoder;
|
||||
* nested plist elements decode as arrays and must count toward the bound.
|
||||
*/
|
||||
const assertXmlPlistDepthWithinLimit = (document: Document): void => {
|
||||
const root = document.documentElement;
|
||||
if (root === null) return;
|
||||
const pending: Array<{ node: Node; depth: number }> = [
|
||||
{ node: root, depth: -1 },
|
||||
];
|
||||
while (pending.length > 0) {
|
||||
const entry = pending.pop();
|
||||
if (entry === undefined) break;
|
||||
const { node, depth } = entry;
|
||||
// Schedule only the next sibling, keeping traversal storage proportional
|
||||
// to depth rather than the number of members in a wide container.
|
||||
if (node.nextSibling !== null)
|
||||
pending.push({ node: node.nextSibling, depth });
|
||||
if (node.nodeType !== node.ELEMENT_NODE) continue;
|
||||
if (node.nodeName !== "key" && depth > MAX_PLIST_DEPTH)
|
||||
throw new RangeError(
|
||||
`Plist exceeds ${String(MAX_PLIST_DEPTH)} nesting levels`,
|
||||
);
|
||||
if (PLIST_CONTAINER_ELEMENTS.has(node.nodeName) && node.firstChild !== null)
|
||||
pending.push({ node: node.firstChild, depth: depth + 1 });
|
||||
}
|
||||
};
|
||||
@@ -1,16 +1,85 @@
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
import { projectPlistValue } from "./apple/plistValue.js";
|
||||
import { UnknownPlistReal } from "./apple/plistNumbers.js";
|
||||
import { UnknownPlistScalar } from "./apple/unknownPlistScalar.js";
|
||||
import { parseXmlPropertyList } from "./propertyListKeys.js";
|
||||
|
||||
it.each([
|
||||
[
|
||||
"plist",
|
||||
"<plist><?dict ignored?><string>demo</string><?rea ignored?></plist>",
|
||||
"demo",
|
||||
],
|
||||
[
|
||||
"array",
|
||||
"<plist><array><?rea ignored?><string>before</string><?integer 9007199254740993?><integer>42</integer><?rea ignored?></array></plist>",
|
||||
["before", 42],
|
||||
],
|
||||
[
|
||||
"dict",
|
||||
"<plist><dict><?key ignored?><key>name</key><?string ignored?><string>demo</string><?rea ignored?><key>count</key><integer>42</integer><?rea ignored?></dict></plist>",
|
||||
{ name: "demo", count: 42 },
|
||||
],
|
||||
] as const)(
|
||||
"ignores processing instructions between %s members",
|
||||
(_kind, xml, value) => {
|
||||
const parsed = parseXmlPropertyList(xml);
|
||||
expect(parsed.value).toEqual(value);
|
||||
expect(parsed.numberLiterals.incomplete).toBe(false);
|
||||
expect(parsed.numberLiterals.unsafeIntegerLiterals).toBe(0);
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps document-level processing instructions outside plist members", () => {
|
||||
const parsed = parseXmlPropertyList(
|
||||
'<?xml version="1.0"?><?rea before?><plist><string>demo</string></plist><?rea after?>',
|
||||
);
|
||||
expect(parsed.value).toBe("demo");
|
||||
expect(parsed.numberLiterals.incomplete).toBe(false);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["CDATA", "<![CDATA[before<?rea content?>after]]>"],
|
||||
["escaped text", "before<?rea content?>after"],
|
||||
])(
|
||||
"preserves literal processing-instruction text from %s",
|
||||
(_kind, content) => {
|
||||
expect(
|
||||
parseXmlPropertyList(`<plist><string>${content}</string></plist>`).value,
|
||||
).toBe("before<?rea content?>after");
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps processing instructions inside numeric content subject to scalar rules", () => {
|
||||
for (const markup of ["<?rea ignored?>", "<!-- ignored -->"]) {
|
||||
const parsed = parseXmlPropertyList(
|
||||
`<plist><integer>4${markup}2</integer></plist>`,
|
||||
);
|
||||
expect(parsed.value).toEqual(
|
||||
new UnknownPlistScalar("integer", `4${markup}2`, "unsupported-markup"),
|
||||
);
|
||||
expect(parsed.numberLiterals.incomplete).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("still rejects an unterminated processing instruction", () => {
|
||||
expect(() =>
|
||||
parseXmlPropertyList("<plist><?rea ignored<string>demo</string></plist>"),
|
||||
).toThrow();
|
||||
});
|
||||
|
||||
it("still rejects an unsupported element beside a processing instruction", () => {
|
||||
expect(() =>
|
||||
parseXmlPropertyList("<plist><?rea ignored?><unsupported/></plist>"),
|
||||
).toThrow("Invalid PLIST tag unsupported");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["0x2A", 42],
|
||||
["0X2a", 42],
|
||||
["+0x2A", 42],
|
||||
["-0x2A", -42],
|
||||
["0x2A", 42],
|
||||
["\n0x2A\n", 42],
|
||||
["\n0x2A", 42],
|
||||
["42", 42],
|
||||
])(
|
||||
"decodes exact integer text %s at the shared boundary",
|
||||
@@ -24,6 +93,20 @@ it.each([
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
["AQID", [1, 2, 3]],
|
||||
[" A Q I D\n", [1, 2, 3]],
|
||||
["AQI", []],
|
||||
["AQ!ID", [1, 2, 3]],
|
||||
["AQ==", [1]],
|
||||
["AQI=", [1, 2]],
|
||||
["AQ= =\n", [1]],
|
||||
["", []],
|
||||
])("decodes data text %j as CFPropertyList does", (text, expected) => {
|
||||
const parsed = parseXmlPropertyList(`<plist><data>${text}</data></plist>`);
|
||||
expect(parsed.value).toEqual(Uint8Array.from(expected));
|
||||
});
|
||||
|
||||
it("collects source integers and CDATA reals before rounding and duplicate assignment", () => {
|
||||
const parsed = parseXmlPropertyList(
|
||||
"<plist><dict><key>a</key><integer>0x20000000000001</integer><key>a</key><integer>9007199254740992</integer><key>b</key><real>9007199254<![CDATA[740992]]></real></dict></plist>",
|
||||
@@ -47,12 +130,24 @@ it("does not invent numeric facts from ignored scalar markup", () => {
|
||||
expect(parsed.numberLiterals.incomplete).toBe(false);
|
||||
});
|
||||
|
||||
it("retains partial numeric values with explicit incomplete metadata for unsupported markup", () => {
|
||||
it("keeps unsupported numeric markup out of numeric source facts", () => {
|
||||
const parsed = parseXmlPropertyList(
|
||||
"<plist><array><integer>0x2A<!-- unsupported --></integer><real>1<string>ignored</string>.25</real></array></plist>",
|
||||
);
|
||||
expect(parsed.value).toEqual([0, 1.25]);
|
||||
expect(parsed.numberLiterals.incomplete).toBe(true);
|
||||
expect(parsed.value).toEqual([
|
||||
new UnknownPlistScalar(
|
||||
"integer",
|
||||
"0x2A<!-- unsupported -->",
|
||||
"unsupported-markup",
|
||||
),
|
||||
new UnknownPlistScalar(
|
||||
"real",
|
||||
"1<string>ignored</string>.25",
|
||||
"unsupported-markup",
|
||||
),
|
||||
]);
|
||||
expect(parsed.numberLiterals.incomplete).toBe(false);
|
||||
expect(parsed.numberLiterals.reals.size).toBe(0);
|
||||
});
|
||||
|
||||
it("decodes a signed integer zero as zero and a negative-zero real with its sign", () => {
|
||||
@@ -67,7 +162,16 @@ it("decodes a signed integer zero as zero and a negative-zero real with its sign
|
||||
true,
|
||||
false,
|
||||
]);
|
||||
expect(parsed.value).toEqual([0, 0, -0, 0]);
|
||||
expect(parsed.value).toEqual([
|
||||
0,
|
||||
new UnknownPlistScalar(
|
||||
"integer",
|
||||
"-0<!-- partial -->",
|
||||
"unsupported-markup",
|
||||
),
|
||||
-0,
|
||||
0,
|
||||
]);
|
||||
});
|
||||
|
||||
it("interprets failures in omitted entries before reporting partial evidence", () => {
|
||||
@@ -110,38 +214,106 @@ it.each([
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps XML numeric overflow and unreadable text explicitly unknown", () => {
|
||||
const parsed = parseXmlPropertyList(
|
||||
"<plist><array><real>1e999</real><real>-1e999</real><real> nan </real><real>-nan</real><real>infinite</real></array></plist>",
|
||||
);
|
||||
expect(parsed.value).toEqual(
|
||||
Array.from({ length: 5 }, () => new UnknownPlistReal()),
|
||||
);
|
||||
expect(parsed.numberLiterals.incomplete).toBe(false);
|
||||
});
|
||||
|
||||
it("does not infer a spelled real from incomplete numeric markup", () => {
|
||||
const parsed = parseXmlPropertyList(
|
||||
"<plist><real>-inf<!-- unsupported --></real></plist>",
|
||||
);
|
||||
expect(parsed.value).toBeInstanceOf(UnknownPlistReal);
|
||||
expect(parsed.numberLiterals.incomplete).toBe(true);
|
||||
});
|
||||
it.each([
|
||||
["integer", "unreadable", "invalid-literal"],
|
||||
["integer", `${"9".repeat(400)}invalid`, "invalid-literal"],
|
||||
["integer", "0x2A", "invalid-literal"],
|
||||
["integer", "42\n", "invalid-literal"],
|
||||
["integer", "", "invalid-literal"],
|
||||
["real", "", "invalid-literal"],
|
||||
["real", "1e999", "unrepresentable"],
|
||||
["real", "-1e999", "unrepresentable"],
|
||||
["real", " nan ", "invalid-literal"],
|
||||
["real", "-nan", "invalid-literal"],
|
||||
["real", "infinite", "invalid-literal"],
|
||||
["real", "Infinitysuffix", "invalid-literal"],
|
||||
["real", "nan<!-- unsupported -->", "unsupported-markup"],
|
||||
["date", "2026-10-17T12:00:00+02:00", "invalid-literal"],
|
||||
["date", "2026-10-17T12:00:00.5Z", "invalid-literal"],
|
||||
["date", "2026-10-17T12:00:00", "invalid-literal"],
|
||||
["date", "2026-13-17T12:00:00Z", "unrepresentable"],
|
||||
["date", "2026-02-31T12:00:00Z", "unrepresentable"],
|
||||
["date", "<![CDATA[2026-10-17T12:00:00Z]]>", "unsupported-markup"],
|
||||
["true", "x", "invalid-literal"],
|
||||
["false", " ", "invalid-literal"],
|
||||
["true", "<!-- empty -->", "unsupported-markup"],
|
||||
] as const)(
|
||||
"keeps unresolved <%s> %j identity explicit",
|
||||
(element, literal, reason) => {
|
||||
const parsed = parseXmlPropertyList(
|
||||
`<plist><${element}>${literal}</${element}></plist>`,
|
||||
);
|
||||
expect(parsed.value).toEqual(
|
||||
new UnknownPlistScalar(element, literal, reason),
|
||||
);
|
||||
const projected = projectPlistValue(parsed.value);
|
||||
expect(projected.value).toEqual({
|
||||
$plist_type: "unknown",
|
||||
element,
|
||||
literal,
|
||||
reason,
|
||||
});
|
||||
expect(projected.unknownScalarCount).toBe(1);
|
||||
expect(projected.nonFiniteRealCount).toBe(0);
|
||||
expect(parsed.numberLiterals.incomplete).toBe(false);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
["integer", "unreadable"],
|
||||
["integer", `${"9".repeat(400)}invalid`],
|
||||
["real", "1e999"],
|
||||
["real", "-1e999"],
|
||||
["real", "Infinitysuffix"],
|
||||
["real", "nan<!-- unsupported -->"],
|
||||
])("keeps unresolved <%s> numeric identity explicit", (tag, text) => {
|
||||
"4<!-- </integer> -->2",
|
||||
"4<?rea </integer> ?>2",
|
||||
"4<![CDATA[</integer>]]>2",
|
||||
'4<integer note=">">2<integer/></integer>3',
|
||||
])(
|
||||
"retains complete source markup %j without treating apparent closing tags as structure",
|
||||
(literal) => {
|
||||
expect(
|
||||
parseXmlPropertyList(`<plist><integer>${literal}</integer></plist>`)
|
||||
.value,
|
||||
).toEqual(new UnknownPlistScalar("integer", literal, "unsupported-markup"));
|
||||
},
|
||||
);
|
||||
|
||||
it("binds raw literals to their nodes across normalized line endings and UTF-16 columns", () => {
|
||||
const parsed = parseXmlPropertyList(
|
||||
`<plist><${tag}>${text}</${tag}></plist>`,
|
||||
'<plist><array><string>💡</string>\r\n<integer note=">">42\r\nabc</integer>\u0085<date>2026-10-17T12:00:00Z</date>\u2028<false>\r\n</false></array></plist>',
|
||||
);
|
||||
expect(parsed.value).toBeInstanceOf(UnknownPlistReal);
|
||||
const projected = projectPlistValue(parsed.value);
|
||||
expect(projected.value).toEqual({ $plist_type: "real", value: null });
|
||||
expect(projected.unknownRealCount).toBe(1);
|
||||
expect(projected.nonFiniteRealCount).toBe(0);
|
||||
expect(parsed.value).toEqual([
|
||||
"💡",
|
||||
new UnknownPlistScalar("integer", "42\r\nabc", "invalid-literal"),
|
||||
new UnknownPlistScalar(
|
||||
"date",
|
||||
"2026-10-17T12:00:00Z",
|
||||
"invalid-literal",
|
||||
),
|
||||
new UnknownPlistScalar("false", "\r\n", "invalid-literal"),
|
||||
]);
|
||||
});
|
||||
|
||||
it.each(["\u0085", "\u2028", "\u2029"])(
|
||||
"does not normalize unsupported leading whitespace %j into an observed scalar",
|
||||
(prefix) => {
|
||||
const parsed = parseXmlPropertyList(
|
||||
`<plist><array><integer>${prefix}42</integer><real>${prefix}1.5</real></array></plist>`,
|
||||
);
|
||||
expect(parsed.value).toEqual([
|
||||
new UnknownPlistScalar("integer", `${prefix}42`, "invalid-literal"),
|
||||
prefix === "\u0085"
|
||||
? 1.5
|
||||
: new UnknownPlistScalar("real", `${prefix}1.5`, "invalid-literal"),
|
||||
]);
|
||||
},
|
||||
);
|
||||
|
||||
it("expands real references outside CDATA without changing scalar source policy", () => {
|
||||
expect(
|
||||
parseXmlPropertyList(
|
||||
"<plist><array><real>1<![CDATA[.5]]></real><real><![CDATA[1.5]]></real><true/><false></false></array></plist>",
|
||||
).value,
|
||||
).toEqual([
|
||||
1.5,
|
||||
new UnknownPlistScalar("real", "<![CDATA[1.5]]>", "invalid-literal"),
|
||||
true,
|
||||
false,
|
||||
]);
|
||||
});
|
||||
@@ -1,51 +1,58 @@
|
||||
import type { Document, Node } from "@xmldom/xmldom";
|
||||
import type { Node } from "@xmldom/xmldom";
|
||||
|
||||
import { readXmlPlistSource } from "./propertyListXmlSource.js";
|
||||
|
||||
import { createPlistNumberCollector } from "./apple/plistNumbers.js";
|
||||
import { UnknownPlistScalar } from "./apple/unknownPlistScalar.js";
|
||||
import {
|
||||
UnknownPlistReal,
|
||||
createPlistNumberCollector,
|
||||
parsePlistIntegerLiteral,
|
||||
parsePlistNonFiniteRealLiteral,
|
||||
} from "./apple/plistNumbers.js";
|
||||
decodeXmlPlistDateLiteral,
|
||||
parseXmlPlistIntegerLiteral,
|
||||
parseXmlPlistRealLiteral,
|
||||
} from "./propertyListXmlScalar.js";
|
||||
|
||||
/** Interpret an already validated, depth-bounded DOM, owning values and number facts together. */
|
||||
export const decodeXmlPropertyList = (document: Document) => {
|
||||
/** Own source interpretation, depth, values and numeric facts together. */
|
||||
export const decodeXmlPropertyList = (text: string) => {
|
||||
const source = readXmlPlistSource(text);
|
||||
const document = source.document;
|
||||
const root = document.documentElement;
|
||||
if (root?.nodeName !== "plist")
|
||||
throw new TypeError("malformed document. First element should be <plist>");
|
||||
const numbers = createPlistNumberCollector();
|
||||
const requireContent = (node: Node): Node => {
|
||||
if (node.firstChild === null)
|
||||
throw new TypeError(`Cannot parse "" as ${node.nodeName}`);
|
||||
return node.firstChild;
|
||||
};
|
||||
const textNode = (node: Node) => node.nodeType === node.TEXT_NODE;
|
||||
const characterNode = (node: Node) =>
|
||||
textNode(node) || node.nodeType === node.CDATA_SECTION_NODE;
|
||||
const content = (node: Node, accepts: (child: Node) => boolean) => {
|
||||
let text = "";
|
||||
let complete = true;
|
||||
for (
|
||||
let child = node.firstChild;
|
||||
child !== null;
|
||||
child = child.nextSibling
|
||||
) {
|
||||
if (accepts(child)) text += child.nodeValue ?? "";
|
||||
else complete = false;
|
||||
}
|
||||
return { text, complete };
|
||||
return text;
|
||||
};
|
||||
const hasUnsupportedMarkup = (
|
||||
node: Node,
|
||||
accepts: (child: Node) => boolean,
|
||||
): boolean => {
|
||||
for (let child = node.firstChild; child !== null; child = child.nextSibling)
|
||||
if (!accepts(child)) return true;
|
||||
return false;
|
||||
};
|
||||
const dictionaryKey = (node: Node): string => {
|
||||
for (let child = node.firstChild; child !== null; child = child.nextSibling)
|
||||
if (child.nodeType === child.ELEMENT_NODE)
|
||||
return String(node.firstChild?.nodeValue ?? null);
|
||||
return content(node, characterNode).text;
|
||||
return content(node, characterNode);
|
||||
};
|
||||
const members = function* (node: Node) {
|
||||
for (let child = node.firstChild; child !== null; child = child.nextSibling)
|
||||
if (
|
||||
child.nodeType !== child.TEXT_NODE &&
|
||||
child.nodeType !== child.CDATA_SECTION_NODE &&
|
||||
child.nodeType !== child.COMMENT_NODE
|
||||
child.nodeType !== child.COMMENT_NODE &&
|
||||
child.nodeType !== child.PROCESSING_INSTRUCTION_NODE
|
||||
)
|
||||
yield child;
|
||||
};
|
||||
@@ -91,61 +98,75 @@ export const decodeXmlPropertyList = (document: Document) => {
|
||||
return value;
|
||||
}
|
||||
case "string":
|
||||
return content(node, characterNode).text;
|
||||
return content(node, characterNode);
|
||||
case "integer": {
|
||||
const first = requireContent(node);
|
||||
const literal = content(node, textNode);
|
||||
const exact = literal.complete
|
||||
? parsePlistIntegerLiteral(literal.text)
|
||||
const raw = source.scalarLiteral(node);
|
||||
if (!raw.ok) throw raw.error;
|
||||
const unsupportedMarkup = hasUnsupportedMarkup(node, textNode);
|
||||
const exact = !unsupportedMarkup
|
||||
? parseXmlPlistIntegerLiteral(raw.value)
|
||||
: undefined;
|
||||
if (exact === undefined) {
|
||||
numbers.markIncomplete();
|
||||
const partial = Number.parseInt(first.nodeValue ?? "", 10);
|
||||
// An integer has no signed zero; keep -0 from reading as a real.
|
||||
return !Number.isFinite(partial)
|
||||
? new UnknownPlistReal()
|
||||
: partial === 0
|
||||
? 0
|
||||
: partial;
|
||||
}
|
||||
if (exact === undefined)
|
||||
return new UnknownPlistScalar(
|
||||
"integer",
|
||||
raw.value,
|
||||
unsupportedMarkup ? "unsupported-markup" : "invalid-literal",
|
||||
);
|
||||
numbers.integer(exact);
|
||||
const decoded = Number(exact);
|
||||
// Keep a finite integer's identity when it cannot fit even a Number
|
||||
// approximation; otherwise it would be mislabeled as a non-finite real.
|
||||
// Arbitrary-precision integers are a REA extension to CF's range.
|
||||
return Number.isFinite(decoded) ? decoded : exact;
|
||||
}
|
||||
case "real": {
|
||||
requireContent(node);
|
||||
const literal = content(node, characterNode);
|
||||
if (!literal.complete) numbers.markIncomplete();
|
||||
// CFPropertyList spells NaN and the infinities; parseFloat would read
|
||||
// the lowercase spellings as NaN and lose the kind and sign.
|
||||
const known = literal.complete
|
||||
? parsePlistNonFiniteRealLiteral(literal.text)
|
||||
: undefined;
|
||||
if (known !== undefined) {
|
||||
numbers.real(known);
|
||||
return known;
|
||||
}
|
||||
const value = Number.parseFloat(literal.text);
|
||||
if (!Number.isFinite(value)) return new UnknownPlistReal();
|
||||
if (literal.complete) numbers.real(value);
|
||||
return value;
|
||||
const raw = source.scalarLiteral(node);
|
||||
if (!raw.ok) throw raw.error;
|
||||
if (hasUnsupportedMarkup(node, characterNode))
|
||||
return new UnknownPlistScalar(
|
||||
"real",
|
||||
raw.value,
|
||||
"unsupported-markup",
|
||||
);
|
||||
const value = parseXmlPlistRealLiteral(raw.value);
|
||||
if (!value.ok)
|
||||
return new UnknownPlistScalar("real", raw.value, value.error);
|
||||
numbers.real(value.value);
|
||||
return value.value;
|
||||
}
|
||||
case "data": {
|
||||
const binary = atob(content(node, textNode).text.replace(/\s+/gu, ""));
|
||||
return Uint8Array.from(binary, (character) => character.charCodeAt(0));
|
||||
const value = source.dataValue(node);
|
||||
if (!value.ok) throw value.error;
|
||||
return value.value;
|
||||
}
|
||||
case "date": {
|
||||
const value = requireContent(node).nodeValue;
|
||||
return new Date(value === null ? 0 : value);
|
||||
const raw = source.scalarLiteral(node);
|
||||
if (!raw.ok) throw raw.error;
|
||||
if (hasUnsupportedMarkup(node, textNode))
|
||||
return new UnknownPlistScalar(
|
||||
"date",
|
||||
raw.value,
|
||||
"unsupported-markup",
|
||||
);
|
||||
const date = decodeXmlPlistDateLiteral(raw.value);
|
||||
return date.ok
|
||||
? date.value
|
||||
: new UnknownPlistScalar("date", raw.value, date.error);
|
||||
}
|
||||
case "null":
|
||||
return null;
|
||||
case "true":
|
||||
return true;
|
||||
case "false":
|
||||
return false;
|
||||
case "false": {
|
||||
const raw = source.scalarLiteral(node);
|
||||
if (!raw.ok) throw raw.error;
|
||||
return raw.value === ""
|
||||
? node.nodeName === "true"
|
||||
: new UnknownPlistScalar(
|
||||
node.nodeName,
|
||||
raw.value,
|
||||
hasUnsupportedMarkup(node, textNode)
|
||||
? "unsupported-markup"
|
||||
: "invalid-literal",
|
||||
);
|
||||
}
|
||||
default:
|
||||
throw new TypeError(`Invalid PLIST tag ${node.nodeName}`);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { parseXmlPropertyList } from "./propertyListKeys.js";
|
||||
|
||||
// Expected bytes were independently read from plutil's binary conversion.
|
||||
it.each([
|
||||
["AQID", "eb9408"],
|
||||
["&AQID", "6a6a40"],
|
||||
["AQ ID", "010d74"],
|
||||
["éAQID", "c44f40"],
|
||||
["====", "00"],
|
||||
["=AQI", "000408"],
|
||||
["AQI=Z", "0102"],
|
||||
["AQ=!==", "0100"],
|
||||
])(
|
||||
"retains CFPropertyList data bytes from source spelling %j",
|
||||
(literal, hex) => {
|
||||
const parsed = parseXmlPropertyList(
|
||||
`<plist><data>${literal}</data></plist>`,
|
||||
);
|
||||
expect(parsed.value).toEqual(Uint8Array.from(Buffer.from(hex, "hex")));
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["é", "😀", "\u00a0", "\u0085", "\u2028", "\u2029"])(
|
||||
"rejects non-ASCII data before source normalization can hide %j",
|
||||
(character) => {
|
||||
expect(() =>
|
||||
parseXmlPropertyList(`<plist><data>AQ${character}ID</data></plist>`),
|
||||
).toThrow(
|
||||
expect.objectContaining({
|
||||
_tag: "XmlPlistDataError",
|
||||
reason: "non-ascii",
|
||||
characterOffset: 2,
|
||||
}),
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
"A<!-- omitted -->QID",
|
||||
"<![CDATA[AQID]]>",
|
||||
"A<?rea ignored?>QID",
|
||||
"<string>AQID</string>",
|
||||
])(
|
||||
"rejects data markup instead of silently decoding a different source: %j",
|
||||
(literal) => {
|
||||
expect(() =>
|
||||
parseXmlPropertyList(`<plist><data>${literal}</data></plist>`),
|
||||
).toThrow(expect.objectContaining({ reason: "markup" }));
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["", ' note=">"', " note='>/'"])(
|
||||
"distinguishes explicit empty data from a self-closing element with attributes %j",
|
||||
(attributes) => {
|
||||
expect(
|
||||
parseXmlPropertyList(`<plist><data${attributes}></data></plist>`).value,
|
||||
).toEqual(new Uint8Array());
|
||||
expect(() =>
|
||||
parseXmlPropertyList(`<plist><data${attributes}/></plist>`),
|
||||
).toThrow(expect.objectContaining({ reason: "empty-element" }));
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["\n", "\r\n", "\r", "\u0085", "\u2028", "\u2029"])(
|
||||
"binds raw data after astral text, entities and line ending %j to the right node",
|
||||
(ending) => {
|
||||
const parsed = parseXmlPropertyList(
|
||||
`<plist><array><string note=">">π😀 ${ending}before</string>${ending}` +
|
||||
"<data>AQID</data><data>AQID</data></array></plist>",
|
||||
);
|
||||
expect(parsed.value).toEqual([
|
||||
expect.any(String),
|
||||
Uint8Array.from([235, 148, 8]),
|
||||
Uint8Array.from([1, 2, 3]),
|
||||
]);
|
||||
},
|
||||
);
|
||||
@@ -0,0 +1,65 @@
|
||||
import { err, ok, type Result } from "./result.js";
|
||||
|
||||
export class XmlPlistDataError extends TypeError {
|
||||
readonly _tag = "XmlPlistDataError";
|
||||
constructor(
|
||||
readonly reason:
|
||||
| "non-ascii"
|
||||
| "markup"
|
||||
| "empty-element"
|
||||
| "source-location",
|
||||
/** UTF-16 offset within the raw data literal, when known. */
|
||||
readonly characterOffset: number | null = null,
|
||||
) {
|
||||
super(`Cannot decode XML plist <data>: ${reason}`);
|
||||
this.name = "XmlPlistDataError";
|
||||
}
|
||||
}
|
||||
|
||||
const BASE64_DIGITS =
|
||||
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
|
||||
const C_WHITESPACE = new Set([" ", "\t", "\n", "\v", "\f", "\r"]);
|
||||
|
||||
/** Interpret each completed quartet once per pass, including CF's padding resets. */
|
||||
const visitDataQuartets = (
|
||||
text: string,
|
||||
visit: (value: number, byteCount: 1 | 2 | 3) => void,
|
||||
): Result<void, XmlPlistDataError> => {
|
||||
let quartet = 0;
|
||||
let digits = 0;
|
||||
let padding = 0;
|
||||
for (let index = 0; index < text.length; index += 1) {
|
||||
if (text.charCodeAt(index) >= 128)
|
||||
return err(new XmlPlistDataError("non-ascii", index));
|
||||
const character = text[index]!;
|
||||
if (character === "=") padding += 1;
|
||||
else if (!C_WHITESPACE.has(character)) padding = 0;
|
||||
const digit = character === "=" ? 0 : BASE64_DIGITS.indexOf(character);
|
||||
if (digit < 0) continue;
|
||||
quartet = ((quartet << 6) | digit) & 0xffffff;
|
||||
digits += 1;
|
||||
if (digits % 4 === 0)
|
||||
visit(quartet, padding >= 2 ? 1 : padding === 1 ? 2 : 3);
|
||||
}
|
||||
return ok(undefined);
|
||||
};
|
||||
|
||||
/** CFPropertyList reads raw ASCII, skips nonalphabet bytes and emits complete quartets. */
|
||||
export const decodeXmlPlistData = (
|
||||
text: string,
|
||||
): Result<Uint8Array, XmlPlistDataError> => {
|
||||
let length = 0;
|
||||
const counted = visitDataQuartets(text, (_value, byteCount) => {
|
||||
length += byteCount;
|
||||
});
|
||||
if (!counted.ok) return counted;
|
||||
// Count actual output before allocation, including padding in every quartet.
|
||||
const bytes = new Uint8Array(length);
|
||||
let offset = 0;
|
||||
const decoded = visitDataQuartets(text, (value, byteCount) => {
|
||||
bytes[offset++] = value >> 16;
|
||||
if (byteCount >= 2) bytes[offset++] = value >> 8;
|
||||
if (byteCount === 3) bytes[offset++] = value;
|
||||
});
|
||||
return decoded.ok ? ok(bytes) : decoded;
|
||||
};
|
||||
@@ -0,0 +1,139 @@
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
decodeXmlPlistDateLiteral,
|
||||
parseXmlPlistIntegerLiteral,
|
||||
parseXmlPlistRealLiteral,
|
||||
} from "./propertyListXmlScalar.js";
|
||||
|
||||
it.each([
|
||||
["42", 42n],
|
||||
[" + 42", 42n],
|
||||
["\n- 0x2a", -42n],
|
||||
["+0XAF", 175n],
|
||||
[`${"9".repeat(80)}`, BigInt("9".repeat(80))],
|
||||
])("parses complete XML plist integer text %j", (raw, expected) => {
|
||||
expect(parseXmlPlistIntegerLiteral(raw)).toBe(expected);
|
||||
});
|
||||
|
||||
it.each(["", "42 ", "42\n", "+", "0x", "0 x2a", "42.0", "42&foo;"])(
|
||||
"rejects incomplete or trailing XML plist integer text %j",
|
||||
(raw) => {
|
||||
expect(parseXmlPlistIntegerLiteral(raw)).toBeUndefined();
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
[" 42.5", 42.5],
|
||||
["\t-1.25e2", -125],
|
||||
[".5", 0.5],
|
||||
["1.", 1],
|
||||
["NaN", Number.NaN],
|
||||
["+inf", Infinity],
|
||||
["-infinity", -Infinity],
|
||||
])("parses complete XML plist real text %j", (raw, expected) => {
|
||||
expect(parseXmlPlistRealLiteral(raw)).toEqual({ ok: true, value: expected });
|
||||
});
|
||||
|
||||
it.each([
|
||||
"",
|
||||
"42.5 ",
|
||||
"42.5\n",
|
||||
"1e",
|
||||
"0x1p2",
|
||||
"1,25",
|
||||
"1suffix",
|
||||
"nan ",
|
||||
"infinite",
|
||||
"-NaN",
|
||||
])("rejects incomplete or unsupported XML plist real text %j", (raw) => {
|
||||
expect(parseXmlPlistRealLiteral(raw)).toEqual({
|
||||
ok: false,
|
||||
error: "invalid-literal",
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
["2001-02-03T04:05:06Z", Date.UTC(2001, 1, 3, 4, 5, 6)],
|
||||
["0000-02-29T00:00:00Z", -62162121600000],
|
||||
])("decodes XML plist date text %s", (raw, expected) => {
|
||||
const date = decodeXmlPlistDateLiteral(raw);
|
||||
expect(date).toEqual({ ok: true, value: new Date(expected) });
|
||||
});
|
||||
|
||||
it.each([
|
||||
"2001-2-03T04:05:06Z",
|
||||
"2001-02-03t04:05:06Z",
|
||||
"2001-02-03T04:05:06+00:00",
|
||||
" 2001-02-03T04:05:06Z",
|
||||
"2001-02-03T04:05:06Z ",
|
||||
"2001-02-03T04:05:06Z&",
|
||||
])("rejects malformed XML plist date text %j", (raw) => {
|
||||
expect(decodeXmlPlistDateLiteral(raw)).toEqual({
|
||||
ok: false,
|
||||
error: "invalid-literal",
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
"2001-02-29T04:05:06Z",
|
||||
"2000-13-01T00:00:00Z",
|
||||
"2000-01-01T24:00:00Z",
|
||||
"2000-01-01T00:00:60Z",
|
||||
"275761-01-01T00:00:00Z",
|
||||
])(
|
||||
"keeps unsupported XML plist date %s distinct from malformed text",
|
||||
(raw) => {
|
||||
const date = decodeXmlPlistDateLiteral(raw);
|
||||
expect(date).toEqual({ ok: false, error: "unrepresentable" });
|
||||
},
|
||||
);
|
||||
|
||||
it("distinguishes finite-spelling overflow from a known infinity", () => {
|
||||
expect(parseXmlPlistRealLiteral("1e999")).toEqual({
|
||||
ok: false,
|
||||
error: "unrepresentable",
|
||||
});
|
||||
expect(parseXmlPlistRealLiteral("-1e999")).toEqual({
|
||||
ok: false,
|
||||
error: "unrepresentable",
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
["\u0085", false, true],
|
||||
["\u00a0", false, true],
|
||||
["\u1680", false, false],
|
||||
["\u2003", true, true],
|
||||
["\u200b", true, true],
|
||||
["\u2028", false, false],
|
||||
["\u202f", false, false],
|
||||
["\u205f", true, false],
|
||||
["\ufeff", false, true],
|
||||
] as const)(
|
||||
"preserves distinct integer and real leading whitespace rules for %j",
|
||||
(prefix, integerAccepted, realAccepted) => {
|
||||
expect(parseXmlPlistIntegerLiteral(`${prefix}42`)).toBe(
|
||||
integerAccepted ? 42n : undefined,
|
||||
);
|
||||
expect(parseXmlPlistRealLiteral(`${prefix}42`)).toEqual(
|
||||
realAccepted
|
||||
? { ok: true, value: 42 }
|
||||
: { ok: false, error: "invalid-literal" },
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
["1-01-01T00:00:00Z", "0001-01-01T00:00:00.000Z"],
|
||||
["-0001-01-01T00:00:00Z", "-000001-01-01T00:00:00.000Z"],
|
||||
["10000-01-01T00:00:00Z", "+010000-01-01T00:00:00.000Z"],
|
||||
])(
|
||||
"interprets year fields as UTC without host string-date inference: %s",
|
||||
(literal, iso8601) => {
|
||||
expect(decodeXmlPlistDateLiteral(literal)).toEqual({
|
||||
ok: true,
|
||||
value: new Date(iso8601),
|
||||
});
|
||||
},
|
||||
);
|
||||
@@ -0,0 +1,139 @@
|
||||
import { err, ok, type Result } from "./result.js";
|
||||
|
||||
// Integer parsing examines UTF-8 bytes; real scanning examines CFString
|
||||
// characters. Native plutil controls distinguish their leading-whitespace sets;
|
||||
// notably, this host rejects U+202F despite its inclusion in older CF source.
|
||||
const INTEGER_WHITESPACE =
|
||||
"[\\u0000-\\u0020\\u007f\\u2000-\\u200b\\u205f\\u3000]*";
|
||||
const REAL_WHITESPACE =
|
||||
"[\\u0000-\\u0020\\u007f-\\u00a0\\u2000-\\u200b\\u3000\\ufeff]*";
|
||||
const INTEGER_LITERAL = new RegExp(
|
||||
`^${INTEGER_WHITESPACE}(?:([+-])${INTEGER_WHITESPACE})?(?:0[xX]([\\da-fA-F]+)|(\\d+))$`,
|
||||
"u",
|
||||
);
|
||||
const REAL_LEADING_WHITESPACE = new RegExp(`^${REAL_WHITESPACE}`, "u");
|
||||
|
||||
/** Parse CF-style decimal or hexadecimal integer text without losing precision. */
|
||||
export const parseXmlPlistIntegerLiteral = (
|
||||
raw: string,
|
||||
): bigint | undefined => {
|
||||
const match = INTEGER_LITERAL.exec(raw);
|
||||
if (match === null) return undefined;
|
||||
|
||||
const magnitude = BigInt(
|
||||
match[2] === undefined ? match[3]! : `0x${match[2]}`,
|
||||
);
|
||||
return match[1] === "-" ? -magnitude : magnitude;
|
||||
};
|
||||
|
||||
/**
|
||||
* Parse UTC fields without JavaScript string-date parsing or calendar rollover.
|
||||
* Syntax failures and unsupported date representations retain distinct reasons.
|
||||
*/
|
||||
export const decodeXmlPlistDateLiteral = (
|
||||
raw: string,
|
||||
): Result<Date, "invalid-literal" | "unrepresentable"> => {
|
||||
const match = /^(-?)(\d+)-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})Z$/u.exec(
|
||||
raw,
|
||||
);
|
||||
if (match === null) return err("invalid-literal");
|
||||
|
||||
const yearMagnitude = Number(match[2]);
|
||||
const year = match[1] === "-" ? -yearMagnitude : yearMagnitude;
|
||||
const month = Number(match[3]);
|
||||
const day = Number(match[4]);
|
||||
const hour = Number(match[5]);
|
||||
const minute = Number(match[6]);
|
||||
const second = Number(match[7]);
|
||||
|
||||
if (
|
||||
!Number.isSafeInteger(year) ||
|
||||
month < 1 ||
|
||||
month > 12 ||
|
||||
hour > 23 ||
|
||||
minute > 59 ||
|
||||
second > 59
|
||||
)
|
||||
return err("unrepresentable");
|
||||
|
||||
const leapYear = year % 4 === 0 && (year % 100 !== 0 || year % 400 === 0);
|
||||
const daysInMonth = [
|
||||
31,
|
||||
leapYear ? 29 : 28,
|
||||
31,
|
||||
30,
|
||||
31,
|
||||
30,
|
||||
31,
|
||||
31,
|
||||
30,
|
||||
31,
|
||||
30,
|
||||
31,
|
||||
][month - 1]!;
|
||||
if (day < 1 || day > daysInMonth) return err("unrepresentable");
|
||||
|
||||
const date = new Date(0);
|
||||
date.setUTCFullYear(year, month - 1, day);
|
||||
date.setUTCHours(hour, minute, second, 0);
|
||||
return Number.isNaN(date.getTime()) ? err("unrepresentable") : ok(date);
|
||||
};
|
||||
|
||||
/** Parse a complete decimal or CF-spelled non-finite real literal. */
|
||||
export const parseXmlPlistRealLiteral = (
|
||||
raw: string,
|
||||
): Result<number, "invalid-literal" | "unrepresentable"> => {
|
||||
// This content has passed XML well-formedness and character-node checks.
|
||||
// Expand references only outside CDATA; retain original line endings, which
|
||||
// the DOM's compatibility normalization would otherwise erase.
|
||||
const text = raw.replace(
|
||||
/<!\[CDATA\[([\s\S]*?)\]\]>|&#(?:x([\da-fA-F]+)|(\d+));|&(amp|lt|gt|apos|quot);/gu,
|
||||
(
|
||||
_token,
|
||||
cdata: string | undefined,
|
||||
hex: string | undefined,
|
||||
decimal: string | undefined,
|
||||
name: string | undefined,
|
||||
) => {
|
||||
if (cdata !== undefined) return cdata;
|
||||
if (hex !== undefined)
|
||||
return String.fromCodePoint(Number.parseInt(hex, 16));
|
||||
if (decimal !== undefined)
|
||||
return String.fromCodePoint(Number.parseInt(decimal, 10));
|
||||
return name === "amp"
|
||||
? "&"
|
||||
: name === "lt"
|
||||
? "<"
|
||||
: name === "gt"
|
||||
? ">"
|
||||
: name === "apos"
|
||||
? "'"
|
||||
: '"';
|
||||
},
|
||||
);
|
||||
// CF compares complete non-finite spellings before scanning a decimal.
|
||||
if (text.length <= 9) {
|
||||
switch (text.toLowerCase()) {
|
||||
case "nan":
|
||||
return ok(Number.NaN);
|
||||
case "+infinity":
|
||||
case "infinity":
|
||||
case "+inf":
|
||||
case "inf":
|
||||
return ok(Number.POSITIVE_INFINITY);
|
||||
case "-infinity":
|
||||
case "-inf":
|
||||
return ok(Number.NEGATIVE_INFINITY);
|
||||
}
|
||||
}
|
||||
|
||||
const numericText = text.replace(REAL_LEADING_WHITESPACE, "");
|
||||
if (
|
||||
!/^[+-]?(?:(?:\d+(?:\.\d*)?)|(?:\.\d+))(?:[eE][+-]?\d+)?$/u.test(
|
||||
numericText,
|
||||
)
|
||||
)
|
||||
return err("invalid-literal");
|
||||
const value = Number(numericText);
|
||||
return Number.isFinite(value) ? ok(value) : err("unrepresentable");
|
||||
};
|
||||
@@ -0,0 +1,191 @@
|
||||
import type { Document, Node } from "@xmldom/xmldom";
|
||||
import { MAX_PLIST_DEPTH } from "./apple/plistValue.js";
|
||||
import { err, ok, type Result } from "./result.js";
|
||||
import { parseXmlDocument } from "./xmlDocument.js";
|
||||
import {
|
||||
decodeXmlPlistData,
|
||||
XmlPlistDataError,
|
||||
} from "./propertyListXmlData.js";
|
||||
|
||||
// Preserve DOM compatibility normalization while retaining raw scalar spans.
|
||||
const LINE_ENDING = /\r[\n\u0085]|[\r\n\u0085\u2028\u2029]/gu;
|
||||
|
||||
interface ScalarSpan {
|
||||
readonly start: number;
|
||||
readonly end: number;
|
||||
readonly selfClosing: boolean;
|
||||
}
|
||||
|
||||
export class XmlPlistSourceError extends TypeError {
|
||||
readonly _tag = "XmlPlistSourceError";
|
||||
constructor() {
|
||||
super("Cannot locate XML plist scalar in its original source");
|
||||
}
|
||||
}
|
||||
|
||||
/** One source owner binds lexical scalar values to their validated DOM nodes. */
|
||||
export const readXmlPlistSource = (text: string) => {
|
||||
const normalized = text.replace(LINE_ENDING, "\n");
|
||||
const parsed = parseXmlDocument(normalized, {
|
||||
doctype: "allow",
|
||||
lineEndings: "preserve",
|
||||
});
|
||||
if (!parsed.ok) throw parsed.error;
|
||||
const document = parsed.value;
|
||||
const lineEndings = new RegExp(LINE_ENDING);
|
||||
let line = 1;
|
||||
let lineStart = 0;
|
||||
// The depth walk visits elements in source order, so location lookup is
|
||||
// linear without an index entry for every line in a potentially large value.
|
||||
const readSpan = (node: Node): Result<ScalarSpan, "source-location"> => {
|
||||
if (node.lineNumber === undefined || node.columnNumber === undefined)
|
||||
return err("source-location");
|
||||
while (line < node.lineNumber) {
|
||||
const ending = lineEndings.exec(text);
|
||||
if (ending === null) return err("source-location");
|
||||
lineStart = ending.index + ending[0].length;
|
||||
line += 1;
|
||||
}
|
||||
if (line !== node.lineNumber) return err("source-location");
|
||||
const start = lineStart + node.columnNumber - 1;
|
||||
const openingEnd = openingTagEnd(text, start);
|
||||
if (openingEnd < 0 || text[start] !== "<") return err("source-location");
|
||||
const selfClosing = text[openingEnd - 1] === "/";
|
||||
const end = selfClosing
|
||||
? openingEnd + 1
|
||||
: closingTagStart(text, openingEnd + 1);
|
||||
return end < 0
|
||||
? err("source-location")
|
||||
: ok({ start: openingEnd + 1, end, selfClosing });
|
||||
};
|
||||
const spans = collectXmlPlistScalarSpans(document, readSpan);
|
||||
return {
|
||||
document,
|
||||
scalarLiteral(node: Node): Result<string, XmlPlistSourceError> {
|
||||
const span = spans.get(node);
|
||||
return span?.ok === true
|
||||
? ok(text.slice(span.value.start, span.value.end))
|
||||
: err(new XmlPlistSourceError());
|
||||
},
|
||||
dataValue(node: Node): Result<Uint8Array, XmlPlistDataError> {
|
||||
const span = spans.get(node);
|
||||
if (node.nodeName !== "data" || span === undefined)
|
||||
return err(new XmlPlistDataError("source-location"));
|
||||
if (!span.ok) return err(new XmlPlistDataError(span.error));
|
||||
for (
|
||||
let child = node.firstChild;
|
||||
child !== null;
|
||||
child = child.nextSibling
|
||||
)
|
||||
if (child.nodeType !== child.TEXT_NODE)
|
||||
return err(new XmlPlistDataError("markup"));
|
||||
if (span.value.selfClosing)
|
||||
return err(new XmlPlistDataError("empty-element"));
|
||||
return decodeXmlPlistData(text.slice(span.value.start, span.value.end));
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
/** Scan validated tag syntax without treating a quoted > as its end. */
|
||||
const openingTagEnd = (source: string, start: number): number => {
|
||||
let quote: string | undefined;
|
||||
for (let cursor = start; cursor < source.length; cursor += 1) {
|
||||
const character = source[cursor];
|
||||
if (quote !== undefined) {
|
||||
if (character === quote) quote = undefined;
|
||||
} else if (character === '"' || character === "'") quote = character;
|
||||
else if (character === ">") return cursor;
|
||||
}
|
||||
return -1;
|
||||
};
|
||||
|
||||
/**
|
||||
* Locate an element's closing tag in validated XML. Comments, CDATA and PIs
|
||||
* can contain apparent tags; nested markup is retained as source, not decoded
|
||||
* or recursively interpreted as additional plist values.
|
||||
*/
|
||||
const closingTagStart = (source: string, contentStart: number): number => {
|
||||
let depth = 1;
|
||||
let cursor = contentStart;
|
||||
while (cursor < source.length) {
|
||||
const start = source.indexOf("<", cursor);
|
||||
if (start < 0) return -1;
|
||||
const delimiter = source.startsWith("<!--", start)
|
||||
? "-->"
|
||||
: source.startsWith("<![CDATA[", start)
|
||||
? "]]>"
|
||||
: source.startsWith("<?", start)
|
||||
? "?>"
|
||||
: undefined;
|
||||
if (delimiter !== undefined) {
|
||||
const end = source.indexOf(delimiter, start + 2);
|
||||
if (end < 0) return -1;
|
||||
cursor = end + delimiter.length;
|
||||
continue;
|
||||
}
|
||||
const end = openingTagEnd(source, start);
|
||||
if (end < 0) return -1;
|
||||
if (source[start + 1] === "/") {
|
||||
depth -= 1;
|
||||
if (depth === 0) return start;
|
||||
} else if (source[end - 1] !== "/") depth += 1;
|
||||
cursor = end + 1;
|
||||
}
|
||||
return -1;
|
||||
};
|
||||
|
||||
const PLIST_LEXICAL_SCALARS = new Set([
|
||||
"data",
|
||||
"integer",
|
||||
"real",
|
||||
"date",
|
||||
"true",
|
||||
"false",
|
||||
]);
|
||||
|
||||
/** Elements whose children the plist value decoder visits recursively. */
|
||||
const PLIST_CONTAINER_ELEMENTS = new Set(["plist", "dict", "array"]);
|
||||
|
||||
/**
|
||||
* Reject XML plists nested deeper than the projection supports before any
|
||||
* recursive value pass runs. Value decoding and prototype omission recurse once
|
||||
* per nesting level, so a deeply nested document exhausts the call stack
|
||||
* before the projection limit can report the resource bound. The failure
|
||||
* matches `projectPlistValue` so callers classify it as a limit failure.
|
||||
*
|
||||
* Depth counts values the way the projection does: the root value is level 0
|
||||
* and every container member is one level deeper. A dictionary key names a
|
||||
* member instead of nesting one, and a self-closing element is the value at
|
||||
* its own level. The outer plist envelope is unwrapped by the decoder;
|
||||
* nested plist elements decode as arrays and must count toward the bound.
|
||||
*/
|
||||
const collectXmlPlistScalarSpans = (
|
||||
document: Document,
|
||||
readSpan: (node: Node) => Result<ScalarSpan, "source-location">,
|
||||
): WeakMap<Node, Result<ScalarSpan, "source-location">> => {
|
||||
const spans = new WeakMap<Node, Result<ScalarSpan, "source-location">>();
|
||||
const root = document.documentElement;
|
||||
if (root === null) return spans;
|
||||
const pending: Array<{ node: Node; depth: number }> = [
|
||||
{ node: root, depth: -1 },
|
||||
];
|
||||
while (pending.length > 0) {
|
||||
const entry = pending.pop();
|
||||
if (entry === undefined) break;
|
||||
const { node, depth } = entry;
|
||||
// Schedule only the next sibling, keeping traversal storage proportional
|
||||
// to depth rather than the number of members in a wide container.
|
||||
if (node.nextSibling !== null)
|
||||
pending.push({ node: node.nextSibling, depth });
|
||||
if (node.nodeType !== node.ELEMENT_NODE) continue;
|
||||
if (node.nodeName !== "key" && depth > MAX_PLIST_DEPTH)
|
||||
throw new RangeError(
|
||||
`Plist exceeds ${String(MAX_PLIST_DEPTH)} nesting levels`,
|
||||
);
|
||||
if (PLIST_LEXICAL_SCALARS.has(node.nodeName))
|
||||
spans.set(node, readSpan(node));
|
||||
if (PLIST_CONTAINER_ELEMENTS.has(node.nodeName) && node.firstChild !== null)
|
||||
pending.push({ node: node.firstChild, depth: depth + 1 });
|
||||
}
|
||||
return spans;
|
||||
};
|
||||
@@ -1,7 +1,7 @@
|
||||
/** Generated from package.json and package-lock.json; do not edit. */
|
||||
export const PACKAGE_METADATA = {
|
||||
name: "rea-agents",
|
||||
version: "6.5.0", // x-release-please-version
|
||||
version: "6.7.0", // x-release-please-version
|
||||
serverSdkVersion: "2.3.1",
|
||||
clientSdkVersion: "2.3.1",
|
||||
coreSdkVersion: "2.3.1",
|
||||
|
||||
+201
-101
@@ -143,6 +143,14 @@ export interface CapturedOutput {
|
||||
readonly complete: boolean;
|
||||
}
|
||||
|
||||
type CapturedOutputRead =
|
||||
| { readonly ok: true; readonly value: CapturedOutput }
|
||||
| {
|
||||
readonly ok: false;
|
||||
readonly error: unknown;
|
||||
readonly partial?: CapturedOutput;
|
||||
};
|
||||
|
||||
/** One completed LLDB observation, before projection into the result contract. */
|
||||
export interface NativeCallTrace {
|
||||
readonly debugger: ResolvedTool & { readonly version: string | null };
|
||||
@@ -231,7 +239,7 @@ export class LldbCallTracer implements NativeCallTracer {
|
||||
try {
|
||||
result = await this.#run(lldb.value, owner, request, signal);
|
||||
} catch (cause: unknown) {
|
||||
const partialObservation = await partialNativeObservation({
|
||||
const partial = await partialNativeObservation({
|
||||
request,
|
||||
files: owner.files,
|
||||
pidPath: paths.pid,
|
||||
@@ -247,21 +255,20 @@ export class LldbCallTracer implements NativeCallTracer {
|
||||
owner.files.hasPending
|
||||
)
|
||||
this.#pendingCleanup = owner;
|
||||
result = err(
|
||||
new ProviderAdapterError(PROVIDER, OPERATION, {
|
||||
cause,
|
||||
diagnostics: { reason: errorReason(cause) },
|
||||
partialObservation,
|
||||
...(this.#pendingCleanup === owner
|
||||
? {
|
||||
cleanup: {
|
||||
reason: "LLDB trace failed while owned resources remain",
|
||||
resources: lldbCleanupResources(owner),
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
}),
|
||||
);
|
||||
const failure = new ProviderAdapterError(PROVIDER, OPERATION, {
|
||||
cause,
|
||||
diagnostics: { reason: errorReason(cause) },
|
||||
partialObservation: partial.observation,
|
||||
...(this.#pendingCleanup === owner
|
||||
? {
|
||||
cleanup: {
|
||||
reason: "LLDB trace failed while owned resources remain",
|
||||
resources: lldbCleanupResources(owner),
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
});
|
||||
result = err(retainPartialReadFailures(failure, partial.readFailures));
|
||||
}
|
||||
if (result.ok && signal?.aborted)
|
||||
result = err(
|
||||
@@ -497,10 +504,11 @@ export class LldbCallTracer implements NativeCallTracer {
|
||||
retained.processStopped =
|
||||
exited.cleanupOwner === undefined && exited.cleanupFailure === undefined;
|
||||
retained.targetStopped = !exited.targetLaunched;
|
||||
const output =
|
||||
const readback =
|
||||
exited.kind === "exited"
|
||||
? await readTracerOutput(paths.result, retained.files)
|
||||
: undefined;
|
||||
const output = readback?.ok ? readback.value : undefined;
|
||||
// An explicit bridge observation can prove no process was created or that
|
||||
// the created process exited. A missing PID alone remains ambiguous.
|
||||
const bridgeObservedExit =
|
||||
@@ -548,7 +556,7 @@ export class LldbCallTracer implements NativeCallTracer {
|
||||
: "tracer-failure";
|
||||
const getPartialObservation = (
|
||||
reason: NativeCallPartialObservation["coverage"]["reason"] = partialReason,
|
||||
): Promise<NativeCallPartialObservation> =>
|
||||
): Promise<PartialNativeObservationRead> =>
|
||||
partialNativeObservation({
|
||||
request,
|
||||
files: retained.files,
|
||||
@@ -559,84 +567,115 @@ export class LldbCallTracer implements NativeCallTracer {
|
||||
version: output?.status === "traced" ? output.version : null,
|
||||
reason,
|
||||
});
|
||||
if (exited.kind === "cancelled")
|
||||
const failureWithPartial = async <Failure extends AnalysisError>(
|
||||
createFailure: (observation: NativeCallPartialObservation) => Failure,
|
||||
reason?: NativeCallPartialObservation["coverage"]["reason"],
|
||||
): Promise<Result<NativeCallTrace, AnalysisError>> => {
|
||||
const partial = await getPartialObservation(reason);
|
||||
return err(
|
||||
new AnalysisCancelledError(OPERATION, {
|
||||
cause: signal?.reason,
|
||||
...(lifecycleCleanup === undefined
|
||||
? {}
|
||||
: { cleanup: lifecycleCleanup }),
|
||||
partialObservation: await getPartialObservation(),
|
||||
}),
|
||||
retainPartialReadFailures(
|
||||
createFailure(partial.observation),
|
||||
partial.readFailures,
|
||||
),
|
||||
);
|
||||
if (exited.kind === "timeout")
|
||||
return err(
|
||||
new AnalysisTimeoutError(
|
||||
OPERATION,
|
||||
input.duration_ms + DEADLINE_GRACE_MS,
|
||||
{
|
||||
};
|
||||
if (exited.kind === "cancelled")
|
||||
return failureWithPartial(
|
||||
(partialObservation) =>
|
||||
new AnalysisCancelledError(OPERATION, {
|
||||
cause: signal?.reason,
|
||||
...(lifecycleCleanup === undefined
|
||||
? {}
|
||||
: { cleanup: lifecycleCleanup }),
|
||||
partialObservation: await getPartialObservation(),
|
||||
},
|
||||
),
|
||||
partialObservation,
|
||||
}),
|
||||
);
|
||||
if (exited.kind === "timeout")
|
||||
return failureWithPartial(
|
||||
(partialObservation) =>
|
||||
new AnalysisTimeoutError(
|
||||
OPERATION,
|
||||
input.duration_ms + DEADLINE_GRACE_MS,
|
||||
{
|
||||
...(lifecycleCleanup === undefined
|
||||
? {}
|
||||
: { cleanup: lifecycleCleanup }),
|
||||
partialObservation,
|
||||
},
|
||||
),
|
||||
);
|
||||
if (output !== undefined && output.status !== "traced")
|
||||
return err(
|
||||
tracerFailure(output, lifecycleCleanup, await getPartialObservation()),
|
||||
return failureWithPartial((partialObservation) =>
|
||||
tracerFailure(output, lifecycleCleanup, partialObservation),
|
||||
);
|
||||
if (exited.cleanupFailure !== undefined)
|
||||
return err(
|
||||
new ProviderCleanupError(
|
||||
PROVIDER,
|
||||
lldbCleanupResources(retained),
|
||||
{
|
||||
reason: "LLDB process-group cleanup could not be verified",
|
||||
cleanup_failure: exited.cleanupFailure,
|
||||
...(lifecycleCleanup === undefined
|
||||
? {}
|
||||
: { target_cleanup: lifecycleCleanup }),
|
||||
},
|
||||
{
|
||||
operation: OPERATION,
|
||||
partialObservation: await getPartialObservation(),
|
||||
},
|
||||
),
|
||||
const processCleanupFailure = exited.cleanupFailure;
|
||||
if (processCleanupFailure !== undefined)
|
||||
return failureWithPartial(
|
||||
(partialObservation) =>
|
||||
new ProviderCleanupError(
|
||||
PROVIDER,
|
||||
lldbCleanupResources(retained),
|
||||
{
|
||||
reason: "LLDB process-group cleanup could not be verified",
|
||||
cleanup_failure: processCleanupFailure,
|
||||
...(lifecycleCleanup === undefined
|
||||
? {}
|
||||
: { target_cleanup: lifecycleCleanup }),
|
||||
},
|
||||
{
|
||||
operation: OPERATION,
|
||||
partialObservation,
|
||||
},
|
||||
),
|
||||
);
|
||||
if (!terminated)
|
||||
return err(
|
||||
new ProviderCleanupError(
|
||||
PROVIDER,
|
||||
lldbCleanupResources(retained),
|
||||
{
|
||||
reason: "LLDB target process termination could not be verified",
|
||||
...(lifecycleCleanup === undefined
|
||||
? {}
|
||||
: { target_cleanup: lifecycleCleanup }),
|
||||
},
|
||||
{
|
||||
operation: OPERATION,
|
||||
partialObservation: await getPartialObservation(),
|
||||
},
|
||||
),
|
||||
return failureWithPartial(
|
||||
(partialObservation) =>
|
||||
new ProviderCleanupError(
|
||||
PROVIDER,
|
||||
lldbCleanupResources(retained),
|
||||
{
|
||||
reason: "LLDB target process termination could not be verified",
|
||||
...(lifecycleCleanup === undefined
|
||||
? {}
|
||||
: { target_cleanup: lifecycleCleanup }),
|
||||
},
|
||||
{
|
||||
operation: OPERATION,
|
||||
partialObservation,
|
||||
},
|
||||
),
|
||||
);
|
||||
if (output === undefined)
|
||||
return err(
|
||||
new ProviderAdapterError(PROVIDER, OPERATION, {
|
||||
diagnostics: {
|
||||
reason: "LLDB exited without a tracer result",
|
||||
exit_code: exited.exitCode,
|
||||
lldb_output: exited.output,
|
||||
return failureWithPartial(
|
||||
(partialObservation) =>
|
||||
new ProviderAdapterError(PROVIDER, OPERATION, {
|
||||
...(readback !== undefined && !readback.ok
|
||||
? { cause: readback.error }
|
||||
: {}),
|
||||
diagnostics: {
|
||||
reason: "LLDB exited without a usable tracer result",
|
||||
...(readback !== undefined && !readback.ok
|
||||
? {
|
||||
result_failure:
|
||||
readback.error instanceof z.ZodError
|
||||
? `Tracer result failed schema validation (${readback.error.issues.length} issues)`
|
||||
: readback.error instanceof Error
|
||||
? readback.error.message
|
||||
: String(readback.error),
|
||||
}
|
||||
: {}),
|
||||
exit_code: exited.exitCode,
|
||||
lldb_output: exited.output,
|
||||
...(lifecycleCleanup === undefined
|
||||
? {}
|
||||
: { target_cleanup: lifecycleCleanup }),
|
||||
},
|
||||
partialObservation,
|
||||
...(lifecycleCleanup === undefined
|
||||
? {}
|
||||
: { target_cleanup: lifecycleCleanup }),
|
||||
},
|
||||
partialObservation: await getPartialObservation(),
|
||||
...(lifecycleCleanup === undefined
|
||||
? {}
|
||||
: { cleanup: lifecycleCleanup }),
|
||||
}),
|
||||
: { cleanup: lifecycleCleanup }),
|
||||
}),
|
||||
);
|
||||
const stdout = await capturedOutput(
|
||||
paths.stdoutCapture,
|
||||
@@ -672,8 +711,8 @@ export class LldbCallTracer implements NativeCallTracer {
|
||||
{
|
||||
debugger: { ...lldb, version: output.version },
|
||||
run: output,
|
||||
stdout: stdout.ok ? stdout.value : null,
|
||||
stderr: stderr.ok ? stderr.value : null,
|
||||
stdout: stdout.ok ? stdout.value : (stdout.partial ?? null),
|
||||
stderr: stderr.ok ? stderr.value : (stderr.partial ?? null),
|
||||
terminated,
|
||||
},
|
||||
"capture-failure",
|
||||
@@ -770,6 +809,31 @@ const tracerFailure = (
|
||||
);
|
||||
};
|
||||
|
||||
interface PartialNativeObservationRead {
|
||||
readonly observation: NativeCallPartialObservation;
|
||||
readonly readFailures: readonly unknown[];
|
||||
}
|
||||
|
||||
const retainPartialReadFailures = <Failure extends AnalysisError>(
|
||||
failure: Failure,
|
||||
readFailures: readonly unknown[],
|
||||
): Failure => {
|
||||
if (readFailures.length === 0) return failure;
|
||||
const readbackFailure = new ProviderAdapterError(PROVIDER, OPERATION, {
|
||||
cause:
|
||||
readFailures.length === 1
|
||||
? readFailures[0]
|
||||
: new AggregateError(readFailures, "LLDB partial readback failed"),
|
||||
diagnostics: {
|
||||
reason: "LLDB could not read all available partial observation data",
|
||||
read_failures: readFailures.map(errorReason),
|
||||
},
|
||||
});
|
||||
return failure.retainSecondaryFailure(readbackFailure, {
|
||||
partial_readback_failure: projectAnalysisError(readbackFailure),
|
||||
});
|
||||
};
|
||||
|
||||
const partialNativeObservation = async (options: {
|
||||
readonly request: Parameters<NativeCallTracer["trace"]>[0];
|
||||
readonly files: LldbReadbackFiles;
|
||||
@@ -779,14 +843,14 @@ const partialNativeObservation = async (options: {
|
||||
readonly observationPath: string;
|
||||
readonly version: string | null;
|
||||
readonly reason: NativeCallPartialObservation["coverage"]["reason"];
|
||||
}): Promise<NativeCallPartialObservation> => {
|
||||
}): Promise<PartialNativeObservationRead> => {
|
||||
const [journal, pid, stdout, stderr] = await Promise.all([
|
||||
readLldbObservationJournal(options.observationPath, options.files),
|
||||
options.files.readTargetPid(options.pidPath),
|
||||
readPartialCapture(options.stdoutPath, options.files),
|
||||
readPartialCapture(options.stderrPath, options.files),
|
||||
]);
|
||||
return projectLldbPartialObservation({
|
||||
const observation = projectLldbPartialObservation({
|
||||
target: {
|
||||
path: options.request.executable,
|
||||
sha256: options.request.expectedSha256,
|
||||
@@ -821,6 +885,14 @@ const partialNativeObservation = async (options: {
|
||||
],
|
||||
reason: options.reason,
|
||||
});
|
||||
const readFailures = [journal, stdout, stderr].flatMap((read) =>
|
||||
!read.ok
|
||||
? [read.error]
|
||||
: read.value.status === "read-failure"
|
||||
? [read.value.readFailure]
|
||||
: [],
|
||||
);
|
||||
return { observation, readFailures };
|
||||
};
|
||||
|
||||
const partialObservationFromTrace = (
|
||||
@@ -855,37 +927,65 @@ const lldbRuntimePaths = (directory: string) => ({
|
||||
const readTracerOutput = async (
|
||||
path: string,
|
||||
files: LldbReadbackFiles,
|
||||
): Promise<z.infer<typeof tracerOutputSchema> | undefined> => {
|
||||
const text = await files.read(path, (file) => file.readFile("utf8"));
|
||||
if (!text.ok) return undefined;
|
||||
const parsed = safeParseJson(text.value);
|
||||
if (!parsed.ok) return undefined;
|
||||
): Promise<Result<z.infer<typeof tracerOutputSchema>, unknown>> => {
|
||||
const bytes = await files.read(path, (file) => file.readFile());
|
||||
if (!bytes.ok) return bytes;
|
||||
let text: string;
|
||||
try {
|
||||
text = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(
|
||||
bytes.value,
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
return err(cause);
|
||||
}
|
||||
const parsed = safeParseJson(text);
|
||||
if (!parsed.ok) return err(parsed.cause);
|
||||
const output = tracerOutputSchema.safeParse(parsed.value);
|
||||
return output.success ? output.data : undefined;
|
||||
return output.success ? ok(output.data) : err(output.error);
|
||||
};
|
||||
|
||||
const capturedOutput = (
|
||||
const capturedOutput = async (
|
||||
path: string,
|
||||
observedBytes: number,
|
||||
truncated: boolean,
|
||||
complete: boolean,
|
||||
files: LldbReadbackFiles,
|
||||
): Promise<Result<CapturedOutput, unknown>> =>
|
||||
files.read(path, async (handle) => {
|
||||
): Promise<CapturedOutputRead> => {
|
||||
const readback = await files.read(path, async (handle) => {
|
||||
const { size } = await handle.stat();
|
||||
const { buffer, bytesRead } = await readBoundedPrefix(
|
||||
const read = await readBoundedPrefix(
|
||||
Math.min(size, MAX_OUTPUT_BYTES),
|
||||
async (target, offset, length, position) =>
|
||||
(await handle.read(target, offset, length, position)).bytesRead,
|
||||
);
|
||||
const { buffer, bytesRead } = read;
|
||||
if (read.status === "read-failure" && bytesRead === 0) throw read.cause;
|
||||
return {
|
||||
text: buffer.subarray(0, bytesRead).toString("utf8"),
|
||||
bytes: observedBytes,
|
||||
truncated: truncated || size > MAX_OUTPUT_BYTES,
|
||||
complete:
|
||||
complete && bytesRead >= Math.min(observedBytes, MAX_OUTPUT_BYTES),
|
||||
status: read.status,
|
||||
capture: {
|
||||
text: buffer.subarray(0, bytesRead).toString("utf8"),
|
||||
bytes: observedBytes,
|
||||
truncated:
|
||||
truncated ||
|
||||
size > MAX_OUTPUT_BYTES ||
|
||||
read.status === "read-failure",
|
||||
complete:
|
||||
read.status !== "read-failure" &&
|
||||
complete &&
|
||||
bytesRead >= Math.min(observedBytes, MAX_OUTPUT_BYTES),
|
||||
},
|
||||
...(read.status === "read-failure" ? { readFailure: read.cause } : {}),
|
||||
};
|
||||
});
|
||||
if (!readback.ok) return readback;
|
||||
if (readback.value.status === "read-failure")
|
||||
return {
|
||||
ok: false,
|
||||
error: readback.value.readFailure,
|
||||
partial: readback.value.capture,
|
||||
};
|
||||
return ok(readback.value.capture);
|
||||
};
|
||||
|
||||
interface RetainedLldbResources {
|
||||
readonly root: PrivateRuntimeRoot;
|
||||
|
||||
@@ -7,6 +7,9 @@ import {
|
||||
} from "../artifacts/ArtifactFileOperation.js";
|
||||
import type { AnalysisCleanupLease } from "../artifacts/AnalysisCleanupLease.js";
|
||||
|
||||
/** PID publication is decimal; allow the largest safe PID and a CRLF ending. */
|
||||
const TARGET_PID_FILE_MAX_BYTES = String(Number.MAX_SAFE_INTEGER).length + 2;
|
||||
|
||||
/** One trace's readback descriptors, separate from the facts already read. */
|
||||
export class LldbReadbackFiles {
|
||||
readonly #resources = new ArtifactResourceScope();
|
||||
@@ -76,10 +79,28 @@ export class LldbReadbackFiles {
|
||||
async readTargetPid(path: string): Promise<number | undefined> {
|
||||
const known = this.#targetPids.get(path);
|
||||
if (known !== undefined) return known;
|
||||
const contents = await this.read(path, async (file) =>
|
||||
(await file.readFile("utf8")).trim(),
|
||||
);
|
||||
if (!contents.ok || !/^[1-9]\d*$/u.test(contents.value)) return undefined;
|
||||
const contents = await this.read(path, async (file) => {
|
||||
const bytes = Buffer.alloc(TARGET_PID_FILE_MAX_BYTES + 1);
|
||||
let length = 0;
|
||||
while (length < bytes.length) {
|
||||
const { bytesRead } = await file.read(
|
||||
bytes,
|
||||
length,
|
||||
bytes.length - length,
|
||||
length,
|
||||
);
|
||||
if (bytesRead === 0) break;
|
||||
length += bytesRead;
|
||||
}
|
||||
if (length > TARGET_PID_FILE_MAX_BYTES) return undefined;
|
||||
return bytes.subarray(0, length).toString("utf8").trim();
|
||||
});
|
||||
if (
|
||||
!contents.ok ||
|
||||
contents.value === undefined ||
|
||||
!/^[1-9]\d*$/u.test(contents.value)
|
||||
)
|
||||
return undefined;
|
||||
const pid = Number(contents.value);
|
||||
if (!Number.isSafeInteger(pid)) return undefined;
|
||||
this.#targetPids.set(path, pid);
|
||||
|
||||
@@ -15,13 +15,47 @@ export interface ParsedLldbJournal {
|
||||
readonly limitations: string[];
|
||||
}
|
||||
|
||||
export interface PartialCaptureRead {
|
||||
interface LldbObservationJournalValue {
|
||||
readonly parsed: ParsedLldbJournal;
|
||||
readonly limitations: readonly string[];
|
||||
}
|
||||
|
||||
export type LldbObservationJournalRead =
|
||||
| (LldbObservationJournalValue & {
|
||||
readonly status: "complete" | "eof";
|
||||
})
|
||||
| (LldbObservationJournalValue & {
|
||||
readonly status: "read-failure";
|
||||
readonly readFailure: unknown;
|
||||
});
|
||||
|
||||
interface PartialCaptureValue {
|
||||
readonly capture: NonNullable<
|
||||
NativeCallPartialObservation["process"]["stdout"]
|
||||
>;
|
||||
readonly limitations: readonly string[];
|
||||
}
|
||||
|
||||
export type PartialCaptureRead =
|
||||
| (PartialCaptureValue & { readonly status: "complete" | "eof" })
|
||||
| (PartialCaptureValue & {
|
||||
readonly status: "read-failure";
|
||||
readonly readFailure: unknown;
|
||||
});
|
||||
|
||||
export type BoundedPrefixRead =
|
||||
| {
|
||||
readonly status: "complete" | "eof";
|
||||
readonly buffer: Buffer;
|
||||
readonly bytesRead: number;
|
||||
}
|
||||
| {
|
||||
readonly status: "read-failure";
|
||||
readonly buffer: Buffer;
|
||||
readonly bytesRead: number;
|
||||
readonly cause: unknown;
|
||||
};
|
||||
|
||||
/** Join safely retained journal and process evidence into its failure contract. */
|
||||
export const projectLldbPartialObservation = (input: {
|
||||
readonly target: NativeCallPartialObservation["target"];
|
||||
@@ -66,20 +100,19 @@ export const projectLldbPartialObservation = (input: {
|
||||
export const parseLldbObservationJournal = (
|
||||
bytes: Uint8Array,
|
||||
): ParsedLldbJournal => {
|
||||
const completeText = Buffer.from(bytes).toString("utf8");
|
||||
const newline = completeText.lastIndexOf("\n");
|
||||
const hasTornTail = newline !== completeText.length - 1;
|
||||
const hasTornTail = bytes.length > 0 && bytes[bytes.length - 1] !== 0x0a;
|
||||
const events: ParsedLldbJournal["events"] = [];
|
||||
const otherStops: string[] = [];
|
||||
const limitations: string[] = [];
|
||||
const decoder = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
|
||||
let malformedCount = 0;
|
||||
let firstMalformedRow: number | undefined;
|
||||
let start = 0;
|
||||
let rowNumber = 0;
|
||||
while (start <= newline && newline >= 0) {
|
||||
const end = completeText.indexOf("\n", start);
|
||||
if (end < 0 || end > newline) break;
|
||||
const line = completeText.slice(start, end);
|
||||
while (start < bytes.length) {
|
||||
const end = bytes.indexOf(0x0a, start);
|
||||
if (end < 0) break;
|
||||
const line = bytes.subarray(start, end);
|
||||
start = end + 1;
|
||||
if (line.length === 0) continue;
|
||||
rowNumber += 1;
|
||||
@@ -89,7 +122,7 @@ export const parseLldbObservationJournal = (
|
||||
};
|
||||
let row: unknown;
|
||||
try {
|
||||
row = JSON.parse(line);
|
||||
row = JSON.parse(decoder.decode(line));
|
||||
} catch {
|
||||
markMalformed();
|
||||
continue;
|
||||
@@ -137,22 +170,31 @@ export const readBoundedPrefix = async (
|
||||
length: number,
|
||||
position: number,
|
||||
) => Promise<number>,
|
||||
): Promise<{ readonly buffer: Buffer; readonly bytesRead: number }> => {
|
||||
): Promise<BoundedPrefixRead> => {
|
||||
const buffer = Buffer.alloc(requestedBytes);
|
||||
let bytesRead = 0;
|
||||
while (bytesRead < requestedBytes) {
|
||||
const count = await readChunk(
|
||||
buffer,
|
||||
bytesRead,
|
||||
requestedBytes - bytesRead,
|
||||
bytesRead,
|
||||
);
|
||||
let count: number;
|
||||
try {
|
||||
count = await readChunk(
|
||||
buffer,
|
||||
bytesRead,
|
||||
requestedBytes - bytesRead,
|
||||
bytesRead,
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
return { status: "read-failure", buffer, bytesRead, cause };
|
||||
}
|
||||
if (count === 0) break;
|
||||
if (count > requestedBytes - bytesRead)
|
||||
throw new Error("The bounded read returned more bytes than requested");
|
||||
bytesRead += count;
|
||||
}
|
||||
return { buffer, bytesRead };
|
||||
return {
|
||||
status: bytesRead < requestedBytes ? "eof" : "complete",
|
||||
buffer,
|
||||
bytesRead,
|
||||
};
|
||||
};
|
||||
|
||||
const readPrefix = (handle: FileHandle, requestedBytes: number) =>
|
||||
@@ -166,17 +208,14 @@ const readPrefix = (handle: FileHandle, requestedBytes: number) =>
|
||||
export const readLldbObservationJournal = (
|
||||
path: string,
|
||||
files: LldbReadbackFiles,
|
||||
): Promise<
|
||||
Result<
|
||||
{ readonly parsed: ParsedLldbJournal; readonly limitations: string[] },
|
||||
unknown
|
||||
>
|
||||
> =>
|
||||
): Promise<Result<LldbObservationJournalRead, unknown>> =>
|
||||
files.read(path, async (handle) => {
|
||||
const { size } = await handle.stat();
|
||||
const length = Math.min(size, LLDB_OBSERVATION_JOURNAL_MAX_BYTES);
|
||||
const { buffer, bytesRead } = await readPrefix(handle, length);
|
||||
return {
|
||||
const read = await readPrefix(handle, length);
|
||||
const { buffer, bytesRead } = read;
|
||||
if (read.status === "read-failure" && bytesRead === 0) throw read.cause;
|
||||
const value: LldbObservationJournalValue = {
|
||||
parsed: parseLldbObservationJournal(buffer.subarray(0, bytesRead)),
|
||||
limitations: [
|
||||
...(size > LLDB_OBSERVATION_JOURNAL_MAX_BYTES
|
||||
@@ -184,13 +223,21 @@ export const readLldbObservationJournal = (
|
||||
"The observation journal exceeded 8 MiB; only its bounded prefix was read.",
|
||||
]
|
||||
: []),
|
||||
...(bytesRead < length
|
||||
...(read.status === "eof"
|
||||
? [
|
||||
"The observation journal became shorter while being read; the available prefix was parsed.",
|
||||
]
|
||||
: []),
|
||||
...(read.status === "read-failure"
|
||||
? [
|
||||
`The observation journal read failed after ${String(bytesRead)} bytes: ${errorReason(read.cause)}`,
|
||||
]
|
||||
: []),
|
||||
],
|
||||
};
|
||||
return read.status === "read-failure"
|
||||
? { ...value, status: read.status, readFailure: read.cause }
|
||||
: { ...value, status: read.status };
|
||||
});
|
||||
|
||||
/** Retain an available output prefix without claiming an unreported total. */
|
||||
@@ -201,19 +248,33 @@ export const readPartialCapture = (
|
||||
files.read(path, async (handle) => {
|
||||
const { size } = await handle.stat();
|
||||
const length = Math.min(size, MAX_CAPTURE_BYTES);
|
||||
const { buffer, bytesRead } = await readPrefix(handle, length);
|
||||
return {
|
||||
const read = await readPrefix(handle, length);
|
||||
const { buffer, bytesRead } = read;
|
||||
if (read.status === "read-failure" && bytesRead === 0) throw read.cause;
|
||||
const value: PartialCaptureValue = {
|
||||
capture: {
|
||||
text: buffer.subarray(0, bytesRead).toString("utf8"),
|
||||
bytes: Math.max(size, bytesRead),
|
||||
truncated: true,
|
||||
complete: false,
|
||||
},
|
||||
limitations:
|
||||
bytesRead < length
|
||||
limitations: [
|
||||
...(read.status === "eof"
|
||||
? [
|
||||
"A target output capture became shorter while being read; its retained prefix may be incomplete.",
|
||||
]
|
||||
: [],
|
||||
: []),
|
||||
...(read.status === "read-failure"
|
||||
? [
|
||||
`A target output capture read failed after ${String(bytesRead)} bytes: ${errorReason(read.cause)}`,
|
||||
]
|
||||
: []),
|
||||
],
|
||||
};
|
||||
return read.status === "read-failure"
|
||||
? { ...value, status: read.status, readFailure: read.cause }
|
||||
: { ...value, status: read.status };
|
||||
});
|
||||
|
||||
const errorReason = (cause: unknown): string =>
|
||||
cause instanceof Error ? cause.message : String(cause);
|
||||
@@ -64,9 +64,9 @@ import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
|
||||
import {
|
||||
projectPlistValue,
|
||||
nonFiniteRealLimitation,
|
||||
unknownRealLimitation,
|
||||
negativeZeroRealLimitation,
|
||||
unknownDateLimitation,
|
||||
unknownScalarLimitation,
|
||||
} from "../domain/apple/plistValue.js";
|
||||
import { createPlistNumberProjection } from "../domain/apple/plistNumbers.js";
|
||||
import type { NativeUiObservationResult } from "../domain/native/nativeUiObservation.js";
|
||||
@@ -118,6 +118,7 @@ import {
|
||||
createNativeUiHelperRuntime,
|
||||
type NativeUiHelperRuntime,
|
||||
} from "./NativeUiHelperRuntime.js";
|
||||
import { NativeUiInterruptionControl } from "./NativeUiInterruptionControl.js";
|
||||
|
||||
type SignatureRuntimeRootFactory = () => Promise<
|
||||
Pick<PrivateRuntimeRoot, "path" | "close">
|
||||
@@ -362,57 +363,57 @@ class NativeMacOSClient implements AnalysisClient {
|
||||
this.#nativeUiCleanupFailure(earlierCleanupFailures, operation),
|
||||
);
|
||||
const runtime = createNativeUiHelperRuntime(this.environment);
|
||||
const observationSignal =
|
||||
signal === undefined
|
||||
? runtime.signal
|
||||
: AbortSignal.any([signal, runtime.signal]);
|
||||
this.#pendingNativeUiRuntimes.add(runtime);
|
||||
let result: Awaited<ReturnType<typeof observeNativeUi>>;
|
||||
const interruption = new NativeUiInterruptionControl(
|
||||
signal,
|
||||
runtime.signal,
|
||||
);
|
||||
try {
|
||||
result = await observeNativeUi(this.target, operation, parameters, {
|
||||
environment: this.environment,
|
||||
signal: observationSignal,
|
||||
runtime,
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
result = err(
|
||||
cause instanceof AnalysisError
|
||||
? cause
|
||||
: new ProviderAdapterError(IDENTITY.id, operation, { cause }),
|
||||
);
|
||||
this.#pendingNativeUiRuntimes.add(runtime);
|
||||
let result: Awaited<ReturnType<typeof observeNativeUi>>;
|
||||
try {
|
||||
result = await observeNativeUi(this.target, operation, parameters, {
|
||||
environment: this.environment,
|
||||
runtime,
|
||||
interruption,
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
result = err(
|
||||
cause instanceof AnalysisError
|
||||
? cause
|
||||
: new ProviderAdapterError(IDENTITY.id, operation, { cause }),
|
||||
);
|
||||
}
|
||||
try {
|
||||
await runtime.close();
|
||||
this.#pendingNativeUiRuntimes.delete(runtime);
|
||||
} catch (cause: unknown) {
|
||||
result = finishNativeUiObservation(result, operation, interruption);
|
||||
const cleanup = this.#nativeUiCleanupFailure(
|
||||
[{ runtime, cause }],
|
||||
operation,
|
||||
result.ok ? result.value : undefined,
|
||||
);
|
||||
return err(
|
||||
result.ok
|
||||
? cleanup
|
||||
: analysisErrorWithCleanupFailure(
|
||||
result.error,
|
||||
cleanup,
|
||||
operation,
|
||||
).bindCleanupDependencies([result.error, cleanup]),
|
||||
);
|
||||
}
|
||||
result = finishNativeUiObservation(result, operation, interruption);
|
||||
return result.ok
|
||||
? ok(
|
||||
createAnalysisExecution(result.value, IDENTITY, {
|
||||
limitations: result.value.limitations,
|
||||
}),
|
||||
)
|
||||
: result;
|
||||
} finally {
|
||||
interruption.dispose();
|
||||
}
|
||||
try {
|
||||
await runtime.close();
|
||||
this.#pendingNativeUiRuntimes.delete(runtime);
|
||||
} catch (cause: unknown) {
|
||||
result = finishNativeUiObservation(
|
||||
result,
|
||||
operation,
|
||||
observationSignal,
|
||||
);
|
||||
const cleanup = this.#nativeUiCleanupFailure(
|
||||
[{ runtime, cause }],
|
||||
operation,
|
||||
result.ok ? result.value : undefined,
|
||||
);
|
||||
return err(
|
||||
result.ok
|
||||
? cleanup
|
||||
: analysisErrorWithCleanupFailure(
|
||||
result.error,
|
||||
cleanup,
|
||||
operation,
|
||||
).bindCleanupDependencies([result.error, cleanup]),
|
||||
);
|
||||
}
|
||||
result = finishNativeUiObservation(result, operation, observationSignal);
|
||||
return result.ok
|
||||
? ok(
|
||||
createAnalysisExecution(result.value, IDENTITY, {
|
||||
limitations: result.value.limitations,
|
||||
}),
|
||||
)
|
||||
: result;
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1243,15 +1244,15 @@ const parseEntitlements = (
|
||||
...(projected.nonFiniteRealCount === 0
|
||||
? []
|
||||
: [nonFiniteRealLimitation(projected.nonFiniteRealCount)]),
|
||||
...(projected.unknownRealCount === 0
|
||||
? []
|
||||
: [unknownRealLimitation(projected.unknownRealCount)]),
|
||||
...(projected.negativeZeroRealCount === 0
|
||||
? []
|
||||
: [negativeZeroRealLimitation(projected.negativeZeroRealCount)]),
|
||||
...(projected.unknownDateCount === 0
|
||||
? []
|
||||
: [unknownDateLimitation(projected.unknownDateCount)]),
|
||||
...(projected.unknownScalarCount === 0
|
||||
? []
|
||||
: [unknownScalarLimitation(projected.unknownScalarCount)]),
|
||||
],
|
||||
};
|
||||
};
|
||||
|
||||
Loaded 100 of 135 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user