diff --git a/experiments/analysis-efficiency/README.md b/experiments/analysis-efficiency/README.md index 9fbd777f9c..25438895da 100644 --- a/experiments/analysis-efficiency/README.md +++ b/experiments/analysis-efficiency/README.md @@ -44,8 +44,9 @@ acceptance conditions for every optimization. experiment control; it does not itself establish reduced CPU consumption. Migrate domain by domain with focused real-artifact checks and relevant shared -contract checks. The domain matrix is a growing roadmap; native PE and DOS, for -example, remain unverified in this study. A passed lane does not imply all +contract checks. The native follow-up verifies selected PE and DOS MZ entry facts; +DOS COM and complete native application behavior remain unverified. The domain +matrix is a growing roadmap. A passed lane does not imply all advertised platforms/providers have passed. Newly discovered REA bugs remain in the separate local handoff document for later team repair. diff --git a/experiments/analysis-efficiency/evm-descriptor.md b/experiments/analysis-efficiency/evm-descriptor.md index 9ba0dbb140..8e3611f282 100644 --- a/experiments/analysis-efficiency/evm-descriptor.md +++ b/experiments/analysis-efficiency/evm-descriptor.md @@ -21,7 +21,7 @@ real EVMole/WABT claims. Nine service and 26 provider-boundary cases also pass; two unsupported-host scenarios are skipped on this Linux host. The existing seven WASM MCP scenarios pass. Full `npm run check` passes; its approximately 3 GiB sampled peak and `memory.high` events are excluded from performance claims. -Fresh candidate real-artifact/catalog and model comparisons are still pending. +Candidate real-artifact/catalog and model observations are recorded below. ## Public inputs and independent checks @@ -94,5 +94,88 @@ and `--output` selecting a new private receipt. It acquires no dependencies and does not overwrite an existing receipt. The [template model question](questions/evm-uniswap.txt) requires the same complete -inferred interface facts and explicit unknowns. Real model trials and a candidate -recommendation are pending; an ABI-count match is not used as a model grade. +inferred interface facts and explicit unknowns. An ABI-count match is not used as +a model grade. + +## Candidate compatibility and real-provider verification + +Candidate: `152984da771450048ee9d60db8a410994a125878`, frozen runtime file-table +SHA-256 `725563640c27eb50ddb6f977d2dfbe1bdaddf7e05c02897318bceaa36fa602d7`. +Two repetitions for each EVM input produce six passing paired fact/settings +comparisons with the same driver and explicit limits. Each report preserves both +original Evidence reference maps. The complete actual SDK catalog is byte-identical +on every pair, including order, names, descriptions, schemas and annotations; +catalog SHA-256 is +`8664e42fd5225645cfd910f440f731c67ac198d3d53aa85b5c485374c74cf603`. +Result byte counts remain unchanged. Candidate Factory/Pool selectors also match +the independent published ABI oracle. + +| Question | Control seconds, repetitions 1 / 2 | Candidate seconds, repetitions 1 / 2 | +| -------- | ---------------------------------: | -----------------------------------: | +| Factory | 3.628 / 3.650 | 3.708 / 3.605 | +| Pool | 11.211 / 3.591 | 3.749 / 3.581 | +| WETH | 11.432 / 3.694 | 3.572 / 3.677 | + +First controls preceded wiring; each second pair ran candidate then control. +Every measurement starts a new MCP process; host cache/page state is not fully +controlled. Slower initial controls do not establish a descriptor speedup. +Second-pair latency and last-observed CPU are similar. Sampled EVM SDK family peaks +range from 510,255,104 to 527,622,144 bytes. No scope abort occurred; final counters +remain unknown, so these measurements are descriptive. + +The candidate passes all 59 existing real CLI/MCP verifier cases with solc 0.8.30 +and unchanged EVMole 0.9.3. Actual syscall traces report target execution and network +sockets absent for those scenarios, and the verifier reports no remaining +descendants. The scope took 177.456 seconds, sampled peak 1,237,020,672 bytes and +last-observed CPU 144.913 seconds. Different totals from the earlier control are +not attributed to unchanged provider algorithms. + +Post-descriptor public MCP regression uses the original pinned shipped corpus: + +| Domain / selected question | Actual provider / scope | Calls | Seconds | +| ------------------------------------------- | -------------------------------------------------- | ----: | ------: | +| SQLite WASM counts / exports | WABT 1.0.42; selected summary and retained read | 2 | 5.516 | +| ShareX managed metadata / native boundaries | Static PE/CLI, no assembly execution | 2 | 3.710 | +| Obsidian manifest / inventory | Complete inventory retained, no AST claim | 4 | 5.376 | +| NewPipe package / MainActivity methods | Real headless JADX 0.7.1 | 3 | 21.669 | +| OpenWrt regions | Real firmware signature scan, no device execution | 1 | 3.581 | +| jq native entry | Ghidra 12.1.4, default analysis, ephemeral Program | 4 | 270.986 | + +All six workflows pass named SDK output validation and original digest checks; +input bytes remain unchanged. Maximum sampled family peak is 1,284,648,960 bytes. +They establish those available Linux workflows, not paired performance or +Windows/macOS, IDA/Hopper, Android-device, DOS COM, browser or JavaScript AST +behavior. Browser/AST were not repeated for this narrow migration. + +## Real model complete-output calibration + +Six serial Codex CLI 0.161.0 trials request `gpt-5.6-terra`, high reasoning, with +isolated temporary profiles pointing to the frozen candidate. Resolved server-side +model identity is not inferred. Only the EVM inspection tool is enabled, matching +the earlier WETH calibration; this is not full-catalog tool selection verification. +Every trial makes one successful call and no failed call. + +| Trial | Total input | Uncached input | Output | Seconds | +| --------- | ----------: | -------------: | -----: | ------: | +| Factory 1 | 66,158 | 36,974 | 1,329 | 43.483 | +| Pool 1 | 65,600 | 36,416 | 2,307 | 69.546 | +| WETH 1 | 54,700 | 25,516 | 1,144 | 56.372 | +| Factory 2 | 65,964 | 36,780 | 1,453 | 26.412 | +| Pool 2 | 65,753 | 36,569 | 1,939 | 74.706 | +| WETH 2 | 54,502 | 25,318 | 1,208 | 35.137 | + +Every trial reports 29,184 cached input tokens. Selected fact/reference/lifecycle +grades all pass: original/decoded identities, every inferred selector, offset, +dispatch, argument/mutability string including null, explicit unknowns, no runtime +execution, observed Evidence references, unchanged bytes and removed profiles. +Raw transcripts and grades remain private. Sampled peaks are +594,853,888–776,335,360 bytes; last-observed CPU is 6.181–7.034 seconds. Strict +overall grades remain unverified because final resource counters are unknown; +they are not relabeled as wholly passing resource-qualified runs. + +This calibrates the current complete path after compatible wiring, not a paired +model performance experiment. The next token experiment should answer the same +interface question through an explicitly selected precise view, retain full +bytecode/original Evidence under the existing lifetime, and preserve complete +defaults. Discover support before sending new options to an older server. Include +discovery, follow-up, failures and retries in the complete task cost.