18 Commits
Author SHA1 Message Date
morluto ab720b4216 test(evaluation): preserve partial coverage in the Notes oracle 2026-10-25 09:07:57 +08:00
morluto be698d4761 test: verify assertion contracts and the production evaluation gate 2026-10-25 08:27:16 +08:00
morluto 92f32981da refactor(mcp): remove the duplicate result delivery class
Construct the canonical ToolResultDelivery through the native continuation policy factory. Move type and lifecycle spy consumers to the core owner while preserving explicit response budgets and the deferred runtime composition.

Validation: runtime build and 335 focused tests passed; 41 Linux-only firmware cases were skipped on macOS. Tests use authored local fixtures, not real external providers.
2026-10-25 02:28:09 +08:00
gensokyo-dev 9895f1870c test(evaluation): find failure gate within cleanup callbacks (#6988) 2026-10-24 10:50:50 +00:00
morluto 3ad465a666 refactor(evaluation): move agent grading into development ownership
Keep the maintained manual evaluator, fixture rubrics and semantic tests together under tests/evaluation/model-evals instead of shipping them in dist. Load the same typed owners from the manual runner and update the architecture and client guide. Grading/report contracts and executable logic are unchanged.
2026-10-24 11:50:11 +08:00
morluto d3700f709b fix(evaluation): align scenario prompts and outcome gates 2026-10-24 02:36:00 +08:00
morluto 79dea89f1f fix(evaluation): own transcript result parsing and error certainty 2026-10-22 17:00:42 +08:00
morluto 443c7aff77 fix(evaluation): require explicit REA server attribution 2026-10-14 09:47:57 +08:00
morluto 4d4631ac94 test: harden verification fixtures and runtime coverage
Use exact discovered test identities, production MCP sessions and byte-exact setup backups. Add CLI signal cleanup coverage and independent Ghidra, Go and firmware verification assertions; prune redundant test configuration and preserve platform limits.
2026-10-14 08:59:27 +08:00
Huamnanxandmorluto ffd8c9c03d test(evaluation): grade managed agent fixture facts (#1924)
* test(evaluation): grade managed agent fixture facts

* test(evaluation): cover distinct managed artifact deliveries

* fix(evaluation): require unfiltered managed type inventories

* fix(evaluation): preserve selected entry point grounding

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-13 23:34:17 +08:00
morluto 1447fd2f99 fix(evaluation): share completed transcript facts across graders 2026-10-13 13:12:12 +08:00
Ysopwnable 844fa52772 fix(evaluation): authenticate delivered Evidence citations
Authenticate completed successful REA Evidence records for citation accounting and shared fixture grading. Preserve genuine structured and complete text-only delivery, including authenticated inventory substeps.

Closes #1728.
2026-10-11 14:48:46 +08:00
Ysopwnable 2653b194b6 feat(skills)!: improve Codex onboarding and task recovery (#1613)
Guide task-centered recovery from observed client, registration, PATH and
provider blockers. Install the recovery reference, document exact JavaScript
module selectors, and require clarification before searching for an unnamed
app. Validate public setup in isolated Codex accounts and preserve failed
or unrecognized tool outcomes in the existing evaluator.

Refs: #1587

BREAKING CHANGE: Optional verify:agent reports now use schemaVersion 4.
Consumers must accept expectedFirstTool: null and targetClarificationPassed
for missing-target scenarios. Production CLI/MCP contracts are unchanged.
2026-10-11 03:13:56 +08:00
morluto 63bf91405c refactor!: enforce canonical evidence and provider boundaries
Deliver complete Evidence directly through MCP with required nullable analysis profiles and immutable per-server delivery settings. Bind selected environments at construction and require typed cleanup outcomes with retained retry ownership.

Remove test-only routing, duplicate session and snapshot metadata, producer-field repairs, and ambiguous provider arguments. Separate artifact byte identity from occurrence classification and retain typed source-map and partial native facts across CLI, MCP, and verifier consumers.

BREAKING CHANGE: obsolete Evidence envelopes, session aliases, incomplete snapshots, and missing normalized producer facts are no longer accepted.
2026-10-09 20:18:52 +08:00
YouWishandN0zoM1z0 27a9ceade7 fix!: restore caching, preserve failures, and verify fixture claims (#1211)
Restore profile-bound Ghidra snapshot reuse, preserve direct-analysis results through cleanup failures, and grade configured agent fixture claims against returned evidence.

Register the compiled agent-evaluation fixture entry for static analysis.

Co-authored-by: YouWish <29498139+Y0UW1SH@users.noreply.github.com>
Co-authored-by: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com>
2026-10-09 14:49:09 +08:00
morluto 0a4caf4a0a fix(evaluation): canonicalize Unicode argument keys (#980) 2026-10-07 23:19:49 +08:00
morluto 2a6eb41147 refactor(tools): return complete analysis results inline 2026-09-29 14:53:50 +08:00
morluto 513d9b1555 test: overhaul deterministic suite architecture 2026-07-29 22:00:38 +00:00