Commit Graph
6 Commits
Author SHA1 Message Date
morluto 3ad465a666 refactor(evaluation): move agent grading into development ownership
Keep the maintained manual evaluator, fixture rubrics and semantic tests together under tests/evaluation/model-evals instead of shipping them in dist. Load the same typed owners from the manual runner and update the architecture and client guide. Grading/report contracts and executable logic are unchanged.
2026-10-24 11:50:11 +08:00
morluto 4d4631ac94 test: harden verification fixtures and runtime coverage
Use exact discovered test identities, production MCP sessions and byte-exact setup backups. Add CLI signal cleanup coverage and independent Ghidra, Go and firmware verification assertions; prune redundant test configuration and preserve platform limits.
2026-10-14 08:59:27 +08:00
Huamnanxandmorluto ffd8c9c03d test(evaluation): grade managed agent fixture facts (#1924)
* test(evaluation): grade managed agent fixture facts

* test(evaluation): cover distinct managed artifact deliveries

* fix(evaluation): require unfiltered managed type inventories

* fix(evaluation): preserve selected entry point grounding

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-13 23:34:17 +08:00
Ysopwnable 844fa52772 fix(evaluation): authenticate delivered Evidence citations
Authenticate completed successful REA Evidence records for citation accounting and shared fixture grading. Preserve genuine structured and complete text-only delivery, including authenticated inventory substeps.

Closes #1728.
2026-10-11 14:48:46 +08:00
morluto 63bf91405c refactor!: enforce canonical evidence and provider boundaries
Deliver complete Evidence directly through MCP with required nullable analysis profiles and immutable per-server delivery settings. Bind selected environments at construction and require typed cleanup outcomes with retained retry ownership.

Remove test-only routing, duplicate session and snapshot metadata, producer-field repairs, and ambiguous provider arguments. Separate artifact byte identity from occurrence classification and retain typed source-map and partial native facts across CLI, MCP, and verifier consumers.

BREAKING CHANGE: obsolete Evidence envelopes, session aliases, incomplete snapshots, and missing normalized producer facts are no longer accepted.
2026-10-09 20:18:52 +08:00
YouWishandN0zoM1z0 27a9ceade7 fix!: restore caching, preserve failures, and verify fixture claims (#1211)
Restore profile-bound Ghidra snapshot reuse, preserve direct-analysis results through cleanup failures, and grade configured agent fixture claims against returned evidence.

Register the compiled agent-evaluation fixture entry for static analysis.

Co-authored-by: YouWish <29498139+Y0UW1SH@users.noreply.github.com>
Co-authored-by: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com>
2026-10-09 14:49:09 +08:00