Commit Graph
23 Commits
Author SHA1 Message Date
Mert Kayaandmorluto cad3209533 fix(javascript): attribute call-path unknowns only where the walk can reach Electron (#4915)
* test(javascript): cover the isolated-world probe outside Electron identity

* fix(javascript): expand Electron call paths only with an Electron reference

* test(javascript): cover Electron reach through resolved names, aliases and worker URLs

* fix(javascript): attribute call-path unknowns only where the walk can reach Electron

* test(javascript): cover a hoisted read that precedes a loader alias

* fix(javascript): keep call-path unknowns for late-initialized loader aliases

* test(javascript): drop a duplicate loader-alias case

* test(javascript): cover loader aliases reached through bindings and calls

* fix(javascript): prove a late initializer loader-free before dropping its unknown

* test(javascript): cover loaders behind default parameters, wrappers and long alias chains

* fix(javascript): resolve identifiers inside late initializers and bound the loader proof

* fix(javascript): bound late loader proofs and preserve input uncertainty

* fix(javascript): fail closed for receiver and mutated loaders

* fix(javascript): close loader-free proof gaps

* fix(javascript): retain uncertain loader sources

* test(javascript): cover remaining loader proof gaps

* fix(javascript): bound alias writes and preserve loader uncertainty

* test(javascript): cover demand-driven loader mutation facts

* perf(javascript): index loader mutation facts on demand

* style(javascript): apply oxfmt to the loader proof changes

* test(javascript): cover inherited loaders and rebound alias writes

* fix(javascript): preserve inherited loaders and rebound aliases

* test(evaluation): update scoped feature coverage oracle

* test(javascript): cover inherited loader proof gaps

* fix(javascript): preserve projected and inherited loaders

* fix(javascript): prove safe parameter member reads

* test(javascript): cover loader proof review gaps

* fix(javascript): close remaining loader proof gaps

* fix(javascript): preserve qualified global replacements

* fix(javascript): preserve spread rest uncertainty

* fix(javascript): retain retried handler uncertainty

* fix(javascript): retain import.meta loader uncertainty

* fix(javascript): detach fresh constructed aliases

* fix(javascript): check inherited members of ambient loaders

* test(javascript): parse call-path gate fixtures as established CommonJS

* fix(javascript): prove projected locals inside late initializers

A function-local binding was accepted by scope alone, so a destructuring
projection such as `const { r } = {}` inside a late initializer never met
the inherited-member check that a top-level projection goes through. Route
projected locals through the binding proof; locals with inline initializers
keep the scope shortcut, since the walk already visits their expressions.

* fix(javascript): keep late results with changed prototypes or import.meta uncertain

The loader-free proof inspected member expressions inside a late function
but not what its caller does with the result. With `Object.prototype.load =
require`, `var load = () => ({})` and `load().load("electron")` reach the
loader through a member selected after the call, and a bare `import.meta`
returned after `import.meta.load = ...` does the same. Check each result
expression: a literal fails when its prototype chain gained members, a bare
`import.meta` when any of its members changed, and a result of unknown shape
when either happened. Promise and generator results inherit from
Object.prototype as well. `SemanticPropertyPathCoverage.touches` answers
whether any retained write or escape sits below a prefix.

* fix(javascript): detach replacements held by fresh bindings

`alias = replacement` with `const replacement = {}` selected an identifier
initializer, so the later `alias.other = 1` stayed a shared write on the
original alias group and the resolver treated `base` as overwritten,
dropping its bridge call path. Resolve the identifier to its binding and
accept it when that binding only ever received one fresh allocation.
Bindings that can hold the original object keep the shared write.

* fix(javascript): charge loader proofs through the wrapper-visit budget

#11319 replaced the wrapper-visit counter with a Budget, so the loader proof charge no longer compiled on current main. Spend it through the shared helper, which records the same limit.

* test(javascript): keep the alias bound off the quadratic write index

The 8,000 by 8,000 alias bound spent its time in the ambient write index, which walks every store to an object for each member write. That cost is the same on main and is not what the bound covers. Reach the same 64 million alias and write pairs with 32,000 aliases and 2,000 writes, which runs in about 2.5 seconds instead of 8.5 on the same host.

Drop the 30 second overrides on both alias bounds too. They halved the hosted CI deadline the source project gives them, and the testing guide asks tests to leave deadlines to the project.

* refactor(javascript): ask the effect log for loader proof effects

Main now guards reads of the effect indexes outside javascriptSemanticEffectLog.ts. The loader proof read them directly in 15 places. Route those reads through the log: the existing ambientPathMayHaveChanged and bindingHasMemberWrites queries, plus three timeless queries for a path or any member below it, an ambient binding write, and an escape at a site other than the call being proved. The answers are unchanged.

* fix(javascript): retain uncertainty for changed ambient values

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-11-19 05:37:00 +00:00
morluto 48e27e19b7 test: prune redundant and implementation-coupled coverage 2026-11-17 18:48:24 -05:00
morluto 1f24b3fa57 feat(evidence)!: name the Evidence subject by its target identity (format revision 2) (#11416)
* feat(evidence)!: name the Evidence subject by its target identity (format revision 2)

The Evidence subject carries subject.identity (kind, sha256, format,
architecture, selection) and the Evidence ID commits to it, so raw/DOL
entry points and ESP flash offsets of the same bytes are different
subjects. Every record declares evidence_format: 2; revision 1 bundles,
standalone records and snapshots are refused with the recreate guidance.
binaryTargetIdentity is the one projection of an open target, used by
Evidence and snapshot comparison.

BREAKING CHANGE: Evidence format revision 2. Evidence IDs commit to the
subject's target identity (subject.identity, including the raw/DOL entry
point or ESP flash offsets), and every record declares evidence_format: 2.
Evidence bundles, records and analysis snapshots from earlier releases are
refused; recreate the observations with this release.

* refactor(analysis-view): read parent subjects through the Evidence identity accessors (#11423)

Analysis views compare their parent's file identity (evidenceFileTarget)
and take path and digest from evidenceSubjectTarget instead of reading
the subject's digest, format and architecture members, which duplicate
subject.identity until they are removed. A counted guard
(evidence-subject-declaration-read) keeps new direct reads out and
records the 20 remaining files.

* fix(evidence): carry format refusals through Evidence readers and bind target selections

- safeParseEvidence/safeParseImmutableEvidence return the format refusal as
  an expected failure; every consumer maps it to the one integrity refusal
  with the recreate guidance. An unmarked object is an earlier record only
  when it has every member records have always carried.
- ArtifactProvider inventory roots and provider subjects that name the
  target's own bytes carry the target's selection (ESP flash offsets).
- Snapshot rows bind only Evidence whose identity, selection included,
  matches the snapshot target.
- Entry-point selections use the raw-binary entry point form.

* fix(evidence): refuse unauthenticated or earlier-format records instead of reading them as absent

- recognizeEvidence (evidenceValidation) replaces tryParseEvidence: input
  without an evidence_id is not Evidence (ok(undefined)); a record that names
  one but fails authentication or is from another format revision is an
  EvidenceIntegrityError carrying the recreate guidance, never undefined.
- Direct analysis retains that refusal as a secondary failure instead of
  silently skipping source capture; cleanup projection, source-closure
  annotation and the flutter, debugger and release-exposure recoveries use
  the same owner (the last three previously read the unauthenticated schema).
- A desktop capture.json from an earlier format is refused with the recreate
  guidance instead of "not desktop application capture Evidence".

* fix(evidence): advertise revision 2 ledger examples and rank delivery omissions by serialized size

- The obligation ledger and reconstruction coverage examples named
  evidence_format 1, which revision 2 refuses; they now use
  EVIDENCE_FORMAT_REVISION.
- MCP delivery chose which unretained recovery to omit by a parsed-storage
  lower bound clipped at the response budget. Once revision 2 subjects made
  the source record's estimate cross the budget, both candidates tied at the
  clip and the smaller source was omitted ahead of the larger completed
  observation. Candidates are now ranked by serialized size, which is what
  the budget counts.

* refactor(native): compare function snapshots by their Evidence target identity (#11450)

Function snapshots carry the subject's file target identity instead of the
duplicated subject declaration. Call paths and function comparisons compare
targets through sameTargetIdentity, so the same bytes opened at a different
raw entry point or ESP flash offset no longer count as one coordinate space:
build_call_path refuses the mix, and compare_functions no longer matches
entries by address across selections. Structural candidates and native
function matching read the architecture and digest through
evidenceFileTarget.

Refs #11417

* refactor(artifacts): compare artifact and WASM summaries by their Evidence target identity (#11461)

Artifact comparison authenticates a summary against its retained inventory
through sameTargetIdentity and rebinds the alias with the whole target,
selection included, so a flash-selected inventory summary taken under an
alias path is no longer refused. WASM comparison, the analysis view
artifact projection and the inventory root check read the digest, format
and display path through the Evidence identity accessors.

Refs #11417

* refactor(evidence): read record subjects through the identity accessors in JS, managed, ledger and bundle readers (#11470)

JavaScript application graphs, runtime reconciliation, managed inspection
and native verification sources, the record summary filter, the ledger's
residual-unknown target and bundle scope selection read the subject digest,
format and display path through evidenceFileTarget and
evidenceSubjectDisplay. No comparison changes: each reader matches one
record's facts against its own payload or a caller filter.

Refs #11417

* refactor(evidence)!: name the Evidence subject only by its target identity (#11497)

* refactor(evidence)!: name the Evidence subject only by its target identity

The subject keeps `identity` and the display facts `name` and `local_path`.
The duplicated `digest`, `format` and `architecture` members are removed; the
format-owned declaration now validates the identity itself. Readers the type
checker found move to the owner accessors; untyped reads in verify scripts,
the efficiency harness and test matchers move to `subject.identity`. The
`evidence-subject-declaration-read` text guard and its baseline are deleted.

Fixes #11473
Refs #11417

BREAKING CHANGE: the Evidence subject no longer has `digest`, `format` or
`architecture`; read `subject.identity.sha256`, `subject.identity.format` and
`subject.identity.architecture`. A record that still carries them is refused.

* test(evidence): give the Flutter verifier fixture the identity subject and check the DOL declaration without the ID

* test(evidence): drop the removed subject members from the declaration matrix and tamper fixtures

* feat(evidence)!: give every Evidence record an explicit subject identity (#11536)

* feat(evidence)!: give every Evidence record an explicit subject identity

A producer that cannot name a file passes an `unknown` target identity
with a closed reason (derived-from-linked-evidence, not-observed,
unsupported-subject) instead of no subject. The subject is non-null;
unknown subjects have no display facts. The generic missing-subject
limitation and `subjectUnavailableReason` are removed; producer-specific
explanations are ordinary limitations. Comparison identity requires the
derived unknown, and null-subject consumers ask the owner accessors.

Refs #11417

BREAKING CHANGE: every Evidence record has a subject. A record without a
file target names `subject.identity` `{ "kind": "unknown", "reason": … }`
with `name` and `local_path` null, instead of `subject: null` and the
limitation "Artifact identity is unavailable for this observation.".
Their Evidence IDs change.

* fix(evidence): name explicit unknown subjects in JavaScript callers and keep a managed graph's file subject

The verification scripts are not type-checked, so five createEvidence
callers still passed undefined (or a missing first target) after the subject
became required. They now pass unknownTarget("not-observed").

A managed application graph chose the first present input subject. Every
subject is now present, so an unknown artifact subject hid a file subject on
the member or boundary Evidence. The graph now takes the first input subject
that names a file.

* test(evidence): assert the subject kind instead of a presence the type now guarantees

Every Evidence record has a subject after this change, so test guards and
assertions that checked it for null always passed. Each now checks that the
fixture subject names a file, or is removed where a later fileIdentity,
fileSubjectTarget or fileSubjectDisplay call already requires that. Non-null
assertions on Evidence subjects are removed; analysis-view inputs, whose
subject is optional, keep an undefined check.

* test(evidence): expect the Evidence subject format under its identity

Two boundary tests still matched subject.format, which slice 2e moved into
subject.identity. toMatchObject is not type-checked against the Evidence
shape, so they failed only when run (the signature test on macOS).
2026-11-16 06:28:01 +08:00
morluto 7f920465c5 fix(ci): align verification with current lifecycle contracts 2026-11-11 15:10:51 -05:00
morluto 7bcefead69 test: strengthen and prune audited test coverage (#8573)
* test: strengthen and prune audited test coverage

* test: require delivered DeepSeek skill instructions

* test: keep schema fixture types private

* test: verify Gemini receives skill instructions

* test: bind Qwen skill evidence to its tool result

* test: report NativeAOT verifier checks precisely

* fix: verify Ghidra cleanup paths reliably

* fix: verify Command Code skill activation

* test: tighten cleanup and fixture checks

* test: verify packet capture evidence is recorded

* fix: report HTTP3 capture provenance accurately

* test: preserve resource coverage through MCP

* test: prune repeated assertions and retain cancellation identity

* fix: validate client verifier fixture paths

* test: skip Copilot verifier cases on Windows

* test: keep verifier setup target relative
2026-11-02 02:13:32 +08:00
morluto ab720b4216 test(evaluation): preserve partial coverage in the Notes oracle 2026-10-25 09:07:57 +08:00
morluto be698d4761 test: verify assertion contracts and the production evaluation gate 2026-10-25 08:27:16 +08:00
morluto 92f32981da refactor(mcp): remove the duplicate result delivery class
Construct the canonical ToolResultDelivery through the native continuation policy factory. Move type and lifecycle spy consumers to the core owner while preserving explicit response budgets and the deferred runtime composition.

Validation: runtime build and 335 focused tests passed; 41 Linux-only firmware cases were skipped on macOS. Tests use authored local fixtures, not real external providers.
2026-10-25 02:28:09 +08:00
gensokyo-dev 9895f1870c test(evaluation): find failure gate within cleanup callbacks (#6988) 2026-10-24 10:50:50 +00:00
morluto 3ad465a666 refactor(evaluation): move agent grading into development ownership
Keep the maintained manual evaluator, fixture rubrics and semantic tests together under tests/evaluation/model-evals instead of shipping them in dist. Load the same typed owners from the manual runner and update the architecture and client guide. Grading/report contracts and executable logic are unchanged.
2026-10-24 11:50:11 +08:00
morluto d3700f709b fix(evaluation): align scenario prompts and outcome gates 2026-10-24 02:36:00 +08:00
morluto 79dea89f1f fix(evaluation): own transcript result parsing and error certainty 2026-10-22 17:00:42 +08:00
morluto 443c7aff77 fix(evaluation): require explicit REA server attribution 2026-10-14 09:47:57 +08:00
morluto 4d4631ac94 test: harden verification fixtures and runtime coverage
Use exact discovered test identities, production MCP sessions and byte-exact setup backups. Add CLI signal cleanup coverage and independent Ghidra, Go and firmware verification assertions; prune redundant test configuration and preserve platform limits.
2026-10-14 08:59:27 +08:00
Huamnanxandmorluto ffd8c9c03d test(evaluation): grade managed agent fixture facts (#1924)
* test(evaluation): grade managed agent fixture facts

* test(evaluation): cover distinct managed artifact deliveries

* fix(evaluation): require unfiltered managed type inventories

* fix(evaluation): preserve selected entry point grounding

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-13 23:34:17 +08:00
morluto 1447fd2f99 fix(evaluation): share completed transcript facts across graders 2026-10-13 13:12:12 +08:00
Ysopwnable 844fa52772 fix(evaluation): authenticate delivered Evidence citations
Authenticate completed successful REA Evidence records for citation accounting and shared fixture grading. Preserve genuine structured and complete text-only delivery, including authenticated inventory substeps.

Closes #1728.
2026-10-11 14:48:46 +08:00
Ysopwnable 2653b194b6 feat(skills)!: improve Codex onboarding and task recovery (#1613)
Guide task-centered recovery from observed client, registration, PATH and
provider blockers. Install the recovery reference, document exact JavaScript
module selectors, and require clarification before searching for an unnamed
app. Validate public setup in isolated Codex accounts and preserve failed
or unrecognized tool outcomes in the existing evaluator.

Refs: #1587

BREAKING CHANGE: Optional verify:agent reports now use schemaVersion 4.
Consumers must accept expectedFirstTool: null and targetClarificationPassed
for missing-target scenarios. Production CLI/MCP contracts are unchanged.
2026-10-11 03:13:56 +08:00
morluto 63bf91405c refactor!: enforce canonical evidence and provider boundaries
Deliver complete Evidence directly through MCP with required nullable analysis profiles and immutable per-server delivery settings. Bind selected environments at construction and require typed cleanup outcomes with retained retry ownership.

Remove test-only routing, duplicate session and snapshot metadata, producer-field repairs, and ambiguous provider arguments. Separate artifact byte identity from occurrence classification and retain typed source-map and partial native facts across CLI, MCP, and verifier consumers.

BREAKING CHANGE: obsolete Evidence envelopes, session aliases, incomplete snapshots, and missing normalized producer facts are no longer accepted.
2026-10-09 20:18:52 +08:00
YouWishandN0zoM1z0 27a9ceade7 fix!: restore caching, preserve failures, and verify fixture claims (#1211)
Restore profile-bound Ghidra snapshot reuse, preserve direct-analysis results through cleanup failures, and grade configured agent fixture claims against returned evidence.

Register the compiled agent-evaluation fixture entry for static analysis.

Co-authored-by: YouWish <29498139+Y0UW1SH@users.noreply.github.com>
Co-authored-by: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com>
2026-10-09 14:49:09 +08:00
morluto 0a4caf4a0a fix(evaluation): canonicalize Unicode argument keys (#980) 2026-10-07 23:19:49 +08:00
morluto 2a6eb41147 refactor(tools): return complete analysis results inline 2026-09-29 14:53:50 +08:00
morluto 513d9b1555 test: overhaul deterministic suite architecture 2026-07-29 22:00:38 +00:00