diff --git a/docs/architecture/evaluator-rag-prototype.md b/docs/architecture/evaluator-rag-prototype.md index cb442ae0f..b8dac1b5d 100644 --- a/docs/architecture/evaluator-rag-prototype.md +++ b/docs/architecture/evaluator-rag-prototype.md @@ -123,6 +123,11 @@ Current corpus: expected-output comparisons, representative repository/commit histories, and regression commands before deep-analysis behavior can be promoted. +- `connector-diagnostics-gap`: inventories an already-used connector, separates + observed gaps from unavailable research, and requires evidence before a new + background component is proposed. Decisions are illustrative, not executed + connector checks. + ## ECC Tools Mapping ECC Tools already flags missing RAG/evaluator evidence for retrieval, diff --git a/examples/evaluator-rag-prototype/connector-diagnostics-gap/candidate-playbook.md b/examples/evaluator-rag-prototype/connector-diagnostics-gap/candidate-playbook.md new file mode 100644 index 000000000..9c6950465 --- /dev/null +++ b/examples/evaluator-rag-prototype/connector-diagnostics-gap/candidate-playbook.md @@ -0,0 +1,78 @@ +# Connector Diagnostics Gap Playbook + +Candidate id: `bounded-event-summary` + +This is a synthetic, read-only worked example for the existing +[search-first skill](../../../skills/search-first/SKILL.md). The five artifacts +record expected rubric decisions, not a connector execution or model benchmark. +The motivating [feedback](https://github.com/affaan-m/ECC/issues/3680) reports a +corrected workflow; exact ECC activation, version and causal contribution are +unknown. The fixture does not reconstruct the original connector or deployment. + +## Worked Comparison + +| Requirement | Inspected fixture evidence | Remaining work | +| --- | --- | --- | +| Connection failures | Reachability and last connection error | Reuse existing path | +| Permission denials | Access diagnostic details | Reuse existing path | +| Repeated throttling | Individual throttle events | Check retention and aggregate over the interval | +| Deployed settings | Read unavailable | Keep unknown; do not infer absence | + +Accepted draft: “The inspected connector already exposes connection and access +diagnostics plus throttle events. The missing behavior in the inspected path is +an interval summary. Check event retention and query support, then reuse that +query or add a thin summary operation. A separate collector is justified only if +required data or collection timing cannot be supplied by the existing connector.” + +Rejected draft: “Settings search failed, so install a new background monitor for +all three failure classes.” This duplicates observed capabilities and converts +an unknown into a negative finding. + +## Evaluation Controls + +- With sufficient retained events, prefer reuse or a bounded summary extension. +- If retained events lack the required interval, name that evidence and compare + the smallest connector extension with a separate component's added lifecycle. +- If research is unavailable, preserve the uncertainty and next targeted read; + do not claim the feature is missing or demand a new subsystem by default. +- Do not deploy components or change connector configuration in this example. + +## Management Preparation Variant + +The [additional field example](https://github.com/affaan-m/ECC/issues/3680#issuecomment-6436640503) +contributes a synthetic partial-dependency control. Existing MCP/API management +reads are available, continuous collection is not configured, and server A SSH +is temporarily unavailable. Defer only host-level checks on server A; continue +monitoring templates, capability mapping and inspection of the existing +notification channel before proposing another installation or channel. + +Reject “SSH is unavailable, so all monitoring preparation is blocked.” Also +reject “management reads succeeded, so monitoring and notifications are ready.” +Management access establishes neither continuous collection readiness nor +notification delivery or authority for arbitrary changes. These are expected +fixture decisions, not live checks or a runtime correction of the reported plan. + +## Target Contract Variant + +The [target-contract report](https://github.com/affaan-m/ECC/issues/3680#issuecomment-6455507941) +and its [scope-revision follow-up](https://github.com/affaan-m/ECC/issues/3680#issuecomment-6456519404) +contribute a second synthetic dependency control. Two import variants are +documented, but the required target sample is unread and mandatory fields are +unconfirmed. General development authorization does not confirm the active target contract. +Before dependent document filling, obtain a supported variant/field mapping or +state the precise dependency hold; preserve independent preparation findings. + +An explicit stop halts the requested work and leaves its candidate unfinished, +without promoting it to accepted state. In a later phase, an explicit user scope +revision separately authorizes fact storage and validation. Continue that +independent work; keep future document filling dependent on its target contract. +Do not let the unconfirmed document contract block the newly authorized scope, +or treat later code acceptance as erasing the earlier sequencing mistake. +These are expected fixture decisions, not execution of the reported application, +a new approval ritual, or evidence of a deployed prevention fix. + +## Repository Validation + +Run `node tests/docs/evaluator-rag-prototype.test.js`, Markdown lint on touched +docs, and `git diff --check`. Those checks validate repository documentation; +they do not execute the connector or establish the historical report's cause. diff --git a/examples/evaluator-rag-prototype/connector-diagnostics-gap/report.json b/examples/evaluator-rag-prototype/connector-diagnostics-gap/report.json new file mode 100644 index 000000000..8ff6aab8c --- /dev/null +++ b/examples/evaluator-rag-prototype/connector-diagnostics-gap/report.json @@ -0,0 +1,61 @@ +{ + "schema_version": "ecc.evaluator-rag.report.v1", + "scenario_id": "connector-diagnostics-gap", + "run_id": "illustrative-connector-gap", + "read_only": true, + "result": "illustrative_expected_result", + "findings": [ + { + "id": "remaining-gap", + "severity": "info", + "summary": "Only interval aggregation is absent from the inspected path; retention and deployed settings remain unknown." + }, + { + "id": "no-daemon-evidence", + "severity": "warning", + "summary": "A separate monitor needs evidence that a bounded query or thin extension cannot meet the requirement." + }, + { + "id": "partial-management-dependency", + "severity": "warning", + "summary": "Management-preparation variant separates existing MCP/API access from unconfigured continuous collection: defer only SSH-dependent host checks and continue independent templates, capability mapping and notification-channel inspection. Management reads do not establish collection readiness, notification delivery or change authority." + }, + { + "id": "target-contract-dependency", + "severity": "warning", + "summary": "An unread target sample blocks dependent document filling, not independently authorized fact storage after explicit scope revision. A stopped candidate is not promoted." + } + ], + "recommended_next_action": { + "candidate_id": "bounded-event-summary", + "action": "Check event retention and query support, then propose the smallest necessary summary extension." + }, + "scoring": { + "status": "not_performed", + "scores": null, + "reason": "Illustrative expected decisions only; no model benchmark or connector checks were executed." + }, + "evidence_coverage": { + "kind": "synthetic_fixture_only", + "sources": [ + "scenario.json#/inputs/inspected_capabilities", + "scenario.json#/inputs/unavailable_channels", + "trace.json#/events", + "candidate-playbook.md#worked-comparison", + "scenario.json#/inputs/management_preparation", + "scenario.json#/inputs/target_contract" + ], + "covered": [ + "Connection and permission diagnostics plus per-request throttle events support the bounded-summary decision.", + "Unavailable settings are preserved as unknown rather than evidence of absent capability.", + "Management-preparation inputs separate blocked host checks from independent preparation and preserve collection, delivery and authority limits.", + "Synthetic target-contract sequence preserves the explicit stop, reassesses dependencies after revised authorization, and keeps later acceptance separate from the earlier sequencing mistake." + ], + "gaps": [ + "Event retention and deployed settings are unknown.", + "The historical incident and ECC causal contribution are unverified; no connector was executed.", + "Management access, collection, SSH and notification behavior are synthetic inputs, not live checks.", + "No target sample, original candidate code, source-system execution or deployed recovery was inspected." + ] + } +} diff --git a/examples/evaluator-rag-prototype/connector-diagnostics-gap/scenario.json b/examples/evaluator-rag-prototype/connector-diagnostics-gap/scenario.json new file mode 100644 index 000000000..53cd89c3e --- /dev/null +++ b/examples/evaluator-rag-prototype/connector-diagnostics-gap/scenario.json @@ -0,0 +1,106 @@ +{ + "schema_version": "ecc.evaluator-rag.scenario.v1", + "scenario_id": "connector-diagnostics-gap", + "title": "Inspect an existing connector before designing another monitor", + "mode": "read_only_prototype", + "objective": "Identify the smallest demonstrated diagnostics gap before recommending a new subsystem.", + "sources": [ + { + "kind": "repo_doc", + "path": "skills/search-first/SKILL.md", + "purpose": "Existing research, reuse and minimal-extension guidance" + }, + { + "kind": "public_feedback", + "url": "https://github.com/affaan-m/ECC/issues/3680", + "purpose": "Reported workflow observation; not an ECC reproduction" + }, + { + "kind": "public_feedback", + "url": "https://github.com/affaan-m/ECC/issues/3680#issuecomment-6436640503", + "purpose": "Reported management-transport versus collection example; synthetic variant, not historical reproduction" + }, + { + "kind": "public_feedback", + "url": "https://github.com/affaan-m/ECC/issues/3680#issuecomment-6455507941", + "purpose": "Reported premature target-dependent work and explicit stop; synthetic control only" + }, + { + "kind": "public_feedback", + "url": "https://github.com/affaan-m/ECC/issues/3680#issuecomment-6456519404", + "purpose": "Later explicit scope revision separates fact storage from document filling; synthetic continuation only" + } + ], + "evidence_limits": "The reported connector configuration, original request, ECC activation/version and causal contribution are unverified. All connector capabilities below are synthetic evaluation inputs.", + "inputs": { + "request": "Explain connection failures, permission denials and repeated throttling using the connector already in use.", + "inspected_capabilities": [ + { + "path": "connector/diagnostics/connection", + "observed": "Returns endpoint reachability and last connection error" + }, + { + "path": "connector/diagnostics/access", + "observed": "Returns permission-denial details" + }, + { + "path": "connector/diagnostics/requests", + "observed": "Returns per-request throttle events, without an aggregate count in the inspected path" + } + ], + "unavailable_channels": [ + "Historical deployment settings could not be read" + ], + "remaining_requirement": "A count of throttle events over the requested interval", + "management_preparation": { + "management_access": "existing_mcp_api_reads", + "continuous_collection": "not_configured", + "unavailable_path": { + "path": "server A SSH", + "blocks": [ + "host-level checks on server A" + ] + }, + "independent_preparation": [ + "prepare monitoring templates", + "map existing management capabilities", + "inspect the existing notification channel" + ] + }, + "target_contract": { + "development_authorized": true, + "documented_variants": [ + "separate payment document", + "document payment section" + ], + "target_sample": "unread", + "mandatory_fields": "unconfirmed", + "independent_work": [ + "fact storage", + "fact validation" + ], + "scope_sequence": [ + "development authorized", + "explicit stop", + "explicit scope revision authorizes fact storage and validation; document filling remains separate" + ] + } + }, + "retrieval_questions": [ + "Which actual connector paths were examined?", + "What requested behavior is missing from those paths?", + "Can the existing events satisfy the gap with a bounded query?", + "What evidence would justify a separate background component?" + ], + "forbidden_actions": [ + "deploying a background monitor from this evaluator", + "treating unavailable research as proof of missing capability", + "claiming the reported incident is a reproduced ECC defect" + ], + "acceptance_gates": [ + "name examined capability and evidence scope", + "state the observed limitation separately from unknown settings", + "choose the smallest extension supported by available events", + "justify any new background component with a demonstrated unmet requirement" + ] +} diff --git a/examples/evaluator-rag-prototype/connector-diagnostics-gap/trace.json b/examples/evaluator-rag-prototype/connector-diagnostics-gap/trace.json new file mode 100644 index 000000000..6e7eaf525 --- /dev/null +++ b/examples/evaluator-rag-prototype/connector-diagnostics-gap/trace.json @@ -0,0 +1,43 @@ +{ + "schema_version": "ecc.evaluator-rag.trace.v1", + "scenario_id": "connector-diagnostics-gap", + "run_id": "illustrative-connector-gap", + "read_only": true, + "illustrative": true, + "events": [ + { + "phase": "observation", + "summary": "Synthetic request needs three diagnostics; the connector is already in use." + }, + { + "phase": "retrieval", + "summary": "The supplied fixture covers connection and permission diagnostics and per-request throttle events; deployment settings remain unknown." + }, + { + "phase": "proposal", + "candidate_ids": [ + "bounded-event-summary", + "new-monitor-without-inventory" + ] + }, + { + "phase": "verification", + "summary": "Expected rubric comparison only: accept a bounded aggregate over existing events; reject a separate daemon justified only by unavailable settings." + }, + { + "phase": "verification", + "variant": "management_preparation", + "summary": "Expected rubric only: existing MCP/API reads support template/map preparation and inspection of the existing notification channel. Unavailable server A SSH blocks only its host-level checks; collection readiness, delivery and change authority remain unestablished." + }, + { + "phase": "verification", + "variant": "target_contract", + "summary": "Expected rubric only: hold document filling for the target sample and mandatory-field mapping; explicit stop leaves the candidate unfinished. Later explicit scope revision permits independent fact storage and validation while document filling remains separate; it does not erase the earlier sequencing error." + }, + { + "phase": "promotion", + "promoted_candidate_id": "bounded-event-summary", + "summary": "Illustrative playbook selection; no connector, service or settings are changed." + } + ] +} diff --git a/examples/evaluator-rag-prototype/connector-diagnostics-gap/verifier-result.json b/examples/evaluator-rag-prototype/connector-diagnostics-gap/verifier-result.json new file mode 100644 index 000000000..1e9f7c586 --- /dev/null +++ b/examples/evaluator-rag-prototype/connector-diagnostics-gap/verifier-result.json @@ -0,0 +1,73 @@ +{ + "schema_version": "ecc.evaluator-rag.verifier.v1", + "scenario_id": "connector-diagnostics-gap", + "run_id": "illustrative-connector-gap", + "read_only": true, + "evaluation_status": "illustrative_expected_decisions_not_executed_connector_checks", + "candidates": [ + { + "candidate_id": "bounded-event-summary", + "decision": "accepted", + "reasons": [ + "Names all inspected diagnostic paths and their observed coverage", + "Uses existing throttle events for the remaining interval-summary requirement", + "Keeps retention and deployed settings unknown until checked", + "Requires evidence before introducing a new background component" + ] + }, + { + "candidate_id": "new-monitor-without-inventory", + "decision": "rejected", + "reasons": [ + "Duplicates existing connection and access diagnostics", + "Infers absent capability from an unavailable settings channel", + "Provides no evidence that existing events cannot satisfy the remaining gap" + ] + } + ], + "promoted_candidate_id": "bounded-event-summary", + "rollback": { + "current_run": "Not applicable: No connector state is changed by this read-only illustrative selection.", + "future_extension": "If later implemented, revert the summary extension and its query/configuration changes to restore the existing diagnostics paths; verify connection, access and per-request event behavior before redeployment." + }, + "management_preparation": { + "decision": "continue_independent_preparation", + "blocked_work": [ + "host-level checks on server A" + ], + "continue_work": [ + "prepare monitoring templates", + "map existing management capabilities", + "inspect the existing notification channel" + ], + "not_established": [ + "continuous collection readiness", + "notification delivery", + "authority for arbitrary changes" + ], + "rejected_inferences": [ + "unavailable SSH blocks all monitoring preparation", + "existing management access requires redundant installation", + "successful management reads prove collection or notification readiness" + ] + }, + "target_contract": { + "before_stop": { + "document_filling": "hold_for_target_mapping", + "reason": "General documentation does not identify the active variant and mandatory fields; preserve independent findings." + }, + "after_stop": { + "action": "stop_requested_work", + "candidate_state": "unfinished_not_promoted" + }, + "after_scope_revision": { + "authorization_source": "explicit_user_scope_revision", + "continue_work": [ + "fact storage", + "fact validation" + ], + "document_filling": "separate_phase_awaiting_target_contract", + "erases_earlier_sequencing_error": false + } + } +} diff --git a/tests/docs/evaluator-rag-prototype.test.js b/tests/docs/evaluator-rag-prototype.test.js index 075ec7dce..129b2d5c7 100644 --- a/tests/docs/evaluator-rag-prototype.test.js +++ b/tests/docs/evaluator-rag-prototype.test.js @@ -409,6 +409,117 @@ test('deep analyzer evidence scenario rejects no-corpus analyzer changes', () => assert.ok(playbook.includes('Deep Analyzer Evidence')); }); +test('connector gap artifacts share identity and preserve read-only scope', () => { + const scenario = readFixtureJson('connector-diagnostics-gap/scenario.json'); + assert.strictEqual(scenario.scenario_id, 'connector-diagnostics-gap'); + assert.strictEqual(scenario.mode, 'read_only_prototype'); + for (const [file, schema] of [ + ['trace.json', 'trace'], ['report.json', 'report'], ['verifier-result.json', 'verifier'] + ]) { + const artifact = readFixtureJson(`connector-diagnostics-gap/${file}`); + assert.strictEqual(artifact.schema_version, `ecc.evaluator-rag.${schema}.v1`); + assert.strictEqual(artifact.scenario_id, scenario.scenario_id); + assert.strictEqual(artifact.run_id, 'illustrative-connector-gap'); + assert.strictEqual(artifact.read_only, true); + } + assert.strictEqual(scenario.schema_version, 'ecc.evaluator-rag.scenario.v1'); +}); + +test('connector gap decisions reject unsupported monitors and preserve uncertainty', () => { + const scenario = readFixtureJson('connector-diagnostics-gap/scenario.json'); + const trace = readFixtureJson('connector-diagnostics-gap/trace.json'); + const verifier = readFixtureJson('connector-diagnostics-gap/verifier-result.json'); + const playbook = read('examples/evaluator-rag-prototype/connector-diagnostics-gap/candidate-playbook.md'); + const accepted = verifier.candidates.find(candidate => candidate.candidate_id === 'bounded-event-summary'); + const rejected = verifier.candidates.find(candidate => candidate.candidate_id === 'new-monitor-without-inventory'); + assert.ok(accepted, 'Missing bounded summary candidate'); + assert.ok(rejected, 'Missing unsupported monitor candidate'); + assert.strictEqual(accepted.decision, 'accepted'); + assert.strictEqual(rejected.decision, 'rejected'); + assert.strictEqual(verifier.promoted_candidate_id, accepted.candidate_id); + assert.ok(trace.events.some(event => event.promoted_candidate_id === accepted.candidate_id)); + assert.strictEqual(trace.illustrative, true); + assert.strictEqual(verifier.evaluation_status, 'illustrative_expected_decisions_not_executed_connector_checks'); + assert.ok(accepted.reasons.includes('Keeps retention and deployed settings unknown until checked')); + assert.ok(rejected.reasons.includes('Infers absent capability from an unavailable settings channel')); + assert.ok(scenario.forbidden_actions.includes('treating unavailable research as proof of missing capability')); + assert.ok(scenario.forbidden_actions.includes('claiming the reported incident is a reproduced ECC defect')); + assert.ok(scenario.forbidden_actions.includes('deploying a background monitor from this evaluator')); + assert.ok(scenario.acceptance_gates.includes('state the observed limitation separately from unknown settings')); + assert.ok(playbook.includes('If research is unavailable, preserve the uncertainty and next targeted read')); + assert.ok(playbook.includes('Do not deploy components or change connector configuration')); +}); + +test('connector report records unscored fixture coverage and verifier rollback', () => { + const report = readFixtureJson('connector-diagnostics-gap/report.json'); + const verifier = readFixtureJson('connector-diagnostics-gap/verifier-result.json'); + assert.strictEqual(report.scoring.status, 'not_performed'); + assert.strictEqual(report.scoring.scores, null); + assert.strictEqual(report.evidence_coverage.kind, 'synthetic_fixture_only'); + assert.deepStrictEqual(report.evidence_coverage.sources, [ + 'scenario.json#/inputs/inspected_capabilities', + 'scenario.json#/inputs/unavailable_channels', + 'trace.json#/events', + 'candidate-playbook.md#worked-comparison', + 'scenario.json#/inputs/management_preparation', + 'scenario.json#/inputs/target_contract' + ]); + assert.ok(report.evidence_coverage.gaps.includes('Event retention and deployed settings are unknown.')); + assert.strictEqual(report.recommended_next_action.candidate_id, verifier.promoted_candidate_id); + assert.ok(verifier.rollback.current_run.includes('No connector state is changed')); + assert.ok(verifier.rollback.future_extension.includes('revert the summary extension')); +}); + +test('connector preparation scopes unavailable SSH to dependent host checks', () => { + const scenario = readFixtureJson('connector-diagnostics-gap/scenario.json'); + const verifier = readFixtureJson('connector-diagnostics-gap/verifier-result.json'); + const variant = scenario.inputs.management_preparation; + assert.strictEqual(variant.management_access, 'existing_mcp_api_reads'); + assert.strictEqual(variant.continuous_collection, 'not_configured'); + assert.deepStrictEqual(variant.unavailable_path.blocks, ['host-level checks on server A']); + assert.deepStrictEqual(variant.independent_preparation, [ + 'prepare monitoring templates', 'map existing management capabilities', + 'inspect the existing notification channel' + ]); + const decision = verifier.management_preparation; + assert.strictEqual(decision.decision, 'continue_independent_preparation'); + assert.deepStrictEqual(decision.blocked_work, variant.unavailable_path.blocks); + assert.deepStrictEqual(decision.continue_work, variant.independent_preparation); + assert.deepStrictEqual(decision.not_established, [ + 'continuous collection readiness', 'notification delivery', 'authority for arbitrary changes' + ]); + assert.deepStrictEqual(decision.rejected_inferences, [ + 'unavailable SSH blocks all monitoring preparation', + 'existing management access requires redundant installation', + 'successful management reads prove collection or notification readiness' + ]); +}); + +test('target contract dependencies are reassessed only after explicit scope revision', () => { + const scenario = readFixtureJson('connector-diagnostics-gap/scenario.json'); + const verifier = readFixtureJson('connector-diagnostics-gap/verifier-result.json'); + const trace = readFixtureJson('connector-diagnostics-gap/trace.json'); + const report = readFixtureJson('connector-diagnostics-gap/report.json'); + const playbook = read('examples/evaluator-rag-prototype/connector-diagnostics-gap/candidate-playbook.md'); + const input = scenario.inputs.target_contract; + assert.strictEqual(input.target_sample, 'unread'); + assert.strictEqual(input.mandatory_fields, 'unconfirmed'); + assert.strictEqual(input.development_authorized, true); + assert.deepStrictEqual(input.documented_variants, ['separate payment document', 'document payment section']); + const result = verifier.target_contract; + assert.strictEqual(result.before_stop.document_filling, 'hold_for_target_mapping'); + assert.strictEqual(result.after_stop.candidate_state, 'unfinished_not_promoted'); + assert.strictEqual(result.after_stop.action, 'stop_requested_work'); + assert.strictEqual(result.after_scope_revision.authorization_source, 'explicit_user_scope_revision'); + assert.deepStrictEqual(result.after_scope_revision.continue_work, ['fact storage', 'fact validation']); + assert.strictEqual(result.after_scope_revision.document_filling, 'separate_phase_awaiting_target_contract'); + assert.strictEqual(result.after_scope_revision.erases_earlier_sequencing_error, false); + assert.ok(trace.events.some(event => event.variant === 'target_contract')); + assert.ok(report.evidence_coverage.sources.includes('scenario.json#/inputs/target_contract')); + assert.ok(playbook.includes('General development authorization does not confirm the active target contract')); + assert.ok(playbook.includes('An explicit stop halts the requested work')); +}); + if (failed > 0) { console.log(`\nFailed: ${failed}`); process.exit(1);