maint: integrate reviewed contribution #3749

docs/architecture/evaluator-rag-prototype.md

@@ -123,6 +123,11 @@ Current corpus:
   expected-output comparisons, representative repository/commit histories, and
   regression commands before deep-analysis behavior can be promoted.
 
+- `connector-diagnostics-gap`: inventories an already-used connector, separates
+  observed gaps from unavailable research, and requires evidence before a new
+  background component is proposed. Decisions are illustrative, not executed
+  connector checks.
+
 ## ECC Tools Mapping
 
 ECC Tools already flags missing RAG/evaluator evidence for retrieval,

examples/evaluator-rag-prototype/connector-diagnostics-gap/candidate-playbook.md

@@ -0,0 +1,78 @@
+# Connector Diagnostics Gap Playbook
+
+Candidate id: `bounded-event-summary`
+
+This is a synthetic, read-only worked example for the existing
+[search-first skill](../../../skills/search-first/SKILL.md). The five artifacts
+record expected rubric decisions, not a connector execution or model benchmark.
+The motivating [feedback](https://github.com/affaan-m/ECC/issues/3680) reports a
+corrected workflow; exact ECC activation, version and causal contribution are
+unknown. The fixture does not reconstruct the original connector or deployment.
+
+## Worked Comparison
+
+| Requirement | Inspected fixture evidence | Remaining work |
+| --- | --- | --- |
+| Connection failures | Reachability and last connection error | Reuse existing path |
+| Permission denials | Access diagnostic details | Reuse existing path |
+| Repeated throttling | Individual throttle events | Check retention and aggregate over the interval |
+| Deployed settings | Read unavailable | Keep unknown; do not infer absence |
+
+Accepted draft: “The inspected connector already exposes connection and access
+diagnostics plus throttle events. The missing behavior in the inspected path is
+an interval summary. Check event retention and query support, then reuse that
+query or add a thin summary operation. A separate collector is justified only if
+required data or collection timing cannot be supplied by the existing connector.”
+
+Rejected draft: “Settings search failed, so install a new background monitor for
+all three failure classes.” This duplicates observed capabilities and converts
+an unknown into a negative finding.
+
+## Evaluation Controls
+
+- With sufficient retained events, prefer reuse or a bounded summary extension.
+- If retained events lack the required interval, name that evidence and compare
+  the smallest connector extension with a separate component's added lifecycle.
+- If research is unavailable, preserve the uncertainty and next targeted read;
+  do not claim the feature is missing or demand a new subsystem by default.
+- Do not deploy components or change connector configuration in this example.
+
+## Management Preparation Variant
+
+The [additional field example](https://github.com/affaan-m/ECC/issues/3680#issuecomment-6436640503)
+contributes a synthetic partial-dependency control. Existing MCP/API management
+reads are available, continuous collection is not configured, and server A SSH
+is temporarily unavailable. Defer only host-level checks on server A; continue
+monitoring templates, capability mapping and inspection of the existing
+notification channel before proposing another installation or channel.
+
+Reject “SSH is unavailable, so all monitoring preparation is blocked.” Also
+reject “management reads succeeded, so monitoring and notifications are ready.”
+Management access establishes neither continuous collection readiness nor
+notification delivery or authority for arbitrary changes. These are expected
+fixture decisions, not live checks or a runtime correction of the reported plan.
+
+## Target Contract Variant
+
+The [target-contract report](https://github.com/affaan-m/ECC/issues/3680#issuecomment-6455507941)
+and its [scope-revision follow-up](https://github.com/affaan-m/ECC/issues/3680#issuecomment-6456519404)
+contribute a second synthetic dependency control. Two import variants are
+documented, but the required target sample is unread and mandatory fields are
+unconfirmed. General development authorization does not confirm the active target contract.
+Before dependent document filling, obtain a supported variant/field mapping or
+state the precise dependency hold; preserve independent preparation findings.
+
+An explicit stop halts the requested work and leaves its candidate unfinished,
+without promoting it to accepted state. In a later phase, an explicit user scope
+revision separately authorizes fact storage and validation. Continue that
+independent work; keep future document filling dependent on its target contract.
+Do not let the unconfirmed document contract block the newly authorized scope,
+or treat later code acceptance as erasing the earlier sequencing mistake.
+These are expected fixture decisions, not execution of the reported application,
+a new approval ritual, or evidence of a deployed prevention fix.
+
+## Repository Validation
+
+Run `node tests/docs/evaluator-rag-prototype.test.js`, Markdown lint on touched
+docs, and `git diff --check`. Those checks validate repository documentation;
+they do not execute the connector or establish the historical report's cause.

examples/evaluator-rag-prototype/connector-diagnostics-gap/report.json

@@ -0,0 +1,61 @@
+{
+  "schema_version": "ecc.evaluator-rag.report.v1",
+  "scenario_id": "connector-diagnostics-gap",
+  "run_id": "illustrative-connector-gap",
+  "read_only": true,
+  "result": "illustrative_expected_result",
+  "findings": [
+    {
+      "id": "remaining-gap",
+      "severity": "info",
+      "summary": "Only interval aggregation is absent from the inspected path; retention and deployed settings remain unknown."
+    },
+    {
+      "id": "no-daemon-evidence",
+      "severity": "warning",
+      "summary": "A separate monitor needs evidence that a bounded query or thin extension cannot meet the requirement."
+    },
+    {
+      "id": "partial-management-dependency",
+      "severity": "warning",
+      "summary": "Management-preparation variant separates existing MCP/API access from unconfigured continuous collection: defer only SSH-dependent host checks and continue independent templates, capability mapping and notification-channel inspection. Management reads do not establish collection readiness, notification delivery or change authority."
+    },
+    {
+      "id": "target-contract-dependency",
+      "severity": "warning",
+      "summary": "An unread target sample blocks dependent document filling, not independently authorized fact storage after explicit scope revision. A stopped candidate is not promoted."
+    }
+  ],
+  "recommended_next_action": {
+    "candidate_id": "bounded-event-summary",
+    "action": "Check event retention and query support, then propose the smallest necessary summary extension."
+  },
+  "scoring": {
+    "status": "not_performed",
+    "scores": null,
+    "reason": "Illustrative expected decisions only; no model benchmark or connector checks were executed."
+  },
+  "evidence_coverage": {
+    "kind": "synthetic_fixture_only",
+    "sources": [
+      "scenario.json#/inputs/inspected_capabilities",
+      "scenario.json#/inputs/unavailable_channels",
+      "trace.json#/events",
+      "candidate-playbook.md#worked-comparison",
+      "scenario.json#/inputs/management_preparation",
+      "scenario.json#/inputs/target_contract"
+    ],
+    "covered": [
+      "Connection and permission diagnostics plus per-request throttle events support the bounded-summary decision.",
+      "Unavailable settings are preserved as unknown rather than evidence of absent capability.",
+      "Management-preparation inputs separate blocked host checks from independent preparation and preserve collection, delivery and authority limits.",
+      "Synthetic target-contract sequence preserves the explicit stop, reassesses dependencies after revised authorization, and keeps later acceptance separate from the earlier sequencing mistake."
+    ],
+    "gaps": [
+      "Event retention and deployed settings are unknown.",
+      "The historical incident and ECC causal contribution are unverified; no connector was executed.",
+      "Management access, collection, SSH and notification behavior are synthetic inputs, not live checks.",
+      "No target sample, original candidate code, source-system execution or deployed recovery was inspected."
+    ]
+  }
+}

examples/evaluator-rag-prototype/connector-diagnostics-gap/scenario.json

@@ -0,0 +1,106 @@
+{
+  "schema_version": "ecc.evaluator-rag.scenario.v1",
+  "scenario_id": "connector-diagnostics-gap",
+  "title": "Inspect an existing connector before designing another monitor",
+  "mode": "read_only_prototype",
+  "objective": "Identify the smallest demonstrated diagnostics gap before recommending a new subsystem.",
+  "sources": [
+    {
+      "kind": "repo_doc",
+      "path": "skills/search-first/SKILL.md",
+      "purpose": "Existing research, reuse and minimal-extension guidance"
+    },
+    {
+      "kind": "public_feedback",
+      "url": "https://github.com/affaan-m/ECC/issues/3680",
+      "purpose": "Reported workflow observation; not an ECC reproduction"
+    },
+    {
+      "kind": "public_feedback",
+      "url": "https://github.com/affaan-m/ECC/issues/3680#issuecomment-6436640503",
+      "purpose": "Reported management-transport versus collection example; synthetic variant, not historical reproduction"
+    },
+    {
+      "kind": "public_feedback",
+      "url": "https://github.com/affaan-m/ECC/issues/3680#issuecomment-6455507941",
+      "purpose": "Reported premature target-dependent work and explicit stop; synthetic control only"
+    },
+    {
+      "kind": "public_feedback",
+      "url": "https://github.com/affaan-m/ECC/issues/3680#issuecomment-6456519404",
+      "purpose": "Later explicit scope revision separates fact storage from document filling; synthetic continuation only"
+    }
+  ],
+  "evidence_limits": "The reported connector configuration, original request, ECC activation/version and causal contribution are unverified. All connector capabilities below are synthetic evaluation inputs.",
+  "inputs": {
+    "request": "Explain connection failures, permission denials and repeated throttling using the connector already in use.",
+    "inspected_capabilities": [
+      {
+        "path": "connector/diagnostics/connection",
+        "observed": "Returns endpoint reachability and last connection error"
+      },
+      {
+        "path": "connector/diagnostics/access",
+        "observed": "Returns permission-denial details"
+      },
+      {
+        "path": "connector/diagnostics/requests",
+        "observed": "Returns per-request throttle events, without an aggregate count in the inspected path"
+      }
+    ],
+    "unavailable_channels": [
+      "Historical deployment settings could not be read"
+    ],
+    "remaining_requirement": "A count of throttle events over the requested interval",
+    "management_preparation": {
+      "management_access": "existing_mcp_api_reads",
+      "continuous_collection": "not_configured",
+      "unavailable_path": {
+        "path": "server A SSH",
+        "blocks": [
+          "host-level checks on server A"
+        ]
+      },
+      "independent_preparation": [
+        "prepare monitoring templates",
+        "map existing management capabilities",
+        "inspect the existing notification channel"
+      ]
+    },
+    "target_contract": {
+      "development_authorized": true,
+      "documented_variants": [
+        "separate payment document",
+        "document payment section"
+      ],
+      "target_sample": "unread",
+      "mandatory_fields": "unconfirmed",
+      "independent_work": [
+        "fact storage",
+        "fact validation"
+      ],
+      "scope_sequence": [
+        "development authorized",
+        "explicit stop",
+        "explicit scope revision authorizes fact storage and validation; document filling remains separate"
+      ]
+    }
+  },
+  "retrieval_questions": [
+    "Which actual connector paths were examined?",
+    "What requested behavior is missing from those paths?",
+    "Can the existing events satisfy the gap with a bounded query?",
+    "What evidence would justify a separate background component?"
+  ],
+  "forbidden_actions": [
+    "deploying a background monitor from this evaluator",
+    "treating unavailable research as proof of missing capability",
+    "claiming the reported incident is a reproduced ECC defect"
+  ],
+  "acceptance_gates": [
+    "name examined capability and evidence scope",
+    "state the observed limitation separately from unknown settings",
+    "choose the smallest extension supported by available events",
+    "justify any new background component with a demonstrated unmet requirement"
+  ]
+}

examples/evaluator-rag-prototype/connector-diagnostics-gap/trace.json

@@ -0,0 +1,43 @@
+{
+  "schema_version": "ecc.evaluator-rag.trace.v1",
+  "scenario_id": "connector-diagnostics-gap",
+  "run_id": "illustrative-connector-gap",
+  "read_only": true,
+  "illustrative": true,
+  "events": [
+    {
+      "phase": "observation",
+      "summary": "Synthetic request needs three diagnostics; the connector is already in use."
+    },
+    {
+      "phase": "retrieval",
+      "summary": "The supplied fixture covers connection and permission diagnostics and per-request throttle events; deployment settings remain unknown."
+    },
+    {
+      "phase": "proposal",
+      "candidate_ids": [
+        "bounded-event-summary",
+        "new-monitor-without-inventory"
+      ]
+    },
+    {
+      "phase": "verification",
+      "summary": "Expected rubric comparison only: accept a bounded aggregate over existing events; reject a separate daemon justified only by unavailable settings."
+    },
+    {
+      "phase": "verification",
+      "variant": "management_preparation",
+      "summary": "Expected rubric only: existing MCP/API reads support template/map preparation and inspection of the existing notification channel. Unavailable server A SSH blocks only its host-level checks; collection readiness, delivery and change authority remain unestablished."
+    },
+    {
+      "phase": "verification",
+      "variant": "target_contract",
+      "summary": "Expected rubric only: hold document filling for the target sample and mandatory-field mapping; explicit stop leaves the candidate unfinished. Later explicit scope revision permits independent fact storage and validation while document filling remains separate; it does not erase the earlier sequencing error."
+    },
+    {
+      "phase": "promotion",
+      "promoted_candidate_id": "bounded-event-summary",
+      "summary": "Illustrative playbook selection; no connector, service or settings are changed."
+    }
+  ]
+}

examples/evaluator-rag-prototype/connector-diagnostics-gap/verifier-result.json

@@ -0,0 +1,73 @@
+{
+  "schema_version": "ecc.evaluator-rag.verifier.v1",
+  "scenario_id": "connector-diagnostics-gap",
+  "run_id": "illustrative-connector-gap",
+  "read_only": true,
+  "evaluation_status": "illustrative_expected_decisions_not_executed_connector_checks",
+  "candidates": [
+    {
+      "candidate_id": "bounded-event-summary",
+      "decision": "accepted",
+      "reasons": [
+        "Names all inspected diagnostic paths and their observed coverage",
+        "Uses existing throttle events for the remaining interval-summary requirement",
+        "Keeps retention and deployed settings unknown until checked",
+        "Requires evidence before introducing a new background component"
+      ]
+    },
+    {
+      "candidate_id": "new-monitor-without-inventory",
+      "decision": "rejected",
+      "reasons": [
+        "Duplicates existing connection and access diagnostics",
+        "Infers absent capability from an unavailable settings channel",
+        "Provides no evidence that existing events cannot satisfy the remaining gap"
+      ]
+    }
+  ],
+  "promoted_candidate_id": "bounded-event-summary",
+  "rollback": {
+    "current_run": "Not applicable: No connector state is changed by this read-only illustrative selection.",
+    "future_extension": "If later implemented, revert the summary extension and its query/configuration changes to restore the existing diagnostics paths; verify connection, access and per-request event behavior before redeployment."
+  },
+  "management_preparation": {
+    "decision": "continue_independent_preparation",
+    "blocked_work": [
+      "host-level checks on server A"
+    ],
+    "continue_work": [
+      "prepare monitoring templates",
+      "map existing management capabilities",
+      "inspect the existing notification channel"
+    ],
+    "not_established": [
+      "continuous collection readiness",
+      "notification delivery",
+      "authority for arbitrary changes"
+    ],
+    "rejected_inferences": [
+      "unavailable SSH blocks all monitoring preparation",
+      "existing management access requires redundant installation",
+      "successful management reads prove collection or notification readiness"
+    ]
+  },
+  "target_contract": {
+    "before_stop": {
+      "document_filling": "hold_for_target_mapping",
+      "reason": "General documentation does not identify the active variant and mandatory fields; preserve independent findings."
+    },
+    "after_stop": {
+      "action": "stop_requested_work",
+      "candidate_state": "unfinished_not_promoted"
+    },
+    "after_scope_revision": {
+      "authorization_source": "explicit_user_scope_revision",
+      "continue_work": [
+        "fact storage",
+        "fact validation"
+      ],
+      "document_filling": "separate_phase_awaiting_target_contract",
+      "erases_earlier_sequencing_error": false
+    }
+  }
+}

tests/docs/evaluator-rag-prototype.test.js

@@ -409,6 +409,117 @@ test('deep analyzer evidence scenario rejects no-corpus analyzer changes', () =>
   assert.ok(playbook.includes('Deep Analyzer Evidence'));
 });
 
+test('connector gap artifacts share identity and preserve read-only scope', () => {
+  const scenario = readFixtureJson('connector-diagnostics-gap/scenario.json');
+  assert.strictEqual(scenario.scenario_id, 'connector-diagnostics-gap');
+  assert.strictEqual(scenario.mode, 'read_only_prototype');
+  for (const [file, schema] of [
+    ['trace.json', 'trace'], ['report.json', 'report'], ['verifier-result.json', 'verifier']
+  ]) {
+    const artifact = readFixtureJson(`connector-diagnostics-gap/${file}`);
+    assert.strictEqual(artifact.schema_version, `ecc.evaluator-rag.${schema}.v1`);
+    assert.strictEqual(artifact.scenario_id, scenario.scenario_id);
+    assert.strictEqual(artifact.run_id, 'illustrative-connector-gap');
+    assert.strictEqual(artifact.read_only, true);
+  }
+  assert.strictEqual(scenario.schema_version, 'ecc.evaluator-rag.scenario.v1');
+});
+
+test('connector gap decisions reject unsupported monitors and preserve uncertainty', () => {
+  const scenario = readFixtureJson('connector-diagnostics-gap/scenario.json');
+  const trace = readFixtureJson('connector-diagnostics-gap/trace.json');
+  const verifier = readFixtureJson('connector-diagnostics-gap/verifier-result.json');
+  const playbook = read('examples/evaluator-rag-prototype/connector-diagnostics-gap/candidate-playbook.md');
+  const accepted = verifier.candidates.find(candidate => candidate.candidate_id === 'bounded-event-summary');
+  const rejected = verifier.candidates.find(candidate => candidate.candidate_id === 'new-monitor-without-inventory');
+  assert.ok(accepted, 'Missing bounded summary candidate');
+  assert.ok(rejected, 'Missing unsupported monitor candidate');
+  assert.strictEqual(accepted.decision, 'accepted');
+  assert.strictEqual(rejected.decision, 'rejected');
+  assert.strictEqual(verifier.promoted_candidate_id, accepted.candidate_id);
+  assert.ok(trace.events.some(event => event.promoted_candidate_id === accepted.candidate_id));
+  assert.strictEqual(trace.illustrative, true);
+  assert.strictEqual(verifier.evaluation_status, 'illustrative_expected_decisions_not_executed_connector_checks');
+  assert.ok(accepted.reasons.includes('Keeps retention and deployed settings unknown until checked'));
+  assert.ok(rejected.reasons.includes('Infers absent capability from an unavailable settings channel'));
+  assert.ok(scenario.forbidden_actions.includes('treating unavailable research as proof of missing capability'));
+  assert.ok(scenario.forbidden_actions.includes('claiming the reported incident is a reproduced ECC defect'));
+  assert.ok(scenario.forbidden_actions.includes('deploying a background monitor from this evaluator'));
+  assert.ok(scenario.acceptance_gates.includes('state the observed limitation separately from unknown settings'));
+  assert.ok(playbook.includes('If research is unavailable, preserve the uncertainty and next targeted read'));
+  assert.ok(playbook.includes('Do not deploy components or change connector configuration'));
+});
+
+test('connector report records unscored fixture coverage and verifier rollback', () => {
+  const report = readFixtureJson('connector-diagnostics-gap/report.json');
+  const verifier = readFixtureJson('connector-diagnostics-gap/verifier-result.json');
+  assert.strictEqual(report.scoring.status, 'not_performed');
+  assert.strictEqual(report.scoring.scores, null);
+  assert.strictEqual(report.evidence_coverage.kind, 'synthetic_fixture_only');
+  assert.deepStrictEqual(report.evidence_coverage.sources, [
+    'scenario.json#/inputs/inspected_capabilities',
+    'scenario.json#/inputs/unavailable_channels',
+    'trace.json#/events',
+    'candidate-playbook.md#worked-comparison',
+    'scenario.json#/inputs/management_preparation',
+    'scenario.json#/inputs/target_contract'
+  ]);
+  assert.ok(report.evidence_coverage.gaps.includes('Event retention and deployed settings are unknown.'));
+  assert.strictEqual(report.recommended_next_action.candidate_id, verifier.promoted_candidate_id);
+  assert.ok(verifier.rollback.current_run.includes('No connector state is changed'));
+  assert.ok(verifier.rollback.future_extension.includes('revert the summary extension'));
+});
+
+test('connector preparation scopes unavailable SSH to dependent host checks', () => {
+  const scenario = readFixtureJson('connector-diagnostics-gap/scenario.json');
+  const verifier = readFixtureJson('connector-diagnostics-gap/verifier-result.json');
+  const variant = scenario.inputs.management_preparation;
+  assert.strictEqual(variant.management_access, 'existing_mcp_api_reads');
+  assert.strictEqual(variant.continuous_collection, 'not_configured');
+  assert.deepStrictEqual(variant.unavailable_path.blocks, ['host-level checks on server A']);
+  assert.deepStrictEqual(variant.independent_preparation, [
+    'prepare monitoring templates', 'map existing management capabilities',
+    'inspect the existing notification channel'
+  ]);
+  const decision = verifier.management_preparation;
+  assert.strictEqual(decision.decision, 'continue_independent_preparation');
+  assert.deepStrictEqual(decision.blocked_work, variant.unavailable_path.blocks);
+  assert.deepStrictEqual(decision.continue_work, variant.independent_preparation);
+  assert.deepStrictEqual(decision.not_established, [
+    'continuous collection readiness', 'notification delivery', 'authority for arbitrary changes'
+  ]);
+  assert.deepStrictEqual(decision.rejected_inferences, [
+    'unavailable SSH blocks all monitoring preparation',
+    'existing management access requires redundant installation',
+    'successful management reads prove collection or notification readiness'
+  ]);
+});
+
+test('target contract dependencies are reassessed only after explicit scope revision', () => {
+  const scenario = readFixtureJson('connector-diagnostics-gap/scenario.json');
+  const verifier = readFixtureJson('connector-diagnostics-gap/verifier-result.json');
+  const trace = readFixtureJson('connector-diagnostics-gap/trace.json');
+  const report = readFixtureJson('connector-diagnostics-gap/report.json');
+  const playbook = read('examples/evaluator-rag-prototype/connector-diagnostics-gap/candidate-playbook.md');
+  const input = scenario.inputs.target_contract;
+  assert.strictEqual(input.target_sample, 'unread');
+  assert.strictEqual(input.mandatory_fields, 'unconfirmed');
+  assert.strictEqual(input.development_authorized, true);
+  assert.deepStrictEqual(input.documented_variants, ['separate payment document', 'document payment section']);
+  const result = verifier.target_contract;
+  assert.strictEqual(result.before_stop.document_filling, 'hold_for_target_mapping');
+  assert.strictEqual(result.after_stop.candidate_state, 'unfinished_not_promoted');
+  assert.strictEqual(result.after_stop.action, 'stop_requested_work');
+  assert.strictEqual(result.after_scope_revision.authorization_source, 'explicit_user_scope_revision');
+  assert.deepStrictEqual(result.after_scope_revision.continue_work, ['fact storage', 'fact validation']);
+  assert.strictEqual(result.after_scope_revision.document_filling, 'separate_phase_awaiting_target_contract');
+  assert.strictEqual(result.after_scope_revision.erases_earlier_sequencing_error, false);
+  assert.ok(trace.events.some(event => event.variant === 'target_contract'));
+  assert.ok(report.evidence_coverage.sources.includes('scenario.json#/inputs/target_contract'));
+  assert.ok(playbook.includes('General development authorization does not confirm the active target contract'));
+  assert.ok(playbook.includes('An explicit stop halts the requested work'));
+});
+
 if (failed > 0) {
   console.log(`\nFailed: ${failed}`);
   process.exit(1);