diff --git a/AGENTS.md b/AGENTS.md index 5585ecadd8..d60f2b07f6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,8 +21,8 @@ REA is a layered ESM TypeScript application. Dependencies flow inward from pure - `src/domain/` owns pure provider-neutral semantics; `src/contracts/` owns caller-visible schemas and the canonical tool inventory. - `src/hopper/`, `src/ghidra/`, `src/ida/`, `src/browser/`, `src/inspector/`, `src/native/`, `src/artifacts/`, and `src/dotnet/` own provider-specific boundaries. Keep provider protocols out of domain and application code. -- `src/application/` composes shared CLI/MCP workflows; `src/server/` translates MCP requests; `src/cli.ts` and `src/main.ts` are the CLI and MCP entry points. -- `src/process/` owns shared process lifecycle primitives, not provider wire protocols. `bridge/` contains provider-side adapters. +- `src/application/` composes shared CLI/MCP workflows, including Evidence provenance, unknowns, and eligible snapshot bindings; `src/server/` translates MCP requests; `src/cli.ts` and `src/main.ts` are the CLI and MCP entry points. +- `src/process/` owns shared process lifecycle primitives, not provider wire protocols. Reuse its supervision and identity primitives before adding provider-local lifecycle code; a PID and executable pathname alone do not establish ownership after exit or reuse. `bridge/` contains provider-side adapters. - `tests/` contains unit, composition, boundary, acceptance, and conformance tests. `scripts/verify-*` and capability directories under `scripts/verify/` contain real-toolchain checks. - `docs/public/product-catalog.json`, `docs/verification/managed-conformance-*.json`, and `skills/` are ignored build outputs. Update source contracts and authored instructions in `skill-src/`, then run `npm run build:cached`; never commit derived catalog digests or portable conformance projections. - `src/generatedMcpToolCatalog.ts` is build-generated and gitignored. Never commit it; resolve any trace of it in merges by deleting it and running `npm run build:cached`. @@ -56,9 +56,9 @@ Treat a boundary as a contract between the producer's actual representation and When changing boundary behavior, inspect adjacent input representations, failure paths, and affected callers, and correct the underlying assumption across those cases. -Keep portable evidence and scenario validation distinct from host-native execution checks. Absolute filesystem paths, file URLs, and HTTP paths have different semantics; do not substitute one platform's syntax for the domain concept. Interpret provider metadata according to its documented or observed producer behavior. When a transformation loses information, preserve the reported value and an explicit unknown rather than guessing a canonical identity. +Keep portable evidence and scenario validation distinct from host-native execution checks. Absolute filesystem paths, file URLs, and HTTP paths have different semantics; do not substitute one platform's syntax for the domain concept. Interpret provider metadata according to its documented or observed producer behavior. Keep values used for identity, provenance, matching, and path resolution separate from display formatting. When normalization loses information, preserve the source value and an explicit unknown; display placeholders must not feed back into lookup or selection. -Preserve meaningful failure reasons through application and adapter layers. Malformed input is distinct from an unsupported target, unavailable provider, or host operating-system permission denial. Diagnostics should identify the failed constraint and the target or lifecycle request it applies to. Recovery advice must address that reason and point to an available workflow; generic catches must not erase actionable validation details. +Preserve meaningful failure reasons through application and adapter layers. Malformed input is distinct from an unsupported target, unavailable provider, or host operating-system permission denial. Diagnostics should identify the failed constraint and the target or lifecycle request it applies to. Recovery advice must address that reason and point to an available workflow; generic catches must not erase actionable validation details. Preserve collected observations on execution failure independently of whether cleanup succeeds. Leave meaningful target, action, capture, and output choices to the agent. A selected operation already expresses intent; do not require approval booleans or repeated permission declarations. Trace each setting to its consumer: remove ignored options and single-value confirmations, derive built-in lifecycle behavior, and supply defaults for omitted optional metadata. Report actual effects and limitations where they help interpret results rather than asking callers to restate them. @@ -82,7 +82,7 @@ Let agents compose experiments with ordinary commands, scripts, and local fixtur - Use **observe/capture** only when runtime activity is required, and declare authority and lifecycle effects in the contract. - Extend an existing tool when intent and result contract are unchanged. Add a tool for a distinct analyst outcome or materially different authority. - Keep caller-facing names and results provider-neutral. Put engine-specific behavior in provider adapters and report each provider's exact coverage. -- Keep results complete by default. Add a limit only when it follows from a real format, protocol, authority, or resource-safety constraint; explain truncation and unsupported facets. Keep observed, derived, inferred, and unknown results distinct. +- Keep results complete by default. Add a limit only when it follows from a real format, protocol, authority, or resource-safety constraint; explain truncation and unsupported facets. Account for representation expansion and products of independently bounded dimensions before allocating or retaining output. Derive facet completeness from what was examined and exhausted, not an empty failure list. Keep observed, derived, inferred, and unknown results distinct. - Include artifact identity, source locations, Evidence references, actionable errors, and relevant limitations when they affect conclusions. Return the evidence needed for the next analysis inline rather than requiring a resource or identifier lookup. - Implement shared application workflows behind CLI and MCP adapters. Update canonical contracts and generated catalog together. diff --git a/docs/testing.md b/docs/testing.md index 7abebdae83..bbd5df1fa6 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -425,7 +425,7 @@ behavior, and complete CI evidence before merging. | `npm run test:watch:all` | Changed tests from every project; builds at startup, so rebuild after production edits before relying on compiled tests | | `npm run check:changed` | Cached typecheck/lint and branch-related source feedback | | `npm run check:pr` | Opt-in complete local deterministic gate and generated-file checks | -| `npm run docs:check` | Committed generated-document freshness | +| `npm run docs:check` | Generated-document validation from current source and build outputs | For example: @@ -460,19 +460,21 @@ broad changes or diagnosing CI, rather than after every edit. Package/install changes additionally need package verification; provider changes need actual provider evidence. -Local full-suite Vitest runs use up to two workers and schedule projects one -at a time. Process, acceptance and process-global projects serialize their -files to prevent competing lifecycle observations. CI retains its two-worker -budget. +Each Vitest project uses up to two workers, bounded by available host parallelism. +Default-group projects can run together. Only `process-boundary` runs in the +later sequence group and serializes its files because process-tree observations +share host sampling resources. Acceptance and process-global files retain isolated +forks but are not serialized; isolation does not imply serial scheduling. CI uses +the same per-project worker bound. The pure domain/contracts and recording-port service projects share one worker module context because their tests own no mutable runtime resources. MCP boundary files also share the immutable server module graph while creating and closing independent in-memory sessions. Adapter, composition, acceptance, process-global, and other boundary projects retain per-file isolation. -`npm test`, `npm run docs:check`, and `npm run docs:generate` share -repository-local locks and fail fast when the same class of command is already -running. The `npm test` build is inside that lock. `check:pr` runs its test task -before starting generated-document validation. +Build and documentation writers use checkout-local locks for their respective +output files. `npm test` delegates its build to the build lock; the Vitest suite +does not hold a test-command lock. `check:pr` runs its test task before starting +generated-document validation. Vitest and Node persistent compile caches are deliberately not enabled by default. To evaluate repeated local runs, opt in for both cold and warm diff --git a/docs/tool-design.md b/docs/tool-design.md index 965e4243cb..4a5c2fab65 100644 --- a/docs/tool-design.md +++ b/docs/tool-design.md @@ -80,7 +80,18 @@ or model-authored loop when one call can answer the question. ## Implement and evaluate - Put shared user workflows in the application layer; keep MCP translation in - the server adapter and CLI behavior aligned with the same workflow. + the server adapter and CLI behavior aligned with the same workflow. Share + Evidence provenance, unknown projection, and eligible snapshot binding through + application/session owners. Matching result payloads alone do not establish + equivalent retention or replay behavior. +- Derive each facet's coverage from examined records, unsupported representations, + and exhaustiveness. Keep issues structured until presentation; do not select a + facet by matching diagnostic prose or infer completeness from an empty failure + list. Preserve available observations independently of cleanup success. +- Measure the resource consumed before choosing a limit. Value alternatives, + retained records, trace-frame products, output bytes, and temporary-file growth + need different accounting. Enforce a justified budget before the costly step + and report its effect; an input-byte or graph-node cap does not bound every stage. - Update the canonical contract inventory, output schemas, examples, generated catalog, and relevant docs together. - Verify valid, malformed, boundary, unavailable, cancellation, and partial diff --git a/package.json b/package.json index e855494980..30c592d1a7 100644 --- a/package.json +++ b/package.json @@ -186,7 +186,7 @@ "docs:build": "vitepress build docs", "docs:preview": "vitepress preview docs", "docs:generate": "node scripts/run-exclusive.mjs docs npm run docs:generate:unlocked", - "docs:generate:unlocked": "npm run metadata:generate && npm run build:cached && node scripts/generate-skill-metadata.mjs && node scripts/generate-product-catalog.mjs && node scripts/generate-completion-ledger.mjs && node scripts/generate-error-schema.mjs", + "docs:generate:unlocked": "npm run metadata:generate && npm run build:cached && node scripts/generate-error-schema.mjs", "docs:check": "node scripts/run-exclusive.mjs docs turbo run docs:check:built", "docs:check:built": "node scripts/generate-package-metadata.mjs --check && node scripts/generate-mcp-tool-catalog.mjs --check && node scripts/generate-skill-metadata.mjs --check && node scripts/generate-product-catalog.mjs --check && node scripts/generate-completion-ledger.mjs --check && node scripts/check-doc-facts.mjs && node scripts/generate-error-schema.mjs --check", "docs:error-schema": "npm run build:cached && node scripts/generate-error-schema.mjs", diff --git a/src/application/DirectAnalysis.ts b/src/application/DirectAnalysis.ts index ea7ed8f6cd..a798a835b5 100644 --- a/src/application/DirectAnalysis.ts +++ b/src/application/DirectAnalysis.ts @@ -30,11 +30,7 @@ import { snapshotEvidenceForQuery, snapshotMatchesTarget, } from "../domain/analysisSnapshot.js"; -import { - analysisProfileSchema, - committedProviderSchema, - type AnalysisProfileCommitment, -} from "../domain/analysisProfile.js"; +import type { AnalysisProfileCommitment } from "../domain/analysisProfile.js"; import { err, ok, type Result } from "../domain/result.js"; import type { AnalysisSnapshot } from "../domain/analysisSnapshot.js"; import type { @@ -45,6 +41,10 @@ import { REA_WORKFLOW_PROVIDER, workflowAnalysisProfile, } from "./InvestigationProviders.js"; +import { + createWorkflowEvidence, + workflowSnapshotRecord, +} from "./WorkflowEvidence.js"; import type { AnalysisProviderSelector } from "../contracts/providerSelection.js"; import { artifactInspectionResultSchema } from "../domain/artifactInspection.js"; @@ -268,35 +268,13 @@ const runAnalysis = async ( tool !== "trace_native_ui_action" && snapshotPath !== undefined && evidence !== undefined && - "analysis_profile" in evidence && session.allowsSnapshotReplay(tool) ) { - const recorded = session.recordWorkflowSnapshot({ - operation: tool, - parameters: arguments_, - execution: { - result: evidence.normalized_result, - rawResult: evidence.raw_result, - provider: committedProviderSchema.parse(evidence.provider), - analysisProfile: analysisProfileSchema.parse( - evidence.analysis_profile, - ), - limitations: evidence.limitations, - locations: evidence.locations, - subject: - evidence.subject === null - ? null - : { - path: evidence.subject.local_path, - sha256: evidence.subject.digest.sha256, - format: evidence.subject.format, - ...(evidence.subject.architecture === null - ? {} - : { architecture: evidence.subject.architecture }), - }, - }, - }); - if (!recorded.ok) return cliError(recorded.error); + const workflowRecord = workflowSnapshotRecord(evidence, tool); + if (workflowRecord !== undefined) { + const recorded = session.recordWorkflowSnapshot(workflowRecord); + if (!recorded.ok) return cliError(recorded.error); + } } if ( tool !== "trace_native_ui_action" && @@ -356,15 +334,12 @@ const executeAnalysisTool = async (input: { signal, ); if (!result.ok) return { output: cliError(result.error) }; - const evidence = createEvidence(input.openedTarget, REA_WORKFLOW_PROVIDER, { + const evidence = createWorkflowEvidence({ + target: input.openedTarget, operation: tool, parameters: input.arguments, result: result.value, - ...(evidenceProfile === undefined - ? {} - : { analysisProfile: evidenceProfile }), - confidence: "derived", - limitations: ["Derived by an REA composed workflow."], + upstreamProfile: evidenceProfile, }); return { output: evidence, evidence }; } @@ -493,8 +468,7 @@ const analysisProfileForEvidence = ( | DirectAnalysisTool, ): AnalysisProfileCommitment | undefined => { if (!isWorkflowEvidenceTool(tool)) return session.analysisProfile(tool); - const upstream = session.analysisProfile(); - return upstream === undefined ? undefined : workflowAnalysisProfile(upstream); + return session.analysisProfile(); }; const isWorkflowEvidenceTool = ( diff --git a/src/application/WorkflowEvidence.ts b/src/application/WorkflowEvidence.ts new file mode 100644 index 0000000000..7944206a93 --- /dev/null +++ b/src/application/WorkflowEvidence.ts @@ -0,0 +1,64 @@ +import { + REA_WORKFLOW_PROVIDER, + workflowAnalysisProfile, +} from "./InvestigationProviders.js"; +import type { AnalysisOperation } from "./AnalysisProvider.js"; +import type { AnalysisProfileCommitment } from "../domain/analysisProfile.js"; +import type { WorkflowSnapshotRecordInput } from "./binary/BinarySessionRecords.js"; +import { + analysisProfileSchema, + committedProviderSchema, +} from "../domain/analysisProfile.js"; +import type { BinaryTarget } from "../domain/binaryTarget.js"; +import { createEvidence, type Evidence } from "../domain/evidence.js"; +import type { JsonValue } from "../domain/jsonValue.js"; + +/** Build the shared Evidence record for a successful composed binary workflow. */ +export const createWorkflowEvidence = (input: { + readonly target: BinaryTarget | undefined; + readonly operation: AnalysisOperation; + readonly parameters: Readonly>; + readonly result: JsonValue; + readonly upstreamProfile: AnalysisProfileCommitment | undefined; +}): Evidence => + createEvidence(input.target, REA_WORKFLOW_PROVIDER, { + operation: input.operation, + parameters: input.parameters, + result: input.result, + ...(input.upstreamProfile === undefined + ? {} + : { analysisProfile: workflowAnalysisProfile(input.upstreamProfile) }), + confidence: "derived", + limitations: ["Derived by an REA composed workflow."], + }); + +/** Convert workflow Evidence into its exact replay binding, when it has a profile. */ +export const workflowSnapshotRecord = ( + evidence: Evidence, + operation: AnalysisOperation, +): WorkflowSnapshotRecordInput | undefined => { + if (!("analysis_profile" in evidence)) return undefined; + return { + operation, + parameters: evidence.parameters, + execution: { + result: evidence.normalized_result, + rawResult: evidence.raw_result, + provider: committedProviderSchema.parse(evidence.provider), + analysisProfile: analysisProfileSchema.parse(evidence.analysis_profile), + limitations: evidence.limitations, + locations: evidence.locations, + subject: + evidence.subject === null + ? null + : { + path: evidence.subject.local_path, + sha256: evidence.subject.digest.sha256, + format: evidence.subject.format, + ...(evidence.subject.architecture === null + ? {} + : { architecture: evidence.subject.architecture }), + }, + }, + }; +}; diff --git a/src/application/binary/BinarySessionPort.ts b/src/application/binary/BinarySessionPort.ts index a8bd37a043..fb7317f000 100644 --- a/src/application/binary/BinarySessionPort.ts +++ b/src/application/binary/BinarySessionPort.ts @@ -7,6 +7,8 @@ import type { AnalysisProfileCommitment } from "../../domain/analysisProfile.js" import type { AnalysisProviderSelector } from "../../contracts/providerSelection.js"; import type { AnalysisError } from "../../domain/analysisErrorBase.js"; import type { Result } from "../../domain/result.js"; +import type { EvidenceIntegrityError } from "../../domain/evidenceErrors.js"; +import type { WorkflowSnapshotRecordInput } from "./BinarySessionRecords.js"; import type { AnalysisOperation, @@ -43,6 +45,10 @@ export interface BinarySessionPort analysisProfile( operation?: AnalysisOperation, ): AnalysisProfileCommitment | undefined; + allowsSnapshotReplay(operation: AnalysisOperation): boolean; + recordWorkflowSnapshot( + input: WorkflowSnapshotRecordInput, + ): Result; openCompatibility(): Readonly>; onAvailabilityChanged?(listener: () => void | Promise): () => void; onAnalysisSnapshotChanged?(listener: () => void | Promise): () => void; diff --git a/src/application/binary/BinarySessionRecords.ts b/src/application/binary/BinarySessionRecords.ts index ee97d88492..581f45b846 100644 --- a/src/application/binary/BinarySessionRecords.ts +++ b/src/application/binary/BinarySessionRecords.ts @@ -3,6 +3,7 @@ import type { AnalysisSnapshot } from "../../domain/analysisSnapshot.js"; import type { BinaryTarget } from "../../domain/binaryTarget.js"; import type { Evidence } from "../../domain/evidence.js"; import type { EvidenceBundle } from "../../domain/evidenceBundle.js"; +import type { JsonValue } from "../../domain/jsonValue.js"; import { evidenceBundleForTarget } from "../../domain/evidenceBundle.js"; import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js"; import { type AnalysisError } from "../../domain/analysisErrorBase.js"; @@ -26,6 +27,15 @@ export interface ActiveAnalysisBinding { readonly profile: AnalysisProfileCommitment | null; } +/** Validated composed-workflow payload bound to the active target and profile. */ +export interface WorkflowSnapshotRecordInput { + readonly operation: AnalysisOperation; + readonly parameters: Readonly>; + readonly execution: Parameters< + AnalysisSnapshotCache["recordWorkflow"] + >[0]["execution"]; +} + /** Binary snapshot owner and compatibility facade for composed investigation records. */ export abstract class BinarySessionRecords { readonly #records: InvestigationRecords; @@ -161,15 +171,9 @@ export abstract class BinarySessionRecords { } /** Retain one derived workflow result alongside its provider cache entries. */ - recordWorkflowSnapshot(input: { - readonly operation: string; - readonly parameters: Readonly< - Record - >; - readonly execution: Parameters< - AnalysisSnapshotCache["recordWorkflow"] - >[0]["execution"]; - }): Result { + recordWorkflowSnapshot( + input: WorkflowSnapshotRecordInput, + ): Result { const active = this.activeAnalysisBinding(); if (active === undefined || active.profile === null) return err( diff --git a/src/domain/analysisErrorProjection.ts b/src/domain/analysisErrorProjection.ts index bd8e7e9c9b..3025dbb474 100644 --- a/src/domain/analysisErrorProjection.ts +++ b/src/domain/analysisErrorProjection.ts @@ -393,16 +393,27 @@ const lifecycleErrorDetails = ( ), }), }; - if ( - error._tag === "ProcessCaptureError" && - error.userCategory === "cancelled" - ) - return { operation: "process_capture", cleanup: "complete" }; - if ( - error._tag === "ProcessCaptureError" && - error.executionFailure !== undefined - ) - return { execution_failure: error.executionFailure }; + if (error._tag === "ProcessCaptureError") { + const details = { + ...(error.userCategory === "cancelled" + ? { operation: "process_capture", cleanup: "complete" } + : {}), + ...(error.cleanupReport === undefined + ? {} + : { cleanup_report: jsonValueSchema.parse(error.cleanupReport) }), + ...(error.executionFailure === undefined + ? {} + : { execution_failure: error.executionFailure }), + ...(error.partialObservation === undefined + ? {} + : { + partial_observation: jsonValueSchema.parse( + error.partialObservation, + ), + }), + }; + return Object.keys(details).length === 0 ? undefined : details; + } if (error instanceof BinaryTargetError) return { path: error.path, diff --git a/src/domain/process/processCapture.ts b/src/domain/process/processCapture.ts index 283706b125..9739cba666 100644 --- a/src/domain/process/processCapture.ts +++ b/src/domain/process/processCapture.ts @@ -578,7 +578,7 @@ const incompleteCaptureObservationSchema = z.strictObject({ observations: incompleteProcessCaptureObservationsSchema, }); -/** Validated, non-comparable process observations attached to cleanup errors. */ +/** Validated, non-comparable process observations attached to execution or cleanup errors. */ export const partialProcessCaptureObservationSchema = z .union([ completedCaptureObservationSchema.extend( @@ -589,15 +589,15 @@ export const partialProcessCaptureObservationSchema = z ), ]) .superRefine((partial, context) => { - if ( - !Object.values(partial.cleanup).some( - ({ state }) => state === "failed" || state === "unverified", - ) - ) + const cleanupIncomplete = Object.values(partial.cleanup).some( + ({ state }) => state === "failed" || state === "unverified", + ); + if (!cleanupIncomplete && partial.execution_failure === null) context.addIssue({ code: "custom", - path: ["cleanup"], - message: "partial observations require incomplete resource cleanup", + path: ["execution_failure"], + message: + "partial observations require incomplete cleanup or an execution failure", }); if ("observations" in partial) { diff --git a/src/process/capture/ProcessCaptureLifecycle.ts b/src/process/capture/ProcessCaptureLifecycle.ts index 50f9d8e1d5..d0b22abc9b 100644 --- a/src/process/capture/ProcessCaptureLifecycle.ts +++ b/src/process/capture/ProcessCaptureLifecycle.ts @@ -846,6 +846,52 @@ export const resolveProcessResult = ( }, ): ProcessCapture => { const cleanupFailure = cleanupReportFailure(cleanup); + const executionFailureReason = + describeProcessCaptureExecutionFailure(executionFailure); + let partialObservation: PartialProcessCaptureObservation | undefined; + if ( + (cleanupFailure !== undefined || executionFailure !== undefined) && + (capture !== undefined || observations !== undefined) + ) { + const normalizedObservations = + observations === undefined || partialContext === undefined + ? observations + : normalizePartialProcessObservations(observations, partialContext); + if (capture === undefined) { + if (normalizedObservations !== undefined) { + const partialResult = partialProcessCaptureObservationSchema.safeParse({ + observations: normalizedObservations, + cleanup, + execution_failure: executionFailureReason ?? null, + }); + if (partialResult.success) partialObservation = partialResult.data; + } + } else { + const captureData = + "cleanup" in capture + ? (() => { + const { cleanup: _cleanup, ...data } = capture; + void _cleanup; + return data; + })() + : capture; + const settlement = + cleanupFailure !== undefined + ? { ...capture.settlement, cleanup_outcome: "failed" as const } + : capture.settlement.state === "quiesced" + ? { + ...capture.settlement, + cleanup_outcome: "not_required" as const, + } + : { ...capture.settlement, cleanup_outcome: "cleaned" as const }; + const partialResult = partialProcessCaptureObservationSchema.safeParse({ + capture: { ...captureData, settlement }, + cleanup, + execution_failure: executionFailureReason ?? null, + }); + if (partialResult.success) partialObservation = partialResult.data; + } + } if (cleanupFailure !== undefined) { const cleanupResources = Object.entries(cleanup) .filter( @@ -853,44 +899,6 @@ export const resolveProcessResult = ( outcome.state === "failed" || outcome.state === "unverified", ) .map(([resource]) => resource); - const executionFailureReason = - describeProcessCaptureExecutionFailure(executionFailure); - let partialObservation: PartialProcessCaptureObservation | undefined; - if (capture !== undefined || observations !== undefined) { - const normalizedObservations = - observations === undefined || partialContext === undefined - ? observations - : normalizePartialProcessObservations(observations, partialContext); - const partialPayload = - capture === undefined - ? { observations: normalizedObservations } - : { capture }; - const pendingCapture = - capture === undefined - ? undefined - : { - ...capture, - settlement: { - ...capture.settlement, - cleanup_outcome: "failed" as const, - }, - }; - const partialResult = partialProcessCaptureObservationSchema.safeParse({ - ...partialPayload, - ...(pendingCapture === undefined - ? {} - : { - capture: Object.fromEntries( - Object.entries(pendingCapture).filter( - ([key]) => key !== "cleanup", - ), - ), - }), - cleanup, - execution_failure: executionFailureReason ?? null, - }); - if (partialResult.success) partialObservation = partialResult.data; - } throw new ProcessCaptureError(cleanupFailure, { cause: executionFailure, reason: "cleanup_incomplete", @@ -902,15 +910,39 @@ export const resolveProcessResult = ( cleanupReport: cleanup, }); } - if (executionFailure instanceof ProcessCaptureError) throw executionFailure; + if (executionFailure instanceof ProcessCaptureError) { + const retainedExecutionFailure = + executionFailure.executionFailure ?? executionFailureReason; + const retainedPartialObservation = + executionFailure.partialObservation ?? partialObservation; + throw new ProcessCaptureError(executionFailure.message, { + cause: executionFailure, + ...(executionFailure.userMessage === undefined + ? {} + : { userMessage: executionFailure.userMessage }), + ...(executionFailure.userCategory === undefined + ? {} + : { userCategory: executionFailure.userCategory }), + reason: executionFailure.reason, + cleanupResources: executionFailure.cleanupResources, + ...(retainedExecutionFailure === undefined + ? {} + : { executionFailure: retainedExecutionFailure }), + ...(retainedPartialObservation === undefined + ? {} + : { partialObservation: retainedPartialObservation }), + cleanupReport: executionFailure.cleanupReport ?? cleanup, + }); + } if (executionFailure !== undefined) { - const executionFailureReason = - describeProcessCaptureExecutionFailure(executionFailure); throw new ProcessCaptureError("process capture failed", { cause: executionFailure, + reason: "capture_failed", ...(executionFailureReason === undefined ? {} : { executionFailure: executionFailureReason }), + ...(partialObservation === undefined ? {} : { partialObservation }), + cleanupReport: cleanup, }); } if (capture === undefined) diff --git a/src/process/capture/ProcessHarness.test.ts b/src/process/capture/ProcessHarness.test.ts index 961276809a..78a3c48ee0 100644 --- a/src/process/capture/ProcessHarness.test.ts +++ b/src/process/capture/ProcessHarness.test.ts @@ -16,6 +16,7 @@ import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "../../domain/process/processCaptu import { projectAnalysisError } from "../../domain/analysisErrorProjection.js"; import { emptyProcessCapture } from "../../domain/process/processCapture.fixture.js"; import { analysisErrorProjectionSchema } from "../../contracts/errorSchemas.js"; +import { processCaptureCancelled } from "./ProcessCaptureError.js"; import { releaseProcessResources, resolveProcessResult, @@ -352,8 +353,20 @@ it("retains observations and both causes when process cleanup is unverifiable", terminal_renderer: { state: "cleaned", reason: null }, temporary_root: { state: "cleaned", reason: null }, }, + execution_failure: null, }).success, ).toBe(false); + expect( + partialProcessCaptureObservationSchema.safeParse({ + ...partialObservation, + cleanup: { + owned_process_group: { state: "cleaned", reason: null }, + terminal_renderer: { state: "cleaned", reason: null }, + temporary_root: { state: "cleaned", reason: null }, + }, + execution_failure: "capture ended after a fixture error", + }).success, + ).toBe(true); if (!("capture" in partialObservation)) throw new Error("expected completed partial capture observations"); const partialCapture = partialObservation.capture; @@ -392,6 +405,95 @@ it("projects execution and cleanup failures when capture never completed", () => }); }); +it("retains completed observations when finalization fails after clean cleanup", () => { + const capture = parseProcessCapture({ + ...emptyProcessCapture(), + frames: [{ sequence: 0, at_ms: 0, data: "observed before finalization" }], + event_journal: [], + }); + const executionFailure = new Error("final filesystem snapshot failed"); + const cleanup = { + owned_process_group: { state: "cleaned" as const, reason: null }, + terminal_renderer: { state: "cleaned" as const, reason: null }, + temporary_root: { state: "cleaned" as const, reason: null }, + }; + + let error: ProcessCaptureError | undefined; + try { + resolveProcessResult(capture, executionFailure, cleanup); + } catch (cause: unknown) { + if (!(cause instanceof ProcessCaptureError)) throw cause; + error = cause; + } + expect(error).toBeDefined(); + if (error === undefined) throw new Error("expected capture failure"); + + expect(error).toMatchObject({ + reason: "capture_failed", + cleanupIncomplete: false, + executionFailure: "final filesystem snapshot failed", + cleanupReport: cleanup, + }); + expect(error.cause).toBe(executionFailure); + expect(error.partialObservation).toMatchObject({ + capture: { + frames: [{ data: "observed before finalization" }], + settlement: { cleanup_outcome: "not_required" }, + }, + cleanup, + execution_failure: "final filesystem snapshot failed", + }); + expect(projectAnalysisError(error)).toMatchObject({ + code: "process_capture_failed", + details: { + cleanup_report: cleanup, + execution_failure: "final filesystem snapshot failed", + partial_observation: { + capture: { frames: [{ data: "observed before finalization" }] }, + }, + }, + }); + expect( + analysisErrorProjectionSchema.safeParse(projectAnalysisError(error)) + .success, + ).toBe(true); +}); + +it("preserves cancellation while projecting observations and successful cleanup", () => { + const capture = parseProcessCapture({ + ...emptyProcessCapture(), + frames: [{ sequence: 0, at_ms: 0, data: "observed before cancellation" }], + event_journal: [], + }); + const cleanup = { + owned_process_group: { state: "cleaned" as const, reason: null }, + terminal_renderer: { state: "cleaned" as const, reason: null }, + temporary_root: { state: "cleaned" as const, reason: null }, + }; + let error: ProcessCaptureError | undefined; + try { + resolveProcessResult(capture, processCaptureCancelled(), cleanup); + } catch (cause: unknown) { + if (!(cause instanceof ProcessCaptureError)) throw cause; + error = cause; + } + expect(error).toBeDefined(); + if (error === undefined) throw new Error("expected cancellation"); + + expect(projectAnalysisError(error)).toMatchObject({ + code: "cancelled", + message: "Process capture was cancelled. Start it again when ready.", + details: { + operation: "process_capture", + cleanup: "complete", + cleanup_report: cleanup, + partial_observation: { + capture: { frames: [{ data: "observed before cancellation" }] }, + }, + }, + }); +}); + it("fails closed on Windows before resolving or launching scenario paths", async () => { const scenario = processScenarioSchema.parse({ executable: "Z:/missing/should-never-be-resolved.exe", diff --git a/src/server/createServer.ts b/src/server/createServer.ts index d011e5ac85..fd71d210cd 100644 --- a/src/server/createServer.ts +++ b/src/server/createServer.ts @@ -349,6 +349,14 @@ const registerBinaryAnalysisTools = ({ ...analysisOptions, analysisProfile: session === undefined ? undefined : () => session.analysisProfile(), + allowsSnapshotReplay: + session === undefined + ? undefined + : (operation) => session.allowsSnapshotReplay(operation), + recordWorkflowSnapshot: + session === undefined + ? undefined + : (input) => session.recordWorkflowSnapshot(input), }); const evidenceOptions = { logger, activeTarget, recordEvidence }; registerNativeTools(server, analysis, evidenceOptions); diff --git a/src/server/registerEnhancedTools.ts b/src/server/registerEnhancedTools.ts index 3a6c156afc..4797998a9e 100644 --- a/src/server/registerEnhancedTools.ts +++ b/src/server/registerEnhancedTools.ts @@ -9,19 +9,19 @@ import type { } from "@modelcontextprotocol/server"; import type { AnalysisOperationPort } from "../application/AnalysisProvider.js"; +import type { BinarySessionPort } from "../application/binary/BinarySessionPort.js"; import { EnhancedTools, type ValidatedEnhancedCall, } from "../application/EnhancedTools.js"; import { - REA_WORKFLOW_PROVIDER, - workflowAnalysisProfile, -} from "../application/InvestigationProviders.js"; + createWorkflowEvidence, + workflowSnapshotRecord, +} from "../application/WorkflowEvidence.js"; import { toolContract, type ToolContract } from "../contracts/toolContracts.js"; import type { AnalysisProfileCommitment } from "../domain/analysisProfile.js"; import type { BinaryTarget } from "../domain/binaryTarget.js"; import { UnknownRegistryError } from "../domain/unknownRegistryError.js"; -import { createEvidence } from "../domain/evidence.js"; import type { JsonValue } from "../domain/jsonValue.js"; import type { Logger } from "../logger.js"; import { mcpProgressReporter } from "./mcpProgress.js"; @@ -39,6 +39,14 @@ export interface EnhancedToolRegistration { | undefined; readonly recordEvidence: EvidenceWriter["recordEvidence"] | undefined; readonly recordUnknown: UnknownRegistryPort["recordUnknown"] | undefined; + readonly allowsSnapshotReplay: + | ((operation: ValidatedEnhancedCall["name"]) => boolean) + | undefined; + readonly recordWorkflowSnapshot: + | (( + input: Parameters[0], + ) => ReturnType) + | undefined; } /** Register composed workflows against the same port as direct bridge tools. */ @@ -235,28 +243,27 @@ const executeEnhancedTool = async ( terminal: true, }); if (result.ok) { - const upstreamProfile = registration.analysisProfile?.(); - const evidence = createEvidence( - registration.activeTarget?.(), - REA_WORKFLOW_PROVIDER, - { - operation: name, - parameters, - result: result.value, - ...(upstreamProfile === undefined - ? {} - : { - analysisProfile: workflowAnalysisProfile(upstreamProfile), - }), - confidence: "derived", - limitations: [ - "Derived by an REA workflow from one or more provider observations.", - ], - }, - ); + const evidence = createWorkflowEvidence({ + target: registration.activeTarget?.(), + operation: name, + parameters, + result: result.value, + upstreamProfile: registration.analysisProfile?.(), + }); const recorded = registration.recordEvidence?.(evidence); if (recorded !== undefined && !recorded.ok) return toCallToolResult(recorded, contract); + if ( + name !== "trace_native_ui_action" && + registration.allowsSnapshotReplay?.(name) === true + ) { + const workflowRecord = workflowSnapshotRecord(evidence, name); + if (workflowRecord !== undefined) { + const snapshot = registration.recordWorkflowSnapshot?.(workflowRecord); + if (snapshot !== undefined && !snapshot.ok) + return toCallToolResult(snapshot, contract); + } + } const unknowns = recordWorkflowUnknowns({ name, result: result.value, diff --git a/tests/boundary/mcp/toolResult.test.ts b/tests/boundary/mcp/toolResult.test.ts index eb00c78743..7a48a8c861 100644 --- a/tests/boundary/mcp/toolResult.test.ts +++ b/tests/boundary/mcp/toolResult.test.ts @@ -47,6 +47,54 @@ const contract: ToolContract = { }; describe("completed partial process capture MCP projection", () => { + it("preserves observations when execution fails after cleanup succeeds", () => { + const cleanup = { + owned_process_group: { state: "cleaned" as const, reason: null }, + terminal_renderer: { state: "cleaned" as const, reason: null }, + temporary_root: { state: "cleaned" as const, reason: null }, + }; + const capture = { + ...emptyProcessCapture(), + frames: [{ sequence: 0, at_ms: 0, data: "observed output" }], + event_journal: [], + }; + let failure: ProcessCaptureError | undefined; + try { + resolveProcessResult( + capture, + new Error("final filesystem snapshot failed"), + cleanup, + ); + } catch (cause: unknown) { + if (!(cause instanceof ProcessCaptureError)) throw cause; + failure = cause; + } + if (failure === undefined) + throw new Error("expected process capture failure"); + + const result = toCallToolResult( + err(failure), + toolContract("capture_process_scenario"), + ); + expect(result.isError).toBe(true); + expect(result.structuredContent).toMatchObject({ + error: { + code: "process_capture_failed", + details: { + cleanup_report: cleanup, + execution_failure: "final filesystem snapshot failed", + partial_observation: { + capture: { + frames: [{ data: "observed output" }], + settlement: { cleanup_outcome: "not_required" }, + }, + execution_failure: "final filesystem snapshot failed", + }, + }, + }, + }); + }); + it("preserves partial capture details over the MCP SDK transport", async () => { const executionFailure = new Error("capture ended after a fixture error"); const cleanup = { diff --git a/tests/boundary/mcp/workflowSnapshotReplayMcp.test.ts b/tests/boundary/mcp/workflowSnapshotReplayMcp.test.ts new file mode 100644 index 0000000000..2a985d61e2 --- /dev/null +++ b/tests/boundary/mcp/workflowSnapshotReplayMcp.test.ts @@ -0,0 +1,230 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; +import { afterEach, describe, expect, it } from "vitest"; +import { z } from "zod"; + +import { + createAnalysisExecution, + type AnalysisProvider, + type CapabilityDescriptor, +} from "../../../src/application/AnalysisProvider.js"; +import type { DirectAnalysisDependencies } from "../../../src/application/DirectAnalysisDependencies.js"; +import { runDirectAnalysis } from "../../../src/application/DirectAnalysis.js"; +import { createTestBinarySession } from "../../fixtures/binarySession.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { createAnalysisProfile } from "../../../src/domain/analysisProfile.js"; +import { readAnalysisSnapshot } from "../../../src/application/binary/AnalysisSnapshotFiles.js"; +import { ok } from "../../../src/domain/result.js"; +import { silentLogger } from "../../../src/logger.js"; +import { createServer } from "../../../src/server/createServer.js"; +import { toolContract } from "../../../src/contracts/toolContracts.js"; + +const identity = { id: "fixture", name: "Fixture", version: "1" } as const; +const profile = createAnalysisProfile(identity, { fixture: true }); +const operations = [ + "list_segments", + "list_documents", + "list_procedures", + "list_strings", +] as const; + +const resources: Array<{ close(): Promise }> = []; + +afterEach(async () => { + await Promise.all(resources.splice(0).map((resource) => resource.close())); +}); + +describe("MCP composed workflow snapshot replay", () => { + it("exports an MCP workflow binding and replays its Evidence without provider startup or calls", async () => { + const directory = await createTestTempDirectory( + "rea-mcp-workflow-snapshot-", + ); + const targetPath = join(directory, "fixture.hop"); + const snapshotPath = join(directory, "analysis.json"); + await writeFile(targetPath, "fixture binary"); + const starts: string[] = []; + const calls: string[] = []; + const provider = makeProvider(starts, calls); + const session = createTestBinarySession(provider, { + resolveAnalysisProfile: () => + Promise.resolve(ok({ profile, compatibility: {} })), + }); + const server = createServer(session, session, { logger: silentLogger }); + const mcp = new Client({ name: "workflow-snapshot", version: "1.0.0" }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + resources.push(mcp, server); + await server.connect(serverTransport); + await mcp.connect(clientTransport); + + const opened = await mcp.callTool({ + name: "open_binary", + arguments: { path: targetPath }, + }); + expect(opened.isError).not.toBe(true); + const analyzed = await mcp.callTool({ + name: "binary_overview", + arguments: {}, + }); + expect(analyzed.isError, JSON.stringify(analyzed.content)).not.toBe(true); + const output = toolContract("binary_overview").outputSchema.parse( + analyzed.structuredContent, + ); + const workflowEvidence = z + .object({ + operation: z.literal("binary_overview"), + analysis_profile: z.object({ digest: z.string() }), + }) + .passthrough() + .parse(output.evidence); + const afterAnalysisCalls = [...calls]; + const afterAnalysisStarts = [...starts]; + + const closed = await mcp.callTool({ + name: "close_binary", + arguments: { snapshot_path: snapshotPath }, + }); + expect(closed.isError, JSON.stringify(closed.content)).not.toBe(true); + const snapshot = await readAnalysisSnapshot(snapshotPath); + if (!snapshot.ok) throw snapshot.error; + expect(snapshot.value.workflow_entries).toHaveLength(1); + expect(snapshot.value.evidence_bundle.records).toContainEqual( + expect.objectContaining({ + evidence_id: workflowEvidence.evidence_id, + operation: "binary_overview", + }), + ); + + const dependencies: DirectAnalysisDependencies = { + createBinarySession: () => + createTestBinarySession(provider, { + resolveAnalysisProfile: () => + Promise.resolve(ok({ profile, compatibility: {} })), + }), + createManagedBinarySession: () => + createTestBinarySession(provider, { + resolveAnalysisProfile: () => + Promise.resolve(ok({ profile, compatibility: {} })), + }), + }; + const replay = await runDirectAnalysis( + dependencies, + targetPath, + "binary_overview", + {}, + { snapshotPath }, + ); + + expect(replay).toEqual(output.evidence); + expect(calls).toEqual(afterAnalysisCalls); + expect(starts).toEqual(afterAnalysisStarts); + }); + + it("does not bind a workflow when an upstream operation is live", async () => { + const directory = await createTestTempDirectory( + "rea-mcp-live-workflow-snapshot-", + ); + const targetPath = join(directory, "fixture.hop"); + const snapshotPath = join(directory, "analysis.json"); + await writeFile(targetPath, "fixture binary"); + const starts: string[] = []; + const calls: string[] = []; + const session = createTestBinarySession( + makeProvider(starts, calls, "list_strings"), + { + resolveAnalysisProfile: () => + Promise.resolve(ok({ profile, compatibility: {} })), + }, + ); + const server = createServer(session, session, { logger: silentLogger }); + const mcp = new Client({ + name: "live-workflow-snapshot", + version: "1.0.0", + }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + resources.push(mcp, server); + await server.connect(serverTransport); + await mcp.connect(clientTransport); + + expect( + ( + await mcp.callTool({ + name: "open_binary", + arguments: { path: targetPath }, + }) + ).isError, + ).not.toBe(true); + const analyzed = await mcp.callTool({ + name: "binary_overview", + arguments: {}, + }); + expect(analyzed.isError, JSON.stringify(analyzed.content)).not.toBe(true); + expect( + toolContract("binary_overview").outputSchema.parse( + analyzed.structuredContent, + ).evidence, + ).toMatchObject({ operation: "binary_overview" }); + const closed = await mcp.callTool({ + name: "close_binary", + arguments: { snapshot_path: snapshotPath }, + }); + expect(closed.isError, JSON.stringify(closed.content)).not.toBe(true); + + const snapshot = await readAnalysisSnapshot(snapshotPath); + if (!snapshot.ok) throw snapshot.error; + expect(snapshot.value.workflow_entries ?? []).toEqual([]); + }); +}); + +const makeProvider = ( + starts: string[], + calls: string[], + liveOperation?: (typeof operations)[number], +): AnalysisProvider => { + const capabilities: CapabilityDescriptor[] = operations.map((operation) => ({ + operation, + provider: identity, + available: true, + reason: null, + cachePolicy: operation === liveOperation ? "live" : "snapshot", + effects: { + mutatesArtifact: false, + launchesProcess: true, + mayShowUi: false, + mayAccessNetwork: false, + mayWriteFilesystem: false, + changesPermissions: false, + requiresRoot: false, + }, + limitations: [], + })); + return { + identity: () => identity, + capabilities: () => capabilities, + resolveAnalysisProfile: () => + Promise.resolve(ok({ profile, compatibility: {} })), + createClient: () => { + starts.push("start"); + return { + execute: async (operation) => { + calls.push(operation); + const result = + operation === "health" + ? null + : operation === "list_segments" + ? [{ name: "__TEXT", start: "0x1000", end: "0x2000" }] + : operation === "list_documents" + ? ["fixture"] + : operation === "list_procedures" + ? ["0x1000"] + : { "0x1000": "fixture" }; + return ok(createAnalysisExecution(result, identity)); + }, + close: () => Promise.resolve(), + }; + }, + }; +}; diff --git a/tests/boundary/process/processCaptureLifecycle.test.ts b/tests/boundary/process/processCaptureLifecycle.test.ts index 53e42aaa6f..566c00597e 100644 --- a/tests/boundary/process/processCaptureLifecycle.test.ts +++ b/tests/boundary/process/processCaptureLifecycle.test.ts @@ -423,11 +423,20 @@ itWithCaptureCapability( ); const timedOutCapture = captureObservations(timedOut); expect(timedOutCapture.capture.exit.reason).toBe("timeout"); - if (!timedOutCapture.cleanupIncomplete && timedOut.ok) - expect(timedOut.value.cleanup).toEqual({ + if (!timedOutCapture.cleanupIncomplete && timedOut.ok) { + expect(timedOut.value.cleanup).toMatchObject({ owned_process_group: "verified", temporary_root: "removed", }); + for (const { pid } of timedOut.value.cleanup.unverified_processes ?? []) { + expect(timedOut.value.residual_unknowns).toContainEqual({ + scope: "process", + reason: expect.stringContaining( + `${String(pid)} could not be verified`, + ), + }); + } + } const controller = new AbortController(); setTimeout(() => controller.abort(), 50);