From d109a3910522db9927e4160f9b22669cb2cda0d4 Mon Sep 17 00:00:00 2001 From: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> Date: Tue, 27 Oct 2026 08:47:36 +0800 Subject: [PATCH] feat(cli): validate browser scenarios offline (#7343) Add validate-browser-scenario for offline schema and semantic checks of inline JSON, local files and stdin. Return compact validity/action/secret counts without reading secret values or discovering, launching, attaching to or navigating a browser. Execution and validation share CLI input admission and the canonical domain schema. Preserve existing capture lifecycle, diagnostics, cancellation and caller-relative executable paths. Integrates #7149 with attribution. Fixes #5488. Validation includes 24 focused tests, static/build/docs checks, and the preserved real PocketBase scenario with its backend stopped and secret variables unset. Verification and resource/token measurements are recorded in docs/verification/browser-scenario-validation-2026-10-27.md. Co-authored-by: Ace <1160684883@qq.com> --- docs/browser-scenario-contract.md | 24 ++ .../browser-scenario-validation-2026-10-27.md | 57 +++ src/cliBrowserScenarioCommands.ts | 100 ++++- src/cliCommandNames.ts | 1 + .../cli/browserScenarioValidation.test.ts | 400 ++++++++++++++++++ 5 files changed, 562 insertions(+), 20 deletions(-) create mode 100644 docs/verification/browser-scenario-validation-2026-10-27.md create mode 100644 tests/boundary/cli/browserScenarioValidation.test.ts diff --git a/docs/browser-scenario-contract.md b/docs/browser-scenario-contract.md index eaee148e7e..5c557d34f3 100644 --- a/docs/browser-scenario-contract.md +++ b/docs/browser-scenario-contract.md @@ -51,6 +51,30 @@ covers endpoint discovery only. The following WebSocket handshake and `Target.attachToTarget` each use `command_timeout_ms`. `capture.limits` controls retained event and network-content budgets; it does not change those liveness deadlines. +## Validate a scenario before running it + +`validate-browser-scenario INPUT_JSON` checks one scenario against the same +`browserScenarioSchema` refinement that `capture_browser_scenario` admits, and +stops there. `INPUT_JSON` may be inline JSON, a JSON file path, or `-` for +standard input. + +```bash +rea validate-browser-scenario ./browser-scenario.json --json +``` + +A valid scenario reports `{"valid":true,"actions":,"secrets":}` +and exits 0. An invalid one exits 1 and reports the same field paths capture +would report, so a correction made here is the correction capture needs: +duplicate or reserved step IDs, duplicate secret identities and +environment-variable names, undeclared secret references, request-step +violations, and unknown fields. + +Validation performs no browser discovery, launch, attachment, or navigation and +constructs no provider. It does not read secret environment variables, so a +scenario whose declared variables are unset still validates; the missing-value +failure belongs to capture. A schema-valid scenario does not prove that the +executable or target is available on the host. + ## Minimal launch example Save this as `browser-scenario.json`, replacing the executable path with an diff --git a/docs/verification/browser-scenario-validation-2026-10-27.md b/docs/verification/browser-scenario-validation-2026-10-27.md new file mode 100644 index 0000000000..c0f3e1e89f --- /dev/null +++ b/docs/verification/browser-scenario-validation-2026-10-27.md @@ -0,0 +1,57 @@ +# Browser scenario offline validation + +Executed on Linux x64 with Node 24.22.1/npm 11.16.0, based on main +`3fcbabe593611435ebc26e1eac86706fe60abb6e`. Integrates Ace's #7149 +(`463c8bd73`), with one shared CLI input/admission helper and stronger boundary +coverage. The domain scenario contract remains the semantic owner. + +The input is the preserved scenario from the successful real PocketBase 0.40.6 +CLI workflow described in [saved-values verification](browser-saved-values-2026-10-26.md). +Its SHA-256 is `cfbb3428292250519a5a6f7f9c4b5fea494eb9e3ceccf7b777fd44ffe0f12223`. +The backend was stopped and its loopback endpoint confirmed unreachable before +these checks. Both declared secret environment variables were removed from each +child environment. The original scenario file was verified unchanged. + +Each invocation used the public `validate-browser-scenario FILE --json` command. +All returned `{"valid":true,"actions":7,"secrets":2}` and exit 0. The seven +caller actions exclude the implicit initial capture step. + +| Input / fresh CLI process | Elapsed s | CPU s | Sampled family peak bytes | +| ----------------------------------------------- | --------: | ----: | ------------------------: | +| Recorded scenario | 1.712 | 2.013 | 217,907,200 | +| Same input repeated | 1.722 | 1.989 | 217,239,552 | +| Browser replaced with a nonexistent executable | 1.667 | 1.933 | 219,025,408 | +| Browser replaced with disconnected loopback CDP | 1.557 | 1.880 | 218,779,648 | + +Two CPUs, a 3 GiB Node heap and an 8 GiB family guard bounded execution. Family +RSS is the sum sampled every 100 ms; shared pages can be counted twice and short +peaks can be missed. No resource guard stopped a task. These are fresh processes +with warm filesystem caches, not cold-OS measurements or an analysis-cache claim. +There was no equivalent earlier public offline workflow, so no comparative +speedup is claimed. Profiling/tokenizer driver overhead is outside these tasks. + +Each JSON reply is 38 bytes / **14 estimated o200k tokens**. The complete Node executable/script/CLI argument array plus parsed reply, +encoded as compact JSON `{argv, result}`, uses 75–77 estimated tokens, including +the local paths. Shell/profiler framing is unmeasured. The documents themselves are 1,926–1,942 bytes / 500–508 tokens if +read into context; this file-based task does not require returning them. There +is no model invocation/billing or MCP schema/discovery in these CLI tasks. + +Boundary checks cover unavailable launch/connect resources, throwing provider +factory and enumerable secret getter spies, duplicate/reserved IDs, undeclared +secrets, duplicate secret IDs/environment names, ordered saved references, +unknown fields, unreadable/malformed files, caller-relative executable paths, +stdin and public CLI exit codes. Existing capture diagnostics and summary +publication remain covered. Initial test-helper type errors were corrected +before the final static gate; no passing result is claimed for that failed run. + +Validation establishes schema and cross-reference validity. It does not establish +browser availability, secret availability, selector correctness or successful +execution, and it emits no analysis Evidence. Real-browser capture was verified +in the preceding feature; these offline checks construct no browser provider. + +To reproduce with an existing scenario: + +```bash +rea validate-browser-scenario ./browser-scenario.json --json +cat ./browser-scenario.json | rea validate-browser-scenario - --json +``` diff --git a/src/cliBrowserScenarioCommands.ts b/src/cliBrowserScenarioCommands.ts index 11234a26ac..51a412f3ed 100644 --- a/src/cliBrowserScenarioCommands.ts +++ b/src/cliBrowserScenarioCommands.ts @@ -14,14 +14,18 @@ import { parseCliJsonInput, resolveCliJsonPaths } from "./cliJsonInput.js"; import { logCliCommand } from "./cliLogging.js"; import { projectAnalysisError } from "./domain/analysisErrorProjection.js"; import { analysisInputErrorFromIssues } from "./domain/inputIssueProjection.js"; -import { browserScenarioSchema } from "./domain/browserScenario.js"; +import { + browserScenarioSchema, + type BrowserScenario, +} from "./domain/browserScenario.js"; +import { err, ok, type Result } from "./domain/result.js"; import type { BrowserScenarioCapturePort } from "./application/BrowserScenarioCapturePort.js"; import type { JsonValue } from "./domain/jsonValue.js"; import type { Logger } from "pino"; const OPERATION = "capture_browser_scenario"; -/** Register the one-shot controlled browser scenario command. */ +/** Register controlled browser execution and offline validation commands. */ export const registerBrowserScenarioCommands = ( cli: ReturnType, logger: Logger, @@ -38,7 +42,9 @@ export const registerBrowserScenarioCommands = ( args: z.object({ inputJson: z .string() - .describe("Inline browser scenario JSON or JSON file path"), + .describe( + "Inline browser scenario JSON, JSON file path, or '-' for stdin", + ), }), options: z.object(analysisEvidenceDeliveryOptions), run: ({ args, options: delivery }) => @@ -50,24 +56,12 @@ export const registerBrowserScenarioCommands = ( operation: OPERATION, }); if (deliveryError !== undefined) return cliError(deliveryError); - const input = await parseCliJsonInput( + const scenario = await parseBrowserScenarioInput( args.inputJson, - OPERATION, signal, + "Browser scenario capture failed", ); - if (!input.ok) return input.error; - const scenario = browserScenarioSchema.safeParse( - resolveCliJsonPaths(input.value, [["browser", "executable_path"]]), - ); - if (!scenario.success) - return cliError( - analysisInputErrorFromIssues( - OPERATION, - scenario.error.issues, - input.value, - { cause: scenario.error }, - ), - ); + if (!scenario.ok) return scenario.error; // Load Playwright only for this command, not during CLI startup. const provider = options.createProvider === undefined @@ -77,7 +71,7 @@ export const registerBrowserScenarioCommands = ( ) : options.createProvider(environment); const result = await finishAnalysisCommand( - () => captureBrowserScenario(provider, scenario.data, { signal }), + () => captureBrowserScenario(provider, scenario.value, { signal }), { close: async () => { await provider?.close?.(); @@ -101,11 +95,77 @@ export const registerBrowserScenarioCommands = ( }), ), }); + registerBrowserScenarioValidationCommand(cli, logger); +}; + +/** Validate the execution contract without provider or environment access. */ +const registerBrowserScenarioValidationCommand = ( + cli: ReturnType, + logger: Logger, +): void => { + cli.command(CLI_COMMANDS.validateBrowserScenario, { + description: + "Validate one browser scenario without launching or attaching to a browser", + args: z.object({ + inputJson: z + .string() + .describe( + "Inline browser scenario JSON, JSON file path, or '-' for stdin", + ), + }), + run: ({ args }) => + withCommandCancellation((signal) => + logCliCommand( + logger, + CLI_COMMANDS.validateBrowserScenario, + async () => { + const scenario = await parseBrowserScenarioInput( + args.inputJson, + signal, + "Browser scenario validation failed", + ); + if (!scenario.ok) return scenario.error; + return { + valid: true, + actions: scenario.value.actions.length, + secrets: scenario.value.secrets.length, + }; + }, + ), + ), + }); +}; + +/** Keep caller-path normalization and canonical diagnostics identical for both commands. */ +const parseBrowserScenarioInput = async ( + inputJson: string, + signal: AbortSignal, + headline: string, +): Promise> => { + const input = await parseCliJsonInput(inputJson, OPERATION, signal); + if (!input.ok) return input; + const scenario = browserScenarioSchema.safeParse( + resolveCliJsonPaths(input.value, [["browser", "executable_path"]]), + ); + return scenario.success + ? ok(scenario.data) + : err( + cliError( + analysisInputErrorFromIssues( + OPERATION, + scenario.error.issues, + input.value, + { cause: scenario.error }, + ), + headline, + ), + ); }; const cliError = ( error: Parameters[0], + headline = "Browser scenario capture failed", ): JsonValue => ({ - error: "Browser scenario capture failed", + error: headline, ...projectAnalysisError(error), }); diff --git a/src/cliCommandNames.ts b/src/cliCommandNames.ts index b56abae118..b0d397d019 100644 --- a/src/cliCommandNames.ts +++ b/src/cliCommandNames.ts @@ -158,6 +158,7 @@ export const CLI_COMMANDS = Object.freeze({ captureWebScreenshot: "capture-web-screenshot", compareWebScreenshots: "compare-web-screenshots", captureBrowserScenario: "capture-browser-scenario", + validateBrowserScenario: "validate-browser-scenario", listElectronTargets: "list-electron-targets", inspectElectronPage: "inspect-electron-page", inspectElectronFuses: "inspect-electron-fuses", diff --git a/tests/boundary/cli/browserScenarioValidation.test.ts b/tests/boundary/cli/browserScenarioValidation.test.ts new file mode 100644 index 0000000000..b54431036e --- /dev/null +++ b/tests/boundary/cli/browserScenarioValidation.test.ts @@ -0,0 +1,400 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { Cli } from "incur"; +import pino from "pino"; +import { expect, it, vi } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import { registerBrowserScenarioCommands } from "../../../src/cliBrowserScenarioCommands.js"; +import { BROWSER_SCENARIO_START_STEP_ID } from "../../../src/domain/browserScenarioValues.js"; + +const serveJson = async ( + registered: ReturnType, + command: string, + input: unknown, +): Promise => { + let stdout = ""; + await registered.cli.serve([command, JSON.stringify(input), "--json"], { + env: {}, + stdout: (text) => { + stdout += text; + }, + exit: () => undefined, + }); + expect(registered.createProvider).not.toHaveBeenCalled(); + expect(registered.readSecret).not.toHaveBeenCalled(); + return JSON.parse(stdout) as unknown; +}; + +/** Register only the scenario commands, with a provider that must never run. */ +const registerValidatingCli = () => { + const cli = Cli.create("rea-scenario-validation-test", { sync: false }); + const createProvider = vi.fn((): never => { + throw new Error("validation must not construct a browser provider"); + }); + const readSecret = vi.fn((): never => { + throw new Error("validation must not resolve secret environment values"); + }); + const environment = Object.defineProperty({}, "SESSION_TOKEN", { + enumerable: true, + get: readSecret, + }); + registerBrowserScenarioCommands(cli, pino({ level: "silent" }), environment, { + createProvider, + }); + return { cli, createProvider, readSecret }; +}; + +const launchScenario = { + browser: { mode: "launch", executable_path: "browser" }, + start_url: { url: "http://127.0.0.1:3000" }, + actions: [{ step_id: "settle", action: "wait_for_timeout", duration_ms: 1 }], +} as const; + +it("accepts a valid scenario without launching or attaching to a browser", async () => { + const output = await serveJson( + registerValidatingCli(), + "validate-browser-scenario", + launchScenario, + ); + expect(output).toEqual({ valid: true, actions: 1, secrets: 0 }); +}); + +it("counts declared secrets without resolving their environment values", async () => { + const cli = registerValidatingCli(); + const output = await serveJson(cli, "validate-browser-scenario", { + ...launchScenario, + actions: [ + { + step_id: "sign-in", + action: "fill", + locator: { kind: "css", selector: "#token" }, + value: { source: "secret", secret_id: "session_token" }, + }, + ], + secrets: [ + { secret_id: "session_token", environment_variable: "SESSION_TOKEN" }, + { secret_id: "api_key", environment_variable: "API_KEY" }, + ], + }); + expect(output).toEqual({ valid: true, actions: 1, secrets: 2 }); +}); + +it("rejects a duplicate action step identity with the offending field path", async () => { + const output = await serveJson( + registerValidatingCli(), + "validate-browser-scenario", + { + ...launchScenario, + actions: [ + { step_id: "settle", action: "wait_for_timeout", duration_ms: 1 }, + { step_id: "settle", action: "wait_for_timeout", duration_ms: 2 }, + ], + }, + ); + expect(output).toMatchObject({ + error: "Browser scenario validation failed", + code: "invalid_request", + category: "invalid_input", + details: { + issues: [ + expect.objectContaining({ + path: ["actions"], + reason: "invalid_value", + message: "Action step IDs must be unique", + }), + ], + }, + }); +}); + +it("rejects an action that reuses the reserved initial step identity", async () => { + const output = await serveJson( + registerValidatingCli(), + "validate-browser-scenario", + { + ...launchScenario, + actions: [ + { + step_id: BROWSER_SCENARIO_START_STEP_ID, + action: "wait_for_timeout", + duration_ms: 1, + }, + ], + }, + ); + expect(output).toMatchObject({ + error: "Browser scenario validation failed", + code: "invalid_request", + category: "invalid_input", + details: { + issues: [ + expect.objectContaining({ + path: ["actions", 0, "step_id"], + reason: "invalid_value", + }), + ], + }, + }); +}); + +it("rejects an undeclared secret reference without exposing any secret value", async () => { + const cli = registerValidatingCli(); + const output = await serveJson(cli, "validate-browser-scenario", { + ...launchScenario, + actions: [ + { + step_id: "sign-in", + action: "fill", + locator: { kind: "css", selector: "#token" }, + value: { source: "secret", secret_id: "session_token" }, + }, + ], + }); + const serialized = JSON.stringify(output); + expect(output).toMatchObject({ + error: "Browser scenario validation failed", + code: "invalid_request", + category: "invalid_input", + details: { + issues: [ + expect.objectContaining({ + path: ["secrets"], + reason: "invalid_value", + message: expect.stringContaining("session_token"), + }), + ], + }, + }); + // The declaration is reported by identity only; no environment variable name + // or resolved value belongs in a validation reply. + expect(serialized).not.toContain("SESSION_TOKEN"); + expect(serialized).not.toContain("environment_variable"); +}); + +it("rejects an unknown field instead of ignoring it", async () => { + const output = await serveJson( + registerValidatingCli(), + "validate-browser-scenario", + { + ...launchScenario, + navigation: { timeout_ms: 5 }, + }, + ); + expect(output).toMatchObject({ + error: "Browser scenario validation failed", + code: "invalid_request", + category: "invalid_input", + details: { + issues: [ + expect.objectContaining({ + path: ["navigation"], + reason: "unknown_argument", + }), + ], + }, + }); +}); + +it("rejects a scenario whose input file cannot be read", async () => { + const cli = registerValidatingCli(); + let stdout = ""; + await cli.cli.serve( + ["validate-browser-scenario", "missing-scenario.json", "--json"], + { + env: {}, + stdout: (text) => { + stdout += text; + }, + exit: () => undefined, + }, + ); + expect(cli.createProvider).not.toHaveBeenCalled(); + expect(cli.readSecret).not.toHaveBeenCalled(); + expect(JSON.parse(stdout) as unknown).toMatchObject({ + input_path: "missing-scenario.json", + input_reason: "read-failed", + }); +}); + +it("accepts a disconnected CDP target without constructing a provider", async () => { + expect( + await serveJson(registerValidatingCli(), "validate-browser-scenario", { + ...launchScenario, + browser: { + mode: "connect", + cdp_endpoint: "http://127.0.0.1:1", + target_id: "unavailable-target", + }, + }), + ).toEqual({ valid: true, actions: 1, secrets: 0 }); +}); + +it.each([ + [ + { secret_id: "token", environment_variable: "FIRST" }, + { secret_id: "token", environment_variable: "SECOND" }, + ], + [ + { secret_id: "first", environment_variable: "SAME" }, + { secret_id: "second", environment_variable: "SAME" }, + ], +])( + "rejects duplicate secret identities or environment references (%j)", + async (first, second) => { + expect( + await serveJson(registerValidatingCli(), "validate-browser-scenario", { + ...launchScenario, + secrets: [first, second], + }), + ).toMatchObject({ + code: "invalid_request", + details: { + issues: [ + expect.objectContaining({ + path: ["secrets"], + reason: "invalid_value", + }), + ], + }, + }); + }, +); + +const savedSteps = [ + { + step_id: "create", + action: "request", + method: "POST", + path: "/api/items", + json: { title: "test" }, + save: [{ name: "id", json_pointer: "/id" }], + }, + { + step_id: "read", + action: "request", + method: "GET", + path: "/api/items/{saved:id}", + }, +]; +it("accepts saved response references only after their producing step", async () => { + const valid = { ...launchScenario, actions: savedSteps }; + expect( + await serveJson( + registerValidatingCli(), + "validate-browser-scenario", + valid, + ), + ).toEqual({ valid: true, actions: 2, secrets: 0 }); + expect( + await serveJson(registerValidatingCli(), "validate-browser-scenario", { + ...valid, + actions: [...savedSteps].reverse(), + }), + ).toMatchObject({ + code: "invalid_request", + details: { + issues: [ + expect.objectContaining({ + path: ["actions", 0], + message: expect.stringContaining("no earlier step saves"), + }), + ], + }, + }); +}); + +cliTest.for(["launch", "connect"] as const)( + "validates %s through the public CLI while runtime resources are unavailable", + async (mode, { cli }) => { + const result = await cli.run({ + arguments: [ + "validate-browser-scenario", + JSON.stringify({ + ...launchScenario, + browser: + mode === "launch" + ? { mode, executable_path: "/nonexistent/rea-validation-browser" } + : { + mode, + cdp_endpoint: "http://127.0.0.1:1", + target_id: "not-attached", + }, + secrets: [ + { + secret_id: "session_token", + environment_variable: "SESSION_TOKEN", + }, + ], + }), + "--json", + ], + environment: { + REA_LOG_LEVEL: "silent", + SESSION_TOKEN: "validation-unread-secret", + }, + }); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.json).toEqual({ valid: true, actions: 1, secrets: 1 }); + expect(result.stdout).not.toContain("validation-unread-secret"); + }, +); + +cliTest( + "validates stdin and caller-relative file paths with canonical diagnostics", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-scenario-validation-"); + const input = JSON.stringify(launchScenario); + await writeFile(join(root, "valid.json"), input); + await writeFile( + join(root, "invalid.json"), + JSON.stringify({ + ...launchScenario, + actions: [...launchScenario.actions, ...launchScenario.actions], + }), + ); + await writeFile(join(root, "malformed.json"), '{"private-value":'); + for (const arguments_ of [ + ["validate-browser-scenario", "valid.json", "--json"], + ["validate-browser-scenario", "-", "--json"], + ]) { + const result = await cli.run({ + arguments: arguments_, + cwd: root, + ...(arguments_[1] === "-" ? { input } : {}), + }); + expect(result.exitCode, result.stderr).toBe(0); + expect(result.json).toEqual({ valid: true, actions: 1, secrets: 0 }); + } + const rejected = await cli.run({ + arguments: ["validate-browser-scenario", "invalid.json", "--json"], + cwd: root, + }); + expect(rejected.exitCode).toBe(1); + expect(rejected.json).toMatchObject({ + error: "Browser scenario validation failed", + code: "invalid_request", + details: { + issues: [ + expect.objectContaining({ + path: ["actions"], + message: "Action step IDs must be unique", + }), + ], + }, + }); + const malformed = await cli.run({ + arguments: ["validate-browser-scenario", "malformed.json", "--json"], + cwd: root, + }); + expect(malformed.exitCode).toBe(1); + expect(malformed.json).toMatchObject({ + code: "invalid_request", + input_path: "malformed.json", + input_reason: "invalid-json", + }); + expect(malformed.stdout).not.toContain("private-value"); + }, +);