feat(desktop): build several tasks from one capture, and measure each task's pass rate within a budget (#9216)

* docs(desktop): clarify gzip scrub limits * fix(setup): own desktop build process cleanup * fix(doctor): compare desktop settings with resolved defaults * fix(desktop): refuse a build runtime whose export takes no task ends Several tasks pass --task-end to the build computer's export, so a runtime packed before it now fails at the start of the capture with the repack fix, as a runtime without synthetic seeds already did, rather than at export. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(desktop): format the noisy-record draft files with the pinned oxfmt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): give the root-array draft test its own describe The records describe grew past the 150-line lint limit with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(doctor): inspect client-recorded desktop runners * fix(doctor): reuse matching runner probes * fix(setup): propagate cancellation to desktop actions * fix(desktop): rewrite a download's launch keys written with spaces around = The desktop entry parser accepts `Exec = notes %F` and an indented `TryExec =notes`, so the install script now matches those keys the same way instead of leaving the original program in place. Adds the feature's tasks.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): keep why a measured run failed in its solve record A solve that returned an outcome dropped the runner's error, so a run that failed to deploy, solve or verify was retried with no reason recorded. The record, the measure result and difficulty.json now carry it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(setup): stop Hopper on cancel, never report a cancelled setup ready, and keep a failed action's client setting - The Hopper install gets setup's signal: Linux already honoured it, and the macOS installer now stops its downloads and stops before mounting or copying. - runSetup checks the signal after the final doctor and close, so a cancellation there raises instead of returning ready. - A failed pack or agentenv install no longer writes its archive or runner to client configs, so the setting the client already had stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(setup): keep the completed setup result in a cancellation after cleanup A cancellation during the final doctor or host close carried only the applied actions, dropping each client's result and backup path and the doctor report. It now carries the whole result as setup_result, as doctor's cancellation carries its report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): move launch execution to boundary suite * fix(desktop): redact build schema failure output * refactor(desktop): normalize schema failure once * fix(desktop): omit unsafe schema parser snippets * fix(desktop): preserve draft change and selected path evidence * test(desktop): group retained dconf note regressions * fix(desktop): respect archive entry format in draft checks * test(desktop): separate unowned-file draft note cases * fix(desktop): preserve explicitly selected journal documents * fix(xml): avoid scanning selected ancestors per tag * fix(desktop): exclude covered text runs from draft counts * fix: preserve text when drafting desktop checks * fix: restore nested desktop state aliases * refactor: share UUID recognition across draft checks * fix: retain multiline XML attribute context * fix: retain relationship identifiers in task drafts * docs: clarify desktop draft identifier handling * test(desktop): type formatting run cases as pairs * fix(desktop): carry document format into selector evaluation * test(desktop): separate markup boundary regression group * fix(desktop): propagate formats through draft selectors * fix(desktop): switch selector format within archive entries * test(desktop): remove duplicate seed filename case * fix(setup): retain diagnostics when cancellation overlaps failure * refactor(setup): keep command cancellation at its owner * test(setup): verify cancellation at the application boundary * test(desktop): pass JSON format in exported index parity * docs(desktop): clarify task checks before scratch export * fix(desktop): respell selected paths in file URLs * test(desktop): group captured document spelling cases * test(desktop): run installer scripts at boundary * style(desktop): remove trailing blank line * fix(desktop): bound draft preview serialization * fix(desktop): restore selected state aliases * fix(desktop): respect entry format in draft noise * test(desktop): match exported alias evidence at its JSON boundary * fix(desktop): retain app state logs in exported seeds * fix: align desktop numeric seed comparisons * test(desktop): keep one draft exclusion fixture import * test(desktop): retain log eligibility across draft test groups * fix(doctor): resolve empty registered desktop runner * style: fix desktop launch test formatting * style: format native command path schema * fix(desktop): map selected external launch paths * style(desktop): format seed document regressions * test(desktop): group launch URL regressions * fix(desktop): omit redundant nested state aliases * fix(desktop): compare added free text as a whole * fix(desktop): check recompressed seeds as shipped * docs(desktop): clarify recovered seed format * test(desktop): group shipped gzip seed checks * fix: retain quoted identifier text in task drafts * fix: recognize Unicode XML in desktop state readers * fix(desktop): bound nested draft checks * fix(desktop): explain missing arrays in truncated views * fix(desktop): find XML type references in archives * test(desktop): remove duplicated capture assertions * fix(desktop): retain nested omitted text boundaries * style(desktop): format draft value imports * test(desktop): group model and draft value checks * fix(desktop): map captured launch paths once * refactor: remove unused whole-text guard state * refactor: stop passing unused text children * test: align free-text checks with whole-note policy * fix: validate recompressed desktop archive seeds * style(desktop): format archive seed regression * fix(desktop): retain placed-item sibling guards * fix(desktop): preserve archive paths and bound doctor reads * fix(doctor): refuse non-files before descriptor admission * fix(setup): track remaining cleanup resources across retries * test(desktop): specify XML format in heap pairing fixture * fix(setup): skip Hopper launch after cancelled copy * fix: preserve task measurement limits and cleanup * fix(desktop): retain reported solve sandbox cleanup * test(desktop): check retained log seed admission * fix(doctor): query agentenv for model endpoint * fix(doctor): preserve uncertainty for empty endpoint override * fix(desktop): redact runtime schema errors * fix(desktop): normalize captured file URLs * fix(desktop): report seeded files by capture path * fix(desktop): retain startup readiness failure stage * fix(desktop): retain JSON date checks * fix(desktop): preserve archive formatting checks * fix(desktop): preserve explicitly selected journal files * fix(desktop): report build command transport limits * docs(desktop): clarify unreadable dconf versions * test(desktop): remove duplicate history regression * fix(desktop): align XML names with exported reader * fix(desktop): create selected state alias targets * fix(desktop): inspect supported runtime archives * fix(desktop): keep ordinary text checks exact * fix(desktop): verify registered build host paths * fix(desktop): retain record identity in draft checks * test(desktop): preserve text reference numbers * fix(desktop): bound download install input reads * fix(desktop): match placed notes by whole text * test(desktop): match multiple task export results * fix(desktop): retain failed measurement file closes * fix(setup): retain completed actions on cancellation * test(setup): retain cancelled installer result * fix(desktop): scrub XML independently of reader compatibility * fix(desktop): preserve file aliases and reader diagnostics * style(desktop): format review fixes * fix(desktop): inspect archives with registered runner * test(desktop): verify registered archive interpreter * style(desktop): format capture review fixes * style(desktop): format review changes * feat(desktop): target agentenv-framework 0.9.1464 Desktop bundles, setup's --install-agentenv and the doctor's exact-version check now target 0.9.1464, which keeps an exported app's data/get call, holds the computer fixture above boot dialogs, leaves an unresolved setting's build unverified rather than failed, and mounts lock packages instead of copying them into the image. The web environment lane stays on 0.9.1431. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): retain per-solve sandbox cleanup roots * test(desktop): expect the solve cleanup helper among the shipped agentenv helpers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(desktop): clean up a stopped solve's sandboxes through agentenv 0.9.1482 agentenv-framework 0.9.1482 lists the local sandboxes under a root (local_sandbox_ids) and rebuilds one killed while it wrote its port map for teardown. The solve cleanup helper now terminates each sandbox agentenv lists, keeping its Docker and host-process checks, and reports anything else left under the root. Desktop bundles, setup's --install-agentenv and the doctor's version check target 0.9.1482. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): verify a killed solve's cleanup once its sandbox root is empty A solve's sandboxes are all local under its private root, and agentenv now keeps a sandbox's folder until its teardown is confirmed. So when the helper ends without a report and REA's cleanup empties that root, the cleanup is verified: a cancelled solve ends as a cancellation and a solve past its deadline as a failed run, rather than as cleanup the measure cannot vouch for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(desktop): target agentenv-framework 0.9.1485 0.9.1485 keeps a local sandbox's folder until its teardown is confirmed, which the solve cleanup's verified result relies on. (0.9.1484 failed to publish.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): record a solve stopped before it reports as failed, not as a refused request A helper killed from outside or stopped at its deadline left no report, so the measure refused with invalid input and told the caller to fix its model endpoint. A helper a signal ended now gives a failed solve of unknown cost, which the measure records and stops on; one that exits on its own without a report still ends the measure as a solve that could not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): check verifier behavior after selector changes * test(windows): verify rejected writes during cleanup by effect * fix(desktop): preserve selected alias target kinds in exports * fix(cli): retain caller capture paths in environment builds * fix(ghidra): classify oversized endpoint records consistently * test: align model and Evidence expectations with current contracts * fix(desktop): reconcile upstream sandbox cleanup recovery * fix(docs): link cleanup qualification as repository evidence --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 16:57:19 +00:00 · 2026-11-10 05:26:28 -05:00
1 parent cb4e5cfbff
commit d2f9b8b7de
@@ -0,0 +1,430 @@
+import { createHash } from "node:crypto";
+import { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
+import {
+  ArtifactFileCleanupFailure,
+  bindArtifactFileCleanup,
+} from "../../artifacts/ArtifactFileOperation.js";
+import { RegularFileCapacityError } from "../../filesystem/readStableFileBytes.js";
+import {
+  guestFileAnswerCeiling,
+  guestFileAnswerWorkingBytes,
+} from "../../desktop/AgentEnvComputer.js";
+import { writeTextFile } from "../JsonFiles.js";
+import { join, resolve } from "node:path";
+
+import { z } from "zod";
+
+import { AnalysisError } from "../../domain/analysisErrorBase.js";
+import {
+  AnalysisCancelledError,
+  AnalysisInputError,
+  AnalysisResourceConstraintError,
+} from "../../domain/analysisErrorCore.js";
+import {
+  difficultyRecord,
+  difficultySection,
+  nextSolve,
+  withDifficultySection,
+  type DifficultyRecord,
+  type SolveRecord,
+  type StopReason,
+} from "../../domain/desktop/taskDifficulty.js";
+import { ProviderAdapterError } from "../../domain/providerAdapterError.js";
+import { err, ok, type Result } from "../../domain/result.js";
+import { safeParseUtf8Json } from "../../domain/safeJson.js";
+import { AGENTENV_PROVIDER } from "../../desktop/AgentEnvComputerHelper.js";
+import {
+  runSolveHelper,
+  SolveCleanupError,
+  SolveCancelledError,
+  type SolveReport,
+  type SolveCleanupOwner,
+} from "../../desktop/AgentEnvSolveHelper.js";
+import {
+  resolveBuildComputerEnvironment,
+  type BuildComputerDefaults,
+} from "../../desktop/BuildComputerEnvironment.js";
+import { readHostFile } from "../../desktop/HostFileRead.js";
+import { PRODUCT_IDENTITY } from "../../identity.js";
+import {
+  silentProgressReporter,
+  type ProgressReporter,
+} from "../ProgressReporter.js";
+import { errorText } from "./DesktopBuildCommands.js";
+
+const MEASURE = "measure_build_task";
+/** The file at the bundle's root that records each measured task. */
+const DIFFICULTY_FILE = "difficulty.json";
+const README_FILE = "README.md";
+/** The most bytes REA reads of a task's steps, verifier, README or difficulty file. */
+const FILE_BYTES = 4 * 1024 * 1024;
+
+/** What the agent asks the measure for. */
+export interface MeasureRequest {
+  readonly bundle: string;
+  readonly task: string;
+  readonly model: string;
+  readonly agent: string;
+  readonly budget_usd: number;
+}
+
+/** One solve of the task in a fresh computer. */
+export type Solver = (
+  request: Pick<MeasureRequest, "bundle" | "task" | "model" | "agent">,
+  signal: AbortSignal,
+) => Promise<SolveReport>;
+
+/** What the measure reports: the record it wrote and where. */
+export type MeasuredTask = Omit<DifficultyRecord, "solves" | "versions"> & {
+  readonly bundle: string;
+  readonly task: string;
+  readonly solves: DifficultyRecord["solves"];
+  readonly difficulty_file: string;
+};
+
+const refused = (userMessage: string, remediationAction: string) =>
+  new AnalysisInputError(MEASURE, {
+    userMessage,
+    remediationAction,
+    conformsToInputContract: true,
+  });
+
+/** The records a difficulty file holds, each read only as far as the README names it. */
+const recordsSchema = z.looseObject({
+  tasks: z.record(
+    z.string(),
+    z.looseObject({
+      model: z.string(),
+      passes: z.int(),
+      k: z.int(),
+      band: z.enum(["in_band", "too_easy", "too_hard", "undecided"]),
+      usd: z.number(),
+    }),
+  ),
+});
+
+/** The task's steps and verifier as the export wrote them, digested, or why the bundle has none. */
+const taskDigest = async (
+  bundle: string,
+  task: string,
+  signal: AbortSignal,
+): Promise<Result<string, AnalysisError>> => {
+  const paths = [`tasks/${task}.json`, `artifacts/${task}-check/verify.py`];
+  const hash = createHash("sha256");
+  const maximum = guestFileAnswerWorkingBytes(guestFileAnswerCeiling());
+  for (const path of paths) {
+    try {
+      const file = await readHostFile(
+        join(bundle, path),
+        maximum,
+        signal,
+        true,
+      );
+      if (!Buffer.isBuffer(file))
+        return err(
+          refused(
+            `The bundle at ${bundle} holds no readable ${path}.`,
+            "Name a task the bundle's export wrote, with its steps and verifier.",
+          ),
+        );
+      hash.update(file);
+    } catch (cause: unknown) {
+      if (cause instanceof RegularFileCapacityError)
+        return err(
+          new AnalysisResourceConstraintError(
+            MEASURE,
+            "capacity",
+            "Task file exceeds the host read budget",
+            { max_bytes: maximum, path: cause.path },
+            { cause },
+          ),
+        );
+      throw cause;
+    }
+  }
+  return ok(hash.digest("hex"));
+};
+
+/** One finished report as the record keeps it. */
+const solveRecord = (
+  report: Extract<SolveReport, { outcome: string }>,
+): SolveRecord => ({
+  outcome: report.outcome,
+  turns: report.turns,
+  usd: report.usd,
+  seconds: report.seconds,
+  failed_checks:
+    report.outcome === "passed" || report.outcome === "scored below 1"
+      ? report.checks.filter((check) => !check.passed).map((check) => check.id)
+      : [],
+  instance_id: report.instance_id,
+  error: report.error,
+});
+
+/** The bundle's difficulty records and README as they stand. */
+interface BundleRecords {
+  readonly tasks: Readonly<
+    Record<string, z.infer<typeof recordsSchema>["tasks"][string]>
+  >;
+  readonly readme: string;
+}
+
+/** The bundle's records, a missing file read as none; a file REA cannot read or parse is refused, so it is never replaced. */
+const readRecords = async (
+  bundle: string,
+  signal: AbortSignal,
+): Promise<Result<BundleRecords, AnalysisError>> => {
+  const unreadable = (file: string, why: string) =>
+    err(
+      refused(
+        `The bundle's ${file} ${why}, so REA would replace records it cannot keep.`,
+        `Repair or move ${join(bundle, file)} and measure again.`,
+      ),
+    );
+  const records = await readHostFile(
+    join(bundle, DIFFICULTY_FILE),
+    FILE_BYTES,
+    signal,
+  );
+  const readme = await readHostFile(
+    join(bundle, README_FILE),
+    FILE_BYTES,
+    signal,
+  );
+  if (records === "unreadable")
+    return unreadable(DIFFICULTY_FILE, "cannot be read");
+  if (readme === "unreadable") return unreadable(README_FILE, "cannot be read");
+  let tasks: BundleRecords["tasks"] = {};
+  if (records !== "missing") {
+    const json = safeParseUtf8Json(records);
+    const parsed = json.ok ? recordsSchema.safeParse(json.value) : undefined;
+    if (parsed?.success !== true)
+      return unreadable(DIFFICULTY_FILE, "is not a difficulty file REA reads");
+    tasks = parsed.data.tasks;
+  }
+  return ok({
+    tasks,
+    readme: readme === "missing" ? "" : readme.toString("utf8"),
+  });
+};
+
+/** The record writes under way, by bundle, so a measure merges with the one before it rather than racing it. */
+const recording = new Map<string, Promise<unknown>>();
+
+/** Run `write` after every earlier write to the same bundle by this REA. */
+const afterEarlierWrites = async <Value>(
+  bundle: string,
+  write: () => Promise<Value>,
+): Promise<Value> => {
+  const key = resolve(bundle);
+  const earlier = recording.get(key) ?? Promise.resolve();
+  const mine = earlier.then(write, write);
+  const settled = mine.catch(() => undefined);
+  recording.set(key, settled);
+  try {
+    return await mine;
+  } finally {
+    if (recording.get(key) === settled) recording.delete(key);
+  }
+};
+
+/** Record the task's measure in the bundle's difficulty file, beside other tasks', and in its README. */
+const recordMeasure = (
+  bundle: string,
+  task: string,
+  record: DifficultyRecord,
+  signal: AbortSignal,
+  resources: ArtifactResourceScope,
+): Promise<Result<string, AnalysisError>> =>
+  afterEarlierWrites(bundle, async () => {
+    const current = await readRecords(bundle, signal);
+    if (!current.ok) return current;
+    const tasks = { ...current.value.tasks, [task]: record };
+    const path = join(bundle, DIFFICULTY_FILE);
+    const saved = await writeTextFile(
+      `${JSON.stringify({ tasks }, null, 2)}\n`,
+      path,
+      true,
+      resources,
+    );
+    if (!saved.ok) return saved;
+    const readme = await writeTextFile(
+      withDifficultySection(current.value.readme, difficultySection(tasks)),
+      join(bundle, README_FILE),
+      true,
+      resources,
+    );
+    if (!readme.ok) return readme;
+    return ok(path);
+  });
+
+/**
+ * Measure a bundle's task: solve it in fresh computers, 3 times, then 2 more
+ * unless they passed none or all, stopping before a solve the budget cannot
+ * cover, and record the band, cost, turns and failures per check in the
+ * bundle's difficulty file and README.
+ */
+export const measureBuildTask = async (
+  request: MeasureRequest,
+  solver: Solver,
+  signal: AbortSignal,
+  progress: ProgressReporter = silentProgressReporter,
+  resources: ArtifactResourceScope = new ArtifactResourceScope(),
+): Promise<Result<MeasuredTask, AnalysisError>> => {
+  const solves: SolveRecord[] = [];
+  const retainSolves = (failure: AnalysisError): AnalysisError => {
+    if (solves.length > 0)
+      failure.retainPartialObservation({
+        kind: "partial-task-measurement",
+        bundle: request.bundle,
+        task: request.task,
+        solves: [...solves],
+      });
+    return failure;
+  };
+  let agentenv: string | null = null;
+  let stopped: StopReason = "decided";
+  try {
+    const digest = await taskDigest(request.bundle, request.task, signal);
+    if (!digest.ok) return digest;
+    // Refuse records REA could not keep before any solve is paid for.
+    const records = await readRecords(request.bundle, signal);
+    if (!records.ok) return records;
+    for (;;) {
+      const decision = nextSolve(solves, request.budget_usd);
+      if (!decision.next) {
+        stopped = decision.stopped;
+        break;
+      }
+      await progress
+        .report({
+          phase: MEASURE,
+          completed: solves.length,
+          total: null,
+          message: `solve ${String(solves.length + 1)} of ${request.task} with ${request.model}`,
+        })
+        .catch(() => undefined);
+      const report = await solver(request, signal);
+      if (!("outcome" in report))
+        return err(
+          retainSolves(
+            refused(
+              `A solve of ${request.task} did not run: ${report.error}`,
+              "Fix what the error names, such as the model endpoint agentenv needs (LITELLM_BASE_URL and LITELLM_API_KEY where REA starts, or [model] in .agentenv/config.toml), and measure again.",
+            ),
+          ),
+        );
+      agentenv = report.agentenv ?? agentenv;
+      solves.push(solveRecord(report));
+    }
+    const record = difficultyRecord({
+      model: request.model,
+      agent: request.agent,
+      solves,
+      budgetUsd: request.budget_usd,
+      stopped,
+      taskSha256: digest.value,
+      versions: { rea: PRODUCT_IDENTITY.packageVersion, agentenv },
+      measuredAt: new Date().toISOString(),
+    });
+    const file = await recordMeasure(
+      request.bundle,
+      request.task,
+      record,
+      signal,
+      resources,
+    );
+    if (!file.ok) return err(retainSolves(file.error));
+    const { versions: _versions, ...reported } = record;
+    return ok({
+      bundle: request.bundle,
+      task: request.task,
+      ...reported,
+      difficulty_file: file.value,
+    });
+  } catch (cause: unknown) {
+    if (
+      (cause instanceof SolveCleanupError ||
+        cause instanceof SolveCancelledError) &&
+      cause.report !== undefined &&
+      "outcome" in cause.report
+    )
+      solves.push(solveRecord(cause.report));
+    if (cause instanceof ArtifactFileCleanupFailure) {
+      const cleanup = cause.cleanup ?? {
+        reason: "an input handle could not be closed",
+        resources: [cause.fileCleanupFailure.path],
+      };
+      for (const lease of cause.cleanupLeases ?? [cause.cleanupLease])
+        resources.retain({
+          kind: "lease",
+          resource: cause.fileCleanupFailure.path,
+          lease,
+        });
+      const cancelled =
+        cause.cancellation !== undefined ||
+        (signal.aborted &&
+          cause.fileCleanupFailure.outcome.kind === "completed");
+      const failure = cancelled
+        ? new AnalysisCancelledError(MEASURE, {
+            cause:
+              cause.cancellation === undefined
+                ? signal.reason
+                : cause.cancellation.reason,
+            cleanup,
+          })
+        : new ProviderAdapterError(AGENTENV_PROVIDER, MEASURE, {
+            userMessage: `Measuring ${request.task} could not close an input file.`,
+            cause,
+            cleanup,
+          });
+      bindArtifactFileCleanup(cause, failure);
+      return err(retainSolves(failure));
+    }
+    const failure =
+      cause instanceof AnalysisError
+        ? cause
+        : signal.aborted
+          ? new AnalysisCancelledError(MEASURE, { cause: signal.reason })
+          : new ProviderAdapterError(AGENTENV_PROVIDER, MEASURE, {
+              userMessage: `Measuring ${request.task} failed after ${String(solves.length)} solve(s): ${errorText(cause)}`,
+              remediationAction:
+                "Check the bundle files and host I/O error, then retry or retain the completed solve results.",
+              cause,
+            });
+    return err(retainSolves(failure));
+  }
+};
+
+/** Measure a task with each solve run by REA's helper in the build host's agentenv, with the build settings' defaults. */
+export const measureInFreshComputers = async (
+  request: MeasureRequest,
+  host: {
+    readonly environment: Readonly<NodeJS.ProcessEnv>;
+    readonly defaults: BuildComputerDefaults;
+    readonly retainCleanup: (owner: SolveCleanupOwner) => void;
+    readonly resources: ArtifactResourceScope;
+  },
+  signal: AbortSignal,
+  progress?: ProgressReporter,
+): Promise<Result<MeasuredTask, AnalysisError>> => {
+  const environment = await resolveBuildComputerEnvironment(
+    host.environment,
+    process.platform,
+    host.defaults,
+  );
+  return measureBuildTask(
+    request,
+    (solve, turn) =>
+      runSolveHelper({
+        ...solve,
+        environment,
+        signal: turn,
+        operation: MEASURE,
+        retainCleanup: host.retainCleanup,
+      }),
+    signal,
+    progress,
+    host.resources,
+  );
+};