mirror of
https://github.com/morluto/rea.git
synced 2026-11-16 19:53:50 +00:00
* fix(native): keep unreported value-flow truncation and archive decode coverage unknown * test(verify): count multiline optional-fact defaults and pin the guard's exclusions * fix(domain): keep unknown counts and costs unknown in comparison and coverage summaries (#11325) * fix(domain): keep unknown counts and costs unknown in comparison and coverage summaries * refactor(verify): exempt legitimate optional-fact defaults with a reasoned marker (#11355) * refactor(verify): exempt legitimate optional-fact defaults with a reasoned marker The optional-fact-default guard counted request defaults and absence that is modelled elsewhere as hidden unknowns. A comment `// optional-fact-default: <reason>` on the default's line or the line before now exempts it; the guard reads unstripped source for the marker and ignores matches inside comment lines. Marked: the two evidence paging offsets and four Swift literal reads whose absence swiftLiteralCoverage reports. Removed instead of marked: jsdiff 9 types Change.count as required, so its two `?? 0` defaults were dead; the evidence link traversal now carries retained records, so it no longer re-looks them up behind `?? []`. Baseline 31 -> 22. * fix(verify): blank comment spans instead of skipping comment-led lines in the optional-fact guard Skipping a match whose line starts like a comment hid live code after a leading block comment and counted defaults inside a block comment without leading stars. The counter now blanks comment text (keeping offsets and line breaks, skipping string literals) and matches the blanked code, while exemption markers are still read from the original lines. * refactor(domain): mark optional-fact defaults whose absence is not an observation (#11389) * refactor(domain): mark optional-fact defaults whose absence is not an observation Per-site review of the remaining optional-fact-default entries. Each marked default is a caller option (trace ignore_fields, environment rules), a list whose producer omits the empty case (unknown_size_frames, the schema's min-1 representation_node_ids), caveat text whose completeness another field carries (call diagnostics, unresolved calls, semantic resource limits), or absence that already turns the result unknown (Wasm interface inventory, WebMCP discovery, managed ExportedType rows, legacy integrity mismatch IDs). Left counted: webCaptureDiff redirects (2; the CDP producer always records them, so absence is a legacy record that should compare as unknown, not as no redirects) and HopperCallerReconciliation (2; index-miss semantics not yet verified). Baseline 22 -> 4. * fix(browser): compare redirects as unknown when a capture predates redirect recording (#11402) * fix(browser): compare redirects as unknown when a capture predates redirect recording The CDP producer records every request's redirect chain, empty when there was none; a record without one predates redirect recording (#986). Comparing it against a current capture reported the network dimension as unchanged, treating the missing chain as "no redirects". Redirects are now compared only when both captures recorded them, and the network dimension is otherwise unknown with a reason; proven membership changes are still reported. * refactor(hopper): interpret a caller-index miss once at the lookup (#11403) The forward index holds observations from the examined procedures only, and relationship coverage stays unknown with the scan facts as its reason. A miss therefore adds no callers without showing there are none; the lookup now says so once instead of four optional reads defaulting at each site. The optional-fact-default baseline is now empty. * docs(javascript): state why an omitted representation list adds no nodes * fix(verify): count optional-fact defaults from parsed TypeScript, not comment-blanked text The guard blanked comments with a hand-written scanner and matched a regular expression on the rest, so a default quoted in a string counted and a regular expression holding /* hid the code after it. It now walks the TypeScript syntax tree for property reads defaulted with ?? 0, ?? [] or ?? false, and reads exemption markers only from comment trivia. * fix(verify): tokenize optional-fact defaults without a compiler and count chained defaults * fix(verify): match whole optional-fact default expressions across calls and operators * fix(verify): count optional-fact defaults in template substitutions and after postfix updates, and tie markers to their own expression * refactor(verify): exempt generation and sample caps from the constant-maximum-slice guard by reason marker (#11454) * refactor(verify): exempt generation and sample caps from the constant-maximum-slice guard by reason marker * fix(verify): read reason markers only from real line comments * fix(verify): read reason markers from the shared source tokenizer * fix(verify): read a regular expression after a control head as a statement start The shared tokenizer treated every closing parenthesis as the end of an operand, so in `if (ready) /[/*]/.test(v)` it read `/[` as division and `/*` inside the regex as a block comment, hiding the rest of the file from the marker-aware guards. Parentheses are now tracked: the `)` closing an if/while/for/with head allows a regular expression, while a call's `)` still means division. * fix(verify): keep a keyword-named member call out of control-head detection `policy.if(ready) / (…)` was read as an if-statement head, so the division was taken for a regular expression and the denominator's optional-fact default was not counted. A control keyword after `.` or `?.` is a member call. * refactor(verify): exempt budget-arithmetic cursors and tallies by a not-an-allowance marker (#11493) * refactor(verify): exempt budget-arithmetic cursors and tallies by a not-an-allowance marker The budget-arithmetic guard cannot tell a scan cursor, an observed total, a string being assembled or a tail buffer's retained length from an allowance when each is compared with a limit. A field declaration marked `// not-an-allowance: <reason>` (read through the shared reason-marker scan) is no longer counted. The four remaining sites carry a marker, so the baseline is empty and removed. * refactor(verify): exempt detail levels, lifecycle states and domain vocabularies from the completeness-enum guard by reason marker (#11510) A zod enum containing "complete" is counted as a local coverage vocabulary. Eleven are not coverage: a caller-chosen detail level, how far a command or session ran, and format states (GPU decode depth, WASM name section, SQLite column details). They now carry `// detail-level:`, `// lifecycle-state:` or `// domain-vocabulary:` with a reason, read through the shared reason-marker scan (which gains an optional predicate for the matches a guard counts). local-completeness-enum: 22 -> 11 on this branch. * fix(verify): bind a not-an-allowance marker to its own class field Exemptions were a file-wide set of names, so a marked tally in one class hid an unmarked allowance with the same name in another, and a marked object property could exempt a class field. Markers now exempt only a member declaration (at class member depth) of the top-level class that contains it, and the named- allowance count is taken per class so the same scoping applies to it. * fix(artifacts): keep main's resources import in the Deno standalone reader after merging main * fix(verify): start an expression after a template substitution and a non-member keyword, and accept any marker reason The shared tokenizer read the `/` after `${` as division, so a regular expression holding `/*` in a substitution hid the rest of the file from every tokenizer-based guard, and it treated property names such as `row.return` as keywords. Tokens now record whether template text opens a substitution and whether a name is an expression keyword (not after `.` or `?.`), and the regular-expression rule and the optional-fact-default operand rule read those facts. The optional-fact-default guard reads markers through reason-marker's isReasonMarker, so any non-empty reason exempts, as for the other reason markers. * fix(verify): blank comments with the shared tokenizer before matching architecture guards (#11573) * fix(verify): blank comments with the shared tokenizer before matching architecture guards The guards stripped comments with a regular expression that took a `/*` inside a string, template or regular expression for a comment opener and deleted code up to the next `*/`. src/flutter/FlutterBuildProvider.ts's "lib/*/libapp.so" text hid four --pool-policy remediation strings from cli-option-text-outside-cli. blankComments in source-tokens.mjs now owns comment removal for the architecture guards and the reason-marker scan: it blanks comment tokens in place (newlines kept) and keeps strings, templates and regular expressions. The scan behind it retains only the last tokens it reads, and whitespace runs are skipped at once. The cli-option-text-outside-cli baseline records FlutterBuildProvider.ts at its true count of 4; no other baseline changed. Fixes #11477 * fix(verify): read a regular expression after a statement block and keep comment offsets for wide characters The tokenizer now records whether each `{` opens a statement block or an object literal (from the token before it) and marks the `}` that closes a block, after which a statement, and so a regular expression, can start; an object literal's `}` still ends an operand. blankComments replaces each run of comment text with one space per UTF-16 unit, so a surrogate pair no longer shifts later offsets. * chore(verify): drop the Flutter CLI-text baseline entry now that main names no option there
584 lines
19 KiB
TypeScript
584 lines
19 KiB
TypeScript
import * as buildEnvironment from "../../desktop/BuildComputerEnvironment.js";
|
|
import { ArtifactResourceScope } from "../../resources/ArtifactResourceScope.js";
|
|
import { renderSmokeTask } from "../../domain/desktop/appExport.js";
|
|
import { MEASURE_BUILD_TASK_CONTRACT } from "../../contracts/desktopBuildMeasureContract.js";
|
|
import {
|
|
guestFileAnswerCeiling,
|
|
guestFileAnswerWorkingBytes,
|
|
} from "../../desktop/AgentEnvComputer.js";
|
|
import { OwnedStagedOutput } from "../../filesystem/OwnedStagedOutput.js";
|
|
import { DesktopBuildOwnedWork } from "./DesktopBuildOwnedWork.js";
|
|
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
|
|
import { mkdir, readFile, truncate, writeFile } from "node:fs/promises";
|
|
import { join } from "node:path";
|
|
|
|
import { describe, expect, it, vi } from "vitest";
|
|
import * as hostFiles from "../../desktop/HostFileRead.js";
|
|
import { RegularFileCleanupFailure } from "../../filesystem/OwnedRegularFile.js";
|
|
import {
|
|
SolveCancelledError,
|
|
SolveCleanupError,
|
|
} from "../../desktop/AgentEnvSolveHelper.js";
|
|
|
|
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
|
import type { SolveReport } from "../../desktop/AgentEnvSolveHelper.js";
|
|
import {
|
|
measureBuildTask,
|
|
measureInFreshComputers,
|
|
type Solver,
|
|
} from "./DesktopBuildSolves.js";
|
|
|
|
/** A bundle holding stand-in steps and verifiers for these tasks, and a README. */
|
|
const bundleWith = async (...tasks: string[]) => {
|
|
const bundle = await createTestTempDirectory("measured-");
|
|
for (const task of tasks) {
|
|
await mkdir(join(bundle, "tasks"), { recursive: true });
|
|
await mkdir(join(bundle, "artifacts", `${task}-check`), {
|
|
recursive: true,
|
|
});
|
|
await writeFile(join(bundle, "tasks", `${task}.json`), `["${task}"]\n`);
|
|
await writeFile(
|
|
join(bundle, "artifacts", `${task}-check`, "verify.py"),
|
|
`# ${task}\n`,
|
|
);
|
|
}
|
|
await writeFile(join(bundle, "README.md"), "# App\n");
|
|
return bundle;
|
|
};
|
|
|
|
const passed: SolveReport = {
|
|
outcome: "passed",
|
|
instance_id: null,
|
|
seconds: 1,
|
|
checks: [{ id: "year", passed: true, description: "year" }],
|
|
turns: 10,
|
|
usd: 0.5,
|
|
agentenv: "0.9.1431",
|
|
error: null,
|
|
};
|
|
|
|
/** A solver that passes every solve, counting its calls. */
|
|
const counting = () => {
|
|
const calls: string[] = [];
|
|
const solver: Solver = async (request) => {
|
|
calls.push(request.task);
|
|
await new Promise((resolve) => setTimeout(resolve, 1));
|
|
return passed;
|
|
};
|
|
return { calls, solver };
|
|
};
|
|
|
|
const request = (bundle: string, task: string) => ({
|
|
bundle,
|
|
task,
|
|
model: "m",
|
|
agent: "claude-code",
|
|
budget_usd: 10,
|
|
});
|
|
|
|
describe("measureBuildTask's record of a bundle", () => {
|
|
it("reports a host capacity refusal rather than claiming the task is missing", async () => {
|
|
const bundle = await bundleWith("large");
|
|
await truncate(
|
|
join(bundle, "tasks", "large.json"),
|
|
guestFileAnswerWorkingBytes(guestFileAnswerCeiling()) + 1,
|
|
);
|
|
const { calls, solver } = counting();
|
|
const result = await measureBuildTask(
|
|
request(bundle, "large"),
|
|
solver,
|
|
new AbortController().signal,
|
|
);
|
|
expect(result.ok).toBe(false);
|
|
if (result.ok) throw new Error("Expected capacity refusal");
|
|
expect(result.error).toMatchObject({
|
|
_tag: "AnalysisResourceConstraintError",
|
|
resource: "capacity",
|
|
});
|
|
expect(calls).toEqual([]);
|
|
});
|
|
|
|
it("measures an exported task larger than 4 MiB", async () => {
|
|
const bundle = await bundleWith("large");
|
|
await writeFile(
|
|
join(bundle, "tasks", "large.json"),
|
|
JSON.stringify(["x".repeat(4 * 1024 * 1024)]),
|
|
);
|
|
const { calls, solver } = counting();
|
|
const result = await measureBuildTask(
|
|
request(bundle, "large"),
|
|
solver,
|
|
new AbortController().signal,
|
|
);
|
|
expect(result.ok).toBe(true);
|
|
expect(calls).toHaveLength(3);
|
|
});
|
|
|
|
it("records an unknown total cost and still measures the bundle's next task", async () => {
|
|
const bundle = await bundleWith("add-eggs", "add-bread");
|
|
const signal = new AbortController().signal;
|
|
const unpriced: Solver = async () => ({ ...passed, usd: null });
|
|
const first = await measureBuildTask(
|
|
request(bundle, "add-eggs"),
|
|
unpriced,
|
|
signal,
|
|
);
|
|
expect(first).toMatchObject({
|
|
ok: true,
|
|
value: { usd: null, stopped: "cost_unknown" },
|
|
});
|
|
const recorded = JSON.parse(
|
|
await readFile(join(bundle, "difficulty.json"), "utf8"),
|
|
) as { tasks: Record<string, { usd: unknown }> };
|
|
expect(recorded.tasks["add-eggs"]?.usd).toBeNull();
|
|
|
|
// The saved record with an unknown cost is read back, not refused.
|
|
const second = await measureBuildTask(
|
|
request(bundle, "add-bread"),
|
|
counting().solver,
|
|
signal,
|
|
);
|
|
expect(second.ok).toBe(true);
|
|
const readme = await readFile(join(bundle, "README.md"), "utf8");
|
|
expect(readme).toContain("cost unknown.");
|
|
expect(readme).toContain("$1.50 in all.");
|
|
});
|
|
|
|
it("keeps both records when two tasks of one bundle are measured at once", async () => {
|
|
const bundle = await bundleWith("add-eggs", "add-bread");
|
|
const { solver } = counting();
|
|
const signal = new AbortController().signal;
|
|
|
|
const results = await Promise.all([
|
|
measureBuildTask(request(bundle, "add-eggs"), solver, signal),
|
|
measureBuildTask(request(bundle, "add-bread"), solver, signal),
|
|
]);
|
|
|
|
for (const result of results) expect(result.ok).toBe(true);
|
|
const recorded = JSON.parse(
|
|
await readFile(join(bundle, "difficulty.json"), "utf8"),
|
|
) as { tasks: Record<string, unknown> };
|
|
expect(Object.keys(recorded.tasks).sort()).toEqual([
|
|
"add-bread",
|
|
"add-eggs",
|
|
]);
|
|
const readme = await readFile(join(bundle, "README.md"), "utf8");
|
|
expect(readme).toContain("`add-eggs`");
|
|
expect(readme).toContain("`add-bread`");
|
|
});
|
|
|
|
it.each([
|
|
["not JSON", "{not json"],
|
|
["records it cannot read", '{"tasks": {"add-bread": {"model": 1}}}'],
|
|
])(
|
|
"refuses a difficulty file holding %s before any solve, and leaves it as it was",
|
|
async (_label, content) => {
|
|
const bundle = await bundleWith("add-eggs");
|
|
await writeFile(join(bundle, "difficulty.json"), content);
|
|
const { calls, solver } = counting();
|
|
|
|
const result = await measureBuildTask(
|
|
request(bundle, "add-eggs"),
|
|
solver,
|
|
new AbortController().signal,
|
|
);
|
|
|
|
expect(result.ok).toBe(false);
|
|
if (!result.ok)
|
|
expect(result.error.userMessage).toContain("difficulty.json");
|
|
expect(calls).toEqual([]);
|
|
expect(await readFile(join(bundle, "difficulty.json"), "utf8")).toBe(
|
|
content,
|
|
);
|
|
expect(await readFile(join(bundle, "README.md"), "utf8")).toBe("# App\n");
|
|
},
|
|
);
|
|
});
|
|
|
|
it("retains a failed staged write and retries its cleanup on build shutdown", async () => {
|
|
const bundle = await bundleWith("add-eggs");
|
|
const work = new DesktopBuildOwnedWork();
|
|
const primary = new Error("publication failed");
|
|
const cleanup = new Error("cleanup failed");
|
|
const close = OwnedStagedOutput.prototype.close;
|
|
let owner: OwnedStagedOutput | undefined;
|
|
let attempts = 0;
|
|
const publication = vi
|
|
.spyOn(OwnedStagedOutput.prototype, "assertReadyForPublication")
|
|
.mockRejectedValueOnce(primary);
|
|
const release = vi
|
|
.spyOn(OwnedStagedOutput.prototype, "close")
|
|
.mockImplementation(async function (this: OwnedStagedOutput) {
|
|
owner = this;
|
|
if (++attempts === 1) throw cleanup;
|
|
await close.call(this);
|
|
});
|
|
try {
|
|
const result = await measureBuildTask(
|
|
request(bundle, "add-eggs"),
|
|
counting().solver,
|
|
new AbortController().signal,
|
|
undefined,
|
|
work.resources,
|
|
);
|
|
expect(result.ok).toBe(false);
|
|
if (result.ok) throw new Error("Expected failure");
|
|
expect(result.error.cleanupIncomplete).toBe(true);
|
|
expect(result.error.cleanup?.reason).toContain("cleanup failed");
|
|
expect(result.error.cause).toMatchObject({
|
|
outcome: { ok: false, error: primary },
|
|
});
|
|
expect(result.error.partialObservation).toMatchObject({
|
|
solves: [{}, {}, {}],
|
|
});
|
|
expect(await work.close()).toEqual([]);
|
|
expect(result.error.cleanupIncomplete).toBe(false);
|
|
expect(attempts).toBe(2);
|
|
} finally {
|
|
publication.mockRestore();
|
|
release.mockRestore();
|
|
await owner?.close();
|
|
}
|
|
});
|
|
|
|
it("retains completed paid solves when a later helper cannot be cleaned up", async () => {
|
|
const bundle = await bundleWith("add-eggs");
|
|
let calls = 0;
|
|
const solver: Solver = async () => {
|
|
calls += 1;
|
|
if (calls === 1) return passed;
|
|
throw new SolveCleanupError(
|
|
passed,
|
|
["process group 123"],
|
|
{ reason: "cleanup incomplete" },
|
|
"measure_build_task",
|
|
);
|
|
};
|
|
const result = await measureBuildTask(
|
|
request(bundle, "add-eggs"),
|
|
solver,
|
|
new AbortController().signal,
|
|
);
|
|
expect(result.ok).toBe(false);
|
|
if (result.ok) throw new Error("Expected cleanup failure");
|
|
expect(result.error.cleanupIncomplete).toBe(true);
|
|
expect(projectAnalysisError(result.error)).toMatchObject({
|
|
details: {
|
|
partial_observation: {
|
|
kind: "partial-task-measurement",
|
|
solves: [{ usd: 0.5 }, { usd: 0.5 }],
|
|
},
|
|
},
|
|
});
|
|
expect(result.error.partialObservation).toMatchObject({
|
|
kind: "partial-task-measurement",
|
|
bundle,
|
|
task: "add-eggs",
|
|
solves: [
|
|
{ outcome: "passed", turns: 10, usd: 0.5 },
|
|
{ outcome: "passed", turns: 10, usd: 0.5 },
|
|
],
|
|
});
|
|
});
|
|
|
|
it("returns a typed failure if difficulty preflight encounters an I/O error", async () => {
|
|
const bundle = await bundleWith("add-eggs");
|
|
const original = hostFiles.readHostFile;
|
|
const spy = vi
|
|
.spyOn(hostFiles, "readHostFile")
|
|
.mockImplementation(async (path, ...rest) => {
|
|
if (path.endsWith("difficulty.json"))
|
|
throw Object.assign(new Error("disk failure"), { code: "EIO" });
|
|
return original(path, ...rest);
|
|
});
|
|
try {
|
|
const { calls, solver } = counting();
|
|
const result = await measureBuildTask(
|
|
request(bundle, "add-eggs"),
|
|
solver,
|
|
new AbortController().signal,
|
|
);
|
|
expect(result.ok).toBe(false);
|
|
if (result.ok) throw new Error("Expected I/O failure");
|
|
expect(result.error).toMatchObject({
|
|
_tag: "ProviderAdapterError",
|
|
cause: expect.objectContaining({ code: "EIO" }),
|
|
});
|
|
expect(calls).toEqual([]);
|
|
} finally {
|
|
spy.mockRestore();
|
|
}
|
|
});
|
|
|
|
it.each([
|
|
["tasks/add-eggs.json", 1, false],
|
|
["difficulty.json", 1, false],
|
|
["README.md", 2, true],
|
|
] as const)(
|
|
"retains the failed close of %s, including cancellation during finalization",
|
|
async (suffix, occurrence, cancel) => {
|
|
const bundle = await bundleWith("add-eggs");
|
|
const work = new DesktopBuildOwnedWork();
|
|
const controller = new AbortController();
|
|
const reason = new Error("cancel during close");
|
|
const original = hostFiles.readHostFile;
|
|
let reads = 0;
|
|
let closes = 0;
|
|
const spy = vi
|
|
.spyOn(hostFiles, "readHostFile")
|
|
.mockImplementation(async (path, ...rest) => {
|
|
if (path.endsWith(suffix) && ++reads === occurrence) {
|
|
if (cancel) controller.abort(reason);
|
|
throw hostFiles.hostReadFailure(
|
|
new RegularFileCleanupFailure({
|
|
path,
|
|
owner: {
|
|
close: async () => {
|
|
if (++closes === 1) throw new Error("still busy");
|
|
},
|
|
} as never,
|
|
outcome: { kind: "completed", value: Buffer.from("read") },
|
|
cleanupCause: new Error("close failed"),
|
|
...(cancel ? { completionCancellation: { reason } } : {}),
|
|
}),
|
|
);
|
|
}
|
|
return original(path, ...rest);
|
|
});
|
|
try {
|
|
const result = await measureBuildTask(
|
|
request(bundle, "add-eggs"),
|
|
counting().solver,
|
|
controller.signal,
|
|
undefined,
|
|
work.resources,
|
|
);
|
|
expect(result.ok).toBe(false);
|
|
if (result.ok) throw new Error("Expected cleanup failure");
|
|
expect(result.error.cleanupIncomplete).toBe(true);
|
|
expect(result.error.cleanupResources).toContain(join(bundle, suffix));
|
|
if (cancel) {
|
|
expect(result.error._tag).toBe("AnalysisCancelledError");
|
|
expect(result.error.cause).toBe(reason);
|
|
expect(result.error.partialObservation).toMatchObject({
|
|
kind: "partial-task-measurement",
|
|
solves: [{}, {}, {}],
|
|
});
|
|
}
|
|
expect(await work.close()).toHaveLength(1);
|
|
expect(result.error.cleanupIncomplete).toBe(true);
|
|
expect(await work.close()).toEqual([]);
|
|
expect(result.error.cleanupIncomplete).toBe(false);
|
|
} finally {
|
|
spy.mockRestore();
|
|
await work.close();
|
|
}
|
|
},
|
|
);
|
|
|
|
it("keeps an input read failure primary when cancellation also arrives during close", async () => {
|
|
const bundle = await bundleWith("add-eggs");
|
|
const work = new DesktopBuildOwnedWork();
|
|
const controller = new AbortController();
|
|
const original = hostFiles.readHostFile;
|
|
const spy = vi
|
|
.spyOn(hostFiles, "readHostFile")
|
|
.mockImplementation(async (path, ...rest) => {
|
|
if (!path.endsWith("tasks/add-eggs.json")) return original(path, ...rest);
|
|
controller.abort(new Error("cancel"));
|
|
throw hostFiles.hostReadFailure(
|
|
new RegularFileCleanupFailure({
|
|
path,
|
|
owner: { close: async () => {} } as never,
|
|
outcome: { kind: "failed", cause: new Error("read failed") },
|
|
cleanupCause: new Error("close failed"),
|
|
}),
|
|
);
|
|
});
|
|
try {
|
|
const result = await measureBuildTask(
|
|
request(bundle, "add-eggs"),
|
|
counting().solver,
|
|
controller.signal,
|
|
undefined,
|
|
work.resources,
|
|
);
|
|
expect(result.ok).toBe(false);
|
|
if (result.ok) throw new Error("Expected failure");
|
|
expect(result.error._tag).toBe("ProviderAdapterError");
|
|
expect(result.error.cleanupIncomplete).toBe(true);
|
|
expect(await work.close()).toEqual([]);
|
|
expect(result.error.cleanupIncomplete).toBe(false);
|
|
} finally {
|
|
spy.mockRestore();
|
|
await work.close();
|
|
}
|
|
});
|
|
|
|
it("retains a solve when cancellation arrives after the helper report", async () => {
|
|
const bundle = await bundleWith("add-eggs");
|
|
const reason = { stage: "final cleanup" };
|
|
const result = await measureBuildTask(
|
|
request(bundle, "add-eggs"),
|
|
async () => {
|
|
throw new SolveCancelledError(passed, "measure_build_task", reason);
|
|
},
|
|
new AbortController().signal,
|
|
);
|
|
expect(result.ok).toBe(false);
|
|
if (result.ok) throw new Error("Expected cancellation");
|
|
expect(result.error).toMatchObject({
|
|
_tag: "AnalysisCancelledError",
|
|
cause: reason,
|
|
partialObservation: {
|
|
kind: "partial-task-measurement",
|
|
solves: [{ turns: 10, usd: 0.5 }],
|
|
},
|
|
});
|
|
});
|
|
|
|
it("retains completed solves when the destination becomes unreadable before publication", async () => {
|
|
const bundle = await bundleWith("add-eggs");
|
|
let calls = 0;
|
|
const result = await measureBuildTask(
|
|
request(bundle, "add-eggs"),
|
|
async () => {
|
|
calls += 1;
|
|
if (calls === 3)
|
|
await writeFile(join(bundle, "difficulty.json"), "invalid");
|
|
return passed;
|
|
},
|
|
new AbortController().signal,
|
|
);
|
|
expect(result.ok).toBe(false);
|
|
if (result.ok) throw new Error("Expected refusal");
|
|
expect(result.error.partialObservation).toMatchObject({
|
|
kind: "partial-task-measurement",
|
|
solves: [{ usd: 0.5 }, { usd: 0.5 }, { usd: 0.5 }],
|
|
});
|
|
expect(await readFile(join(bundle, "difficulty.json"), "utf8")).toBe(
|
|
"invalid",
|
|
);
|
|
});
|
|
|
|
it.each(["../../outside", "nested/task", "nested\\task", ".", ".."])(
|
|
"refuses invalid selected task names before any file read or solve: %s",
|
|
async (task) => {
|
|
const read = vi
|
|
.spyOn(hostFiles, "readHostFile")
|
|
.mockRejectedValue(new Error("invalid task must not reach filesystem"));
|
|
const solve = vi.fn<Solver>();
|
|
try {
|
|
const result = await measureBuildTask(
|
|
request("/unused-bundle", task),
|
|
solve,
|
|
new AbortController().signal,
|
|
);
|
|
expect(result).toMatchObject({
|
|
ok: false,
|
|
error: { _tag: "AnalysisInputError", issues: [{ path: ["task"] }] },
|
|
});
|
|
expect(read).not.toHaveBeenCalled();
|
|
expect(solve).not.toHaveBeenCalled();
|
|
} finally {
|
|
read.mockRestore();
|
|
}
|
|
},
|
|
);
|
|
|
|
it("keeps the portable smoke name selectable independently of authoring reservations", async () => {
|
|
const bundle = await bundleWith("smoke");
|
|
const { solver, calls } = counting();
|
|
const result = await measureBuildTask(
|
|
request(bundle, "smoke"),
|
|
solver,
|
|
new AbortController().signal,
|
|
);
|
|
expect(result.ok).toBe(true);
|
|
expect(calls).toEqual(["smoke", "smoke", "smoke"]);
|
|
});
|
|
|
|
it("refuses invalid measurement identity before resolving the fresh-computer environment", async () => {
|
|
const environment = vi
|
|
.spyOn(buildEnvironment, "resolveBuildComputerEnvironment")
|
|
.mockRejectedValue(
|
|
new Error("invalid task must not resolve an environment"),
|
|
);
|
|
const read = vi.spyOn(hostFiles, "readHostFile");
|
|
const resources = new ArtifactResourceScope();
|
|
try {
|
|
const result = await measureInFreshComputers(
|
|
request("/unused-bundle", "../../outside"),
|
|
{
|
|
environment: {},
|
|
defaults: {},
|
|
retainCleanup: () => undefined,
|
|
resources,
|
|
},
|
|
new AbortController().signal,
|
|
);
|
|
expect(result).toMatchObject({
|
|
ok: false,
|
|
error: { _tag: "AnalysisInputError", issues: [{ path: ["task"] }] },
|
|
});
|
|
expect(environment).not.toHaveBeenCalled();
|
|
expect(read).not.toHaveBeenCalled();
|
|
} finally {
|
|
environment.mockRestore();
|
|
read.mockRestore();
|
|
await resources.close();
|
|
}
|
|
});
|
|
|
|
it("advertises the same portable lookup names without changing their spelling", () => {
|
|
for (const task of ["add-eggs", "123-task", "a".repeat(63), "smoke"])
|
|
expect(
|
|
MEASURE_BUILD_TASK_CONTRACT.inputSchema.parse(
|
|
request("/unused-bundle", task),
|
|
).task,
|
|
).toBe(task);
|
|
for (const task of [
|
|
"../../outside",
|
|
"nested/task",
|
|
"nested\\task",
|
|
".",
|
|
"..",
|
|
"con",
|
|
"123",
|
|
"a".repeat(64),
|
|
])
|
|
expect(
|
|
MEASURE_BUILD_TASK_CONTRACT.inputSchema.safeParse(
|
|
request("/unused-bundle", task),
|
|
).success,
|
|
).toBe(false);
|
|
});
|
|
|
|
it("keeps a generated deploy-only smoke task inapplicable without its measurement verifier", async () => {
|
|
const bundle = await createTestTempDirectory("measured-smoke-");
|
|
await mkdir(join(bundle, "tasks"));
|
|
await mkdir(join(bundle, "artifacts", "check_app"), { recursive: true });
|
|
await writeFile(
|
|
join(bundle, "tasks", "smoke.json"),
|
|
renderSmokeTask({ name: "app" }),
|
|
);
|
|
await writeFile(
|
|
join(bundle, "artifacts", "check_app", "verify.py"),
|
|
"# generated deployment check\n",
|
|
);
|
|
const solve = vi.fn<Solver>();
|
|
const result = await measureBuildTask(
|
|
request(bundle, "smoke"),
|
|
solve,
|
|
new AbortController().signal,
|
|
);
|
|
expect(result).toMatchObject({
|
|
ok: false,
|
|
error: {
|
|
_tag: "AnalysisInputError",
|
|
userMessage: expect.stringContaining("artifacts/smoke-check/verify.py"),
|
|
},
|
|
});
|
|
expect(solve).not.toHaveBeenCalled();
|
|
});
|