Merge main ca3fd7ba into feat/recorder-saved-values

This commit is contained in:
Edgar Arakelyan committed 2026-10-26 14:26:51 -07:00
commit 3de21ef5da
83 files changed
+3685 -584

No files matched your search

+3
View File
@@ -753,6 +753,9 @@ jobs:
- name: Run native Apple and process boundary regressions
run: >-
npm run test:focused --
tests/acceptance/artifactDmgAttachmentRecoveryCli.test.ts
tests/boundary/providers/native/nativeDmgArtifactReader.test.ts
tests/process-global/dmgCancellationReason.test.ts
tests/boundary/filesystem/installScript.test.ts
tests/boundary/filesystem/nibNativeHierarchy.test.ts
tests/boundary/filesystem/interfaceBuilderBundle.test.ts
+5 -1
View File
@@ -183,6 +183,7 @@ what to record next; hints never change what is served:
| `side-effect` | The write changed something besides its own entity, which the environment does not model. |
| `unconfirmed` | No later read showed the effect. Read the entity and its list after the write. |
| `incomplete-entities` | The write is confirmed, but the views that show its entity cannot render it in full. Read the written entity through every view that lists it. |
| `unknown-clock` | The learned rules generate timestamps, but the capture establishes no global served clock start. Record usable start times and completed durations for every request. |
The bundle ships `evidence.json`, the whole capture Evidence, response bodies
included. Inspection excludes the values of the standard authentication headers
@@ -771,7 +772,10 @@ any. A refused call leaves the state unchanged. Values a write generates are
deterministic: the clock starts at `write_clock` and advances one second per
served write, sequences take one past both the largest held value and the
largest they generated, and UUIDs and tokens derive from the state Evidence
id, tool, field and write count, skipping held values. `data/reset` and
id, tool, field and write count, skipping held values. If `write_clock` is null,
plans needing `clock`, `held-clock`, or `nested-clock` retain their learned rules
but are withheld with `unknown-clock`; other confirmed plans can still be served.
`data/reset` and
`data/add` restart the clock, counter and sequences, so the same calls after a
reset give the same state. Answers carry `state: {"source": "state", "view":
"create" | "update" | "delete", …}` or `"document-write"`; a bulk answer adds
+3 -2
View File
@@ -589,8 +589,9 @@ the browser's wall-clock start, and `elapsed_ms`, its time from that start to
the end of the response, when the browser reports them. The environment views
time scenario records from these, as HAR entries are timed: the browser's start
where it reported one, else `observed_at`, which trails it by up to tens of
milliseconds while REA reads other responses, and a duration of 0 where the
browser reported none. Captures made before these fields existed lack them and
milliseconds while REA reads other responses. A duration is retained only from a
unique terminal event that reports it; missing or ambiguous durations stay
omitted, while an explicitly reported zero remains zero. Captures made before these fields existed lack them and
are read in record order. REA versions before them refuse captures that carry
them. Scenario comparisons ignore all three (`ignore_network_request_timing`).
Requests sharing a URL remain distinct. Redirects link to the preceding
+1 -1
View File
@@ -53,7 +53,7 @@ Every collection supports `fields`, `offset` and `limit`; its stable row ID is
always included. Selection changes delivery, not source inspection. Pages keep
source coverage and parent Evidence links. Page exhaustion does not establish
absence in a partial source. Complete Evidence remains accessible through
`get_evidence` and Evidence bundle export.
`get_evidence_bundle` and Evidence bundle export.
The summary's `probe_issues` counts files with reported issues and their retained
issue entries. It previews at most two such files, including stable `file_id`,
+1 -1
View File
@@ -275,7 +275,7 @@ Additional lanes are explicit, so the base lane stays small:
REA_FIRMWARE_VERIFY_EXT4=1 npm run fixtures:firmware
REA_FIRMWARE_VERIFY_EXT4=1 npm run verify:firmware
# Existing Ghidra 12.1.x and its declared JDK required.
# Existing Ghidra 12.x (>=12.1) and its declared JDK required.
REA_FIRMWARE_VERIFY_GHIDRA=1 npm run verify:firmware
```
+1 -1
View File
@@ -3,7 +3,7 @@
REA can analyze DOS MZ executables and explicitly selected COM images through the bring-your-own Ghidra adapter.
The Linux x64 and macOS arm64 verification lanes exercise actual 16-bit disassembly and
decompilation through the CLI and stdio MCP. Provider admission accepts Ghidra
12.1.x and that installation's declared JDK range. The verification lane uses
12.x (>=12.1) and that installation's declared JDK range. The verification lane uses
Ghidra 12.1.4 and JDK 21; no DOS emulator or cross-compiler is required.
## Open and inspect
+4 -3
View File
@@ -982,7 +982,7 @@ copying license secrets into logs, or killing unrelated Hopper processes.
REA connects to an existing Ghidra installation on Linux x64/arm64, macOS x64/arm64,
or experimental Windows x64 P0.
It accepts Ghidra 12.1.x and the 64-bit full JDK declared by that installation's
It accepts Ghidra 12.x (>=12.1) and the 64-bit full JDK declared by that installation's
`application.java.min` and `application.java.max`. The JDK's reported platform must also
map to the same native decompiler directory as the host: a JDK running under
emulation (for example x86_64 under Rosetta on Apple silicon) or reporting a
@@ -990,7 +990,8 @@ non-host `os.name` fails the doctor
even when it is an otherwise accepted 64-bit full JDK. Select a matching JDK
through `JAVA_HOME`; adding another decompiler in the JVM-selected directory
does not satisfy this host-validated admission rule. Current 12.1 releases require
JDK 21 or newer and set no maximum. The bridge is verified with Ghidra 12.1.4
JDK 21 or newer; Ghidra 12.2 declares JDK 25 or newer. Neither sets a maximum.
The bridge is verified with Ghidra 12.1.4
and JDK 21. Each installation must include the native decompiler for the host
architecture in `Ghidra/Features/Decompiler/os/<platform>/` or the corresponding
`build/os/<platform>/` directory. Linux ARM64 uses `linux_arm_64`; official
@@ -1105,7 +1106,7 @@ the application metadata and headless launcher. An explicitly supplied
REA reports that error instead of silently selecting another installation.
This discovery is read-only and does not execute the Ghidra launcher.
Doctor validates the platform, architecture, Ghidra 12.1.x application version,
Doctor validates the platform, architecture, Ghidra 12.x (>=12.1) application version,
`support/analyzeHeadless` or `support/analyzeHeadless.bat`, the installation's
Java major range, 64-bit JDK bitness, and the presence of `javac`/`javac.exe`.
It also checks that the reported JVM `os.arch` and `os.name` map to the same native decompiler
+11 -3
View File
@@ -125,9 +125,17 @@ REA writes Kuna's exports only inside `REA_KUNA_CACHE_DIR`:
- `trash/` holds retired builds and evicted exports awaiting removal.
- `home/` is Kuna's HOME.
The cache key includes the selected launcher and resolved executable paths.
Older exports without that identity are rebuilt automatically and retired through
normal cache cleanup; no manual migration is needed.
The cache key includes the selected launcher and resolved executable paths, plus
the target's native basename: that name determines the export filenames and
appears in generated source. REA accepts one current manifest layout. Older
exports are rebuilt on demand and retired through normal owned-cache cleanup;
no manual deletion or migration is needed. Manifests must name the producer's
exact flat file layout, with matching basenames and producing inputs. Replaced
entry, project or manifest symlinks are refused before their targets are read;
an unverified `projects/` parent grants no removal authority. Warm reuse checks
the manifest's file identity as well as the export files. These refusal and
migration cases are verified with controlled Kuna-format exports; they do not
establish behavior of an independently installed Kuna release.
Builds publish by one rename, so a reader sees a complete export or none. If later staging retirement fails, the typed cleanup error retains the completed export's key, directory and manifest in `published_export`. The session adopts that verified project; later queries reuse its immutable files while `project_export.cleanup_incomplete` reports the retained owner's live state. `close_binary` retries only the remaining cleanup, and a successful retry resolves prior cleanup errors. A failed export followed by failed staging cleanup retains both failures; it does not invent a publication receipt. A session keeps the export it uses from eviction only within its own REA process. If another REA process sharing the cache evicts it, the session rebuilds it; meanwhile pseudocode falls back to single-function runs. A damaged or edited export is removed and rebuilt. REA retains the file state verified with each digest, checks it before warm-cache reuse, and checks open descriptors before and after reading text. Assembly indexes are checked against the same state; a changed file makes the whole report retry rather than mixing facets from different exports. Staging moves to `trash/` for removal only after REA proves its Kuna process stopped. Staging of a process it cannot prove stopped is never swept; REA keeps it owned, retries the stop on `close_binary`, and reports it until the stop is proven. Failed file removal also stays bound to its export and is retried by `close_binary`; closing does not report success while that cleanup remains. `rea doctor --provider kuna` reports the cache directory without creating it.
+1 -1
View File
@@ -1005,7 +1005,7 @@ is present. A denied trace reports the `debugger-attach-denied` reason.
## Provider and verification boundaries
Install a Ghidra 12.1.x release and the 64-bit full JDK it declares, then configure REA to
Install a Ghidra 12.x (>=12.1) release and the 64-bit full JDK it declares, then configure REA to
use them. Ghidra analysis supports Linux x64/arm64 and macOS x64/arm64; each installation requires
the matching native decompiler. Experimental Windows x64 P0 admits native
x86, x86-64, ARMNT (Thumb-2) and ARM64 PE applications and DLLs, plus explicitly selected
@@ -17,6 +17,7 @@
6. Tests: the collector stamps both fields, records take them, the state learns
the HAR's clock and verdicts from a timed scenario and none from an untimed
or partly untimed one, a request that never ended or ended twice falls back
to its observation, comparisons ignore the times (also against an untimed
to its observation with unknown duration omitted, an explicit zero remains
observed, comparisons ignore the times (also against an untimed
capture), and the real-Chrome network lane checks every request is observed
and every finished request carries the browser's start and duration.
@@ -27,9 +27,9 @@ scenario capture learns the clock a HAR does.
Evidence that carries them.
- Scenario records take `startedDateTime` from the transaction's single
finished or failed event's `started_at`, else from `observed_at`, and `time`
from its `elapsed_ms` (0 without one), and say so in a limitation; a capture
whose requests are all timed gets the recorded-start-time timeline and a
write clock.
from its `elapsed_ms`, omitted without a unique reported duration, and say so
in a limitation. An explicit zero remains zero. Complete starts establish the
recorded-start-time timeline; a write clock also requires known completions.
- Scenario comparisons ignore both fields, as a committed built-in rule
(`ignore_network_request_timing`).
- The added metadata is charged to the event budgets as other metadata is.
@@ -543,7 +543,11 @@ contradicts it. Otherwise the reason, in this precedence, is:
never showed), so it could answer only for entities the environment itself
created. An update's answer renders the entity it wrote, so a field one of
its rules writes counts as held, except an increment's, which adds only to
a held value.
a held value;
6. `unknown-clock`: an otherwise servable plan needs `clock`, `held-clock`, or
`nested-clock`, but the capture establishes no global served clock start.
Its learned rules and confirming reads remain retained, and plans without
this clock dependency remain eligible for serving.
`checked_record_ordinals` lists the confirming reads.
+1 -1
View File
@@ -989,7 +989,7 @@ commands. Check prerequisites before starting expensive work and name the
missing command, target, and lane in any failure message. A lane must not imply
that a host or target is covered when it was skipped.
Provider admission accepts Ghidra 12.1.x and the JDK range declared by that
Provider admission accepts Ghidra 12.x (>=12.1) and the JDK range declared by that
installation (`application.java.min` through `application.java.max`). Current
12.1 releases require JDK 21 or newer and set no maximum. The lanes below still
prove behavior on the verified Ghidra 12.1.4 and JDK 21 build.
+3 -2
View File
@@ -855,10 +855,11 @@ checked only on the members they sent; a mismatch no write explains marks the re
`side-effect`. A plan is `served` when a read confirmed it and none
contradicted it; otherwise its `reason` is the first of its build reason
(`no-call`, `body-unusable`, `no-observation`, `unexplained-change`),
`contradicted` (with `contradicted_by`), `side-effect`, `unconfirmed` and `incomplete-entities` (an update or delete that answers with the entity, when no initial entity holds every field its recorded answer shows, counting for an update the fields its rules write, so it could answer only for entities the environment created).
`contradicted` (with `contradicted_by`), `side-effect`, `unconfirmed` and `incomplete-entities` (an update or delete that answers with the entity, when no initial entity holds every field its recorded answer shows, counting for an update the fields its rules write, so it could answer only for entities the environment created). An otherwise served plan whose learned rules generate clock values is withheld with `unknown-clock` when no global clock start is established; its learned rules and confirming reads remain retained.
`checked_record_ordinals` lists the confirming reads, `insert` places created
entities (sorted by a field the collection's recorded order shows, else last),
and `write_clock` is the end of the last timed record, to the next second.
and `write_clock` is the end of the last timed record, to the next second, when
all request completions are known; otherwise it is null.
Values in the state are recorded data, so the view refuses the same parents as
the manifest, including those inspected with `sensitive_values`, and a
+1 -1
View File
@@ -15,7 +15,7 @@ additional permission flag or degraded mode is required.
- Windows 10+ x64 with fixed local NTFS target and temporary volumes;
- Node.js 22.x (>=22.23.4), 24.x (>=24.22.1), or 26.12.1+;
- an operator-installed Ghidra 12.1.x distribution (verified with 12.1.4);
- an operator-installed Ghidra 12.x (>=12.1) distribution (verified with 12.1.4);
- a 64-bit full JDK inside that installation's declared Java range (JDK 21 or newer, with no maximum, for current 12.1 releases);
- an explicit native, non-managed x86, x86-64, ARMNT (Thumb-2) or ARM64 PE application/DLL, or a
[raw NMOS 6502 image](raw-binary.md#nmos-6502) with a caller-supplied entry; and
+4
View File
@@ -67,6 +67,7 @@ export const SCOPES = {
],
// Apple artifact CLI/MCP lane on macOS, including its focused regressions.
apple: [
"src/artifacts/NativeDmg*.ts",
"src/native/Apple*",
"src/native/NativeMacOS*",
"src/native/NativeUi*",
@@ -74,6 +75,9 @@ export const SCOPES = {
"bridge/native/**",
"bridge/process/**",
"scripts/verify/apple/**",
"tests/acceptance/artifactDmgAttachmentRecoveryCli.test.ts",
"tests/boundary/providers/native/nativeDmgArtifactReader.test.ts",
"tests/process-global/dmgCancellationReason.test.ts",
"tests/boundary/filesystem/installScript.test.ts",
"tests/boundary/filesystem/nibNativeHierarchy.test.ts",
"tests/boundary/filesystem/interfaceBuilderBundle.test.ts",
+26 -15
View File
@@ -71,22 +71,33 @@ export const managedAnalysisCleanupFailure = (
result: Result<AnalysisExecution, AnalysisError>,
error: AnalysisError,
): Result<AnalysisExecution, AnalysisError> => {
if (!result.ok)
return err(analysisErrorWithCleanupFailure(result.error, error));
// A close rejection without a live cleanup state has unknown completion;
// an empty dependency set must not turn that rejection into confirmed cleanup.
if (!result.ok) {
const failure = analysisErrorWithCleanupFailure(result.error, error);
return err(
error.cleanupIncomplete
? failure.bindCleanupDependencies([result.error, error])
: failure,
);
}
const cleanup = projectAnalysisError(error);
return err(
new ProviderAdapterError(
error instanceof ProviderAdapterError ? error.providerId : "rea",
"close_binary",
{
cause: error,
remediationAction: cleanup.remediation.action,
cleanup: { reason: cleanup.message, resources: error.cleanupResources },
diagnostics: {
cleanup_error: cleanup,
partial_observation: jsonValueSchema.parse(result.value),
},
const failure = new ProviderAdapterError(
error instanceof ProviderAdapterError ? error.providerId : "rea",
"close_binary",
{
cause: error,
remediationAction: cleanup.remediation.action,
cleanup: { reason: cleanup.message, resources: error.cleanupResources },
diagnostics: {
cleanup_error: cleanup,
partial_observation: jsonValueSchema.parse(result.value),
},
),
},
);
return err(
error.cleanupIncomplete
? failure.bindCleanupDependencies([error])
: failure,
);
};
+1 -3
View File
@@ -10,7 +10,6 @@ import {
spawnOwnedProviderProcess,
} from "../process/ProviderProcess.js";
import { cleanupOwnedProcessGroup } from "../process/ProcessOwnership.js";
import { waitForAbortableDelay } from "../process/ProviderDeadline.js";
import { ownedCommandResources } from "../process/OwnedCommand.js";
import { tmpdir } from "node:os";
import { BoundedOutputExcerpt } from "../process/BoundedOutputExcerpt.js";
@@ -193,8 +192,7 @@ export const launchRegistrationRunnerCommand: RegistrationRunnerLauncher = (
launched.process.stderr?.on("data", (chunk: Buffer) =>
stderr.append(chunk),
);
while (!supervisor.observedOutputClose() && !controller.signal.aborted)
await waitForAbortableDelay(25, controller.signal);
await supervisor.waitForOutputClose(undefined, controller.signal);
} catch (cause: unknown) {
if (!controller.signal.aborted)
spawnError = cause instanceof Error ? cause.message : String(cause);
@@ -683,6 +683,11 @@ const run = (value: unknown) =>
describe("probe request chronology", () => {
it.each([
["Tue, 20 Oct 2026 09:00:01 GMT", "Tue, 20 Oct 2026 09:00:00 GMT"],
[
"Tue, 20 Oct 2026 14:30:01 GMT+0530",
"Tue, 20 Oct 2026 14:30:00 GMT+0530",
],
["2026-10-20T09:00:00.0002Z", "2026-10-20T09:00:00.0001Z"],
["2026-10-20T09:00:00.0000000002Z", "2026-10-20T09:00:00.0000000001Z"],
["0001-01-01T00:00:00.0002Z", "0001-01-01T01:00:00.0001+01:00"],
@@ -18,8 +18,7 @@ import { parseExactJson } from "../../domain/exactJsonNumber.js";
import type { JsonValue } from "../../domain/jsonValue.js";
import {
compareOffsetDateTimeInstants,
offsetDateTimeInstantFromEpochMilliseconds,
inspectRecordedDateTime,
readRecordedStart,
type OffsetDateTimeInstant,
} from "../../domain/offsetDateTime.js";
import type { WebNetworkCaptureRecord } from "../../domain/webNetworkCapture.js";
@@ -55,14 +54,7 @@ const startedInstant = (
): OffsetDateTimeInstant | undefined => {
const started = startedSchema.safeParse(record.reported);
if (!started.success) return undefined;
const reading = inspectRecordedDateTime(started.data.startedDateTime);
if (reading.kind === "valid") return reading.reading.instant;
if (reading.kind === "invalid") return undefined;
// Preserve native-Date admission for legacy spellings, at its original precision.
const ms = Date.parse(started.data.startedDateTime);
return Number.isFinite(ms)
? offsetDateTimeInstantFromEpochMilliseconds(ms)
: undefined;
return readRecordedStart(started.data.startedDateTime)?.instant;
};
/** Whether a probe makes calls that carry a request body: a POST, PUT or PATCH. */
@@ -1,3 +1,7 @@
import { EnvironmentStateStore } from "./environmentStateStore.js";
import { environmentToolAnswer } from "./environmentToolAnswer.js";
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import type { StateWritePlan } from "../../domain/analysisView/environmentStateWriteRules.js";
import { describe, expect, it } from "vitest";
import type { TimedCaptureRow } from "../../../tests/fixtures/analysisView.js";
@@ -274,3 +278,48 @@ describe("the state size limit", () => {
expect(books.snapshot()).toBe(before);
});
});
it.each(["clock", "held-clock", "nested-clock"] as const)(
"refuses a legacy served %s plan with an unknown clock before changing state or generators",
(rule) => {
const fixture = servedFrom();
const original = fixture.state.writes.get("patch_books_by_id");
if (original === undefined || !original.served)
throw new Error("Expected served fixture update");
const format = { kind: "epoch", unit: "ms" } as const;
const plan: StateWritePlan = {
...original,
rules:
rule === "nested-clock"
? [{ field: "updated_at", rule, format, members: ["at"] }]
: [{ field: "updated_at", rule, format }],
};
const writes = new Map(fixture.state.writes).set(plan.tool, plan);
const state = { ...fixture.state, writeClock: null, writes };
const store = new EnvironmentStateStore(state);
const respond = environmentToolAnswer(fixture.source, store);
const tool = fixture.source.manifest.tools.find(
(candidate) => candidate.name === plan.tool,
);
const deletion = fixture.source.manifest.tools.find(
(candidate) => candidate.name === "delete_books_by_id",
);
if (tool === undefined || deletion === undefined)
throw new Error("Expected fixture tools");
const before = JSON.stringify(store.snapshot());
const pending = store.pendingWrite();
const result = respond(tool, { id: "1", body: { name: "Refused" } });
if (result.ok) throw new Error("Expected unknown clock refusal");
expect(result.error._tag).toBe("AnalysisInputError");
expect(projectAnalysisError(result.error)).toMatchObject({
code: "invalid_request",
message: expect.stringContaining("requires a known write clock"),
});
expect(JSON.stringify(store.snapshot())).toBe(before);
expect(store.pendingWrite()).toEqual(pending);
expect(store.state.writes.get(plan.tool)).toBe(plan);
// Independent writes keep working; the guard does not erase imported metadata.
expect(respond(deletion, { id: "1" }).ok).toBe(true);
expect(JSON.stringify(store.snapshot())).not.toBe(before);
},
);
@@ -1,5 +1,6 @@
import { usesWriteClock } from "../../domain/analysisView/environmentStateWriteRules.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import type { Result } from "../../domain/result.js";
import { err, type Result } from "../../domain/result.js";
import {
replayEnvironmentTool,
type ReplayArguments,
@@ -9,7 +10,7 @@ import type {
EnvironmentManifestTool,
EnvironmentSource,
} from "./environmentSource.js";
import { answerFromState } from "./environmentStateAnswer.js";
import { answerFromState, refused } from "./environmentStateAnswer.js";
import { answerBulk } from "./environmentStateBulkAnswer.js";
import type { EnvironmentStateStore } from "./environmentStateStore.js";
import { answerWrite } from "./environmentStateWriteAnswer.js";
@@ -34,6 +35,13 @@ export const environmentToolAnswer = (
const plan = store.state.writes.get(tool.name);
if (plan?.served !== true)
return answerFromState({ source, store }, tool, args, observedQuery);
if (store.state.writeClock === null && usesWriteClock(plan))
return err(
refused(
tool,
`Tool ${tool.name} requires a known write clock; this state's capture completion timing is unknown. Capture complete timing and derive the state again before calling this tool.`,
),
);
const request = pinnedRequest(
{ source, store },
{
@@ -146,14 +146,10 @@ export const reconstructJavaScriptArtifact = async (
const earlyCleanup = await scope.release(readerOwner);
readerOpen = false;
if (earlyCleanup.kind === "failed")
throw ArtifactReaderFailure.withCleanup(
earlyCleanup.cause,
ArtifactReaderFailure.cleanupObservation(
earlyCleanup.cause,
readerOwner.resource,
),
{ kind: "artifact-inventory", inventory: snapshot },
);
throw scope.cleanupFailure({ kind: "completed" }, [earlyCleanup], {
kind: "artifact-inventory",
inventory: snapshot,
});
await reportPhase(
"parse_javascript_sources",
`Processing ${String(files.files.length - (files.source_selection?.unselected_files ?? 0))} selected application files`,
@@ -410,19 +406,13 @@ export const reconstructJavaScriptArtifact = async (
if (readerOpen) {
const cleanupAttempt = await scope.release(readerOwner);
if (cleanupAttempt.kind === "failed") {
const cleanup = ArtifactReaderFailure.cleanupObservation(
cleanupAttempt.cause,
readerOwner.resource,
);
const partialObservation: ArtifactInventoryPartialObservation = {
kind: "artifact-inventory",
inventory: snapshot,
};
throw ArtifactReaderFailure.withCleanup(
reconstructionOutcome.kind === "failed"
? reconstructionOutcome.cause
: cleanupAttempt.cause,
cleanup,
throw scope.cleanupFailure(
reconstructionOutcome,
[cleanupAttempt],
partialObservation,
);
}
+10 -3
View File
@@ -49,7 +49,12 @@ export class ArtifactFileSource {
/** Bind verified bytes to the inventory's expected whole-carrier identity. */
bindSha256(expectedSha256: string, signal?: AbortSignal): Promise<void> {
return this.#scope.run(() =>
this.#bind(expectedSha256, signal, () => this.#verify(signal)),
this.#bind(
expectedSha256,
signal,
() => this.#verify(signal),
() => this.#verify(),
),
);
}
@@ -77,7 +82,7 @@ export class ArtifactFileSource {
expectedSha256: string,
signal: AbortSignal | undefined,
verify: () => Promise<StableRegularFileDescriptor>,
verifyCompletion: () => Promise<StableRegularFileDescriptor> = verify,
verifyCompletion: () => Promise<StableRegularFileDescriptor>,
): Promise<void> {
const { handle } = await verify();
const observed =
@@ -96,8 +101,9 @@ export class ArtifactFileSource {
signal,
)
).sha256;
// A completed digest needs source validation before late cancellation can
// decide the outcome; unverified bytes must not establish a contradiction.
await verifyCompletion();
assertArtifactReadActive(signal);
if (observed !== expectedSha256)
throw new ArtifactReaderFailure(
"integrity",
@@ -110,6 +116,7 @@ export class ArtifactFileSource {
false,
),
);
assertArtifactReadActive(signal);
this.#boundSha256 = observed;
}
+19 -1
View File
@@ -32,6 +32,25 @@ export const hashReadable = async (
signal?: AbortSignal,
integrityAlgorithm?: "sha256" | "md5",
asarHeaderAdmission?: AsarHeaderAdmission,
): Promise<HashResult> => {
const completed = await hashReadableToCompletion(
stream,
signal,
integrityAlgorithm,
asarHeaderAdmission,
);
abortIfNeeded(signal);
return completed;
};
/** Keep a fully delivered digest so its owner can validate before late cancellation.
* Source stability remains the stream or caller's responsibility.
*/
export const hashReadableToCompletion = async (
stream: Readable,
signal?: AbortSignal,
integrityAlgorithm?: "sha256" | "md5",
asarHeaderAdmission?: AsarHeaderAdmission,
): Promise<HashResult> => {
const hash = createHash("sha256");
const md5 = integrityAlgorithm === "md5" ? createHash("md5") : undefined;
@@ -45,7 +64,6 @@ export const hashReadable = async (
md5?.update(chunk);
probe.write(chunk);
}
abortIfNeeded(signal);
return {
sha256: hash.digest("hex"),
...(md5 === undefined ? {} : { md5: md5.digest("hex") }),
@@ -23,3 +23,25 @@ it.each(["capacity", "file-size", "memory"] as const)(
});
},
);
it("keeps producer-owned recovery through cleanup wrapping and public projection", () => {
const userMessage = "Caller-selected reader recovery";
const failure = new ArtifactReaderFailure(
"unavailable",
"Exact command diagnostic",
{
userMessage,
},
);
const wrapped = ArtifactReaderFailure.withCleanup(failure, {
reason: "Owned staging cleanup needs retry",
resources: ["owned staging"],
});
const projected = projectAnalysisError(
translateArtifactFailure("inspect_artifact", wrapped),
);
expect(projected.message).toBe(userMessage);
expect(projected.remediation.action).toBe(userMessage);
expect(projected.details?.detail).toBe("Exact command diagnostic");
expect(projected.details?.cleanup).toBeDefined();
});
@@ -19,6 +19,9 @@ export const translateArtifactFailure = (
cause.details,
cause.message,
{
...(cause.userMessage === undefined
? {}
: { userMessage: cause.userMessage }),
cause:
cause.reason === "cancelled"
? artifactFileCancellationCause(cause)
+5
View File
@@ -22,6 +22,8 @@ import type {
} from "../domain/analysisErrorBase.js";
export interface ArtifactReaderFailureOptions extends ErrorOptions {
/** Recovery guidance established by the producing adapter, separate from diagnostics. */
readonly userMessage?: string | undefined;
readonly cancellation?: { readonly reason: unknown } | undefined;
readonly cleanup?: AnalysisCleanupObservation | undefined;
/** Exact authority for all reported cleanup; undefined leaves ownership unknown. */
@@ -195,6 +197,7 @@ export interface ArtifactReader {
/** Typed adapter failure translated at provider boundary. */
export class ArtifactReaderFailure extends Error {
readonly userMessage: string | undefined;
readonly cancellation: { readonly reason: unknown } | undefined;
readonly cleanup: AnalysisCleanupObservation | undefined;
readonly cleanupLeases: readonly AnalysisCleanupLease[] | undefined;
@@ -219,6 +222,7 @@ export class ArtifactReaderFailure extends Error {
) {
super(message, options);
this.name = "ArtifactReaderFailure";
this.userMessage = options?.userMessage;
this.cancellation =
reason === "cancelled"
? (options?.cancellation ?? { reason: options?.cause })
@@ -269,6 +273,7 @@ export class ArtifactReaderFailure extends Error {
primary.message,
{
cause: primary,
userMessage: primary.userMessage,
cancellation: primary.cancellation,
cleanup: combineCleanup(primary.cleanup, cleanup),
cleanupLeases:
+39
View File
@@ -31,6 +31,9 @@ import {
type NativeDmgInfo,
} from "./NativeDmgOwnership.js";
const HDIUTIL_COMMAND_TIMEOUT_MS = 120_000;
/** Conditional recovery guidance; a path match never grants attachment ownership. */
export const DMG_EXISTING_ATTACHMENT_RECOVERY =
"hdiutil refused a private read-only DMG attachment after listing an existing attachment of this image path. If it is still attached, ask its owner to release it when safe, then retry. REA does not borrow or detach existing attachments.";
const DMG_CHECKSUM_ABSENT_LIMITATION =
"hdiutil verify found no stored checksum in this disk image, so its integrity was not verified; the root SHA-256 identifies the inventoried bytes.";
const VERIFY_SETTLE_DELAYS_MS = [250, 500] as const;
@@ -492,6 +495,16 @@ export class NativeDmgArtifactReader implements ArtifactReader {
this.#attachMayHaveMounted = false;
this.#ownershipUncertainty = undefined;
}
if (
cause instanceof NativeDmgCommandFailure &&
cause.reason === "unavailable" &&
cause.exitCode !== null &&
(await existingImagePathListed(this.path, baseline))
)
throw new ArtifactReaderFailure("unavailable", cause.message, {
cause,
userMessage: DMG_EXISTING_ATTACHMENT_RECOVERY,
});
throw cause;
}
}
@@ -663,6 +676,32 @@ export class NativeDmgArtifactReader implements ArtifactReader {
}
}
/** A path match supports recovery advice, never attachment ownership or byte identity. */
const existingImagePathListed = async (
path: string,
info: NativeDmgInfo,
): Promise<boolean> => {
let target: string;
try {
target = await realpath(path);
} catch {
return false;
}
for (const image of info.images) {
if (
image["image-path"] === undefined ||
image["system-entities"].length === 0
)
continue;
try {
if ((await realpath(image["image-path"])) === target) return true;
} catch {
// An unresolved path does not identify this image; keep checking other entries.
}
}
return false;
};
const failureMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
+1
View File
@@ -15,6 +15,7 @@ const attachOutputSchema = z.object({
const infoOutputSchema = z.object({
images: z.array(
z.object({
"image-path": z.string().optional(),
"system-entities": z.array(systemEntitySchema),
}),
),
+107 -57
View File
@@ -7,69 +7,119 @@ import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory
import { scanCanonicalArtifactInventory } from "../../tests/fixtures/artifactInventory.js";
import {
NativeDmgArtifactReader,
DMG_EXISTING_ATTACHMENT_RECOVERY,
type NativeDmgHost,
} from "./NativeDmgArtifactReader.js";
import { translateArtifactFailure } from "./ArtifactOperationFailure.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import { analysisErrorUserMessage } from "../domain/analysisErrorPresentation.js";
import { ArtifactOperationError } from "../domain/artifactOperationError.js";
it("retains authenticated carrier facts and unknown child coverage when native attachment is unavailable", async () => {
const root = await createTestTempDirectory("rea-dmg-unavailable-root-");
const path = join(root, "image.dmg");
const bytes = Buffer.alloc(1024);
bytes.write("koly", bytes.length - 512, "ascii");
await writeFile(path, bytes);
const host: NativeDmgHost = {
run: async (args) => {
if (args[0] === "verify") return { stdout: "", exitCode: 0 };
if (args[0] === "info")
return { stdout: build({ images: [] }), exitCode: 0 };
if (args[0] === "attach")
return {
stdout: "",
stderr: "hdiutil: attach failed - Resource busy",
exitCode: 1,
};
throw new Error(`Unexpected command ${String(args[0])}`);
},
};
let failure: unknown;
try {
await scanCanonicalArtifactInventory(
path,
{},
() => new NativeDmgArtifactReader(path, {}, host),
it.each(["same", "other", "missing-path", "no-path", "empty-group"] as const)(
"retains DMG root facts and limits existing-attachment guidance to a resolved image path: %s",
async (listed) => {
const root = await createTestTempDirectory("rea-dmg-unavailable-root-");
const path = join(root, "image.dmg");
const bytes = Buffer.alloc(1024);
bytes.write("koly", bytes.length - 512, "ascii");
await writeFile(path, bytes);
const other = join(root, "other.dmg");
await writeFile(other, bytes);
const calls: string[] = [];
const host: NativeDmgHost = {
run: async (args) => {
calls.push(args[0] ?? "");
if (args[0] === "verify") return { stdout: "", exitCode: 0 };
if (args[0] === "info")
return {
stdout: build({
images: [
{
...(listed === "no-path"
? {}
: {
"image-path":
listed === "same" || listed === "empty-group"
? path
: listed === "other"
? other
: join(root, "missing.dmg"),
}),
"system-entities":
listed === "empty-group"
? []
: [
{
"dev-entry": "/dev/disk800",
"mount-point": join(root, "preexisting"),
},
],
},
],
}),
exitCode: 0,
};
if (args[0] === "attach")
return {
stdout: "",
stderr: "hdiutil: attach failed - Resource busy",
exitCode: 1,
};
throw new Error(`Unexpected command ${String(args[0])}`);
},
};
let failure: unknown;
try {
await scanCanonicalArtifactInventory(
path,
{},
() => new NativeDmgArtifactReader(path, {}, host),
);
} catch (cause: unknown) {
failure = cause;
}
expect(failure).toBeDefined();
const projected = projectAnalysisError(
translateArtifactFailure("inspect_artifact", failure),
);
} catch (cause: unknown) {
failure = cause;
}
expect(failure).toBeDefined();
const projected = projectAnalysisError(
translateArtifactFailure("inspect_artifact", failure),
);
expect(projected).toMatchObject({
category: "unavailable",
details: {
partial_observation: {
kind: "artifact-inventory",
inventory: {
manifest: {
root_sha256: createHash("sha256").update(bytes).digest("hex"),
root_format: "dmg",
},
coverage: { status: "unknown" },
occurrences: [{ logical_path: "." }],
provenance: [
{ arguments: ["verify", path], exit_code: 0 },
{
arguments: expect.arrayContaining(["attach", path]),
exit_code: 1,
expect(projected).toMatchObject({
category: "unavailable",
details: {
partial_observation: {
kind: "artifact-inventory",
inventory: {
manifest: {
root_sha256: createHash("sha256").update(bytes).digest("hex"),
root_format: "dmg",
},
],
coverage: { status: "unknown" },
occurrences: [{ logical_path: "." }],
provenance: [
{ arguments: ["verify", path], exit_code: 0 },
{
arguments: expect.arrayContaining(["attach", path]),
exit_code: 1,
},
],
},
},
},
},
});
expect(projected.details?.partial_observation).toEqual(
expect.objectContaining({ kind: "artifact-inventory" }),
);
});
});
expect(projected.details?.partial_observation).toEqual(
expect.objectContaining({ kind: "artifact-inventory" }),
);
if (listed === "same") {
expect(projected.message).toBe(DMG_EXISTING_ATTACHMENT_RECOVERY);
expect(projected.remediation.action).toBe(
DMG_EXISTING_ATTACHMENT_RECOVERY,
);
} else {
expect(projected.message).toBe(
analysisErrorUserMessage(
new ArtifactOperationError("inspect_artifact", "unavailable"),
),
);
}
expect(calls).not.toContain("detach");
},
);
+1
View File
@@ -202,6 +202,7 @@ const scanCanonical = async (
),
cleanup: cause.cleanup,
cleanupLeases: cause.cleanupLeases,
userMessage: cause.userMessage,
...(retained === undefined
? {}
: {
@@ -16,7 +16,7 @@ import {
type ArtifactReader,
} from "../ArtifactReader.js";
import { streamChunkToBuffer } from "../StreamBytes.js";
import { hashReadable } from "../ArtifactHash.js";
import { hashReadableToCompletion } from "../ArtifactHash.js";
import type { ArtifactOccurrence } from "../../domain/artifactGraph.js";
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import { AnalysisInputError } from "../../domain/analysisErrorCore.js";
@@ -400,17 +400,13 @@ const verifyEntryBytes = async (
expected: Pick<ExpectedFile, "path" | "sha256" | "bytes">,
signal?: AbortSignal,
): Promise<void> => {
try {
const digest = await hashReadable(stream, signal);
if (digest.sha256 !== expected.sha256 || digest.bytes !== expected.bytes)
throw new ArtifactReaderFailure(
"integrity",
`Artifact entry changed after inventory: ${expected.path}`,
);
} catch (cause: unknown) {
abortIfNeeded(signal);
throw cause;
}
const digest = await hashReadableToCompletion(stream, signal);
if (digest.sha256 !== expected.sha256 || digest.bytes !== expected.bytes)
throw new ArtifactReaderFailure(
"integrity",
`Artifact entry changed after inventory: ${expected.path}`,
);
abortIfNeeded(signal);
};
const expectedInventory = (
+7
View File
@@ -1,3 +1,5 @@
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import { redactSensitiveFailure } from "../domain/explicitSensitiveFailure.js";
import { AnalysisInputError } from "../domain/analysisErrorCore.js";
import { err, ok, type Result } from "../domain/result.js";
import { createHash } from "node:crypto";
@@ -457,6 +459,11 @@ export class BrowserScenarioSecrets {
return Math.ceil(length * this.redactionCostPerCharacter);
}
redactFailure(error: AnalysisError): AnalysisError {
if (this.textPatterns.length === 0) return error;
return redactSensitiveFailure(error, (value) => this.redact(value));
}
redact(value: string): string {
const patterns = this.textPatterns;
if (patterns.length === 0) return value;
+7 -1
View File
@@ -41,11 +41,17 @@ export interface BrowserScenarioSessionPort {
redactError(error: unknown): string;
}
/** Retain the exact release authority and its producer-owned diagnostic projection. */
export type RetainBrowserScenarioCleanup = (
lease: AnalysisCleanupLease,
redactText?: (value: string) => string,
) => void;
/** Acquire one scenario session whose lifecycle the caller owns. */
export type BrowserScenarioSessionOpener = (
scenario: BrowserScenario,
options: {
readonly signal?: AbortSignal;
readonly retainCleanup?: (lease: AnalysisCleanupLease) => void;
readonly retainCleanup?: RetainBrowserScenarioCleanup;
},
) => Promise<BrowserScenarioSessionPort>;
@@ -42,6 +42,7 @@ import {
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { err, ok, type Result } from "../domain/result.js";
import type {
RetainBrowserScenarioCleanup,
BrowserScenarioSessionOpener,
BrowserScenarioSessionPort,
} from "./BrowserScenarioSessionPort.js";
@@ -252,7 +253,7 @@ const runScenario = async (
openSession: BrowserScenarioSessionOpener,
scenario: BrowserScenario,
options: ExecutionOptions,
retainCleanup: (lease: AnalysisCleanupLease) => void,
retainCleanup: RetainBrowserScenarioCleanup,
): Promise<BrowserScenarioCapture> => {
if (options.signal?.aborted)
throw new BrowserObservationError(OPERATION, "cancelled", {
@@ -402,7 +403,10 @@ export class PlaywrightBrowserScenarioProvider implements BrowserScenarioCapture
#tail: Promise<void> = Promise.resolve();
#closing = false;
#closePromise: Promise<void> | undefined;
readonly #pendingCleanup = new Set<AnalysisCleanupLease>();
readonly #pendingCleanup = new Map<
AnalysisCleanupLease,
(value: string) => string
>();
constructor(private readonly openSession: BrowserScenarioSessionOpener) {}
@@ -436,10 +440,15 @@ export class PlaywrightBrowserScenarioProvider implements BrowserScenarioCapture
const retained: AnalysisCleanupLease[] = [];
try {
return ok(
await runScenario(this.openSession, scenario, options, (lease) => {
this.#pendingCleanup.add(lease);
retained.push(lease);
}),
await runScenario(
this.openSession,
scenario,
options,
(lease, redactText = (value) => value) => {
this.#pendingCleanup.set(lease, redactText);
retained.push(lease);
},
),
);
} catch (cause: unknown) {
const primary =
@@ -453,11 +462,15 @@ export class PlaywrightBrowserScenarioProvider implements BrowserScenarioCapture
if (retained.length === 0) return err(primary);
const cleanup = new ProviderCleanupError(
PLAYWRIGHT_BROWSER_SCENARIO_PROVIDER_IDENTITY.id,
retained.flatMap((lease) => lease.cleanupResources ?? []),
retained.flatMap((lease) =>
(lease.cleanupResources ?? []).map(
this.#pendingCleanup.get(lease) ?? ((value) => value),
),
),
{ reason: "Browser scenario cleanup capabilities remain retained." },
{ operation: OPERATION, cause },
);
new AnalysisCleanupLeaseGroup(cleanup, retained);
this.#bindCleanup(cleanup, retained);
return err(
analysisErrorWithCleanupFailure(
primary,
@@ -489,9 +502,33 @@ export class PlaywrightBrowserScenarioProvider implements BrowserScenarioCapture
});
}
#bindCleanup(
error: AnalysisError,
leases: readonly AnalysisCleanupLease[],
): void {
new AnalysisCleanupLeaseGroup(error, leases);
const projections = this.#pendingCleanup;
error.bindCleanupResourceOwner({
get cleanupResources() {
const pending = leases.filter((lease) => !lease.released);
if (pending.some((lease) => lease.cleanupResources === undefined))
return undefined;
return [
...new Set(
pending.flatMap((lease) =>
(lease.cleanupResources ?? []).map(
projections.get(lease) ?? ((value) => value),
),
),
),
];
},
});
}
async #retryCleanup(): Promise<AnalysisError | undefined> {
const failures: AnalysisError[] = [];
for (const pending of this.#pendingCleanup) {
for (const [pending, redactText] of this.#pendingCleanup) {
try {
await withPlaywrightExecutionBoundary(
() => pending.release(),
@@ -501,11 +538,15 @@ export class PlaywrightBrowserScenarioProvider implements BrowserScenarioCapture
} catch (cause: unknown) {
const failure = new ProviderCleanupError(
PLAYWRIGHT_BROWSER_SCENARIO_PROVIDER_IDENTITY.id,
pending.cleanupResources ?? [],
{ reason: cause instanceof Error ? cause.message : String(cause) },
(pending.cleanupResources ?? []).map(redactText),
{
reason: redactText(
cause instanceof Error ? cause.message : String(cause),
),
},
{ operation: OPERATION, cause },
);
new AnalysisCleanupLeaseGroup(failure, [pending]);
this.#bindCleanup(failure, [pending]);
failures.push(failure);
}
}
+34 -3
View File
@@ -11,7 +11,10 @@ import { PLAYWRIGHT_BROWSER_SCENARIO_PROVIDER_IDENTITY } from "./providerIdentit
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { AnalysisError } from "../domain/analysisErrorBase.js";
import { BrowserObservationError } from "../domain/browserObservationError.js";
import type { BrowserScenarioSessionPort } from "./BrowserScenarioSessionPort.js";
import type {
RetainBrowserScenarioCleanup,
BrowserScenarioSessionPort,
} from "./BrowserScenarioSessionPort.js";
import { BrowserScenarioSecrets } from "./BrowserScenarioSecrets.js";
import {
failBrowserScenarioOperation,
@@ -158,13 +161,41 @@ export class PlaywrightScenarioSession implements BrowserScenarioSessionPort {
environment: Readonly<Record<string, string | undefined>>,
options: {
readonly signal?: AbortSignal;
readonly retainCleanup?: (lease: AnalysisCleanupLease) => void;
readonly retainCleanup?: RetainBrowserScenarioCleanup;
},
): Promise<PlaywrightScenarioSession> {
throwIfScenarioCancelled(options.signal);
const resolved = BrowserScenarioSecrets.resolve(scenario, environment);
if (!resolved.ok) throw resolved.error;
const secrets = resolved.value;
const projectedOptions = {
...options,
retainCleanup: (lease: AnalysisCleanupLease) =>
options.retainCleanup?.(lease, (value) => secrets.redact(value)),
};
try {
return await this.openResolved(
scenario,
environment,
secrets,
projectedOptions,
);
} catch (cause: unknown) {
throw cause instanceof AnalysisError
? secrets.redactFailure(cause)
: cause;
}
}
private static async openResolved(
scenario: BrowserScenario,
environment: Readonly<Record<string, string | undefined>>,
secrets: BrowserScenarioSecrets,
options: {
readonly signal?: AbortSignal;
readonly retainCleanup?: RetainBrowserScenarioCleanup;
},
): Promise<PlaywrightScenarioSession> {
const startedAt = performance.now();
const openingCleanup = new Set<AnalysisCleanupLease>();
const opening = openPlaywrightScenarioBrowser(scenario, environment, {
@@ -292,7 +323,7 @@ export class PlaywrightScenarioSession implements BrowserScenarioSessionPort {
secrets: BrowserScenarioSecrets,
options: {
readonly signal?: AbortSignal;
readonly retainCleanup?: (lease: AnalysisCleanupLease) => void;
readonly retainCleanup?: RetainBrowserScenarioCleanup;
},
): Promise<PlaywrightScenarioSession> {
let events: PlaywrightScenarioEvents | undefined;
@@ -4,8 +4,7 @@ import { parseExactJson } from "../exactJsonNumber.js";
import { MAX_JSON_DEPTH, type JsonValue } from "../jsonValue.js";
import {
compareOffsetDateTimeInstants,
readRecordedDateTime,
type OffsetDateTimeInstant,
readRecordedStart,
} from "../offsetDateTime.js";
import { ok, type Result } from "../result.js";
import {
@@ -86,10 +85,8 @@ export const stateBodyJson = (
: value;
};
const timingSchema = z.object({
startedDateTime: z.string(),
time: z.number().nonnegative(),
});
const startSchema = z.object({ startedDateTime: z.string() });
const durationSchema = z.object({ time: z.number().nonnegative() });
const redactedWriteSchema = z.object({
request: z.object({ method: z.string() }),
response: z.object({ status: z.number() }).optional(),
@@ -99,8 +96,9 @@ interface RecordEntry {
/** Null for a write whose URL was redacted: its target is unknown, so it bounds every type. */
readonly origin: string | null;
readonly contentType: string;
/** Exact ordering fact; request interval coordinates remain milliseconds. */
readonly startInstant: OffsetDateTimeInstant | undefined;
readonly startInterpretation:
| NonNullable<ReturnType<typeof readRecordedStart>>["interpretation"]
| undefined;
readonly record: WebNetworkCaptureRecord;
}
@@ -143,43 +141,51 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
record: WebNetworkCaptureRecord,
line: RecordedRequestLine,
): Result<void, AnalysisError> {
const timing = timingSchema.safeParse(record.reported);
const started = startSchema.safeParse(record.reported);
const duration = durationSchema.safeParse(record.reported);
// Admit normalization and fraction parsing before materializing their strings.
if (
timing.success &&
started.success &&
!this.scratch(
RECORD_SCRATCH_BYTES +
textBytes(timing.data.startedDateTime.length + 6) * 3,
textBytes(started.data.startedDateTime.length + 6) * 3,
)
)
return this.overBudget();
const startInstant = timing.success
? readRecordedDateTime(timing.data.startedDateTime)?.instant
const reading = started.success
? readRecordedStart(started.data.startedDateTime)
: undefined;
const startInstant = reading?.instant;
// Keep the established millisecond interval policy independently of sorting.
const start =
const startMs =
startInstant === undefined
? record.ordinal
? undefined
: startInstant.epochSecond * 1000 +
Number(startInstant.fraction.slice(0, 3).padEnd(3, "0"));
const end =
startMs === undefined || !duration.success
? undefined
: startMs + duration.data.time;
const timing: StateRecordFacts["timing"] = {
start:
startInstant === undefined || startMs === undefined
? undefined
: { instant: startInstant, ms: startMs },
endMs: end !== undefined && Number.isFinite(end) ? end : undefined,
};
// A fraction substring may retain its normalized timestamp's backing store,
// including trailing zeroes removed from the exact comparison value.
const instantBytes =
timing.success && startInstant !== undefined
? ENTRY_BYTES + textBytes(timing.data.startedDateTime.length + 6)
started.success && startInstant !== undefined
? ENTRY_BYTES + textBytes(started.data.startedDateTime.length + 6)
: 0;
if (line.kind === "excluded")
return line.reason === "url-redacted"
? this.#redactedWrite(
record,
timing.success && startInstant !== undefined
? {
start,
end: start + timing.data.time,
startInstant,
instantBytes,
}
: undefined,
timing,
instantBytes,
reading?.interpretation,
)
: ok(undefined);
// The entry keeps the line's strings and a reference to the already admitted record.
@@ -204,15 +210,11 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
path: line.path,
query: line.query === "" ? undefined : line.query,
status: line.status,
start,
end:
timing.success && startInstant !== undefined
? start + timing.data.time
: record.ordinal,
timing,
},
origin: line.origin,
contentType: line.rawMime,
startInstant,
startInterpretation: reading?.interpretation,
record,
});
return ok(undefined);
@@ -221,14 +223,9 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
/** Keep a successful write whose URL was redacted, with an empty path whose root relates to every type. */
#redactedWrite(
record: WebNetworkCaptureRecord,
timing:
| {
readonly start: number;
readonly end: number;
readonly startInstant: OffsetDateTimeInstant;
readonly instantBytes: number;
}
| undefined,
timing: StateRecordFacts["timing"],
instantBytes: number,
startInterpretation: RecordEntry["startInterpretation"],
): Result<void, AnalysisError> {
const parsed = redactedWriteSchema.safeParse(record.reported);
if (!parsed.success) return ok(undefined);
@@ -239,15 +236,12 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
path: "",
query: undefined,
status: status > 0 ? status : null,
start: timing?.start ?? record.ordinal,
end: timing?.end ?? record.ordinal,
timing,
};
if (!isStateWrite(facts)) return ok(undefined);
if (
!this.retain(
ENTRY_BYTES * 8 +
textBytes(facts.method.length) +
(timing?.instantBytes ?? 0),
ENTRY_BYTES * 8 + textBytes(facts.method.length) + instantBytes,
)
)
return this.overBudget();
@@ -255,7 +249,7 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
facts,
origin: null,
contentType: "",
startInstant: timing?.startInstant,
startInterpretation,
record,
});
return ok(undefined);
@@ -358,7 +352,7 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
).length;
const timeline: StateTimeline =
entries.length > 0 &&
entries.every((entry) => entry.startInstant !== undefined)
entries.every((entry) => entry.facts.timing.start !== undefined)
? "recorded-start-time"
: "record-order";
const records = entries
@@ -366,16 +360,11 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
timeline === "record-order"
? left.facts.ordinal - right.facts.ordinal
: compareOffsetDateTimeInstants(
left.startInstant!,
right.startInstant!,
left.facts.timing.start!.instant,
right.facts.timing.start!.instant,
) || left.facts.ordinal - right.facts.ordinal,
)
.map((entry) => entry.facts)
.map((facts) =>
timeline === "record-order"
? { ...facts, start: facts.ordinal, end: facts.ordinal }
: facts,
);
.map((entry) => entry.facts);
const state = new EnvironmentStateExtraction({
tools: manifest.value.tools,
records,
@@ -391,6 +380,11 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
}).run();
if (state === null) return this.overBudget();
const limitations = this.addLimitations([
...(entries.some((entry) => entry.startInterpretation === "native-date")
? [
"Some recorded start spellings use native Date parsing at millisecond precision; spellings without an explicit zone depend on the host timezone.",
]
: []),
...(redactedWrites > 0
? [
`${String(redactedWrites)} of the recorded writes had a redacted URL; each ends the initial state of every type, since its target is unknown.`,
@@ -401,6 +395,12 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
"Some records carry no start time, so the state's timeline is record order and overlapping requests are not detected.",
]
: []),
...(timeline === "recorded-start-time" &&
records.some((record) => record.timing.endMs === undefined)
? [
"Some records carry no usable request duration; their completion and overlap remain unknown.",
]
: []),
...(state.views.some((view) => view.shape === "recorded")
? [
"Tools whose recordings do not reproduce from the extracted state are listed as recorded views with the reason; they replay their recordings.",
@@ -1,3 +1,4 @@
import { StateRecordChronology } from "./StateRecordChronology.js";
import { jsonByteLength } from "../jsonSerialization.js";
import type { JsonValue } from "../jsonValue.js";
import {
@@ -107,14 +108,13 @@ const memberNames = (body: JsonValue, pointer: string): string[] => {
export class EnvironmentStateExtraction {
readonly #tools: readonly StateToolFacts[];
readonly #records: readonly StateRecordFacts[];
readonly #timeline: StateTimeline;
readonly #chronology: StateRecordChronology;
readonly #budget: StateBudget;
readonly #position: ReadonlyMap<number, number>;
readonly #byOrdinal: ReadonlyMap<number, StateRecordFacts>;
/** POST tools whose calls the capture shows read state rather than change it. */
#readOnly: ReadonlySet<string> = new Set();
#writes: readonly StateRecordFacts[] = [];
#writeWithin: ReturnType<typeof writeSpanCheck> = writeSpanCheck([]);
#writeWithin: ReturnType<typeof writeSpanCheck> = () => false;
readonly #bases: ReadonlyMap<string, number>;
#writeCutoffs: StateWriteCutoffs | undefined;
readonly #initLimits = new Map<string, StateRecordFacts | undefined>();
@@ -146,11 +146,8 @@ export class EnvironmentStateExtraction {
}) {
this.#tools = input.tools;
this.#records = input.records;
this.#timeline = input.timeline;
this.#chronology = new StateRecordChronology(input.records, input.timeline);
this.#budget = input.budget;
this.#position = new Map(
input.records.map((record, index) => [record.ordinal, index]),
);
this.#byOrdinal = new Map(
input.records.map((record) => [record.ordinal, record]),
);
@@ -169,9 +166,10 @@ export class EnvironmentStateExtraction {
request: (ordinal) => this.#requestBody(ordinal),
retain: (bytes) => this.#retain(bytes),
},
this.#chronology,
));
release();
this.#writeWithin = writeSpanCheck(this.#writes);
this.#writeWithin = writeSpanCheck(this.#writes, this.#chronology);
}
/** The extracted state, or null when the representation budget is exhausted. */
@@ -183,8 +181,8 @@ export class EnvironmentStateExtraction {
this.#writes,
this.#bases,
(left, right) =>
this.#position.get(left.ordinal)! <
this.#position.get(right.ordinal)!,
this.#chronology.positions.get(left.ordinal)! <
this.#chronology.positions.get(right.ordinal)!,
(bytes) => this.#retain(bytes),
);
this.#creations();
@@ -240,7 +238,7 @@ export class EnvironmentStateExtraction {
const split = withoutVariants(plan, {
shapeOf: (record) => shapes.get(record.ordinal),
queryOf: (ordinal) => this.#byOrdinal.get(ordinal)?.query,
position: (ordinal) => this.#position.get(ordinal)!,
position: (ordinal) => this.#chronology.positions.get(ordinal)!,
retain: (bytes) => this.#retain(bytes),
checkpoint: () => this.#budget.checkpoint(),
});
@@ -259,6 +257,7 @@ export class EnvironmentStateExtraction {
this.#spans.filter((span) => this.#rewritten.has(span.type)),
this.#writes,
this.#byOrdinal,
this.#chronology,
);
if (this.#cuts.size === 0) return false;
for (const map of [this.#initLimits, this.#identity, this.#views])
@@ -282,7 +281,7 @@ export class EnvironmentStateExtraction {
tools: this.#tools,
records: this.#records,
readOnly: this.#readOnly,
timed: this.#timeline === "recorded-start-time",
chronology: this.#chronology,
volatileOf: (type) => this.#volatileOf(type),
io: {
withBody: (ordinal, use) => this.#withBody(ordinal, use),
@@ -355,9 +354,10 @@ export class EnvironmentStateExtraction {
this.#creations(),
{
type: template,
chronology: this.#chronology,
tools: this.#tools,
records: this.#byOrdinal,
position: (record) => this.#position.get(record.ordinal)!,
position: (record) => this.#chronology.positions.get(record.ordinal)!,
withBody: (ordinal, use) => this.#withBody(ordinal, use),
retain: (bytes) => this.#retain(bytes),
},
@@ -367,7 +367,8 @@ export class EnvironmentStateExtraction {
const limit =
cut !== undefined &&
(bound === undefined ||
this.#position.get(cut.ordinal)! < this.#position.get(bound.ordinal)!)
this.#chronology.positions.get(cut.ordinal)! <
this.#chronology.positions.get(bound.ordinal)!)
? cut
: bound;
this.#retain(ENTRY_BYTES + textBytes(template.length));
@@ -391,9 +392,9 @@ export class EnvironmentStateExtraction {
): boolean {
return (
limit === undefined ||
(this.#position.get(record.ordinal)! <
this.#position.get(limit.ordinal)! &&
(this.#timeline === "record-order" || record.end <= limit.start))
(this.#chronology.positions.get(record.ordinal)! <
this.#chronology.positions.get(limit.ordinal)! &&
this.#chronology.completedBefore(record, limit))
);
}
@@ -412,8 +413,8 @@ export class EnvironmentStateExtraction {
)
.sort(
(left, right) =>
this.#position.get(left.ordinal)! -
this.#position.get(right.ordinal)!,
this.#chronology.positions.get(left.ordinal)! -
this.#chronology.positions.get(right.ordinal)!,
);
}
@@ -553,7 +554,9 @@ export class EnvironmentStateExtraction {
this.#identity.set(type, { field, basis: "path-parameter" });
const late = this.#addLateProjections(tool, type, projected);
projected.sort(
(left, right) => this.#position.get(left)! - this.#position.get(right)!,
(left, right) =>
this.#chronology.positions.get(left)! -
this.#chronology.positions.get(right)!,
);
this.#addPlan(candidate, pointer, kept, { projected, late });
}
@@ -709,7 +712,7 @@ export class EnvironmentStateExtraction {
if (id === undefined) return;
const collection = collectionPathOf(record, plan.item);
const key = entityKey(plan.type, collection, id);
const position = this.#position.get(record.ordinal)!;
const position = this.#chronology.positions.get(record.ordinal)!;
let entity = this.#entities.get(key);
if (entity === undefined) {
this.#retain(ENTRY_BYTES * 4 + textBytes(key.length));
@@ -1,9 +1,12 @@
import { StateRecordChronology } from "./StateRecordChronology.js";
import { OmittedFields } from "./environmentStateWriteOmitted.js";
import { checkedLines, servedLine } from "./environmentStateCheckedLines.js";
import { describe, expect, it, vi } from "vitest";
import {
timedCaptureEvidence,
analysisViewCaptureEvidence,
environmentManifestCaptureFixture,
type TimedCaptureRow,
} from "../../../tests/fixtures/analysisView.js";
import {
@@ -1298,8 +1301,7 @@ describe("encoded parent spellings", () => {
path: "/api/orgs/uid/acme1/books",
query: undefined,
status: 200,
start: 0,
end: 0,
timing: { start: undefined, endMs: undefined },
};
const lines = checkedLines({
tools: [tool],
@@ -1502,8 +1504,7 @@ describe("resolved encoded item observations", () => {
path: "/api/orgs/acme1/books/%61",
query: undefined,
status: 204,
start: 0,
end: 0,
timing: { start: undefined, endMs: undefined },
};
const target = itemTarget(state, "/api/orgs/{org_id}/books", write);
const itemView = {
@@ -1516,19 +1517,20 @@ describe("resolved encoded item observations", () => {
checked_record_ordinals: [1],
envelope_ordinal: 1,
};
const records = [
write,
{
ordinal: 1,
method: "GET",
path: "/api/orgs/acme1/books/%61",
query: undefined,
status: 200,
timing: { start: undefined, endMs: undefined },
},
];
const search: ObservationSearch = {
records: [
write,
{
ordinal: 1,
method: "GET",
path: "/api/orgs/acme1/books/%61",
query: undefined,
status: 200,
start: 1,
end: 1,
},
],
records,
chronology: new StateRecordChronology(records, "record-order"),
from: 0,
viewOf: () => itemView,
withBody: (_ordinal, use) => use({ id: "%61", name: "After" }),
@@ -2130,7 +2132,54 @@ describe("what an unmodelled write explains", () => {
});
});
it("leaves the global served clock unknown when any record completion is unknown", () => {
const rows = (duration: number): TimedCaptureRow[] => [
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
// Even an earlier record with an unknown end may finish after the final known record.
["GET", "/api/books/1", FIRST_BOOK, 200, undefined, duration],
["GET", "/api/books/2", SECOND_BOOK],
];
expect(
environmentStateViewOf(timedCaptureEvidence(rows(5))).write_clock,
).toBe(new Date(bookClock(3)).toISOString());
expect(
environmentStateViewOf(timedCaptureEvidence(rows(-1))).write_clock,
).toBeNull();
});
describe("reads and calls on an entity an unmodelled write left unknown", () => {
it.each([
{ duration: 5, target: "1", served: true },
{ duration: -1, target: "1", served: false },
{ duration: -1, target: "2", served: true },
])(
"requires unmodelled completion before resync and later confirmation: $target duration=$duration",
({ duration, target, served }) => {
const rows: TimedCaptureRow[] = [
[
"GET",
"/api/books",
[
{ id: 1, name: "A" },
{ id: 2, name: "B" },
],
],
["GET", "/api/books/1", { id: 1, name: "A" }],
["GET", "/api/books/2", { id: 2, name: "B" }],
["POST", `/api/books/${target}/archive`, null, 204, {}, duration],
// This read may re-establish a baseline only after the related action finished.
["GET", "/api/books/1", { id: 1, name: "A" }],
["PATCH", "/api/books/1", { id: 1, name: "Z" }, 200, { name: "Z" }],
["GET", "/api/books/1", { id: 1, name: "Z" }],
];
expect(gated(rows, "patch_books_by_id")).toMatchObject({
served,
checked_record_ordinals: served ? [6] : [],
contradicted_by: null,
});
},
);
const named = (name: string) => ({ id: 1, name });
const other = { id: 2, name: "B" };
const ARCHIVE: TimedCaptureRow = [
@@ -2307,8 +2356,7 @@ describe("entities an unmodelled write made", () => {
path: "/api/books/3",
query: undefined,
status: 200,
start: 1,
end: 1,
timing: { start: undefined, endMs: undefined },
};
expect(
unseen.adopt(view as unknown as StateItemView, { record, index: 1 }),
@@ -2333,8 +2381,7 @@ describe("entities an unmodelled write made", () => {
path: "/api/books/3",
query: undefined,
status: 200,
start: index,
end: index,
timing: { start: undefined, endMs: undefined },
},
index,
});
@@ -3042,8 +3089,7 @@ it("learns omitted fields using the raw ID resolved by the item owner", () => {
path: "/api/books/%63",
query: undefined,
status: 200,
start: 1,
end: 1,
timing: { start: undefined, endMs: undefined },
};
const target = itemTarget(state, view.type, record);
expect(target.id).toBe("%63");
@@ -3066,3 +3112,133 @@ it("learns omitted fields using the raw ID resolved by the item owner", () => {
required_fields: ["id", "name"],
});
});
it.each([
{
label: "point read with simultaneous unknown action",
archiveStart: 4,
duration: 0,
archiveDuration: -1,
target: "1",
served: false,
},
{
label: "point read with simultaneous point action",
archiveStart: 4,
duration: 0,
archiveDuration: 0,
target: "1",
served: true,
},
{
label: "point read with unrelated unknown action",
archiveStart: 4,
duration: 0,
archiveDuration: -1,
target: "2",
served: true,
},
{
label: "same-start later ordinal",
archiveStart: 4,
duration: 5,
target: "1",
served: false,
},
{
label: "later start within response interval",
archiveStart: 5,
duration: 1500,
target: "1",
served: false,
},
{
label: "unknown response completion",
archiveStart: 5,
duration: -1,
target: "1",
served: false,
},
{
label: "known disjoint response interval",
archiveStart: 6,
duration: 1500,
target: "1",
served: true,
},
{
label: "unrelated overlapping write",
archiveStart: 5,
duration: 1500,
target: "2",
served: true,
},
])(
"requires a read free of captured write overlap: $label",
({ archiveStart, duration, target, served, ...options }) => {
const rows: TimedCaptureRow[] = [
[
"GET",
"/api/books",
[
{ id: 1, name: "A" },
{ id: 2, name: "B" },
],
],
["GET", "/api/books/1", { id: 1, name: "A" }],
["GET", "/api/books/2", { id: 2, name: "B" }],
["PATCH", "/api/books/1", { id: 1, name: "Z" }, 200, { name: "Z" }],
["GET", "/api/books/1", { id: 1, name: "Z" }, 200, undefined, duration],
[
"POST",
`/api/books/${target}/archive`,
null,
204,
{},
"archiveDuration" in options ? options.archiveDuration : 5,
],
];
const entries = rows.map(
(
[method, path, body = null, status = 200, request, time = 5],
index,
) => ({
startedDateTime: new Date(
Date.parse(bookClock(index === 5 ? archiveStart : index)),
).toISOString(),
time,
request: {
method,
url: `http://app.example${path}`,
bodySize:
request === undefined
? 0
: Buffer.byteLength(JSON.stringify(request)),
...(request === undefined
? {}
: {
postData: {
mimeType: "application/json",
text: JSON.stringify(request),
},
}),
},
response: {
status,
content: { mimeType: "application/json", text: JSON.stringify(body) },
},
}),
);
const capture = analysisViewCaptureEvidence(
environmentManifestCaptureFixture(entries),
);
const plan = environmentStateViewOf(capture).writes.find(
(write) => write.tool === "patch_books_by_id",
);
expect(plan).toMatchObject({
served,
checked_record_ordinals: served ? [4] : [],
contradicted_by: null,
});
},
);
@@ -1,3 +1,4 @@
import { CapturedWriteCompletion } from "./environmentStateWriteCompletion.js";
import type { JsonValue } from "../jsonValue.js";
import {
at,
@@ -114,6 +115,7 @@ export class EnvironmentStateWriteSimulation {
readonly #touchedSince: PendingTargets = new Map();
/** Timeline indexes of writes the gate does not model. */
readonly #unmodelledAt: number[] = [];
readonly #writeCompletion: CapturedWriteCompletion;
readonly #history: WriteHistory = {
touched: this.#touched,
pending: this.#touchedSince,
@@ -135,6 +137,7 @@ export class EnvironmentStateWriteSimulation {
constructor(input: WriteSimulationInput) {
this.#input = input;
this.#writeCompletion = new CapturedWriteCompletion(input.io.retain);
this.#state = new SimulatedState(input.entities, input.documents);
this.#unseen = new UnseenEntities(input.entities, {
state: this.#state,
@@ -225,6 +228,15 @@ export class EnvironmentStateWriteSimulation {
linked: link.collection,
});
}
// Reads may finish after writes with the same or a later start: index the
// complete capture before replay so ordinal tie-breaking cannot create proof.
for (const record of this.#input.records)
if (this.#isWrite(record))
this.#writeCompletion.write(
record.path,
this.#input.chronology.endMs(record),
this.#input.chronology.startMs(record),
);
this.#input.records.forEach((record, index) => this.#step(record, index));
const views = new Map(
this.views([...this.#input.views.values()]).map((view) => [
@@ -281,7 +293,7 @@ export class EnvironmentStateWriteSimulation {
: this.#input.io.requestBody(ordinal),
},
records: this.#input.records,
timed: this.#input.timed,
chronology: this.#input.chronology,
viewOf: (ordinal) => {
const view = this.#input.views.get(
this.#toolOf.get(ordinal)?.name ?? "",
@@ -355,7 +367,7 @@ export class EnvironmentStateWriteSimulation {
index,
tool: tool.name,
path: record.path,
end: record.end,
end: this.#input.chronology.endMs(record),
});
};
// Item deletes use their path; calls outside a built plan's query never interpret their body.
@@ -371,7 +383,7 @@ export class EnvironmentStateWriteSimulation {
readonly index: number;
readonly tool: string;
readonly path: string;
readonly end: number;
readonly end: number | undefined;
},
): void {
if (result.kind === "unknown") {
@@ -428,6 +440,7 @@ export class EnvironmentStateWriteSimulation {
/** Whether an item read shows an entity an unmodelled write made, or one left unknown, again rather than checks it. */
#resynced(view: StateView, record: StateRecordFacts, index: number): boolean {
if (view.shape !== "item") return false;
if (this.#writeOverlapsRead(record)) return true;
const adopted = this.#unseen.adopt(view, { record, index });
if (adopted !== undefined) {
this.#taint.known(adopted.type, adopted.collection, adopted.id);
@@ -437,7 +450,14 @@ export class EnvironmentStateWriteSimulation {
if (!this.#taint.unknown(view.type, collection, id)) return false;
// A read begun while a modelled write to the entity still ran may show either state, so it neither checks nor shows it.
const target = entityTarget(view.type, collection, id);
if (runningAt(this.#touchedSince, target, record.start)) return true;
if (
runningAt(
this.#touchedSince,
target,
this.#input.chronology.startMs(record),
)
)
return true;
// A failed read that is no absence shows nothing of the entity, so it stays unknown for a usable read.
if (!succeeded(record) && !isAbsence(record.status)) return true;
const entity = this.#state.entity(view.type, collection, id);
@@ -531,6 +551,7 @@ export class EnvironmentStateWriteSimulation {
if (original === undefined) return;
let view = this.#sorted.comparisonView(this.omitted.view(original));
if (projectedRead(view, record, this.#input.io)) return;
if (this.#writeOverlapsRead(record)) return;
const borrowers = this.#observations.get(record.ordinal);
if (
this.#viewOf(original).shape !== "recorded" &&
@@ -545,7 +566,11 @@ export class EnvironmentStateWriteSimulation {
targets.some(
(key) =>
this.#taint.tainted(key) ||
runningAt(this.#touchedSince, key, record.start),
runningAt(
this.#touchedSince,
key,
this.#input.chronology.startMs(record),
),
);
if (blocked(keys)) return;
// A read the gate counts can show the view lists a written array sorted, or rule that out.
@@ -651,6 +676,15 @@ export class EnvironmentStateWriteSimulation {
return shown.unseen;
}
/** An overlapping captured write cannot supply a stable read baseline. */
#writeOverlapsRead(read: StateRecordFacts): boolean {
return this.#writeCompletion.runningAt(
read.path,
this.#input.chronology.startMs(read),
this.#input.chronology.endMs(read),
);
}
#unmodelled(path: string, index: number): void {
this.#taint.write(path);
this.#unmodelledAt.push(index);
@@ -0,0 +1,36 @@
import type {
StateRecordFacts,
StateTimeline,
} from "./environmentStateRules.js";
/** Own recorded interval facts separately from the explicit ordinal fallback policy. */
export class StateRecordChronology {
readonly positions: ReadonlyMap<number, number>;
constructor(
records: readonly StateRecordFacts[],
readonly timeline: StateTimeline,
) {
this.positions = new Map(
records.map((record, index) => [record.ordinal, index]),
);
}
startMs(record: StateRecordFacts): number | undefined {
return this.timeline === "record-order"
? this.positions.get(record.ordinal)
: record.timing.start?.ms;
}
endMs(record: StateRecordFacts): number | undefined {
return this.timeline === "record-order"
? this.positions.get(record.ordinal)
: record.timing.endMs;
}
completedBefore(left: StateRecordFacts, right: StateRecordFacts): boolean {
const end = this.endMs(left);
const start = this.startMs(right);
return end !== undefined && start !== undefined && end <= start;
}
}
@@ -46,6 +46,8 @@ export const WITHHELD_WRITE_HINTS = {
"No later read showed the effect. Read the entity and its list after the write.",
"incomplete-entities":
"The write is confirmed, but the views that show its entity cannot render it in full. Read the written entity through every view that lists it.",
"unknown-clock":
"The write's learned rules generate timestamps, but the capture establishes no global served clock start. Record usable start times and completed durations for every request.",
} as const satisfies Record<WriteReason, string>;
const writeReasons = Object.keys(WITHHELD_WRITE_HINTS) as [
@@ -21,8 +21,7 @@ const check = (
path: `/api/books/${ordinal + 1}`,
query: undefined,
status: 200,
start: ordinal,
end: ordinal,
timing: { start: undefined, endMs: undefined },
}));
const bodies: JsonObject[] = clocks.map(([, stamp], index) => ({
id: index + 1,
@@ -1,3 +1,5 @@
import { StateRecordChronology } from "./StateRecordChronology.js";
import { parseOffsetDateTimeInstant } from "../offsetDateTime.js";
import { describe, expect, it } from "vitest";
import type { JsonValue } from "../jsonValue.js";
@@ -7,6 +9,7 @@ import {
} from "./environmentStateQueryCalls.js";
import type {
StateRecordFacts,
StateTimeline,
StateToolFacts,
} from "./environmentStateRules.js";
@@ -20,7 +23,7 @@ interface Call {
readonly tool: string;
readonly template?: string;
readonly start?: number;
readonly end?: number;
readonly end?: number | null;
}
/** How the read-only pass parses response bodies and charges what it keeps of them, when a test watches either. */
@@ -30,15 +33,26 @@ interface Bodies {
}
/** The read-only tools of a timeline of calls, ordinals in order; a call spans its index unless it gives its times. */
const readOnlyOf = (calls: readonly Call[], bodies: Bodies = {}): string[] => {
const readOnlyOf = (
calls: readonly Call[],
bodies: Bodies = {},
timeline: StateTimeline = "recorded-start-time",
): string[] => {
const records: StateRecordFacts[] = calls.map((call, index) => ({
ordinal: index,
method: call.method,
path: call.path,
query: call.query,
status: call.status === undefined ? 200 : call.status,
start: call.start ?? index,
end: call.end ?? index,
timing: {
start: {
instant: parseOffsetDateTimeInstant(
new Date(call.start ?? index).toISOString(),
),
ms: call.start ?? index,
},
endMs: call.end === null ? undefined : (call.end ?? index),
},
}));
const tools = new Map<string, StateToolFacts>();
calls.forEach((call, index) => {
@@ -53,11 +67,16 @@ const readOnlyOf = (calls: readonly Call[], bodies: Bodies = {}): string[] => {
const withBody: QueryCallIo["withBody"] =
bodies.withBody ?? ((ordinal, use) => use(calls[ordinal]?.response));
return [
...readOnlyTools([...tools.values()], records, {
withBody,
request: (ordinal) => calls[ordinal]?.request,
retain: bodies.retain ?? (() => undefined),
}),
...readOnlyTools(
[...tools.values()],
records,
{
withBody,
request: (ordinal) => calls[ordinal]?.request,
retain: bodies.retain ?? (() => undefined),
},
new StateRecordChronology(records, timeline),
),
].sort();
};
@@ -316,6 +335,31 @@ describe("POSTs that read state but stay writes", () => {
});
describe("the reads either side of a POST", () => {
it("requires known read completion without discarding independent known intervals", () => {
const known: Call = {
method: "GET",
path: "/api/members",
tool: "get_members",
response: JOBS,
start: 0,
end: 5,
};
const unknown = { ...known, start: 6, end: null };
const post: Call = {
method: "POST",
path: "/api/members",
tool: "post_members",
response: JOBS,
start: 10,
end: 11,
};
expect(readOnlyOf([unknown, post])).toEqual([]);
expect(readOnlyOf([known, unknown, post])).toEqual(["post_members"]);
expect(readOnlyOf([unknown, post], {}, "record-order")).toEqual([
"post_members",
]);
});
it("compares a POST only with reads that finished before it began or began after it ended", () => {
const note = (title: string): Call => ({
method: "GET",
@@ -454,8 +498,15 @@ describe("read-only decisions over many calls", () => {
path: read ? "/api/members" : "/api/members/ids",
query: undefined,
status: 200,
start: index,
end: index,
timing: {
start: {
instant: parseOffsetDateTimeInstant(
new Date(index).toISOString(),
),
ms: index,
},
endMs: index,
},
};
const method = read ? "GET" : "POST";
return Object.defineProperty(record, "method", {
@@ -487,11 +538,16 @@ describe("read-only decisions over many calls", () => {
];
visits = 0;
expect([
...readOnlyTools(tools, records, {
withBody: (_ordinal, use) => use([]),
request: () => undefined,
retain: () => undefined,
}),
...readOnlyTools(
tools,
records,
{
withBody: (_ordinal, use) => use([]),
request: () => undefined,
retain: () => undefined,
},
new StateRecordChronology(records, "recorded-start-time"),
),
]).toEqual(["post_ids"]);
expect(visits).toBeLessThan(50 * records.length);
});
@@ -510,8 +566,15 @@ describe("read-only decisions over a long run of calls", () => {
path: "/v1/jobs/statuses",
query: undefined,
status: 200,
start: index,
end: index,
timing: {
start: {
instant: parseOffsetDateTimeInstant(
new Date(index).toISOString(),
),
ms: index,
},
endMs: index,
},
};
const method = read ? "GET" : "POST";
return Object.defineProperty(record, "method", {
@@ -543,11 +606,16 @@ describe("read-only decisions over a long run of calls", () => {
];
visits = 0;
expect([
...readOnlyTools(tools, records, {
withBody: (_ordinal, use) => use(JOBS),
request: () => ({ jobs: ["night-watch"] }),
retain: () => undefined,
}),
...readOnlyTools(
tools,
records,
{
withBody: (_ordinal, use) => use(JOBS),
request: () => ({ jobs: ["night-watch"] }),
retain: () => undefined,
},
new StateRecordChronology(records, "recorded-start-time"),
),
]).toEqual(["post_statuses"]);
expect(visits).toBeLessThan(50 * records.length);
});
@@ -566,8 +634,15 @@ describe("read-only decisions over many lines", () => {
method: read ? "GET" : "POST",
path: read ? "/api/members" : "/api/members/ids",
status: 200,
start: index,
end: index,
timing: {
start: {
instant: parseOffsetDateTimeInstant(
new Date(index).toISOString(),
),
ms: index,
},
endMs: index,
},
};
const query = read ? `?page=${String(index)}` : undefined;
return Object.defineProperty(record, "query", {
@@ -596,11 +671,16 @@ describe("read-only decisions over many lines", () => {
},
];
expect([
...readOnlyTools(tools, records, {
withBody: (_ordinal, use) => use([]),
request: () => undefined,
retain: () => undefined,
}),
...readOnlyTools(
tools,
records,
{
withBody: (_ordinal, use) => use([]),
request: () => undefined,
retain: () => undefined,
},
new StateRecordChronology(records, "recorded-start-time"),
),
]).toEqual(["post_ids"]);
expect(reads).toBeLessThan(20 * records.length);
});
@@ -1,3 +1,4 @@
import type { StateRecordChronology } from "./StateRecordChronology.js";
import { digestCanonicalValue } from "../canonicalDigest.js";
import { identifierWords } from "./environmentStateWords.js";
import type { JsonValue } from "../jsonValue.js";
@@ -176,10 +177,9 @@ export const readOnlyTools = (
tools: readonly StateToolFacts[],
records: readonly StateRecordFacts[],
io: QueryCallIo,
chronology: StateRecordChronology,
): Set<string> => {
const position = new Map(
records.map((record, index) => [record.ordinal, index]),
);
const position = chronology.positions;
const response = responseDigests(io);
/** The timeline indexes of the nearest state writes either side of an index, the skipped calls aside: -1 and the length when none, found in one pass each way. */
const nearestWrites = (
@@ -220,6 +220,7 @@ export const readOnlyTools = (
if (!calls.some(succeeded) || calls.some(unresolved)) continue;
const timeline = {
position,
chronology,
...nearestWrites(new Set(tool.record_ordinals)),
response,
};
@@ -234,6 +235,7 @@ export const readOnlyTools = (
for (let vetoed = true; vetoed;) {
const timeline = {
position,
chronology,
...nearestWrites(
new Set([...candidates.values()].flat().map((call) => call.ordinal)),
),
@@ -250,6 +252,7 @@ export const readOnlyTools = (
};
interface Timeline {
readonly chronology: StateRecordChronology;
readonly position: ReadonlyMap<number, number>;
/** The nearest other writes either side of a timeline index. */
readonly writes: (at: number) => {
@@ -266,12 +269,14 @@ interface Timeline {
const lastFinishedBefore = (
reads: readonly StateRecordFacts[],
call: StateRecordFacts,
chronology: StateRecordChronology,
): StateRecordFacts | undefined => {
let latest: StateRecordFacts | undefined;
for (const read of reads)
if (
read.end <= call.start &&
(latest === undefined || read.end >= latest.end)
chronology.completedBefore(read, call) &&
(latest === undefined ||
chronology.endMs(read)! >= chronology.endMs(latest)!)
)
latest = read;
return latest;
@@ -281,10 +286,10 @@ const lastFinishedBefore = (
const repeatsRead = (
calls: readonly StateRecordFacts[],
records: readonly StateRecordFacts[],
{ position, isWrite, response }: Timeline,
{ position, isWrite, response, chronology }: Timeline,
): boolean => {
const answering = calls.filter(succeeded);
const reads = lastReads(answering, records, position, isWrite);
const reads = lastReads(answering, records, position, isWrite, chronology);
return answering.every((call) => {
const read = reads.get(call.ordinal);
if (read === undefined) return false;
@@ -302,6 +307,7 @@ const lastReads = (
records: readonly StateRecordFacts[],
position: ReadonlyMap<number, number>,
isWrite: (index: number) => boolean,
chronology: StateRecordChronology,
): Map<number, StateRecordFacts> => {
const paths = new Set(calls.map((call) => call.path));
const callAt = new Map(
@@ -317,12 +323,12 @@ const lastReads = (
for (let index = 0; index <= lastCall; index += 1) {
const record = records[index]!;
const call = callAt.get(index);
if (call !== undefined) {
if (call !== undefined && chronology.startMs(call) !== undefined) {
const ready = waiting.get(call.path);
for (
let read = ready?.popEndedBy(call.start);
let read = ready?.popEndedBy(chronology.startMs(call)!);
read !== undefined;
read = ready?.popEndedBy(call.start)
read = ready?.popEndedBy(chronology.startMs(call)!)
)
latest.set(call.path, read);
const read = latest.get(call.path);
@@ -333,9 +339,11 @@ const lastReads = (
} else if (
record.method === "GET" &&
succeeded(record) &&
paths.has(record.path)
paths.has(record.path) &&
chronology.endMs(record) !== undefined
) {
const heap = waiting.get(record.path) ?? new ReadsByEnd(position);
const heap =
waiting.get(record.path) ?? new ReadsByEnd(position, chronology);
heap.push(record);
waiting.set(record.path, heap);
}
@@ -347,12 +355,15 @@ const lastReads = (
class ReadsByEnd {
readonly #reads: StateRecordFacts[] = [];
constructor(private readonly position: ReadonlyMap<number, number>) {}
constructor(
private readonly position: ReadonlyMap<number, number>,
private readonly chronology: StateRecordChronology,
) {}
#before(left: StateRecordFacts, right: StateRecordFacts): boolean {
return (
left.end < right.end ||
(left.end === right.end &&
this.chronology.endMs(left)! < this.chronology.endMs(right)! ||
(this.chronology.endMs(left) === this.chronology.endMs(right) &&
this.position.get(left.ordinal)! < this.position.get(right.ordinal)!)
);
}
@@ -372,7 +383,8 @@ class ReadsByEnd {
popEndedBy(time: number): StateRecordFacts | undefined {
const reads = this.#reads;
const first = reads[0];
if (first === undefined || first.end > time) return undefined;
if (first === undefined || this.chronology.endMs(first)! > time)
return undefined;
const last = reads.pop()!;
if (reads.length > 0) {
reads[0] = last;
@@ -431,7 +443,7 @@ const lastMeaningfulWord = (segment: string): string | undefined =>
const changesReadsAcross = (
calls: readonly StateRecordFacts[],
records: readonly StateRecordFacts[],
{ position, writes, response }: Timeline,
{ position, writes, response, chronology }: Timeline,
): boolean => {
const stretches = new Map<
string,
@@ -451,7 +463,8 @@ const changesReadsAcross = (
else stretch.calls.push(call);
}
for (const stretch of stretches.values())
if (changesInStretch(stretch, records, { position, response })) return true;
if (changesInStretch(stretch, records, { position, response, chronology }))
return true;
return false;
};
@@ -464,7 +477,11 @@ const changesInStretch = (
readonly calls: readonly StateRecordFacts[];
},
records: readonly StateRecordFacts[],
{ position, response }: Pick<Timeline, "position" | "response">,
{
position,
response,
chronology,
}: Pick<Timeline, "position" | "response" | "chronology">,
): boolean => {
const shown = (record: StateRecordFacts): string | symbol =>
succeeded(record) ? (response(record.ordinal) ?? UNKNOWN) : ABSENT;
@@ -493,9 +510,12 @@ const changesInStretch = (
const earlier = lastFinishedBefore(
reads.filter((read) => position.get(read.ordinal)! < at),
call,
chronology,
);
const later = reads.find(
(read) => position.get(read.ordinal)! > at && read.start >= call.end,
(read) =>
position.get(read.ordinal)! > at &&
chronology.completedBefore(call, read),
);
if (earlier === undefined || later === undefined) return false;
const [was, now] = [shown(earlier), shown(later)];
@@ -515,11 +535,12 @@ export const stateWritesOf = (
tools: readonly StateToolFacts[],
records: readonly StateRecordFacts[],
io: QueryCallIo,
chronology: StateRecordChronology,
): {
readonly readOnly: ReadonlySet<string>;
readonly writes: readonly StateRecordFacts[];
} => {
const readOnly = readOnlyTools(tools, records, io);
const readOnly = readOnlyTools(tools, records, io, chronology);
const reads = new Set(
tools
.filter((tool) => readOnly.has(tool.name))
@@ -0,0 +1,112 @@
import { expect, it } from "vitest";
import { parseOffsetDateTimeInstant } from "../offsetDateTime.js";
import { StateRecordChronology } from "./StateRecordChronology.js";
import { rewriteCuts } from "./environmentStateRewrites.js";
import {
writeSpanCheck,
writeSpanIndex,
type StateRecordFacts,
} from "./environmentStateRules.js";
const record = (
ordinal: number,
method: string,
start: number,
endMs: number | undefined,
): StateRecordFacts => ({
ordinal,
method,
path: "/api/items/1",
query: undefined,
status: 200,
timing: {
start: {
instant: parseOffsetDateTimeInstant(new Date(start).toISOString()),
ms: start,
},
endMs,
},
});
it.each([
{ label: "unknown duration", start: 10, end: undefined, possible: true },
{ label: "completed before the reads", start: 10, end: 19, possible: false },
{
label: "completed at the first read's start",
start: 10,
end: 20,
possible: false,
},
{
label: "known zero duration at the boundary",
start: 20,
end: 20,
possible: false,
},
{ label: "known overlap", start: 10, end: 21, possible: true },
])(
"keeps rewrite cuts conservative for an earlier write with $label",
({ start, end, possible }) => {
const write = record(0, "PUT", start, end);
const earlier = record(1, "GET", 20, 25);
const later = record(2, "GET", 30, 35);
const records = [write, earlier, later];
const chronology = new StateRecordChronology(
records,
"recorded-start-time",
);
const cuts = rewriteCuts(
[{ type: "/api/items", earlier: earlier.ordinal, later: later.ordinal }],
[write],
new Map(records.map((value) => [value.ordinal, value])),
chronology,
);
expect(writeSpanCheck([write], chronology)(earlier, later)).toBe(possible);
expect(cuts.get("/api/items")).toBe(possible ? write : undefined);
if (end === undefined) {
// A possible explanation prevents an independence proof; it does not prove
// that the write completed, or that it caused the observed difference.
expect(chronology.completedBefore(write, earlier)).toBe(false);
}
},
);
it("selects the earliest possibly interfering write without replacing a known completed predecessor", () => {
const finished = record(0, "PUT", 0, 5);
const unresolved = record(1, "PUT", 10, undefined);
const earlier = record(2, "GET", 20, 25);
const later = record(3, "GET", 30, 35);
const records = [finished, unresolved, earlier, later];
const chronology = new StateRecordChronology(records, "recorded-start-time");
const writes = [finished, unresolved];
expect(writeSpanIndex(writes, chronology)(earlier, later)).toBe(1);
expect(
rewriteCuts(
[{ type: "/api/items", earlier: earlier.ordinal, later: later.ordinal }],
writes,
new Map(records.map((value) => [value.ordinal, value])),
chronology,
).get("/api/items"),
).toBe(unresolved);
});
it.each([35, undefined])(
"does not invent a later read's completion when locating possible writes (end=%s)",
(end) => {
const earlier = record(0, "GET", 20, 25);
const later = record(1, "GET", 30, end);
const write = record(2, "PUT", 40, 45);
const records = [earlier, later, write];
const chronology = new StateRecordChronology(
records,
"recorded-start-time",
);
const cuts = rewriteCuts(
[{ type: "/api/items", earlier: earlier.ordinal, later: later.ordinal }],
[write],
new Map(records.map((value) => [value.ordinal, value])),
chronology,
);
expect(cuts.get("/api/items")).toBe(end === undefined ? write : undefined);
},
);
@@ -1,4 +1,8 @@
import type { StateRecordFacts } from "./environmentStateRules.js";
import {
writeSpanIndex,
type StateRecordFacts,
} from "./environmentStateRules.js";
import type { StateRecordChronology } from "./StateRecordChronology.js";
/** Two recordings of a type that disagree across a write. */
export interface RewriteSpan {
@@ -8,40 +12,24 @@ export interface RewriteSpan {
}
/**
* Per type whose reads disagree across writes, the earliest write within any such span: one that
* began before the later recording ended and ended after the earlier began, in timeline order.
* Per type whose reads disagree, the earliest write that may intersect the span.
* Unknown interval bounds cannot establish that the observations were independent
* of a write; choosing a conservative cutoff does not prove the write caused a change.
*/
export const rewriteCuts = (
spans: readonly RewriteSpan[],
writes: readonly StateRecordFacts[],
byOrdinal: ReadonlyMap<number, StateRecordFacts>,
chronology: StateRecordChronology,
): Map<string, StateRecordFacts> => {
// The latest end among each prefix of writes, so the first write ending after a time is found by bisection.
const reach: number[] = [];
for (const write of writes)
reach.push(Math.max(reach.at(-1) ?? Number.NEGATIVE_INFINITY, write.end));
const starts = writes.map((write) => write.start);
/** The first index whose value passes `test`, which once true stays true. */
const first = (
values: readonly number[],
test: (value: number) => boolean,
) => {
let [low, high] = [0, values.length];
while (low < high) {
const middle = (low + high) >>> 1;
if (test(values[middle]!)) high = middle;
else low = middle + 1;
}
return low;
};
const possibleWrite = writeSpanIndex(writes, chronology);
const cuts = new Map<string, number>();
for (const span of spans) {
const earlier = byOrdinal.get(span.earlier);
const later = byOrdinal.get(span.later);
if (earlier === undefined || later === undefined) continue;
// Writes come in start order: those begun before the later recording ended are a prefix.
const index = first(reach, (end) => end > earlier.start);
if (index < first(starts, (start) => start >= later.end))
const index = possibleWrite(earlier, later);
if (index !== undefined)
cuts.set(span.type, Math.min(index, cuts.get(span.type) ?? index));
}
return new Map([...cuts].map(([type, index]) => [type, writes[index]!]));
@@ -1,3 +1,5 @@
import type { OffsetDateTimeInstant } from "../offsetDateTime.js";
import type { StateRecordChronology } from "./StateRecordChronology.js";
import type { ListQuery } from "./environmentStateListQuery.js";
import { canonicalJson } from "../comparisonSemantics.js";
import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js";
@@ -21,8 +23,12 @@ export interface StateRecordFacts {
/** WHATWG query spelling including `?`, or undefined when the request had none. */
readonly query: string | undefined;
readonly status: number | null;
readonly start: number;
readonly end: number;
readonly timing: {
readonly start:
| { readonly instant: OffsetDateTimeInstant; readonly ms: number }
| undefined;
readonly endMs: number | undefined;
};
}
/** The manifest tool facts extraction uses. */
@@ -392,26 +398,54 @@ export const isStateWrite = (record: StateRecordFacts): boolean =>
changesState(record.method, record.path) &&
(record.status === null || (record.status >= 200 && record.status < 400));
/** Whether any write overlaps the span from one record's start to a later record's end; writes come in timeline order. */
export const writeSpanCheck = (
/** Index possible write interference without inventing completion for unknown intervals. */
export const writeSpanIndex = (
writes: readonly StateRecordFacts[],
): ((earlier: StateRecordFacts, later: StateRecordFacts) => boolean) => {
// The latest end among each prefix of writes, so one search answers a span.
const reach: number[] = [];
for (const write of writes)
reach.push(Math.max(reach.at(-1) ?? Number.NEGATIVE_INFINITY, write.end));
chronology: StateRecordChronology,
): ((
earlier: StateRecordFacts,
later: StateRecordFacts,
) => number | undefined) => {
const reach: (number | undefined)[] = [];
const starts: (number | undefined)[] = [];
let maximum: number | undefined;
let unknown = false;
for (const write of writes) {
const end = chronology.endMs(write);
if (end === undefined) unknown = true;
else maximum = maximum === undefined ? end : Math.max(maximum, end);
reach.push(unknown ? undefined : maximum);
starts.push(chronology.startMs(write));
}
return (earlier, later) => {
const beginning = chronology.startMs(earlier);
const ending = chronology.endMs(later);
let low = 0;
let high = writes.length;
while (low < high) {
const middle = (low + high) >>> 1;
if (writes[middle]!.start < later.end) low = middle + 1;
else high = middle;
const end = reach[middle];
if (beginning === undefined || end === undefined || end > beginning)
high = middle;
else low = middle + 1;
}
return low > 0 && reach[low - 1]! > earlier.start;
const start = starts[low];
return low < writes.length &&
(ending === undefined || start === undefined || start < ending)
? low
: undefined;
};
};
/** Whether a write may overlap the span; unknown completion cannot prove absence. */
export const writeSpanCheck = (
writes: readonly StateRecordFacts[],
chronology: StateRecordChronology,
): ((earlier: StateRecordFacts, later: StateRecordFacts) => boolean) => {
const index = writeSpanIndex(writes, chronology);
return (earlier, later) => index(earlier, later) !== undefined;
};
/** Literal leading segments shared by every template with the same first segment: each path's API base. */
export const apiBaseLengths = (
templates: readonly string[],
@@ -143,18 +143,185 @@ it("falls back to the observed time with no duration when a request never ended
);
expect(records.records[0]?.reported).toMatchObject({
startedDateTime: "2026-10-17T18:00:00.015Z",
time: 0,
});
expect(records.records[1]?.reported).toMatchObject({
startedDateTime: "2026-10-17T18:00:01.015Z",
time: 0,
});
expect(records.records[0]?.reported).not.toHaveProperty("time");
expect(records.records[1]?.reported).not.toHaveProperty("time");
expect(records.limitations).toContain(
"2 scenario request(s) had no unique reported duration; their duration is omitted and does not establish zero-length execution.",
);
expect(records.records[2]?.reported).toMatchObject({
startedDateTime: "2026-10-17T18:00:02.000Z",
time: 5,
});
const state = environmentStateViewOf(
scenarioCaptureEvidence({
...capture,
events: { ...capture.events, items },
}),
);
expect(state.timeline).toBe("recorded-start-time");
expect(state.write_clock).toBeNull();
const clockPlans = state.writes.filter((plan) =>
plan.rules.some((rule) => rule.rule === "clock"),
);
expect(clockPlans.length).toBeGreaterThan(0);
expect(clockPlans.every((plan) => !plan.served && plan.reason !== null)).toBe(
true,
);
expect(clockPlans.some((plan) => plan.reason === "unknown-clock")).toBe(true);
});
it.each(["manifest", "replay"] as const)(
"preserves independent duration and explicit zero in %s projection",
(purpose) => {
const capture = timed(scenarioCaptureFixture(EXCHANGES));
const first = capture.events.items.find(
(event) => event.kind === "request",
);
if (first === undefined) throw new Error("Expected first request");
const items = capture.events.items.map((event) => {
if (
!("transaction_id" in event) ||
event.transaction_id !== first.transaction_id
)
return event;
if (event.kind === "request") {
const { observed_at: _omitted, ...withoutStart } = event;
return withoutStart;
}
if (event.kind === "request-finished") {
const { started_at: _omitted, ...withoutStart } = event;
return { ...withoutStart, elapsed_ms: 0 };
}
return event;
});
const source = { ...capture, events: { ...capture.events, items } };
const before = structuredClone(source);
const records = scenarioCaptureRecords(source, purpose);
expect(records.records[0]?.reported).toHaveProperty("time", 0);
expect(records.records[0]?.reported).not.toHaveProperty("startedDateTime");
expect(records.records[0]?.location).toEqual({
kind: "json-pointer",
pointer: `/events/items/${capture.events.items.indexOf(first)}`,
});
expect(records.limitations.join(" ")).not.toContain(
"had no unique reported duration",
);
expect(source).toEqual(before);
},
);
it.each([
"known zero",
"missing terminal",
"duplicate terminal",
"terminal without duration",
] as const)(
"requires a completed scenario action before resync and write proof: %s",
(mode) => {
const exchanges: ScenarioExchange[] = [
{
method: "GET",
url: "/api/books",
body: JSON.stringify([
{ id: 1, name: "A" },
{ id: 2, name: "B" },
]),
},
{
method: "GET",
url: "/api/books/1",
body: JSON.stringify({ id: 1, name: "A" }),
},
{
method: "GET",
url: "/api/books/2",
body: JSON.stringify({ id: 2, name: "B" }),
},
{
method: "POST",
url: "/api/books/1/archive",
status: 204,
body: "",
requestBody: "{}",
requestMediaType: "application/json",
},
{
method: "GET",
url: "/api/books/1",
body: JSON.stringify({ id: 1, name: "A" }),
},
{
method: "PATCH",
url: "/api/books/1",
body: JSON.stringify({ id: 1, name: "Z" }),
requestBody: JSON.stringify({ name: "Z" }),
requestMediaType: "application/json",
},
{
method: "GET",
url: "/api/books/1",
body: JSON.stringify({ id: 1, name: "Z" }),
},
].map((exchange) => ({
...exchange,
status: exchange.status ?? 200,
mediaType: "application/json",
}));
const capture = timed(scenarioCaptureFixture(exchanges));
const action = capture.events.items.find(
(event) => event.kind === "request" && event.method === "POST",
);
if (action?.kind !== "request") throw new Error("Expected action request");
const terminal = capture.events.items.find(
(event) =>
event.kind === "request-finished" &&
event.transaction_id === action?.transaction_id,
);
if (terminal?.kind !== "request-finished")
throw new Error("Expected action terminal");
const items = capture.events.items.flatMap<
BrowserScenarioCapture["events"]["items"][number]
>((event) => {
if (event !== terminal) return [event];
if (mode === "missing terminal") return [];
if (mode === "terminal without duration") {
const { elapsed_ms: _omitted, ...withoutDuration } = event;
return [withoutDuration];
}
return [{ ...event, elapsed_ms: 0 }];
});
if (mode === "duplicate terminal")
items.push({
...terminal,
sequence: Math.max(...items.map((event) => event.sequence)) + 1,
});
const source = {
...capture,
events: { ...capture.events, items, retained: items.length },
};
const projected = scenarioCaptureRecords(source, "manifest");
if (mode === "known zero")
expect(projected.records[3]?.reported).toHaveProperty("time", 0);
else expect(projected.records[3]?.reported).not.toHaveProperty("time");
const state = environmentStateViewOf(scenarioCaptureEvidence(source));
expect(state.timeline).toBe("recorded-start-time");
const plan = state.writes.find(
(write) => write.tool === "patch_books_by_id",
);
expect(plan).toMatchObject({
served: mode === "known zero",
checked_record_ordinals: mode === "known zero" ? [6] : [],
contradicted_by: null,
});
if (mode === "known zero") expect(state.write_clock).not.toBeNull();
else expect(state.write_clock).toBeNull();
},
);
it("keeps record order and learns no clock when any request is untimed", () => {
const capture = timed(scenarioCaptureFixture(EXCHANGES));
const items = capture.events.items.map((event, index) => {
@@ -41,6 +41,8 @@ import {
type LineShape,
} from "./environmentStateVariants.js";
import { rewriteCuts } from "./environmentStateRewrites.js";
import { StateRecordChronology } from "./StateRecordChronology.js";
import { offsetDateTimeInstantFromEpochMilliseconds } from "../offsetDateTime.js";
import { ENTRY_BYTES } from "./webNetworkRouteTemplates.js";
type State = Extract<AnalysisViewResult, { kind: "environment-state" }>;
@@ -839,8 +841,7 @@ it("leaves every line checked rather than compare more lines than the variant pa
path: "/api/items",
query: `?view=${String(ordinal)}`,
status: 200,
start: ordinal,
end: ordinal,
timing: { start: undefined, endMs: undefined },
}));
const shape = {
entities: new Map(
@@ -867,8 +868,7 @@ it("tells apart member names that hold the separator, so a line spelling them ot
path: "/api/items/1",
query,
status: 200,
start: ordinal,
end: ordinal,
timing: { start: undefined, endMs: undefined },
});
const observed = (ordinal: number, object: JsonObject) => ({
position: ordinal,
@@ -908,8 +908,7 @@ it("leaves every line checked when comparing many mostly empty lines would excee
method: "GET",
path: "/api/items",
status: 200,
start: ordinal,
end: ordinal,
timing: { start: undefined, endMs: undefined },
};
return Object.defineProperty(record, "query", {
enumerable: true,
@@ -950,8 +949,7 @@ const expandedItemLines = (items: number) => {
path: `/api/items/${String(ordinal % items)}`,
query: ordinal < items ? undefined : "?expand=shelf",
status: 200,
start: ordinal,
end: ordinal,
timing: { start: undefined, endMs: undefined },
}),
);
const shapes = new Map(
@@ -1137,11 +1135,21 @@ it("finds each span's earliest write without scanning the writes before it", ()
path: "/api/log",
query: undefined,
status: 200,
timing: {
start: {
instant: offsetDateTimeInstantFromEpochMilliseconds(start),
get ms() {
reads += 1;
return start;
},
},
get endMs() {
reads += 1;
return end;
},
},
};
return Object.defineProperties(record, {
start: { enumerable: true, get: () => ((reads += 1), start) },
end: { enumerable: true, get: () => ((reads += 1), end) },
}) as typeof record & { readonly start: number; readonly end: number };
return record;
};
const count = 2000;
const writes = Array.from({ length: count }, (_, index) =>
@@ -1162,7 +1170,11 @@ it("finds each span's earliest write without scanning the writes before it", ()
later: earlier,
}));
reads = 0;
const cuts = rewriteCuts(spans, writes, reads_);
const chronology = new StateRecordChronology(
[...writes, ...reads_.values()],
"recorded-start-time",
);
const cuts = rewriteCuts(spans, writes, reads_, chronology);
expect(cuts.size).toBe(count - 1);
expect(reads).toBeLessThan(200 * count);
});
@@ -1423,6 +1435,125 @@ it.each<{
},
);
it.each([
{ duration: undefined, redactedWrite: false },
{ duration: null, redactedWrite: false },
{ duration: -1, redactedWrite: false },
{ duration: "0", redactedWrite: false },
{ duration: undefined, redactedWrite: true },
{ duration: null, redactedWrite: true },
])(
"retains chronological initial state with duration $duration and redacted write $redactedWrite",
({ duration, redactedWrite }) => {
const entry = (
method: string,
path: string | null,
second: string,
body: JsonValue,
) => ({
startedDateTime: `2026-10-20T09:00:${second}Z`,
time: 0,
request: {
method,
url: path === null ? null : `http://app.example${path}`,
},
response: {
status: 200,
content: { mimeType: "application/json", text: JSON.stringify(body) },
},
});
const { time: _duration, ...laterRead } = entry(
"GET",
"/api/settings",
"02",
{ theme: "dark" },
);
const entries = [
entry("GET", "/api/settings", "00", { theme: "light" }),
{ ...laterRead, ...(duration === undefined ? {} : { time: duration }) },
entry("PATCH", redactedWrite ? null : "/api/settings", "01", {
theme: "dark",
}),
entry("GET", "/api/health", "00.1", { healthy: true }),
entry("GET", "/api/version", "00.2", { version: "1" }),
];
const capture = webNetworkCaptureSchema.parse(
environmentManifestCaptureFixture(entries),
);
const before = structuredClone(capture);
const state = stateOf(parentOf(analysisViewCaptureEvidence(capture)));
expect(state.timeline).toBe("recorded-start-time");
expect(state.limitations).toContain(
"Some records carry no usable request duration; their completion and overlap remain unknown.",
);
expect(state.documents).toEqual(
expect.arrayContaining([
expect.objectContaining({
path: "/api/settings",
value: { theme: "light" },
record_ordinal: 0,
}),
expect.objectContaining({
path: "/api/health",
value: { healthy: true },
record_ordinal: 3,
}),
expect.objectContaining({
path: "/api/version",
value: { version: "1" },
record_ordinal: 4,
}),
]),
);
expect(capture).toEqual(before);
if (redactedWrite)
expect(state.limitations).toContain(
"1 of the recorded writes had a redacted URL; each ends the initial state of every type, since its target is unknown.",
);
},
);
it("does not confirm completion of an untimed delete while preserving unrelated known reads", () => {
const source = captureOf([
["GET", "/api/items", [{ id: 1 }, { id: 2 }]],
["GET", "/api/items/2", { id: 2 }],
["GET", "/api/health", { healthy: true }],
["GET", "/api/version", { version: "1" }],
["DELETE", "/api/items/2", "", 204],
["GET", "/api/items/2", null, 404],
]);
const known = stateOf(source);
expect(known.writes).toContainEqual(
expect.objectContaining({ tool: "delete_items_by_id", served: true }),
);
const result = structuredClone(source.normalizedResult) as {
records: { reported: { time?: JsonValue } }[];
};
delete result.records[4]!.reported.time;
const before = structuredClone(result);
const state = stateOf({ ...source, normalizedResult: result });
expect(state.timeline).toBe("recorded-start-time");
expect(state.limitations).toContain(
"Some records carry no usable request duration; their completion and overlap remain unknown.",
);
expect(state.writes).toContainEqual(
expect.objectContaining({ tool: "delete_items_by_id", served: false }),
);
expect(state.documents).toEqual(
expect.arrayContaining([
expect.objectContaining({
path: "/api/health",
value: { healthy: true },
}),
expect.objectContaining({
path: "/api/version",
value: { version: "1" },
}),
]),
);
expect(result).toEqual(before);
});
it("bounds a parameterized namespace by concrete writes before publishing new items", () => {
const result = projectAnalysisView(
captureOf([
@@ -1819,3 +1950,68 @@ it("preserves own prototype-named entity and document fields through capture pro
expect.objectContaining({ value: document }),
);
});
it.each([
{
starts: [
"Tue, 20 Oct 2026 09:00:00 GMT",
"Tue, 20 Oct 2026 09:02:00 GMT",
"Tue, 20 Oct 2026 09:01:00 GMT",
],
missingDuration: false,
},
{
starts: [
"Tue, 20 Oct 2026 14:30:00 GMT+0530",
"Tue, 20 Oct 2026 14:32:00 GMT+0530",
"Tue, 20 Oct 2026 14:31:00 GMT+0530",
],
missingDuration: true,
},
{
starts: [
"October 20, 2026 09:00:00",
"October 20, 2026 09:02:00",
"October 20, 2026 09:01:00",
],
missingDuration: true,
},
])(
"preserves native chronological state and discloses interpretation ($missingDuration)",
({ starts, missingDuration }) => {
const entries = starts.map((startedDateTime, ordinal) => ({
startedDateTime,
...(missingDuration && ordinal === 2 ? {} : { time: 0 }),
request: {
method: ordinal === 2 ? "PATCH" : "GET",
url: "http://app.example/api/settings",
},
response: {
status: 200,
content: {
mimeType: "application/json",
text: ordinal === 0 ? '{"theme":"light"}' : '{"theme":"dark"}',
},
},
}));
const capture = webNetworkCaptureSchema.parse(
environmentManifestCaptureFixture(entries),
);
const original = structuredClone(capture.records);
const state = stateOf(parentOf(analysisViewCaptureEvidence(capture)));
expect(state.timeline).toBe("recorded-start-time");
expect(state.documents).toEqual([
{
tool: "get_settings",
path: "/api/settings",
value: { theme: "light" },
record_ordinal: 0,
},
]);
expect(state.limitations).toContain(
"Some recorded start spellings use native Date parsing at millisecond precision; spellings without an explicit zone depend on the host timezone.",
);
expect(capture.records).toEqual(original);
if (missingDuration) expect(state.write_clock).toBeNull();
},
);
@@ -347,6 +347,7 @@ const writePlanSchema = z.strictObject({
"side-effect",
"unconfirmed",
"incomplete-entities",
"unknown-clock",
])
.nullable(),
exemplar_ordinal: ordinalSchema
@@ -0,0 +1,44 @@
import { expect, it } from "vitest";
import {
forgetChecked,
runningAt,
touchTarget,
WriteEffects,
type PendingTargets,
type TouchedTargets,
} from "./environmentStateWriteAttribution.js";
it("keeps an unknown write completion pending alongside later known completions", () => {
const touched: TouchedTargets = new Map();
const pending: PendingTargets = new Map();
for (const [index, end] of [undefined, 20].entries())
touchTarget(touched, pending, "entity", {
index,
tool: `write-${String(index)}`,
end,
fields: ["name"],
});
expect(runningAt(pending, "entity", 30)).toBe(true);
expect(runningAt(pending, "entity", undefined)).toBe(true);
expect(runningAt(pending, "absent", undefined)).toBe(false);
forgetChecked(
{ touched, since: pending, effects: new WriteEffects() },
"entity",
undefined,
);
expect(runningAt(pending, "entity", 30)).toBe(false);
});
it("requires a known read start to establish a known pending write completed", () => {
const touched: TouchedTargets = new Map();
const pending: PendingTargets = new Map();
touchTarget(touched, pending, "entity", {
index: 0,
tool: "write",
end: 20,
fields: ["name"],
});
expect(runningAt(pending, "entity", undefined)).toBe(true);
expect(runningAt(pending, "entity", 19)).toBe(true);
expect(runningAt(pending, "entity", 20)).toBe(false);
});
@@ -28,12 +28,14 @@ export type TouchedTargets = Map<string, Map<string, readonly string[] | null>>;
/** Per target, when the latest modelled write since its last check ended, and each tool's latest call there. */
export type PendingTargets = Map<
string,
{ until: number; readonly at: Map<string, number> }
{ until: number | undefined; readonly at: Map<string, number> }
>;
/** Whether a write that ended at `until` was still running when a read began at `start`, so the read may show either state. */
export const overlapsWrite = (until: number, start: number): boolean =>
until > start;
export const overlapsWrite = (
until: number | undefined,
start: number | undefined,
): boolean => until === undefined || start === undefined || until > start;
/** What the gate tracks of the writes since each target's last check, and when each unmodelled write ran. */
export interface WriteHistory {
@@ -76,8 +78,11 @@ export const interference = (
export const runningAt = (
pending: PendingTargets,
key: string,
start: number,
): boolean => overlapsWrite(pending.get(key)?.until ?? -Infinity, start);
start: number | undefined,
): boolean => {
const held = pending.get(key);
return held !== undefined && overlapsWrite(held.until, start);
};
/** The tools whose writes a read exercises: those that changed its targets since they were last checked, less those that borrowed the read to learn their effect. */
export const exercisedBy = (
@@ -100,13 +105,18 @@ export const touchTarget = (
write: {
readonly index: number;
readonly tool: string;
readonly end: number;
readonly end: number | undefined;
readonly fields: readonly string[] | undefined;
},
): void => {
const held = pending.get(target);
pending.set(target, {
until: Math.max(held?.until ?? -Infinity, write.end),
// One unknown completion keeps this target pending until a real check clears it.
until:
write.end === undefined ||
(held !== undefined && held.until === undefined)
? undefined
: Math.max(held?.until ?? -Infinity, write.end),
at: (held?.at ?? new Map()).set(write.tool, write.index),
});
const tools = touched.get(target) ?? new Map();
@@ -0,0 +1,112 @@
import { expect, it, vi } from "vitest";
import * as paths from "./environmentStateRules.js";
import { prefixRelated } from "./environmentStateWriteRules.js";
import { CapturedWriteCompletion } from "./environmentStateWriteCompletion.js";
it("preserves the existing prefix relation, including non-transitive decoding and parameters", () => {
const spellings = [
"",
"api",
"a",
"%61",
"%2561",
"%2F",
"%252F",
"{id}",
"%ZZ",
];
const candidates = [
"/",
...spellings.flatMap((value) => [
`/api/${value}`,
`/api/${value}/child`,
`/${value}/other`,
]),
];
for (const write of candidates) {
const completion = new CapturedWriteCompletion(() => {});
completion.write(write, undefined);
for (const read of candidates)
expect(completion.runningAt(read, 100), `${write} -> ${read}`).toBe(
prefixRelated(write, read),
);
}
});
it("retains unknown completion and isolates unrelated writes while aggregating known ends", () => {
const completion = new CapturedWriteCompletion(() => {});
completion.write("/api/books/1", 20);
completion.write("/api/books/1", 30);
completion.write("/api/books/2", undefined);
completion.write("/api/books/2", 10);
expect(completion.runningAt("/api/books/1", 29)).toBe(true);
expect(completion.runningAt("/api/books/1", 30)).toBe(false);
expect(completion.runningAt("/api/books/1", undefined)).toBe(true);
expect(completion.runningAt("/api/books/2", 100)).toBe(true);
expect(completion.runningAt("/api/books", 100)).toBe(true);
expect(completion.runningAt("/api/authors", undefined)).toBe(false);
completion.write("/", undefined);
expect(completion.runningAt("/api/authors", 100)).toBe(true);
});
it("uses explicit record-order coordinates without treating absent writes as pending", () => {
const completion = new CapturedWriteCompletion(() => {});
expect(completion.runningAt("/api/books", undefined)).toBe(false);
completion.write("/api/books", 1);
expect(completion.runningAt("/api/books/1", 2)).toBe(false);
});
it("looks up a concrete path without reinterpreting every retained write path", () => {
const completion = new CapturedWriteCompletion(() => {});
for (let id = 0; id < 128; id += 1)
completion.write(`/api/books/${String(id)}`, undefined);
const decoded = vi.spyOn(paths, "decodedSegment");
try {
expect(completion.runningAt("/api/books/64", 100)).toBe(true);
// Matching interprets only the three read segments; prior write segments were indexed once.
expect(decoded).toHaveBeenCalledTimes(3);
} finally {
decoded.mockRestore();
}
});
it("reserves retained prefix growth before publishing an unbudgeted edge", () => {
const refused = new Error("state retained-memory budget exhausted");
let remaining = Infinity;
const completion = new CapturedWriteCompletion((bytes) => {
if (bytes > remaining) throw refused;
remaining -= bytes;
});
completion.write("/api/books/1", 10);
remaining = 0;
// Updating an indexed path retains no new keys or nodes.
completion.write("/api/books/1", 20);
expect(completion.runningAt("/api/books/1", 19)).toBe(true);
expect(() => completion.write("/api/authors/1", undefined)).toThrow(refused);
expect(completion.runningAt("/api/authors/1", 100)).toBe(false);
expect(completion.runningAt("/api", 100)).toBe(false);
expect(completion.runningAt("/", 100)).toBe(false);
expect(completion.runningAt("/api/{collection}", 100)).toBe(false);
expect(completion.runningAt("/api/books/1", 20)).toBe(false);
});
it("queries both interval boundaries across future writes in chronology order", () => {
const completion = new CapturedWriteCompletion(() => {});
completion.write("/api/books/1", 15, 10);
completion.write("/api/books/1", undefined, 30);
expect(completion.runningAt("/api/books/1", 5, 10)).toBe(false);
expect(completion.runningAt("/api/books/1", 5, 11)).toBe(true);
expect(completion.runningAt("/api/books/1", 15, 30)).toBe(false);
expect(completion.runningAt("/api/books/1", 15, 31)).toBe(true);
expect(completion.runningAt("/api/books/1", 15, undefined)).toBe(true);
expect(completion.runningAt("/api/authors", 5, undefined)).toBe(false);
});
it("keeps a simultaneous unknown write ambiguous for a point read, unlike a known point write", () => {
const completion = new CapturedWriteCompletion(() => {});
completion.write("/api/books/1", undefined, 10);
completion.write("/api/books/2", 10, 10);
expect(completion.runningAt("/api/books/1", 10, 10)).toBe(true);
expect(completion.runningAt("/api/books/2", 10, 10)).toBe(false);
expect(completion.runningAt("/api/books/3", 10, 10)).toBe(false);
});
@@ -0,0 +1,149 @@
import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js";
import {
decodedSegment,
isParam,
segmentsOf,
} from "./environmentStateRules.js";
/** Ordered write starts and the latest completion among writes through each start. */
class CompletionHistory {
readonly #starts: number[] = [];
readonly #ends: (number | undefined)[] = [];
constructor(private readonly retain: (bytes: number) => void) {}
reserve(start: number): void {
if (this.#starts.at(-1) !== start) this.retain(ENTRY_BYTES);
}
append(start: number, end: number | undefined): void {
const previous = this.#ends.at(-1);
const maximum =
end === undefined || (this.#ends.length > 0 && previous === undefined)
? undefined
: Math.max(previous ?? -Infinity, end);
if (this.#starts.at(-1) === start)
this.#ends[this.#ends.length - 1] = maximum;
else {
this.#starts.push(start);
this.#ends.push(maximum);
}
}
overlaps(start: number | undefined, end: number | undefined): boolean {
// Admit same-start writes for a point observation; a known zero-length write
// is still excluded by its completion below. Otherwise starts at read end are disjoint.
// Find writes begun before the read completed. Unknown read completion
// admits every captured write; all histories are built in chronology order.
let low = 0;
let high = this.#starts.length;
while (low < high) {
const middle = Math.floor((low + high) / 2);
if (
end === undefined ||
this.#starts[middle]! < end ||
(start === end && this.#starts[middle] === end)
)
low = middle + 1;
else high = middle;
}
if (low === 0) return false;
const completion = this.#ends[low - 1];
return (
completion === undefined || start === undefined || completion > start
);
}
}
class CompletionPrefix {
terminal: CompletionHistory | undefined;
subtree: CompletionHistory | undefined;
readonly children: Map<string, CompletionPrefix>;
readonly decoded: Map<string, Set<CompletionPrefix>>;
parameter: CompletionPrefix | undefined;
constructor(private readonly retain: (bytes: number) => void) {
// Node, child maps, parameter link and both history owners with their arrays.
retain(ENTRY_BYTES * 10);
this.children = new Map();
this.decoded = new Map();
}
child(segment: string): CompletionPrefix {
if (isParam(segment))
return (this.parameter ??= new CompletionPrefix(this.retain));
const held = this.children.get(segment);
if (held !== undefined) return held;
// Reserve edge entries, inverse-match membership and both retained spellings
// before decoding or allocating. A once-decoded segment cannot grow in length.
this.retain(ENTRY_BYTES * 4 + textBytes(segment.length) * 2);
const child = new CompletionPrefix(this.retain);
const decoded = decodedSegment(segment);
this.children.set(segment, child);
this.decoded.set(
decoded,
(this.decoded.get(decoded) ?? new Set()).add(child),
);
return child;
}
/** Keep verbatim, once-decoded and inverse-decoded matches distinct: decoding is not transitive. */
matching(segment: string): Set<CompletionPrefix> {
if (isParam(segment))
return new Set([
...this.children.values(),
...(this.parameter === undefined ? [] : [this.parameter]),
]);
const matches = new Set(this.decoded.get(segment));
for (const spelling of [segment, decodedSegment(segment)]) {
const child = this.children.get(spelling);
if (child !== undefined) matches.add(child);
}
if (this.parameter !== undefined) matches.add(this.parameter);
return matches;
}
}
/** Index captured write intervals by the same segment-prefix relation as write attribution. */
export class CapturedWriteCompletion {
readonly #root: CompletionPrefix;
constructor(private readonly retain: (bytes: number) => void) {
this.#root = new CompletionPrefix(retain);
}
/** Admit writes in nondecreasing chronology start order. */
write(path: string, end: number | undefined, start = -Infinity): void {
let node = this.#root;
const ancestors = [node];
// Finish budgeted path admission before publishing any completion summary.
// A refused edge may leave reserved empty prefixes, but no write facts.
for (const segment of segmentsOf(path)) {
node = node.child(segment);
ancestors.push(node);
}
const histories = ancestors.map(
(ancestor) => (ancestor.subtree ??= new CompletionHistory(this.retain)),
);
histories.push((node.terminal ??= new CompletionHistory(this.retain)));
for (const history of histories) history.reserve(start);
for (const history of histories) history.append(start, end);
}
/** Unknown completion or read start cannot prove that a related write finished. */
runningAt(path: string, start: number | undefined, end = Infinity): boolean {
const running = (history: CompletionHistory | undefined): boolean =>
history?.overlaps(start, end) === true;
let nodes = new Set([this.#root]);
for (const segment of segmentsOf(path)) {
const next = new Set<CompletionPrefix>();
for (const node of nodes) {
if (running(node.terminal)) return true;
for (const child of node.matching(segment)) next.add(child);
}
if (next.size === 0) return false;
nodes = next;
}
return [...nodes].some((node) => running(node.subtree));
}
}
@@ -4,6 +4,7 @@ import {
rootOf,
type StateRecordFacts,
} from "./environmentStateRules.js";
import { offsetDateTimeInstantFromEpochMilliseconds } from "../offsetDateTime.js";
import { StateWriteCutoffs } from "./environmentStateWriteCutoffs.js";
const write = (path: string, ordinal: number): StateRecordFacts => ({
@@ -12,8 +13,13 @@ const write = (path: string, ordinal: number): StateRecordFacts => ({
method: "POST",
query: undefined,
status: 200,
start: ordinal,
end: ordinal,
timing: {
start: {
instant: offsetDateTimeInstantFromEpochMilliseconds(ordinal),
ms: ordinal,
},
endMs: ordinal,
},
});
it("matches prefix and parameter semantics against an independent linear oracle", () => {
@@ -46,12 +52,12 @@ it("matches prefix and parameter semantics against an independent linear oracle"
const index = new StateWriteCutoffs(
selection,
bases,
(a, b) => a.start < b.start,
(a, b) => a.timing.start!.ms < b.timing.start!.ms,
() => {},
);
for (const root of patterns) {
const expected = [...selection]
.sort((a, b) => a.start - b.start)
.sort((a, b) => a.timing.start!.ms - b.timing.start!.ms)
.find((record) => {
const candidate = rootOf(record.path, bases);
const shared = Math.min(root.length, candidate.length);
@@ -1,3 +1,4 @@
import type { StateRecordChronology } from "./StateRecordChronology.js";
import { jsonTextCodeUnits } from "../jsonSerialization.js";
import type { JsonValue } from "../jsonValue.js";
import {
@@ -143,7 +144,7 @@ export interface WriteSimulationInput {
readonly records: readonly StateRecordFacts[];
/** POST tools whose calls read state rather than change it. */
readonly readOnly: ReadonlySet<string>;
readonly timed: boolean;
readonly chronology: StateRecordChronology;
readonly views: ReadonlyMap<string, StateView>;
readonly types: readonly StateType[];
readonly entities: readonly StateEntity[];
@@ -157,7 +158,7 @@ export interface EffectContext {
readonly state: SimulatedState;
readonly io: WriteIo;
readonly records: readonly StateRecordFacts[];
readonly timed: boolean;
readonly chronology: StateRecordChronology;
/** Whether a record may change state: a write the capture does not show reading only. */
readonly isWrite: (record: StateRecordFacts) => boolean;
readonly viewOf: (ordinal: number) => StateView | undefined;
@@ -286,12 +287,15 @@ export const referencedIn =
export const startOf = (
context: EffectContext,
step: WriteStep,
): number | undefined => (context.timed ? step.record.start : undefined);
): number | undefined =>
context.chronology.timeline === "recorded-start-time"
? step.record.timing.start?.ms
: undefined;
export const searchFrom = (context: EffectContext, step: WriteStep) => ({
records: context.records,
from: step.index,
...(context.timed ? { after: step.record.end } : {}),
chronology: context.chronology,
viewOf: context.viewOf,
withBody: context.io.withBody,
queryOf: context.io.queryOf,
@@ -1,3 +1,5 @@
import { StateRecordChronology } from "./StateRecordChronology.js";
import { parseOffsetDateTimeInstant } from "../offsetDateTime.js";
import { describe, expect, it } from "vitest";
import type { JsonValue } from "../jsonValue.js";
@@ -15,8 +17,13 @@ const record = (
path,
query: undefined,
status: 200,
start,
end,
timing: {
start: {
instant: parseOffsetDateTimeInstant(new Date(start).toISOString()),
ms: start,
},
endMs: end,
},
});
/** The creation limit of `/api/projects` over list reads and a POST returning `created`, in ordinal order. */
@@ -35,6 +42,10 @@ const projectsLimit = (
creationLimit(
[{ record: post, ids: created }],
{
chronology: new StateRecordChronology(
[...reads, post],
"recorded-start-time",
),
type: "/api/projects",
tools: [
{
@@ -52,6 +63,52 @@ const projectsLimit = (
undefined,
);
describe("short-id creation limits", () => {
it("does not use an unknown completion as the prior read needed to link a short id", () => {
const prior = record(0, "GET", "/api/projects", [0, 5]);
const unknown = {
...prior,
timing: { ...prior.timing, endMs: undefined },
};
const post = record(1, "POST", "/api/imports", [10, 11]);
const later = record(2, "GET", "/api/projects/7", [20, 21]);
const limit = (earlier: StateRecordFacts) =>
creationLimit(
[{ record: post, ids: ["7"] }],
{
type: "/api/projects",
chronology: new StateRecordChronology(
[earlier, post, later],
"recorded-start-time",
),
tools: [
{
name: "list",
method: "GET",
path_template: "/api/projects",
record_ordinals: [0],
},
{
name: "item",
method: "GET",
path_template: "/api/projects/{id}",
record_ordinals: [2],
},
],
records: new Map(
[earlier, post, later].map((entry) => [entry.ordinal, entry]),
),
position: (entry) => entry.ordinal,
withBody: (ordinal, use) => use(ordinal === 0 ? [] : { id: "7" }),
retain: () => undefined,
},
undefined,
);
expect(limit(unknown)).toBeUndefined();
expect(limit(prior)).toBe(post);
});
});
describe("creation limits", () => {
it("reads the ids of a list body longer than a call's arguments can spread", () => {
// A million elements overflow the argument limit of a spread, here and in the test workers' larger stack.
@@ -67,6 +124,10 @@ describe("creation limits", () => {
const limit = creationLimit(
[{ record: post, ids: ["1003"] }],
{
chronology: new StateRecordChronology(
[list, post, later],
"recorded-start-time",
),
type: "/api/metrics",
tools: [
{
@@ -107,7 +168,9 @@ describe("creation limits", () => {
}),
).toBeUndefined();
});
});
describe("the entity identities creation limits use", () => {
it("reads a list's ids from a suffix-named field when no identity name identifies its entities", () => {
const [list, post, later] = [
record(0, "GET", "/api/projects"),
@@ -163,6 +226,10 @@ describe("creation limits", () => {
const limit = creationLimit(
[{ record: post, ids: ["proj-0002"] }],
{
chronology: new StateRecordChronology(
[item, post, later],
"recorded-start-time",
),
type: "/api/projects",
tools: [
{
@@ -282,6 +349,10 @@ describe("the reads creation limits parse", () => {
const limit = creationLimit(
[{ record: post, ids: ["proj-0002"] }],
{
chronology: new StateRecordChronology(
[...reads, post, until],
"recorded-start-time",
),
type: "/api/projects",
tools: [
{
@@ -1,3 +1,4 @@
import type { StateRecordChronology } from "./StateRecordChronology.js";
import type { JsonValue } from "../jsonValue.js";
import {
ownMember,
@@ -140,7 +141,7 @@ const keptIds = (
/** One read of a type: where it ran, when it finished, whether it reads one item, and the ids it shows. */
interface ShownRead {
readonly at: number;
readonly end: number;
readonly end: number | undefined;
readonly item: boolean;
readonly ids: ReadonlySet<string>;
}
@@ -154,11 +155,14 @@ interface ShownRead {
const freshIds = (
ids: readonly string[],
shown: readonly ShownRead[],
post: { readonly at: number; readonly start: number },
post: { readonly at: number; readonly start: number | undefined },
until: number,
): string[] => {
const before = (read: ShownRead): boolean =>
read.at < post.at && read.end <= post.start;
read.at < post.at &&
read.end !== undefined &&
post.start !== undefined &&
read.end <= post.start;
const seen = shown.some(before);
return ids.filter((id) => {
const short = id.length < MIN_ID_LENGTH;
@@ -215,6 +219,7 @@ export const creationLimit = (
creations: readonly CreatedIds[],
reads: {
readonly type: string;
readonly chronology: StateRecordChronology;
readonly tools: readonly StateToolFacts[];
readonly records: ReadonlyMap<number, StateRecordFacts>;
readonly position: (record: StateRecordFacts) => number;
@@ -249,7 +254,7 @@ export const creationLimit = (
return [
{
at,
end: record.end,
end: reads.chronology.endMs(record),
item,
ids: keptIds(ids, returned, reads.retain),
},
@@ -261,7 +266,7 @@ export const creationLimit = (
)) {
const at = reads.position(creation.record);
if (at >= end) break;
const post = { at, start: creation.record.start };
const post = { at, start: reads.chronology.startMs(creation.record) };
if (freshIds(creation.ids, shown, post, end).length > 0)
return creation.record;
}
@@ -276,6 +281,7 @@ export const creationLimit = (
*/
export const linkedCreate = (input: {
readonly tool: StateToolFacts;
readonly chronology: StateRecordChronology;
readonly records: readonly StateRecordFacts[];
readonly isWrite: (record: StateRecordFacts) => boolean;
readonly viewOf: (ordinal: number) => StateView | undefined;
@@ -296,7 +302,7 @@ export const linkedCreate = (input: {
// Only a read finished before the call began shows what was there before it, as `freshIds` decides.
const before = records
.slice(0, first)
.filter((record) => record.end <= post.start);
.filter((record) => input.chronology.completedBefore(record, post));
for (const record of records.slice(first + 1)) {
// Another write may have made what a later read shows, so the search ends there.
if (!own.has(record.ordinal) && input.isWrite(record)) return undefined;
@@ -417,6 +423,7 @@ export const linkedTarget = (
classified: { readonly effect: WriteEffect; readonly target: string | null },
input: {
readonly records: readonly StateRecordFacts[];
readonly chronology: StateRecordChronology;
readonly isWrite: (record: StateRecordFacts) => boolean;
readonly viewOf: (ordinal: number) => StateView | undefined;
readonly entities: readonly StateEntity[];
@@ -437,6 +444,7 @@ export const linkedTarget = (
? linkedCreate({
tool,
records: input.records,
chronology: input.chronology,
isWrite: input.isWrite,
viewOf: input.viewOf,
entities: input.entities,
@@ -1,5 +1,5 @@
import type { JsonValue } from "../jsonValue.js";
import { overlapsWrite } from "./environmentStateWriteAttribution.js";
import type { StateRecordChronology } from "./StateRecordChronology.js";
import {
ownMember,
at,
@@ -126,8 +126,8 @@ export const shownBefore = (
export interface ObservationSearch {
readonly records: readonly StateRecordFacts[];
readonly from: number;
/** A timed write's completion. A read begun while it still ran may show the prior state. */
readonly after?: number;
/** Completion must be established before a read may supply a write observation. */
readonly chronology: StateRecordChronology;
readonly viewOf: (ordinal: number) => StateView | undefined;
readonly withBody: <T>(
ordinal: number,
@@ -192,7 +192,9 @@ export const laterObservation = (
for (const record of search.records.slice(search.from + 1)) {
if (target.isWrite(record)) return undefined;
// The same overlap the check path ignores: the write was still running when this read began.
if (search.after !== undefined && overlapsWrite(search.after, record.start))
if (
!search.chronology.completedBefore(search.records[search.from]!, record)
)
continue;
const view = search.viewOf(record.ordinal);
if (
@@ -33,7 +33,17 @@ export type WriteReason =
| "contradicted"
| "side-effect"
| "unconfirmed"
| "incomplete-entities";
| "incomplete-entities"
| "unknown-clock";
/** Whether serving a plan generates a value from the environment's global clock. */
export const usesWriteClock = (plan: Pick<StateWritePlan, "rules">): boolean =>
plan.rules.some(
(rule) =>
rule.rule === "clock" ||
rule.rule === "held-clock" ||
rule.rule === "nested-clock",
);
/** Rules that generate a value when served; the gate takes the recorded value instead. */
export type GeneratedRule =
@@ -52,6 +52,40 @@ const ISO_SECONDS = {
const stateOf = (rows: readonly TimedCaptureRow[]): StateView =>
environmentStateViewOf(timedCaptureEvidence(rows));
it("retains learned clock rules but withholds their serving when the global clock is unknown", () => {
const baseline = stateOf(BOOK_ROWS);
const unknown = stateOf([
...BOOK_ROWS,
["GET", "/api/health", { ready: true }, 200, undefined, -1],
]);
expect(baseline.write_clock).not.toBeNull();
expect(unknown.write_clock).toBeNull();
const generatesClock = (plan: StateView["writes"][number]) =>
plan.rules.some((rule) =>
["clock", "held-clock", "nested-clock"].includes(rule.rule),
);
const clockPlans = baseline.writes.filter(
(plan) => plan.served && generatesClock(plan),
);
expect(clockPlans.length).toBeGreaterThan(0);
for (const plan of clockPlans)
expect(
unknown.writes.find((candidate) => candidate.tool === plan.tool),
).toEqual({
...plan,
served: false,
reason: "unknown-clock",
});
const independent = baseline.writes.filter(
(plan) => plan.served && !generatesClock(plan),
);
expect(independent.length).toBeGreaterThan(0);
for (const plan of independent)
expect(
unknown.writes.find((candidate) => candidate.tool === plan.tool),
).toEqual(plan);
});
it.each(["delete", "mismatched-query"])(
"does not parse unused request content for %s under a bounded state allowance",
(kind) => {
@@ -1,6 +1,7 @@
import type { ExtractedState } from "./environmentStateRules.js";
import { EnvironmentStateWriteSimulation } from "./EnvironmentStateWriteSimulation.js";
import type { WriteSimulationInput } from "./environmentStateWriteEffects.js";
import { usesWriteClock } from "./environmentStateWriteRules.js";
/** The latest end a record can have and still round up to a whole second a date can hold. */
const LATEST_END_MS = 8_640_000_000_000_000 - 1000;
@@ -8,14 +9,15 @@ const LATEST_END_MS = 8_640_000_000_000_000 - 1000;
/** The instant a served clock starts from: the end of the last timed record a date can hold, to the next whole second. */
const clockStart = (
records: WriteSimulationInput["records"],
timed: boolean,
): string | null => {
if (!timed) return null;
// The clock is global: any unknown completion may extend beyond every known end.
if (records.some((record) => record.timing.endMs === undefined)) return null;
// A record timed past the range of a date carries no usable time, so the clock starts from the others.
const end = records.reduce(
(latest, record) =>
Math.abs(record.end) <= LATEST_END_MS
? Math.max(latest, record.end)
record.timing.endMs !== undefined &&
Math.abs(record.timing.endMs) <= LATEST_END_MS
? Math.max(latest, record.timing.endMs)
: latest,
Number.NEGATIVE_INFINITY,
);
@@ -45,7 +47,17 @@ export const extractWrites = (
entities: state.entities,
documents: state.documents,
});
const writes = simulation.run();
const writeClock =
input.chronology.timeline === "recorded-start-time"
? clockStart(input.records)
: null;
const writes = simulation
.run()
.map((plan) =>
plan.served && writeClock === null && usesWriteClock(plan)
? { ...plan, served: false, reason: "unknown-clock" as const }
: plan,
);
const views = simulation.views(state.views);
const backed = new Set<string>();
const backedTypes = new Set<string>();
@@ -56,7 +68,7 @@ export const extractWrites = (
}
return {
writes,
write_clock: clockStart(input.records, input.timed),
write_clock: writeClock,
views,
types: state.types.filter((type) =>
backedTypes.has(type.collection_template),
@@ -132,18 +132,25 @@ const timingByTransaction = (
};
/**
* HAR timing for a record: the browser's start when it reported one, else when the request was observed, and the
* browser's duration (0 when it reported none).
* HAR timing for a record: the browser's start when it reported one, else the
* request observation. Preserve duration independently; omission is not zero.
*/
const recordTiming = (
request: RequestEvent,
timing: ReadonlyMap<string, BrowserTiming>,
): { readonly startedDateTime: string; readonly time: number } | undefined => {
):
| { readonly startedDateTime?: string; readonly time?: number }
| undefined => {
const browser = timing.get(request.transaction_id);
const started = browser?.started_at ?? request.observed_at;
return started === undefined
return started === undefined && browser?.elapsed_ms === undefined
? undefined
: { startedDateTime: started, time: browser?.elapsed_ms ?? 0 };
: {
...(started === undefined ? {} : { startedDateTime: started }),
...(browser?.elapsed_ms === undefined
? {}
: { time: browser.elapsed_ms }),
};
};
const record = (
@@ -235,9 +242,11 @@ export const scenarioCaptureRecords = (
const records: WebNetworkCaptureRecord[] = [];
const responseBodies = new Map<number, ScenarioResponseBodyRetention>();
let timed = false;
let unknownDurations = 0;
for (const [item, event] of capture.events.items.entries()) {
if (event.kind !== "request") continue;
const timing = recordTiming(event, timings);
if (timing?.time === undefined) unknownDurations += 1;
timed ||= timing !== undefined;
const projected = record(
records.length,
@@ -258,9 +267,14 @@ export const scenarioCaptureRecords = (
"Scenario records are the capture's network transactions in request order; each record ordinal is the transaction's zero-based position.",
...(timed
? [
"Scenario record start times are the browser's own, from when the request ended, or when the provider observed the request where the browser reported none; durations are the browser's own, or 0 where it reported none.",
"Scenario record start times, when available, are the browser's own from a unique terminal event, or when the provider observed the request where the browser reported none; durations retain only values reported by a unique terminal event.",
]
: []),
...(unknownDurations === 0
? []
: [
`${String(unknownDurations)} scenario request(s) had no unique reported duration; their duration is omitted and does not establish zero-length execution.`,
]),
...(capture.events.dropped > 0
? [
`The scenario capture omitted ${String(capture.events.dropped)} events; the manifest covers only retained transactions.`,
+38 -30
View File
@@ -1,3 +1,4 @@
import { BrowserObservationError } from "./browserObservationError.js";
import { EvidenceFileError } from "./evidenceErrors.js";
import type { AnalysisError } from "./analysisErrorBase.js";
import {
@@ -24,7 +25,7 @@ import {
/** Exclude whole identity values; redacted text is no longer a path or digest. */
const excludeSnapshotIdentity = (
value: unknown,
values: readonly string[],
text: (value: string) => string,
): FileSnapshotPartialObservation | undefined => {
if (
value === null ||
@@ -37,10 +38,7 @@ const excludeSnapshotIdentity = (
if (!parsed.success) return undefined;
const snapshot = parsed.data;
const exclude = (identity: string | null): string | null =>
identity !== null &&
values.some((literal) => literal !== "" && identity.includes(literal))
? null
: identity;
identity !== null && text(identity) !== identity ? null : identity;
const path = exclude(snapshot.path);
if (snapshot.kind === "file-snapshot-hash") {
const sha256 = exclude(snapshot.sha256);
@@ -54,17 +52,18 @@ const excludeSnapshotIdentity = (
export const redactExplicitPointer = (
pointer: string,
values: readonly string[],
): string =>
redactSensitivePointer(pointer, (value) => redactExplicitText(value, values));
const redactSensitivePointer = (
pointer: string,
text: (value: string) => string,
): string => {
let parent = "";
for (const segment of pointer.split("/").slice(1)) {
const decoded = segment.replaceAll("~1", "/").replaceAll("~0", "~");
const child = `${parent}/${segment}`;
if (
values.some(
(literal) => decoded.includes(literal) || child.includes(literal),
)
)
return parent;
if (text(decoded) !== decoded || text(child) !== child) return parent;
parent = child;
}
return parent;
@@ -76,25 +75,29 @@ export const redactExplicitFailure = (
values: readonly string[],
): AnalysisError => {
if (values.length === 0) return error;
const snapshot = excludeSnapshotIdentity(error.partialObservation, values);
return redactSensitiveFailure(error, (value) =>
redactExplicitText(value, values),
);
};
/** Apply the producer's parsed sensitivity rules without exporting resolved secrets. */
export const redactSensitiveFailure = (
error: AnalysisError,
text: (value: string) => string,
): AnalysisError => {
const snapshot = excludeSnapshotIdentity(error.partialObservation, text);
if (snapshot !== undefined) error.partialObservation = snapshot;
const redacted = redactFailure(error, values);
const redacted = redactFailure(error, text);
// Other observation owners already apply their format-aware exclusions.
// Keep those facts across diagnostic replacements, including input/capacity.
if (redacted !== error && error.partialObservation !== undefined)
redacted.retainPartialObservation(error.partialObservation);
return redacted === error
? error
: mirrorCleanup(error, redacted, (value) =>
redactExplicitText(value, values),
);
return redacted === error ? error : mirrorCleanup(error, redacted, text);
};
const redactFailure = (
error: AnalysisError,
values: readonly string[],
text: (value: string) => string,
): AnalysisError => {
const text = (value: string): string => redactExplicitText(value, values);
const path = (
parts: readonly (string | number)[],
ordinaryNames: boolean,
@@ -104,11 +107,11 @@ const redactFailure = (
if (typeof part !== "string") {
retained.push(part);
} else if (!ordinaryNames && part.startsWith("/")) {
const pointer = redactExplicitPointer(part, values);
const pointer = redactSensitivePointer(part, text);
retained.push(pointer);
if (pointer !== part) break;
} else {
if (values.some((literal) => part.includes(literal))) break;
if (text(part) !== part) break;
retained.push(part);
}
}
@@ -119,11 +122,11 @@ const redactFailure = (
): Record<string, JsonValue> =>
Object.fromEntries(
Object.entries(value)
.filter(([key]) => !values.some((literal) => key.includes(literal)))
.filter(([key]) => text(key) === key)
.map(([key, item]) => [key, visit(item)]),
);
const visit = (value: JsonValue): JsonValue => {
const snapshot = excludeSnapshotIdentity(value, values);
const snapshot = excludeSnapshotIdentity(value, text);
if (snapshot !== undefined) return snapshot;
if (typeof value === "string") return text(value);
if (Array.isArray(value)) return value.map(visit);
@@ -148,7 +151,7 @@ const redactFailure = (
...(error.cleanupPrimary === undefined
? {}
: {
cleanupPrimary: redactExplicitFailure(error.cleanupPrimary, values),
cleanupPrimary: redactSensitiveFailure(error.cleanupPrimary, text),
}),
...(error.diagnostics === undefined
? {}
@@ -171,12 +174,17 @@ const redactFailure = (
},
}),
};
if (error instanceof BrowserObservationError)
return new BrowserObservationError(error.operation, error.reason, {
...context,
...(error.detail === undefined ? {} : { detail: text(error.detail) }),
...(error.remediationAction === undefined
? {}
: { remediationAction: text(error.remediationAction) }),
});
if (error instanceof EvidenceFileError) {
const identity = (value: string | undefined): string | undefined =>
value !== undefined &&
values.some((literal) => literal !== "" && value.includes(literal))
? undefined
: value;
value !== undefined && text(value) !== value ? undefined : value;
const retainedPath = identity(error.path);
const publishedPath = identity(error.published?.path);
return new EvidenceFileError(error.operation, error.reason, {
+33
View File
@@ -5,6 +5,8 @@ import {
inspectRecordedDateTime,
offsetDateTimeSchema,
readRecordedDateTime,
readRecordedStart,
offsetDateTimeInstantFromEpochMilliseconds,
utcDateTimeSchema,
} from "./offsetDateTime.js";
@@ -244,3 +246,34 @@ const findPattern = (value: unknown): string | undefined => {
}
return undefined;
};
it.each([
"Tue, 20 Oct 2026 09:00:00 GMT",
"Tue, 20 Oct 2026 14:30:00 GMT+0530",
"October 20, 2026 09:00:00",
])("preserves native start interpretation for %s", (value) => {
const milliseconds = Date.parse(value);
expect(Number.isFinite(milliseconds)).toBe(true);
expect(readRecordedStart(value)).toEqual({
instant: offsetDateTimeInstantFromEpochMilliseconds(milliseconds),
interpretation: "native-date",
});
});
it.each([
"2026-02-30T09:00:00Z",
"1900-02-29 09:00:00",
"2026-10-20T24:00:00Z",
"not a recorded start",
])("keeps invalid start %s unknown without native rollover", (value) => {
expect(readRecordedStart(value)).toBeUndefined();
});
it("keeps every ISO start fraction digit and treats its absent zone as UTC", () => {
const value = "2026-10-20 09:00:00.000000000123";
expect(readRecordedStart(value)).toEqual({
instant: readRecordedDateTime(`${value}Z`)!.instant,
interpretation: "recorded-iso",
});
expect(readRecordedStart(value)?.instant.fraction).toBe("000000000123");
});
+25
View File
@@ -109,6 +109,31 @@ export const readRecordedDateTime = (
return inspected.kind === "valid" ? inspected.reading : undefined;
};
/** Interpret recorded starts once, preserving exact ISO facts and legacy native-Date admission. */
export const readRecordedStart = (
value: string,
):
| {
readonly instant: OffsetDateTimeInstant;
readonly interpretation: "recorded-iso" | "native-date";
}
| undefined => {
const inspected = inspectRecordedDateTime(value);
if (inspected.kind === "valid")
return {
instant: inspected.reading.instant,
interpretation: "recorded-iso",
};
if (inspected.kind === "invalid") return undefined;
const milliseconds = Date.parse(value);
return Number.isFinite(milliseconds)
? {
instant: offsetDateTimeInstantFromEpochMilliseconds(milliseconds),
interpretation: "native-date",
}
: undefined;
};
/** Convert a Date.now-style integer into the same exact comparison shape. */
export const offsetDateTimeInstantFromEpochMilliseconds = (
epochMilliseconds: number,
+37 -2
View File
@@ -979,7 +979,7 @@ describe("Ghidra installation version admission", () => {
},
{
name: "different Ghidra line",
readText: () => "application.version=12.2.0\napplication.java.min=21\n",
readText: () => "application.version=13.0.0\napplication.java.min=21\n",
probeJava: () => javaProbeOk<GhidraJavaObservation>(JAVA),
failed: "version",
code: "unsupported_version",
@@ -1024,7 +1024,7 @@ describe("Ghidra installation version admission", () => {
);
expect(result.status).toBe("unavailable");
if (result.status !== "unavailable") return;
expect(result.rejection.reason).toContain("12.1.x");
expect(result.rejection.reason).toContain("12.x (>=12.1)");
expect(result.rejection.reason).toContain(SUPPORTED_GHIDRA_VERSION);
expect(result.checks.find(({ name }) => name === "java")).toMatchObject({
status: "failed",
@@ -1055,6 +1055,41 @@ describe("Ghidra installation version admission", () => {
),
).toMatchObject({ status: "available", javaVersion: "17.0.20.1" });
});
it.each([21, 25])(
"admits Ghidra 12.2 independently of its declared Java 25 minimum (JDK %s)",
async (major) => {
const result = await inspectGhidraInstallation(
{
resourceOwner: inspectionResources.owner,
environment: {},
installDir: INSTALL,
platform: "linux",
architecture: "x64",
},
host({
readText: () => "application.version=12.2\napplication.java.min=25\n",
probeJava: () =>
javaProbeOk<GhidraJavaObservation>({
...JAVA,
major,
version: `${String(major)}.0.5`,
home: `/usr/lib/jvm/jdk-${String(major)}`,
}),
}),
);
expect(result.providerVersion).toBe("12.2");
expect(
result.checks.find(({ name }) => name === "version"),
).toMatchObject({
status: "passed",
});
expect(result.checks.find(({ name }) => name === "java")).toMatchObject({
status: major === 25 ? "passed" : "failed",
});
expect(result.status).toBe(major === 25 ? "available" : "unavailable");
},
);
});
describe("Ghidra configuration paths", () => {
+80
View File
@@ -0,0 +1,80 @@
import { describe, expect, it } from "vitest";
import {
SUPPORTED_GHIDRA_VERSION,
ghidraReleaseAccepted,
ghidraReleaseCompatibility,
ghidraReleaseLine,
ghidraVersionDetail,
unverifiedGhidraBuildLimitation,
} from "./GhidraReleasePolicy.js";
describe("ghidraReleaseCompatibility", () => {
it("marks the verified build", () => {
const compatibility = ghidraReleaseCompatibility("12.1.4");
expect(compatibility.status).toBe("verified");
if (compatibility.status !== "verified") return;
expect(compatibility.raw).toBe("12.1.4");
});
it("marks the rest of the verified line compatible", () => {
expect(ghidraReleaseCompatibility("12.1.2").status).toBe("compatible");
});
it("admits a newer minor release as compatible", () => {
const compatibility = ghidraReleaseCompatibility("12.2");
expect(compatibility.status).toBe("compatible");
if (compatibility.status !== "compatible") return;
expect(compatibility.raw).toBe("12.2");
});
it("rejects other majors and older minors", () => {
expect(ghidraReleaseCompatibility("12.0.4").status).toBe("unsupported");
expect(ghidraReleaseCompatibility("11.4").status).toBe("unsupported");
expect(ghidraReleaseCompatibility("13.0").status).toBe("unsupported");
});
it("leaves unparseable releases unresolved", () => {
expect(ghidraReleaseCompatibility("latest").status).toBe("unresolved");
expect(ghidraReleaseCompatibility(null).status).toBe("unresolved");
});
});
describe("ghidraReleaseAccepted", () => {
it("accepts the verified build and newer compatible releases", () => {
expect(ghidraReleaseAccepted("12.1.4")).toBe(true);
expect(ghidraReleaseAccepted("12.2")).toBe(true);
});
it("rejects unsupported releases", () => {
expect(ghidraReleaseAccepted("13.0")).toBe(false);
expect(ghidraReleaseAccepted("latest")).toBe(false);
});
});
describe("unverifiedGhidraBuildLimitation", () => {
it("reports a newer compatible build against the verified build", () => {
const limitation = unverifiedGhidraBuildLimitation("12.2");
expect(limitation).toBeDefined();
expect(limitation).toContain("12.2");
expect(limitation).toContain(SUPPORTED_GHIDRA_VERSION);
});
it("stays silent for the verified build and unsupported releases", () => {
expect(unverifiedGhidraBuildLimitation("12.1.4")).toBeUndefined();
expect(unverifiedGhidraBuildLimitation("13.0")).toBeUndefined();
});
});
describe("ghidraReleaseLine", () => {
it("names the accepted release floor", () => {
expect(ghidraReleaseLine()).toBe("12.x (>=12.1)");
});
});
describe("ghidraVersionDetail", () => {
it("describes a newer compatible release against the verified build", () => {
const detail = ghidraVersionDetail("12.2");
expect(detail).toContain("compatible");
expect(detail).toContain(SUPPORTED_GHIDRA_VERSION);
});
});
+7 -3
View File
@@ -43,7 +43,11 @@ export const ghidraReleaseCompatibility = (
if (raw === SUPPORTED_GHIDRA_VERSION)
return { status: "verified", version, raw };
const verified = verifiedGhidraRelease();
return version.major === verified.major && version.minor === verified.minor
// Admit the verified line and anything newer on this major: the observed
// version is reported on the provider identity and sessions run with the
// unverified-build limitation, so a newer release like 12.2 is usable
// while the verified build stays pinned.
return version.major === verified.major && version.minor >= verified.minor
? { status: "compatible", version, raw }
: { status: "unsupported", version, raw };
};
@@ -60,10 +64,10 @@ export const unverifiedGhidraBuildLimitation = (
return `This session uses Ghidra ${providerVersion}. The bridge is verified against ${SUPPORTED_GHIDRA_VERSION}; other ${ghidraReleaseLine()} builds are accepted and the observed version is reported.`;
};
/** Release line accepted by this bridge, such as `12.1.x`. */
/** Accepted release floor on this major, such as `12.x (>=12.1)`. */
export const ghidraReleaseLine = (): string => {
const verified = verifiedGhidraRelease();
return `${String(verified.major)}.${String(verified.minor)}.x`;
return `${String(verified.major)}.x (>=${String(verified.major)}.${String(verified.minor)})`;
};
/** Detail for the installation version check, including the accepted line. */
+191 -55
View File
@@ -1,4 +1,9 @@
import type { RegularFileState } from "../filesystem/RegularFile.js";
import {
regularFileState,
sameRegularFileState,
type RegularFileState,
} from "../filesystem/RegularFile.js";
import type { Stats } from "node:fs";
import {
assertKunaProjectState,
type KunaProjectFileStates,
@@ -6,16 +11,16 @@ import {
import { createHash, randomUUID } from "node:crypto";
import {
mkdir,
lstat,
mkdtemp,
readdir,
realpath,
readFile,
rename,
rm,
stat,
utimes,
} from "node:fs/promises";
import { join } from "node:path";
import { basename, join } from "node:path";
import canonicalize from "canonicalize";
import writeFileAtomic from "write-file-atomic";
@@ -30,6 +35,7 @@ import {
AnalysisResourceConstraintError,
} from "../domain/analysisErrorCore.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { err, ok, type Result } from "../domain/result.js";
import {
observeStableFile,
@@ -43,6 +49,7 @@ import {
import type { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
import type { KunaInstallation } from "./KunaInstallation.js";
import {
kunaExportFileNames,
exportKunaProject,
KUNA_EXPORT_OPERATION,
kunaExportFailure,
@@ -59,7 +66,7 @@ import type {
import { KUNA_PROVIDER_ID } from "./KunaRelease.js";
/** Bumped whenever the entry layout or manifest meaning changes. */
const CACHE_LAYOUT = 2;
const CACHE_LAYOUT = 3;
/** A manifest names five files and their digests; anything larger is not ours. */
const MANIFEST_MAX_BYTES = 64 * 1024;
@@ -71,6 +78,7 @@ const SHA256_PATTERN = /^[0-9a-f]{64}$/u;
/** Inputs that determine an export's content. Each one is part of the cache key. */
export interface KunaProjectKeyInputs {
readonly targetSha256: string;
readonly targetBasename: string;
readonly kunaSha256: string;
readonly kunaCommand: string;
readonly kunaExecutable: string;
@@ -89,6 +97,7 @@ export const kunaProjectKey = (inputs: KunaProjectKeyInputs): string =>
canonicalize({
layout: CACHE_LAYOUT,
target_sha256: inputs.targetSha256,
target_basename: inputs.targetBasename,
kuna_sha256: inputs.kunaSha256,
kuna_command: inputs.kunaCommand,
kuna_executable: inputs.kunaExecutable,
@@ -104,30 +113,74 @@ const fileSchema = z.strictObject({
sha256: z.string().regex(SHA256_PATTERN),
});
const manifestSchema = z.strictObject({
layout: z.literal(CACHE_LAYOUT),
key: z.string().regex(KEY_PATTERN),
inputs: z.strictObject({
target_sha256: z.string().regex(SHA256_PATTERN),
kuna_sha256: z.string().regex(SHA256_PATTERN),
kuna_command: z.string(),
kuna_executable: z.string(),
kuna_version: z.string().min(1),
jobs: z.int().positive(),
}),
files: z.strictObject({
c: fileSchema,
h: fileSchema,
asm: fileSchema,
readme: fileSchema,
inventory: fileSchema,
}),
summary: z.strictObject({
decompiled: z.int().nonnegative(),
failed: z.int().nonnegative(),
}),
created_at: utcDateTimeSchema,
});
const manifestSchema = z
.strictObject({
layout: z.literal(CACHE_LAYOUT),
key: z.string().regex(KEY_PATTERN),
inputs: z.strictObject({
target_sha256: z.string().regex(SHA256_PATTERN),
target_basename: z.string().min(1),
kuna_sha256: z.string().regex(SHA256_PATTERN),
kuna_command: z.string(),
kuna_executable: z.string(),
kuna_version: z.string().min(1),
jobs: z.int().positive(),
}),
files: z.strictObject({
c: fileSchema,
h: fileSchema,
asm: fileSchema,
readme: fileSchema,
inventory: fileSchema,
}),
summary: z.strictObject({
decompiled: z.int().nonnegative(),
failed: z.int().nonnegative(),
}),
created_at: utcDateTimeSchema,
})
.superRefine(({ files, inputs, key }, context) => {
if (
key !==
kunaProjectKey({
targetSha256: inputs.target_sha256,
targetBasename: inputs.target_basename,
kunaSha256: inputs.kuna_sha256,
kunaCommand: inputs.kuna_command,
kunaExecutable: inputs.kuna_executable,
kunaVersion: inputs.kuna_version,
jobs: inputs.jobs,
})
)
context.addIssue({
code: "custom",
path: ["key"],
message: "Export cache key does not match its producing inputs",
});
const stem = inputs.target_basename;
if (
stem.length === 0 ||
stem === "." ||
stem === ".." ||
stem.includes("\0") ||
basename(stem) !== stem
) {
context.addIssue({
code: "custom",
path: ["files", "c", "path"],
message: "Export source must use the target's native basename",
});
return;
}
const expected = kunaExportFileNames(stem);
for (const role of Object.keys(expected) as (keyof typeof expected)[])
if (files[role].path !== expected[role])
context.addIssue({
code: "custom",
path: ["files", role, "path"],
message: "Export file does not match its producer role and basename",
});
});
/** Immutable description of one published export, written once at publish. */
export type KunaProjectManifest = z.infer<typeof manifestSchema>;
@@ -287,7 +340,13 @@ export class KunaProjectCache {
readonly #inflight = new Map<string, InflightBuild>();
/** Every build not yet settled, including abandoned ones a retry replaced. */
readonly #unfinished = new Set<InflightBuild>();
readonly #verified = new Map<string, KunaProject>();
readonly #verified = new Map<
string,
{
readonly project: KunaProject;
readonly manifestState: RegularFileState;
}
>();
readonly #pins = new Map<string, number>();
/** One eviction decision at a time; exports themselves remain concurrent. */
#eviction: Promise<void> = Promise.resolve();
@@ -364,6 +423,11 @@ export class KunaProjectCache {
key: string,
signal?: AbortSignal,
): Promise<KunaProject | undefined> {
if (!KEY_PATTERN.test(key))
throw new ProviderAdapterError(KUNA_PROVIDER_ID, KUNA_EXPORT_OPERATION, {
userMessage:
"Kuna cache lookup requires a content-addressed project key",
});
await this.#retryHashFiles(key);
const cleanup = this.#cleanupResult(this.#hashCleanupFailures(key));
if (!cleanup.ok) throw cleanup.error;
@@ -374,13 +438,27 @@ export class KunaProjectCache {
if (isMissing(cause)) return undefined;
throw cause;
}
const verified = this.#verified.get(key);
const projects = await this.#admitProjects(root);
if (projects === undefined) return undefined;
const admitted = await this.#admitEntry(projects, key);
if (admitted === undefined) {
await this.#discard(key);
return undefined;
}
let verified = this.#verified.get(key);
if (
verified !== undefined &&
!sameRegularFileState(verified.manifestState, admitted.manifest)
) {
this.#verified.delete(key);
verified = undefined;
}
if (verified !== undefined) {
try {
await assertKunaProjectState(
verified.directory,
verified.manifest.files,
verified.fileStates,
verified.project.directory,
verified.project.manifest.files,
verified.project.fileStates,
signal,
);
} catch (cause: unknown) {
@@ -388,19 +466,17 @@ export class KunaProjectCache {
await this.#discard(key);
return undefined;
}
if (await this.#touch(key)) return verified;
if (await this.#touch(key)) return verified.project;
this.#verified.delete(key);
return undefined;
}
const entry = join(root, "projects", key);
let text: string;
try {
text = await this.#readBounded(join(entry, "manifest.json"));
text = await readFile(join(entry, "manifest.json"), "utf8");
} catch (cause: unknown) {
if (isMissing(cause)) return undefined;
if (!(cause instanceof RangeError)) throw cause;
await this.#discard(key);
return undefined;
throw cause;
}
const manifest = parseManifest(text);
if (manifest?.key !== key) {
@@ -464,7 +540,10 @@ export class KunaProjectCache {
...parsed.value,
};
if (!(await this.#touch(key))) return undefined;
this.#verified.set(key, project);
this.#verified.set(key, {
project,
manifestState: regularFileState(admitted.manifest),
});
return project;
}
@@ -480,6 +559,7 @@ export class KunaProjectCache {
return err(new AnalysisCancelledError(KUNA_EXPORT_OPERATION));
const key = kunaProjectKey({
targetSha256: request.targetSha256,
targetBasename: basename(request.targetPath),
kunaSha256: request.installation.sha256,
kunaCommand: request.installation.configuration.command,
kunaExecutable: request.installation.executable,
@@ -952,6 +1032,7 @@ export class KunaProjectCache {
key,
inputs: {
target_sha256: request.targetSha256,
target_basename: basename(request.targetPath),
kuna_sha256: request.installation.sha256,
kuna_command: request.installation.configuration.command,
kuna_executable: request.installation.executable,
@@ -981,6 +1062,9 @@ export class KunaProjectCache {
`${JSON.stringify(manifest, null, 2)}\n`,
{ mode: 0o600 },
);
const manifestState = regularFileState(
await lstat(join(staging, "entry", "manifest.json")),
);
const destination = join(await this.#resolveRoot(), "projects", key);
try {
await rename(join(staging, "entry"), destination);
@@ -998,7 +1082,7 @@ export class KunaProjectCache {
functions: exported.functions,
inventory: exported.inventory,
};
this.#verified.set(key, project);
this.#verified.set(key, { project, manifestState });
return ok(project);
}
@@ -1043,21 +1127,31 @@ export class KunaProjectCache {
async #evictUnpinned(keep: string | null): Promise<void> {
const root = await this.#resolveRoot();
const projects = await this.#admitProjects(root);
if (projects === undefined) {
this.#capacity = "within_limit";
return;
}
// A successful build must reach its callers before they can pin its entry.
// Protect every unfinished build while reading this eviction snapshot.
const building = new Set([...this.#unfinished].map(({ key }) => key));
const entries: { key: string; bytes: number; mtimeMs: number }[] = [];
for (const name of await readdir(join(root, "projects"))) {
for (const name of await readdir(projects)) {
if (!KEY_PATTERN.test(name)) continue;
const path = join(root, "projects", name);
const status = await stat(path).catch(() => undefined);
if (status === undefined) continue;
const manifest = await this.#readBounded(join(path, "manifest.json"))
.then(parseManifest)
.catch(() => undefined);
const status = await this.#admitEntry(projects, name);
const manifest =
status === undefined
? undefined
: await readFile(join(path, "manifest.json"), "utf8")
.then(parseManifest)
.catch((cause: unknown) => {
if (isMissing(cause)) return undefined;
throw cause;
});
// An owned entry without a valid manifest is unusable; evict it first.
entries.push(
manifest === undefined
manifest === undefined || status === undefined
? { key: name, bytes: 0, mtimeMs: -1 }
: {
key: name,
@@ -1090,10 +1184,12 @@ export class KunaProjectCache {
async #discard(key: string): Promise<void> {
this.#verified.delete(key);
const root = await this.#resolveRoot();
const projects = await this.#admitProjects(root);
if (projects === undefined) return;
const trash = join(root, "trash", `${key}-${randomUUID()}`);
try {
await mkdir(join(root, "trash"), { recursive: true, mode: 0o700 });
await rename(join(root, "projects", key), trash);
await rename(join(projects, key), trash);
} catch (cause: unknown) {
if (isMissing(cause)) return;
throw cause;
@@ -1126,13 +1222,53 @@ export class KunaProjectCache {
}
}
async #readBounded(path: string): Promise<string> {
const status = await stat(path);
if (status.size > MANIFEST_MAX_BYTES)
throw new RangeError(
`${path} exceeds ${String(MANIFEST_MAX_BYTES)} bytes`,
);
return readFile(path, "utf8");
/** A replaced projects parent does not confer ownership of its children. */
async #admitProjects(root: string): Promise<string | undefined> {
const projects = join(root, "projects");
const parent = await lstat(projects).catch((cause: unknown) => {
if (isMissing(cause)) return undefined;
throw cause;
});
if (parent === undefined) return undefined;
if (!parent.isDirectory() || parent.isSymbolicLink())
throw new ProviderAdapterError(KUNA_PROVIDER_ID, KUNA_EXPORT_OPERATION, {
userMessage: `Kuna cache project ownership is unknown: ${projects} is not a regular directory`,
});
return projects;
}
/** Admit the owned entry layout before reading any manifest or project bytes. */
async #admitEntry(
projects: string,
key: string,
): Promise<
| {
readonly mtimeMs: number;
readonly manifest: Stats;
}
| undefined
> {
const entry = join(projects, key);
try {
const status = await lstat(entry);
if (!status.isDirectory() || status.isSymbolicLink()) return undefined;
const project = await lstat(join(entry, "project"));
if (!project.isDirectory() || project.isSymbolicLink()) return undefined;
const manifest = await lstat(join(entry, "manifest.json"));
if (
!manifest.isFile() ||
manifest.isSymbolicLink() ||
manifest.size > MANIFEST_MAX_BYTES
)
return undefined;
return {
mtimeMs: status.mtimeMs,
manifest,
};
} catch (cause: unknown) {
if (isMissing(cause)) return undefined;
throw cause;
}
}
#retainFileCleanup(
+13 -8
View File
@@ -54,6 +54,18 @@ export const KUNA_EXPORT_OPERATION = "kuna:decompile-project";
const INVENTORY_FILE = "functions.json";
/** Exact flat file layout emitted for the selected target's native basename. */
export const kunaExportFileNames = (targetPath: string) => {
const stem = basename(targetPath);
return {
c: `${stem}.c`,
h: `${stem}.h`,
asm: `${stem}.asm`,
readme: "README.md",
inventory: INVENTORY_FILE,
} as const;
};
/** Identity of one file an export consists of. */
export interface KunaExportFile {
readonly path: string;
@@ -292,14 +304,7 @@ export const exportKunaProject = async (
return unusable(
"Kuna did not print its `functions: N ok, M failed` summary.",
);
const stem = basename(request.targetPath);
const names = {
c: `${stem}.c`,
h: `${stem}.h`,
asm: `${stem}.asm`,
readme: "README.md",
inventory: INVENTORY_FILE,
} as const;
const names = kunaExportFileNames(request.targetPath);
const files: Partial<Record<keyof KunaExportFiles, KunaExportFile>> = {};
const fileStates: Partial<
Record<keyof KunaProjectFileStates, RegularFileState>
+2
View File
@@ -1,3 +1,4 @@
import { basename } from "node:path";
import {
regularFileCancellation,
RegularFileCleanupFailure,
@@ -1561,6 +1562,7 @@ export class KunaSessionClient implements AnalysisClient {
async #lookupOrBuild(installation: KunaInstallation): Promise<void> {
const key = kunaProjectKey({
targetSha256: this.#target.sha256,
targetBasename: basename(this.#target.path),
kunaSha256: installation.sha256,
kunaCommand: installation.configuration.command,
kunaExecutable: installation.executable,
@@ -0,0 +1,143 @@
import { createHash } from "node:crypto";
import { mkdir, readFile, realpath, writeFile } from "node:fs/promises";
import { join, sep } from "node:path";
import { expect } from "vitest";
import {
NativeDmgArtifactReader,
DMG_EXISTING_ATTACHMENT_RECOVERY,
} from "../../src/artifacts/NativeDmgArtifactReader.js";
import { parseNativeDmgInfo } from "../../src/artifacts/NativeDmgOwnership.js";
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
import { cliTest } from "../support/cli/cliFixture.js";
cliTest.skipIf(process.platform !== "darwin")(
"DMG recovery preserves the caller's real attachment and succeeds after its owner releases it",
async ({ cli, processes }) => {
const root = await createTestTempDirectory("rea-dmg-attachment-recovery-");
const source = join(root, "source");
await mkdir(source);
await writeFile(join(source, "fixture.txt"), "caller-owned image\n");
const image = join(root, "fixture.dmg");
const created = await processes.run(
"/usr/bin/hdiutil",
[
"create",
"-quiet",
"-srcfolder",
source,
"-volname",
"ReaRecoveryFixture",
"-size",
"20m",
"-format",
"UDZO",
image,
],
{ timeoutMs: 30_000 },
);
expect(created.exitCode, created.stderr).toBe(0);
const digest = createHash("sha256")
.update(await readFile(image))
.digest("hex");
const resolvedImage = await realpath(image);
const owner = new NativeDmgArtifactReader(image, process.env);
let mountRoot: string | undefined;
const attachedImages = async () => {
const info = await processes.run("/usr/bin/hdiutil", ["info", "-plist"], {
timeoutMs: 10_000,
});
expect(info.exitCode, info.stderr).toBe(0);
return parseNativeDmgInfo(info.stdout).images;
};
const attachedImagePaths = async () =>
Promise.all(
(await attachedImages()).map((entry) =>
entry["image-path"] === undefined
? undefined
: realpath(entry["image-path"]).catch(() => undefined),
),
);
const mountedDevices = async (): Promise<string[]> =>
(await attachedImages())
.flatMap((entry) => entry["system-entities"])
.filter(
(entry) =>
mountRoot !== undefined &&
entry["mount-point"]?.startsWith(`${mountRoot}${sep}`),
)
.map((entry) => entry["dev-entry"])
.sort();
try {
const entries: string[] = [];
for await (const entry of owner.entries()) entries.push(entry.path);
expect(entries.some((path) => path.endsWith("/fixture.txt"))).toBe(true);
const attach = owner
.provenance()
.find((command) => command.arguments[0] === "attach");
if (attach === undefined)
throw new Error("Fixture attachment was not recorded");
mountRoot = attach.arguments[attach.arguments.indexOf("-mountroot") + 1];
expect(mountRoot).toBeDefined();
const before = await mountedDevices();
expect(before.length).toBeGreaterThan(0);
expect(await attachedImagePaths()).toContain(resolvedImage);
const busy = await cli.run({
arguments: ["inspect-artifact", image, "--json"],
timeoutMs: 30_000,
});
expect(busy.exitCode, busy.stdout).toBe(1);
expect(busy.json).toMatchObject({
code: "artifact_operation_failed",
category: "unavailable",
message: DMG_EXISTING_ATTACHMENT_RECOVERY,
remediation: {
action: DMG_EXISTING_ATTACHMENT_RECOVERY,
},
details: {
reason: "unavailable",
partial_observation: {
kind: "artifact-inventory",
inventory: {
manifest: { root_sha256: digest, root_format: "dmg" },
coverage: { status: "unknown" },
occurrences: [{ logical_path: "." }],
provenance: expect.arrayContaining([
expect.objectContaining({
arguments: expect.arrayContaining(["attach", image]),
exit_code: 1,
}),
]),
},
},
},
});
expect(await mountedDevices()).toEqual(before);
} finally {
await owner.close();
}
expect(await mountedDevices()).toEqual([]);
const control = await cli.run({
arguments: ["inspect-artifact", image, "--json"],
timeoutMs: 30_000,
});
expect(control.exitCode, control.stdout).toBe(0);
expect(control.json).toMatchObject({
normalized_result: {
substeps: [
{
operation: "inventory_artifact",
evidence: {
normalized_result: {
manifest: { root_sha256: digest },
coverage: { scope: "container-entries", status: "complete" },
},
},
},
],
},
});
expect(await attachedImagePaths()).not.toContain(resolvedImage);
},
60_000,
);
@@ -730,6 +730,22 @@ it.each([
"src/native/AppleExample.ts",
{ code: true, runtime: true, apple: true, package: false, evm: false },
],
[
"src/artifacts/NativeDmgArtifactReader.ts",
{ code: true, runtime: true, apple: true, package: false, evm: false },
],
[
"tests/acceptance/artifactDmgAttachmentRecoveryCli.test.ts",
{ code: true, apple: true, package: false, runtime: false },
],
[
"tests/process-global/dmgCancellationReason.test.ts",
{ code: true, apple: true, package: false, runtime: false },
],
[
"tests/boundary/providers/native/nativeDmgArtifactReader.test.ts",
{ code: true, apple: true, package: false, runtime: false },
],
[
"scripts/verify/apple/example.mjs",
{ code: true, apple: true, package: false, runtime: false },
@@ -1,6 +1,8 @@
import { writeFile, stat } from "node:fs/promises";
import { writeFile, stat, open } from "node:fs/promises";
import { createHash } from "node:crypto";
import { join } from "node:path";
import { expect, it } from "vitest";
import { expect, it, vi } from "vitest";
import { GodotPckArtifactReader } from "../../../src/artifacts/GodotPckArtifactReader.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { buildGodotPck, pckU64 } from "../../fixtures/godotPck.js";
import {
@@ -10,6 +12,83 @@ import {
import { parseBinaryTarget } from "../../../src/application/BinaryTargetResolver.js";
import { elf } from "../../../src/domain/binaryTarget.fixture.js";
it.each(
[true, false].flatMap((matchingDigest) =>
["stable", "changed", "unavailable", "read failure"].map((sourceState) => ({
matchingDigest,
sourceState,
})),
),
)(
"preserves verified carrier binding through final cancellation (matching=$matchingDigest, source=$sourceState)",
async ({ matchingDigest, sourceState }) => {
const root = await createTestTempDirectory("rea-pck-binding-finalization-");
const path = join(root, "game.pck");
const bytes = buildGodotPck(4, [
{ path: "res://main.gd", data: "extends Node\n" },
]);
await writeFile(path, bytes);
const handle = await open(path);
const initial = await handle.stat();
const observedSha256 = createHash("sha256").update(bytes).digest("hex");
const expectedSha256 = matchingDigest ? observedSha256 : "0".repeat(64);
const controller = new AbortController();
const reason = { stage: "carrier final source validation" };
const physical = Object.assign(
new Error("carrier source I/O unavailable"),
{ code: "EIO" },
);
const nativeStat = handle.stat.bind(handle);
let validations = 0;
vi.spyOn(handle, "stat").mockImplementation(async () => {
const observed = await nativeStat();
if (++validations === 2) {
controller.abort(reason);
if (sourceState === "unavailable") throw physical;
if (sourceState === "changed") observed.ino += 1;
}
return observed;
});
if (sourceState === "read failure")
vi.spyOn(handle, "read").mockImplementationOnce(async () => {
controller.abort(reason);
throw physical;
});
const reader = new GodotPckArtifactReader(path, { handle, initial });
try {
const failure: unknown = await reader
.prepareContainer(expectedSha256, controller.signal)
.catch((cause: unknown) => cause);
expect(controller.signal.reason).toBe(reason);
if (sourceState === "unavailable" || sourceState === "read failure")
expect(failure).toBe(physical);
else if (sourceState === "changed")
expect(failure).toMatchObject({
reason: "integrity",
details: undefined,
message: expect.stringContaining("source identity changed"),
});
else if (matchingDigest)
expect(failure).toMatchObject({ reason: "cancelled", cause: reason });
else
expect(failure).toMatchObject({
reason: "integrity",
details: {
declaredSha256: expectedSha256,
calculatedSha256: observedSha256,
},
});
await reader.close();
// The admitted descriptor is borrowed; only its original caller closes it.
expect(handle.fd).toBeGreaterThanOrEqual(0);
} finally {
await reader.close();
await handle.close();
vi.restoreAllMocks();
}
},
);
it("binds extraction to the original carrier even if a changed directory names identical payloads", async () => {
const root = await createTestTempDirectory("rea-pck-extraction-identity-");
const source = join(root, "game.pck");
+153 -1
View File
@@ -22,7 +22,10 @@ import { BinaryLayoutService } from "../../../src/application/binaryDiagnostics/
import { toolContract } from "../../../src/contracts/toolContracts.js";
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
import { createAnalysisProfile } from "../../../src/domain/analysisProfile.js";
import { jsonObjectSchema } from "../../../src/domain/jsonValue.js";
import {
jsonObjectSchema,
type JsonValue,
} from "../../../src/domain/jsonValue.js";
import { functionDossierSchema } from "../../../src/domain/hopperValues.js";
import { ghidraFunctionDossier } from "../../../src/domain/ghidraValues.fixture.js";
import { ok } from "../../../src/domain/result.js";
@@ -44,7 +47,9 @@ import {
analysisViewNativeEvidence,
environmentManifestCaptureFixture,
environmentStateCaptureFixture,
timedCaptureEvidence,
} from "../../fixtures/analysisView.js";
import { BOOK_ROWS } from "../../fixtures/environmentStateWrites.js";
import { createTestBinarySession } from "../../fixtures/binarySession.js";
import { createLinkedIl2cppMetadataFixture } from "../../fixtures/il2cpp/metadata.js";
import {
@@ -1390,6 +1395,153 @@ it.each(PUBLISHED_EXAMPLES)(
},
);
it.each([false, true])(
"keeps known chronology and unrelated facts with unknown delete completion %s through MCP",
async (unknownCompletion) => {
const { client, session } = await connect();
const entry = (
method: string,
path: string,
second: string,
body: JsonValue,
status = 200,
) => ({
startedDateTime: `2026-10-20T09:00:${second}Z`,
time: 0,
request: { method, url: `http://app.example${path}` },
response: {
status,
content: { mimeType: "application/json", text: JSON.stringify(body) },
},
});
const { time: _readDuration, ...laterRead } = entry(
"GET",
"/api/settings",
"02",
{ theme: "dark" },
);
const deletion = entry("DELETE", "/api/items/2", "03", "", 204);
const { time: _deleteDuration, ...unknownDeletion } = deletion;
const capture = environmentManifestCaptureFixture([
entry("GET", "/api/settings", "00", { theme: "light" }),
laterRead,
entry("PATCH", "/api/settings", "01", { theme: "dark" }),
entry("GET", "/api/health", "00.1", { healthy: true }),
entry("GET", "/api/version", "00.2", { version: "1" }),
entry("GET", "/api/items", "00.3", [{ id: 1 }, { id: 2 }]),
entry("GET", "/api/items/2", "00.4", { id: 2 }),
unknownCompletion ? unknownDeletion : deletion,
entry("GET", "/api/items/2", "04", null, 404),
]);
const parent = analysisViewCaptureEvidence(capture);
const before = structuredClone(parent);
expect(session.recordEvidence(parent).ok).toBe(true);
const response = await client.callTool({
name: "inspect_analysis_view",
arguments: {
source: { kind: "retained-evidence", evidence_id: parent.evidence_id },
view: { kind: "environment-state" },
},
});
expect(response.isError, JSON.stringify(response.content)).not.toBe(true);
const state = analysisViewResultSchema.parse(
parseEvidence(response.structuredContent).normalized_result,
);
expect(state).toMatchObject({
kind: "environment-state",
timeline: "recorded-start-time",
documents: expect.arrayContaining([
expect.objectContaining({
path: "/api/settings",
value: { theme: "light" },
}),
expect.objectContaining({
path: "/api/health",
value: { healthy: true },
}),
expect.objectContaining({
path: "/api/version",
value: { version: "1" },
}),
]),
writes: expect.arrayContaining([
expect.objectContaining({
tool: "delete_items_by_id",
served: !unknownCompletion,
}),
]),
limitations: expect.arrayContaining([
"Some records carry no usable request duration; their completion and overlap remain unknown.",
]),
});
expect(parent).toEqual(before);
expect(session.evidenceById(parent.evidence_id)).toEqual(before);
},
);
it("advertises unserved learned clock rules when a retained capture has no global clock", async () => {
const { client, session } = await connect();
const { ajv, outputSchema } = await advertisedMcpToolSchemas(
client,
"inspect_analysis_view",
schemaValidation.assertions,
);
const parents = [
timedCaptureEvidence(BOOK_ROWS),
timedCaptureEvidence([
...BOOK_ROWS,
["GET", "/api/health", { ready: true }, 200, undefined, -1],
]),
];
const states = [];
for (const parent of parents) {
const original = structuredClone(parent);
expect(session.recordEvidence(parent).ok).toBe(true);
const response = await client.callTool({
name: "inspect_analysis_view",
arguments: {
source: { kind: "retained-evidence", evidence_id: parent.evidence_id },
view: { kind: "environment-state" },
},
});
expect(response.isError, JSON.stringify(response.content)).not.toBe(true);
expect(
ajv.validate(outputSchema, response.structuredContent),
JSON.stringify(ajv.errors),
).toBe(true);
const state = analysisViewResultSchema.parse(
parseEvidence(response.structuredContent).normalized_result,
);
if (state.kind !== "environment-state")
throw new Error("Expected environment-state result");
states.push(state);
expect(parent).toEqual(original);
expect(session.evidenceById(parent.evidence_id)).toEqual(original);
}
const [baseline, unknown] = states;
if (baseline === undefined || unknown === undefined)
throw new Error("Expected both clock controls");
expect(baseline.write_clock).not.toBeNull();
expect(unknown.write_clock).toBeNull();
const clockPlans = baseline.writes.filter(
(plan) =>
plan.served &&
plan.rules.some((rule) =>
["clock", "held-clock", "nested-clock"].includes(rule.rule),
),
);
expect(clockPlans.length).toBeGreaterThan(0);
for (const plan of clockPlans)
expect(
unknown.writes.find((candidate) => candidate.tool === plan.tool),
).toEqual({ ...plan, served: false, reason: "unknown-clock" });
const independent = baseline.writes.find(
(plan) => plan.tool === "delete_books_by_id",
);
expect(independent).toMatchObject({ served: true });
expect(unknown.writes).toContainEqual(independent);
});
it("extracts an environment state identically from retained, portable and file MCP Evidence", async () => {
const { client, session } = await connect();
const { ajv, inputSchema, outputSchema } = await advertisedMcpToolSchemas(
@@ -13,7 +13,7 @@ import {
utimes,
writeFile,
} from "node:fs/promises";
import { join } from "node:path";
import { basename, dirname, join, relative } from "node:path";
import canonicalize from "canonicalize";
import { expect, it as test, vi } from "vitest";
@@ -86,6 +86,7 @@ const newCache = (installation: KunaInstallation, maxBytes = 64 * 1024 ** 2) =>
const keyOf = (installation: KunaInstallation, target: Target) =>
kunaProjectKey({
targetSha256: target.targetSha256,
targetBasename: basename(target.targetPath),
kunaSha256: installation.sha256,
kunaCommand: installation.configuration.command,
kunaExecutable: installation.executable,
@@ -118,84 +119,103 @@ const deferred = <T>() => {
return { promise, resolve, reject };
};
it("regenerates an obsolete export once and retires it without touching unrelated cache resources", async () => {
const { kuna, installation, cacheRoot, cache, target } = await fixture(
"rea-kuna-cache-format-transition-",
);
const input = await target("program");
const initial = await cache.build({ installation, ...input });
if (!initial.ok) throw initial.error;
expect((await cache.close()).ok).toBe(true);
// Seed the previously shipped manifest and content-key format using a real
// authored export; omitted launcher identity must never be invented or reused.
const legacyKey = createHash("sha256")
.update(
canonicalize({
layout: 1,
target_sha256: input.targetSha256,
kuna_sha256: installation.sha256,
kuna_version: installation.version,
export: {
command: "decompile-project",
it.each([1, 2])(
"regenerates layout %i once and retires it without touching unrelated cache resources",
async (layout) => {
const { kuna, installation, cacheRoot, cache, target } = await fixture(
"rea-kuna-cache-format-transition-",
);
const input = await target("program");
const initial = await cache.build({ installation, ...input });
if (!initial.ok) throw initial.error;
expect((await cache.close()).ok).toBe(true);
// Seed the previously shipped manifest and content-key format using a real
// authored export; omitted launcher identity must never be invented or reused.
const legacyKey = createHash("sha256")
.update(
canonicalize({
layout,
target_sha256: input.targetSha256,
kuna_sha256: installation.sha256,
...(layout === 2
? {
kuna_command: installation.configuration.command,
kuna_executable: installation.executable,
}
: {}),
kuna_version: installation.version,
export: {
command: "decompile-project",
jobs: installation.configuration.jobs,
},
}) ?? "",
)
.digest("hex");
const legacyEntry = join(cacheRoot, "projects", legacyKey);
await rename(join(cacheRoot, "projects", initial.value.key), legacyEntry);
await writeFile(
join(legacyEntry, "manifest.json"),
JSON.stringify({
...initial.value.manifest,
layout,
key: legacyKey,
inputs: {
target_sha256: input.targetSha256,
kuna_sha256: installation.sha256,
...(layout === 2
? {
kuna_command: installation.configuration.command,
kuna_executable: installation.executable,
}
: {}),
kuna_version: installation.version,
jobs: installation.configuration.jobs,
},
}) ?? "",
)
.digest("hex");
const legacyEntry = join(cacheRoot, "projects", legacyKey);
await rename(join(cacheRoot, "projects", initial.value.key), legacyEntry);
await writeFile(
join(legacyEntry, "manifest.json"),
JSON.stringify({
...initial.value.manifest,
layout: 1,
key: legacyKey,
inputs: {
target_sha256: input.targetSha256,
kuna_sha256: installation.sha256,
kuna_version: installation.version,
jobs: installation.configuration.jobs,
},
}),
);
const unknown = join(cacheRoot, "projects", "unrelated-directory");
const unresolved = join(cacheRoot, "staging", "build-unresolved");
await mkdir(unknown);
await mkdir(unresolved);
await writeFile(join(unknown, "sentinel"), "preserve");
await writeFile(join(unresolved, "sentinel"), "preserve");
const migrated = newCache(installation);
try {
const built = await migrated.build({ installation, ...input });
if (!built.ok) throw built.error;
await expect(access(legacyEntry)).rejects.toMatchObject({ code: "ENOENT" });
expect(built.value.key).not.toBe(legacyKey);
expect(built.value.manifest).toMatchObject({
layout: 2,
inputs: {
kuna_command: installation.configuration.command,
kuna_executable: installation.executable,
},
});
expect(await callsOf(kuna, "decompile-project")).toHaveLength(2);
const fresh = newCache(installation);
try {
const reused = await fresh.build({ installation, ...input });
if (!reused.ok) throw reused.error;
expect(reused.value.manifest).toEqual(built.value.manifest);
expect(await callsOf(kuna, "decompile-project")).toHaveLength(2);
} finally {
expect((await fresh.close()).ok).toBe(true);
}
expect(await readFile(join(unknown, "sentinel"), "utf8")).toBe("preserve");
expect(await readFile(join(unresolved, "sentinel"), "utf8")).toBe(
"preserve",
}),
);
expect(await entries(cacheRoot, "trash")).toEqual([]);
} finally {
expect((await migrated.close()).ok).toBe(true);
}
});
const unknown = join(cacheRoot, "projects", "unrelated-directory");
const unresolved = join(cacheRoot, "staging", "build-unresolved");
await mkdir(unknown);
await mkdir(unresolved);
await writeFile(join(unknown, "sentinel"), "preserve");
await writeFile(join(unresolved, "sentinel"), "preserve");
const migrated = newCache(installation);
try {
const built = await migrated.build({ installation, ...input });
if (!built.ok) throw built.error;
await expect(access(legacyEntry)).rejects.toMatchObject({
code: "ENOENT",
});
expect(built.value.key).not.toBe(legacyKey);
expect(built.value.manifest).toMatchObject({
layout: 3,
inputs: {
kuna_command: installation.configuration.command,
kuna_executable: installation.executable,
},
});
expect(await callsOf(kuna, "decompile-project")).toHaveLength(2);
const fresh = newCache(installation);
try {
const reused = await fresh.build({ installation, ...input });
if (!reused.ok) throw reused.error;
expect(reused.value.manifest).toEqual(built.value.manifest);
expect(await callsOf(kuna, "decompile-project")).toHaveLength(2);
} finally {
expect((await fresh.close()).ok).toBe(true);
}
expect(await readFile(join(unknown, "sentinel"), "utf8")).toBe(
"preserve",
);
expect(await readFile(join(unresolved, "sentinel"), "utf8")).toBe(
"preserve",
);
expect(await entries(cacheRoot, "trash")).toEqual([]);
} finally {
expect((await migrated.close()).ok).toBe(true);
}
},
);
const causesOf = (cause: unknown): unknown[] => {
const pending = [cause];
@@ -235,7 +255,7 @@ it("publishes a verified export and serves later requests from the cache", async
value: {
key: keyOf(installation, input),
manifest: {
layout: 2,
layout: 3,
inputs: {
target_sha256: input.targetSha256,
kuna_sha256: installation.sha256,
@@ -1248,3 +1268,246 @@ it("invalidates a warm export when same-length content changes", async () => {
expect(await cache.close()).toMatchObject({ ok: true });
}
});
it("refuses escaped manifest paths before reading export files and preserves the outside project", async () => {
const { installation, cacheRoot, cache, target } = await fixture(
"rea-kuna-escaped-manifest-",
);
const built = await cache.build({
installation,
...(await target("program")),
});
if (!built.ok) throw built.error;
expect((await cache.close()).ok).toBe(true);
const outside = join(dirname(cacheRoot), "saved-project");
await rename(built.value.directory, outside);
await mkdir(built.value.directory);
const manifest = structuredClone(built.value.manifest);
for (const file of Object.values(manifest.files))
file.path = relative(built.value.directory, join(outside, file.path));
await writeFile(
join(dirname(built.value.directory), "manifest.json"),
JSON.stringify(manifest),
);
const hashes = vi.spyOn(stableFileHash, "observeStableFile");
const reopened = newCache(installation);
try {
expect(await reopened.lookup(built.value.key)).toBeUndefined();
expect(hashes).not.toHaveBeenCalled();
await expect(access(dirname(built.value.directory))).rejects.toMatchObject({
code: "ENOENT",
});
expect(await readFile(join(outside, "program.c"), "utf8")).toContain(
"// Function: main",
);
} finally {
hashes.mockRestore();
expect((await reopened.close()).ok).toBe(true);
}
});
it.each(["h", "readme", "inventory"] as const)(
"refuses a manifest's mismatched %s role before hashing files",
async (role) => {
const { installation, cache, target } = await fixture(
"rea-kuna-role-manifest-",
);
const built = await cache.build({
installation,
...(await target("program")),
});
if (!built.ok) throw built.error;
expect((await cache.close()).ok).toBe(true);
const manifest = structuredClone(built.value.manifest);
manifest.files[role].path =
role === "h" ? "different.h" : `saved-${manifest.files[role].path}`;
await writeFile(
join(dirname(built.value.directory), "manifest.json"),
JSON.stringify(manifest),
);
const hashes = vi.spyOn(stableFileHash, "observeStableFile");
const reopened = newCache(installation);
try {
expect(await reopened.lookup(built.value.key)).toBeUndefined();
expect(hashes).not.toHaveBeenCalled();
} finally {
hashes.mockRestore();
expect((await reopened.close()).ok).toBe(true);
}
},
);
it.each(["entry", "project", "manifest"] as const)(
"refuses a symlinked %s on cold and verified lookup without touching its target",
async (kind) => {
for (const verified of [false, true]) {
const { installation, cacheRoot, cache, target } = await fixture(
"rea-kuna-linked-entry-",
);
const built = await cache.build({
installation,
...(await target("program")),
});
if (!built.ok) throw built.error;
expect((await cache.close()).ok).toBe(true);
const reopened = newCache(installation);
if (verified)
expect(await reopened.lookup(built.value.key)).toBeDefined();
const entry = dirname(built.value.directory);
const selected =
kind === "entry"
? entry
: kind === "project"
? built.value.directory
: join(entry, "manifest.json");
const outside = join(dirname(cacheRoot), `saved-${kind}`);
await rename(selected, outside);
await symlink(outside, selected);
const preserved =
kind === "entry"
? join(outside, "project", "program.c")
: kind === "project"
? join(outside, "program.c")
: outside;
const before = await readFile(preserved);
const hashes = vi.spyOn(stableFileHash, "observeStableFile");
try {
expect(await reopened.lookup(built.value.key)).toBeUndefined();
expect(hashes).not.toHaveBeenCalled();
expect((await readFile(preserved)).equals(before)).toBe(true);
await expect(access(entry)).rejects.toMatchObject({ code: "ENOENT" });
} finally {
hashes.mockRestore();
expect((await reopened.close()).ok).toBe(true);
}
}
},
);
it("keeps an empty symlinked projects parent outside its removal authority", async () => {
const { installation, cacheRoot, cache, target } = await fixture(
"rea-kuna-linked-projects-",
);
const built = await cache.build({
installation,
...(await target("program")),
});
if (!built.ok) throw built.error;
expect((await cache.close()).ok).toBe(true);
const outside = join(dirname(cacheRoot), "empty-projects");
await mkdir(outside);
await rename(join(cacheRoot, "projects"), join(cacheRoot, "saved-projects"));
await symlink(outside, join(cacheRoot, "projects"));
const reopened = newCache(installation);
try {
await expect(reopened.lookup(built.value.key)).rejects.toMatchObject({
_tag: "ProviderAdapterError",
userMessage: expect.stringContaining("ownership is unknown"),
});
expect(await readdir(outside)).toEqual([]);
await access(
join(cacheRoot, "saved-projects", built.value.key, "manifest.json"),
);
} finally {
expect((await reopened.close()).ok).toBe(true);
}
});
it("preserves a legitimate POSIX backslash in the common export basename", async () => {
const { installation, cache, target } = await fixture(
"rea-kuna-native-basename-",
);
const built = await cache.build({
installation,
...(await target("program\\name")),
});
if (!built.ok) throw built.error;
expect((await cache.close()).ok).toBe(true);
const reopened = newCache(installation);
try {
expect(await reopened.lookup(built.value.key)).toMatchObject({
manifest: {
files: {
c: { path: "program\\name.c" },
h: { path: "program\\name.h" },
asm: { path: "program\\name.asm" },
},
},
});
} finally {
expect((await reopened.close()).ok).toBe(true);
}
});
it.each(["invalid JSON", "changed producing inputs", "valid rewrite"] as const)(
"revalidates a warm cache after manifest %s",
async (change) => {
const { installation, cache, target } = await fixture(
"rea-kuna-warm-manifest-",
);
const built = await cache.build({
installation,
...(await target("program")),
});
if (!built.ok) throw built.error;
expect(await cache.lookup(built.value.key)).toBeDefined();
const manifest = structuredClone(built.value.manifest);
if (change === "changed producing inputs")
manifest.inputs.target_sha256 = "f".repeat(64);
await writeFile(
join(dirname(built.value.directory), "manifest.json"),
change === "invalid JSON"
? "{not JSON}"
: JSON.stringify(manifest, null, 3),
);
const hashes = vi.spyOn(stableFileHash, "observeStableFile");
try {
const found = await cache.lookup(built.value.key);
if (change === "valid rewrite") {
expect(found).toMatchObject({ key: built.value.key });
expect(hashes).toHaveBeenCalledTimes(5);
} else {
expect(found).toBeUndefined();
expect(hashes).not.toHaveBeenCalled();
}
} finally {
hashes.mockRestore();
expect((await cache.close()).ok).toBe(true);
}
},
);
it("separates equal target bytes whose native basenames change the export", async () => {
const { kuna, installation, cache, target } = await fixture(
"rea-kuna-basename-key-",
);
const first = await cache.build({
installation,
...(await target("first", "same bytes")),
});
const secondInput = await target("second", "same bytes");
const second = await cache.build({ installation, ...secondInput });
if (!first.ok) throw first.error;
if (!second.ok) throw second.error;
expect(second.value.key).not.toBe(first.value.key);
expect(first.value.manifest.inputs.target_basename).toBe("first");
expect(second.value.manifest.inputs.target_basename).toBe("second");
expect(first.value.manifest.files.c.path).toBe("first.c");
expect(second.value.manifest.files.c.path).toBe("second.c");
expect(first.value.manifest.files.c.sha256).not.toBe(
second.value.manifest.files.c.sha256,
);
expect(
await readFile(join(second.value.directory, "second.c"), "utf8"),
).toContain('#include "second.h"');
expect((await cache.close()).ok).toBe(true);
const reopened = newCache(installation);
try {
const reused = await reopened.build({ installation, ...secondInput });
if (!reused.ok) throw reused.error;
expect(reused.value.manifest).toEqual(second.value.manifest);
expect(await callsOf(kuna, "decompile-project")).toHaveLength(2);
} finally {
expect((await reopened.close()).ok).toBe(true);
}
});
@@ -0,0 +1,162 @@
import { BrowserObservationError } from "../../src/domain/browserObservationError.js";
import { PlaywrightScenarioSession } from "../../src/browser/PlaywrightScenarioSession.js";
import { chromium, type BrowserContext } from "playwright-core";
import { expect, it, vi } from "vitest";
import { captureBrowserScenario } from "../../src/application/BrowserScenarioCaptureService.js";
import { createBrowserScenarioProvider } from "../../src/composition/browserScenario.js";
import { browserScenarioSchema } from "../../src/domain/browserScenario.js";
import { projectAnalysisError } from "../../src/domain/analysisErrorProjection.js";
import {
AnalysisError,
primaryAnalysisError,
} from "../../src/domain/analysisErrorBase.js";
it.each([
{ late: false, secret: "declared-startup-private-value" },
{ late: false, secret: "browser_profile" },
{ late: false, secret: "startup private/value" },
{ late: true, secret: "declared-startup-private-value" },
{ late: true, secret: "browser_profile" },
])(
"excludes declared startup secrets across opening cleanup and retries: %j",
async ({ late, secret }) => {
const scenario = browserScenarioSchema.parse({
browser: { mode: "launch", executable_path: process.execPath },
start_url: { url: "https://app.test/" },
actions: [
{ step_id: "wait", action: "wait_for_timeout", duration_ms: 1 },
],
secrets: [
{ secret_id: "credential", environment_variable: "REA_STARTUP_SECRET" },
],
});
const diagnosticSecret = encodeURIComponent(secret).replace(
/%[0-9A-F]{2}/gu,
(escape) => escape.toLowerCase(),
);
let blocked = true;
let closeCalls = 0;
const context = {
// An unusable launch reply fails before a PlaywrightScenarioSession exists.
browser: () => null,
close: async () => {
closeCalls += 1;
if (blocked)
throw new Error(`startup cleanup failed ${diagnosticSecret}`);
},
} as unknown as BrowserContext;
let finishOpening: () => void = () => undefined;
const opening = new Promise<void>((resolve) => {
finishOpening = resolve;
});
let launchStarted: () => void = () => undefined;
const started = new Promise<void>((resolve) => {
launchStarted = resolve;
});
const launch = vi
.spyOn(chromium, "launchPersistentContext")
.mockImplementation(async () => {
launchStarted();
if (late) await opening;
return context;
});
const provider = createBrowserScenarioProvider({
REA_STARTUP_SECRET: secret,
});
if (provider.close === undefined)
throw new Error("Production scenario provider must expose cleanup");
const closeProvider = provider.close.bind(provider);
const cancellation = new AbortController();
const reason = { caller: "cancelled opening" };
try {
const capturing = captureBrowserScenario(provider, scenario, {
signal: cancellation.signal,
});
await started;
if (late) cancellation.abort(reason);
const result = await capturing;
if (result.ok) throw new Error("Expected startup failure");
expect(result.error.cleanupIncomplete).toBe(true);
const initialProjection = JSON.stringify(
projectAnalysisError(result.error),
);
expect(initialProjection).not.toContain(secret);
expect(initialProjection).not.toContain(diagnosticSecret);
if (!late && secret === "browser_profile")
expect(result.error.cleanupResources).toContain(
"[REDACTED:credential]",
);
if (late) {
expect(primaryAnalysisError(result.error).cause).toBe(reason);
expect(projectAnalysisError(result.error).code).toBe(
"cleanup_incomplete",
);
finishOpening();
}
let retry: unknown;
try {
await closeProvider();
} catch (error) {
retry = error;
}
if (!(retry instanceof AnalysisError))
throw new Error("Expected retained startup cleanup failure");
expect(retry.cleanupIncomplete).toBe(true);
const retryProjection = JSON.stringify(projectAnalysisError(retry));
expect(retryProjection).not.toContain(secret);
expect(retryProjection).not.toContain(diagnosticSecret);
if (secret === "browser_profile")
expect(retry.cleanupResources).toContain("[REDACTED:credential]");
expect(closeCalls).toBeGreaterThan(0);
blocked = false;
await closeProvider();
expect(result.error.cleanupIncomplete).toBe(false);
expect(retry.cleanupIncomplete).toBe(false);
expect(result.error.cleanupResources).toEqual([]);
} finally {
blocked = false;
finishOpening();
await closeProvider();
launch.mockRestore();
}
},
);
it.each([false, true])(
"preserves the original startup failure when no effective secret is declared: %s",
async (emptyDeclaration) => {
const scenario = browserScenarioSchema.parse({
browser: { mode: "launch", executable_path: process.execPath },
start_url: { url: "https://app.test/" },
actions: [
{ step_id: "wait", action: "wait_for_timeout", duration_ms: 1 },
],
secrets: emptyDeclaration
? [{ secret_id: "empty", environment_variable: "REA_EMPTY_SECRET" }]
: [],
});
const original = new BrowserObservationError(
"capture_browser_scenario",
"protocol_error",
{ detail: "ordinary observed startup failure" },
);
const close = vi.fn(async () => undefined);
const launch = vi
.spyOn(chromium, "launchPersistentContext")
.mockResolvedValue({
browser: () => {
throw original;
},
close,
} as unknown as BrowserContext);
try {
await expect(
PlaywrightScenarioSession.open(scenario, { REA_EMPTY_SECRET: "" }, {}),
).rejects.toBe(original);
expect(close).toHaveBeenCalledOnce();
expect(original.cleanupIncomplete).toBe(false);
} finally {
launch.mockRestore();
}
},
);
+150 -1
View File
@@ -1,4 +1,11 @@
import { mkdir, readFile, readdir, stat, writeFile } from "node:fs/promises";
import {
mkdir,
open,
readFile,
readdir,
stat,
writeFile,
} from "node:fs/promises";
import { dirname, join } from "node:path";
import { describe, expect, it, vi } from "vitest";
import { createPackageWithOptions } from "@electron/asar";
@@ -41,6 +48,7 @@ import {
createTestBinarySession,
} from "../fixtures/binarySession.js";
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
import { AnalysisCleanupLease } from "../../src/artifacts/AnalysisCleanupLease.js";
const cleanupError = new ProviderCleanupError(
"fixture",
@@ -727,3 +735,144 @@ describe("managed one-shot cleanup outcomes", () => {
});
});
});
describe("managed one-shot cleanup retry ownership", () => {
it.each(["success", "timeout", "independent primary"] as const)(
"follows exact cleanup completion after a %s outcome",
async (mode) => {
const [path] = await createBinarySessionTargets();
const resource = join(dirname(path), "retained-descriptor");
const handle = await open(resource, "w+");
const cleanup = new ProviderCleanupError("fixture", [resource], {
reason: "controlled first close refusal",
});
let attempts = 0;
const lease = new AnalysisCleanupLease(async () => {
if (++attempts === 1) throw new Error("first close refused");
await handle.close();
}, [resource]);
lease.bindCleanupError(cleanup);
const primaryResource = join(dirname(path), "primary-descriptor");
const primaryHandle =
mode === "independent primary"
? await open(primaryResource, "w+")
: undefined;
const primary =
mode === "success"
? undefined
: new AnalysisTimeoutError("inspect_managed_members", 100, {
capturedOutput: {
stdout: "retained timeout observation",
stderr: "",
truncated: false,
},
...(primaryHandle === undefined
? {}
: {
cleanup: {
reason: "independent primary cleanup",
resources: [primaryResource],
},
}),
});
const primaryLease =
primaryHandle === undefined
? undefined
: new AnalysisCleanupLease(
() => primaryHandle.close(),
[primaryResource],
);
if (primary !== undefined) primaryLease?.bindCleanupError(primary);
const session = createTestBinarySession(() => ({
execute: async (operation) =>
operation === "health"
? observed(null)
: primary === undefined
? observed({ retained: true })
: err(primary),
close: async () => {
try {
await lease.release();
return ok(null);
} catch {
return err(cleanup);
}
},
}));
try {
const result = await runManagedProviderExecution(
{ createManagedBinarySession: () => session },
path,
"inspect_managed_members",
);
if (result.ok) throw new Error("Expected retained cleanup failure");
expect(projectAnalysisError(result.error).code).toBe(
"cleanup_incomplete",
);
expect(handle.fd).toBeGreaterThanOrEqual(0);
expect(await session.close()).toMatchObject({ ok: true });
expect(handle.fd).toBe(-1);
expect(cleanup.cleanupIncomplete).toBe(false);
if (primaryLease !== undefined) {
expect(result.error.cleanupIncomplete).toBe(true);
expect(result.error.cleanupResources).toEqual([primaryResource]);
expect(primaryHandle?.fd).toBeGreaterThanOrEqual(0);
await primaryLease.release();
}
expect(result.error.cleanupIncomplete).toBe(false);
const projected = projectAnalysisError(result.error);
if (primary === undefined)
expect(projected).toMatchObject({
details: {
diagnostics: {
partial_observation: { result: { retained: true } },
},
},
});
else
expect(projected).toMatchObject({
code: "provider_timeout",
details: {
captured_output: { stdout: "retained timeout observation" },
},
});
} finally {
await session.close();
await lease.release();
await primaryLease?.release();
await handle.close();
await primaryHandle?.close();
}
},
);
it.each([false, true])(
"keeps rejected cleanup without a live completion owner unknown (primary failure: %s)",
async (primaryFailure) => {
const [path] = await createBinarySessionTargets();
const unowned = new AnalysisInputError("close_binary");
const { dependencies } = factories({
...(primaryFailure
? {
executionError: new AnalysisTimeoutError(
"inspect_managed_members",
100,
),
}
: {}),
cleanupError: unowned,
});
const result = await runManagedProviderExecution(
dependencies,
path,
"inspect_managed_members",
);
if (result.ok) throw new Error("Expected cleanup rejection");
expect(unowned.cleanupIncomplete).toBe(false);
expect(result.error.cleanupIncomplete).toBe(true);
expect(projectAnalysisError(result.error).code).toBe(
"cleanup_incomplete",
);
},
);
});
+1 -1
View File
@@ -62,7 +62,7 @@ binaries and Ghidra projects are never committed.
`REA_CC`, `REA_CLANG`, and `REA_LLD_LINK` can select alternate commands. The
verifier requires the verified bring-your-own Ghidra 12.1.4 installation through
`GHIDRA_INSTALL_DIR` and validates header classification before starting the
provider. Provider admission accepts the rest of the 12.1.x line; this lane does
provider. Provider admission accepts newer 12.x (>=12.1) builds with an unverified-build limitation; this lane does
not.
Conformance compares semantic facts: provider/profile identity, function
@@ -0,0 +1,177 @@
import type { FileHandle } from "node:fs/promises";
import { open, realpath, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it, onTestFinished, vi } from "vitest";
import { analyzeJavaScriptApplication } from "../../src/application/javascript/JavaScriptApplicationService.js";
import { ArtifactResourceScope } from "../../src/artifacts/ArtifactResourceScope.js";
import { OwnedFileHandle } from "../../src/filesystem/OwnedFileHandle.js";
import { DirectoryArtifactReader } from "../../src/artifacts/DirectoryArtifactReader.js";
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
const injected = vi.hoisted(() => ({
opened: undefined as ((path: unknown, file: FileHandle) => void) | undefined,
}));
vi.mock("node:fs/promises", async (importOriginal) => {
const actual = await importOriginal<typeof import("node:fs/promises")>();
return {
...actual,
open: async (...args: Parameters<typeof actual.open>) => {
const file = await actual.open(...args);
injected.opened?.(args[0], file);
return file;
},
};
});
const causesOf = (error: unknown): unknown[] => {
if (!(error instanceof Error)) return [error];
return [
error,
...causesOf(error.cause),
...(error instanceof AggregateError ? error.errors.flatMap(causesOf) : []),
];
};
it.each(
["addon.node", "unselected.js"].flatMap((filename) =>
[
"physical failure",
"failed cleanup",
"completed cleanup",
"abort only",
"changed bytes",
"completed mismatch cancellation",
"completed matching cancellation",
].map((outcome) => ({ filename, outcome })),
),
)(
"preserves $filename verifier provenance through $outcome",
async ({ filename, outcome }) => {
const root = await realpath(
await createTestTempDirectory("rea-js-verifier-failure-"),
);
const path = join(root, filename);
await writeFile(path, "original payload");
await writeFile(join(root, "main.js"), "module.exports = 42;\n");
const scope = new ArtifactResourceScope();
const controller = new AbortController();
const reason = { source: "artifact source acquisition" };
const physical = Object.assign(
new Error("physical acquisition read denied"),
{ code: "EIO", errno: -5, syscall: "read" },
);
let acquisition = false;
let blocked =
outcome === "failed cleanup" || outcome === "completed cleanup";
let independentBlocked = false;
let descriptor: FileHandle | undefined;
const openEntry = DirectoryArtifactReader.prototype.open;
vi.spyOn(DirectoryArtifactReader.prototype, "open").mockImplementation(
async function (this: DirectoryArtifactReader, entry, signal) {
const stream = await openEntry.call(this, entry, signal);
if (
acquisition &&
entry.adapterKey === path &&
outcome.startsWith("completed ") &&
outcome !== "completed cleanup"
)
stream.once("end", () => controller.abort(reason));
return stream;
},
);
injected.opened = (candidate, file) => {
if (!acquisition || String(candidate) !== path) return;
descriptor = file;
const close = file.close.bind(file);
vi.spyOn(file, "close").mockImplementation(async () => {
if (blocked) throw new Error("acquisition descriptor close denied");
await close();
});
if (outcome === "abort only") controller.abort(reason);
if (outcome === "physical failure" || outcome === "failed cleanup")
vi.spyOn(file, "read").mockImplementationOnce(async () => {
controller.abort(reason);
throw physical;
});
};
onTestFinished(async () => {
injected.opened = undefined;
blocked = false;
independentBlocked = false;
await scope.close();
vi.restoreAllMocks();
});
const result = await analyzeJavaScriptApplication(
{ input_path: root, source_paths: ["main.js"] },
scope,
{
signal: controller.signal,
progress: {
report: async (update) => {
if (update.phase !== "read_javascript_artifacts") return;
acquisition = true;
if (
outcome === "changed bytes" ||
outcome === "completed mismatch cancellation"
)
await writeFile(path, "modified payload");
},
},
},
);
if (result.ok) throw new Error("Expected acquisition failure");
if (
outcome === "abort only" ||
outcome === "completed matching cancellation"
)
expect(result.error).toMatchObject({
reason: "cancelled",
cause: reason,
});
else if (
outcome === "changed bytes" ||
outcome === "completed mismatch cancellation"
)
expect(result.error).toMatchObject({ reason: "integrity" });
else {
expect(result.error).toMatchObject({ reason: "io" });
if (outcome !== "completed cleanup")
expect(causesOf(result.error)).toContain(physical);
}
expect(descriptor).toBeDefined();
if (blocked) {
const independentPath = join(root, "independent-owner");
await writeFile(independentPath, "separately retained bytes");
const independent = await open(independentPath);
const closeIndependent = independent.close.bind(independent);
independentBlocked = true;
vi.spyOn(independent, "close").mockImplementation(async () => {
if (independentBlocked)
throw new Error("independent owner close denied");
await closeIndependent();
});
scope.retain({
kind: "file-handle",
resource: independentPath,
handle: new OwnedFileHandle(independent),
});
expect(result.error.cleanupIncomplete).toBe(true);
expect(result.error.partialObservation).toMatchObject({
kind: "artifact-inventory",
});
await expect(scope.close()).rejects.toBeDefined();
expect(descriptor!.fd).toBeGreaterThanOrEqual(0);
blocked = false;
await expect(scope.close()).rejects.toBeDefined();
expect(descriptor!.fd).toBe(-1);
expect(result.error.cleanupIncomplete).toBe(false);
expect(independent.fd).toBeGreaterThanOrEqual(0);
independentBlocked = false;
await scope.close();
expect(independent.fd).toBe(-1);
}
await scope.close();
expect(descriptor!.fd).toBe(-1);
expect(result.error.cleanupIncomplete).toBe(false);
},
);