mirror of
https://github.com/morluto/rea.git
synced 2026-11-10 00:36:35 +00:00
feat(environment): compare an environment with probe captures of its application
Add rea environment-compare. It takes an environment (manifest and capture Evidence) and probe captures, each one call sequence recorded against a freshly reset application, and answers every probe request as the served environment would from a reset state. Each request is matched to the tool whose template fits it with the most literal segments; live ids are mapped to the environment's by a natural key shared by paired objects and by the identifier fields of two answers, and translated before the environment answers. Answers are compared after mapping, with clocks compared by format and generated values normalised, and the report gives agreement per step, tool and probe with each difference categorised. The command reads files only and makes no network call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d337fc5b6a
commit
e9e16a9a57
12 files changed
+1310
-2
No files matched your search
@@ -691,6 +691,53 @@ state is unavailable, and that neither is emulation and served
|
||||
content is untrusted data. It states that the recorded content belongs to the
|
||||
captured application and must not be published without the rights to it.
|
||||
|
||||
## Measure fidelity against the application
|
||||
|
||||
An environment agrees with its own capture by construction. What matters to an agent is how it answers the calls the
|
||||
capture did not make, what it shows after a write, and how it fails. `rea environment-compare` measures that against the
|
||||
application itself:
|
||||
|
||||
```sh
|
||||
rea environment-compare manifest.json --evidence capture.json \
|
||||
--probe probe-crud.json --probe probe-reads.json --json
|
||||
```
|
||||
|
||||
Each `--probe` is capture Evidence (`inspect-web-network-capture` of a HAR, or `capture-browser-scenario`) of one call
|
||||
sequence made against the live application, seeded as it was for the capture and reset before the sequence. The command
|
||||
makes no network call: whoever records the probes drives the application with its own credentials. For each probe it
|
||||
resets the environment's state, takes the requests in the order they were made, matches each to the manifest tool of the
|
||||
same method whose path template fits with the most literal segments (a format-rule parameter must fit its format),
|
||||
answers it as the served environment would, and compares that answer with the application's.
|
||||
|
||||
A re-seeded application holds the same entities under other ids, so the comparison maps them. Objects in the two answers
|
||||
to one call pair by a natural key, a string field unique on each side that both share, such as a name or a slug; their
|
||||
identifier fields (`id`, `uuid`, names ending in `id` or `_id`), and those of two single answers such as a create's, map
|
||||
the live value to the environment's. A live id in a later request's path, query or body is translated before the
|
||||
environment answers. After mapping, two clock values compare by format only (ISO-8601 strings, and epoch numbers under
|
||||
time-named fields), and two different values each side generated on its own compare equal: both UUIDs, or one length of
|
||||
eight or more with the same character classes and a digit. The report counts each kind of normalised value.
|
||||
|
||||
A step agrees when the statuses and the normalised bodies are equal. The report gives agreement overall, per probe and
|
||||
per tool, and each step's differences, at most 20, with a JSON pointer and both values cut short:
|
||||
|
||||
| Category | Meaning |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| `status` | The statuses differ |
|
||||
| `missing-field` | The application's object has a member the environment's lacks |
|
||||
| `extra-field` | The environment's object has a member the application's lacks |
|
||||
| `value` | Values differ, or one is an object or array and the other is not |
|
||||
| `length` | Arrays differ in length, or not every item pairs |
|
||||
| `order` | Paired items come in a different order |
|
||||
| `clock-format` | Both are clocks, in different formats |
|
||||
| `refused` | The environment refused the call |
|
||||
| `not-served` | The request's tool is one the environment does not offer, such as a withheld write |
|
||||
| `no-tool` | No manifest tool matches the request |
|
||||
| `unmapped-id` | The application answered with success and the environment did not, for a path id the comparison never mapped |
|
||||
|
||||
Agreement covers only the calls the probes make, after the mapping the answers allow. Values compared by form, clocks
|
||||
and generated values, can still differ in content, and an equal answer after a write shows the state matched for that
|
||||
read, not that the write had no other effect.
|
||||
|
||||
## Compatibility verification
|
||||
|
||||
`npm run verify:agentenv:environment` is the hosted `real-agentenv` lane
|
||||
|
||||
+15
@@ -456,6 +456,21 @@ of the serve command's stopped document. See
|
||||
[agent environments](agent-environments.md#export-a-bundle) for the folder
|
||||
layout and what the smoke task checks.
|
||||
|
||||
## Compare an environment with its application
|
||||
|
||||
`rea environment-compare <manifest.json> --evidence <capture.json> --probe <probe.json> [--probe <probe.json> ...]`
|
||||
compares the environment the pair serves with probe captures of the application it came from. Each probe is capture
|
||||
Evidence of one call sequence recorded against the application after a reset; the environment's state is reset before
|
||||
each probe, every request is answered as the served environment answers it, and the answer is compared with the
|
||||
application's after ids are mapped and clocks and generated values are normalised. The command reads files only and
|
||||
makes no network call. It prints the report and exits `0`, or prints a structured `invalid_request` error and exits `1`
|
||||
for the same pair refusals as `environment-serve` or a probe that is not capture Evidence. See
|
||||
[agent environments](agent-environments.md#measure-fidelity-against-the-application) for the report.
|
||||
|
||||
```sh
|
||||
rea environment-compare manifest.json --evidence capture.json --probe probe.json --json
|
||||
```
|
||||
|
||||
## Output and exit status
|
||||
|
||||
`rea mcp doctor --json` checks production MCP startup and reports child logs in
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
timedCaptureEvidence,
|
||||
type TimedCaptureRow,
|
||||
} from "../../../tests/fixtures/analysisView.js";
|
||||
import {
|
||||
environmentServeSource,
|
||||
environmentStateServeFixture,
|
||||
} from "../../../tests/fixtures/environmentServe.js";
|
||||
import {
|
||||
BOOK_NOT_FOUND,
|
||||
BOOK_ROWS,
|
||||
book,
|
||||
renamedBook,
|
||||
} from "../../../tests/fixtures/environmentStateWrites.js";
|
||||
import { compareEnvironment } from "./environmentCompare.js";
|
||||
import { matchProbeRequest } from "./environmentCompareMatch.js";
|
||||
|
||||
const fixture = environmentStateServeFixture(timedCaptureEvidence(BOOK_ROWS));
|
||||
const source = environmentServeSource(fixture);
|
||||
|
||||
/** The books application re-seeded: the same books under other ids and creation times. */
|
||||
const LIVE_FIRST = book(101, "First Book", "2026-10-20T09:00:00Z");
|
||||
const LIVE_SECOND = book(102, "Second Book", "2026-10-20T10:00:00Z");
|
||||
const LIVE_FOURTH = book(103, "Fourth Book", "2026-10-20T11:00:00Z");
|
||||
const LIVE_RENAMED = renamedBook(LIVE_FIRST, "Renamed Book", 5);
|
||||
|
||||
const compare = (...probes: (readonly TimedCaptureRow[])[]) => {
|
||||
const report = compareEnvironment({
|
||||
source,
|
||||
captureValue: fixture.capture,
|
||||
probes: probes.map((rows, index) => ({
|
||||
label: `probe-${String(index + 1)}`,
|
||||
value: timedCaptureEvidence(rows),
|
||||
})),
|
||||
operation: "environment-compare",
|
||||
});
|
||||
if (!report.ok) throw report.error;
|
||||
return report.value;
|
||||
};
|
||||
|
||||
describe("matchProbeRequest", () => {
|
||||
it("prefers literal segments, requires format-rule parameters to fit, and maps query and body arguments", () => {
|
||||
const item = matchProbeRequest(source.manifest, {
|
||||
method: "GET",
|
||||
path: "/api/books/7",
|
||||
query: "?expand=all",
|
||||
body: undefined,
|
||||
});
|
||||
expect(item?.tool.path_template).toBe("/api/books/{numeric}");
|
||||
expect(item?.args).toEqual({ id: "7" });
|
||||
expect(item?.ignoredQuery).toEqual(["expand"]);
|
||||
const create = matchProbeRequest(source.manifest, {
|
||||
method: "POST",
|
||||
path: "/api/books",
|
||||
query: null,
|
||||
body: { name: "X" },
|
||||
});
|
||||
expect(create?.args).toEqual({ body: { name: "X" } });
|
||||
expect(
|
||||
matchProbeRequest(source.manifest, {
|
||||
method: "GET",
|
||||
path: "/api/books/x",
|
||||
query: null,
|
||||
body: undefined,
|
||||
}),
|
||||
).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("compareEnvironment", () => {
|
||||
it("agrees with a re-seeded application once seeded and created ids are mapped", () => {
|
||||
const report = compare([
|
||||
["GET", "/api/books", [LIVE_FIRST, LIVE_SECOND]],
|
||||
["GET", "/api/books/101", LIVE_FIRST],
|
||||
["POST", "/api/books", LIVE_FOURTH, 201, { name: "Fourth Book" }],
|
||||
["GET", "/api/books/103", LIVE_FOURTH],
|
||||
["PATCH", "/api/books/101", LIVE_RENAMED, 200, { name: "Renamed Book" }],
|
||||
["GET", "/api/books/101", LIVE_RENAMED],
|
||||
["DELETE", "/api/books/103", "", 204],
|
||||
["GET", "/api/books/103", BOOK_NOT_FOUND, 404],
|
||||
["GET", "/api/books", [LIVE_RENAMED, LIVE_SECOND]],
|
||||
]);
|
||||
expect(report.environment.state).toBe("served");
|
||||
expect(report.probes[0]?.results.filter((step) => !step.equal)).toEqual([]);
|
||||
expect(report.summary).toMatchObject({
|
||||
probes: 1,
|
||||
steps: 9,
|
||||
equal: 9,
|
||||
agreement: 1,
|
||||
});
|
||||
expect(report.probes[0]?.ids_mapped).toBe(3);
|
||||
expect(report.summary.normalised.ids).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("resets the state per probe and categorises every kind of disagreement", () => {
|
||||
const report = compare(
|
||||
[
|
||||
["GET", "/api/books", [LIVE_FIRST, LIVE_SECOND]],
|
||||
["DELETE", "/api/books/101", { detail: "Book is referenced." }, 409],
|
||||
["GET", "/api/authors", []],
|
||||
["GET", "/api/books/555", LIVE_FIRST],
|
||||
],
|
||||
[["GET", "/api/books", [LIVE_SECOND, LIVE_FIRST, LIVE_FOURTH]]],
|
||||
);
|
||||
const [first, second] = report.probes;
|
||||
expect(first?.results.map((step) => step.categories)).toEqual([
|
||||
[],
|
||||
["status", "value"],
|
||||
["no-tool"],
|
||||
["unmapped-id", "status", "missing-field", "extra-field"],
|
||||
]);
|
||||
expect(second?.results[0]?.categories).toEqual(["length", "order"]);
|
||||
expect(report.summary.categories).toMatchObject({
|
||||
status: 2,
|
||||
"no-tool": 1,
|
||||
"unmapped-id": 1,
|
||||
length: 1,
|
||||
order: 1,
|
||||
});
|
||||
expect(report.tools.map((tool) => tool.tool)).toContain(
|
||||
"delete_books_by_id",
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,382 @@
|
||||
import { z } from "zod";
|
||||
import { isJsonLike } from "../../domain/analysisView/EnvironmentManifestAccumulator.js";
|
||||
import {
|
||||
mediaTypeEssence,
|
||||
readRecordedRequestLine,
|
||||
} from "../../domain/analysisView/webNetworkRouteTemplates.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { parseExactJson } from "../../domain/exactJsonNumber.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import type { WebNetworkCaptureRecord } from "../../domain/webNetworkCapture.js";
|
||||
import {
|
||||
compareAnswers,
|
||||
DIFFERENCE_CATEGORIES,
|
||||
STEP_DIFFERENCE_LIMIT,
|
||||
type Answer,
|
||||
type Difference,
|
||||
type DifferenceCategory,
|
||||
type Normalised,
|
||||
} from "./environmentCompareDiff.js";
|
||||
import { IdMap, learnIds, translateValue } from "./environmentCompareIds.js";
|
||||
import {
|
||||
matchProbeRequest,
|
||||
type ProbeRequest,
|
||||
type ProbeToolCall,
|
||||
} from "./environmentCompareMatch.js";
|
||||
import {
|
||||
inputError,
|
||||
parseCaptureEvidence,
|
||||
recordedExchange,
|
||||
replayedBody,
|
||||
type EnvironmentSource,
|
||||
type ReplayedBody,
|
||||
} from "./environmentSource.js";
|
||||
import { prepareServedState, servedCatalog } from "./environmentState.js";
|
||||
import { EnvironmentStateStore } from "./environmentStateStore.js";
|
||||
import {
|
||||
environmentToolAnswer,
|
||||
type EnvironmentToolAnswer,
|
||||
} from "./environmentToolAnswer.js";
|
||||
|
||||
/** One probe capture file: its label for the report and its parsed JSON. */
|
||||
export interface CompareProbeInput {
|
||||
readonly label: string;
|
||||
readonly value: unknown;
|
||||
}
|
||||
|
||||
/** One probe request, how the environment answered it, and how that differs from the application. */
|
||||
export interface CompareStep {
|
||||
readonly ordinal: number;
|
||||
readonly method: string;
|
||||
readonly path: string;
|
||||
readonly tool: string | null;
|
||||
readonly live_status: number | null;
|
||||
readonly env_status: number | null;
|
||||
readonly equal: boolean;
|
||||
readonly categories: DifferenceCategory[];
|
||||
readonly differences: Difference[];
|
||||
readonly ignored_query: string[];
|
||||
}
|
||||
|
||||
export interface CompareProbeReport {
|
||||
readonly label: string;
|
||||
readonly evidence_id: string;
|
||||
readonly steps: number;
|
||||
readonly equal: number;
|
||||
readonly ids_mapped: number;
|
||||
readonly id_conflicts: number;
|
||||
readonly normalised: Normalised;
|
||||
readonly results: CompareStep[];
|
||||
}
|
||||
|
||||
export interface CompareReport {
|
||||
readonly status: "complete";
|
||||
readonly environment: {
|
||||
readonly name: string;
|
||||
readonly manifest_evidence_id: string;
|
||||
readonly evidence_id: string;
|
||||
readonly state: "served" | "replay-only";
|
||||
};
|
||||
readonly summary: {
|
||||
readonly probes: number;
|
||||
readonly steps: number;
|
||||
readonly equal: number;
|
||||
readonly agreement: number | null;
|
||||
readonly categories: Record<DifferenceCategory, number>;
|
||||
readonly normalised: Normalised;
|
||||
};
|
||||
readonly tools: { tool: string; steps: number; equal: number }[];
|
||||
readonly probes: CompareProbeReport[];
|
||||
}
|
||||
|
||||
const startedSchema = z.object({ startedDateTime: z.string() });
|
||||
const postDataSchema = z.object({
|
||||
request: z.object({
|
||||
postData: z.object({ mimeType: z.string(), text: z.string() }),
|
||||
}),
|
||||
});
|
||||
|
||||
const startedMs = (record: WebNetworkCaptureRecord): number => {
|
||||
const started = startedSchema.safeParse(record.reported);
|
||||
const ms = started.success ? Date.parse(started.data.startedDateTime) : NaN;
|
||||
return Number.isFinite(ms) ? ms : 0;
|
||||
};
|
||||
|
||||
/** The JSON a request body holds, its text when it is not JSON, or undefined without one. */
|
||||
const requestBody = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
): JsonValue | undefined => {
|
||||
const parsed = postDataSchema.safeParse(record.reported);
|
||||
if (!parsed.success) return undefined;
|
||||
const { mimeType, text } = parsed.data.request.postData;
|
||||
return isJsonLike(mediaTypeEssence(mimeType))
|
||||
? (parseExactJson(text) ?? text)
|
||||
: text;
|
||||
};
|
||||
|
||||
/** The application's body as an environment answer carries it: JSON, text, or null when there is none or it is empty. */
|
||||
const liveBody = (body: ReplayedBody): JsonValue => {
|
||||
if (body.kind === "json") return body.value;
|
||||
if (body.kind === "none") return null;
|
||||
if (body.kind === "base64") return body.encoded;
|
||||
return body.text === "" ? null : body.text;
|
||||
};
|
||||
|
||||
/** The probe's requests in the order they were made, each with the application's answer. */
|
||||
const probeRequests = (
|
||||
records: readonly WebNetworkCaptureRecord[],
|
||||
): { ordinal: number; request: ProbeRequest; live: Answer }[] =>
|
||||
[...records]
|
||||
.sort(
|
||||
(left, right) =>
|
||||
startedMs(left) - startedMs(right) || left.ordinal - right.ordinal,
|
||||
)
|
||||
.flatMap((record) => {
|
||||
const line = readRecordedRequestLine(record.reported, true, () => true);
|
||||
const exchange = recordedExchange(record);
|
||||
if (line === null || line.kind !== "line" || exchange === undefined)
|
||||
return [];
|
||||
return [
|
||||
{
|
||||
ordinal: record.ordinal,
|
||||
request: {
|
||||
method: line.method,
|
||||
path: exchange.path,
|
||||
query: exchange.query,
|
||||
body: requestBody(record),
|
||||
},
|
||||
live: {
|
||||
status: exchange.status,
|
||||
body: liveBody(replayedBody(exchange)),
|
||||
},
|
||||
},
|
||||
];
|
||||
});
|
||||
|
||||
const isSuccess = (status: number | null): boolean =>
|
||||
status !== null && status >= 200 && status < 300;
|
||||
|
||||
interface ProbeRun {
|
||||
readonly answer: EnvironmentToolAnswer;
|
||||
readonly store: EnvironmentStateStore | undefined;
|
||||
readonly served: ReadonlySet<string>;
|
||||
readonly map: IdMap;
|
||||
readonly normalised: Normalised;
|
||||
}
|
||||
|
||||
const heldIds = (
|
||||
store: EnvironmentStateStore | undefined,
|
||||
): ReadonlySet<string> =>
|
||||
new Set(
|
||||
store?.snapshot().rea_state.entities.map((entity) => entity.id) ?? [],
|
||||
);
|
||||
|
||||
const stepOf = (
|
||||
base: Pick<CompareStep, "ordinal" | "method" | "path" | "live_status">,
|
||||
call: ProbeToolCall | undefined,
|
||||
outcome: { env_status: number | null; differences: Difference[] },
|
||||
): CompareStep => {
|
||||
const categories = [...new Set(outcome.differences.map((d) => d.category))];
|
||||
return {
|
||||
...base,
|
||||
tool: call?.tool.name ?? null,
|
||||
env_status: outcome.env_status,
|
||||
equal: outcome.differences.length === 0,
|
||||
categories,
|
||||
differences: outcome.differences.slice(0, STEP_DIFFERENCE_LIMIT),
|
||||
ignored_query: [...(call?.ignoredQuery ?? [])],
|
||||
};
|
||||
};
|
||||
|
||||
/** Answer one probe request as the environment would and compare it with the application's answer. */
|
||||
const compareStep = (
|
||||
run: ProbeRun,
|
||||
manifest: EnvironmentSource["manifest"],
|
||||
probe: { ordinal: number; request: ProbeRequest; live: Answer },
|
||||
): CompareStep => {
|
||||
const { request, live } = probe;
|
||||
const base = {
|
||||
ordinal: probe.ordinal,
|
||||
method: request.method,
|
||||
path: request.path,
|
||||
live_status: live.status,
|
||||
};
|
||||
const call = matchProbeRequest(manifest, request);
|
||||
if (call === undefined)
|
||||
return stepOf(base, call, {
|
||||
env_status: null,
|
||||
differences: [{ category: "no-tool", pointer: "" }],
|
||||
});
|
||||
if (!run.served.has(call.tool.name))
|
||||
return stepOf(base, call, {
|
||||
env_status: null,
|
||||
differences: [{ category: "not-served", pointer: "" }],
|
||||
});
|
||||
const held = heldIds(run.store);
|
||||
const unknownPath = call.pathArguments.some((name) => {
|
||||
const value = call.args[name];
|
||||
return (
|
||||
typeof value === "string" && !run.map.hasLive(value) && !held.has(value)
|
||||
);
|
||||
});
|
||||
const args = Object.fromEntries(
|
||||
Object.entries(call.args).map(([name, value]) => [
|
||||
name,
|
||||
translateValue(value, run.map),
|
||||
]),
|
||||
);
|
||||
const answered = run.answer(call.tool, args);
|
||||
if (!answered.ok)
|
||||
return stepOf(base, call, {
|
||||
env_status: null,
|
||||
differences: [
|
||||
{
|
||||
category: "refused",
|
||||
pointer: "",
|
||||
env: answered.error.message.slice(0, 200),
|
||||
},
|
||||
],
|
||||
});
|
||||
const env: Answer = {
|
||||
status: answered.value.status,
|
||||
body: answered.value.body,
|
||||
};
|
||||
learnIds(live.body, env.body, run.map, heldIds(run.store));
|
||||
const compared = compareAnswers(live, env, run.map);
|
||||
for (const key of ["ids", "clocks", "generated"] as const)
|
||||
run.normalised[key] += compared.normalised[key];
|
||||
const differences = [...compared.differences];
|
||||
if (unknownPath && isSuccess(live.status) && !isSuccess(env.status))
|
||||
differences.unshift({ category: "unmapped-id", pointer: "" });
|
||||
return stepOf(base, call, { env_status: env.status, differences });
|
||||
};
|
||||
|
||||
const probeReport = (
|
||||
label: string,
|
||||
evidenceId: string,
|
||||
run: ProbeRun,
|
||||
results: CompareStep[],
|
||||
): CompareProbeReport => ({
|
||||
label,
|
||||
evidence_id: evidenceId,
|
||||
steps: results.length,
|
||||
equal: results.filter((step) => step.equal).length,
|
||||
ids_mapped: run.map.size,
|
||||
id_conflicts: run.map.conflicts,
|
||||
normalised: run.normalised,
|
||||
results,
|
||||
});
|
||||
|
||||
const summaryOf = (
|
||||
probes: readonly CompareProbeReport[],
|
||||
): CompareReport["summary"] => {
|
||||
const steps = probes.flatMap((probe) => probe.results);
|
||||
const categories = Object.fromEntries(
|
||||
DIFFERENCE_CATEGORIES.map((category) => [category, 0]),
|
||||
) as Record<DifferenceCategory, number>;
|
||||
for (const step of steps)
|
||||
for (const category of step.categories) categories[category] += 1;
|
||||
const equal = steps.filter((step) => step.equal).length;
|
||||
const normalised = { ids: 0, clocks: 0, generated: 0 };
|
||||
for (const probe of probes)
|
||||
for (const key of ["ids", "clocks", "generated"] as const)
|
||||
normalised[key] += probe.normalised[key];
|
||||
return {
|
||||
probes: probes.length,
|
||||
steps: steps.length,
|
||||
equal,
|
||||
agreement:
|
||||
steps.length === 0
|
||||
? null
|
||||
: Math.round((equal / steps.length) * 1000) / 1000,
|
||||
categories,
|
||||
normalised,
|
||||
};
|
||||
};
|
||||
|
||||
const toolsOf = (
|
||||
probes: readonly CompareProbeReport[],
|
||||
): CompareReport["tools"] => {
|
||||
const tools = new Map<
|
||||
string,
|
||||
{ tool: string; steps: number; equal: number }
|
||||
>();
|
||||
for (const step of probes.flatMap((probe) => probe.results)) {
|
||||
if (step.tool === null) continue;
|
||||
const entry = tools.get(step.tool) ?? {
|
||||
tool: step.tool,
|
||||
steps: 0,
|
||||
equal: 0,
|
||||
};
|
||||
entry.steps += 1;
|
||||
if (step.equal) entry.equal += 1;
|
||||
tools.set(step.tool, entry);
|
||||
}
|
||||
return [...tools.values()].sort(
|
||||
(left, right) =>
|
||||
left.equal / left.steps - right.equal / right.steps ||
|
||||
(left.tool < right.tool ? -1 : 1),
|
||||
);
|
||||
};
|
||||
|
||||
/** Compare an environment with probe captures of the application: each probe from a reset state, every request answered and compared. */
|
||||
export const compareEnvironment = (input: {
|
||||
readonly source: EnvironmentSource;
|
||||
readonly captureValue: unknown;
|
||||
readonly probes: readonly CompareProbeInput[];
|
||||
readonly operation: string;
|
||||
}): Result<CompareReport, AnalysisError> => {
|
||||
const { source, operation } = input;
|
||||
const state = prepareServedState(source, input.captureValue, {
|
||||
operation,
|
||||
}).state;
|
||||
const catalog = servedCatalog(source, state, source.manifest.name, operation);
|
||||
if (!catalog.ok) return catalog;
|
||||
const served = new Set(catalog.value.tools.map((tool) => tool.name));
|
||||
const probes: CompareProbeReport[] = [];
|
||||
for (const [index, probe] of input.probes.entries()) {
|
||||
const capture = parseCaptureEvidence(operation, probe.value);
|
||||
if (!capture.ok)
|
||||
return err(
|
||||
inputError(
|
||||
operation,
|
||||
[
|
||||
{
|
||||
path: ["probe", index],
|
||||
reason: "invalid_value",
|
||||
message: `${probe.label}: ${capture.error.message}`,
|
||||
},
|
||||
],
|
||||
"A probe file is not capture Evidence.",
|
||||
),
|
||||
);
|
||||
const store =
|
||||
state === undefined ? undefined : new EnvironmentStateStore(state);
|
||||
const run: ProbeRun = {
|
||||
answer: environmentToolAnswer(source, store),
|
||||
store,
|
||||
served,
|
||||
map: new IdMap(),
|
||||
normalised: { ids: 0, clocks: 0, generated: 0 },
|
||||
};
|
||||
const results = probeRequests(capture.value.records).map((request) =>
|
||||
compareStep(run, source.manifest, request),
|
||||
);
|
||||
probes.push(
|
||||
probeReport(probe.label, capture.value.evidenceId, run, results),
|
||||
);
|
||||
}
|
||||
return ok({
|
||||
status: "complete",
|
||||
environment: {
|
||||
name: source.manifest.name,
|
||||
manifest_evidence_id: source.manifestEvidenceId,
|
||||
evidence_id: source.evidenceId,
|
||||
state: state === undefined ? "replay-only" : "served",
|
||||
},
|
||||
summary: summaryOf(probes),
|
||||
tools: toolsOf(probes),
|
||||
probes,
|
||||
});
|
||||
};
|
||||
@@ -0,0 +1,86 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { compareAnswers } from "./environmentCompareDiff.js";
|
||||
import { IdMap } from "./environmentCompareIds.js";
|
||||
|
||||
const compare = (
|
||||
live: unknown,
|
||||
env: unknown,
|
||||
map = new IdMap(),
|
||||
status: [number, number] = [200, 200],
|
||||
) =>
|
||||
compareAnswers(
|
||||
{ status: status[0], body: live as never },
|
||||
{ status: status[1], body: env as never },
|
||||
map,
|
||||
);
|
||||
|
||||
describe("compareAnswers", () => {
|
||||
it("agrees on mapped ids, clocks of one format and generated values, and counts each", () => {
|
||||
const map = new IdMap();
|
||||
map.learn(101, 1);
|
||||
const result = compare(
|
||||
{
|
||||
id: 101,
|
||||
created_at: "2026-10-20T10:00:00Z",
|
||||
token: "3f2a9c8d7e",
|
||||
uuid: "6f1c2b9e-2a3d-4f5e-8a9b-0c1d2e3f4a5b",
|
||||
},
|
||||
{
|
||||
id: 1,
|
||||
created_at: "2026-10-17T18:00:03Z",
|
||||
token: "9b8e7d6c5a",
|
||||
uuid: "0a1b2c3d-4e5f-4a6b-9c7d-8e9f0a1b2c3d",
|
||||
},
|
||||
map,
|
||||
);
|
||||
expect(result.differences).toEqual([]);
|
||||
expect(result.normalised).toEqual({ ids: 1, clocks: 1, generated: 2 });
|
||||
});
|
||||
|
||||
it("categorises status, missing and extra fields, values and clock formats", () => {
|
||||
const result = compare(
|
||||
{ name: "A", gone: 1, at: "2026-10-20T10:00:00.123+00:00" },
|
||||
{ name: "B", added: true, at: "2026-10-20T10:00:00Z" },
|
||||
new IdMap(),
|
||||
[409, 204],
|
||||
);
|
||||
expect(result.differences.map((d) => [d.category, d.pointer])).toEqual([
|
||||
["status", ""],
|
||||
["value", "/name"],
|
||||
["missing-field", "/gone"],
|
||||
["clock-format", "/at"],
|
||||
["extra-field", "/added"],
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports array length and order and compares paired items", () => {
|
||||
const result = compare(
|
||||
[
|
||||
{ name: "B", n: 2 },
|
||||
{ name: "A", n: 1 },
|
||||
{ name: "C", n: 3 },
|
||||
],
|
||||
[
|
||||
{ name: "A", n: 1 },
|
||||
{ name: "B", n: 5 },
|
||||
],
|
||||
);
|
||||
expect(result.differences.map((d) => [d.category, d.pointer])).toEqual([
|
||||
["length", ""],
|
||||
["order", ""],
|
||||
["value", "/0/n"],
|
||||
]);
|
||||
});
|
||||
|
||||
it("treats an epoch number as a clock only under a time-named field", () => {
|
||||
expect(
|
||||
compare({ updated: 1760000000 }, { updated: 1760000999 }).differences,
|
||||
).toEqual([]);
|
||||
expect(
|
||||
compare({ size: 1760000000 }, { size: 1760000999 }).differences.map(
|
||||
(d) => d.category,
|
||||
),
|
||||
).toEqual(["value"]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,226 @@
|
||||
import type { JsonObject } from "../../domain/analysisView/environmentStateRules.js";
|
||||
import {
|
||||
isUuid,
|
||||
readClock,
|
||||
type ClockFormat,
|
||||
} from "../../domain/analysisView/environmentStateWriteValues.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import { pairElements, type IdMap } from "./environmentCompareIds.js";
|
||||
|
||||
export const DIFFERENCE_CATEGORIES = [
|
||||
"status",
|
||||
"missing-field",
|
||||
"extra-field",
|
||||
"value",
|
||||
"length",
|
||||
"order",
|
||||
"clock-format",
|
||||
"refused",
|
||||
"not-served",
|
||||
"no-tool",
|
||||
"unmapped-id",
|
||||
] as const;
|
||||
export type DifferenceCategory = (typeof DIFFERENCE_CATEGORIES)[number];
|
||||
|
||||
/** One way an environment answer differs from the live one: where, and both values cut short. */
|
||||
export interface Difference {
|
||||
readonly category: DifferenceCategory;
|
||||
readonly pointer: string;
|
||||
readonly live?: string;
|
||||
readonly env?: string;
|
||||
}
|
||||
|
||||
/** Values the comparison treated as equal although they differ: mapped ids, clocks of one format, generated values. */
|
||||
export interface Normalised {
|
||||
ids: number;
|
||||
clocks: number;
|
||||
generated: number;
|
||||
}
|
||||
|
||||
/** At most this many differences are kept per step. */
|
||||
export const STEP_DIFFERENCE_LIMIT = 20;
|
||||
const VALUE_CHARS = 120;
|
||||
const TIME_FIELD = /(at|time|date|timestamp|created|updated|modified)$/iu;
|
||||
|
||||
const isObject = (value: JsonValue | undefined): value is JsonObject =>
|
||||
typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
|
||||
const shown = (value: JsonValue | undefined): string => {
|
||||
const text = value === undefined ? "<absent>" : JSON.stringify(value);
|
||||
return text.length > VALUE_CHARS ? `${text.slice(0, VALUE_CHARS)}…` : text;
|
||||
};
|
||||
|
||||
const escapePointer = (key: string): string =>
|
||||
key.replaceAll("~", "~0").replaceAll("/", "~1");
|
||||
|
||||
const formatKey = (format: ClockFormat): string =>
|
||||
format.kind === "epoch"
|
||||
? `epoch-${format.unit}`
|
||||
: `iso${format.separator}${String(format.fraction)}${format.zone === "" ? "" : format.zone === "Z" ? "Z" : "offset"}`;
|
||||
|
||||
/** The format of a clock value: an ISO-8601 string, or an epoch number under a time-named field. */
|
||||
const clockFormat = (
|
||||
value: JsonValue,
|
||||
key: string | undefined,
|
||||
): string | undefined => {
|
||||
if (typeof value === "number" && (key === undefined || !TIME_FIELD.test(key)))
|
||||
return undefined;
|
||||
const clock = readClock(value);
|
||||
return clock === undefined ? undefined : formatKey(clock.format);
|
||||
};
|
||||
|
||||
const characterClasses = (text: string): string =>
|
||||
[
|
||||
...new Set(
|
||||
text
|
||||
.replaceAll(/\d/gu, "0")
|
||||
.replaceAll(/[a-z]/gu, "a")
|
||||
.replaceAll(/[A-Z]/gu, "A"),
|
||||
),
|
||||
]
|
||||
.sort()
|
||||
.join("");
|
||||
|
||||
/** Two values each side generated on its own: both UUIDs, or one length of eight or more with the same character classes and a digit. */
|
||||
const generatedPair = (live: string, env: string): boolean =>
|
||||
(isUuid(live) && isUuid(env)) ||
|
||||
(live.length === env.length &&
|
||||
live.length >= 8 &&
|
||||
/\d/u.test(live) &&
|
||||
/\d/u.test(env) &&
|
||||
characterClasses(live) === characterClasses(env));
|
||||
|
||||
class Differ {
|
||||
readonly differences: Difference[] = [];
|
||||
readonly normalised: Normalised = { ids: 0, clocks: 0, generated: 0 };
|
||||
|
||||
constructor(readonly map: IdMap) {}
|
||||
|
||||
add(difference: Difference): void {
|
||||
this.differences.push(difference);
|
||||
}
|
||||
|
||||
scalar(live: JsonValue, env: JsonValue, pointer: string, key?: string): void {
|
||||
if (live === env) return;
|
||||
if (
|
||||
(typeof live === "string" || typeof live === "number") &&
|
||||
(typeof env === "string" || typeof env === "number")
|
||||
) {
|
||||
const mapped = this.map.env(live);
|
||||
if (mapped !== undefined && String(mapped) === String(env)) {
|
||||
this.normalised.ids += 1;
|
||||
return;
|
||||
}
|
||||
const liveClock = clockFormat(live, key);
|
||||
const envClock = clockFormat(env, key);
|
||||
if (liveClock !== undefined && envClock !== undefined) {
|
||||
if (liveClock === envClock) this.normalised.clocks += 1;
|
||||
else
|
||||
this.add({
|
||||
category: "clock-format",
|
||||
pointer,
|
||||
live: shown(live),
|
||||
env: shown(env),
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (
|
||||
typeof live === "string" &&
|
||||
typeof env === "string" &&
|
||||
!this.map.hasLive(live) &&
|
||||
!this.map.hasEnv(env) &&
|
||||
generatedPair(live, env)
|
||||
) {
|
||||
this.normalised.generated += 1;
|
||||
return;
|
||||
}
|
||||
}
|
||||
this.add({
|
||||
category: "value",
|
||||
pointer,
|
||||
live: shown(live),
|
||||
env: shown(env),
|
||||
});
|
||||
}
|
||||
|
||||
array(live: JsonValue[], env: JsonValue[], pointer: string): void {
|
||||
const pairs = pairElements(live, env);
|
||||
if (live.length !== env.length || pairs.length !== live.length)
|
||||
this.add({
|
||||
category: "length",
|
||||
pointer,
|
||||
live: `${String(live.length)} items`,
|
||||
env: `${String(env.length)} items (${String(pairs.length)} paired)`,
|
||||
});
|
||||
const envOrder = pairs.map(([, right]) => right);
|
||||
if (
|
||||
envOrder.some((right, index) => index > 0 && right < envOrder[index - 1]!)
|
||||
)
|
||||
this.add({ category: "order", pointer });
|
||||
for (const [left, right] of pairs)
|
||||
this.value(live[left]!, env[right]!, `${pointer}/${String(left)}`);
|
||||
}
|
||||
|
||||
object(live: JsonObject, env: JsonObject, pointer: string): void {
|
||||
for (const [key, value] of Object.entries(live)) {
|
||||
const child = `${pointer}/${escapePointer(key)}`;
|
||||
if (!Object.hasOwn(env, key))
|
||||
this.add({
|
||||
category: "missing-field",
|
||||
pointer: child,
|
||||
live: shown(value),
|
||||
});
|
||||
else this.value(value, env[key]!, child, key);
|
||||
}
|
||||
for (const [key, value] of Object.entries(env))
|
||||
if (!Object.hasOwn(live, key))
|
||||
this.add({
|
||||
category: "extra-field",
|
||||
pointer: `${pointer}/${escapePointer(key)}`,
|
||||
env: shown(value),
|
||||
});
|
||||
}
|
||||
|
||||
value(live: JsonValue, env: JsonValue, pointer: string, key?: string): void {
|
||||
if (Array.isArray(live) && Array.isArray(env))
|
||||
this.array(live, env, pointer);
|
||||
else if (isObject(live) && isObject(env)) this.object(live, env, pointer);
|
||||
else if (
|
||||
Array.isArray(live) ||
|
||||
Array.isArray(env) ||
|
||||
isObject(live) ||
|
||||
isObject(env)
|
||||
)
|
||||
this.add({
|
||||
category: "value",
|
||||
pointer,
|
||||
live: shown(live),
|
||||
env: shown(env),
|
||||
});
|
||||
else this.scalar(live, env, pointer, key);
|
||||
}
|
||||
}
|
||||
|
||||
/** One answer: its status, and its JSON body, its text, or null without one. */
|
||||
export interface Answer {
|
||||
readonly status: number | null;
|
||||
readonly body: JsonValue;
|
||||
}
|
||||
|
||||
/** Compare a live answer with the environment's after normalisation. */
|
||||
export const compareAnswers = (
|
||||
live: Answer,
|
||||
env: Answer,
|
||||
map: IdMap,
|
||||
): { readonly differences: Difference[]; readonly normalised: Normalised } => {
|
||||
const differ = new Differ(map);
|
||||
if (live.status !== env.status)
|
||||
differ.add({
|
||||
category: "status",
|
||||
pointer: "",
|
||||
live: String(live.status),
|
||||
env: String(env.status),
|
||||
});
|
||||
differ.value(live.body, env.body, "");
|
||||
return { differences: differ.differences, normalised: differ.normalised };
|
||||
};
|
||||
@@ -0,0 +1,93 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
IdMap,
|
||||
learnIds,
|
||||
pairElements,
|
||||
translateValue,
|
||||
} from "./environmentCompareIds.js";
|
||||
|
||||
describe("IdMap", () => {
|
||||
it("keeps the first pairing of either side and counts conflicting ones", () => {
|
||||
const map = new IdMap();
|
||||
map.learn(101, 1);
|
||||
map.learn("101", 1);
|
||||
map.learn(101, 2);
|
||||
map.learn(102, 1);
|
||||
expect(map.env(101)).toBe(1);
|
||||
expect(map.env("101")).toBe(1);
|
||||
expect(map.hasEnv(1)).toBe(true);
|
||||
expect(map.size).toBe(1);
|
||||
expect(map.conflicts).toBe(2);
|
||||
});
|
||||
|
||||
it("ignores a pair of equal values", () => {
|
||||
const map = new IdMap();
|
||||
map.learn("same", "same");
|
||||
expect(map.size).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("pairElements", () => {
|
||||
it("pairs objects by the shared unique string field that pairs the most", () => {
|
||||
const live = [
|
||||
{ id: 102, name: "Second", kind: "book" },
|
||||
{ id: 101, name: "First", kind: "book" },
|
||||
];
|
||||
const env = [
|
||||
{ id: 1, name: "First", kind: "book" },
|
||||
{ id: 2, name: "Second", kind: "book" },
|
||||
{ id: 3, name: "Third", kind: "book" },
|
||||
];
|
||||
expect(pairElements(live, env)).toEqual([
|
||||
[0, 1],
|
||||
[1, 0],
|
||||
]);
|
||||
});
|
||||
|
||||
it("pairs by position without a shared key or for scalar items", () => {
|
||||
expect(pairElements([{ n: 1 }, { n: 2 }], [{ n: 3 }])).toEqual([[0, 0]]);
|
||||
expect(pairElements(["a", "b"], ["c", "d"])).toEqual([
|
||||
[0, 0],
|
||||
[1, 1],
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("learnIds", () => {
|
||||
it("maps identifier fields of paired list items and of two single answers", () => {
|
||||
const map = new IdMap();
|
||||
learnIds(
|
||||
[
|
||||
{ id: 102, name: "Second", owner_id: "u-9" },
|
||||
{ id: 101, name: "First", owner_id: "u-9" },
|
||||
],
|
||||
[
|
||||
{ id: 1, name: "First", owner_id: "u-1" },
|
||||
{ id: 2, name: "Second", owner_id: "u-1" },
|
||||
],
|
||||
map,
|
||||
new Set(),
|
||||
);
|
||||
expect([map.env(101), map.env(102), map.env("u-9")]).toEqual([1, 2, "u-1"]);
|
||||
learnIds(
|
||||
{ data: { uid: "x9", ref: "r-1" } },
|
||||
{ data: { uid: "e3", ref: "r-2" } },
|
||||
map,
|
||||
new Set(["r-2"]),
|
||||
);
|
||||
expect(map.env("r-1")).toBe("r-2");
|
||||
expect(map.env("x9")).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("translateValue", () => {
|
||||
it("replaces mapped live identifiers at any depth in the original value's type", () => {
|
||||
const map = new IdMap();
|
||||
map.learn(101, 1);
|
||||
map.learn("abc", "xyz");
|
||||
expect(
|
||||
translateValue({ ids: [101, "101", "abc"], name: "abc def", n: 5 }, map),
|
||||
).toEqual({ ids: [1, "1", "xyz"], name: "abc def", n: 5 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,155 @@
|
||||
import type { JsonObject } from "../../domain/analysisView/environmentStateRules.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
|
||||
/** Field names that hold an identifier: `id`, `uuid`, `guid`, or names ending in `id` or `_id`. */
|
||||
export const ID_FIELD = /^(id|uuid|guid)$|(_id|Id|ID|_uuid|Uuid)$/u;
|
||||
|
||||
const isObject = (value: JsonValue | undefined): value is JsonObject =>
|
||||
typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
|
||||
const isScalarId = (value: JsonValue | undefined): value is string | number =>
|
||||
(typeof value === "string" && value !== "") ||
|
||||
(typeof value === "number" && Number.isFinite(value));
|
||||
|
||||
/** Live identifier values and the environment's values they stand for, in both directions. */
|
||||
export class IdMap {
|
||||
readonly #toEnv = new Map<string, string | number>();
|
||||
readonly #toLive = new Map<string, string | number>();
|
||||
#conflicts = 0;
|
||||
|
||||
/** Pair a live value with an environment value; the first pairing of either side wins. */
|
||||
learn(live: string | number, env: string | number): void {
|
||||
const liveKey = String(live);
|
||||
const envKey = String(env);
|
||||
if (liveKey === envKey) return;
|
||||
const known = this.#toEnv.get(liveKey);
|
||||
if (known !== undefined) {
|
||||
if (String(known) !== envKey) this.#conflicts += 1;
|
||||
return;
|
||||
}
|
||||
if (this.#toLive.has(envKey)) {
|
||||
this.#conflicts += 1;
|
||||
return;
|
||||
}
|
||||
this.#toEnv.set(liveKey, env);
|
||||
this.#toLive.set(envKey, live);
|
||||
}
|
||||
|
||||
env(live: string | number): string | number | undefined {
|
||||
return this.#toEnv.get(String(live));
|
||||
}
|
||||
|
||||
hasLive(value: string | number): boolean {
|
||||
return this.#toEnv.has(String(value));
|
||||
}
|
||||
|
||||
hasEnv(value: string | number): boolean {
|
||||
return this.#toLive.has(String(value));
|
||||
}
|
||||
|
||||
get size(): number {
|
||||
return this.#toEnv.size;
|
||||
}
|
||||
|
||||
get conflicts(): number {
|
||||
return this.#conflicts;
|
||||
}
|
||||
}
|
||||
|
||||
/** A field every object element has as a non-empty string, unique on its side; not an identifier, which differs across sides. */
|
||||
const keyCandidates = (items: readonly JsonObject[]): Set<string> => {
|
||||
const candidates = new Set<string>();
|
||||
const first = items[0];
|
||||
if (first === undefined) return candidates;
|
||||
for (const key of Object.keys(first)) {
|
||||
if (ID_FIELD.test(key)) continue;
|
||||
const values = items.map((item) => item[key]);
|
||||
if (
|
||||
values.every((value) => typeof value === "string" && value !== "") &&
|
||||
new Set(values).size === values.length
|
||||
)
|
||||
candidates.add(key);
|
||||
}
|
||||
return candidates;
|
||||
};
|
||||
|
||||
/**
|
||||
* Pair two arrays' object elements by the natural key that pairs the most, as [live index, env index]; arrays of other
|
||||
* values, or with no shared key, pair by position up to the shorter length.
|
||||
*/
|
||||
export const pairElements = (
|
||||
live: readonly JsonValue[],
|
||||
env: readonly JsonValue[],
|
||||
): Array<readonly [number, number]> => {
|
||||
const byPosition = (): Array<readonly [number, number]> =>
|
||||
Array.from({ length: Math.min(live.length, env.length) }, (_, index) => [
|
||||
index,
|
||||
index,
|
||||
]);
|
||||
if (!live.every(isObject) || !env.every(isObject)) return byPosition();
|
||||
const liveItems = live as readonly JsonObject[];
|
||||
const envItems = env as readonly JsonObject[];
|
||||
const liveKeys = keyCandidates(liveItems);
|
||||
const envKeys = keyCandidates(envItems);
|
||||
let best:
|
||||
| { key: string; pairs: Array<readonly [number, number]> }
|
||||
| undefined;
|
||||
for (const key of [...liveKeys].sort()) {
|
||||
if (!envKeys.has(key)) continue;
|
||||
const envIndex = new Map(envItems.map((item, index) => [item[key], index]));
|
||||
const pairs: Array<readonly [number, number]> = [];
|
||||
for (const [index, item] of liveItems.entries()) {
|
||||
const match = envIndex.get(item[key]);
|
||||
if (match !== undefined) pairs.push([index, match]);
|
||||
}
|
||||
if (pairs.length > (best?.pairs.length ?? 0)) best = { key, pairs };
|
||||
}
|
||||
return best?.pairs ?? byPosition();
|
||||
};
|
||||
|
||||
/** Learn identifier pairs from the live and environment answers to one call: identifier fields of objects that pair, at any depth. */
|
||||
export const learnIds = (
|
||||
live: JsonValue | undefined,
|
||||
env: JsonValue | undefined,
|
||||
map: IdMap,
|
||||
heldIds: ReadonlySet<string>,
|
||||
): void => {
|
||||
if (Array.isArray(live) && Array.isArray(env)) {
|
||||
for (const [left, right] of pairElements(live, env))
|
||||
learnIds(live[left], env[right], map, heldIds);
|
||||
return;
|
||||
}
|
||||
if (!isObject(live) || !isObject(env)) return;
|
||||
for (const [key, value] of Object.entries(live)) {
|
||||
if (!Object.hasOwn(env, key)) continue;
|
||||
const other = env[key];
|
||||
if (
|
||||
isScalarId(value) &&
|
||||
isScalarId(other) &&
|
||||
(ID_FIELD.test(key) || heldIds.has(String(other)))
|
||||
)
|
||||
map.learn(value, other);
|
||||
else learnIds(value, other, map, heldIds);
|
||||
}
|
||||
};
|
||||
|
||||
/** A value with every string or number equal to a mapped live identifier replaced by the environment's value, in its own type. */
|
||||
export const translateValue = (value: JsonValue, map: IdMap): JsonValue => {
|
||||
if (Array.isArray(value))
|
||||
return value.map((item) => translateValue(item, map));
|
||||
if (isObject(value))
|
||||
return Object.fromEntries(
|
||||
Object.entries(value).map(([key, item]) => [
|
||||
key,
|
||||
translateValue(item, map),
|
||||
]),
|
||||
);
|
||||
if (!isScalarId(value)) return value;
|
||||
const env = map.env(value);
|
||||
if (env === undefined) return value;
|
||||
return typeof value === "string"
|
||||
? String(env)
|
||||
: typeof env === "number"
|
||||
? env
|
||||
: value;
|
||||
};
|
||||
@@ -0,0 +1,103 @@
|
||||
import { segmentRule } from "../../domain/analysisView/webNetworkRouteTemplates.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import { decodedPathSegment } from "./environmentReplay.js";
|
||||
import type {
|
||||
EnvironmentManifest,
|
||||
EnvironmentManifestTool,
|
||||
} from "./environmentSource.js";
|
||||
import { environmentToolArguments } from "./environmentTools.js";
|
||||
|
||||
const FORMAT_RULES: ReadonlySet<string> = new Set([
|
||||
"numeric",
|
||||
"uuid",
|
||||
"hex",
|
||||
"iso-date",
|
||||
]);
|
||||
|
||||
/** One request of a probe capture, as the comparison replays it. */
|
||||
export interface ProbeRequest {
|
||||
readonly method: string;
|
||||
readonly path: string;
|
||||
/** WHATWG query spelling including `?`, or null. */
|
||||
readonly query: string | null;
|
||||
readonly body: JsonValue | undefined;
|
||||
}
|
||||
|
||||
/** A probe request as a call of one manifest tool. */
|
||||
export interface ProbeToolCall {
|
||||
readonly tool: EnvironmentManifestTool;
|
||||
readonly args: Record<string, JsonValue>;
|
||||
/** Argument names of the path values, in path order. */
|
||||
readonly pathArguments: readonly string[];
|
||||
/** Query names the tool does not take; the call drops them. */
|
||||
readonly ignoredQuery: readonly string[];
|
||||
}
|
||||
|
||||
interface Fit {
|
||||
readonly tool: EnvironmentManifestTool;
|
||||
readonly literals: number;
|
||||
}
|
||||
|
||||
/** How a path fits a tool's template: the literal segments it matches, or undefined when it does not fit. */
|
||||
const fitOf = (
|
||||
tool: EnvironmentManifestTool,
|
||||
segments: readonly string[],
|
||||
): Fit | undefined => {
|
||||
const template = tool.path_template.slice(1).split("/");
|
||||
if (template.length !== segments.length) return undefined;
|
||||
const params = new Map(tool.params.map((param) => [param.index, param]));
|
||||
let literals = 0;
|
||||
for (const [index, part] of template.entries()) {
|
||||
const segment = segments[index]!;
|
||||
const param = params.get(index);
|
||||
if (param === undefined) {
|
||||
if (part !== segment && part !== decodedPathSegment(segment))
|
||||
return undefined;
|
||||
literals += 1;
|
||||
} else if (
|
||||
FORMAT_RULES.has(param.rule) &&
|
||||
segmentRule(segment) !== param.rule
|
||||
)
|
||||
return undefined;
|
||||
else if (segment === "") return undefined;
|
||||
}
|
||||
return { tool, literals };
|
||||
};
|
||||
|
||||
/** The tool of the request's method whose template fits the path with the most literal segments; the manifest's order breaks ties. */
|
||||
export const matchProbeRequest = (
|
||||
manifest: EnvironmentManifest,
|
||||
request: ProbeRequest,
|
||||
): ProbeToolCall | undefined => {
|
||||
const segments = request.path.slice(1).split("/");
|
||||
let best: Fit | undefined;
|
||||
for (const tool of manifest.tools) {
|
||||
if (tool.method !== request.method) continue;
|
||||
const fit = fitOf(tool, segments);
|
||||
if (
|
||||
fit !== undefined &&
|
||||
(best === undefined || fit.literals > best.literals)
|
||||
)
|
||||
best = fit;
|
||||
}
|
||||
if (best === undefined) return undefined;
|
||||
const { tool } = best;
|
||||
const names = environmentToolArguments(tool);
|
||||
const args: Record<string, JsonValue> = {};
|
||||
const pathArguments: string[] = [];
|
||||
for (const param of tool.params) {
|
||||
const name = names.path.get(param.name) ?? param.name;
|
||||
args[name] = decodedPathSegment(segments[param.index]!);
|
||||
pathArguments.push(name);
|
||||
}
|
||||
const ignoredQuery: string[] = [];
|
||||
if (request.query !== null)
|
||||
for (const [key, value] of new URLSearchParams(request.query.slice(1))) {
|
||||
const name = names.query.get(key);
|
||||
if (name === undefined) ignoredQuery.push(key);
|
||||
else if (!Object.hasOwn(args, name)) args[name] = value;
|
||||
}
|
||||
if (names.body !== undefined && request.body !== undefined)
|
||||
args[names.body] = request.body;
|
||||
return { tool, args, pathArguments, ignoredQuery };
|
||||
};
|
||||
@@ -155,7 +155,8 @@ export const parseEvidenceFile = (
|
||||
}
|
||||
};
|
||||
|
||||
const recordedExchange = (
|
||||
/** One capture record reduced to its request line and retained response body, or undefined without a request line. */
|
||||
export const recordedExchange = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
): RecordedExchange | undefined => {
|
||||
const line = readRecordedRequestLine(record.reported, true, () => true);
|
||||
@@ -214,7 +215,7 @@ export const parseViewEvidence = <Kind extends AnalysisViewResult["kind"]>(
|
||||
};
|
||||
|
||||
/** Authenticate the evidence file and require complete capture Evidence of either kind. */
|
||||
const parseCaptureEvidence = (
|
||||
export const parseCaptureEvidence = (
|
||||
operation: string,
|
||||
value: unknown,
|
||||
): Result<
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { basename } from "node:path";
|
||||
import { absoluteCallerPath } from "../filesystem/callerPath.js";
|
||||
import type { EnvironmentExportResult } from "../application/environment/environmentExport.js";
|
||||
import type { EnvironmentSource } from "../application/environment/environmentSource.js";
|
||||
@@ -299,6 +300,77 @@ const evidenceOption = z
|
||||
"Parent capture Evidence JSON path (inspect-web-network-capture or capture-browser-scenario output) the manifest cites",
|
||||
);
|
||||
|
||||
interface CompareArguments {
|
||||
readonly manifest: string;
|
||||
readonly evidence: string;
|
||||
readonly probes: readonly string[];
|
||||
}
|
||||
|
||||
/** Read the environment and every probe, then compare; a refusal is the command's output. */
|
||||
const compareEnvironmentCommand = async (
|
||||
input: CompareArguments,
|
||||
): Promise<JsonValue> => {
|
||||
const read = (path: string) =>
|
||||
parseCliJsonInput(path, CLI_COMMANDS.environmentCompare, undefined, {
|
||||
snapshot: true,
|
||||
});
|
||||
const manifest = await read(input.manifest);
|
||||
if (!manifest.ok) return manifest.error;
|
||||
const evidence = await read(input.evidence);
|
||||
if (!evidence.ok) return evidence.error;
|
||||
const probes = [];
|
||||
for (const path of input.probes) {
|
||||
const probe = await read(path);
|
||||
if (!probe.ok) return probe.error;
|
||||
probes.push({ label: basename(path), value: probe.value });
|
||||
}
|
||||
const { loadEnvironmentSource } =
|
||||
await import("../application/environment/environmentSource.js");
|
||||
const source = loadEnvironmentSource(
|
||||
manifest.value,
|
||||
evidence.value,
|
||||
CLI_COMMANDS.environmentCompare,
|
||||
);
|
||||
if (!source.ok) return cliError(source.error);
|
||||
const { compareEnvironment } =
|
||||
await import("../application/environment/environmentCompare.js");
|
||||
const report = compareEnvironment({
|
||||
source: source.value,
|
||||
captureValue: evidence.value,
|
||||
probes,
|
||||
operation: CLI_COMMANDS.environmentCompare,
|
||||
});
|
||||
return report.ok
|
||||
? (report.value as unknown as JsonValue)
|
||||
: cliError(report.error);
|
||||
};
|
||||
|
||||
/** Register the comparison of an environment with probe captures of its application. */
|
||||
const registerCompareCommand = (cli: CliInstance, logger: Logger): void => {
|
||||
cli.command(CLI_COMMANDS.environmentCompare, {
|
||||
description:
|
||||
"Compare an environment with probe captures of the application it was captured from: each probe is one call sequence recorded against a freshly reset application, answered by the environment from a reset state, with agreement reported per step, tool and probe",
|
||||
args: z.object({ manifest: manifestArgument }),
|
||||
options: z.object({
|
||||
evidence: evidenceOption,
|
||||
probe: z
|
||||
.array(z.string())
|
||||
.min(1)
|
||||
.describe(
|
||||
"Probe capture Evidence JSON path (inspect-web-network-capture or capture-browser-scenario output) of one call sequence made against a freshly reset application; repeat once per sequence",
|
||||
),
|
||||
}),
|
||||
run: ({ args, options }) =>
|
||||
logCliCommand(logger, CLI_COMMANDS.environmentCompare, () =>
|
||||
compareEnvironmentCommand({
|
||||
manifest: args.manifest,
|
||||
evidence: options.evidence,
|
||||
probes: options.probe,
|
||||
}),
|
||||
),
|
||||
});
|
||||
};
|
||||
|
||||
/** Register the one-shot bundle export command. */
|
||||
const registerExportCommand = (cli: CliInstance, logger: Logger): void => {
|
||||
cli.command(CLI_COMMANDS.environmentExport, {
|
||||
@@ -347,6 +419,7 @@ export const registerEnvironmentCommands = (
|
||||
environment: Readonly<NodeJS.ProcessEnv>,
|
||||
): void => {
|
||||
registerExportCommand(cli, logger);
|
||||
registerCompareCommand(cli, logger);
|
||||
cli.command(CLI_COMMANDS.environmentServe, {
|
||||
description:
|
||||
"Serve an environment manifest as an agent environment over streamable HTTP until SIGINT or SIGTERM: read tools, and write tools whose effect the capture confirmed, answer from the state the capture observed, derived at startup, or every tool replays retained capture records with --replay-only or when the state is unavailable",
|
||||
|
||||
@@ -128,6 +128,7 @@ export const CLI_COMMANDS = Object.freeze({
|
||||
evidenceExport: "evidence-export",
|
||||
environmentServe: "environment-serve",
|
||||
environmentExport: "environment-export",
|
||||
environmentCompare: "environment-compare",
|
||||
compare: "compare",
|
||||
captureProcess: "capture-process",
|
||||
compareProcessCaptures: "compare-process-captures",
|
||||
|
||||
Reference in new issue
Block a user