mirror of
https://github.com/morluto/rea.git
synced 2026-10-26 17:35:11 +00:00
feat(environment): compare an environment with probe captures of its application
Add environment-compare against probe captures of the served application, keeping the rebased write effects and main's state-file ownership. Squashed rebase of #5699 onto the rebased #5600 tip.
This commit is contained in:
Cursor Agent committed
1 parent
ad8f674d22
commit
12d2bc3cb2
21 files changed
+2881
-27
No files matched your search
@@ -696,6 +696,68 @@ state is unavailable, and that neither is emulation and served
|
||||
content is untrusted data. It states that the recorded content belongs to the
|
||||
captured application and must not be published without the rights to it.
|
||||
|
||||
## Measure fidelity against the application
|
||||
|
||||
An environment agrees with its own capture by construction. What matters to an agent is how it answers the calls the
|
||||
capture did not make, what it shows after a write, and how it fails. `rea environment-compare` measures that against the
|
||||
application itself:
|
||||
|
||||
```sh
|
||||
rea environment-compare manifest.json --evidence capture.json \
|
||||
--probe probe-crud.json --probe probe-reads.json --json
|
||||
```
|
||||
|
||||
Each `--probe` is capture Evidence (`inspect-web-network-capture` of a HAR, or `capture-browser-scenario`) of one call
|
||||
sequence made against the live application, seeded as it was for the capture and reset before the sequence. The command
|
||||
makes no network call: whoever records the probes drives the application with its own credentials. For each probe it
|
||||
resets the environment's state, takes the requests in the order they were made (record order when any lacks a start time), matches each to the manifest tool of the
|
||||
same method whose path template fits with the most literal segments (a format-rule parameter must fit its format once
|
||||
percent-decoded),
|
||||
answers it as the served environment would, and compares that answer with the application's.
|
||||
|
||||
A re-seeded application holds the same entities under other ids, so the comparison maps them. Objects in the two answers
|
||||
to one call pair by a natural key, a string field unique on each side that both share, such as a name or a slug; their
|
||||
identifier fields (`id`, `uuid`, `guid`, and names ending in `_id`, `Id`, `ID`, `_uuid` or `Uuid`), and those of two single
|
||||
answers such as a create's, map the live value to the environment's. Ids are learned only from an answer the environment
|
||||
gave that call itself, from its state or a recording of that very call, request line and body, never from another call's
|
||||
recording replayed in its place, and the answer's own `id`, `uuid` or `guid` also maps within the collection the call names, since
|
||||
two collections can reuse a live id. A live id in a later request's path, query or body is translated before the
|
||||
environment answers, a path parameter by the collection it names, except a query value that pages a list, such as `limit`
|
||||
or `page`. Each call is checked against the served tool's input schema first, as the MCP server checks it. After mapping, two clock values compare by format only (ISO-8601 strings, and epoch numbers under
|
||||
time-named fields), and two different values each side generated on its own compare equal: both UUIDs, or one length of
|
||||
eight or more with the same character classes and a digit. The report counts each kind of normalised value.
|
||||
|
||||
A step agrees when the statuses and the normalised bodies are equal. The report gives agreement overall, per probe and
|
||||
per tool, `agreement_ignoring_order` for steps that differ only in item order, and each step's differences, at most 20,
|
||||
with a JSON pointer and both values cut short:
|
||||
|
||||
| Category | Meaning |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| `status` | The statuses differ |
|
||||
| `missing-field` | The application's object has a member the environment's lacks |
|
||||
| `extra-field` | The environment's object has a member the application's lacks |
|
||||
| `value` | Values differ, or one is an object or array and the other is not |
|
||||
| `length` | Arrays differ in length, or not every item pairs |
|
||||
| `order` | Paired items come in a different order |
|
||||
| `clock-format` | Both are clocks, in different formats |
|
||||
| `refused` | The environment refused the call |
|
||||
| `not-served` | The request's tool is one the environment does not offer, such as a withheld write |
|
||||
| `no-tool` | No manifest tool matches the request |
|
||||
| `unmapped-id` | The application answered with success and the environment did not, for a path id the comparison never mapped |
|
||||
|
||||
What a probe did not observe is not counted as the application's behaviour. A record without a usable request line,
|
||||
such as a redacted URL or a transaction the capture cannot attribute, is not replayed, nor is a request to another
|
||||
origin than the application's (`other-origin`: the probe's origin with the most JSON-like records, as the manifest
|
||||
chooses its own), and a request with no recorded
|
||||
answer is made, since it may have changed the application, but not compared: each probe lists them under `unobserved` with the reason, and the summary counts them. A live
|
||||
body the probe did not record compares by status only and is counted as `unobserved_bodies`, unless the answer never
|
||||
carries one (204, 205, 304, `HEAD`). A browser probe captured without `capture.network.response_body` is refused, as
|
||||
is one that makes POST, PUT or PATCH calls without `capture.network.request_body`.
|
||||
|
||||
Agreement covers only the calls the probes make, after the mapping the answers allow. Values compared by form, clocks
|
||||
and generated values, can still differ in content, and an equal answer after a write shows the state matched for that
|
||||
read, not that the write had no other effect.
|
||||
|
||||
## Compatibility verification
|
||||
|
||||
`npm run verify:agentenv:environment` is the hosted `real-agentenv` lane
|
||||
|
||||
+16
@@ -456,6 +456,22 @@ of the serve command's stopped document. See
|
||||
[agent environments](agent-environments.md#export-a-bundle) for the folder
|
||||
layout and what the smoke task checks.
|
||||
|
||||
## Compare an environment with its application
|
||||
|
||||
`rea environment-compare <manifest.json> --evidence <capture.json> --probe <probe.json> [--probe <probe.json> ...]`
|
||||
compares the environment the pair serves with probe captures of the application it came from. Each probe is capture
|
||||
Evidence of one call sequence recorded against the application after a reset; the environment's state is reset before
|
||||
each probe, every request to the application's origin is answered as the served environment answers it, and the
|
||||
answer is compared with the application's after ids are mapped and clocks and generated values are normalised. The command reads files only and
|
||||
makes no network call. It prints the report and exits `0`, or prints a structured `invalid_request` error and exits `1`
|
||||
for the same pair refusals as `environment-serve`, a probe that is not capture Evidence, or a browser probe captured
|
||||
without response bodies, or without the request bodies its writes need. See
|
||||
[agent environments](agent-environments.md#measure-fidelity-against-the-application) for the report.
|
||||
|
||||
```sh
|
||||
rea environment-compare manifest.json --evidence capture.json --probe probe.json --json
|
||||
```
|
||||
|
||||
## Output and exit status
|
||||
|
||||
`rea mcp doctor --json` checks production MCP startup and reports child logs in
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
# Plan
|
||||
|
||||
1. Export the capture parser from `environmentSource.ts` so a probe loads like a capture.
|
||||
2. `environmentCompareMatch.ts`: a probe request to a tool and its arguments.
|
||||
3. `environmentCompareIds.ts`: the id map, natural-key pairing, learning from two answers, translating arguments.
|
||||
4. `environmentCompareDiff.ts`: normalisation and categorised differences.
|
||||
5. `environmentCompare.ts`: per probe, reset the state, answer each request, compare and learn; the report.
|
||||
6. `rea environment-compare` in `src/cli/environmentCommands.ts`, documented in `docs/agent-environments.md` and
|
||||
`docs/cli.md`.
|
||||
7. Tests: matching, pairing, normalisation and categories as unit tests; the command over a fixture capture and probes.
|
||||
8. Verification on VMs: six applications brought up and seeded as for their capture, reset from a snapshot per sequence,
|
||||
driven by a script that records each sequence as a HAR; baseline reports per environment.
|
||||
@@ -0,0 +1,75 @@
|
||||
# Compare an environment with the application it came from
|
||||
|
||||
Issue: https://github.com/morluto/rea/issues/4933
|
||||
|
||||
## Outcome
|
||||
|
||||
An environment is only as useful as it is faithful: an agent trained or graded on it should meet the behaviour of the
|
||||
application it was captured from. REA checks an environment only against its own capture, so nothing says how far it
|
||||
drifts from the application on calls the capture did not make, after writes, or on errors.
|
||||
|
||||
`rea environment-compare` measures that. It takes an environment (a manifest and its capture Evidence) and one or more
|
||||
**probe captures**, each a recording of one call sequence made against the live application, and reports how many of the
|
||||
probe's calls the environment answers as the application did, which tools disagree and how.
|
||||
|
||||
The command only reads Evidence: it makes no network call. Whoever records the probes drives the application with its own
|
||||
credentials and resets it between sequences.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Inputs: the manifest Evidence, its capture Evidence (`--evidence`), and one or more probe capture Evidence files
|
||||
(`--probe`, repeatable), each `inspect-web-network-capture` or `capture-browser-scenario` Evidence of one sequence.
|
||||
- Each probe is one sequence run against a freshly reset application. Before each probe the environment's state is reset,
|
||||
so sequences are independent on both sides.
|
||||
- Requests are taken in the probe's order (start time, then ordinal; ordinal alone when any record lacks a start
|
||||
time, as state extraction orders them). A request is matched to the manifest tool of the
|
||||
same method whose path template it fits with the most literal segments; format-rule parameters must fit their format
|
||||
once percent-decoded, as replay reads them.
|
||||
Path values, query values the tool takes and a JSON body become the tool's arguments.
|
||||
- A record without a usable request line (a redacted URL, a transaction the capture cannot attribute, an unparseable
|
||||
URL or no method) is not replayed, nor is a request to another origin than the probe's application (`other-origin`),
|
||||
which is the origin with the most JSON-like records, as the manifest chooses its own. A request with no recorded
|
||||
answer is made, since it may have changed the application, but not compared. Each is listed per probe with its
|
||||
reason and left out of agreement.
|
||||
- A request no tool matches is reported `no-tool`; a matched tool the environment does not offer, such as a write it
|
||||
withholds, is reported `not-served`. Neither is answered.
|
||||
- Ids differ between a re-seeded application and the capture. The comparison maps them:
|
||||
- seeded entities pair across the two answers by a natural key: a field present on every element of both arrays whose
|
||||
string values are unique on each side and shared between them;
|
||||
- identifier fields (`id`, `uuid`, `guid`, and names ending in `_id`, `Id`, `ID`, `_uuid` or `Uuid`) of paired objects,
|
||||
and of the two answers to the same call, map the live value to the environment's value; a create's two answers map
|
||||
the new ids. The `id`, `uuid` or `guid` of the answer's own entities (at the read view's entity or list location, a
|
||||
write's response location, or else the root) also maps within the collection the call names,
|
||||
since two collections can reuse a live id for different entities. Outside identifier fields, a string maps when the
|
||||
environment's state holds it as an id, and the keys of two objects keyed by ids map when their values pair (keys
|
||||
that partly overlap only when they are digits, UUIDs or their entries' identities, and every value pairs);
|
||||
- ids are learned only from an answer the environment gave this call, from its state or from a recording of exactly
|
||||
this call, its whole request line and its request body: a recording of another call that replay answers with
|
||||
instead holds other entities;
|
||||
- path, query and body values equal to a mapped live id are translated before the environment answers, a path
|
||||
parameter by the collection it names, except query values that page a list (`limit`, `offset`, `page` and the
|
||||
like), which count items rather than name one.
|
||||
A path value the map does not know, and the environment's state does not hold, is reported `unmapped-id`.
|
||||
- A call is checked against the served tool's input schema first, as the MCP server checks it; a call it refuses is
|
||||
reported `refused`. A browser probe keeps its request bodies; one captured without response bodies is refused, since
|
||||
only statuses would compare, and so is one that makes POST, PUT or PATCH calls without recorded request bodies, since
|
||||
those calls would be made empty.
|
||||
- Answers compare after normalisation: empty response text equals no body (a JSON empty string stays one); mapped ids
|
||||
compare equal, in scalar sets as well, an entity's own identity within the call's collection and any other id by the
|
||||
probe's mapping; two clock values (ISO-8601 strings, or epoch numbers under
|
||||
time-named fields) compare by format only; two different generated values (both UUIDs, or both of one length and the
|
||||
same character classes with a digit) that neither side maps compare equal and are counted. A body over the replay
|
||||
limit, text or binary, compares by status only, as does a live body the probe did not record, unless the answer has
|
||||
none by definition (204, 205, 304, or an answer to `HEAD`). Both are counted.
|
||||
- A step agrees when the statuses are equal and the normalised bodies are. Differences are categorised: `status`,
|
||||
`missing-field` (the application had it), `extra-field`, `value`, `length`, `order`, `clock-format`, `refused` (the
|
||||
environment refused the call), `not-served`, `no-tool` and `unmapped-id`, each with a JSON pointer and short values.
|
||||
- The report gives agreement overall, per probe and per tool, agreement ignoring item order beside it, the counts per
|
||||
category, the ids mapped and the values normalised, the records left uncompared, and each step's differences (at most
|
||||
20 per step).
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No network access, no credentials, no reset of the application: probes are recorded outside REA.
|
||||
- No generation of call sequences; a later change generates them where the environment disagrees.
|
||||
- No change to how environments serve.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Tasks
|
||||
|
||||
- [x] Record the outcome, requirements and non-goals.
|
||||
- [x] Load probes as captures.
|
||||
- [x] Match probe requests to tools.
|
||||
- [x] Map ids and translate arguments.
|
||||
- [x] Normalise and categorise differences.
|
||||
- [x] Compare per probe and report.
|
||||
- [x] Add the command and its documentation.
|
||||
- [x] Add unit and command tests.
|
||||
- [x] Run the gates.
|
||||
- [x] Measure six environments against their live applications and record the results.
|
||||
@@ -0,0 +1,146 @@
|
||||
# Verification
|
||||
|
||||
Verified commit: 235b22f2 on the write-plan branch at 7056f5ff. Executed on macOS with Node v22.23.1 (the repository
|
||||
pins 24.18.0 in `.nvmrc`, which was not installed); the live applications ran on six disposable Linux VMs.
|
||||
|
||||
## Gates
|
||||
|
||||
- `npm run build:cached`, `npm run check:fast` (typecheck and lint), `npm run format:check`, `npx knip`,
|
||||
`npm run scan:todos`, `npm run docs:check`, `npm run verify:test-discovery` and `npm run mcp-catalog:check`: passed.
|
||||
- `npm run jscpd`: 3.46% duplicated lines over 3609 files, under the 3.5% gate.
|
||||
- `npx vitest run --project domain --project services`: 7487 passed, 5 pre-existing skips.
|
||||
- `npm run test:prepare`, then `npm run test:focused --` the environment-compare, export and serve CLI boundaries, the
|
||||
serve MCP boundary and the tool schema validity boundary: 5 files, 30 passed.
|
||||
|
||||
New tests: matching, id mapping and translation, pairing guards, normalisation and every category as unit tests
|
||||
(`environmentCompareIds`, `environmentCompareDiff`, `environmentCompare`, 19 tests), and the command through the built
|
||||
CLI (`environmentCompareCli`, 2 tests): a probe of the books application re-seeded under other ids agrees on every step
|
||||
with 2 ids mapped, and a probe that is not capture Evidence is refused with its index.
|
||||
|
||||
## Six environments against their live applications
|
||||
|
||||
Six published environments were compared with the application each was captured from: NetBox 4.7.3, Komga 1.29.1,
|
||||
Mealie 3.28.0, n8n 1.123.85, Shlink 5.1.7 and self-hosted Sentry 26.9.0, the same images, brought up and seeded by the
|
||||
same scripts as for their captures, on loopback. Each was snapshotted after seeding and restored from that snapshot
|
||||
before every sequence (7 to 80 s), so every sequence met the same application.
|
||||
|
||||
A driver outside REA ran three kinds of sequence per application, each starting with the capture's read-only prefix:
|
||||
the capture re-run in order; per served write type, a list, create, read, update, read, delete, read and list, then a
|
||||
second create with changed free text; and a read of every seeded entity through its item tool. It translated the
|
||||
environment's ids to the application's as it learned them, signed in with the application's own credentials, and wrote
|
||||
each sequence as a HAR without headers or cookies, which `inspect-web-network-capture` turned into probe Evidence. The
|
||||
environments' manifests use `prefix-variance`, so the comparison ran on a local build of this branch with that rule's
|
||||
branch merged.
|
||||
|
||||
| Environment | Steps | Agreement | Ignoring order | Against its own capture |
|
||||
| ----------- | ----: | --------: | -------------: | ----------------------: |
|
||||
| NetBox | 370 | 99.2% | 99.2% | 100% |
|
||||
| Komga | 356 | 93.8% | 93.8% | 93.1% |
|
||||
| Mealie | 301 | 96.0% | 97.0% | 97.1% |
|
||||
| Shlink | 133 | 89.5% | 92.5% | 92.4% |
|
||||
| n8n | 202 | 74.8% | 87.6% | 87.5% |
|
||||
| Sentry | 230 | 23.9% | 46.5% | 84.7% |
|
||||
|
||||
The last column compares each environment with its own capture as the probe: what it cannot reproduce even of the
|
||||
session it was built from. The disagreements fall into four groups.
|
||||
|
||||
- Environment gaps, the ones to fix:
|
||||
- write tools the environment withholds, and the reads after them: readlists and collections (Komga), recipes (Mealie), short URLs
|
||||
(Shlink), workflow and tag writes (n8n);
|
||||
- preconditions: n8n refuses to delete a workflow that is not archived and the environment deletes it;
|
||||
- lists: a create does not appear in a list that keeps its recorded members (Komga), a filtered list is answered
|
||||
without the filter (Shlink), a replayed search ignores its body (Komga);
|
||||
- a domain-scoped short URL answered without its `domain` parameter, where Shlink answers 404;
|
||||
- state rendering: a project field two recorded views disagree on is rendered with one value, and an organisation
|
||||
read whose shape depends on a query parameter is answered the same either way (Sentry);
|
||||
- a tag read refused for want of the fields its item view always carried (Mealie);
|
||||
- time: per-hour event counts move with the current time, while the environment's clock is frozen at the capture
|
||||
(Sentry);
|
||||
- a field the capture's scrubbing replaced, answered as `"<scrubbed>"` (Sentry).
|
||||
- The application differing from itself between seedings: Sentry returns permission lists in no fixed order and
|
||||
reseeding changed release first-event times, feature lists and profile flags; n8n's executions and NetBox's running
|
||||
worker count differ from the captured instance.
|
||||
- The probe observing itself: Komga and Sentry report the driver's own sign-in user agent.
|
||||
- Driver gaps: ids the driver never mapped (Mealie units, Sentry events by hash) were sent as the environment's ids.
|
||||
|
||||
## What the runs fixed in the comparison
|
||||
|
||||
Each live run found a way the comparison itself was wrong, fixed with a test before the next run: requests are matched
|
||||
to tools in the environment's terms, so a tool whose path holds a seeded id literally matches; ids are learned outside
|
||||
identifier fields only from strings, after NetBox's worker count of 1 mapped a live 0 to entity 1; items paired only by
|
||||
position teach ids only when no key is shared and the lengths agree, after a filtered Shlink list mapped one short URL to
|
||||
another; objects keyed by ids pair their entries (n8n data tables); reordered scalar sets differ in order only and epoch
|
||||
numbers in arrays are clocks (Sentry); and bodies over the replay limit compare by status.
|
||||
|
||||
## After the review round
|
||||
|
||||
The branch merged the write-plan branch at b6c3b168 (four serving and gate fixes its review found), then changed the
|
||||
comparison in nine places: ids are learned only from answers to the call itself and also per collection, paging values
|
||||
are never translated, calls go through the served tools' input schemas, browser probes keep their request bodies, empty
|
||||
bodies and large binary bodies compare as the server answers them, scalar sets compare after mapping, and id-keyed
|
||||
objects pair when some keys overlap. Each new test fails on the previous head. The gates above passed again (domain and
|
||||
services 7504 passed with 5 pre-existing skips; the focused boundaries 80; conformance; `jscpd` 3.46%). Re-run on the
|
||||
same probes with this branch and the template rule's merged, the six environments compare exactly as in the table above,
|
||||
step for step, with the same ids mapped and the same values normalised.
|
||||
|
||||
A second round, on Greptile's review of those changes, learned ids from a replay only when the whole recorded line and
|
||||
request body match the call, compared an entity's own identity within the call's collection and other ids by the probe's
|
||||
mapping, read overlapping keys as ids only when they evidently are ids, and kept a JSON empty string apart from empty
|
||||
text. Its four tests fail on the previous head; the six environments again compare exactly as in the table.
|
||||
|
||||
A third round located the answer's own entities where the served environment does, at the read view's entity or list
|
||||
pointer or the write's response pointer, so an entity inside an envelope keeps its collection's mapping. Its test fails
|
||||
on the previous head; the gates passed again (domain and services 7509, boundaries 80, conformance, `jscpd` 3.46%) and
|
||||
the six environments again compare exactly as in the table.
|
||||
|
||||
A fourth round, on Greptile's next review, kept what a probe did not observe out of the agreement figures. Records
|
||||
without a usable request line, and requests without a recorded answer, are listed per probe with the reason and left
|
||||
uncompared. A live body the probe did not record compares by status only and is counted. A browser probe captured
|
||||
without response bodies is refused. A format-rule path parameter fits by its decoded value, as replay reads it. The
|
||||
branch also merged the write-plan branch at 1d1c4787, whose gate no longer confirms a write by a read of a state that a
|
||||
later write replaced. Each new test fails on the previous head. The gates passed again (domain and services
|
||||
7647, 5 pre-existing skips; the compare, export, serve and analysis-view boundaries 69; conformance 279; `jscpd` 3.44%).
|
||||
|
||||
On the same probes, the six environments have no unobserved records or bodies, and five compare exactly as before.
|
||||
Komga now withholds its collection create and patch. The only read the gate could check them by is the 404 after the
|
||||
collection is deleted, because the read after the patch is the patch's own observation. Its agreement falls from
|
||||
96.9% to 93.8%: the three creates and two patches the probes make are not served, and the six reads and deletes of
|
||||
those collections that follow fail. Against its own capture it falls from 96.6% to 93.1%. The table shows the new
|
||||
figures.
|
||||
|
||||
A fifth round merged the write-plan branch at dc5e2aad, which weighs an update's derived changes when a later write
|
||||
replaces it and resets collection spellings on reset or load. It also makes a request whose answer the probe lost,
|
||||
still uncompared, so that a later read sees its effect: a lost delete followed by a 404 now agrees. Its test fails on
|
||||
the previous head. The six environments compare as in the table.
|
||||
|
||||
A sixth round merged the write-plan branch at fe8ab584, whose gate erases the fields later writes and recorded
|
||||
outcomes overwrite from earlier write effects. The six environments again compare exactly as in the table.
|
||||
|
||||
A seventh round merged the write-plan branch at 3580d85c and main d8a9673f. That brought the gate's fix for overwritten
|
||||
document writes, which Greptile raised here against the write-plan code, among others. It also fixed two findings of
|
||||
its own:
|
||||
|
||||
- a probe with any undated record keeps its recorded order, as state extraction does;
|
||||
- a browser probe that makes POST, PUT or PATCH calls without recorded request bodies is refused.
|
||||
|
||||
Both tests fail on the previous head. The six environments compare exactly as in the table, on the probes and against
|
||||
their own captures.
|
||||
|
||||
An eighth round merged the write-plan branch at 43d98ea9 and 264ce410. They bring the gate's fixes for three findings
|
||||
Greptile raised here against the write-plan code: a passing read that shows nothing of a write leaves it pending, an
|
||||
update or delete finds its entity as an item read does, and a built document plan writes later calls at its own
|
||||
object. It also fixed one finding of its own: a request to another origin than the probe's application is neither
|
||||
made nor compared, so another server's `DELETE /api/books/2` no longer deletes the environment's book. Its test fails
|
||||
on the previous head. The six environments compare exactly as in the table, on the probes and against their own
|
||||
captures, with no unobserved records.
|
||||
|
||||
It then merged the write-plan branch at 0ae64400 and 6257de93: an update's identity member never takes a rule, a page
|
||||
size of zero bounds nothing, and the paging rules, with this branch's `isPagingParameter`, move to
|
||||
`environmentStatePaging.ts`. The six environments again compare exactly as in the table.
|
||||
|
||||
A ninth round merged the write-plan branch at 466ed8b4, which fixes two findings Greptile raised here against its
|
||||
code (entity fields that look like pointers match by name; a zero page size bounds nothing only where a recording
|
||||
listed items under it) among others. The six environments compare exactly as in the table, on the probes and against
|
||||
their own captures. It then merged 7cef2fb2, whose fixes settle three more findings Greptile raised here against that branch's code:
|
||||
zero page sizes erasing another bound, partial write responses taken for removals, and interference timed by a tool's
|
||||
call elsewhere. The six environments again compare exactly as in the table.
|
||||
@@ -0,0 +1,445 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import {
|
||||
analysisViewCaptureEvidence,
|
||||
environmentManifestCaptureFixture,
|
||||
timedCaptureEvidence,
|
||||
type TimedCaptureRow,
|
||||
} from "../../../tests/fixtures/analysisView.js";
|
||||
import {
|
||||
environmentServeSource,
|
||||
environmentStateServeFixture,
|
||||
} from "../../../tests/fixtures/environmentServe.js";
|
||||
import {
|
||||
BOOK_NOT_FOUND,
|
||||
BOOK_ROWS,
|
||||
FIRST_BOOK,
|
||||
SECOND_BOOK,
|
||||
book,
|
||||
renamedBook,
|
||||
} from "../../../tests/fixtures/environmentStateWrites.js";
|
||||
import {
|
||||
scenarioCaptureEvidence,
|
||||
scenarioCaptureFixture,
|
||||
scenarioHeaderOnlyCaptureEvidence,
|
||||
} from "../../../tests/fixtures/browserScenarioCapture.js";
|
||||
import { compareEnvironment } from "./environmentCompare.js";
|
||||
import { environmentArguments, IdMap } from "./environmentCompareIds.js";
|
||||
import { matchProbeRequest } from "./environmentCompareMatch.js";
|
||||
import { isCut } from "./environmentCompareProbe.js";
|
||||
import { parseCaptureEvidence } from "./environmentSource.js";
|
||||
|
||||
const fixture = environmentStateServeFixture(timedCaptureEvidence(BOOK_ROWS));
|
||||
const source = environmentServeSource(fixture);
|
||||
|
||||
/** The books application re-seeded: the same books under other ids and creation times. */
|
||||
const LIVE_FIRST = book(101, "First Book", "2026-10-20T09:00:00Z");
|
||||
const LIVE_SECOND = book(102, "Second Book", "2026-10-20T10:00:00Z");
|
||||
const LIVE_FOURTH = book(103, "Fourth Book", "2026-10-20T11:00:00Z");
|
||||
const LIVE_RENAMED = renamedBook(LIVE_FIRST, "Renamed Book", 5);
|
||||
const ANN = { id: 1, name: "Ann" };
|
||||
const BOB = { id: 2, name: "Bob" };
|
||||
|
||||
const compare = (...probes: (readonly TimedCaptureRow[])[]) => {
|
||||
const report = compareEnvironment({
|
||||
source,
|
||||
captureValue: fixture.capture,
|
||||
probes: probes.map((rows, index) => ({
|
||||
label: `probe-${String(index + 1)}`,
|
||||
value: timedCaptureEvidence(rows),
|
||||
})),
|
||||
operation: "environment-compare",
|
||||
});
|
||||
if (!report.ok) throw report.error;
|
||||
return report.value;
|
||||
};
|
||||
|
||||
describe("matchProbeRequest", () => {
|
||||
it("prefers literal segments, requires format-rule parameters to fit, and maps query and body arguments", () => {
|
||||
const item = matchProbeRequest(source.manifest, {
|
||||
method: "GET",
|
||||
path: "/api/books/7",
|
||||
query: "?expand=all",
|
||||
body: undefined,
|
||||
});
|
||||
expect(item?.tool.path_template).toBe("/api/books/{numeric}");
|
||||
expect(item?.args).toEqual({ id: "7" });
|
||||
expect(item?.ignoredQuery).toEqual(["expand"]);
|
||||
const create = matchProbeRequest(source.manifest, {
|
||||
method: "POST",
|
||||
path: "/api/books",
|
||||
query: null,
|
||||
body: { name: "X" },
|
||||
});
|
||||
expect(create?.args).toEqual({ body: { name: "X" } });
|
||||
expect(
|
||||
matchProbeRequest(source.manifest, {
|
||||
method: "GET",
|
||||
path: "/api/books/x",
|
||||
query: null,
|
||||
body: undefined,
|
||||
}),
|
||||
).toBeUndefined();
|
||||
});
|
||||
|
||||
it("fits a format-rule parameter by its decoded value, as replay matches it", () => {
|
||||
const item = matchProbeRequest(source.manifest, {
|
||||
method: "GET",
|
||||
path: "/api/books/%31",
|
||||
query: null,
|
||||
body: undefined,
|
||||
});
|
||||
expect(item?.tool.path_template).toBe("/api/books/{numeric}");
|
||||
expect(item?.args).toEqual({ id: "1" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("compareEnvironment", () => {
|
||||
it("agrees with a re-seeded application once seeded and created ids are mapped", () => {
|
||||
const report = compare([
|
||||
["GET", "/api/books", [LIVE_FIRST, LIVE_SECOND]],
|
||||
["GET", "/api/books/101", LIVE_FIRST],
|
||||
["POST", "/api/books", LIVE_FOURTH, 201, { name: "Fourth Book" }],
|
||||
["GET", "/api/books/103", LIVE_FOURTH],
|
||||
["PATCH", "/api/books/101", LIVE_RENAMED, 200, { name: "Renamed Book" }],
|
||||
["GET", "/api/books/101", LIVE_RENAMED],
|
||||
["DELETE", "/api/books/103", "", 204],
|
||||
["GET", "/api/books/103", BOOK_NOT_FOUND, 404],
|
||||
["GET", "/api/books", [LIVE_RENAMED, LIVE_SECOND]],
|
||||
]);
|
||||
expect(report.environment.state).toBe("served");
|
||||
expect(report.probes[0]?.results.filter((step) => !step.equal)).toEqual([]);
|
||||
expect(report.summary).toMatchObject({
|
||||
probes: 1,
|
||||
steps: 9,
|
||||
equal: 9,
|
||||
agreement: 1,
|
||||
});
|
||||
expect(report.probes[0]?.ids_mapped).toBe(3);
|
||||
expect(report.summary.normalised.ids).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("resets the state per probe and categorises every kind of disagreement", () => {
|
||||
const report = compare(
|
||||
[
|
||||
["GET", "/api/books", [LIVE_FIRST, LIVE_SECOND]],
|
||||
["DELETE", "/api/books/101", { detail: "Book is referenced." }, 409],
|
||||
["GET", "/api/authors", []],
|
||||
["GET", "/api/books/555", LIVE_FIRST],
|
||||
],
|
||||
[["GET", "/api/books", [LIVE_SECOND, LIVE_FIRST, LIVE_FOURTH]]],
|
||||
);
|
||||
const [first, second] = report.probes;
|
||||
expect(first?.results.map((step) => step.categories)).toEqual([
|
||||
[],
|
||||
["status", "value"],
|
||||
["no-tool"],
|
||||
["unmapped-id", "status", "missing-field", "extra-field"],
|
||||
]);
|
||||
expect(second?.results[0]?.categories).toEqual(["length", "order"]);
|
||||
expect(second?.results[0]?.equal_ignoring_order).toBe(false);
|
||||
expect(report.summary.agreement_ignoring_order).toBe(
|
||||
report.summary.agreement,
|
||||
);
|
||||
expect(report.summary.categories).toMatchObject({
|
||||
status: 2,
|
||||
"no-tool": 1,
|
||||
"unmapped-id": 1,
|
||||
length: 1,
|
||||
order: 1,
|
||||
});
|
||||
expect(report.tools.map((tool) => tool.tool)).toContain(
|
||||
"delete_books_by_id",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("what the comparison translates and learns", () => {
|
||||
/** The books capture with a paged, filtered list and reports the state does not model, so other reports replay this one. */
|
||||
const extra = environmentStateServeFixture(
|
||||
timedCaptureEvidence([
|
||||
...BOOK_ROWS,
|
||||
["GET", "/api/books?limit=2&author=7", [FIRST_BOOK, SECOND_BOOK]],
|
||||
["GET", "/api/reports/1", [{ owner_id: 7 }]],
|
||||
["GET", "/api/authors", [ANN, BOB]],
|
||||
["GET", "/api/authors/1", ANN],
|
||||
["GET", "/api/authors/2", BOB],
|
||||
["GET", "/api/ping", ""],
|
||||
["POST", "/api/books/search", [FIRST_BOOK], 200, { q: "First" }],
|
||||
["POST", "/api/tags/search", [{ id: 1 }], 200, { q: "first" }],
|
||||
["GET", "/api/motd", '""'],
|
||||
]),
|
||||
);
|
||||
const extraSource = environmentServeSource(extra);
|
||||
const compareExtra = (rows: readonly TimedCaptureRow[]) => {
|
||||
const report = compareEnvironment({
|
||||
source: extraSource,
|
||||
captureValue: extra.capture,
|
||||
probes: [{ label: "probe", value: timedCaptureEvidence(rows) }],
|
||||
operation: "environment-compare",
|
||||
});
|
||||
if (!report.ok) throw report.error;
|
||||
return report.value.probes[0]!;
|
||||
};
|
||||
|
||||
it("translates a path id by the collection it names when two collections reuse a live id", () => {
|
||||
// Live book 101 is the environment's book 1, while live author 101 is its author 2.
|
||||
const probe = compareExtra([
|
||||
["GET", "/api/books", [LIVE_FIRST, LIVE_SECOND]],
|
||||
[
|
||||
"GET",
|
||||
"/api/authors",
|
||||
[
|
||||
{ ...ANN, id: 102 },
|
||||
{ ...BOB, id: 101 },
|
||||
],
|
||||
],
|
||||
["GET", "/api/authors/101", { ...BOB, id: 101 }],
|
||||
]);
|
||||
expect(probe.results.map((step) => step.equal)).toEqual([true, true, true]);
|
||||
});
|
||||
|
||||
it("refuses arguments the served tool's input schema refuses, as the server would", () => {
|
||||
// A search replays its recording whatever it is sent, but the server refuses a body that is not an object.
|
||||
const [step] = compareExtra([
|
||||
["POST", "/api/books/search", [FIRST_BOOK], 200, ["First"]],
|
||||
]).results;
|
||||
expect(step?.categories).toEqual(["refused"]);
|
||||
expect(step?.differences[0]?.env).toContain("Invalid arguments");
|
||||
});
|
||||
|
||||
it("agrees on an empty response however each side spells it, and keeps a JSON empty string", () => {
|
||||
expect(compareExtra([["GET", "/api/ping", ""]]).results[0]?.equal).toBe(
|
||||
true,
|
||||
);
|
||||
expect(compareExtra([["GET", "/api/motd", '""']]).results[0]?.equal).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it("learns ids from a replayed recording only of the same call, body included", () => {
|
||||
expect(
|
||||
compareExtra([
|
||||
["POST", "/api/tags/search", [{ id: 102 }], 200, { q: "second" }],
|
||||
]).ids_mapped,
|
||||
).toBe(0);
|
||||
expect(
|
||||
compareExtra([
|
||||
["POST", "/api/tags/search", [{ id: 102 }], 200, { q: "first" }],
|
||||
]).ids_mapped,
|
||||
).toBe(1);
|
||||
});
|
||||
|
||||
it("compares a large binary body only by status, as it does large text", () => {
|
||||
expect(isCut({ kind: "base64", encoded: "AAAA", whole: false })).toBe(true);
|
||||
expect(isCut({ kind: "base64", encoded: "AAAA", whole: true })).toBe(false);
|
||||
expect(isCut({ kind: "cut", text: "a" })).toBe(true);
|
||||
});
|
||||
|
||||
it("translates mapped ids in query values except those that page a list", () => {
|
||||
const call = matchProbeRequest(extraSource.manifest, {
|
||||
method: "GET",
|
||||
path: "/api/books",
|
||||
query: "?limit=101&author=101",
|
||||
body: undefined,
|
||||
});
|
||||
const map = new IdMap();
|
||||
map.learn(101, 1);
|
||||
expect(call?.pagingArguments).toEqual(["limit"]);
|
||||
expect(call && environmentArguments(call, map)).toEqual({
|
||||
limit: "101",
|
||||
author: "1",
|
||||
});
|
||||
});
|
||||
|
||||
it("learns no ids from another call's recording replayed in place of an answer", () => {
|
||||
const report = compareEnvironment({
|
||||
source: extraSource,
|
||||
captureValue: extra.capture,
|
||||
probes: [
|
||||
{
|
||||
label: "reports",
|
||||
value: timedCaptureEvidence([
|
||||
["GET", "/api/reports/2", [{ owner_id: 107 }]],
|
||||
]),
|
||||
},
|
||||
],
|
||||
operation: "environment-compare",
|
||||
});
|
||||
if (!report.ok) throw report.error;
|
||||
const [probe] = report.value.probes;
|
||||
expect(probe?.ids_mapped).toBe(0);
|
||||
expect(probe?.results[0]?.differences).toEqual([
|
||||
{ category: "value", pointer: "/0/owner_id", live: "107", env: "7" },
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("browser probes", () => {
|
||||
it("keeps a browser probe's request bodies, which are its call arguments", () => {
|
||||
const post = (purpose?: "manifest") => {
|
||||
const parsed = parseCaptureEvidence(
|
||||
"environment-compare",
|
||||
scenarioCaptureEvidence(),
|
||||
purpose,
|
||||
);
|
||||
if (!parsed.ok) throw parsed.error;
|
||||
return parsed.value.records.find((record) =>
|
||||
JSON.stringify(record.reported).includes('"method":"POST"'),
|
||||
)?.reported;
|
||||
};
|
||||
expect(JSON.stringify(post("manifest"))).toContain("postData");
|
||||
expect(JSON.stringify(post())).not.toContain("postData");
|
||||
});
|
||||
});
|
||||
|
||||
describe("what a probe did not observe", () => {
|
||||
const at = (second: number): string =>
|
||||
new Date(
|
||||
Date.parse("2026-10-20T09:00:00.000Z") + second * 1000,
|
||||
).toISOString();
|
||||
const exchange = (
|
||||
second: number,
|
||||
method: string,
|
||||
url: string | null,
|
||||
status: number,
|
||||
) => ({
|
||||
startedDateTime: at(second),
|
||||
request: { method, url },
|
||||
response: { status, content: { mimeType: "application/json" } },
|
||||
});
|
||||
const run = (value: unknown) =>
|
||||
compareEnvironment({
|
||||
source,
|
||||
captureValue: fixture.capture,
|
||||
probes: [{ label: "probe-1", value }],
|
||||
operation: "environment-compare",
|
||||
});
|
||||
|
||||
it("compares an unrecorded body by status only and lists the records it could not compare", () => {
|
||||
const report = run(
|
||||
analysisViewCaptureEvidence(
|
||||
environmentManifestCaptureFixture([
|
||||
exchange(0, "GET", "http://app.example/api/books/1", 200),
|
||||
exchange(1, "GET", null, 200),
|
||||
{ startedDateTime: at(2) },
|
||||
exchange(3, "GET", "http://app.example/api/books", 0),
|
||||
// A 204 carries no body, so its missing body is known to be empty.
|
||||
exchange(4, "DELETE", "http://app.example/api/books/2", 204),
|
||||
]),
|
||||
),
|
||||
);
|
||||
if (!report.ok) throw report.error;
|
||||
expect(report.value.probes[0]?.unobserved).toEqual([
|
||||
{ ordinal: 1, reason: "url-redacted" },
|
||||
{ ordinal: 2, reason: "request-unavailable" },
|
||||
{ ordinal: 3, reason: "no-response" },
|
||||
]);
|
||||
expect(report.value.summary).toMatchObject({
|
||||
steps: 2,
|
||||
equal: 2,
|
||||
unobserved: 3,
|
||||
});
|
||||
expect(report.value.summary.normalised.unobserved_bodies).toBe(1);
|
||||
});
|
||||
|
||||
it("makes a request whose answer was lost, so later reads see its effect", () => {
|
||||
const report = run(
|
||||
analysisViewCaptureEvidence(
|
||||
environmentManifestCaptureFixture([
|
||||
exchange(0, "DELETE", "http://app.example/api/books/2", 0),
|
||||
{
|
||||
...exchange(1, "GET", "http://app.example/api/books/2", 404),
|
||||
response: {
|
||||
status: 404,
|
||||
content: {
|
||||
mimeType: "application/json",
|
||||
text: JSON.stringify(BOOK_NOT_FOUND),
|
||||
},
|
||||
},
|
||||
},
|
||||
]),
|
||||
),
|
||||
);
|
||||
if (!report.ok) throw report.error;
|
||||
expect(report.value.probes[0]?.unobserved).toEqual([
|
||||
{ ordinal: 0, reason: "no-response" },
|
||||
]);
|
||||
expect(report.value.summary).toMatchObject({ steps: 1, equal: 1 });
|
||||
});
|
||||
|
||||
it("makes no request to another origin, so it cannot change the environment", () => {
|
||||
const answered = (second: number, url: string, body: JsonValue) => ({
|
||||
...exchange(second, "GET", url, 200),
|
||||
response: {
|
||||
status: 200,
|
||||
content: { mimeType: "application/json", text: JSON.stringify(body) },
|
||||
},
|
||||
});
|
||||
const report = run(
|
||||
analysisViewCaptureEvidence(
|
||||
environmentManifestCaptureFixture([
|
||||
// The probe also called another server's API, at a path the application's tools match.
|
||||
exchange(0, "DELETE", "http://cdn.example/api/books/2", 204),
|
||||
answered(1, "http://app.example/api/books/1", FIRST_BOOK),
|
||||
answered(2, "http://app.example/api/books/2", SECOND_BOOK),
|
||||
]),
|
||||
),
|
||||
);
|
||||
if (!report.ok) throw report.error;
|
||||
expect(report.value.probes[0]?.unobserved).toEqual([
|
||||
{ ordinal: 0, reason: "other-origin" },
|
||||
]);
|
||||
expect(report.value.summary).toMatchObject({ steps: 2, equal: 2 });
|
||||
});
|
||||
|
||||
it("keeps the recorded order when a record has no start time, as state extraction does", () => {
|
||||
const { startedDateTime: _undated, ...read } = {
|
||||
...exchange(1, "GET", "http://app.example/api/books/2", 404),
|
||||
response: {
|
||||
status: 404,
|
||||
content: {
|
||||
mimeType: "application/json",
|
||||
text: JSON.stringify(BOOK_NOT_FOUND),
|
||||
},
|
||||
},
|
||||
};
|
||||
const report = run(
|
||||
analysisViewCaptureEvidence(
|
||||
environmentManifestCaptureFixture([
|
||||
exchange(0, "DELETE", "http://app.example/api/books/2", 204),
|
||||
read,
|
||||
]),
|
||||
),
|
||||
);
|
||||
if (!report.ok) throw report.error;
|
||||
expect(report.value.summary).toMatchObject({ steps: 2, equal: 2 });
|
||||
});
|
||||
|
||||
it("refuses a browser probe that makes writes without recorded request bodies", () => {
|
||||
const report = run(
|
||||
scenarioCaptureEvidence(
|
||||
scenarioCaptureFixture(undefined, {
|
||||
request_body: false,
|
||||
response_body: true,
|
||||
header_values: true,
|
||||
}),
|
||||
),
|
||||
);
|
||||
expect(report.ok).toBe(false);
|
||||
if (report.ok) return;
|
||||
expect(JSON.stringify(report.error)).toContain(
|
||||
"needs capture.network.request_body selected",
|
||||
);
|
||||
});
|
||||
|
||||
it("refuses a browser probe captured without response bodies", () => {
|
||||
const report = run(scenarioHeaderOnlyCaptureEvidence());
|
||||
expect(report.ok).toBe(false);
|
||||
if (report.ok) return;
|
||||
expect(JSON.stringify(report.error)).toContain(
|
||||
"probe-1: a browser probe needs capture.network.response_body selected",
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,487 @@
|
||||
import { fromJsonSchema } from "@modelcontextprotocol/server";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import {
|
||||
compareAnswers,
|
||||
DIFFERENCE_CATEGORIES,
|
||||
STEP_DIFFERENCE_LIMIT,
|
||||
type Answer,
|
||||
type Difference,
|
||||
type DifferenceCategory,
|
||||
type Normalised,
|
||||
} from "./environmentCompareDiff.js";
|
||||
import {
|
||||
environmentArguments,
|
||||
IdMap,
|
||||
learnIds,
|
||||
translatePath,
|
||||
} from "./environmentCompareIds.js";
|
||||
import {
|
||||
collectionScope,
|
||||
matchProbeRequest,
|
||||
parameterScopes,
|
||||
type ProbeToolCall,
|
||||
} from "./environmentCompareMatch.js";
|
||||
import {
|
||||
probeRequests,
|
||||
requestBody,
|
||||
sendsBodies,
|
||||
type ProbeStep,
|
||||
type UnobservedRecord,
|
||||
} from "./environmentCompareProbe.js";
|
||||
import {
|
||||
inputError,
|
||||
parseCaptureEvidence,
|
||||
replayedBody,
|
||||
type EnvironmentManifestTool,
|
||||
type EnvironmentSource,
|
||||
} from "./environmentSource.js";
|
||||
import {
|
||||
selectsRecordedLine,
|
||||
type ReplayArguments,
|
||||
type ReplayResult,
|
||||
} from "./environmentReplay.js";
|
||||
import { prepareServedState, servedCatalog } from "./environmentState.js";
|
||||
import type { StateAnswer } from "./environmentStateAnswer.js";
|
||||
import { EnvironmentStateStore } from "./environmentStateStore.js";
|
||||
import {
|
||||
environmentToolAnswer,
|
||||
type EnvironmentToolAnswer,
|
||||
} from "./environmentToolAnswer.js";
|
||||
import {
|
||||
environmentToolArguments,
|
||||
type EnvironmentToolDefinition,
|
||||
} from "./environmentTools.js";
|
||||
import { same } from "../../domain/analysisView/environmentStateRules.js";
|
||||
|
||||
/** One probe capture file: its label for the report and its parsed JSON. */
|
||||
export interface CompareProbeInput {
|
||||
readonly label: string;
|
||||
readonly value: unknown;
|
||||
}
|
||||
|
||||
/** One probe request, how the environment answered it, and how that differs from the application. */
|
||||
export interface CompareStep {
|
||||
readonly ordinal: number;
|
||||
readonly method: string;
|
||||
readonly path: string;
|
||||
readonly tool: string | null;
|
||||
readonly live_status: number | null;
|
||||
readonly env_status: number | null;
|
||||
readonly equal: boolean;
|
||||
/** Equal but for item order, which an application that returns sets in no fixed order cannot reproduce either. */
|
||||
readonly equal_ignoring_order: boolean;
|
||||
readonly categories: DifferenceCategory[];
|
||||
readonly differences: Difference[];
|
||||
readonly ignored_query: string[];
|
||||
}
|
||||
|
||||
export interface CompareProbeReport {
|
||||
readonly label: string;
|
||||
readonly evidence_id: string;
|
||||
readonly steps: number;
|
||||
readonly equal: number;
|
||||
readonly ids_mapped: number;
|
||||
readonly id_conflicts: number;
|
||||
readonly normalised: Normalised;
|
||||
readonly results: CompareStep[];
|
||||
/** Records left uncompared, in probe order, each with its reason. */
|
||||
readonly unobserved: UnobservedRecord[];
|
||||
}
|
||||
|
||||
export interface CompareReport {
|
||||
readonly status: "complete";
|
||||
readonly environment: {
|
||||
readonly name: string;
|
||||
readonly manifest_evidence_id: string;
|
||||
readonly evidence_id: string;
|
||||
readonly state: "served" | "replay-only";
|
||||
};
|
||||
readonly summary: {
|
||||
readonly probes: number;
|
||||
readonly steps: number;
|
||||
readonly equal: number;
|
||||
readonly agreement: number | null;
|
||||
readonly agreement_ignoring_order: number | null;
|
||||
/** Probe records left uncompared, outside the steps agreement counts. */
|
||||
readonly unobserved: number;
|
||||
readonly categories: Record<DifferenceCategory, number>;
|
||||
readonly normalised: Normalised;
|
||||
};
|
||||
readonly tools: { tool: string; steps: number; equal: number }[];
|
||||
readonly probes: CompareProbeReport[];
|
||||
}
|
||||
|
||||
const NORMALISED_KEYS = [
|
||||
"ids",
|
||||
"clocks",
|
||||
"generated",
|
||||
"large_bodies",
|
||||
"unobserved_bodies",
|
||||
] as const;
|
||||
const emptyNormalised = (): Normalised => ({
|
||||
ids: 0,
|
||||
clocks: 0,
|
||||
generated: 0,
|
||||
large_bodies: 0,
|
||||
unobserved_bodies: 0,
|
||||
});
|
||||
|
||||
const isSuccess = (status: number | null): boolean =>
|
||||
status !== null && status >= 200 && status < 300;
|
||||
|
||||
/** Where a call's answer holds its own entities, and the collection they belong to: a read view's entity or list pointer, a write's response pointer, else the root. */
|
||||
const entityOf = (
|
||||
run: ProbeRun,
|
||||
tool: EnvironmentManifestTool,
|
||||
): { readonly scope: string; readonly at: string } => {
|
||||
const view = run.store?.state.views.get(tool.name);
|
||||
const plan = run.store?.state.writes.get(tool.name);
|
||||
const at =
|
||||
view?.shape === "item" || view?.shape === "list"
|
||||
? view.pointer
|
||||
: (plan?.response?.pointer ?? plan?.observation?.pointer ?? "");
|
||||
return { scope: collectionScope(tool), at };
|
||||
};
|
||||
|
||||
const fromState = (answer: ReplayResult): boolean =>
|
||||
(answer as Partial<StateAnswer>).state?.source === "state";
|
||||
|
||||
/** The environment's body as the application's is read: a replayed recording of empty text, unlike a JSON empty string, is no body. */
|
||||
const envBody = (run: ProbeRun, answer: ReplayResult): JsonValue => {
|
||||
if (answer.body !== "" || fromState(answer)) return answer.body;
|
||||
const exchange = run.source.exchanges.get(answer.replay.record_ordinal);
|
||||
return exchange !== undefined && replayedBody(exchange).kind === "json"
|
||||
? answer.body
|
||||
: null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Whether the environment answered this call itself: from its state, or from a recording of exactly this call, every
|
||||
* recorded query value supplied and the body it sent, rather than replaying another call's recording.
|
||||
*/
|
||||
const answersCall = (
|
||||
run: ProbeRun,
|
||||
call: ProbeToolCall,
|
||||
args: Readonly<Record<string, JsonValue>>,
|
||||
answer: ReplayResult,
|
||||
): boolean => {
|
||||
if (fromState(answer)) return true;
|
||||
const exchange = run.source.exchanges.get(answer.replay.record_ordinal);
|
||||
if (
|
||||
!answer.replay.exact_match ||
|
||||
exchange === undefined ||
|
||||
!selectsRecordedLine(call.tool, exchange, args)
|
||||
)
|
||||
return false;
|
||||
const body = environmentToolArguments(call.tool).body;
|
||||
const sent = body === undefined ? undefined : args[body];
|
||||
return (
|
||||
sent === undefined || same(run.recordedBodies.get(exchange.ordinal), sent)
|
||||
);
|
||||
};
|
||||
|
||||
/** A served tool's input check: the first problem with a call's arguments, or undefined. */
|
||||
type InputCheck = (args: ReplayArguments) => string | undefined;
|
||||
|
||||
/** The input schema check the MCP server applies before a served tool answers, so a call it refuses is not compared as answered. */
|
||||
const inputCheck = (definition: EnvironmentToolDefinition): InputCheck => {
|
||||
const schema = fromJsonSchema<ReplayArguments>(definition.inputSchema)[
|
||||
"~standard"
|
||||
];
|
||||
return (args) => {
|
||||
const checked = schema.validate(args);
|
||||
return checked instanceof Promise
|
||||
? undefined
|
||||
: checked.issues?.[0]?.message;
|
||||
};
|
||||
};
|
||||
|
||||
interface ProbeRun {
|
||||
readonly source: EnvironmentSource;
|
||||
/** The request body of each capture record, by ordinal, that a replayed answer's call is compared with. */
|
||||
readonly recordedBodies: ReadonlyMap<number, JsonValue | undefined>;
|
||||
readonly answer: EnvironmentToolAnswer;
|
||||
readonly store: EnvironmentStateStore | undefined;
|
||||
/** The tools the environment offers, by name, with their input checks. */
|
||||
readonly served: ReadonlyMap<string, InputCheck>;
|
||||
readonly map: IdMap;
|
||||
readonly normalised: Normalised;
|
||||
}
|
||||
|
||||
const heldIds = (
|
||||
store: EnvironmentStateStore | undefined,
|
||||
): ReadonlySet<string> =>
|
||||
new Set(
|
||||
store?.snapshot().rea_state.entities.map((entity) => entity.id) ?? [],
|
||||
);
|
||||
|
||||
const stepOf = (
|
||||
base: Pick<CompareStep, "ordinal" | "method" | "path" | "live_status">,
|
||||
call: ProbeToolCall | undefined,
|
||||
outcome: { env_status: number | null; differences: Difference[] },
|
||||
): CompareStep => {
|
||||
const categories = [...new Set(outcome.differences.map((d) => d.category))];
|
||||
return {
|
||||
...base,
|
||||
tool: call?.tool.name ?? null,
|
||||
env_status: outcome.env_status,
|
||||
equal: outcome.differences.length === 0,
|
||||
equal_ignoring_order: outcome.differences.every(
|
||||
(difference) => difference.category === "order",
|
||||
),
|
||||
categories,
|
||||
differences: outcome.differences.slice(0, STEP_DIFFERENCE_LIMIT),
|
||||
ignored_query: [...(call?.ignoredQuery ?? [])],
|
||||
};
|
||||
};
|
||||
|
||||
/** Answer one probe request as the environment would and compare it with the application's answer; one without an answer is only made. */
|
||||
const compareStep = (
|
||||
run: ProbeRun,
|
||||
manifest: EnvironmentSource["manifest"],
|
||||
probe: ProbeStep,
|
||||
): CompareStep | undefined => {
|
||||
const { request, live } = probe;
|
||||
const base = {
|
||||
ordinal: probe.ordinal,
|
||||
method: request.method,
|
||||
path: request.path,
|
||||
live_status: live?.status ?? null,
|
||||
};
|
||||
const report = (
|
||||
call: ProbeToolCall | undefined,
|
||||
outcome: { env_status: number | null; differences: Difference[] },
|
||||
): CompareStep | undefined =>
|
||||
live === undefined ? undefined : stepOf(base, call, outcome);
|
||||
// Matched in the environment's terms, so a tool whose path holds a seeded id literally still matches; each path
|
||||
// parameter translates by the collection it names, where the request already fits a tool.
|
||||
const fitted = matchProbeRequest(manifest, request);
|
||||
const scopes =
|
||||
fitted === undefined ? undefined : parameterScopes(fitted.tool);
|
||||
const call = matchProbeRequest(manifest, {
|
||||
...request,
|
||||
path: translatePath(request.path, run.map, (index) => scopes?.get(index)),
|
||||
});
|
||||
if (call === undefined)
|
||||
return report(call, {
|
||||
env_status: null,
|
||||
differences: [{ category: "no-tool", pointer: "" }],
|
||||
});
|
||||
const check = run.served.get(call.tool.name);
|
||||
if (check === undefined)
|
||||
return report(call, {
|
||||
env_status: null,
|
||||
differences: [{ category: "not-served", pointer: "" }],
|
||||
});
|
||||
const held = heldIds(run.store);
|
||||
const unknownPath = call.pathArguments.some((name) => {
|
||||
const value = call.args[name];
|
||||
return (
|
||||
typeof value === "string" && !run.map.hasEnv(value) && !held.has(value)
|
||||
);
|
||||
});
|
||||
const refusedWith = (message: string): CompareStep | undefined =>
|
||||
report(call, {
|
||||
env_status: null,
|
||||
differences: [
|
||||
{ category: "refused", pointer: "", env: message.slice(0, 200) },
|
||||
],
|
||||
});
|
||||
const args = environmentArguments(call, run.map);
|
||||
const problem = check(args);
|
||||
if (problem !== undefined)
|
||||
return refusedWith(`Invalid arguments: ${problem}`);
|
||||
const answered = run.answer(call.tool, args);
|
||||
if (!answered.ok) return refusedWith(answered.error.message);
|
||||
if (live === undefined) return undefined;
|
||||
const env: Answer = {
|
||||
status: answered.value.status,
|
||||
body: envBody(run, answered.value),
|
||||
};
|
||||
const entity = entityOf(run, call.tool);
|
||||
// Another call's recording holds other entities, whose ids would map wrongly and, the first pairing winning, for good.
|
||||
if (answersCall(run, call, args, answered.value))
|
||||
learnIds(live.body, env.body, run.map, {
|
||||
heldIds: heldIds(run.store),
|
||||
scope: entity.scope,
|
||||
entityAt: entity.at,
|
||||
});
|
||||
const compared = compareAnswers(live, env, run.map, entity);
|
||||
for (const key of NORMALISED_KEYS)
|
||||
run.normalised[key] += compared.normalised[key];
|
||||
const differences = [...compared.differences];
|
||||
if (unknownPath && isSuccess(live.status) && !isSuccess(env.status))
|
||||
differences.unshift({ category: "unmapped-id", pointer: "" });
|
||||
return stepOf(base, call, { env_status: env.status, differences });
|
||||
};
|
||||
|
||||
const probeReport = (
|
||||
probe: { readonly label: string; readonly evidenceId: string },
|
||||
run: ProbeRun,
|
||||
results: CompareStep[],
|
||||
unobserved: UnobservedRecord[],
|
||||
): CompareProbeReport => ({
|
||||
label: probe.label,
|
||||
evidence_id: probe.evidenceId,
|
||||
steps: results.length,
|
||||
equal: results.filter((step) => step.equal).length,
|
||||
ids_mapped: run.map.size,
|
||||
id_conflicts: run.map.conflicts,
|
||||
normalised: run.normalised,
|
||||
results,
|
||||
unobserved,
|
||||
});
|
||||
|
||||
const summaryOf = (
|
||||
probes: readonly CompareProbeReport[],
|
||||
): CompareReport["summary"] => {
|
||||
const steps = probes.flatMap((probe) => probe.results);
|
||||
const categories = Object.fromEntries(
|
||||
DIFFERENCE_CATEGORIES.map((category) => [category, 0]),
|
||||
) as Record<DifferenceCategory, number>;
|
||||
for (const step of steps)
|
||||
for (const category of step.categories) categories[category] += 1;
|
||||
const equal = steps.filter((step) => step.equal).length;
|
||||
const ratio = (count: number): number | null =>
|
||||
steps.length === 0
|
||||
? null
|
||||
: Math.round((count / steps.length) * 1000) / 1000;
|
||||
const normalised = emptyNormalised();
|
||||
for (const probe of probes)
|
||||
for (const key of NORMALISED_KEYS) normalised[key] += probe.normalised[key];
|
||||
return {
|
||||
probes: probes.length,
|
||||
steps: steps.length,
|
||||
equal,
|
||||
agreement: ratio(equal),
|
||||
agreement_ignoring_order: ratio(
|
||||
steps.filter((step) => step.equal_ignoring_order).length,
|
||||
),
|
||||
unobserved: probes.reduce((sum, probe) => sum + probe.unobserved.length, 0),
|
||||
categories,
|
||||
normalised,
|
||||
};
|
||||
};
|
||||
|
||||
const toolsOf = (
|
||||
probes: readonly CompareProbeReport[],
|
||||
): CompareReport["tools"] => {
|
||||
const tools = new Map<
|
||||
string,
|
||||
{ tool: string; steps: number; equal: number }
|
||||
>();
|
||||
for (const step of probes.flatMap((probe) => probe.results)) {
|
||||
if (step.tool === null) continue;
|
||||
const entry = tools.get(step.tool) ?? {
|
||||
tool: step.tool,
|
||||
steps: 0,
|
||||
equal: 0,
|
||||
};
|
||||
entry.steps += 1;
|
||||
if (step.equal) entry.equal += 1;
|
||||
tools.set(step.tool, entry);
|
||||
}
|
||||
return [...tools.values()].sort(
|
||||
(left, right) =>
|
||||
left.equal / left.steps - right.equal / right.steps ||
|
||||
(left.tool < right.tool ? -1 : 1),
|
||||
);
|
||||
};
|
||||
|
||||
/** Compare an environment with probe captures of the application: each probe from a reset state, every request answered and compared. */
|
||||
export const compareEnvironment = (input: {
|
||||
readonly source: EnvironmentSource;
|
||||
readonly captureValue: unknown;
|
||||
readonly probes: readonly CompareProbeInput[];
|
||||
readonly operation: string;
|
||||
}): Result<CompareReport, AnalysisError> => {
|
||||
const { source, operation } = input;
|
||||
const state = prepareServedState(source, input.captureValue, {
|
||||
operation,
|
||||
}).state;
|
||||
const catalog = servedCatalog(source, state, source.manifest.name, operation);
|
||||
if (!catalog.ok) return catalog;
|
||||
const served = new Map(
|
||||
catalog.value.tools.map((definition) => [
|
||||
definition.name,
|
||||
inputCheck(definition),
|
||||
]),
|
||||
);
|
||||
const recorded = parseCaptureEvidence(
|
||||
operation,
|
||||
input.captureValue,
|
||||
"manifest",
|
||||
);
|
||||
const recordedBodies = new Map(
|
||||
recorded.ok
|
||||
? recorded.value.records.map((record) => [
|
||||
record.ordinal,
|
||||
requestBody(record),
|
||||
])
|
||||
: [],
|
||||
);
|
||||
const probes: CompareProbeReport[] = [];
|
||||
for (const [index, probe] of input.probes.entries()) {
|
||||
// A probe's request bodies are its call arguments, which a replay projection of a browser capture drops.
|
||||
const capture = parseCaptureEvidence(operation, probe.value, "manifest");
|
||||
// Without response bodies only statuses would compare, and without request bodies writes would be made empty.
|
||||
const problem = !capture.ok
|
||||
? capture.error.message
|
||||
: !capture.value.responseBodies
|
||||
? "a browser probe needs capture.network.response_body selected, or only statuses would compare."
|
||||
: !capture.value.requestBodies && sendsBodies(capture.value.records)
|
||||
? "a browser probe that makes POST, PUT or PATCH calls needs capture.network.request_body selected, or those calls would be made without their bodies."
|
||||
: undefined;
|
||||
if (!capture.ok || problem !== undefined)
|
||||
return err(
|
||||
inputError(
|
||||
operation,
|
||||
[
|
||||
{
|
||||
path: ["probe", index],
|
||||
reason: "invalid_value",
|
||||
message: `${probe.label}: ${problem}`,
|
||||
},
|
||||
],
|
||||
"A probe file cannot be compared.",
|
||||
),
|
||||
);
|
||||
const store =
|
||||
state === undefined ? undefined : new EnvironmentStateStore(state);
|
||||
const run: ProbeRun = {
|
||||
source,
|
||||
recordedBodies,
|
||||
answer: environmentToolAnswer(source, store),
|
||||
store,
|
||||
served,
|
||||
map: new IdMap(),
|
||||
normalised: emptyNormalised(),
|
||||
};
|
||||
const { steps, unobserved } = probeRequests(capture.value.records);
|
||||
const results = steps.flatMap(
|
||||
(step) => compareStep(run, source.manifest, step) ?? [],
|
||||
);
|
||||
probes.push(
|
||||
probeReport(
|
||||
{ label: probe.label, evidenceId: capture.value.evidenceId },
|
||||
run,
|
||||
results,
|
||||
unobserved,
|
||||
),
|
||||
);
|
||||
}
|
||||
return ok({
|
||||
status: "complete",
|
||||
environment: {
|
||||
name: source.manifest.name,
|
||||
manifest_evidence_id: source.manifestEvidenceId,
|
||||
evidence_id: source.evidenceId,
|
||||
state: state === undefined ? "replay-only" : "served",
|
||||
},
|
||||
summary: summaryOf(probes),
|
||||
tools: toolsOf(probes),
|
||||
probes,
|
||||
});
|
||||
};
|
||||
@@ -0,0 +1,202 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import { compareAnswers } from "./environmentCompareDiff.js";
|
||||
import { IdMap, learnIds } from "./environmentCompareIds.js";
|
||||
|
||||
const compare = (
|
||||
live: unknown,
|
||||
env: unknown,
|
||||
map = new IdMap(),
|
||||
status: [number, number] = [200, 200],
|
||||
) =>
|
||||
compareAnswers(
|
||||
{ status: status[0], body: live as never },
|
||||
{ status: status[1], body: env as never },
|
||||
map,
|
||||
);
|
||||
|
||||
describe("compareAnswers", () => {
|
||||
it("agrees on mapped ids, clocks of one format and generated values, and counts each", () => {
|
||||
const map = new IdMap();
|
||||
map.learn(101, 1);
|
||||
const result = compare(
|
||||
{
|
||||
id: 101,
|
||||
created_at: "2026-10-20T10:00:00Z",
|
||||
token: "3f2a9c8d7e",
|
||||
uuid: "6f1c2b9e-2a3d-4f5e-8a9b-0c1d2e3f4a5b",
|
||||
},
|
||||
{
|
||||
id: 1,
|
||||
created_at: "2026-10-17T18:00:03Z",
|
||||
token: "9b8e7d6c5a",
|
||||
uuid: "0a1b2c3d-4e5f-4a6b-9c7d-8e9f0a1b2c3d",
|
||||
},
|
||||
map,
|
||||
);
|
||||
expect(result.differences).toEqual([]);
|
||||
expect(result.normalised).toEqual({
|
||||
ids: 1,
|
||||
clocks: 1,
|
||||
generated: 2,
|
||||
large_bodies: 0,
|
||||
unobserved_bodies: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it("categorises status, missing and extra fields, values and clock formats", () => {
|
||||
const result = compare(
|
||||
{ name: "A", gone: 1, at: "2026-10-20T10:00:00.123+00:00" },
|
||||
{ name: "B", added: true, at: "2026-10-20T10:00:00Z" },
|
||||
new IdMap(),
|
||||
[409, 204],
|
||||
);
|
||||
expect(result.differences.map((d) => [d.category, d.pointer])).toEqual([
|
||||
["status", ""],
|
||||
["value", "/name"],
|
||||
["missing-field", "/gone"],
|
||||
["clock-format", "/at"],
|
||||
["extra-field", "/added"],
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports array length and order and compares paired items", () => {
|
||||
const result = compare(
|
||||
[
|
||||
{ name: "B", n: 2 },
|
||||
{ name: "A", n: 1 },
|
||||
{ name: "C", n: 3 },
|
||||
],
|
||||
[
|
||||
{ name: "A", n: 1 },
|
||||
{ name: "B", n: 5 },
|
||||
],
|
||||
);
|
||||
expect(result.differences.map((d) => [d.category, d.pointer])).toEqual([
|
||||
["length", ""],
|
||||
["order", ""],
|
||||
["value", "/0/n"],
|
||||
]);
|
||||
});
|
||||
|
||||
it("treats an epoch number as a clock only under a time-named field", () => {
|
||||
expect(
|
||||
compare({ updated: 1760000000 }, { updated: 1760000999 }).differences,
|
||||
).toEqual([]);
|
||||
expect(
|
||||
compare({ size: 1760000000 }, { size: 1760000999 }).differences.map(
|
||||
(d) => d.category,
|
||||
),
|
||||
).toEqual(["value"]);
|
||||
});
|
||||
|
||||
it("compares an object keyed by mapped ids under the environment's keys", () => {
|
||||
const map = new IdMap();
|
||||
map.learn("liveTable", "envTable");
|
||||
expect(
|
||||
compare(
|
||||
{ tables: { liveTable: { name: "T" } } },
|
||||
{ tables: { envTable: { name: "T" } } },
|
||||
map,
|
||||
).differences,
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it("reports a reordered set of scalars as order alone and reads epoch array items as clocks", () => {
|
||||
expect(
|
||||
compare(
|
||||
{ access: ["a", "b", "c"] },
|
||||
{ access: ["c", "a", "b"] },
|
||||
).differences.map((d) => [d.category, d.pointer]),
|
||||
).toEqual([["order", "/access"]]);
|
||||
expect(
|
||||
compare({ stats: [[1792450800, 3]] }, { stats: [[1792285200, 3]] })
|
||||
.differences,
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it("compares a live body over the replay limit by status only and counts it", () => {
|
||||
const result = compareAnswers(
|
||||
{ status: 200, body: '{"data":[', cut: true },
|
||||
{ status: 200, body: null },
|
||||
new IdMap(),
|
||||
);
|
||||
expect(result.differences).toEqual([]);
|
||||
expect(result.normalised.large_bodies).toBe(1);
|
||||
});
|
||||
|
||||
it("compares a live body the probe did not record by status only and counts it", () => {
|
||||
const unrecorded = { status: 200, body: null, unobserved: true };
|
||||
const result = compareAnswers(
|
||||
unrecorded,
|
||||
{ status: 200, body: { id: 1 } },
|
||||
new IdMap(),
|
||||
);
|
||||
expect(result.differences).toEqual([]);
|
||||
expect(result.normalised.unobserved_bodies).toBe(1);
|
||||
expect(
|
||||
compareAnswers(unrecorded, { status: 404, body: null }, new IdMap())
|
||||
.differences,
|
||||
).toEqual([{ category: "status", pointer: "", live: "200", env: "404" }]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("scalar sets of mapped ids", () => {
|
||||
it("differ only in order once live ids are mapped", () => {
|
||||
const map = new IdMap();
|
||||
map.learn(101, 1);
|
||||
map.learn(102, 2);
|
||||
const compared = compareAnswers(
|
||||
{ status: 200, body: { ids: [101, 102] } },
|
||||
{ status: 200, body: { ids: [2, 1] } },
|
||||
map,
|
||||
);
|
||||
expect(compared.differences).toEqual([
|
||||
{ category: "order", pointer: "/ids" },
|
||||
]);
|
||||
expect(compared.normalised.ids).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ids two collections reuse", () => {
|
||||
it("compares an entity's own id within the call's collection and any other id by the probe's mapping", () => {
|
||||
const map = new IdMap();
|
||||
map.learn(101, 1, "/api/books");
|
||||
map.learn(101, 2, "/api/authors");
|
||||
const answer = (body: JsonValue) => ({ status: 200, body });
|
||||
expect(
|
||||
compareAnswers(answer({ id: 101 }), answer({ id: 2 }), map, {
|
||||
scope: "/api/authors",
|
||||
}).differences,
|
||||
).toEqual([]);
|
||||
expect(
|
||||
compareAnswers(answer({ book_id: 101 }), answer({ book_id: 2 }), map)
|
||||
.differences,
|
||||
).toEqual([
|
||||
{ category: "value", pointer: "/book_id", live: "101", env: "2" },
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("wrapped answers", () => {
|
||||
it("compares the identity of an entity inside an envelope within the call's collection", () => {
|
||||
const map = new IdMap();
|
||||
map.learn(101, 1, "/api/books");
|
||||
learnIds(
|
||||
{ data: [{ id: 101, name: "Bob" }] },
|
||||
{ data: [{ id: 2, name: "Bob" }] },
|
||||
map,
|
||||
{ heldIds: new Set(), scope: "/api/authors", entityAt: "/data" },
|
||||
);
|
||||
expect(map.env(101, "/api/authors")).toBe(2);
|
||||
expect(
|
||||
compareAnswers(
|
||||
{ status: 200, body: { data: { id: 101, name: "Bob" } } },
|
||||
{ status: 200, body: { data: { id: 2, name: "Bob" } } },
|
||||
map,
|
||||
{ scope: "/api/authors", at: "/data" },
|
||||
).differences,
|
||||
).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,315 @@
|
||||
import type { JsonObject } from "../../domain/analysisView/environmentStateRules.js";
|
||||
import {
|
||||
isUuid,
|
||||
readClock,
|
||||
type ClockFormat,
|
||||
} from "../../domain/analysisView/environmentStateWriteValues.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import {
|
||||
IDENTITY_FIELD,
|
||||
pairElements,
|
||||
translateValue,
|
||||
type IdMap,
|
||||
} from "./environmentCompareIds.js";
|
||||
|
||||
export const DIFFERENCE_CATEGORIES = [
|
||||
"status",
|
||||
"missing-field",
|
||||
"extra-field",
|
||||
"value",
|
||||
"length",
|
||||
"order",
|
||||
"clock-format",
|
||||
"refused",
|
||||
"not-served",
|
||||
"no-tool",
|
||||
"unmapped-id",
|
||||
] as const;
|
||||
export type DifferenceCategory = (typeof DIFFERENCE_CATEGORIES)[number];
|
||||
|
||||
/** One way an environment answer differs from the live one: where, and both values cut short. */
|
||||
export interface Difference {
|
||||
readonly category: DifferenceCategory;
|
||||
readonly pointer: string;
|
||||
readonly live?: string;
|
||||
readonly env?: string;
|
||||
}
|
||||
|
||||
/** Values the comparison treated as equal although they differ: mapped ids, clocks of one format, generated values. */
|
||||
export interface Normalised {
|
||||
ids: number;
|
||||
clocks: number;
|
||||
generated: number;
|
||||
/** Live bodies over the replay limit, compared by status only. */
|
||||
large_bodies: number;
|
||||
/** Live bodies the probe did not record, compared by status only. */
|
||||
unobserved_bodies: number;
|
||||
}
|
||||
|
||||
/** At most this many differences are kept per step. */
|
||||
export const STEP_DIFFERENCE_LIMIT = 20;
|
||||
const VALUE_CHARS = 120;
|
||||
const TIME_FIELD = /(at|time|date|timestamp|created|updated|modified)$/iu;
|
||||
|
||||
const isObject = (value: JsonValue | undefined): value is JsonObject =>
|
||||
typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
|
||||
const shown = (value: JsonValue | undefined): string => {
|
||||
const text = value === undefined ? "<absent>" : JSON.stringify(value);
|
||||
return text.length > VALUE_CHARS ? `${text.slice(0, VALUE_CHARS)}…` : text;
|
||||
};
|
||||
|
||||
const escapePointer = (key: string): string =>
|
||||
key.replaceAll("~", "~0").replaceAll("/", "~1");
|
||||
|
||||
const formatKey = (format: ClockFormat): string =>
|
||||
format.kind === "epoch"
|
||||
? `epoch-${format.unit}`
|
||||
: `iso${format.separator}${String(format.fraction)}${format.zone === "" ? "" : format.zone === "Z" ? "Z" : "offset"}`;
|
||||
|
||||
/** The format of a clock value: an ISO-8601 string, or an epoch number under a time-named field or as an array item. */
|
||||
const clockFormat = (
|
||||
value: JsonValue,
|
||||
key: string | undefined,
|
||||
): string | undefined => {
|
||||
if (typeof value === "number" && key !== undefined && !TIME_FIELD.test(key))
|
||||
return undefined;
|
||||
const clock = readClock(value);
|
||||
return clock === undefined ? undefined : formatKey(clock.format);
|
||||
};
|
||||
|
||||
const characterClasses = (text: string): string =>
|
||||
[
|
||||
...new Set(
|
||||
text
|
||||
.replaceAll(/\d/gu, "0")
|
||||
.replaceAll(/[a-z]/gu, "a")
|
||||
.replaceAll(/[A-Z]/gu, "A"),
|
||||
),
|
||||
]
|
||||
.sort()
|
||||
.join("");
|
||||
|
||||
/** Two values each side generated on its own: both UUIDs, or one length of eight or more with the same character classes and a digit. */
|
||||
const generatedPair = (live: string, env: string): boolean =>
|
||||
(isUuid(live) && isUuid(env)) ||
|
||||
(live.length === env.length &&
|
||||
live.length >= 8 &&
|
||||
/\d/u.test(live) &&
|
||||
/\d/u.test(env) &&
|
||||
characterClasses(live) === characterClasses(env));
|
||||
|
||||
class Differ {
|
||||
readonly differences: Difference[] = [];
|
||||
readonly normalised: Normalised = {
|
||||
ids: 0,
|
||||
clocks: 0,
|
||||
generated: 0,
|
||||
large_bodies: 0,
|
||||
unobserved_bodies: 0,
|
||||
};
|
||||
|
||||
/** The answer's own entities: where they sit (the object, or the array of them, at `at`) and the collection the call names, whose own mapping their identities compare by. */
|
||||
constructor(
|
||||
readonly map: IdMap,
|
||||
readonly entity: {
|
||||
readonly scope: string | undefined;
|
||||
readonly at: string;
|
||||
},
|
||||
) {}
|
||||
|
||||
add(difference: Difference): void {
|
||||
this.differences.push(difference);
|
||||
}
|
||||
|
||||
scalar(
|
||||
live: JsonValue,
|
||||
env: JsonValue,
|
||||
pointer: string,
|
||||
at: { readonly key?: string; readonly own?: boolean } = {},
|
||||
): void {
|
||||
if (live === env) return;
|
||||
const { key } = at;
|
||||
if (
|
||||
(typeof live === "string" || typeof live === "number") &&
|
||||
(typeof env === "string" || typeof env === "number")
|
||||
) {
|
||||
// An entity's own identity compares within the call's collection; any other id by the probe's mapping.
|
||||
const mapped = this.map.env(
|
||||
live,
|
||||
at.own === true && key !== undefined && IDENTITY_FIELD.test(key)
|
||||
? this.entity.scope
|
||||
: undefined,
|
||||
);
|
||||
if (mapped !== undefined && String(mapped) === String(env)) {
|
||||
this.normalised.ids += 1;
|
||||
return;
|
||||
}
|
||||
const liveClock = clockFormat(live, key);
|
||||
const envClock = clockFormat(env, key);
|
||||
if (liveClock !== undefined && envClock !== undefined) {
|
||||
if (liveClock === envClock) this.normalised.clocks += 1;
|
||||
else
|
||||
this.add({
|
||||
category: "clock-format",
|
||||
pointer,
|
||||
live: shown(live),
|
||||
env: shown(env),
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (
|
||||
typeof live === "string" &&
|
||||
typeof env === "string" &&
|
||||
!this.map.hasLive(live) &&
|
||||
!this.map.hasEnv(env) &&
|
||||
generatedPair(live, env)
|
||||
) {
|
||||
this.normalised.generated += 1;
|
||||
return;
|
||||
}
|
||||
}
|
||||
this.add({
|
||||
category: "value",
|
||||
pointer,
|
||||
live: shown(live),
|
||||
env: shown(env),
|
||||
});
|
||||
}
|
||||
|
||||
array(live: JsonValue[], env: JsonValue[], pointer: string): void {
|
||||
const own = pointer === this.entity.at;
|
||||
if (this.sameItems(live, env, pointer)) return;
|
||||
const pairs = pairElements(live, env);
|
||||
if (live.length !== env.length || pairs.length !== live.length)
|
||||
this.add({
|
||||
category: "length",
|
||||
pointer,
|
||||
live: `${String(live.length)} items`,
|
||||
env: `${String(env.length)} items (${String(pairs.length)} paired)`,
|
||||
});
|
||||
const envOrder = pairs.map(([, right]) => right);
|
||||
if (
|
||||
envOrder.some((right, index) => index > 0 && right < envOrder[index - 1]!)
|
||||
)
|
||||
this.add({ category: "order", pointer });
|
||||
for (const [left, right] of pairs)
|
||||
this.value(live[left]!, env[right]!, `${pointer}/${String(left)}`, {
|
||||
own,
|
||||
});
|
||||
}
|
||||
|
||||
/** Arrays of scalars holding the same items in another order differ only in order, as sets an application returns unordered do. */
|
||||
sameItems(live: JsonValue[], env: JsonValue[], pointer: string): boolean {
|
||||
const scalar = (item: JsonValue): boolean =>
|
||||
item === null || typeof item !== "object";
|
||||
if (live.length !== env.length || !live.every(scalar) || !env.every(scalar))
|
||||
return false;
|
||||
// Live ids compare in the environment's terms, as scalar values do.
|
||||
const mapped = live.map((item) => translateValue(item, this.map));
|
||||
const key = (items: JsonValue[]): string =>
|
||||
JSON.stringify(items.map((item) => JSON.stringify(item)).sort());
|
||||
if (key(mapped) !== key(env)) return false;
|
||||
this.normalised.ids += mapped.filter(
|
||||
(item, index) => item !== live[index],
|
||||
).length;
|
||||
if (JSON.stringify(mapped) !== JSON.stringify(env))
|
||||
this.add({ category: "order", pointer });
|
||||
return true;
|
||||
}
|
||||
|
||||
object(
|
||||
live: JsonObject,
|
||||
env: JsonObject,
|
||||
pointer: string,
|
||||
own = false,
|
||||
): void {
|
||||
const self = own || pointer === this.entity.at;
|
||||
// An object keyed by ids compares each member under the environment's key.
|
||||
const envKey = (key: string): string => {
|
||||
const mapped = this.map.env(key);
|
||||
return mapped !== undefined && Object.hasOwn(env, String(mapped))
|
||||
? String(mapped)
|
||||
: key;
|
||||
};
|
||||
const matched = new Set<string>();
|
||||
for (const [key, value] of Object.entries(live)) {
|
||||
const child = `${pointer}/${escapePointer(key)}`;
|
||||
const other = envKey(key);
|
||||
if (!Object.hasOwn(env, other))
|
||||
this.add({
|
||||
category: "missing-field",
|
||||
pointer: child,
|
||||
live: shown(value),
|
||||
});
|
||||
else {
|
||||
matched.add(other);
|
||||
this.value(value, env[other]!, child, { key, own: self });
|
||||
}
|
||||
}
|
||||
for (const [key, value] of Object.entries(env))
|
||||
if (!matched.has(key))
|
||||
this.add({
|
||||
category: "extra-field",
|
||||
pointer: `${pointer}/${escapePointer(key)}`,
|
||||
env: shown(value),
|
||||
});
|
||||
}
|
||||
|
||||
/** `own` marks an item of the entity array, or with `key` a field of an own entity; a field's own children are not. */
|
||||
value(
|
||||
live: JsonValue,
|
||||
env: JsonValue,
|
||||
pointer: string,
|
||||
at: { readonly key?: string; readonly own?: boolean } = {},
|
||||
): void {
|
||||
if (Array.isArray(live) && Array.isArray(env))
|
||||
this.array(live, env, pointer);
|
||||
else if (isObject(live) && isObject(env))
|
||||
this.object(live, env, pointer, at.own === true && at.key === undefined);
|
||||
else if (
|
||||
Array.isArray(live) ||
|
||||
Array.isArray(env) ||
|
||||
isObject(live) ||
|
||||
isObject(env)
|
||||
)
|
||||
this.add({
|
||||
category: "value",
|
||||
pointer,
|
||||
live: shown(live),
|
||||
env: shown(env),
|
||||
});
|
||||
else this.scalar(live, env, pointer, at);
|
||||
}
|
||||
}
|
||||
|
||||
/** One answer: its status, and its JSON body, its text, or null without one. */
|
||||
export interface Answer {
|
||||
readonly status: number | null;
|
||||
readonly body: JsonValue;
|
||||
/** Set when the body was over the replay limit, so only a prefix is known. */
|
||||
readonly cut?: boolean;
|
||||
/** Set when the probe did not record the body, so only the status is known. */
|
||||
readonly unobserved?: boolean;
|
||||
}
|
||||
|
||||
/** Compare a live answer with the environment's after normalisation. */
|
||||
export const compareAnswers = (
|
||||
live: Answer,
|
||||
env: Answer,
|
||||
map: IdMap,
|
||||
entity: { readonly scope?: string; readonly at?: string } = {},
|
||||
): { readonly differences: Difference[]; readonly normalised: Normalised } => {
|
||||
const differ = new Differ(map, { scope: entity.scope, at: entity.at ?? "" });
|
||||
if (live.status !== env.status)
|
||||
differ.add({
|
||||
category: "status",
|
||||
pointer: "",
|
||||
live: String(live.status),
|
||||
env: String(env.status),
|
||||
});
|
||||
if (live.cut === true) differ.normalised.large_bodies += 1;
|
||||
else if (live.unobserved === true) differ.normalised.unobserved_bodies += 1;
|
||||
else differ.value(live.body, env.body, "");
|
||||
return { differences: differ.differences, normalised: differ.normalised };
|
||||
};
|
||||
@@ -0,0 +1,205 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
IdMap,
|
||||
learnIds,
|
||||
pairElements,
|
||||
translatePath,
|
||||
translateValue,
|
||||
} from "./environmentCompareIds.js";
|
||||
|
||||
describe("IdMap", () => {
|
||||
it("keeps the first pairing of either side and counts conflicting ones", () => {
|
||||
const map = new IdMap();
|
||||
map.learn(101, 1);
|
||||
map.learn("101", 1);
|
||||
map.learn(101, 2);
|
||||
map.learn(102, 1);
|
||||
expect(map.env(101)).toBe(1);
|
||||
expect(map.env("101")).toBe(1);
|
||||
expect(map.hasEnv(1)).toBe(true);
|
||||
expect(map.size).toBe(1);
|
||||
expect(map.conflicts).toBe(2);
|
||||
});
|
||||
|
||||
it("ignores a pair of equal values", () => {
|
||||
const map = new IdMap();
|
||||
map.learn("same", "same");
|
||||
expect(map.size).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("pairElements", () => {
|
||||
it("pairs objects by the shared unique string field that pairs the most", () => {
|
||||
const live = [
|
||||
{ id: 102, name: "Second", kind: "book" },
|
||||
{ id: 101, name: "First", kind: "book" },
|
||||
];
|
||||
const env = [
|
||||
{ id: 1, name: "First", kind: "book" },
|
||||
{ id: 2, name: "Second", kind: "book" },
|
||||
{ id: 3, name: "Third", kind: "book" },
|
||||
];
|
||||
expect(pairElements(live, env)).toEqual([
|
||||
[0, 1],
|
||||
[1, 0],
|
||||
]);
|
||||
});
|
||||
|
||||
it("pairs by position without a shared key or for scalar items", () => {
|
||||
expect(pairElements([{ n: 1 }, { n: 2 }], [{ n: 3 }])).toEqual([[0, 0]]);
|
||||
expect(pairElements(["a", "b"], ["c", "d"])).toEqual([
|
||||
[0, 0],
|
||||
[1, 1],
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("learnIds", () => {
|
||||
it("maps identifier fields of paired list items and of two single answers", () => {
|
||||
const map = new IdMap();
|
||||
learnIds(
|
||||
[
|
||||
{ id: 102, name: "Second", owner_id: "u-9" },
|
||||
{ id: 101, name: "First", owner_id: "u-9" },
|
||||
],
|
||||
[
|
||||
{ id: 1, name: "First", owner_id: "u-1" },
|
||||
{ id: 2, name: "Second", owner_id: "u-1" },
|
||||
],
|
||||
map,
|
||||
{ heldIds: new Set() },
|
||||
);
|
||||
expect([map.env(101), map.env(102), map.env("u-9")]).toEqual([1, 2, "u-1"]);
|
||||
learnIds(
|
||||
{ data: { uid: "x9", ref: "r-1" } },
|
||||
{ data: { uid: "e3", ref: "r-2" } },
|
||||
map,
|
||||
{ heldIds: new Set(["r-2"]) },
|
||||
);
|
||||
expect(map.env("r-1")).toBe("r-2");
|
||||
expect(map.env("x9")).toBeUndefined();
|
||||
learnIds({ workers: 0 }, { workers: 1 }, map, { heldIds: new Set(["1"]) });
|
||||
expect(map.env(0)).toBeUndefined();
|
||||
});
|
||||
|
||||
it("learns nothing from items that share a key but no value, or differ in count without one", () => {
|
||||
const map = new IdMap();
|
||||
learnIds([{ code: "kelp", id: 1 }], [{ code: "menu", id: 2 }], map, {
|
||||
heldIds: new Set(["menu"]),
|
||||
});
|
||||
learnIds(
|
||||
[{ n: 1, id: 3 }],
|
||||
[
|
||||
{ n: 1, id: 4 },
|
||||
{ n: 2, id: 5 },
|
||||
],
|
||||
map,
|
||||
{ heldIds: new Set() },
|
||||
);
|
||||
expect(map.size).toBe(0);
|
||||
learnIds([{ n: 1, id: 3 }], [{ n: 1, id: 4 }], map, { heldIds: new Set() });
|
||||
expect(map.env(3)).toBe(4);
|
||||
});
|
||||
});
|
||||
|
||||
describe("learnIds over objects keyed by ids", () => {
|
||||
it("pairs entries by a natural key in their values and maps the keys", () => {
|
||||
const map = new IdMap();
|
||||
learnIds(
|
||||
{
|
||||
data: {
|
||||
liveA: { id: "liveA", name: "Lamp Oil" },
|
||||
liveB: { id: "liveB", name: "Quarry" },
|
||||
},
|
||||
},
|
||||
{
|
||||
data: {
|
||||
envB: { id: "envB", name: "Quarry" },
|
||||
envA: { id: "envA", name: "Lamp Oil" },
|
||||
},
|
||||
},
|
||||
map,
|
||||
{ heldIds: new Set() },
|
||||
);
|
||||
expect([map.env("liveA"), map.env("liveB")]).toEqual(["envA", "envB"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("translateValue", () => {
|
||||
it("replaces mapped live identifiers at any depth in the original value's type", () => {
|
||||
const map = new IdMap();
|
||||
map.learn(101, 1);
|
||||
map.learn("abc", "xyz");
|
||||
expect(
|
||||
translateValue({ ids: [101, "101", "abc"], name: "abc def", n: 5 }, map),
|
||||
).toEqual({ ids: [1, "1", "xyz"], name: "abc def", n: 5 });
|
||||
});
|
||||
});
|
||||
|
||||
describe("translatePath", () => {
|
||||
it("replaces segments that decode to a mapped live id and re-encodes them", () => {
|
||||
const map = new IdMap();
|
||||
map.learn("live@1", "env@1");
|
||||
map.learn(7, 3);
|
||||
expect(translatePath("/api/releases/live%401/deploys/7/", map)).toBe(
|
||||
"/api/releases/env%401/deploys/3/",
|
||||
);
|
||||
expect(translatePath("/api/%E0%A4%A/x", map)).toBe("/api/%E0%A4%A/x");
|
||||
});
|
||||
});
|
||||
|
||||
describe("ids two collections reuse", () => {
|
||||
it("keeps each collection's pairing of a live id and translates a path id by its collection", () => {
|
||||
const map = new IdMap();
|
||||
learnIds([{ id: 101, name: "First" }], [{ id: 1, name: "First" }], map, {
|
||||
heldIds: new Set(),
|
||||
scope: "/api/books",
|
||||
});
|
||||
learnIds([{ id: 101, name: "Bob" }], [{ id: 2, name: "Bob" }], map, {
|
||||
heldIds: new Set(),
|
||||
scope: "/api/authors",
|
||||
});
|
||||
expect(map.env(101, "/api/authors")).toBe(2);
|
||||
expect(map.env(101)).toBe(1);
|
||||
expect(
|
||||
translatePath("/api/authors/101", map, (index) =>
|
||||
index === 2 ? "/api/authors" : undefined,
|
||||
),
|
||||
).toBe("/api/authors/2");
|
||||
});
|
||||
});
|
||||
|
||||
describe("objects keyed by ids that overlap", () => {
|
||||
it("pairs entries by a natural key when some re-seeded keys are reused, and only when every entry pairs", () => {
|
||||
const map = new IdMap();
|
||||
learnIds(
|
||||
{ "2": { name: "A" }, "3": { name: "B" } },
|
||||
{ "1": { name: "A" }, "2": { name: "B" } },
|
||||
map,
|
||||
{ heldIds: new Set() },
|
||||
);
|
||||
expect([map.env("2"), map.env("3")]).toEqual(["1", "2"]);
|
||||
const partial = new IdMap();
|
||||
learnIds(
|
||||
{ "2": { name: "A" }, "3": { name: "C" } },
|
||||
{ "1": { name: "A" }, "2": { name: "B" } },
|
||||
partial,
|
||||
{ heldIds: new Set() },
|
||||
);
|
||||
expect(partial.size).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ordinary objects whose fields overlap", () => {
|
||||
it("does not read their fields as ids, even when every value pairs", () => {
|
||||
const map = new IdMap();
|
||||
learnIds(
|
||||
{ current: { name: "A" }, previous: { name: "B" } },
|
||||
{ current: { name: "B" }, next: { name: "A" } },
|
||||
map,
|
||||
{ heldIds: new Set() },
|
||||
);
|
||||
expect(map.size).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,368 @@
|
||||
import type { JsonObject } from "../../domain/analysisView/environmentStateRules.js";
|
||||
import { isUuid } from "../../domain/analysisView/environmentStateWriteValues.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import type { ProbeToolCall } from "./environmentCompareMatch.js";
|
||||
|
||||
/** Field names that hold an identifier: `id`, `uuid`, `guid`, or names ending in `id` or `_id`. */
|
||||
export const ID_FIELD = /^(id|uuid|guid)$|(_id|Id|ID|_uuid|Uuid)$/u;
|
||||
|
||||
const isObject = (value: JsonValue | undefined): value is JsonObject =>
|
||||
typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
|
||||
const isScalarId = (value: JsonValue | undefined): value is string | number =>
|
||||
(typeof value === "string" && value !== "") ||
|
||||
(typeof value === "number" && Number.isFinite(value));
|
||||
|
||||
/** Entity identity fields, whose values an answer's own collection scopes. */
|
||||
export const IDENTITY_FIELD = /^(id|uuid|guid)$/u;
|
||||
|
||||
/**
|
||||
* Live identifier values and the environment's values they stand for, in both directions: one map for the whole probe,
|
||||
* and one per collection for the ids of its own entities, since two collections can reuse a live id for entities the
|
||||
* environment holds under different ids.
|
||||
*/
|
||||
export class IdMap {
|
||||
readonly #toEnv = new Map<string, string | number>();
|
||||
readonly #toLive = new Map<string, string | number>();
|
||||
readonly #scoped = new Map<string, Map<string, string | number>>();
|
||||
readonly #envValues = new Set<string>();
|
||||
#conflicts = 0;
|
||||
|
||||
/** Pair a live value with an environment value; the first pairing of either side wins, and within a collection the first of that collection. */
|
||||
learn(live: string | number, env: string | number, scope?: string): void {
|
||||
const liveKey = String(live);
|
||||
const envKey = String(env);
|
||||
if (liveKey === envKey) return;
|
||||
if (scope !== undefined) {
|
||||
const scoped = this.#scoped.get(scope) ?? new Map();
|
||||
if (!scoped.has(liveKey)) scoped.set(liveKey, env);
|
||||
this.#scoped.set(scope, scoped);
|
||||
this.#envValues.add(envKey);
|
||||
}
|
||||
const known = this.#toEnv.get(liveKey);
|
||||
if (known !== undefined) {
|
||||
if (String(known) !== envKey) this.#conflicts += 1;
|
||||
return;
|
||||
}
|
||||
if (this.#toLive.has(envKey)) {
|
||||
this.#conflicts += 1;
|
||||
return;
|
||||
}
|
||||
this.#toEnv.set(liveKey, env);
|
||||
this.#toLive.set(envKey, live);
|
||||
this.#envValues.add(envKey);
|
||||
}
|
||||
|
||||
/** The environment's value for a live one: the collection's own when a scope is given and it knows one, else the probe's. */
|
||||
env(live: string | number, scope?: string): string | number | undefined {
|
||||
const liveKey = String(live);
|
||||
return (
|
||||
(scope === undefined
|
||||
? undefined
|
||||
: this.#scoped.get(scope)?.get(liveKey)) ?? this.#toEnv.get(liveKey)
|
||||
);
|
||||
}
|
||||
|
||||
hasLive(value: string | number): boolean {
|
||||
const key = String(value);
|
||||
return (
|
||||
this.#toEnv.has(key) ||
|
||||
[...this.#scoped.values()].some((scoped) => scoped.has(key))
|
||||
);
|
||||
}
|
||||
|
||||
hasEnv(value: string | number): boolean {
|
||||
return this.#envValues.has(String(value));
|
||||
}
|
||||
|
||||
get size(): number {
|
||||
return this.#toEnv.size;
|
||||
}
|
||||
|
||||
get conflicts(): number {
|
||||
return this.#conflicts;
|
||||
}
|
||||
}
|
||||
|
||||
/** A field every object element has as a non-empty string, unique on its side; not an identifier, which differs across sides. */
|
||||
const keyCandidates = (items: readonly JsonObject[]): Set<string> => {
|
||||
const candidates = new Set<string>();
|
||||
const first = items[0];
|
||||
if (first === undefined) return candidates;
|
||||
for (const key of Object.keys(first)) {
|
||||
if (ID_FIELD.test(key)) continue;
|
||||
const values = items.map((item) => item[key]);
|
||||
if (
|
||||
values.every((value) => typeof value === "string" && value !== "") &&
|
||||
new Set(values).size === values.length
|
||||
)
|
||||
candidates.add(key);
|
||||
}
|
||||
return candidates;
|
||||
};
|
||||
|
||||
/** How two arrays pair: by a natural key, by position because no key is shared, or by position although a shared key paired nothing. */
|
||||
interface Pairing {
|
||||
readonly pairs: Array<readonly [number, number]>;
|
||||
readonly by: "key" | "position" | "unmatched";
|
||||
}
|
||||
|
||||
const pairing = (
|
||||
live: readonly JsonValue[],
|
||||
env: readonly JsonValue[],
|
||||
): Pairing => {
|
||||
const byPosition = (by: Pairing["by"]): Pairing => ({
|
||||
pairs: Array.from(
|
||||
{ length: Math.min(live.length, env.length) },
|
||||
(_, index) => [index, index] as const,
|
||||
),
|
||||
by,
|
||||
});
|
||||
if (!live.every(isObject) || !env.every(isObject))
|
||||
return byPosition("position");
|
||||
const liveItems = live as readonly JsonObject[];
|
||||
const envItems = env as readonly JsonObject[];
|
||||
const envKeys = keyCandidates(envItems);
|
||||
const shared = [...keyCandidates(liveItems)].filter((key) =>
|
||||
envKeys.has(key),
|
||||
);
|
||||
let best: Array<readonly [number, number]> = [];
|
||||
for (const key of shared.sort()) {
|
||||
const envIndex = new Map(envItems.map((item, index) => [item[key], index]));
|
||||
const pairs: Array<readonly [number, number]> = [];
|
||||
for (const [index, item] of liveItems.entries()) {
|
||||
const match = envIndex.get(item[key]);
|
||||
if (match !== undefined) pairs.push([index, match]);
|
||||
}
|
||||
if (pairs.length > best.length) best = pairs;
|
||||
}
|
||||
if (best.length > 0) return { pairs: best, by: "key" };
|
||||
return byPosition(shared.length > 0 ? "unmatched" : "position");
|
||||
};
|
||||
|
||||
/**
|
||||
* Pair two arrays' object elements by the natural key that pairs the most, as [live index, env index]; arrays of other
|
||||
* values, or with no shared key, pair by position up to the shorter length.
|
||||
*/
|
||||
export const pairElements = (
|
||||
live: readonly JsonValue[],
|
||||
env: readonly JsonValue[],
|
||||
): Array<readonly [number, number]> => pairing(live, env).pairs;
|
||||
|
||||
/** Whether an object's keys are evidently identifiers: digits or UUIDs, or each the identity of the entry it keys. */
|
||||
const keyedByIds = (
|
||||
entries: readonly (readonly [string, JsonValue])[],
|
||||
): boolean =>
|
||||
entries.every(
|
||||
([key, value]) =>
|
||||
/^\d+$/u.test(key) ||
|
||||
isUuid(key) ||
|
||||
(isObject(value) &&
|
||||
Object.entries(value).some(
|
||||
([field, held]) =>
|
||||
IDENTITY_FIELD.test(field) &&
|
||||
isScalarId(held) &&
|
||||
String(held) === key,
|
||||
)),
|
||||
);
|
||||
|
||||
/**
|
||||
* Two objects keyed by ids, whose object values pair by a natural key: each pair maps the live key to the environment's
|
||||
* and is learned from in turn. Returns whether the objects were read that way.
|
||||
*/
|
||||
const learnKeyedEntries = (
|
||||
live: JsonObject,
|
||||
env: JsonObject,
|
||||
map: IdMap,
|
||||
heldIds: ReadonlySet<string>,
|
||||
): boolean => {
|
||||
const liveEntries = Object.entries(live);
|
||||
const envEntries = Object.entries(env);
|
||||
const shared = liveEntries.filter(([key]) => Object.hasOwn(env, key)).length;
|
||||
if (
|
||||
liveEntries.length === 0 ||
|
||||
envEntries.length === 0 ||
|
||||
shared === liveEntries.length ||
|
||||
!liveEntries.every(([, value]) => isObject(value)) ||
|
||||
!envEntries.every(([, value]) => isObject(value))
|
||||
)
|
||||
return false;
|
||||
const paired = pairing(
|
||||
liveEntries.map(([, value]) => value),
|
||||
envEntries.map(([, value]) => value),
|
||||
);
|
||||
// Re-seeded ids can reuse a key for another entity, but ordinary fields overlap too: keys that overlap are read as ids
|
||||
// only when they evidently are ids and every entry pairs.
|
||||
if (
|
||||
paired.by !== "key" ||
|
||||
(shared > 0 &&
|
||||
(paired.pairs.length !== liveEntries.length ||
|
||||
!keyedByIds(liveEntries) ||
|
||||
!keyedByIds(envEntries)))
|
||||
)
|
||||
return false;
|
||||
for (const [left, right] of paired.pairs) {
|
||||
const [liveKey, liveValue] = liveEntries[left]!;
|
||||
const [envKey, envValue] = envEntries[right]!;
|
||||
map.learn(liveKey, envKey);
|
||||
learnIds(liveValue, envValue, map, { heldIds });
|
||||
}
|
||||
return true;
|
||||
};
|
||||
|
||||
/** What learning reads besides the two values: the map it adds to, the ids the state holds, and where the answer's own entities sit. */
|
||||
interface LearnContext {
|
||||
readonly map: IdMap;
|
||||
readonly heldIds: ReadonlySet<string>;
|
||||
readonly scope: string | undefined;
|
||||
/** JSON pointer of the answer's own entity, or of the array holding its entities. */
|
||||
readonly entityAt: string;
|
||||
}
|
||||
|
||||
/** Two values under one key as a pair of ids, when they are: an identifier field, or outside one only a string the state holds as an id, since a counter that equals an id is not one. */
|
||||
const idPair = (
|
||||
key: string,
|
||||
value: JsonValue,
|
||||
other: JsonValue | undefined,
|
||||
heldIds: ReadonlySet<string>,
|
||||
): readonly [string | number, string | number] | undefined =>
|
||||
isScalarId(value) &&
|
||||
isScalarId(other) &&
|
||||
(ID_FIELD.test(key) ||
|
||||
(typeof value === "string" &&
|
||||
typeof other === "string" &&
|
||||
heldIds.has(other)))
|
||||
? [value, other]
|
||||
: undefined;
|
||||
|
||||
const childPointer = (at: string, key: string | number): string =>
|
||||
`${at}/${String(key).replaceAll("~", "~0").replaceAll("/", "~1")}`;
|
||||
|
||||
const learnAt = (
|
||||
live: JsonValue | undefined,
|
||||
env: JsonValue | undefined,
|
||||
context: LearnContext,
|
||||
place: { readonly at: string; readonly own: boolean },
|
||||
): void => {
|
||||
if (Array.isArray(live) && Array.isArray(env)) {
|
||||
// Items paired only by position stand for the same entities only when nothing could pair them by a key and the
|
||||
// arrays are the same length; otherwise a filtered or reordered list would teach wrong ids.
|
||||
const paired = pairing(live, env);
|
||||
if (
|
||||
paired.by === "unmatched" ||
|
||||
(paired.by === "position" && live.length !== env.length)
|
||||
)
|
||||
return;
|
||||
for (const [left, right] of paired.pairs)
|
||||
learnAt(live[left], env[right], context, {
|
||||
at: childPointer(place.at, left),
|
||||
own: place.at === context.entityAt,
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (!isObject(live) || !isObject(env)) return;
|
||||
if (learnKeyedEntries(live, env, context.map, context.heldIds)) return;
|
||||
const own = place.own || place.at === context.entityAt;
|
||||
for (const [key, value] of Object.entries(live)) {
|
||||
if (!Object.hasOwn(env, key)) continue;
|
||||
const other = env[key];
|
||||
const pair = idPair(key, value, other, context.heldIds);
|
||||
if (pair !== undefined)
|
||||
context.map.learn(
|
||||
...pair,
|
||||
own && IDENTITY_FIELD.test(key) ? context.scope : undefined,
|
||||
);
|
||||
else
|
||||
learnAt(value, other, context, {
|
||||
at: childPointer(place.at, key),
|
||||
own: false,
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Learn identifier pairs from the live and environment answers to one call: identifier fields, and strings the state holds
|
||||
* as ids, of objects that pair, at any depth. The identity fields of the answer's own entities, the object or the items of
|
||||
* the array at `entityAt` (the root by default), are also learned under `scope`, the collection the call names.
|
||||
*/
|
||||
export const learnIds = (
|
||||
live: JsonValue | undefined,
|
||||
env: JsonValue | undefined,
|
||||
map: IdMap,
|
||||
known: {
|
||||
/** Ids the environment's state holds, which mark a string outside identifier fields as an id. */
|
||||
readonly heldIds: ReadonlySet<string>;
|
||||
readonly scope?: string;
|
||||
readonly entityAt?: string;
|
||||
},
|
||||
): void =>
|
||||
learnAt(
|
||||
live,
|
||||
env,
|
||||
{
|
||||
map,
|
||||
heldIds: known.heldIds,
|
||||
scope: known.scope,
|
||||
entityAt: known.entityAt ?? "",
|
||||
},
|
||||
{ at: "", own: false },
|
||||
);
|
||||
|
||||
/** A value with every string or number equal to a mapped live identifier replaced by the environment's value, in its own type. */
|
||||
export const translateValue = (value: JsonValue, map: IdMap): JsonValue => {
|
||||
if (Array.isArray(value))
|
||||
return value.map((item) => translateValue(item, map));
|
||||
if (isObject(value))
|
||||
return Object.fromEntries(
|
||||
Object.entries(value).map(([key, item]) => [
|
||||
key,
|
||||
translateValue(item, map),
|
||||
]),
|
||||
);
|
||||
if (!isScalarId(value)) return value;
|
||||
const env = map.env(value);
|
||||
if (env === undefined) return value;
|
||||
return typeof value === "string"
|
||||
? String(env)
|
||||
: typeof env === "number"
|
||||
? env
|
||||
: value;
|
||||
};
|
||||
|
||||
/** The arguments the environment is called with: path values as matched, already in its terms, paging values as sent, and every other value translated. */
|
||||
export const environmentArguments = (
|
||||
call: ProbeToolCall,
|
||||
map: IdMap,
|
||||
): Record<string, JsonValue> =>
|
||||
Object.fromEntries(
|
||||
Object.entries(call.args).map(([name, value]) => [
|
||||
name,
|
||||
call.pathArguments.includes(name) || call.pagingArguments.includes(name)
|
||||
? value
|
||||
: translateValue(value, map),
|
||||
]),
|
||||
);
|
||||
|
||||
/**
|
||||
* A request path with every segment that decodes to a mapped live identifier replaced by the environment's value,
|
||||
* encoded; `scopeAt` names the collection of the segment at a position after the leading slash, whose own mapping wins.
|
||||
*/
|
||||
export const translatePath = (
|
||||
path: string,
|
||||
map: IdMap,
|
||||
scopeAt: (index: number) => string | undefined = () => undefined,
|
||||
): string =>
|
||||
path
|
||||
.split("/")
|
||||
.map((segment, index) => {
|
||||
let decoded = segment;
|
||||
try {
|
||||
decoded = decodeURIComponent(segment);
|
||||
} catch {
|
||||
return segment;
|
||||
}
|
||||
const env =
|
||||
decoded === "" ? undefined : map.env(decoded, scopeAt(index - 1));
|
||||
return env === undefined ? segment : encodeURIComponent(String(env));
|
||||
})
|
||||
.join("/");
|
||||
@@ -0,0 +1,141 @@
|
||||
import { isPagingParameter } from "../../domain/analysisView/environmentStatePaging.js";
|
||||
import { segmentRule } from "../../domain/analysisView/webNetworkRouteTemplates.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import { decodedPathSegment } from "./environmentReplay.js";
|
||||
import type {
|
||||
EnvironmentManifest,
|
||||
EnvironmentManifestTool,
|
||||
} from "./environmentSource.js";
|
||||
import { environmentToolArguments } from "./environmentTools.js";
|
||||
|
||||
const FORMAT_RULES: ReadonlySet<string> = new Set([
|
||||
"numeric",
|
||||
"uuid",
|
||||
"hex",
|
||||
"iso-date",
|
||||
]);
|
||||
|
||||
/** One request of a probe capture, as the comparison replays it. */
|
||||
export interface ProbeRequest {
|
||||
readonly method: string;
|
||||
readonly path: string;
|
||||
/** WHATWG query spelling including `?`, or null. */
|
||||
readonly query: string | null;
|
||||
readonly body: JsonValue | undefined;
|
||||
}
|
||||
|
||||
/** A probe request as a call of one manifest tool. */
|
||||
export interface ProbeToolCall {
|
||||
readonly tool: EnvironmentManifestTool;
|
||||
readonly args: Record<string, JsonValue>;
|
||||
/** Argument names of the path values, in path order. */
|
||||
readonly pathArguments: readonly string[];
|
||||
/** Argument names of query values that page a list: counts, never ids. */
|
||||
readonly pagingArguments: readonly string[];
|
||||
/** Query names the tool does not take; the call drops them. */
|
||||
readonly ignoredQuery: readonly string[];
|
||||
}
|
||||
|
||||
/** A tool's path segments up to `end`, a parameter as `{}`, so tools naming one parameter differently share collections. */
|
||||
const collectionAt = (tool: EnvironmentManifestTool, end: number): string => {
|
||||
const params = new Set(tool.params.map((param) => param.index));
|
||||
return `/${tool.path_template
|
||||
.slice(1)
|
||||
.split("/")
|
||||
.slice(0, end)
|
||||
.map((segment, index) => (params.has(index) ? "{}" : segment))
|
||||
.join("/")}`;
|
||||
};
|
||||
|
||||
/** The collection a call of the tool names: its path, less the trailing parameter of an item route. */
|
||||
export const collectionScope = (tool: EnvironmentManifestTool): string => {
|
||||
const length = tool.path_template.slice(1).split("/").length;
|
||||
return collectionAt(
|
||||
tool,
|
||||
tool.params.some((param) => param.index === length - 1)
|
||||
? length - 1
|
||||
: length,
|
||||
);
|
||||
};
|
||||
|
||||
/** The collection each path parameter names an entity of: the path before it, by the parameter's position. */
|
||||
export const parameterScopes = (
|
||||
tool: EnvironmentManifestTool,
|
||||
): ReadonlyMap<number, string> =>
|
||||
new Map(
|
||||
tool.params.map((param) => [param.index, collectionAt(tool, param.index)]),
|
||||
);
|
||||
|
||||
interface Fit {
|
||||
readonly tool: EnvironmentManifestTool;
|
||||
readonly literals: number;
|
||||
}
|
||||
|
||||
/** How a path fits a tool's template: the literal segments it matches, or undefined when it does not fit. */
|
||||
const fitOf = (
|
||||
tool: EnvironmentManifestTool,
|
||||
segments: readonly string[],
|
||||
): Fit | undefined => {
|
||||
const template = tool.path_template.slice(1).split("/");
|
||||
if (template.length !== segments.length) return undefined;
|
||||
const params = new Map(tool.params.map((param) => [param.index, param]));
|
||||
let literals = 0;
|
||||
for (const [index, part] of template.entries()) {
|
||||
const segment = segments[index]!;
|
||||
const param = params.get(index);
|
||||
if (param === undefined) {
|
||||
if (part !== segment && part !== decodedPathSegment(segment))
|
||||
return undefined;
|
||||
literals += 1;
|
||||
} else if (
|
||||
// Replay matches a path value decoded, so its format is the decoded value's.
|
||||
FORMAT_RULES.has(param.rule) &&
|
||||
segmentRule(decodedPathSegment(segment)) !== param.rule
|
||||
)
|
||||
return undefined;
|
||||
else if (segment === "") return undefined;
|
||||
}
|
||||
return { tool, literals };
|
||||
};
|
||||
|
||||
/** The tool of the request's method whose template fits the path with the most literal segments; the manifest's order breaks ties. */
|
||||
export const matchProbeRequest = (
|
||||
manifest: EnvironmentManifest,
|
||||
request: ProbeRequest,
|
||||
): ProbeToolCall | undefined => {
|
||||
const segments = request.path.slice(1).split("/");
|
||||
let best: Fit | undefined;
|
||||
for (const tool of manifest.tools) {
|
||||
if (tool.method !== request.method) continue;
|
||||
const fit = fitOf(tool, segments);
|
||||
if (
|
||||
fit !== undefined &&
|
||||
(best === undefined || fit.literals > best.literals)
|
||||
)
|
||||
best = fit;
|
||||
}
|
||||
if (best === undefined) return undefined;
|
||||
const { tool } = best;
|
||||
const names = environmentToolArguments(tool);
|
||||
const args: Record<string, JsonValue> = {};
|
||||
const pathArguments: string[] = [];
|
||||
for (const param of tool.params) {
|
||||
const name = names.path.get(param.name) ?? param.name;
|
||||
args[name] = decodedPathSegment(segments[param.index]!);
|
||||
pathArguments.push(name);
|
||||
}
|
||||
const ignoredQuery: string[] = [];
|
||||
const pagingArguments: string[] = [];
|
||||
if (request.query !== null)
|
||||
for (const [key, value] of new URLSearchParams(request.query.slice(1))) {
|
||||
const name = names.query.get(key);
|
||||
if (name === undefined) ignoredQuery.push(key);
|
||||
else if (!Object.hasOwn(args, name)) {
|
||||
args[name] = value;
|
||||
if (isPagingParameter(key)) pagingArguments.push(name);
|
||||
}
|
||||
}
|
||||
if (names.body !== undefined && request.body !== undefined)
|
||||
args[names.body] = request.body;
|
||||
return { tool, args, pathArguments, pagingArguments, ignoredQuery };
|
||||
};
|
||||
@@ -0,0 +1,177 @@
|
||||
import { z } from "zod";
|
||||
import { isJsonLike } from "../../domain/analysisView/EnvironmentManifestAccumulator.js";
|
||||
import {
|
||||
primaryOrigin,
|
||||
type OriginStats,
|
||||
} from "../../domain/analysisView/environmentManifestOrigin.js";
|
||||
import {
|
||||
mediaTypeEssence,
|
||||
readRecordedRequestLine,
|
||||
type RecordedRequestLine,
|
||||
type RequestLineExclusion,
|
||||
} from "../../domain/analysisView/webNetworkRouteTemplates.js";
|
||||
import { parseExactJson } from "../../domain/exactJsonNumber.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import type { WebNetworkCaptureRecord } from "../../domain/webNetworkCapture.js";
|
||||
import type { Answer } from "./environmentCompareDiff.js";
|
||||
import type { ProbeRequest } from "./environmentCompareMatch.js";
|
||||
import {
|
||||
recordedExchange,
|
||||
replayedBody,
|
||||
type ReplayedBody,
|
||||
} from "./environmentSource.js";
|
||||
|
||||
/** A probe record left uncompared: without a usable request line, or at another origin than the application's, it is not replayed either; without an answer it is replayed, so the state follows, but nothing is compared. */
|
||||
export interface UnobservedRecord {
|
||||
readonly ordinal: number;
|
||||
readonly reason: RequestLineExclusion | "other-origin" | "no-response";
|
||||
}
|
||||
|
||||
/** One probe request with the application's answer, absent when the probe recorded none. */
|
||||
export interface ProbeStep {
|
||||
readonly ordinal: number;
|
||||
readonly request: ProbeRequest;
|
||||
readonly live?: Answer;
|
||||
}
|
||||
|
||||
/** Statuses whose answers carry no body, so a missing body is known to be empty. */
|
||||
const BODYLESS_STATUSES: ReadonlySet<number> = new Set([204, 205, 304]);
|
||||
|
||||
const startedSchema = z.object({ startedDateTime: z.string() });
|
||||
const postDataSchema = z.object({
|
||||
request: z.object({
|
||||
postData: z.object({ mimeType: z.string(), text: z.string() }),
|
||||
}),
|
||||
});
|
||||
|
||||
const methodSchema = z.object({ request: z.object({ method: z.string() }) });
|
||||
|
||||
const startedMs = (record: WebNetworkCaptureRecord): number | undefined => {
|
||||
const started = startedSchema.safeParse(record.reported);
|
||||
const ms = started.success ? Date.parse(started.data.startedDateTime) : NaN;
|
||||
return Number.isFinite(ms) ? ms : undefined;
|
||||
};
|
||||
|
||||
/** Whether a probe makes calls that carry a request body: a POST, PUT or PATCH. */
|
||||
export const sendsBodies = (
|
||||
records: readonly WebNetworkCaptureRecord[],
|
||||
): boolean =>
|
||||
records.some((record) => {
|
||||
const parsed = methodSchema.safeParse(record.reported);
|
||||
return (
|
||||
parsed.success &&
|
||||
["POST", "PUT", "PATCH"].includes(parsed.data.request.method)
|
||||
);
|
||||
});
|
||||
|
||||
/** The JSON a request body holds, its text when it is not JSON, or undefined without one. */
|
||||
export const requestBody = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
): JsonValue | undefined => {
|
||||
const parsed = postDataSchema.safeParse(record.reported);
|
||||
if (!parsed.success) return undefined;
|
||||
const { mimeType, text } = parsed.data.request.postData;
|
||||
return isJsonLike(mediaTypeEssence(mimeType))
|
||||
? (parseExactJson(text) ?? text)
|
||||
: text;
|
||||
};
|
||||
|
||||
/** A body over the replay limit, text or bytes: only a prefix is known, so only the status compares. */
|
||||
export const isCut = (body: ReplayedBody): boolean =>
|
||||
body.kind === "cut" || (body.kind === "base64" && !body.whole);
|
||||
|
||||
/** The application's answer as an environment answer carries it: JSON, text, or null when it is empty; a body the probe did not record is marked. */
|
||||
const liveAnswer = (
|
||||
method: string,
|
||||
status: number,
|
||||
body: ReplayedBody,
|
||||
): Answer => {
|
||||
if (body.kind === "none")
|
||||
return method === "HEAD" || BODYLESS_STATUSES.has(status)
|
||||
? { status, body: null }
|
||||
: { status, body: null, unobserved: true };
|
||||
if (body.kind === "json") return { status, body: body.value };
|
||||
if (body.kind === "base64")
|
||||
return { status, body: body.encoded, cut: isCut(body) };
|
||||
return {
|
||||
status,
|
||||
body: body.text === "" ? null : body.text,
|
||||
cut: isCut(body),
|
||||
};
|
||||
};
|
||||
|
||||
/** The origin a probe calls the application at, chosen as the manifest chooses its own. */
|
||||
const probeOrigin = (
|
||||
lines: ReadonlyMap<number, RecordedRequestLine | null>,
|
||||
): string | null => {
|
||||
const origins = new Map<string, OriginStats>();
|
||||
for (const line of lines.values()) {
|
||||
if (line?.kind !== "line") continue;
|
||||
const stats = origins.get(line.origin) ?? { records: 0, jsonLike: 0 };
|
||||
stats.records += 1;
|
||||
if (isJsonLike(mediaTypeEssence(line.rawMime))) stats.jsonLike += 1;
|
||||
origins.set(line.origin, stats);
|
||||
}
|
||||
return primaryOrigin(origins);
|
||||
};
|
||||
|
||||
/** The probe's requests in the order they were made, each with the application's answer, and the records left uncompared. */
|
||||
export const probeRequests = (
|
||||
records: readonly WebNetworkCaptureRecord[],
|
||||
): { steps: ProbeStep[]; unobserved: UnobservedRecord[] } => {
|
||||
const steps: ProbeStep[] = [];
|
||||
const unobserved: UnobservedRecord[] = [];
|
||||
// Without a start time for every record the probe keeps its recorded order, as state extraction does.
|
||||
const starts = new Map(
|
||||
records.map((record) => [record.ordinal, startedMs(record)]),
|
||||
);
|
||||
const timed = [...starts.values()].every((ms) => ms !== undefined);
|
||||
const ordered = [...records].sort(
|
||||
(left, right) =>
|
||||
(timed ? starts.get(left.ordinal)! - starts.get(right.ordinal)! : 0) ||
|
||||
left.ordinal - right.ordinal,
|
||||
);
|
||||
const lines = new Map(
|
||||
records.map((record) => [
|
||||
record.ordinal,
|
||||
readRecordedRequestLine(record.reported, true, () => true),
|
||||
]),
|
||||
);
|
||||
// Another origin's request goes to some other server, so it neither calls nor changes the environment.
|
||||
const origin = probeOrigin(lines);
|
||||
for (const record of ordered) {
|
||||
const line = lines.get(record.ordinal);
|
||||
const exchange = recordedExchange(record);
|
||||
const { ordinal } = record;
|
||||
if (line?.kind === "line" && line.origin !== origin)
|
||||
unobserved.push({ ordinal, reason: "other-origin" });
|
||||
else if (line?.kind !== "line" || exchange === undefined)
|
||||
unobserved.push({
|
||||
ordinal,
|
||||
reason: line?.kind === "excluded" ? line.reason : "request-unavailable",
|
||||
});
|
||||
else {
|
||||
const request = {
|
||||
method: line.method,
|
||||
path: exchange.path,
|
||||
query: exchange.query,
|
||||
body: requestBody(record),
|
||||
};
|
||||
// A request whose answer was lost may still have changed the application, so it is made but not compared.
|
||||
if (exchange.status === null) {
|
||||
unobserved.push({ ordinal, reason: "no-response" });
|
||||
steps.push({ ordinal, request });
|
||||
} else
|
||||
steps.push({
|
||||
ordinal,
|
||||
request,
|
||||
live: liveAnswer(
|
||||
line.method,
|
||||
exchange.status,
|
||||
replayedBody(exchange),
|
||||
),
|
||||
});
|
||||
}
|
||||
}
|
||||
return { steps, unobserved };
|
||||
};
|
||||
@@ -155,7 +155,8 @@ export const parseEvidenceFile = (
|
||||
}
|
||||
};
|
||||
|
||||
const recordedExchange = (
|
||||
/** One capture record reduced to its request line and retained response body, or undefined without a request line. */
|
||||
export const recordedExchange = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
): RecordedExchange | undefined => {
|
||||
const line = readRecordedRequestLine(record.reported, true, () => true);
|
||||
@@ -213,16 +214,21 @@ export const parseViewEvidence = <Kind extends AnalysisViewResult["kind"]>(
|
||||
});
|
||||
};
|
||||
|
||||
/** Authenticate the evidence file and require complete capture Evidence of either kind. */
|
||||
const parseCaptureEvidence = (
|
||||
/** Authenticate the evidence file and require complete capture Evidence of either kind; a browser capture's request bodies are kept only for the manifest purpose. */
|
||||
export const parseCaptureEvidence = (
|
||||
operation: string,
|
||||
value: unknown,
|
||||
purpose: "manifest" | "replay" = "replay",
|
||||
): Result<
|
||||
{
|
||||
readonly records: readonly WebNetworkCaptureRecord[];
|
||||
readonly evidenceId: string;
|
||||
readonly captureKind: EnvironmentCaptureKind;
|
||||
readonly captureSha256: string | null;
|
||||
/** Whether the capture recorded response bodies: a browser scenario does only when they were selected. */
|
||||
readonly responseBodies: boolean;
|
||||
/** Whether the capture recorded request bodies, likewise. */
|
||||
readonly requestBodies: boolean;
|
||||
},
|
||||
AnalysisError
|
||||
> => {
|
||||
@@ -240,12 +246,14 @@ const parseCaptureEvidence = (
|
||||
.safeParse(evidence.value.parameters);
|
||||
if (scenario.success)
|
||||
return ok({
|
||||
records: scenarioCaptureRecords(scenario.data, "replay").records,
|
||||
records: scenarioCaptureRecords(scenario.data, purpose).records,
|
||||
evidenceId,
|
||||
captureKind: "browser-scenario",
|
||||
captureSha256: definition.success
|
||||
? definition.data.scenario_sha256
|
||||
: null,
|
||||
responseBodies: scenario.data.scenario.network_content.response_body,
|
||||
requestBodies: scenario.data.scenario.network_content.request_body,
|
||||
});
|
||||
} else if (evidence.value.operation === "inspect_web_network_capture") {
|
||||
const records = immutableJsonSnapshotMember(normalized, "records");
|
||||
@@ -278,6 +286,8 @@ const parseCaptureEvidence = (
|
||||
evidenceId,
|
||||
captureKind: "web-network-capture",
|
||||
captureSha256: capture.data.artifact.sha256,
|
||||
responseBodies: true,
|
||||
requestBodies: true,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { basename } from "node:path";
|
||||
import { absoluteCallerPath } from "../filesystem/callerPath.js";
|
||||
import type { EnvironmentExportResult } from "../application/environment/environmentExport.js";
|
||||
import type { EnvironmentSource } from "../application/environment/environmentSource.js";
|
||||
@@ -299,6 +300,77 @@ const evidenceOption = z
|
||||
"Parent capture Evidence JSON path (inspect-web-network-capture or capture-browser-scenario output) the manifest cites",
|
||||
);
|
||||
|
||||
interface CompareArguments {
|
||||
readonly manifest: string;
|
||||
readonly evidence: string;
|
||||
readonly probes: readonly string[];
|
||||
}
|
||||
|
||||
/** Read the environment and every probe, then compare; a refusal is the command's output. */
|
||||
const compareEnvironmentCommand = async (
|
||||
input: CompareArguments,
|
||||
): Promise<JsonValue> => {
|
||||
const read = (path: string) =>
|
||||
parseCliJsonInput(path, CLI_COMMANDS.environmentCompare, undefined, {
|
||||
snapshot: true,
|
||||
});
|
||||
const manifest = await read(input.manifest);
|
||||
if (!manifest.ok) return manifest.error;
|
||||
const evidence = await read(input.evidence);
|
||||
if (!evidence.ok) return evidence.error;
|
||||
const probes = [];
|
||||
for (const path of input.probes) {
|
||||
const probe = await read(path);
|
||||
if (!probe.ok) return probe.error;
|
||||
probes.push({ label: basename(path), value: probe.value });
|
||||
}
|
||||
const { loadEnvironmentSource } =
|
||||
await import("../application/environment/environmentSource.js");
|
||||
const source = loadEnvironmentSource(
|
||||
manifest.value,
|
||||
evidence.value,
|
||||
CLI_COMMANDS.environmentCompare,
|
||||
);
|
||||
if (!source.ok) return cliError(source.error);
|
||||
const { compareEnvironment } =
|
||||
await import("../application/environment/environmentCompare.js");
|
||||
const report = compareEnvironment({
|
||||
source: source.value,
|
||||
captureValue: evidence.value,
|
||||
probes,
|
||||
operation: CLI_COMMANDS.environmentCompare,
|
||||
});
|
||||
return report.ok
|
||||
? (report.value as unknown as JsonValue)
|
||||
: cliError(report.error);
|
||||
};
|
||||
|
||||
/** Register the comparison of an environment with probe captures of its application. */
|
||||
const registerCompareCommand = (cli: CliInstance, logger: Logger): void => {
|
||||
cli.command(CLI_COMMANDS.environmentCompare, {
|
||||
description:
|
||||
"Compare an environment with probe captures of the application it was captured from: each probe is one call sequence recorded against a freshly reset application, answered by the environment from a reset state, with agreement reported per step, tool and probe",
|
||||
args: z.object({ manifest: manifestArgument }),
|
||||
options: z.object({
|
||||
evidence: evidenceOption,
|
||||
probe: z
|
||||
.array(z.string())
|
||||
.min(1)
|
||||
.describe(
|
||||
"Probe capture Evidence JSON path (inspect-web-network-capture or capture-browser-scenario output) of one call sequence made against a freshly reset application; repeat once per sequence",
|
||||
),
|
||||
}),
|
||||
run: ({ args, options }) =>
|
||||
logCliCommand(logger, CLI_COMMANDS.environmentCompare, () =>
|
||||
compareEnvironmentCommand({
|
||||
manifest: args.manifest,
|
||||
evidence: options.evidence,
|
||||
probes: options.probe,
|
||||
}),
|
||||
),
|
||||
});
|
||||
};
|
||||
|
||||
/** Register the one-shot bundle export command. */
|
||||
const registerExportCommand = (cli: CliInstance, logger: Logger): void => {
|
||||
cli.command(CLI_COMMANDS.environmentExport, {
|
||||
@@ -347,6 +419,7 @@ export const registerEnvironmentCommands = (
|
||||
environment: Readonly<NodeJS.ProcessEnv>,
|
||||
): void => {
|
||||
registerExportCommand(cli, logger);
|
||||
registerCompareCommand(cli, logger);
|
||||
cli.command(CLI_COMMANDS.environmentServe, {
|
||||
description:
|
||||
"Serve an environment manifest as an agent environment over streamable HTTP until SIGINT or SIGTERM: read tools, and write tools whose effect the capture confirmed, answer from the state the capture observed, derived at startup, or every tool replays retained capture records with --replay-only or when the state is unavailable",
|
||||
|
||||
@@ -129,6 +129,7 @@ export const CLI_COMMANDS = Object.freeze({
|
||||
evidenceExport: "evidence-export",
|
||||
environmentServe: "environment-serve",
|
||||
environmentExport: "environment-export",
|
||||
environmentCompare: "environment-compare",
|
||||
compare: "compare",
|
||||
captureProcess: "capture-process",
|
||||
compareProcessCaptures: "compare-process-captures",
|
||||
|
||||
@@ -20,6 +20,10 @@ import {
|
||||
toolName,
|
||||
toolNameRegistry,
|
||||
} from "./environmentManifestNames.js";
|
||||
import {
|
||||
primaryOrigin,
|
||||
type OriginStats,
|
||||
} from "./environmentManifestOrigin.js";
|
||||
import {
|
||||
applyTemplateRules,
|
||||
SEGMENT_LIMITED_RULES,
|
||||
@@ -189,7 +193,7 @@ const hasParam = (parts: readonly MutablePart[]): boolean =>
|
||||
export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumulator {
|
||||
readonly #groups = new Map<string, GroupState>();
|
||||
readonly #records: RecordLine[] = [];
|
||||
readonly #origins = new Map<string, { records: number; jsonLike: number }>();
|
||||
readonly #origins = new Map<string, OriginStats>();
|
||||
readonly #rules: ReadonlySet<TemplateRule>;
|
||||
#unrepresented = 0;
|
||||
#unrepresentableBodyKey = false;
|
||||
@@ -385,24 +389,6 @@ export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumula
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
/** The origin with the most JSON-like records, then most records, then lowest code points. */
|
||||
#primaryOrigin(): string | null {
|
||||
let best: { origin: string; records: number; jsonLike: number } | null =
|
||||
null;
|
||||
for (const [origin, stats] of this.#origins) {
|
||||
if (
|
||||
best === null ||
|
||||
stats.jsonLike > best.jsonLike ||
|
||||
(stats.jsonLike === best.jsonLike &&
|
||||
(stats.records > best.records ||
|
||||
(stats.records === best.records &&
|
||||
compare(origin, best.origin) < 0)))
|
||||
)
|
||||
best = { origin, ...stats };
|
||||
}
|
||||
return best?.origin ?? null;
|
||||
}
|
||||
|
||||
#tools(groups: readonly ToolGroup[]): Tool[] {
|
||||
const registry = toolNameRegistry();
|
||||
return applyTemplateRules(groups, this.#rules)
|
||||
@@ -464,7 +450,7 @@ export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumula
|
||||
|
||||
/** Project the complete manifest; tools, records and exclusions are sorted deterministically. */
|
||||
finish(): Result<ManifestFields, AnalysisError> {
|
||||
const primary = this.#primaryOrigin();
|
||||
const primary = primaryOrigin(this.#origins);
|
||||
const toolGroups: GroupState[] = [];
|
||||
const exclusions = new Map<string, Exclusion & { templated: boolean }>();
|
||||
for (const group of this.#groups.values()) {
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js";
|
||||
|
||||
/** Records and JSON-like records seen at one origin. */
|
||||
export interface OriginStats {
|
||||
records: number;
|
||||
jsonLike: number;
|
||||
}
|
||||
|
||||
/** The origin with the most JSON-like records, then most records, then lowest code points: the application a capture records. */
|
||||
export const primaryOrigin = (
|
||||
origins: ReadonlyMap<string, Readonly<OriginStats>>,
|
||||
): string | null => {
|
||||
let best: ({ origin: string } & OriginStats) | null = null;
|
||||
for (const [origin, stats] of origins) {
|
||||
if (
|
||||
best === null ||
|
||||
stats.jsonLike > best.jsonLike ||
|
||||
(stats.jsonLike === best.jsonLike &&
|
||||
(stats.records > best.records ||
|
||||
(stats.records === best.records && compare(origin, best.origin) < 0)))
|
||||
)
|
||||
best = { origin, ...stats };
|
||||
}
|
||||
return best?.origin ?? null;
|
||||
};
|
||||
@@ -18,12 +18,14 @@ const PAGING_PARAMETERS = new Set([
|
||||
"_end",
|
||||
]);
|
||||
|
||||
/** Whether a query parameter pages or bounds a list rather than filtering it. */
|
||||
export const isPagingParameter = (name: string): boolean =>
|
||||
PAGING_PARAMETERS.has(name.toLowerCase());
|
||||
|
||||
/** Whether a query only pages its list: no parameters, or paging parameters alone. */
|
||||
export const pagesOnly = (query: string | undefined): boolean =>
|
||||
query === undefined ||
|
||||
[...parseUrlQueryParameterNames(query)].every((name) =>
|
||||
PAGING_PARAMETERS.has(name.toLowerCase()),
|
||||
);
|
||||
[...parseUrlQueryParameterNames(query)].every(isPagingParameter);
|
||||
|
||||
/** Paging parameters that bound how many items a page holds. */
|
||||
const PAGE_SIZE_PARAMETERS = new Set([
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { describe, expect } from "vitest";
|
||||
|
||||
import { timedCaptureEvidence } from "../../fixtures/analysisView.js";
|
||||
import { environmentStateServeFixture } from "../../fixtures/environmentServe.js";
|
||||
import {
|
||||
BOOK_NOT_FOUND,
|
||||
BOOK_ROWS,
|
||||
book,
|
||||
} from "../../fixtures/environmentStateWrites.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { cliTest } from "../../support/cli/cliFixture.js";
|
||||
|
||||
/** The books environment and a probe of the application re-seeded under other ids, written as Evidence files. */
|
||||
const writeCompareFiles = async (root: string) => {
|
||||
const fixture = environmentStateServeFixture(timedCaptureEvidence(BOOK_ROWS));
|
||||
const first = book(101, "First Book", "2026-10-20T09:00:00Z");
|
||||
const second = book(102, "Second Book", "2026-10-20T10:00:00Z");
|
||||
const probe = timedCaptureEvidence([
|
||||
["GET", "/api/books", [first, second]],
|
||||
["DELETE", "/api/books/102", "", 204],
|
||||
["GET", "/api/books/102", BOOK_NOT_FOUND, 404],
|
||||
]);
|
||||
const paths = {
|
||||
manifest: join(root, "manifest.json"),
|
||||
capture: join(root, "capture.json"),
|
||||
probe: join(root, "probe.json"),
|
||||
};
|
||||
await writeFile(paths.manifest, JSON.stringify(fixture.manifest));
|
||||
await writeFile(paths.capture, JSON.stringify(fixture.capture));
|
||||
await writeFile(paths.probe, JSON.stringify(probe));
|
||||
return paths;
|
||||
};
|
||||
|
||||
describe("rea environment-compare", () => {
|
||||
cliTest(
|
||||
"compares the served environment with a probe of the re-seeded application and prints the report",
|
||||
async ({ cli }) => {
|
||||
const files = await writeCompareFiles(
|
||||
await createTestTempDirectory("rea-environment-compare-cli-"),
|
||||
);
|
||||
const result = await cli.run({
|
||||
arguments: [
|
||||
"environment-compare",
|
||||
files.manifest,
|
||||
"--evidence",
|
||||
files.capture,
|
||||
"--probe",
|
||||
files.probe,
|
||||
"--json",
|
||||
],
|
||||
timeoutMs: 30_000,
|
||||
});
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(result.json).toMatchObject({
|
||||
status: "complete",
|
||||
environment: { state: "served" },
|
||||
summary: { probes: 1, steps: 3, equal: 3, agreement: 1 },
|
||||
probes: [{ label: "probe.json", ids_mapped: 2 }],
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"refuses a probe that is not capture Evidence and names it",
|
||||
async ({ cli }) => {
|
||||
const files = await writeCompareFiles(
|
||||
await createTestTempDirectory("rea-environment-compare-refusal-cli-"),
|
||||
);
|
||||
const result = await cli.run({
|
||||
arguments: [
|
||||
"environment-compare",
|
||||
files.manifest,
|
||||
"--evidence",
|
||||
files.capture,
|
||||
"--probe",
|
||||
files.manifest,
|
||||
"--json",
|
||||
],
|
||||
timeoutMs: 30_000,
|
||||
});
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.json).toMatchObject({
|
||||
code: "invalid_request",
|
||||
details: {
|
||||
operation: "environment-compare",
|
||||
issues: [{ path: ["probe", 0] }],
|
||||
},
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
Reference in new issue
Block a user