OpenSource/reverse-engineering-anything
1
0 Fork 0
mirror of https://github.com/morluto/rea.git synced 2026-10-26 17:35:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(environment): give more write tools a state target (#6385)

Browse Source 
* fix(environment): keep read-only POSTs out of the write list

A POST that repeats the latest read of its path, tunnels only read
operations, names a lookup in its path's last meaningful word, or renews
a session no longer counts as a state write: it is replayed, does not cut
initial windows, and does not end observation searches. A tool stays a
write when a related read line changes across one of its calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): take an entity an unplanned write made when a read first shows it

A successful item read of an entity the gate's copy never held, after a
write no built plan models, puts the entity into the copy instead of
failing the read, so later writes to it are checked as usual. An entity a
modelled write touched is never taken back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): plan a create into the type its returned id is read under

A POST whose path names no state type becomes a create into the type
whose item read first names an id its response returned, when the id is
new to the state; serving creates in that read's collection. A successful
POST that returned an id a type's reads only show after it also ends that
type's initial window, so entities a capture created stay out of the
initial state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): end a type's initial reads at the write its reads disagree across

When a type's reads disagree across a write under another root, the
type's initial window now ends at the earliest write within that change
and the state is extracted again, instead of leaving every view of the
type replayed. The cut composes with the creation cut; the earlier wins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): replay request lines that show entities in another form

A read line that shows the same entities in another form, by an extra
query parameter that adds or drops members, or conflicts with a base
line on a shared entity, is a variant: it leaves the view's checks and
the entity observations, the view names it with the query keys that pick
it out, and a call matching those keys replays its recording. A filter
role whose name matches no field needs two agreeing values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): specify and record the state-target changes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): keep a read-only POST a write when its evidence is incomplete

Body-level container operations are skipped like query ones, a lookup is named only
by the path's last literal segment and never alongside a write word, a tool with a
redirected or unanswered call is not reclassified, and the mutation veto is resolved
over all candidates at once so another read-only call cannot explain a read change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): answer a field selection only when the state shows it exactly

Serving tests a field selection before any variant key: a projection value a replayed
read gave, or one no checked line gave other than `*`, replays. A list sets aside a
line asking for some fields only before the identity proof, and a dropped variant's
membership change no longer rewrites the type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): hold adopted entities and linked creates to what the recording shows

An adopted entity is charged against the view's budget like a created one. A linked
create holds only for its exemplar's path arguments: serving refuses others and the
gate counts such a call as unmodelled. A response's identity-named id links at any
length, but a short one only to a type read before the call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the review fixes' measurements

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): release a discarded planning pass before planning again

A rewrite cut clears the first pass, so the budget rewinds to a checkpoint taken after
the write cutoffs and created ids before the state is extracted again. Rewrite spans are
charged and appended without spreading them into an argument list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): link no create whose id an earlier read showed, and read only whole batches

A linked create is rejected when an earlier list of its type or read of its collection
showed the returned id. Body ids are walked without spreading children into an argument
list. An RPC call reads only when every operations entry names an operation of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): let the gate skip exactly the reads serving replays

Serving's replay decision moves to the domain as replayReason, and the write gate uses it:
a read serving replays can neither approve nor veto a write, and a read of every field
beside a replayed selection is checked. A selection read missing a required field is
still skipped, since part of an entity can neither confirm nor check it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the second review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): read writes, links and creation cuts only from evidence that rules out other causes

- A read-only POST repeats only a read that finished before it began; the veto compares
  the latest read finished before a call with the first begun after it, and counts a
  recorded absence on one side as a change. `modify` and `replace` are write words.
- A redirected or unanswered write lets a later item read adopt the entity it may have made.
- A creation cut counts a read still running when the POST began as after it, and a list
  read shows its listed entities' own ids, not those they nest.
- A linked create's item read must come before any other tool's write.
- Line shapes are charged to the budget as they are built.
- A `fields=*` list line is whole, not a field selection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the third review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): tie linked creates to their returned ids and order state by the lines left

- A linked create is observed only in a read of an id its response returned,
  envelopes included, and serving takes its recorded path values in either spelling.
- A list read shows a suffix-named identity (`projectId`) distinct across its entities.
- A read-only POST compares the read that finished last before it began.
- An entity's first sighting comes from the lines left after variant lines are dropped.
- The pairwise variant pass stops at a work limit, leaving every line checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the fourth review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): keep foreign keys, stale reads and double decoding out of write evidence

- A list read counts a suffix-named field as its entities' ids only when no
  id-named field identifies them, so an `ownerId` beside `id` cuts no window.
- A read-only POST is compared with the read that finished last among those no
  other write follows, so a consuming call stays a write.
- A linked create's source path values compare as spelled on both sides, so no
  spelling is decoded twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the fifth review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): compare linked sources by position, decode path fields once and bound read and variant scans

- A linked create's source values compare segment by segment, so a template
  naming two parameters alike cannot accept a changed first one.
- Serving fills a linked create's path fields with its arguments decoded once,
  as the gate learned them.
- The read-only check reads each related line's records once per call, between
  the nearest other writes.
- Member signatures are JSON arrays, so a name holding the separator still
  conflicts; the variant work limit counts line pairs as well as entities.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the sixth review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): replay late and variant-only items, fill linked path fields from the exemplar and charge membership spans

- An item line first recorded after the initial window that names a new shaping
  parameter and shows other members is a replayed variant, like an initial one.
- An item only a replayed line showed, which the state never started with,
  answers with its own recording; one the state held and lost stays missing.
- A linked create fills its path fields from its exemplar's path decoded once,
  whichever accepted spelling the call sends.
- Each membership change span is charged against the budget before it is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the seventh review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): decide late item lines against the lines left, charge replay keys and bisect rewrite cuts

- Item lines first recorded after the initial reads are decided once variants
  are dropped, against the lines left: a new shaping parameter showing other
  members, or a selection of some fields leaving members out, replays.
- Each replay key is charged against the budget before it is kept.
- A rewrite span's earliest write is found by bisection over the writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the eighth review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): count only reads finished before a linked create began as showing its id held

A list still running when the call began may show what the call made, as the
creation cut already decides, so it no longer refuses the link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): keep served creates' items missing, check same-type earlier reads, and bound late lines and read-only stretches

- An item only a replayed line showed replays only while no served create has
  added to its type, so a created and deleted item stays missing.
- A linked create's earlier item-read veto checks only reads of the candidate's
  type or collection that show the item, so another type's id does not hide it.
- An item read answered with JSON null shows no id to a creation cut.
- Late lines' member names are charged before they are kept.
- The read-only check precomputes nearest writes and passes once over each
  write-free stretch per path, skipping lines whose reads all agree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the ninth review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(environment): link creates past empty item reads, sweep repeated reads once and keep only returned ids for creation cuts

- A linked create passes over a later item read naming its id whose body shows
  no item, such as JSON null, and links the next one.
- Read-only repeated reads are found in one sweep: each path's reads wait by
  end time until a call's start passes them, and a write clears them.
- A creation cut keeps, and charges, only the shown ids some creation returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the tenth review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(environment): compare variant lines through shared entities and parse each compared body once

The variant pass indexes a tool's lines by the entities they show and compares only lines that share one, charging
the index to the budget while it is held; the pairwise work limit still decides when a tool keeps every line checked.
Read-only decisions keep each response body's canonical digest, taken once while its parse is charged, across calls
and veto passes. The creation cut no longer parses reads at or after the write that ends the initial reads, and reads
the ids a body shows inside its lease. Adoption looks up a read's target only after an unplanned write before it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the eleventh review round's checks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(environment): list a type's collections only when the path's own holds no id, and parse variant base lines once

The write gate's item target lists the type's held collections, and the removed targets', only when the path's own
collection holds none of its ids, so a served item view's reads no longer walk every entity of the type. The variant
pass parses the base lines' queries once for all the variants they key, and not at all when it found none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(environment): record the twelfth review round's checks and the sweep's cost

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
Edgar ArakelyanClaude Opus 5.5Kaoru0822
 authored and GitHub committed 2026-10-24 15:58:32 +08:00
1 parent 4a041ae753
commit 977c9727e6
41 files changed
+5404 -236
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/agent-environments.md
+12 -9
View File
@@ -431,12 +431,12 @@ member reports which applied:
Each read tool answers according to its view in the state:
| View | Answer |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `item` | The entity the path arguments name, rendered into the recorded envelope of `envelope_ordinal`. Arguments match as replay matches them: each collection segment equals the held collection's segment as written or after percent-decoding, and the identity is the last argument percent-decoded, then as written; values are never re-split on `/`. Any entity the state holds answers, including one `data/add` loaded; query arguments only decide `replay.exact_match`. An entity the state does not hold answers the tool's latest recorded 400, 404 or 410 response with `state.view` `missing`, or is a tool error when none was recorded; one lacking a field every recording carried is a tool error, unless a read showed the application leaves that field out; a body is never invented. |
| `list` | When the arguments name one of the lines the view checked exactly (its latest recordings before the first related write; every path and query value equal to the recording's, and every recorded query parameter supplied), that recorded envelope with its array rendered from the state: every entity of the collection in state order for `membership: "all"` or a line `all_record_ordinals` names, or the recorded members still held. A call with no query argument answers from such a line of the collection its path arguments name that only pages its list, with `replay.exact_match` false. On a line the view's `query` answers, and for a call whose every query argument has a role there (an argument-less call aside), the answer is the query evaluated over the state: filtered, searched, sorted and paged, with counters at the number listed and `query.totals` at the number matched, and `state.membership: "query"`; such a call answers on a query or whole line of its collection with `replay.exact_match` false. A value an `inert` parameter never recorded, or a parameter without a role, replays its recording. A member lacking a field every recording carried makes the call a tool error rather than a partial list. |
| `document` | When the arguments name the document's checked line exactly, the state's value for it; a tool error when the state holds none. |
| `recorded` | The recording, as replay answers it. |
| View | Answer |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `item` | The entity the path arguments name, rendered into the recorded envelope of `envelope_ordinal`. Arguments match as replay matches them: each collection segment equals the held collection's segment as written or after percent-decoding, and the identity is the last argument percent-decoded, then as written; values are never re-split on `/`. Any entity the state holds answers, including one `data/add` loaded; query arguments only decide `replay.exact_match`, except that a call matching one of the view's `projected_query_keys`, or on a view with `projected_record_ordinals` asking for some fields only with a value other than `*` no checked line gave or one a replayed read gave, replays its recording. An entity the state does not hold answers the tool's latest recorded 400, 404 or 410 response with `state.view` `missing`, or is a tool error when none was recorded; one lacking a field every recording carried is a tool error, unless a read showed the application leaves that field out; a body is never invented. |
| `list` | When the arguments name one of the lines the view checked exactly (its latest recordings before the first related write; every path and query value equal to the recording's, and every recorded query parameter supplied), that recorded envelope with its array rendered from the state: every entity of the collection in state order for `membership: "all"` or a line `all_record_ordinals` names, or the recorded members still held. A call with no query argument answers from such a line of the collection its path arguments name that only pages its list, with `replay.exact_match` false. On a line the view's `query` answers, and for a call whose every query argument has a role there (an argument-less call aside), the answer is the query evaluated over the state: filtered, searched, sorted and paged, with counters at the number listed and `query.totals` at the number matched, and `state.membership: "query"`; such a call answers on a query or whole line of its collection with `replay.exact_match` false. A value an `inert` parameter never recorded, a parameter without a role, a call matching one of the view's `projected_query_keys`, or a field selection the checked lines do not show (as for `item`), replays its recording. A member lacking a field every recording carried makes the call a tool error rather than a partial list. |
| `document` | When the arguments name the document's checked line exactly, the state's value for it; a tool error when the state holds none. |
| `recorded` | The recording, as replay answers it. |
A read with repeated query values uses replay, because the state query model
does not infer repeated-parameter semantics. A repeated-query write is refused
@@ -456,13 +456,16 @@ A list or document call that names no checked line, including a line recorded
after a write or with other query values, answers from the recording. Every
answer has the replay shape plus `state`: `{"source": "state", "view": …}`
naming the entity, collection or document path, or `{"source": "recording",
"reason": …}` with the view's reason, `no-view`, `unrecorded-request`, `page-exceeded` (a call naming a page of a whole list that the collection has outgrown) or `projection` (a call to an item tool whose recordings asked for some fields only, naming such a parameter for an entity the state holds; the answer is a recording, so it does not show served writes). A
"reason": …}` with the view's reason, `no-view`, `unrecorded-request`, `page-exceeded` (a call naming a page of a whole list that the collection has outgrown), `projection` (a call to an item or list tool with replayed reads asking for a selection of fields the checked lines do not show, or leaving out a projection parameter every checked line names) or `variant` (a call matching another `projected_query_keys` key: a parameter, or a value of one, only lines showing the entities in another representation named, or leaving out a parameter every other line named). A `projection` or `variant` answer is a recording, so it does not show served writes. A
state answer carries the base record's status, content type and `replay`
provenance with `truncated: false`; one whose JSON would exceed 256 KiB is
refused as a `resource_constraint` tool error rather than cut. Extraction does
not state-back a document whose rendered JSON exceeds 256 KiB; that read stays
recorded (`rendered-oversize`) and replays the captured body. Query `POST`s
stay served and answer from their recordings with the reason `query-body`.
recorded (`rendered-oversize`) and replays the captured body. Query `POST`s,
and `POST` tools the capture shows only reading (a lookup answering as a `GET`
did, an RPC tunnel naming read operations, a session renewal), stay served and
answer from their recordings with the reason `query-body`; they are not write
tools.
A line establishes whole-collection membership only from the beginning:
`offset`, `start`, `_start` and `page` must be zero when present. Positive or
docs/specs/agent-environments/list-queries/spec.md
+3 -1
View File
@@ -44,7 +44,9 @@ filter or inert. No two parameters of one call take the same `sort`, `direction`
depend on which applied last, and serving supplies a call's values in name order rather than as recorded, so such a
combination explains no line and such a call replays its recording. A line with more than 12 query parameters is left
unexplained, which bounds the search; names such as `__proto__` are kept as the query's own members. The lines' roles
merge: lines that disagree drop the parameter, inert values gather, search fields unite. The merged query keeps the
merge: lines that disagree drop the parameter, inert values gather, search fields unite. A filter on a field not
named like its parameter (`shelf`, or `shelf_id`, for `shelf`) can match by coincidence, so it stands only when lines
giving the parameter two different values agree on it; otherwise the parameter takes no role. The merged query keeps the
lines it reproduces, envelope and fields included; any other line keeps its recorded members. The view's `query` names
the roles, those lines, and the envelope members that count every match before paging (`totals`); a `number`-of-items
member such as `numberOfElements` counts the page.
docs/specs/agent-environments/state-targets/plan.md
+14
View File
@@ -0,0 +1,14 @@
# Implementation plan
1. **Read-only POSTs.** A module decides once per extraction which POST tools only read, used by the write filter,
init cutoffs and the write simulation.
2. **Adoption.** The write simulation tracks held entities and writes no built plan models, and takes an unseen
entity at its first successful item read.
3. **Linked creates.** A module links a POST's returned id to the state type it is first read under; the extraction
cuts that type's initial window at the POST, and the create plan records the read's collection.
4. **Rewrite cuts.** The merge reports the spans behind a type's rewrite; the extraction cuts the type's initial window
at the earliest write within them and plans once more.
5. **Variant lines.** A module classifies request lines by the members their objects show; plans drop variants, views
name them with their query keys, and serving and the write gate replay calls that match.
6. **Tests and docs.** Each step's tests fail on the step before; the state-view, state-writes, list-queries and
write-coverage specs and the environment docs describe the new behaviour.
docs/specs/agent-environments/state-targets/spec.md
+66
View File
@@ -0,0 +1,66 @@
# Give more write tools a state target
Issue: https://github.com/morluto/rea/issues/4933
## Outcome
Across the 50 published capture-built environments, 91 of 195 write tools got no state target: the gate refused them
before looking at their evidence, because their path named no state type, item or document. A diagnosis traced each to
its cause in the recordings and the code. The five largest shared causes:
- **Read-only POSTs counted as writes.** A lookup, an RPC tunnel naming read operations, or a session refresh was
planned as an unserved write and cut initial windows.
- **Entities a write the gate does not model made.** A later read of such an entity failed its check instead of
holding it, so writes to it stayed unobserved.
- **Creates read under another path.** A create whose path is no state type (`POST /Playlists` read as `/Items/{Id}`)
had no target, and its entity leaked into the initial state of the type it is read under.
- **Types whose reads disagree across an unrelated write.** Every view of the type stayed replayed.
- **Request lines that show entities in another form.** An extra query parameter that adds or drops members
(`?counts=true`, `?expand=`, `?fields=`) failed the whole view.
## Behaviour
- **Read-only calls.** A POST is not a state write when its response repeats the read of its path that finished last
before it began, with no write between, it tunnels only read operations, its last literal segment names a lookup and
no write, or it renews a session; it is replayed. It stays a write when one of its calls was redirected or unanswered,
or when a related read line changes across one of its calls, or finds its entity there on one side only, with no write
but other read-only calls between; a read overlapping the call counts on neither side. Each response body compared is
parsed once and kept only as its canonical digest, charged to the view's budget until the decision ends.
- **Adopted entities.** A successful item read of an entity the gate's copy never held, after a write no built plan
models or one redirected or unanswered, puts the entity into the copy within the view's budget; an entity a modelled
write touched is never taken back.
- **Linked creates.** A POST whose path names no state type becomes a create into the type whose item read first names
an id its response returned, before any other tool's write, when the id is new: no read finished before it began
showed it, whether an item read of the id under that type or collection whose body shows the item, a list of the type
or a read of that collection. Its observation is a read of an id its response returned, in an envelope too. Serving
creates in that read's collection, for the exemplar's path arguments only (segment by segment, in either spelling),
and fills path fields with its exemplar's path values decoded once. A successful POST that returned an id a type's
reads only show once it began ends that type's initial window: a read still running then may show it, and a list read
shows its listed entities' own ids (identity-named, or a suffix-named field such as `projectId` distinct across two or
more when no id-named field identifies them), not those they nest. A read at or after the write that ends the initial
reads is not parsed for that cut, since it cannot show a new id.
- **Rewrite cuts.** When a type's reads disagree across a write under another root, the type's initial window ends at
the earliest write within the change and the state is extracted again. The earlier of this and the creation cut wins.
- **Variant lines.** A read line that shows the same entities in another form is a variant: it leaves the view's checks,
the entity observations (an entity's first sighting comes from the lines left) and the membership changes that cut a
type's initial window, and the view names it with the query keys that pick it out (`projected_query_keys`, on item and
list views). Beside a line showing whole entities, a list line asking for some fields only (a projection value other
than `*`) is set aside before the identity proof the same way. A call matching those keys replays its recording, so it
does not show served writes; a 400, 404 or 410 on such a line is still evidence. A call asking for some fields only
replays unless every value it gives is `*` or one a checked line gave, and no replayed read gave it. The write gate
skips exactly the successful reads serving replays, by the same decision. A filter role whose name matches no field
needs two agreeing values. A tool whose lines would take more than 2^24 line pairs and entity lookups to compare
pairwise keeps every line checked; otherwise lines are compared only with those showing one of the same entities,
through an index charged to the budget while it is held. The base lines' queries are parsed once for every variant
they key. An item line first recorded after the initial window is a variant too, judged against the lines left once
variants are dropped: when it names a new shaping parameter and its object carries or lacks members theirs do not, or
selects some fields and leaves out members they always show, and an item only a replayed line showed, which the state
never started with and in whose type no served create has added an entity since, answers with its own recording.
## Acceptance
- Over the 50 published captures, write tools with no state target fall from 91 to 61 and served write tools rise from
46 to 61, with none lost; every served write tool applies.
- Read calls recorded before the first write that state serving answers keep matching their recordings; each newly
state-backed view answers them as recorded, except where replay itself does not.
- The Grafana and Mailpit environments no longer hold, in their initial state, entities their captures created.
docs/specs/agent-environments/state-targets/tasks.md
+10
View File
@@ -0,0 +1,10 @@
# Tasks
- [x] Keep read-only POSTs out of the write list.
- [x] Take an entity a write no plan models made when a read first shows it.
- [x] Plan a create into the type its returned id is read under, and cut that type's initial window.
- [x] End a type's initial reads at the write its reads disagree across.
- [x] Replay request lines that show entities in another form.
- [x] Add tests that fail on the step before each change.
- [x] Measure the 50 published captures, the served smoke and the serving of recorded reads after each step.
- [x] Run static, formatting, duplication, documentation and test gates.
docs/specs/agent-environments/state-targets/verification.md
+98
View File
@@ -0,0 +1,98 @@
# Verification
Executed on macOS with Node v22.23.1, one change at a time on main, over the 50 published capture-built environments
(each app's latest capture). The served smoke rebuilds each manifest with the tree under test and applies every served
write tool; the serving check answers every recorded read made before the capture's first write through state serving
and compares it with the recording. The serving check reads no records from the five browser-scenario captures (Art
Institute collection, LoC photos, Lychee, PhotoPrism, Pinry), so its read counts cover 45 apps; their 41 views count
among the state-backed views.
## Gates
- `npm run build:cached`, `npm run check:fast` (typecheck and lint, no new warnings in the changed files),
`npm run format:check`, `npx knip`, `npm run scan:todos`, `npm run docs:check`, `npm run verify:test-discovery` and
`npm run mcp-catalog:check`: passed.
- `npm run jscpd`: 3.49% duplicated lines over 4074 files.
- `npx vitest run --project domain --project services`: 8992 passed, 6 pre-existing skips, on main `ed069bb9`.
- `npm run test:prepare`, then `npm run test:focused --` the environment serve and export CLI boundaries, the serve
and served-writes MCP boundaries, the analysis-view CLI and MCP boundaries, the compact input schema boundary and
tool schema validity: 8 files, 88 passed.
## Measurements
| Step | Write tools | Served | No state target | State-backed read views | Recorded reads matching |
| --------------- | ----------- | ------ | --------------- | ----------------------- | ----------------------- |
| main | 195 | 46 | 91 | 1,098 | 1,833 of 1,886 |
| read-only POSTs | 189 | 46 | 84 | 1,104 | 1,911 of 1,964 |
| adoption | 189 | 47 | 84 | 1,104 | 1,911 of 1,964 |
| linked creates | 189 | 48 | 80 | 1,105 | 1,911 of 1,964 |
| rewrite cuts | 189 | 51 | 77 | 1,110 | 1,922 of 1,975 |
| variant lines | 189 | 61 | 61 | 1,150 | 2,068 of 2,126 |
| review fixes | 189 | 61 | 61 | 1,151 | 2,075 of 2,133 |
| third round | 189 | 61 | 62 | 1,151 | 2,075 of 2,133 |
The served smoke applies every served write tool at each step. No read view loses its state backing, and no served
write tool is lost.
- **Read-only POSTs:** six tools leave the write list (Mattermost `members/ids`, Miniflux `discover`, NocoDB's two
internal tunnels, Paperless `selection_data`, PocketBase `auth-refresh`); Paperless's notes create gains a target.
Grafana's `query-history` stays a write. Nomad `jobs/statuses` repeats its GET, but its integers exceed JSON's
exact range, so it stays a write.
- **Adoption:** Grocy's product delete is served; Mealie's recipe delete moves from no observation to unconfirmed.
- **Linked creates:** Jellyfin's item delete is served; its playlist create and item update, and Mailpit's send, gain
targets. Grafana's `/api/search` and Mailpit's messages no longer hold the dashboard and message their captures
created.
- **Rewrite cuts:** Tandoor's recipe create, update and delete are served.
- **Variant lines:** Baserow's row create, update and delete, Keycloak's user delete, Miniflux's category create and
update, PocketBase's create, update and delete, and Vikunja's task delete are served. Ghost's post create and update,
Keycloak's user update, Miniflux's category delete, n8n's tag create and Vikunja's task update gain targets.
- **Review fixes:** no write verdict changes. A NocoDB records list whose `fields` line had rejected it becomes
state-backed, and all 7 of its recorded reads match. Without its guard, a short id would cut 23 Tandoor views'
initial windows at a `POST /api/view-log/` returning `{"id":1}`.
The second round (the gate skipping exactly the reads serving replays, earlier reads vetoing a linked create, whole
RPC batches, and releasing a discarded planning pass) changes no measurement.
- **Third round:** Mailpit's send, a linked create that was never observed, loses its target: other writes (marking
messages read, tagging) ran between it and the item read naming its id. No served write, read view or matching read
changes.
- **Fourth round:** no verdict, read view or recorded read changes. A purge of temporary files removed 32 of the local
captures, so this round was measured on captures restored from the published datasets, byte-identical where a local
copy survived; Baserow's, Grocy's and Keycloak's published captures predate the ones measured above. On that set the
previous head and this one both give 187 write tools, 62 served (smoke 62 of 62), 1,151 views and 2,085 of 2,155
recorded reads matching.
- **Fifth round** (on main `6a34be41`): no verdict, read view or recorded read changes on the same restored captures,
once the serving check passes a repeated query value as an array, as main now expects.
- **Sixth round:** no verdict, read view or recorded read changes on the restored captures.
- **Seventh round** (on main `96ea32b7`): WordPress's post create moves from side-effect to unconfirmed, still unserved:
its only later read, `/wp-json/wp/v2/posts/22?context=edit`, shows the edit representation (`generated_slug`), so it
is now replayed rather than checked. No served write, read view or recorded read changes.
- **Eighth round** (on main `3e7e102a`): no verdict, read view or recorded read changes on the restored captures.
- **Ninth round** (on main `b7ebccf8`): no verdict, read view, recorded read or session replay changes on the restored
captures (187 write tools, 62 served, 1,151 views, 2,085 of 2,155 initial reads and 2,764 of 3,022 session reads
matching).
- **Tenth round** (on main `a2235b09`): no verdict, read view, recorded read or session replay changes on the restored
captures.
- **Eleventh round** (on main `2968d1e9`, performance review): variant lines compare only lines that share an entity,
read-only decisions parse each body once and keep its digest, and the creation cut skips reads that cannot count.
Every census, serving-check and session-replay row equals the previous head's. On a synthetic capture of a list and
1,000 items read plainly and with `?expand=`, the variant pass falls from 401 ms to 2 ms and the state build from 659
ms to 216 ms (Node v22.23.1).
- **Twelfth round** (on main `d87e73f2`, second performance review): the write gate lists a type's collections only when
the path's own collection holds no id, and the variant pass parses base lines once. Every census, serving-check and
session-replay row equals the previous head's. The synthetic 1,000-item capture takes 259–264 ms against main's
244–247 ms (378–397 ms before this round; Node v22.23.1, medians of seven warm runs). Over the 50 captures, the
environment-state and environment-manifest views together take 2.96 s wall and 3.82 s CPU with a 540 MiB peak RSS,
against main's 2.61 s, 3.41 s and 516 MiB (one Node v24.22.1 process per sweep, warm medians of three runs; the cold
first run is within 2% of them); the environment-state view alone takes 2,093 ms against 1,791 ms, and its results
grow from 3.69 MB to 3.84 MB. Agent tokens were not measured.
Of the 40 views variant lines make state-backed, 146 of 151 recorded reads before the first write match; the other five
(Sentry projects) are replayed, and replay returns the first of several differing recordings, as before. A paged
Paperless document list answers all five documents, a PocketBase item without `expand` answers without it, and a
NetBox `site_id` filter learned from one line is not adopted.
## Not included
- Making a value that differs between two request lines volatile: it contradicts the spec's distinct query strings and
made 11 recorded reads answer otherwise than replay.
- Item actions (`/{type}/{id}/archive`), bulk writes at a collection path, templating literal ids by value, and
envelope masking: each needs further recording evidence or a separate change.
docs/specs/agent-environments/state-view/spec.md
+50 -8
View File
@@ -25,16 +25,56 @@ later increments.
any start time is missing, and say which. Bound each type's state by the first
write under its root (API base, version segments, one segment); a query
`POST` is not a write, a read overlapping the write is left out, and a write
whose URL was redacted bounds every type, with a limitation counting it.
whose URL was redacted bounds every type, with a limitation counting it. A
successful `POST` elsewhere bounds a type's state too when its response named
an id (an identity member, or a bare or sole scalar of four characters or
more) that the type's reads show only once it began: the `POST` made that
entity, so it is not in the initial state. A read finished before the `POST`
began shows what was there, and one still running may show its effect. A list
read shows the ids its listed entities carry as their own identity members, an
item read its path's id only when its body shows the item (a JSON `null` shows
none), and
across two or more of them, when no id-named field (`id`, `uuid`, …; not `key`,
`slug` or `name`) identifies them, the first suffix-named field (`projectId`)
distinct on each, so a foreign key beside an `id` is none of theirs, nor are the
ids of entities they nest. An id shorter than four characters counts only
when the type was read before the `POST` and an item read names it after.
- Prove an item type's identity field from the requested path value its
responses repeat; otherwise infer it from a fixed list of identity names that
every listed item carries distinctly, which needs at least one listed item.
An item read that asked for some fields only (`fields`, `fields[...]`,
`select`, `$select`) and does not return the item is left out of the proof
and of the view's checks; the view lists it in `projected_record_ordinals`,
and serving replays any call to that tool that names a projection parameter
for an entity the state holds. The write gate also leaves out any read that
names one and leaves out a field its view requires.
and of the view's checks; the view lists it in `projected_record_ordinals`.
Beside a line showing whole entities, a list line asking for some fields only
(a projection value other than `*`, which asks for all) is listed there too,
and proves no identity or membership. So is an item line first recorded after
the type's initial reads, judged against the lines left once variants are
dropped: one that names a shaping parameter none of them named while its
object carries or lacks members theirs do not, such as a later
`?expand=shelf` read, or one that selects some fields and leaves out members
they always show, such as a later `?fields=id,name` read. On a view with
such replayed reads, serving replays a call naming a projection parameter for
an entity the state holds unless every value it gives is `*` or one a checked
line gave, and no replayed read gave it. The write gate leaves out exactly
the successful reads serving replays, and any read that names one and leaves
out a field its view requires, which serving would answer whole.
- A request line of an item or list view is a representation variant when it
shows an entity another line shows with other members, or names a parameter
the base lines do not (paging aside) while its objects carry a member no base
object carries or lack one every base object carries. Base lines are taken
first: lines naming no such parameter, then lines asking for every field,
then the line most others agree with. A variant leaves the view's checks, the
entity observations and the membership changes that would end the type's
initial reads when a query key tells it apart: a parameter only
variants name, a value no base line gives a parameter, or a parameter every
base line names that a variant leaves out. The view lists its latest
recordings in `projected_record_ordinals` and the keys in
`projected_query_keys`; serving replays a call matching a key with the reason
`variant` (`projection` when it leaves out a projection parameter every base
line names; a projection value is tested as above), so the
answer is a recording and shows no served write. The write gate neither
checks nor learns from a successful read serving replays; a 400, 404 or 410
such a read answers is still the absence it checks.
- A field every recording of a view carried is required: serving refuses an
entity lacking it. Once a successful read the write gate checks, naming no
projection parameter, shows the application leaves the field out of an entity
@@ -45,8 +85,9 @@ later increments.
latest values win over list views', differing views keep their own values,
and a value that changed between repeats of one request line with no recorded
write is volatile and keeps the latest; a change across a write under another
root, including between two tools of the type, leaves every tool of the type
recorded. Compare list membership, including empty lists, and field presence before selecting each request line's latest recording; a change across a write also leaves the type recorded. Different query strings remain distinct; omitted fields do not erase their earlier observations on other request lines.
root, including between two tools of the type, ends the type's initial reads
at the earliest write within the change, and the state is extracted again
once; a type whose reads still disagree keeps every tool recorded. Compare list membership, including empty lists, and field presence before selecting each request line's latest recording; a change across a write ends the type's initial reads the same way. Different query strings remain distinct; omitted fields do not erase their earlier observations on other request lines.
- An item or list view whose recordings always spell a clock field's instant
otherwise than the entity's stored value (`Z` against `+00:00`, another
separator or fraction width) names that spelling in `clock_formats`; a field
@@ -68,7 +109,8 @@ later increments.
semantics and write effects.
- Charge the manifest's admission and grouping, every record line, every body
as temporary bytes before decoding and parsing, and every kept value, within
the existing 96 MiB budget.
the existing 96 MiB budget. A planning pass that a rewrite cut discards
releases its charges before the state is extracted again.
- Read response bodies through one domain owner shared with replay, and parse
exact JSON through the owner of the exact-number policy.
docs/specs/agent-environments/state-writes/spec.md
+80 -1
View File
@@ -31,9 +31,49 @@ a reset give the same state.
The view adds two members to its result:
- `writes`: one plan per manifest tool whose method may change state
(`changesState`);
(`changesState`), less the `POST` tools the capture shows only reading (below);
- `write_clock`: the instant the served clock starts from.
**Calls that only read.** A `POST` tool whose calls read state is no write: it
gets no plan, stays a recorded view with the reason `query-body`, is served by
replay, and its calls bound no initial-recording window and end no observation
search. The capture shows it by the first of these that applies:
1. every successful call answers as the successful `GET` of its path that
finished last before the call began, of those with no write between, did (a
status lookup posting the jobs it names);
2. every call names read operations only, in its body's `operation` or
`operations[].operation`, else its query's `operation`, `op` or `action`
(`batch` and similar only wrap them, in either place, so a call needs at
least one other, and every `operations` entry must name one of its own, so
an entry with no string operation or another container keeps the call a
write); an operation reads when its words name one of `list`,
`get`, `count`, `aggregate`, `info`, `search`, `find`, `read`, `query`,
`schema`, `describe`, `fetch` or `lookup`, and none of `create`, `insert`,
`update`, `modify`, `replace`, `delete`, `remove`, `upsert`, `add`, `set`,
`save` and similar, so `findAndModify` and `findOneAndReplace` stay writes;
3. the path's last literal segment has a query word as its last meaningful
word, split on `_`, `-`, `.` and case and skipping trailing `data`, `info`
and `results` (`search`, `query`, `list`, `find`, `lookup`, `count`,
`aggregate`, `stats`, `ids`, `selection`, `discover`, …), and no write word:
`selection_data` names a lookup, while `query-history` names a history,
`create_lookup` a create and `selection_data/import` an import, and each
stays a write;
4. the path's last literal segment renews or opens a session: `auth-refresh`,
`auth-with-password`, `login`, `logout`, `refresh` or `token`.
A tool stays a write when one of its calls was redirected or went unanswered,
since nothing shows what it did. It also stays a write when a `GET` line whose
path its call's path relates to (segment by segment, a parameter matching any
value) reads differently just before and just after one of its calls, or finds
its entity on one side only (a 400, 404 or 410 on the other), with no other
write between; the other read-only tools' calls are no writes there, so the
tools are dropped until none is vetoed. Those reads are the one that finished
last before the call began, of those no other write follows, and the first that
began after it ended, since one overlapping the call may show either state. Reads of unrelated paths do not
count, so a session renewal stays read-only although an endpoint under the
same root regenerates values on every read.
**Effects.** Each plan classifies its tool:
| Effect | Tool | Target |
@@ -44,6 +84,30 @@ The view adds two members to its result:
| `document` | `PUT`, `PATCH` or `POST` to a document view's path | that document |
| `none` | anything else | reason `no-state-target` |
A `POST` whose path gives it no target is a `create` into the type its returned
id is read under: its exemplar's response names an id (an identity member of the
body or of an object member, or a scalar body or sole member of four characters
or more), and the first later successful item read naming that id, whose body
shows the item (a JSON `null` read is passed over), belongs to a state type. That read must come before any other tool's write, which may have
made what it shows, as a publish call makes a note from a draft. The entity must
be new: neither the initial state nor any read finished before the call began
holds that id (one still running may show what the call made): an item read of
it under the item read's type or collection whose body shows the item (a JSON
`null` is an absence), a list read of its type, or any read of that collection, whether an
item read of it, a list read of its type, or any read of the item read's
collection, with or without a query, that lists it under an identity member of
its own (a body with no list counts an identity member at any depth). An id shorter than four characters may be any type's,
so it links only to a type a successful read showed before the call. The plan
records that read's collection, and serving creates there, since the call's own
path names no collection of the type. That collection holds only for the
exemplar's path arguments, compared segment by segment, each as recorded or in
the other spelling once percent-decoded: serving refuses a call with others, and the gate counts such a
recorded call as a write no built plan models. Its path fields take the
exemplar's path values decoded once, the source it accepts, whichever spelling
the call sends. A linked create's call is
observed only in a read of an id its response returned, at the top or in an
envelope member, so another unseen item's read is never taken for it.
**Exemplar.** A plan is built from its first successful (2xx) recorded call,
the exemplar.
@@ -228,6 +292,21 @@ unknown. Writes that changed it since its last check are checked only on the
members they sent, less those the view treats as volatile, which show nothing
of a write.
A write no built plan models can also make an entity the copy never held, such
as a row an import or a nested create added: a write of a tool with no create,
update, delete or document plan, a call of a tool whose plan was never built, a
linked create's call that serving would refuse, or any write redirected or left
unanswered.
A successful item read of an entity the copy has never held (no initial entity,
and none a modelled write created, changed or removed) that names it by the
type's identity, when such a write ran since that entity or its collection was
last checked, shows that entity: the copy takes its recorded values instead of
failing the read, and later writes to it are modelled and checked as usual. The
taken entity counts against the view's budget as a created one does. An
entity a modelled write touched is never taken back this way, and a built plan's
call that serving would not apply, such as a create outside its tied
collection, keeps its evidence.
**Reads.** A read is checked when the served environment would answer it
from the state:
docs/specs/agent-environments/write-coverage/spec.md
+7 -2
View File
@@ -31,11 +31,16 @@ what the writes did:
is left out of the identity proof and the view's checks. The view lists it in `projected_record_ordinals`, and serving
replays a call to that tool naming such a parameter, for an entity the state holds, with the reason `projection`;
the replay shows the recording, not served writes. The write gate neither checks nor learns a write's effect from a
successful read that names such a parameter and leaves out a field its view requires; a 400, 404 or 410 such a read
answers is still the absence it checks.
successful read serving replays, nor from one that names such a parameter and leaves out a field its view requires; a
400, 404 or 410 such a read answers is still the absence it checks.
Identity-free projection evidence after the initial-state cutoff is retained too, without seeding later values
into initial entities. Failed responses are classified before the projection-replay shortcut, so a projected
absence can confirm a delete or contradict an unexplained disappearance.
A list line asking for some fields only beside one showing whole entities is replayed the same way and proves no
identity or membership; an item or list line showing its entities in another representation (an expanding or
reducing query) is replayed when a query key tells it from the view's other lines (`projected_query_keys`, reason
`variant`). A call asking for some fields only replays unless every value it gives is `*` or one a checked line
gave, and no replayed read gave it; such calls replay recordings and do not show served writes.
- **Sequences.** A sequence value may be `-`-separated digit groups of one width of at least two digits, not a date,
version or address. The next value keeps the exemplar's groups and zero padding; digits that outgrow them lead. A
later create must record its value spelled that way. A held value whose digits outgrew the mask (`100-00` after
docs/web-network-captures.md
+17 -5
View File
@@ -652,9 +652,14 @@ collection path: an item view's latest values win over a list view's, a view
whose latest values differ keeps them in `view_fields`, and a field whose value
changed between one tool's observations with no recorded write between is named
in the type's `volatile_fields`, keeping the latest value. A value that changed
across a write under another root may be that write's effect, so every tool of
its type is a recorded view with reason `fields-differ`; so is a value one tool
read before a write and another tool read after it. Repeated reads of one full request line also keep the type recorded when list membership changes across a write, including transitions to or from an empty list, or when entity fields are added or removed. Different query strings remain distinct request lines. Sparse responses do not erase the last observation of a field supplied by another request line. Entities are ordered
across a write under another root may be that write's effect, so the type's
initial reads end at the earliest write within the change, reads that overlap
it are left out, and the state is extracted again once; the same holds for a
value one tool read before a write and another tool read after it, and for
repeated reads of one full request line whose list membership changes across a
write, including transitions to or from an empty list, or whose entity fields
are added or removed. A type whose reads still disagree keeps every tool a
recorded view with reason `fields-differ`. Different query strings remain distinct request lines. Sparse responses do not erase the last observation of a field supplied by another request line. Entities are ordered
by type, collection and first sighting; that order is a list's state order.
Every entity cites its `record_ordinals`.
@@ -670,9 +675,16 @@ Every entity cites its `record_ordinals`.
An item or list plan is kept only when it reproduces the latest initial
recording of each request line from the state, comparing JSON values without
regard to member order; a volatile field compares its recorded value. Each
plan lists the `checked_record_ordinals`. Otherwise the tool is a recorded view
plan lists the `checked_record_ordinals`. A request line that shows the
entities in another representation (an expanding or reducing query, or one
asking for some fields only) is left out of those checks when a query key tells
it from the others: the plan lists its latest recordings in
`projected_record_ordinals` and the keys in `projected_query_keys` (`name`,
with a `value` or `absent: true`), and serving replays calls matching them and
calls asking for fields the checked lines do not show.
Otherwise the tool is a recorded view
with reason `envelope-varies`, `fields-differ` or `list-differs`; the other
reasons are `query-body` (a query `POST`), `no-init-recording`,
reasons are `query-body` (a query `POST`, or one the capture shows only reading), `no-init-recording`,
`body-unusable`, `no-identity` (an item read without a proved identity),
`path-varies` (another read under more than one path, as a template with a
parameter before its end), `query-varies` (another read of one path under
src/application/environment/environmentState.ts
+12 -6
View File
@@ -51,16 +51,22 @@ export interface EnvironmentState {
readonly writeClock: string | null;
}
/** The write plans a catalog serves from: none before a state is derived. */
type ServedWrites = Pick<EnvironmentState, "writes">;
/** The write plans a catalog serves from, and the views naming calls that only read: none before a state is derived. */
type ServedWrites = Pick<EnvironmentState, "writes"> &
Partial<Pick<EnvironmentState, "views">>;
/** Serving from the state serves read tools and the write tools whose effect the capture confirmed. */
/** Serving from the state serves read tools, calls the capture shows reading only, and the write tools whose effect it confirmed. */
const servedFromState = (
tool: EnvironmentManifestTool,
state: ServedWrites,
): boolean =>
!changesState(tool.method, tool.path_template) ||
state.writes.get(tool.name)?.served === true;
): boolean => {
const view = state.views?.get(tool.name);
return (
!changesState(tool.method, tool.path_template) ||
(view?.shape === "recorded" && view.reason === "query-body") ||
state.writes.get(tool.name)?.served === true
);
};
/** The tools an environment serves and the data plane its card declares. */
export interface ServedCatalog {
src/application/environment/environmentStateAnswer.ts
+52 -20
View File
@@ -1,12 +1,8 @@
import {
environmentToolArguments,
hasRepeatedQueryArgument,
} from "./environmentTools.js";
import { hasRepeatedQueryArgument } from "./environmentTools.js";
import { stateBodyJson } from "../../domain/analysisView/EnvironmentStateAccumulator.js";
import {
collectionMatch,
collectionPathOf,
isProjectionParameter,
decodedSegment,
allLines,
listItems,
@@ -25,6 +21,7 @@ import {
type StateView,
} from "../../domain/analysisView/environmentStateRules.js";
import { pageBound } from "../../domain/analysisView/environmentStatePaging.js";
import { replayReason } from "../../domain/analysisView/environmentStateVariants.js";
import { isAbsence } from "../../domain/analysisView/environmentStateWriteChecks.js";
import {
evaluateQuery,
@@ -72,7 +69,8 @@ export type StateProvenance =
| "query-multiplicity"
| "unrecorded-request"
| "page-exceeded"
| "projection";
| "projection"
| "variant";
}
| {
readonly source: "state";
@@ -154,16 +152,16 @@ const fromRecording = (
: replayed;
};
/** Whether a call names a projection parameter of an item view with projected reads, which the state's whole entities cannot answer. */
const asksProjection = (
view: StateItemView,
tool: EnvironmentManifestTool,
args: ReplayArguments,
): boolean =>
view.projected_record_ordinals !== undefined &&
[...environmentToolArguments(tool).query].some(
([name, argument]) =>
isProjectionParameter(name) && args[argument] !== undefined,
/** Why a call replays the view's replayed reads, or undefined when the state answers it. */
const replayedCall = (
source: EnvironmentSource,
view: StateItemView | StateListView,
call: ReadonlyMap<string, string>,
): "projection" | "variant" | undefined =>
replayReason(
view,
new Map([...call].map(([name, value]) => [name, [value]])),
(ordinal) => source.exchanges.get(ordinal)?.query,
);
/** A recorded body the state renders into, usable exactly when extraction could use it. */
@@ -221,6 +219,22 @@ export const itemCall = (
};
/** The entity an item call names, preferring verbatim collection spellings as replay does; the id is tried percent-decoded, then as written. */
/** Whether the state started with the item a call names, which a served delete may since have removed. */
const startedWithItem = (
store: EnvironmentStateStore,
type: string,
call: ItemCall,
): boolean => {
const ids = new Set([decodedSegment(call.id), call.id]);
return store
.startedWith(type)
.some(
(entity) =>
ids.has(entity.id) &&
collectionMatch(entity.collection, call.segments) !== undefined,
);
};
export const namedEntity = (
store: EnvironmentStateStore,
type: string,
@@ -308,15 +322,27 @@ const answerItem = (
if (call === undefined)
return err(refused(tool, `Tool ${tool.name} needs every path argument.`));
const entity = namedEntity(store, view.type, call);
if (entity === undefined)
// A projection or another representation of an entity the state holds replays as the capture's reads did.
const replayed = replayedCall(source, view, callQuery(tool, args));
if (entity === undefined) {
// An item only a replayed line showed, which the state never held, answers with its own recording; a created
// item may since have been deleted, so a type a served create added to keeps the missing answer.
if (
replayed !== undefined &&
!store.createdInto(view.type) &&
!startedWithItem(store, view.type, call) &&
selectReplayExchange(tool, source.exchanges, args, observedQuery)
?.exact === true
)
return fromRecording(source, tool, args, replayed, observedQuery);
return missingAnswer(serving, tool, {
type: view.type,
collection: `/${call.segments.map(({ text }) => text).join("/")}`,
id: call.id,
});
// A projection of an entity the state holds replays as the capture's projected reads did.
if (asksProjection(view, tool, args))
return fromRecording(source, tool, args, "projection", observedQuery);
}
if (replayed !== undefined)
return fromRecording(source, tool, args, replayed, observedQuery);
const shown = shownEntity(
tool,
view,
@@ -489,6 +515,12 @@ const answerCheckedLine = (
args: ReplayArguments,
observedQuery?: ObservedQuery,
): Result<StateAnswer, AnalysisError> => {
const replayed =
view.shape === "list"
? replayedCall(serving.source, view, callQuery(tool, args))
: undefined;
if (replayed !== undefined)
return fromRecording(serving.source, tool, args, replayed, observedQuery);
const named = checkedLine(
serving.source,
tool,
src/application/environment/environmentStateStore.ts
+9
View File
@@ -117,6 +117,8 @@ export class EnvironmentStateStore {
readonly #sequences = new Map<string, number>();
/** The entities the state held when it was last reset or loaded, whose values no sequence reuses either. */
#started: readonly StateEntity[] = [];
/** The types a served create has added an entity to since the last reset or load. */
readonly #createdTypes = new Set<string>();
/** Baseline and post-start spellings kept known after their entities are deleted. */
readonly #knownCollections = new Map<string, Set<string>>();
/** Logical UTF-8 key bytes plus a 64-byte allowance per collection and potential sequence-key record. */
@@ -210,6 +212,7 @@ export class EnvironmentStateStore {
this.#writes = 0;
this.#sequences.clear();
this.#started = this.#content.entities;
this.#createdTypes.clear();
this.#knownCollections.clear();
this.#collectionHistoryBytes = 0;
for (const [type, collections] of this.#typeCollections)
@@ -227,6 +230,11 @@ export class EnvironmentStateStore {
}
/** The entities of a type the state held when it was last reset or loaded. */
/** Whether a served create has added an entity of the type since the last reset or load. */
createdInto(type: string): boolean {
return this.#createdTypes.has(type);
}
startedWith(type: string): readonly StateEntity[] {
return this.#started.filter((entity) => entity.type === type);
}
@@ -294,6 +302,7 @@ export class EnvironmentStateStore {
this.#rememberCollection(entity.type, entity.collection);
this.#collectionHistoryBytes += newCollectionBytes;
}
this.#createdTypes.add(entity.type);
return ok(undefined);
}
src/application/environment/environmentStateWriteAnswer.test.ts
+372
View File
@@ -20,6 +20,7 @@ import {
BOOK_ROWS,
DELETE_NOT_FOUND,
DETAILED_LIST_ROWS,
EXPANDED_LIST_ROWS,
FIRST_BOOK,
OMITTED_FIELD_ROWS,
PROJECTED_ROWS,
@@ -36,6 +37,7 @@ import {
book,
bookClock,
detailedBook,
expandedBook,
UNCHAPTERED_ROWS,
VERSIONED_ROWS,
numberedUuid,
@@ -434,6 +436,19 @@ describe("wider write coverage", () => {
});
});
it("replays a call naming a parameter only the expanded line names, and lists a served create on the plain one", () => {
const books = servedFrom(EXPANDED_LIST_ROWS);
const created = books.answer("post_books", CREATE_NEW_BOOK).body as Book;
expect(books.answer("get_books")).toMatchObject({
body: [FIRST_BOOK, SECOND_BOOK, created],
state: { source: "state", view: "list" },
});
expect(books.answer("get_books", { expand: "shelf" })).toMatchObject({
body: [FIRST_BOOK, SECOND_BOOK].map(expandedBook),
state: { source: "recording", reason: "variant" },
});
});
it("numbers created books after the largest held asset number, in its digit groups", () => {
const books = servedFrom(ASSET_ROWS);
const create = (name: string) =>
@@ -446,6 +461,98 @@ describe("wider write coverage", () => {
});
});
describe("items only a replayed line showed", () => {
it("replays the recorded read of an item only a variant line showed, and still finds an unrecorded one missing", () => {
const books = servedFrom([
["GET", "/api/books", [FIRST_BOOK]],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/1?expand=shelf", expandedBook(FIRST_BOOK)],
["GET", "/api/books/2?expand=shelf", expandedBook(SECOND_BOOK)],
]);
expect(
books.answer("get_books_by_id", { id: "2", expand: "shelf" }),
).toMatchObject({
status: 200,
body: expandedBook(SECOND_BOOK),
state: { source: "recording", reason: "variant" },
});
expect(
books.call("get_books_by_id", { id: "3", expand: "shelf" }),
).not.toMatchObject({ ok: true, value: { status: 200 } });
});
});
describe("items a served write made and removed", () => {
it("finds an item a served create made and a served delete removed missing, though a variant line recorded it", () => {
const created = book(3, "Third Book", bookClock(4));
const books = servedFrom([
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/2", SECOND_BOOK],
["GET", "/api/books/1?expand=shelf", expandedBook(FIRST_BOOK)],
["POST", "/api/books", created, 201, { name: "Third Book" }],
["GET", "/api/books/3", created],
["GET", "/api/books/3?expand=shelf", expandedBook(created)],
["DELETE", "/api/books/3", "", 204],
["GET", "/api/books/3", BOOK_NOT_FOUND, 404],
]);
expect(books.state.writes.get("post_books")).toMatchObject({
served: true,
});
expect(books.state.writes.get("delete_books_by_id")).toMatchObject({
served: true,
});
books.answer("post_books", { body: { name: "Third Book" } });
books.answer("delete_books_by_id", { id: "3" });
const read = books.call("get_books_by_id", { id: "3", expand: "shelf" });
expect(read.ok && read.value.status === 200).toBe(false);
});
});
describe("field selections beside replayed reads", () => {
it("replays a field selection no checked line gave, and answers a selection of every field from the state", () => {
const books = servedFrom([
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
["GET", "/api/books/1?fields=*", FIRST_BOOK],
["GET", "/api/books/2?fields=*", SECOND_BOOK],
["GET", "/api/books/2?fields=name", { name: "Second Book" }],
]);
expect(
books.answer("get_books_by_id", { id: "1", fields: "id" }),
).toMatchObject({ state: { source: "recording", reason: "projection" } });
expect(
books.answer("get_books_by_id", { id: "1", fields: "*" }),
).toMatchObject({
body: FIRST_BOOK,
state: { source: "state", view: "item" },
});
});
it("replays a field selection a replayed read gave, beside a variant key and a checked line giving it too", () => {
// One read asking for a book's name answered it whole; another left the id out.
const books = servedFrom([
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/1?fields=name", FIRST_BOOK],
["GET", "/api/books/2?fields=name", { name: "Second Book" }],
["GET", "/api/books/1?expand=shelf", expandedBook(FIRST_BOOK)],
]);
expect(
books.answer("get_books_by_id", { id: "2", fields: "name" }),
).toMatchObject({
body: { name: "Second Book" },
state: { source: "recording", reason: "projection" },
});
expect(
books.answer("get_books_by_id", { id: "1", expand: "shelf" }),
).toMatchObject({ state: { source: "recording", reason: "variant" } });
expect(books.answer("get_books_by_id", { id: "2" })).toMatchObject({
body: SECOND_BOOK,
state: { source: "state", view: "item" },
});
});
});
describe("write coverage limits", () => {
it("numbers past a mask's width again after the state is saved and loaded", () => {
const books = servedFrom([
@@ -1689,3 +1796,268 @@ describe("asset sequence inference", () => {
).toEqual(gamma);
});
});
describe("served catalog of calls that only read", () => {
it("serves a lookup POST the capture shows reading only, by replay", () => {
const { source, state } = servedFrom([
...BOOK_INIT_ROWS,
["POST", "/api/books/ids", [FIRST_BOOK], 200, { ids: [1] }],
]);
const catalog = servedCatalog(source, state, "books");
if (!catalog.ok) throw catalog.error;
expect(catalog.value.tools.map((tool) => tool.name)).toContain(
"post_books_ids",
);
expect(state.writes.has("post_books_ids")).toBe(false);
});
});
describe("creates read under another path", () => {
it("creates into the collection a linked create's returned id was read under", () => {
const note = (id: number, title: string) => ({ id, title });
const [alpha, beta, gamma] = [
note(1001, "Alpha"),
note(1002, "Beta"),
note(1003, "Gamma"),
];
const notes = servedFrom([
["GET", "/api/notes", [alpha, beta]],
["GET", "/api/notes/1001", alpha],
["GET", "/api/notes/1002", beta],
["POST", "/api/drafts/publish", { id: 1003 }, 201, { title: "Gamma" }],
["GET", "/api/notes/1003", gamma],
["GET", "/api/notes", [alpha, beta, gamma]],
]);
expect(
notes.answer("post_drafts_publish", { body: { title: "Delta" } }),
).toMatchObject({ status: 201 });
// The capture's own create is not in the initial state, so the served one takes the next id.
const delta = note(1003, "Delta");
expect(notes.answer("get_notes_by_id", { id: "1003" }).body).toEqual(delta);
expect(notes.answer("get_notes").body).toEqual([alpha, beta, delta]);
});
it("refuses a linked create called with other path arguments than its recording sent", () => {
const note = (id: number, title: string) => ({ id, title });
const [alpha, beta, gamma] = [
note(1001, "Alpha"),
note(1002, "Beta"),
note(1003, "Gamma"),
];
const notes = servedFrom([
["GET", "/api/projects/11/notes", [alpha, beta]],
["GET", "/api/projects/11/notes/1001", alpha],
["GET", "/api/projects/11/notes/1002", beta],
[
"POST",
"/api/projects/11/publish",
{ id: 1003 },
201,
{ title: "Gamma" },
],
["GET", "/api/projects/11/notes/1003", gamma],
["GET", "/api/projects/11/notes", [alpha, beta, gamma]],
]);
const before = notes.snapshot();
// No read showed where project 12's publish creates, so it is not put into project 11.
expect(
notes.refusal("post_projects_publish_by_id", {
id: "12",
body: { title: "Delta" },
}),
).toMatchObject({
code: "invalid_request",
message: expect.stringContaining(
"creates into /api/projects/11/notes only with the path arguments its recording sent",
),
});
expect(notes.snapshot()).toBe(before);
expect(
notes.answer("post_projects_publish_by_id", {
id: "11",
body: { title: "Delta" },
}),
).toMatchObject({ status: 201 });
});
});
describe("path values of creates read under another path", () => {
it("refuses a linked create that changes one of two path values its template names alike", () => {
const note = (id: number, title: string) => ({ id, title });
const [alpha, beta, gamma] = [
note(1001, "Alpha"),
note(1002, "Beta"),
note(1003, "Gamma"),
];
const group = "/api/projects/11/groups/5";
const notes = servedFrom([
["GET", `${group}/notes`, [alpha, beta]],
["GET", `${group}/notes/1001`, alpha],
["GET", `${group}/notes/1002`, beta],
["POST", `${group}/publish`, { id: 1003 }, 201, { title: "Gamma" }],
["GET", `${group}/notes/1003`, gamma],
["GET", `${group}/notes`, [alpha, beta, gamma]],
]);
const publish = notes.source.manifest.tools.find(
(tool) => tool.method === "POST",
)!;
expect(publish.path_template).toBe(
"/api/projects/{numeric}/groups/{numeric}/publish",
);
const [project, groupArgument] =
environmentToolArguments(publish).path.values();
expect(notes.state.writes.get(publish.name)).toMatchObject({
served: true,
collection: `${group}/notes`,
});
// Project 12's group 5 is another source, though the group value matches.
expect(
notes.refusal(publish.name, {
[project!]: "12",
[groupArgument!]: "5",
body: { title: "Delta" },
}),
).toMatchObject({ code: "invalid_request" });
});
it("fills a linked create's path field with the decoded value, whichever spelling the call uses", () => {
const note = (id: number, title: string) => ({
id,
title,
project: "my org1",
});
const [alpha, beta, gamma] = [
note(1001, "Alpha"),
note(1002, "Beta"),
note(1003, "Gamma"),
];
const other = { id: 2001, title: "Other", project: "my org2" };
const project = "/api/projects/my%20org1";
for (const spelling of ["my%20org1", "my org1"]) {
// A second project makes the project segment a path parameter.
const notes = servedFrom([
["GET", "/api/projects/my%20org2/notes", [other]],
["GET", "/api/projects/my%20org2/notes/2001", other],
["GET", `${project}/notes`, [alpha, beta]],
["GET", `${project}/notes/1001`, alpha],
["GET", `${project}/notes/1002`, beta],
["POST", `${project}/publish`, { id: 1003 }, 201, { title: "Gamma" }],
["GET", `${project}/notes/1003`, gamma],
["GET", `${project}/notes`, [alpha, beta, gamma]],
["POST", "/api/projects/my%20org2/publish", { ok: true }, 200, {}],
]);
notes.answer("post_projects_publish_by_project_id", {
project_id: spelling,
body: { title: "Delta" },
});
const created = notes.store
.snapshot()
.rea_state.entities.filter((entity) => entity.fields.title === "Delta");
expect(created).toMatchObject([{ fields: { project: "my org1" } }]);
}
});
it("counts a linked create's call from a source that names the exemplar's only after decoding twice as another source's", () => {
const note = (id: number, title: string) => ({ id, title });
const [alpha, beta, gamma, delta] = [
note(1001, "Alpha"),
note(1002, "Beta"),
note(1003, "Gamma"),
note(1004, "Delta"),
];
const other = note(2001, "Other");
const project = "/api/projects/team1";
// A second project makes the project segment a path parameter.
const notes = servedFrom([
["GET", "/api/projects/team2/notes", [other]],
["GET", "/api/projects/team2/notes/2001", other],
["GET", `${project}/notes`, [alpha, beta]],
["GET", `${project}/notes/1001`, alpha],
["GET", `${project}/notes/1002`, beta],
["POST", `${project}/publish`, { id: 1003 }, 201, { title: "Gamma" }],
["GET", `${project}/notes/1003`, gamma],
["GET", `${project}/notes`, [alpha, beta, gamma]],
// `team%2531` decodes once to `team%31`, another project than `team1`.
[
"POST",
"/api/projects/team%2531/publish",
delta,
201,
{ title: "Delta" },
],
["GET", `${project}/notes`, [alpha, beta, gamma]],
]);
expect(
notes.state.writes.get("post_projects_publish_by_project_id"),
).toMatchObject({ served: true, contradicted_by: null });
});
it("takes a linked create's recorded path value in either spelling, as other state-backed calls do", () => {
const note = (id: number, title: string) => ({ id, title });
const [alpha, beta, gamma] = [
note(1001, "Alpha"),
note(1002, "Beta"),
note(1003, "Gamma"),
];
const project = "/api/projects/my%20org1";
const other = note(2001, "Other");
// A second project makes the project segment a path parameter.
const notes = servedFrom([
["GET", "/api/projects/my%20org2/notes", [other]],
["GET", "/api/projects/my%20org2/notes/2001", other],
["GET", `${project}/notes`, [alpha, beta]],
["GET", `${project}/notes/1001`, alpha],
["GET", `${project}/notes/1002`, beta],
["POST", `${project}/publish`, { id: 1003 }, 201, { title: "Gamma" }],
["GET", `${project}/notes/1003`, gamma],
["GET", `${project}/notes`, [alpha, beta, gamma]],
["POST", "/api/projects/my%20org2/publish", { ok: true }, 200, {}],
]);
for (const spelling of ["my%20org1", "my org1"])
expect(
notes.answer("post_projects_publish_by_project_id", {
project_id: spelling,
body: { title: "Delta" },
}),
).toMatchObject({ status: 201 });
});
});
describe("path fields of creates read under another path", () => {
it("fills a linked create's path field from its exemplar decoded once, whichever accepted spelling the call uses", () => {
const note = (id: number, title: string) => ({
id,
title,
project: "team%31",
});
const [alpha, beta, gamma] = [
note(1001, "Alpha"),
note(1002, "Beta"),
note(1003, "Gamma"),
];
const other = { id: 2001, title: "Other", project: "team2" };
// The recorded segment `team%2531` names the project `team%31`.
const project = "/api/projects/team%2531";
for (const spelling of ["team%2531", "team%31"]) {
const notes = servedFrom([
["GET", "/api/projects/team2/notes", [other]],
["GET", "/api/projects/team2/notes/2001", other],
["GET", `${project}/notes`, [alpha, beta]],
["GET", `${project}/notes/1001`, alpha],
["GET", `${project}/notes/1002`, beta],
["POST", `${project}/publish`, { id: 1003 }, 201, { title: "Gamma" }],
["GET", `${project}/notes/1003`, gamma],
["GET", `${project}/notes`, [alpha, beta, gamma]],
["POST", "/api/projects/team2/publish", { ok: true }, 200, {}],
]);
notes.answer("post_projects_publish_by_project_id", {
project_id: spelling,
body: { title: "Delta" },
});
const created = notes.store
.snapshot()
.rea_state.entities.filter((entity) => entity.fields.title === "Delta");
expect(created).toMatchObject([{ fields: { project: "team%31" } }]);
}
});
});
src/application/environment/environmentStateWriteAnswer.ts
+88 -8
View File
@@ -14,6 +14,10 @@ import {
nextInteger,
} from "../../domain/analysisView/environmentStateWriteExplain.js";
import { callRenewal } from "../../domain/analysisView/environmentStateWriteGenerated.js";
import {
linkedCollection,
sameLinkedSource,
} from "../../domain/analysisView/environmentStateWriteLinks.js";
import {
pathValuesOf,
type GeneratedRule,
@@ -29,6 +33,7 @@ import {
} from "../../domain/analysisView/environmentStateWriteValues.js";
import {
collectionMatch,
decodedSegment,
isObject,
scalarText,
type JsonObject,
@@ -290,6 +295,85 @@ const writeAnswer = (
return stateAnswer(serving, { exchange, exact }, body.value ?? null, state);
};
/** The path arguments a call supplies as given, by the template segment each fills. */
const pathArguments = (
tool: EnvironmentManifestTool,
args: ReplayArguments,
): Map<number, string> => {
const names = environmentToolArguments(tool).path;
return new Map(
tool.params.flatMap((param) => {
const value = args[names.get(param.name) ?? param.name];
return typeof value === "string" ? [[param.index, value] as const] : [];
}),
);
};
/** The path of the recorded call a plan was built from. */
const exemplarPath = (
serving: StateServing,
plan: StateWritePlan,
): string | undefined =>
plan.exemplar_ordinal === null
? undefined
: serving.source.exchanges.get(plan.exemplar_ordinal)?.path;
/** The collection a create writes into: a linked plan's, only for its exemplar's path arguments, else the one the call's path selects. */
const writtenCollection = (
serving: StateServing,
request: WriteRequest,
): Result<string | undefined, AnalysisError> => {
const { tool, plan } = request;
const linked = linkedCollection(plan, tool.path_template);
if (linked === undefined)
return ok(
createdCollection(serving, tool, request.args, plan.target ?? ""),
);
// No read showed where other path arguments create.
return sameLinkedSource(
tool.path_template,
exemplarPath(serving, plan),
pathArguments(tool, request.args),
)
? ok(linked)
: err(
refused(
tool,
`Tool ${tool.name} creates into ${linked} only with the path arguments its recording sent.`,
),
);
};
/**
* The path values a create's path fields take, decoded once as the gate learned them: a linked
* create accepts its exemplar's source only, so it takes that; another the collection this call
* selects, else its own arguments.
*/
const createdPathValues = (
serving: StateServing,
request: WriteRequest,
call: WriteCall,
collection: string,
): ReadonlyMap<string, string> => {
const { tool, plan } = request;
const exemplar =
linkedCollection(plan, tool.path_template) === undefined
? undefined
: exemplarPath(serving, plan);
return (
(exemplar === undefined
? undefined
: pathValuesOf(tool.path_template, exemplar)) ??
pathValuesOf(tool.path_template, collection) ??
new Map(
[...call.pathValues].map(([name, value]) => [
name,
decodedSegment(value),
]),
)
);
};
const answerCreate = (
serving: StateServing,
request: WriteRequest,
@@ -298,12 +382,9 @@ const answerCreate = (
const { tool, plan } = request;
const type = plan.target ?? "";
const idField = serving.store.type(type)?.id_field ?? "";
const collection = createdCollection(
serving,
request.tool,
request.args,
plan.target ?? "",
);
const target = writtenCollection(serving, request);
if (!target.ok) return target;
const collection = target.value;
const observed = observedEntity(serving, plan);
if (collection === undefined || observed === undefined)
return err(
@@ -325,10 +406,9 @@ const answerCreate = (
collection: members,
observed,
});
// The gate learned path rules from the recorded path, decoded; the collection is this call's.
const created = {
...call,
pathValues: pathValuesOf(tool.path_template, collection) ?? call.pathValues,
pathValues: createdPathValues(serving, request, call, collection),
};
const fields = createdFields(plan, observed, created, values.generate);
if (!fields.ok) return err(refused(tool, fields.error));
src/domain/analysisView/EnvironmentStateAccumulator.ts
+1
View File
@@ -374,6 +374,7 @@ export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator
requestBody: (ordinal) => this.#requestBody(ordinal),
emptyResponse: (ordinal) => this.#emptyResponse(ordinal),
retain: (bytes) => this.retain(bytes),
checkpoint: () => this.retainedCheckpoint(),
release: () => this.releaseTemporary(),
},
}).run();
src/domain/analysisView/EnvironmentStateExtraction.ts
+225 -59
View File
@@ -13,21 +13,18 @@ import {
IDENTITY_NAMES,
identityCandidates,
isObject,
isParam,
isQueryPost,
isStateWrite,
stateValueRetentionBytes,
lastSegmentValue,
STATE_BODY_LIMIT_BYTES,
listItems,
locateItem,
locateList,
normalizedPath,
projectsFields,
rootOf,
selectsSomeFields,
same,
scalarText,
segmentsOf,
writeSpanCheck,
type ExtractedState,
type JsonObject,
@@ -49,14 +46,30 @@ import {
type ObservedEntity,
type StateObservation,
} from "./environmentStateMerge.js";
import { rewriteCuts, type RewriteSpan } from "./environmentStateRewrites.js";
import {
checkItemPlan,
checkListPlan,
type LateLine,
type StateGateContext,
type StatePlan,
} from "./environmentStateGate.js";
import { StateWriteCutoffs } from "./environmentStateWriteCutoffs.js";
import {
createdIdsOf,
creationLimit,
readType,
respondedIds,
type CreatedIds,
} from "./environmentStateWriteLinks.js";
import { extractWrites } from "./environmentStateWrites.js";
import {
dropObservations,
lineShapes,
shapesOtherwise,
withoutVariants,
} from "./environmentStateVariants.js";
import { stateWritesOf } from "./environmentStateQueryCalls.js";
class StateBudgetExhausted extends Error {}
@@ -74,6 +87,12 @@ const entityKey = (type: string, collection: string, id: string): string =>
const requestLine = (record: StateRecordFacts): string =>
record.path + (record.query ?? "");
/** The member names of the object at a pointer, none when it holds no object. */
const memberNames = (body: JsonValue, pointer: string): string[] => {
const object = at(body, pointer);
return isObject(object) ? Object.keys(object) : [];
};
/** Derive the init state and render plans from manifest tools and timeline-ordered records. */
export class EnvironmentStateExtraction {
readonly #tools: readonly StateToolFacts[];
@@ -82,11 +101,15 @@ export class EnvironmentStateExtraction {
readonly #budget: StateBudget;
readonly #position: ReadonlyMap<number, number>;
readonly #byOrdinal: ReadonlyMap<number, StateRecordFacts>;
readonly #writes: readonly StateRecordFacts[];
readonly #writeWithin: ReturnType<typeof writeSpanCheck>;
/** POST tools whose calls the capture shows read state rather than change it. */
#readOnly: ReadonlySet<string> = new Set();
#writes: readonly StateRecordFacts[] = [];
#writeWithin: ReturnType<typeof writeSpanCheck> = writeSpanCheck([]);
readonly #bases: ReadonlyMap<string, number>;
#writeCutoffs: StateWriteCutoffs | undefined;
readonly #initLimits = new Map<string, StateRecordFacts | undefined>();
/** Successful POSTs with the ids their responses named, read once before planning. */
#created: CreatedIds[] | undefined;
readonly #identity = new Map<
string,
{ readonly field: string; readonly basis: StateType["id_basis"] }
@@ -97,6 +120,9 @@ export class EnvironmentStateExtraction {
readonly #volatile = new Map<string, Set<string>>();
/** Types with a value that changed across a recorded write, which may be that write's effect. */
readonly #rewritten = new Set<string>();
/** The recordings either side of each such change, and per type the write its initial reads end at. */
#spans: RewriteSpan[] = [];
#cuts: ReadonlyMap<string, StateRecordFacts> = new Map();
readonly #merged = new Map<string, StateEntity>();
/** Merged entities by type and collection, built once on first use after merging. */
#collections: Map<string, StateEntity[]> | undefined;
@@ -117,15 +143,31 @@ export class EnvironmentStateExtraction {
this.#byOrdinal = new Map(
input.records.map((record) => [record.ordinal, record]),
);
this.#writes = input.records.filter(isStateWrite);
this.#writeWithin = writeSpanCheck(this.#writes);
this.#bases = apiBaseLengths(input.tools.map((tool) => tool.path_template));
}
/** The timeline's state writes, less the calls of tools that only read. */
#findWrites(): void {
// The body digests the decision keeps are dropped with it, so what they were charged is released.
const release = this.#budget.checkpoint();
({ readOnly: this.#readOnly, writes: this.#writes } = stateWritesOf(
this.#tools,
this.#records,
{
withBody: (ordinal, use) => this.#withBody(ordinal, use),
request: (ordinal) => this.#requestBody(ordinal),
retain: (bytes) => this.#retain(bytes),
},
));
release();
this.#writeWithin = writeSpanCheck(this.#writes);
}
/** The extracted state, or null when the representation budget is exhausted. */
run(): ExtractedState | null {
try {
if (this.#writes.length > 0)
this.#findWrites();
if (this.#writes.length > 0) {
this.#writeCutoffs = new StateWriteCutoffs(
this.#writes,
this.#bases,
@@ -134,14 +176,15 @@ export class EnvironmentStateExtraction {
this.#position.get(right.ordinal)!,
(bytes) => this.#retain(bytes),
);
const candidates = this.#candidates();
for (const candidate of candidates.filter((entry) => entry.item))
this.#planItem(candidate);
for (const candidate of candidates.filter((entry) => !entry.item))
this.#planList(candidate);
for (const plan of this.#plans) this.#observePlan(plan);
this.#classifyChanges();
this.#merge();
this.#creations();
}
// A rewrite cut discards the first planning pass, so what it retained is released before the second.
const release = this.#budget.checkpoint();
let candidates = this.#plan();
if (this.#cutRewrites()) {
release();
candidates = this.#plan();
}
const gate = this.#gateContext();
for (const plan of this.#plans)
if (this.#rewritten.has(plan.type))
@@ -158,6 +201,62 @@ export class EnvironmentStateExtraction {
}
}
/** Plan and observe every candidate, then merge the observed entities. */
#plan(): Candidate[] {
const candidates = this.#candidates();
for (const candidate of candidates.filter((entry) => entry.item))
this.#planItem(candidate);
for (const candidate of candidates.filter((entry) => !entry.item))
this.#planList(candidate);
for (const plan of this.#plans) this.#observePlan(plan);
this.#dropVariants();
this.#classifyChanges();
this.#merge();
return candidates;
}
/** Leave each plan's representation variants to replay: out of its lines, the observed entities and the membership changes. */
#dropVariants(): void {
const shapes = lineShapes(this.#entities.values(), (bytes) =>
this.#retain(bytes),
);
const dropped = new Set<number>();
for (const [index, plan] of this.#plans.entries()) {
const split = withoutVariants(plan, {
shapeOf: (record) => shapes.get(record.ordinal),
queryOf: (ordinal) => this.#byOrdinal.get(ordinal)?.query,
position: (ordinal) => this.#position.get(ordinal)!,
retain: (bytes) => this.#retain(bytes),
checkpoint: () => this.#budget.checkpoint(),
});
if (split === undefined) continue;
this.#retain(ENTRY_BYTES * (split.dropped.size + 2));
this.#plans[index] = split.plan;
dropObservations(this.#entities, split.dropped);
for (const ordinal of split.dropped) dropped.add(ordinal);
}
this.#spans = this.#spans.filter((span) => !dropped.has(span.later));
}
/** End each rewritten type's initial reads at the earliest write its reads disagree across, and clear the plans to make them again. */
#cutRewrites(): boolean {
this.#cuts = rewriteCuts(
this.#spans.filter((span) => this.#rewritten.has(span.type)),
this.#writes,
this.#byOrdinal,
);
if (this.#cuts.size === 0) return false;
for (const map of [this.#initLimits, this.#identity, this.#views])
map.clear();
for (const map of [this.#entities, this.#volatile, this.#merged])
map.clear();
this.#rewritten.clear();
this.#plans.length = 0;
this.#spans = [];
this.#collections = undefined;
return true;
}
/** The write plans the capture's own timeline confirms against the extracted state. */
#withWrites(
state: Omit<ExtractedState, "writes" | "write_clock">,
@@ -167,22 +266,26 @@ export class EnvironmentStateExtraction {
...extractWrites(state, {
tools: this.#tools,
records: this.#records,
readOnly: this.#readOnly,
timed: this.#timeline === "recorded-start-time",
volatileOf: (type) => this.#volatileOf(type),
io: {
withBody: (ordinal, use) => this.#withBody(ordinal, use),
requestBody: (ordinal) => {
const parsed = this.#budget.requestBody(ordinal);
if (parsed === null) throw new StateBudgetExhausted();
return parsed.kind === "json" ? parsed.value : undefined;
},
requestBody: (ordinal) => this.#requestBody(ordinal),
emptyResponse: (ordinal) => this.#budget.emptyResponse(ordinal),
retain: (bytes) => this.#retain(bytes),
queryOf: (ordinal) => this.#byOrdinal.get(ordinal)?.query,
},
}),
};
}
#requestBody(ordinal: number): JsonValue | undefined {
const parsed = this.#budget.requestBody(ordinal);
if (parsed === null) throw new StateBudgetExhausted();
return parsed.kind === "json" ? parsed.value : undefined;
}
#retain(bytes: number): void {
if (!this.#budget.retain(bytes)) throw new StateBudgetExhausted();
}
@@ -206,30 +309,54 @@ export class EnvironmentStateExtraction {
#candidates(): Candidate[] {
const candidates: Candidate[] = [];
for (const tool of this.#tools) {
if (isQueryPost(tool.method, tool.path_template))
if (
isQueryPost(tool.method, tool.path_template) ||
this.#readOnly.has(tool.name)
)
this.#recorded(tool, "query-body");
if (tool.method !== "GET") continue;
const segments = segmentsOf(tool.path_template);
const item = segments.length > 0 && isParam(segments.at(-1)!);
candidates.push({
tool,
item,
type: item
? `/${segments.slice(0, -1).join("/")}`
: normalizedPath(tool.path_template),
});
candidates.push({ tool, ...readType(tool.path_template) });
}
return candidates;
}
/** The first write whose root relates to the template's bounds its init recordings. */
/** The first write whose root relates to the template's, or that made one of its entities, bounds its init recordings. */
#initLimit(template: string): StateRecordFacts | undefined {
if (this.#writeCutoffs === undefined) return undefined;
if (this.#initLimits.has(template)) return this.#initLimits.get(template);
const first = this.#writeCutoffs.first(rootOf(template, this.#bases));
const bound =
creationLimit(
this.#creations(),
{
type: template,
tools: this.#tools,
records: this.#byOrdinal,
position: (record) => this.#position.get(record.ordinal)!,
withBody: (ordinal, use) => this.#withBody(ordinal, use),
retain: (bytes) => this.#retain(bytes),
},
first,
) ?? first;
const cut = this.#cuts.get(template);
const limit =
cut !== undefined &&
(bound === undefined ||
this.#position.get(cut.ordinal)! < this.#position.get(bound.ordinal)!)
? cut
: bound;
this.#retain(ENTRY_BYTES + textBytes(template.length));
this.#initLimits.set(template, first);
return first;
this.#initLimits.set(template, limit);
return limit;
}
#creations(): CreatedIds[] {
this.#created ??= createdIdsOf(this.#writes, (ordinal) => {
const ids = this.#withBody(ordinal, respondedIds);
this.#retain(ENTRY_BYTES + textBytes(ids.join("").length));
return ids;
});
return this.#created;
}
/** Initial-state admission is shared with later projection-evidence selection. */
@@ -286,25 +413,31 @@ export class EnvironmentStateExtraction {
candidate: Candidate,
pointer: string,
recordings: readonly StateRecordFacts[],
projected: readonly number[] = [],
replayed: Pick<StatePlan, "projected" | "late"> = {},
): void {
this.#plans.push({
...candidate,
pointer,
recordings,
latest: this.#latestPerLine(recordings),
projected,
projected: replayed.projected ?? [],
late: replayed.late ?? [],
});
}
/** Later identity-free projections are evidence of replay, never initial entities. Read only those query bodies. */
/**
* Later identity-free projections are evidence of replay, never initial entities; later lines that
* show the identity and name a shaping or projection parameter are kept, with their members, for
* the variant pass to decide against the lines left. Read only those query bodies.
*/
#addLateProjections(
tool: StateToolFacts,
type: string,
projected: number[],
): void {
): LateLine[] {
const limit = this.#initLimit(type);
if (limit === undefined) return;
const late: LateLine[] = [];
if (limit === undefined) return late;
for (const ordinal of tool.record_ordinals) {
const record = this.#byOrdinal.get(ordinal);
if (
@@ -312,22 +445,33 @@ export class EnvironmentStateExtraction {
record.status === null ||
record.status < 200 ||
record.status >= 300 ||
!projectsFields(record.query) ||
this.#beforeInitLimit(record, limit)
)
continue;
if (
this.#withBody(
ordinal,
(body) =>
body !== undefined &&
locateItem(body, lastSegmentValue(record.path)) === undefined,
)
) {
const projecting = projectsFields(record.query);
// A selection of every field (`fields=*`) shows the entity whole, so only a partial one may replay.
const selecting = selectsSomeFields(record.query);
const shaping = shapesOtherwise(record.query);
if (!projecting && !shaping) continue;
const line = this.#withBody(ordinal, (body) => {
if (body === undefined) return undefined;
const located = locateItem(body, lastSegmentValue(record.path));
if (located === undefined) return projecting ? null : undefined;
if (!selecting && !shaping) return undefined;
const names = memberNames(body, located.pointer);
// The kept set holds the names, which the body's own reservation no longer covers.
this.#retain(
ENTRY_BYTES * (2 + names.length) +
names.reduce((bytes, name) => bytes + textBytes(name.length), 0),
);
return { record, members: new Set(names), projecting: selecting };
});
if (line === null) {
this.#retain(ENTRY_BYTES);
projected.push(ordinal);
}
} else if (line !== undefined) late.push(line);
}
return late;
}
/** Item tools prove a type's identity field by the path value their bodies repeat. */
@@ -382,11 +526,11 @@ export class EnvironmentStateExtraction {
return;
}
this.#identity.set(type, { field, basis: "path-parameter" });
this.#addLateProjections(tool, type, projected);
const late = this.#addLateProjections(tool, type, projected);
projected.sort(
(left, right) => this.#position.get(left)! - this.#position.get(right)!,
);
this.#addPlan(candidate, pointer, kept, projected);
this.#addPlan(candidate, pointer, kept, { projected, late });
}
/**
@@ -396,8 +540,11 @@ export class EnvironmentStateExtraction {
*/
#planList(candidate: Candidate): void {
const { tool, type } = candidate;
const recordings = this.#initRecordingsOf(candidate);
if (recordings === undefined) return;
const initial = this.#initRecordingsOf(candidate);
if (initial === undefined) return;
// Beside a line showing whole entities, one asking for some fields only is replayed and proves nothing; one asking for all is whole.
const whole = initial.filter((record) => !selectsSomeFields(record.query));
const recordings = whole.length > 0 ? whole : initial;
const proved = this.#identity.get(type)?.field;
const fieldsOf = (items: readonly JsonObject[]): string[] =>
proved === undefined
@@ -447,7 +594,12 @@ export class EnvironmentStateExtraction {
if (field === undefined || pointer === undefined) return;
if (!this.#identity.has(type))
this.#identity.set(type, { field, basis: "field-name" });
this.#addPlan(candidate, pointer, recordings);
this.#retain(ENTRY_BYTES * (initial.length - recordings.length));
this.#addPlan(candidate, pointer, recordings, {
projected: initial
.filter((record) => !recordings.includes(record))
.map((record) => record.ordinal),
});
}
#observePlan(plan: StatePlan): void {
@@ -500,8 +652,19 @@ export class EnvironmentStateExtraction {
for (const object of objects)
if (!previous.ids.delete(scalarText(object[field])!)) changed = true;
if (previous.ids.size > 0) changed = true;
if (earlier !== undefined && changed && this.#writeWithin(earlier, record))
this.#rewritten.add(plan.type);
if (
earlier !== undefined &&
changed &&
this.#writeWithin(earlier, record)
) {
// Each kept span is charged as an entity change's is.
this.#retain(ENTRY_BYTES * 4);
this.#spans.push({
type: plan.type,
earlier: earlier.ordinal,
later: record.ordinal,
});
}
previous.ids.clear();
for (const object of objects) previous.ids.add(scalarText(object[field])!);
previous.record = record;
@@ -540,6 +703,7 @@ export class EnvironmentStateExtraction {
this.#retain(ENTRY_BYTES + stateValueRetentionBytes(object));
entity.observations.push({
position,
index,
ordinal: record.ordinal,
tool: plan.tool.name,
item: plan.item,
@@ -547,7 +711,7 @@ export class EnvironmentStateExtraction {
});
}
/** A change across any recorded write rewrites the type; one between repeats of a line with no write is volatile. */
/** A change across any recorded write, in values or in a kept line's membership, rewrites the type; one between repeats of a line with no write is volatile. */
#classifyChanges(): void {
const recordOf = (observation: StateObservation): StateRecordFacts =>
this.#byOrdinal.get(observation.ordinal)!;
@@ -556,7 +720,9 @@ export class EnvironmentStateExtraction {
(observation) => [observation.tool, requestLine(recordOf(observation))],
(earlier, later) => this.#writeWithin(recordOf(earlier), recordOf(later)),
);
for (const type of changes.rewritten) this.#rewritten.add(type);
this.#retain(ENTRY_BYTES * 4 * changes.spans.length);
for (const span of changes.spans) this.#spans.push(span);
for (const { type } of this.#spans) this.#rewritten.add(type);
for (const [type, names] of changes.volatile)
this.#volatile.set(type, names);
}
src/domain/analysisView/EnvironmentStateWriteSimulation.test.ts
+167 -5
View File
@@ -1,6 +1,6 @@
import { OmittedFields } from "./environmentStateWriteOmitted.js";
import { checkedLines, servedLine } from "./environmentStateCheckedLines.js";
import { describe, expect, it } from "vitest";
import { describe, expect, it, vi } from "vitest";
import {
timedCaptureEvidence,
@@ -25,6 +25,8 @@ import {
itemTarget,
SimulatedState,
} from "./environmentStateWriteChecks.js";
import { UnseenEntities } from "./environmentStateWriteAdoption.js";
import type { StateItemView } from "./environmentStateRules.js";
import {
laterObservation,
type ObservationSearch,
@@ -1244,10 +1246,11 @@ describe("reads named as serving names them", () => {
});
describe("encoded parent spellings", () => {
it("names the plainly held entity from a path that encodes its parent", () => {
const type = "/api/orgs/uid/{uid}/books";
const collection = "/api/orgs/uid/acme1/books";
const state = new SimulatedState(
const type = "/api/orgs/uid/{uid}/books";
const collection = "/api/orgs/uid/acme1/books";
/** A state holding one book under its parent's plain spelling. */
const plainlyHeld = () =>
new SimulatedState(
[
{
type,
@@ -1260,9 +1263,26 @@ describe("encoded parent spellings", () => {
],
[],
);
it("names the plainly held entity from a path that encodes its parent", () => {
expect(
itemTarget(plainlyHeld(), type, {
path: "/api/orgs/uid/%61cme1/books/1",
}),
).toEqual({ collection, id: "1" });
});
it("lists the type's entities only when the path's own collection holds no such id", () => {
const state = plainlyHeld();
const listed = vi.spyOn(state, "ofType");
expect(
itemTarget(state, type, { path: "/api/orgs/uid/acme1/books/1" }),
).toEqual({ collection, id: "1" });
expect(listed).not.toHaveBeenCalled();
expect(
itemTarget(state, type, { path: "/api/orgs/uid/%61cme1/books/1" }),
).toEqual({ collection, id: "1" });
expect(listed).toHaveBeenCalledTimes(1);
});
it("answers an encoded read from the plain checked line", () => {
@@ -1512,6 +1532,7 @@ describe("resolved encoded item observations", () => {
from: 0,
viewOf: () => itemView,
withBody: (_ordinal, use) => use({ id: "%61", name: "After" }),
queryOf: () => undefined,
};
expect(target).toEqual({ collection, id: "%61" });
@@ -2207,6 +2228,147 @@ describe("volatile fields", () => {
});
});
describe("entities an unmodelled write made", () => {
const IMPORT: TimedCaptureRow = [
"POST",
"/api/imports",
{ queued: true },
202,
{ source: "csv" },
];
const RENAME: TimedCaptureRow = [
"PATCH",
"/api/books/1",
renamedBook(FIRST_BOOK, "Renamed Book", 3),
200,
{ name: "Renamed Book" },
];
const deletedAfterRead = (before: readonly TimedCaptureRow[]) => [
...BOOK_INIT_ROWS,
...before,
["GET", "/api/books/3", THIRD_BOOK] as TimedCaptureRow,
["DELETE", "/api/books/3", "", 204] as TimedCaptureRow,
["GET", "/api/books/3", BOOK_NOT_FOUND, 404] as TimedCaptureRow,
];
it("serves a delete of a book an unmodelled import made, which a later item read showed", () => {
// The rename ends the initial reads, so the third book is no initial one.
expect(
gated(deletedAfterRead([RENAME, IMPORT]), "delete_books_by_id"),
).toMatchObject({ served: true, checked_record_ordinals: [7] });
});
it("takes a book the state never held only when a write the gate does not model came first", () => {
expect(
gated(deletedAfterRead([RENAME, IMPORT]), "delete_books_by_id"),
).toMatchObject({ served: true, checked_record_ordinals: [7] });
// Only a modelled rename came first, so nothing explains the third book and the delete finds none.
expect(
gated(deletedAfterRead([RENAME]), "delete_books_by_id"),
).toMatchObject({
served: false,
checked_record_ordinals: [],
});
});
it("takes a book an import may have made though it was redirected or went unanswered", () => {
for (const status of [303, 0]) {
const unresolved: TimedCaptureRow = [
"POST",
"/api/imports",
{ queued: true },
status,
{ source: "csv" },
];
expect(
gated(deletedAfterRead([RENAME, unresolved]), "delete_books_by_id"),
).toMatchObject({ served: true, checked_record_ordinals: [7] });
}
});
it("counts an entity it takes against the view's allowance", () => {
let retained = 0;
const read = { ...THIRD_BOOK, notes: "x".repeat(10_000) };
const unseen = new UnseenEntities([], {
state: new SimulatedState([], []),
io: {
withBody: (_ordinal, use) => use(read),
retain: (bytes) => (retained += bytes),
},
lastCheck: new Map(),
idField: () => "id",
});
unseen.unplanned(0);
retained = 0;
const view = { shape: "item", type: "/api/books", pointer: "" };
const record = {
ordinal: 1,
method: "GET",
path: "/api/books/3",
query: undefined,
status: 200,
start: 1,
end: 1,
};
expect(
unseen.adopt(view as unknown as StateItemView, { record, index: 1 }),
).toMatchObject({ id: "3", fields: read });
expect(retained).toBeGreaterThan(20_000);
});
it("finds no read's target among the held entities until an unplanned write came before the read", () => {
const state = new SimulatedState([], []);
const targets = vi.spyOn(state, "ofType");
const unseen = new UnseenEntities([], {
state,
io: { withBody: (_ordinal, use) => use(THIRD_BOOK), retain: () => 0 },
lastCheck: new Map(),
idField: () => "id",
});
const view = { shape: "item", type: "/api/books", pointer: "" };
const read = (index: number) => ({
record: {
ordinal: index,
method: "GET",
path: "/api/books/3",
query: undefined,
status: 200,
start: index,
end: index,
},
index,
});
const adopt = (index: number) =>
unseen.adopt(view as unknown as StateItemView, read(index));
expect(adopt(1)).toBeUndefined();
// A write after the read cannot have made what it shows either.
unseen.unplanned(2);
expect(adopt(1)).toBeUndefined();
expect(targets).not.toHaveBeenCalled();
expect(adopt(3)).toMatchObject({ id: "3" });
});
it("takes back no book a modelled delete removed, so a list showing it again confirms no delete", () => {
expect(
gated(deletedAfterRead([RENAME, IMPORT]), "delete_books_by_id"),
).toMatchObject({ served: true });
const rows: TimedCaptureRow[] = [
...BOOK_INIT_ROWS,
["POST", "/api/books", THIRD_BOOK, 201, { name: "Third Book" }],
["GET", "/api/books/3", THIRD_BOOK],
["DELETE", "/api/books/3", "", 204],
IMPORT,
// The import put the deleted book back; the copy must not take it as one the import made.
["GET", "/api/books/3", THIRD_BOOK],
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK, THIRD_BOOK]],
];
expect(gated(rows, "delete_books_by_id")).toMatchObject({
served: false,
checked_record_ordinals: [],
});
});
});
describe("an unmodelled write under another collection spelling", () => {
it("keeps a confirmed update when the write spells the held collection plainly", () => {
const acme = (name: string, status: string) => ({
src/domain/analysisView/EnvironmentStateWriteSimulation.ts
+59 -15
View File
@@ -12,6 +12,8 @@ import {
type StateType,
type StateView,
} from "./environmentStateRules.js";
import { UnseenEntities } from "./environmentStateWriteAdoption.js";
import { linkedTarget } from "./environmentStateWriteLinks.js";
import { updatedEntity } from "./environmentStateWriteApply.js";
import { projectedRead } from "./environmentStateWriteObserve.js";
import { OmittedFields } from "./environmentStateWriteOmitted.js";
@@ -54,6 +56,7 @@ import { applyDocument } from "./environmentStateWriteDocument.js";
import {
applyDelete,
applyUpdate,
built,
type EffectContext,
type EffectResult,
type WriteDraft,
@@ -116,10 +119,18 @@ export class EnvironmentStateWriteSimulation {
readonly #inserted = new Map<string, string>();
/** Creates whose insert rule a passing list check relied on, so it is no longer changed. */
readonly #placed = new Set<string>();
/** Entities writes no plan models made, taken when an item read first shows them. */
readonly #unseen: UnseenEntities;
constructor(input: WriteSimulationInput) {
this.#input = input;
this.#state = new SimulatedState(input.entities, input.documents);
this.#unseen = new UnseenEntities(input.entities, {
state: this.#state,
io: input.io,
lastCheck: this.#lastCheck,
idField: (type) => this.#types.get(type)?.id_field,
});
for (const type of input.types)
this.#types.set(type.collection_template, type);
for (const tool of input.tools)
@@ -143,8 +154,18 @@ export class EnvironmentStateWriteSimulation {
documents.set(normalizedPath(tool.path_template), tool.name);
const types = new Set(this.#types.keys());
for (const tool of this.#input.tools) {
if (!changesState(tool.method, tool.path_template)) continue;
const { effect, target } = classifyWrite(tool, types, documents);
if (
!changesState(tool.method, tool.path_template) ||
this.#input.readOnly.has(tool.name)
)
continue;
const link = linkedTarget(tool, classifyWrite(tool, types, documents), {
...this.#input,
isWrite: (record) => this.#isWrite(record),
viewOf: (ordinal) =>
this.#input.views.get(this.#toolOf.get(ordinal)?.name ?? ""),
});
const { effect, target } = link;
this.#input.io.retain(512);
this.#drafts.set(tool.name, {
plan: {
@@ -159,6 +180,7 @@ export class EnvironmentStateWriteSimulation {
calls: 0,
updates: [],
sequences: new Map(),
linked: link.collection,
});
}
this.#input.records.forEach((record, index) => this.#step(record, index));
@@ -169,15 +191,25 @@ export class EnvironmentStateWriteSimulation {
#step(record: StateRecordFacts, index: number): void {
const tool = this.#toolOf.get(record.ordinal);
// A redirected or unanswered write may have changed state, but only a 2xx one shows its effect.
if (isStateWrite(record) && !succeeded(record))
// A redirected or unanswered write may have changed state, even made an entity, but only a 2xx one shows its effect.
if (this.#isWrite(record) && !succeeded(record)) {
this.#unmodelled(record.path, index);
else if (isStateWrite(record)) this.#write(record, index, tool);
this.#unseen.unplanned(index);
} else if (this.#isWrite(record)) this.#write(record, index, tool);
else if (tool?.method === "GET") this.#read(record, index, tool);
}
/** Whether a record may change state: a state write whose tool the capture does not show reading only. */
#isWrite(record: StateRecordFacts): boolean {
return (
isStateWrite(record) &&
!this.#input.readOnly.has(this.#toolOf.get(record.ordinal)?.name ?? "")
);
}
#context(writer: string): EffectContext {
return {
isWrite: (record) => this.#isWrite(record),
state: this.#state,
io: this.#input.io,
records: this.#input.records,
@@ -217,12 +249,20 @@ export class EnvironmentStateWriteSimulation {
effect === "none"
) {
this.#unmodelled(record.path, index);
this.#unseen.unplanned(index);
return;
}
const observeOnly = draft.plan.reason !== null;
if (!observeOnly) draft.calls += 1;
const step: WriteStep = { record, index, tool, observeOnly };
this.#settle(APPLY[effect](this.#context(tool.name), draft, step), {
const result = APPLY[effect](this.#context(tool.name), draft, step);
// A call of a tool whose plan was never built, or one a linked plan does not place, is no more modelled than one of a tool without a plan.
if (
result.kind === "unknown" &&
(!built(draft) || draft.linked !== undefined)
)
this.#unseen.unplanned(index);
this.#settle(result, {
index,
tool: tool.name,
path: record.path,
@@ -244,12 +284,11 @@ export class EnvironmentStateWriteSimulation {
this.#unmodelled(write.path, write.index);
return;
}
if (result.entity !== undefined)
this.#taint.known(
result.entity.type,
result.entity.collection,
result.entity.id,
);
if (result.entity !== undefined) {
this.#unseen.touched(result.entity);
const { type, collection, id } = result.entity;
this.#taint.known(type, collection, id);
}
if (result.kind === "observed") {
this.#effects.observed(this.#touched, result.targets);
// A recorded outcome replaces the entity's values, not its collection's membership.
@@ -289,9 +328,14 @@ export class EnvironmentStateWriteSimulation {
this.#state.put(updatedEntity(entity, { ...entity.fields, ...object }));
}
/** Whether an item read is of an entity left unknown that it shows again rather than checks. */
#resynced(view: StateView, record: StateRecordFacts): boolean {
/** Whether an item read shows an entity an unmodelled write made, or one left unknown, again rather than checks it. */
#resynced(view: StateView, record: StateRecordFacts, index: number): boolean {
if (view.shape !== "item") return false;
const adopted = this.#unseen.adopt(view, { record, index });
if (adopted !== undefined) {
this.#taint.known(adopted.type, adopted.collection, adopted.id);
return true;
}
const { collection, id } = itemTarget(this.#state, view.type, record);
if (!this.#taint.unknown(view.type, collection, id)) return false;
// A read begun while a modelled write to the entity still ran may show either state, so it neither checks nor shows it.
@@ -370,7 +414,7 @@ export class EnvironmentStateWriteSimulation {
const view = this.omitted.view(original);
if (projectedRead(view, record, this.#input.io)) return;
const borrowers = this.#observations.get(record.ordinal);
if (!borrowers && this.#resynced(view, record)) return;
if (!borrowers && this.#resynced(view, record, index)) return;
let outcome = this.#check(view, record);
if (outcome === undefined) return;
let keys = [...outcome.targets, ...outcome.unheld];
src/domain/analysisView/WebNetworkProjectionAccumulator.ts
+8
View File
@@ -48,6 +48,14 @@ export abstract class WebNetworkProjectionAccumulator {
return true;
}
/** Mark the retained bytes; the returned call releases what was retained since, once none of it is held. */
protected retainedCheckpoint(): () => void {
const mark = this.#retainedBytes;
return () => {
this.#retainedBytes = Math.min(this.#retainedBytes, mark);
};
}
/** Per-record strings and descriptors never become cumulative scratch. */
protected releaseTemporary(): void {
this.#temporaryBytes = 0;
src/domain/analysisView/environmentStateGate.ts
+30 -4
View File
@@ -15,7 +15,9 @@ import {
scalarText,
type JsonObject,
type RecordedViewReason,
type ReplayKey,
type StateEntity,
type StateItemView,
type StateRecordFacts,
type StateToolFacts,
type StateView,
@@ -34,6 +36,14 @@ import { startsAtBeginning } from "./environmentStatePaging.js";
import { readClock, type ClockFormat } from "./environmentStateWriteValues.js";
import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js";
/** A successful item read first recorded after the initial reads, with the members its object carries. */
export interface LateLine {
readonly record: StateRecordFacts;
readonly members: ReadonlySet<string>;
/** Whether it asks for some fields (`fields`, `select`, …). */
readonly projecting: boolean;
}
/** A read tool whose recordings locate entities of one type at one pointer. */
export interface StatePlan {
readonly tool: StateToolFacts;
@@ -44,10 +54,27 @@ export interface StatePlan {
readonly recordings: readonly StateRecordFacts[];
/** The latest of them per request line, in timeline order. */
readonly latest: readonly StateRecordFacts[];
/** Item reads left to replay because they asked for some fields only. */
/** Reads left to replay: item reads that asked for some fields only, and representation variants. */
readonly projected?: readonly number[];
/** Item lines first recorded after the initial reads whose identity they show, decided against the lines left once variants are dropped. */
readonly late?: readonly LateLine[];
/** The query keys that pick calls of the replayed reads out. */
readonly projectedKeys?: readonly ReplayKey[];
}
/** A view's replayed reads and their keys, when it has any. */
const replayedReads = (
plan: StatePlan,
): Pick<StateItemView, "projected_record_ordinals" | "projected_query_keys"> =>
plan.projected === undefined || plan.projected.length === 0
? {}
: {
projected_record_ordinals: [...plan.projected],
...(plan.projectedKeys === undefined || plan.projectedKeys.length === 0
? {}
: { projected_query_keys: [...plan.projectedKeys] }),
};
/** What the gate reads from an extraction; `withBody` drops each body's temporary work after `use`. */
export interface StateGateContext {
readonly withBody: <T>(
@@ -257,9 +284,7 @@ export const checkItemPlan = (
required_fields: union.required,
envelope_ordinal: plan.latest[0]!.ordinal,
checked_record_ordinals: plan.latest.map((record) => record.ordinal),
...(plan.projected === undefined || plan.projected.length === 0
? {}
: { projected_record_ordinals: [...plan.projected] }),
...replayedReads(plan),
...spellings.view,
};
};
@@ -548,6 +573,7 @@ export const checkListPlan = (
counters,
...(query === undefined ? {} : { query }),
checked_record_ordinals: plan.latest.map((record) => record.ordinal),
...replayedReads(plan),
...spellings.view,
};
};
src/domain/analysisView/environmentStateListQuery.test.ts
+12
View File
@@ -250,6 +250,18 @@ describe("learning a list's query", () => {
// Serving supplies a call's values in name order.
expect(ids("?limit=1&size=2", learned!.parameters)).toEqual(["1"]);
});
it("adopts a filter on a field named unlike its parameter only once lines with two values agree on it", () => {
const owner = (...lines: QueryLine[]) =>
learnQuery(lines)?.parameters["owner"];
expect(owner(line("?owner=a", [1, 3], 50))).toBeUndefined();
expect(
owner(line("?owner=a", [1, 3], 50), line("?owner=b", [2], 51)),
).toEqual({ role: "filter", field: "shelf" });
expect(learnQuery([line("?shelf_id=a", [1, 3], 52)])?.parameters).toEqual({
shelf_id: { role: "filter", field: "shelf" },
});
});
});
describe("queries the state can always answer", () => {
src/domain/analysisView/environmentStateListQuery.ts
+31 -2
View File
@@ -549,16 +549,45 @@ const mergeRoles = (
return Object.fromEntries(merged);
};
/** Whether a filtered field is named like its parameter, an id suffix aside: `category`, `site_id` for `site`. */
const namedLike = (field: string, name: string): boolean => {
const [left, right] = [normalized(field), normalized(name)];
return left === right || right === `${left}id` || left === `${right}id`;
};
/** The roles less any filter on a field named unlike its parameter that only one recorded value supports, since one line can match by coincidence. */
const trustedRoles = (
parameters: Record<string, QueryParameter>,
explained: readonly {
readonly line: QueryLine;
readonly parameters: Record<string, QueryParameter>;
}[],
): Record<string, QueryParameter> =>
Object.fromEntries(
Object.entries(parameters).filter(([name, role]) => {
if (role.role !== "filter" || namedLike(role.field, name)) return true;
const values = new Set(
explained
.filter((entry) => entry.parameters[name]?.role === "filter")
.map((entry) => entry.line.values.get(name)),
);
return values.size >= 2;
}),
);
/** The query model that lists exactly what each explained line recorded, or undefined when none does. */
export const learnQuery = (
lines: readonly QueryLine[],
): Pick<ListQuery, "parameters" | "record_ordinals"> | undefined => {
const explained = lines.flatMap((line) => {
const parameters = explainLine(line);
return parameters === undefined ? [] : [parameters];
return parameters === undefined ? [] : [{ line, parameters }];
});
if (explained.length === 0) return undefined;
const parameters = mergeRoles(explained);
const parameters = trustedRoles(
mergeRoles(explained.map((entry) => entry.parameters)),
explained,
);
const record_ordinals = lines
.filter((line) => listsAsRecorded(line, parameters))
.map((line) => line.ordinal);
src/domain/analysisView/environmentStateMerge.ts
+22 -4
View File
@@ -9,6 +9,8 @@ import {
/** One entity object as one tool recorded it. */
export interface StateObservation {
readonly position: number;
/** Where in its read the entity was listed; 0 for an item read. */
readonly index: number;
readonly ordinal: number;
readonly tool: string;
readonly item: boolean;
@@ -79,7 +81,8 @@ export const mergeObservedEntity = (entity: ObservedEntity): StateEntity => {
* How observed values changed, per entity in timeline order. A shared field that differs from the
* previous repeat of the same request line is volatile, unless a recorded write lies between them;
* a field that differs from its previous observation on another line across a write rewrites the
* type. Either across a write means the reads do not agree on one initial state.
* type. Either across a write means the reads do not agree on one initial state; each such pair of
* observations is a span the type's initial reads can end before.
*/
export const classifyEntityChanges = (
entities: Iterable<ObservedEntity>,
@@ -88,9 +91,23 @@ export const classifyEntityChanges = (
): {
readonly rewritten: ReadonlySet<string>;
readonly volatile: ReadonlyMap<string, Set<string>>;
readonly spans: readonly {
readonly type: string;
readonly earlier: number;
readonly later: number;
}[];
} => {
const rewritten = new Set<string>();
const volatile = new Map<string, Set<string>>();
const spans: { type: string; earlier: number; later: number }[] = [];
const rewrite = (
type: string,
earlier: StateObservation,
later: StateObservation,
) => {
rewritten.add(type);
spans.push({ type, earlier: earlier.ordinal, later: later.ordinal });
};
const changed = (before: StateObservation, after: StateObservation) =>
Object.keys(after.object).filter(
(name) =>
@@ -113,7 +130,8 @@ export const classifyEntityChanges = (
if (repeat !== undefined) {
if (writeBetween(repeat, after)) {
// Key additions/removals are changes too, even when every shared field is unchanged.
if (!same(repeat.object, after.object)) rewritten.add(entity.type);
if (!same(repeat.object, after.object))
rewrite(entity.type, repeat, after);
} else
for (const name of changed(repeat, after))
volatile.set(
@@ -130,11 +148,11 @@ export const classifyEntityChanges = (
!same(previous.observation.object[name], after.object[name]) &&
writeBetween(previous.observation, after)
)
rewritten.add(entity.type);
rewrite(entity.type, previous.observation, after);
lastOfField.set(name, sighting);
}
lastOfLine.set(line, after);
});
}
return { rewritten, volatile };
return { rewritten, volatile, spans };
};
src/domain/analysisView/environmentStateQueryCalls.test.ts
+678
View File
@@ -0,0 +1,678 @@
import { describe, expect, it } from "vitest";
import type { JsonValue } from "../jsonValue.js";
import {
readOnlyTools,
type QueryCallIo,
} from "./environmentStateQueryCalls.js";
import type {
StateRecordFacts,
StateToolFacts,
} from "./environmentStateRules.js";
interface Call {
readonly method: string;
readonly path: string;
readonly query?: string;
readonly status?: number | null;
readonly response?: JsonValue;
readonly request?: JsonValue;
readonly tool: string;
readonly template?: string;
readonly start?: number;
readonly end?: number;
}
/** How the read-only pass parses response bodies and charges what it keeps of them, when a test watches either. */
interface Bodies {
readonly withBody?: QueryCallIo["withBody"];
readonly retain?: (bytes: number) => void;
}
/** The read-only tools of a timeline of calls, ordinals in order; a call spans its index unless it gives its times. */
const readOnlyOf = (calls: readonly Call[], bodies: Bodies = {}): string[] => {
const records: StateRecordFacts[] = calls.map((call, index) => ({
ordinal: index,
method: call.method,
path: call.path,
query: call.query,
status: call.status === undefined ? 200 : call.status,
start: call.start ?? index,
end: call.end ?? index,
}));
const tools = new Map<string, StateToolFacts>();
calls.forEach((call, index) => {
const known = tools.get(call.tool);
tools.set(call.tool, {
name: call.tool,
method: call.method,
path_template: call.template ?? call.path,
record_ordinals: [...(known?.record_ordinals ?? []), index],
});
});
const withBody: QueryCallIo["withBody"] =
bodies.withBody ?? ((ordinal, use) => use(calls[ordinal]?.response));
return [
...readOnlyTools([...tools.values()], records, {
withBody,
request: (ordinal) => calls[ordinal]?.request,
retain: bodies.retain ?? (() => undefined),
}),
].sort();
};
const JOBS = [{ id: "night-watch", status: "running" }];
describe("POSTs that read state", () => {
it("reads a POST answering as the latest GET of its path did, with no write between", () => {
const get = {
method: "GET",
path: "/v1/jobs/statuses",
tool: "get_statuses",
response: JOBS,
};
const post = {
method: "POST",
path: "/v1/jobs/statuses",
tool: "post_statuses",
request: { jobs: ["night-watch"] },
};
expect(readOnlyOf([get, { ...post, response: JOBS }])).toEqual([
"post_statuses",
]);
expect(readOnlyOf([get, { ...post, response: [] }])).toEqual([]);
const write = {
method: "PUT",
path: "/v1/job/night-watch",
tool: "put_job",
response: {},
};
expect(readOnlyOf([get, write, { ...post, response: JOBS }])).toEqual([]);
});
it("reads an RPC tunnel whose calls name only read operations, in the body or the query", () => {
const tunnel = (operations: string[]): Call => ({
method: "POST",
path: "/api/v2/internal/nc/base1",
query: "?operation=batch",
template: "/api/v2/internal/nc/{uid}",
tool: "post_internal",
request: {
operations: operations.map((operation) => ({ operation, query: {} })),
},
response: { results: [] },
});
expect(readOnlyOf([tunnel(["tableSyncList", "listScripts"])])).toEqual([
"post_internal",
]);
expect(
readOnlyOf([tunnel(["tableList"]), tunnel(["tableCreate"])]),
).toEqual([]);
// A container operation names no operation of its own.
expect(readOnlyOf([{ ...tunnel([]), request: {} }])).toEqual([]);
const queried = {
method: "POST",
path: "/rpc",
query: "?op=getUser",
tool: "post_rpc",
response: {},
};
expect(readOnlyOf([queried])).toEqual(["post_rpc"]);
});
it("skips a container operation in the body as in the query, still needing a real read", () => {
const rpc = (request: JsonValue): Call => ({
method: "POST",
path: "/rpc",
tool: "post_rpc",
request,
response: {},
});
const batch = (operation: string): JsonValue => ({
operation: "batch",
operations: [{ operation }],
});
expect(readOnlyOf([rpc(batch("getUser"))])).toEqual(["post_rpc"]);
expect(readOnlyOf([rpc(batch("createUser"))])).toEqual([]);
expect(readOnlyOf([rpc({ operation: "batch" })])).toEqual([]);
});
it("reads a POST whose path names a lookup word, split on underscores, hyphens and case", () => {
const post = (path: string, tool: string, template = path): Call => ({
method: "POST",
path,
template,
tool,
response: [],
});
expect(
readOnlyOf([
post("/api/documents/selection_data/", "post_selection"),
post(
"/api/v4/channels/c1/members/ids",
"post_member_ids",
"/api/v4/channels/{uid}/members/ids",
),
post("/v1/discover", "post_discover"),
post("/api/members", "post_members"),
]),
).toEqual(["post_discover", "post_member_ids", "post_selection"]);
});
it("reads a lookup by a segment's last meaningful word, so a query history stays a write", () => {
const post = (path: string, tool: string): Call => ({
method: "POST",
path,
tool,
response: {},
});
expect(
readOnlyOf([
post("/api/query-history", "post_query_history"),
post("/api/documents/selection_data/", "post_selection"),
post("/api/search-results", "post_search_results"),
]),
).toEqual(["post_search_results", "post_selection"]);
});
it("reads a lookup only by the last literal segment, so a trailing write action stays a write", () => {
const post = (path: string, tool: string, template = path): Call => ({
method: "POST",
path,
template,
tool,
response: {},
});
expect(
readOnlyOf([
post("/api/documents/selection_data/import", "post_import"),
post("/api/saved-query/create", "post_saved_query"),
post("/api/views/create_lookup", "post_create_lookup"),
post(
"/api/lookup_data/users",
"post_lookup",
"/api/lookup_data/{kind}",
),
]),
).toEqual(["post_lookup"]);
});
it("reads a session renewal, not a create in the same collection tree", () => {
const post = (path: string, tool: string): Call => ({
method: "POST",
path,
tool,
response: { token: "x" },
});
expect(
readOnlyOf([
post("/api/collections/_superusers/auth-refresh", "post_auth_refresh"),
post("/api/collections/notes/records", "post_records"),
]),
).toEqual(["post_auth_refresh"]);
});
});
describe("POSTs that read state but stay writes", () => {
it("keeps a lookup-named POST a write when a read its path relates to changes across it alone", () => {
const read = (members: string[]): Call => ({
method: "GET",
path: "/api/members",
tool: "get_members",
response: members,
});
const post: Call = {
method: "POST",
path: "/api/members/ids",
tool: "post_ids",
response: ["a"],
};
expect(readOnlyOf([read(["a"]), post, read(["a", "b"])])).toEqual([]);
expect(readOnlyOf([read(["a"]), post, read(["a"])])).toEqual(["post_ids"]);
// Another write between explains the change, so the POST stays read-only.
const other: Call = {
method: "PUT",
path: "/api/members/b",
tool: "put_member",
response: {},
};
expect(readOnlyOf([read(["a"]), post, other, read(["a", "b"])])).toEqual([
"post_ids",
]);
});
it("keeps a lookup-named POST a write when only another read-only POST sits between it and a changed read", () => {
const read = (members: string[]): Call => ({
method: "GET",
path: "/api/members",
tool: "get_members",
response: members,
});
const post: Call = {
method: "POST",
path: "/api/members/ids",
tool: "post_ids",
response: ["a"],
};
const refresh: Call = {
method: "POST",
path: "/api/collections/_superusers/auth-refresh",
tool: "post_auth_refresh",
response: { token: "x" },
};
expect(readOnlyOf([read(["a"]), post, refresh, read(["a", "b"])])).toEqual([
"post_auth_refresh",
]);
expect(readOnlyOf([read(["a"]), refresh, post, read(["a", "b"])])).toEqual([
"post_auth_refresh",
]);
});
it("keeps a POST a write when one of its calls was redirected or went unanswered", () => {
const post = (status: number | null): Call => ({
method: "POST",
path: "/api/members/ids",
tool: "post_ids",
status,
response: ["a"],
});
expect(readOnlyOf([post(200), post(302)])).toEqual([]);
expect(readOnlyOf([post(200), post(null)])).toEqual([]);
// A refused call changed nothing, so it does not hold the tool.
expect(readOnlyOf([post(200), post(400)])).toEqual(["post_ids"]);
});
it("keeps a batch a write when one of its entries names no read operation of its own", () => {
const rpc = (operations: JsonValue): Call => ({
method: "POST",
path: "/rpc",
tool: "post_rpc",
request: { operation: "batch", operations },
response: {},
});
const read = { operation: "getUser" };
expect(readOnlyOf([rpc([read, read])])).toEqual(["post_rpc"]);
expect(readOnlyOf([rpc([read, { method: "deleteUser" }])])).toEqual([]);
expect(readOnlyOf([rpc([read, "deleteUser"])])).toEqual([]);
expect(readOnlyOf([rpc([read, { operation: 7 }])])).toEqual([]);
expect(
readOnlyOf([rpc([read, { operation: "batch", operations: [] }])]),
).toEqual([]);
expect(readOnlyOf([rpc({ first: read })])).toEqual([]);
});
it("keeps an RPC call a write when it finds and then modifies or replaces", () => {
const rpc = (operation: string): Call => ({
method: "POST",
path: "/rpc",
tool: "post_rpc",
request: { operation },
response: {},
});
expect(readOnlyOf([rpc("findOne")])).toEqual(["post_rpc"]);
expect(readOnlyOf([rpc("findAndModify")])).toEqual([]);
expect(readOnlyOf([rpc("findOneAndReplace")])).toEqual([]);
});
});
describe("the reads either side of a POST", () => {
it("compares a POST only with reads that finished before it began or began after it ended", () => {
const note = (title: string): Call => ({
method: "GET",
path: "/v1/notes/7",
template: "/v1/notes/{id}",
tool: "get_note",
response: { id: 7, title },
});
const post: Call = {
method: "POST",
path: "/v1/notes/7",
template: "/v1/notes/{id}",
tool: "post_note",
request: { title: "Renamed" },
response: { id: 7, title: "Renamed" },
start: 10,
end: 20,
};
const at = (call: Call, start: number, end: number): Call => ({
...call,
start,
end,
});
const renamed = note("Renamed");
const original = note("Draft");
expect(readOnlyOf([at(renamed, 0, 5), post, at(renamed, 40, 41)])).toEqual([
"post_note",
]);
// A read still running when the POST began may already show its change.
expect(readOnlyOf([at(renamed, 0, 30), post, at(renamed, 40, 41)])).toEqual(
[],
);
// A read begun before the POST ended may not show it yet, so the later read decides.
expect(
readOnlyOf([
at(original, 0, 5),
{ ...post, response: original.response! },
at(original, 15, 25),
at(renamed, 40, 41),
]),
).toEqual([]);
// Of the reads finished before the POST began, the one that finished last decides, not the one begun last.
expect(
readOnlyOf([
at(original, 0, 9),
at(renamed, 1, 2),
post,
at(renamed, 40, 41),
]),
).toEqual([]);
});
it("compares a POST with a read begun after another write, which shows that write's effect", () => {
const message = { id: 1001, body: "hello" };
const get = (
start: number,
end: number,
status = 200,
response: JsonValue = message,
): Call => ({
method: "GET",
path: "/api/messages/1001",
template: "/api/messages/{id}",
tool: "get_message",
status,
response,
start,
end,
});
const put: Call = {
method: "PUT",
path: "/api/messages/1001",
template: "/api/messages/{id}",
tool: "put_message",
response: message,
start: 1,
end: 2,
};
const consume: Call = {
method: "POST",
path: "/api/messages/1001/ids",
template: "/api/messages/{id}/ids",
tool: "post_ids",
response: ["1001"],
start: 10,
end: 11,
};
// The read that finished last began before the PUT; the shorter one after it shows the message survived the PUT.
expect(
readOnlyOf([
get(0, 9),
put,
get(3, 4),
consume,
get(12, 13, 404, { error: "Not found" }),
]),
).toEqual([]);
});
it("keeps a POST a write when a read of its path finds the entity gone after it", () => {
const message = { id: 1001, body: "hello" };
const get = (status: number, response: JsonValue): Call => ({
method: "GET",
path: "/api/messages/1001",
template: "/api/messages/{id}",
tool: "get_message",
status,
response,
});
const post: Call = {
method: "POST",
path: "/api/messages/1001",
template: "/api/messages/{id}",
tool: "post_message",
response: message,
};
expect(readOnlyOf([get(200, message), post, get(200, message)])).toEqual([
"post_message",
]);
expect(
readOnlyOf([get(200, message), post, get(404, { error: "Not found" })]),
).toEqual([]);
});
});
describe("read-only decisions over many calls", () => {
it("visits each write-free stretch once for all of a lookup's calls in it", () => {
let visits = 0;
const calls = 2000;
const records: StateRecordFacts[] = Array.from(
{ length: calls * 2 },
(_, index) => {
const read = index % 2 === 0;
const record = {
ordinal: index,
path: read ? "/api/members" : "/api/members/ids",
query: undefined,
status: 200,
start: index,
end: index,
};
const method = read ? "GET" : "POST";
return Object.defineProperty(record, "method", {
enumerable: true,
get: () => {
visits += 1;
return method;
},
}) as StateRecordFacts;
},
);
const ordinals = (read: boolean) =>
records
.filter((record) => (record.ordinal % 2 === 0) === read)
.map((record) => record.ordinal);
const tools: StateToolFacts[] = [
{
name: "get_members",
method: "GET",
path_template: "/api/members",
record_ordinals: ordinals(true),
},
{
name: "post_ids",
method: "POST",
path_template: "/api/members/ids",
record_ordinals: ordinals(false),
},
];
visits = 0;
expect([
...readOnlyTools(tools, records, {
withBody: (_ordinal, use) => use([]),
request: () => undefined,
retain: () => undefined,
}),
]).toEqual(["post_ids"]);
expect(visits).toBeLessThan(50 * records.length);
});
});
describe("read-only decisions over a long run of calls", () => {
it("finds each call's repeated read without rescanning the run before it", () => {
let visits = 0;
const calls = 2000;
const records: StateRecordFacts[] = Array.from(
{ length: calls * 2 },
(_, index) => {
const read = index % 2 === 0;
const record = {
ordinal: index,
path: "/v1/jobs/statuses",
query: undefined,
status: 200,
start: index,
end: index,
};
const method = read ? "GET" : "POST";
return Object.defineProperty(record, "method", {
enumerable: true,
get: () => {
visits += 1;
return method;
},
}) as StateRecordFacts;
},
);
const ordinals = (read: boolean) =>
records
.filter((record) => (record.ordinal % 2 === 0) === read)
.map((record) => record.ordinal);
const tools: StateToolFacts[] = [
{
name: "get_statuses",
method: "GET",
path_template: "/v1/jobs/statuses",
record_ordinals: ordinals(true),
},
{
name: "post_statuses",
method: "POST",
path_template: "/v1/jobs/statuses",
record_ordinals: ordinals(false),
},
];
visits = 0;
expect([
...readOnlyTools(tools, records, {
withBody: (_ordinal, use) => use(JOBS),
request: () => ({ jobs: ["night-watch"] }),
retain: () => undefined,
}),
]).toEqual(["post_statuses"]);
expect(visits).toBeLessThan(50 * records.length);
});
});
describe("read-only decisions over many lines", () => {
it("reads each related line's records once per call, however many lines the lookup's path relates to", () => {
let reads = 0;
const lines = 2000;
const records: StateRecordFacts[] = Array.from(
{ length: lines + 1 },
(_, index) => {
const read = index < lines;
const record = {
ordinal: index,
method: read ? "GET" : "POST",
path: read ? "/api/members" : "/api/members/ids",
status: 200,
start: index,
end: index,
};
const query = read ? `?page=${String(index)}` : undefined;
return Object.defineProperty(record, "query", {
enumerable: true,
get: () => {
reads += 1;
return query;
},
}) as StateRecordFacts;
},
);
const tools: StateToolFacts[] = [
{
name: "get_members",
method: "GET",
path_template: "/api/members",
record_ordinals: records
.slice(0, lines)
.map((record) => record.ordinal),
},
{
name: "post_ids",
method: "POST",
path_template: "/api/members/ids",
record_ordinals: [lines],
},
];
expect([
...readOnlyTools(tools, records, {
withBody: (_ordinal, use) => use([]),
request: () => undefined,
retain: () => undefined,
}),
]).toEqual(["post_ids"]);
expect(reads).toBeLessThan(20 * records.length);
});
});
describe("the bodies read-only decisions compare", () => {
const read = (members: string[]): Call => ({
method: "GET",
path: "/api/members",
tool: "get_members",
response: members,
});
const lookup: Call = {
method: "POST",
path: "/api/members/ids",
tool: "post_ids",
response: ["a"],
};
const refresh: Call = {
method: "POST",
path: "/api/collections/_superusers/auth-refresh",
tool: "post_auth_refresh",
response: { token: "x" },
};
// The change across the lookup's calls drops it on the first pass, so the refresh is checked again.
const calls = [read(["a"]), lookup, lookup, refresh, read(["a", "b"])];
it("parses each read once across a tool's calls and the passes that drop vetoed tools", () => {
const parses = new Map<number, number>();
expect(
readOnlyOf(calls, {
withBody: (ordinal, use) => {
parses.set(ordinal, (parses.get(ordinal) ?? 0) + 1);
return use(calls[ordinal]?.response);
},
}),
).toEqual(["post_auth_refresh"]);
expect([...parses].filter(([, count]) => count > 1)).toEqual([]);
});
it("compares bodies only while their parse is charged, and charges what it keeps of each", () => {
const [parsed, outside] = [new Set<number>(), new Set<number>()];
let parsing = false;
let retained = 0;
expect(
readOnlyOf(calls, {
withBody: (ordinal, use) => {
parsed.add(ordinal);
const body = calls[ordinal]?.response;
parsing = true;
try {
return use(
typeof body === "object" && body !== null
? new Proxy(body, {
get: (target, key, receiver) => {
if (!parsing) outside.add(ordinal);
return Reflect.get(target, key, receiver) as unknown;
},
})
: body,
);
} finally {
parsing = false;
}
},
retain: (bytes) => {
retained += bytes;
},
}),
).toEqual(["post_auth_refresh"]);
expect([...outside]).toEqual([]);
// Each body is kept as its digest's hex text, charged before it is.
expect(retained).toBeGreaterThanOrEqual(parsed.size * 64);
});
});
src/domain/analysisView/environmentStateQueryCalls.ts
+538
View File
@@ -0,0 +1,538 @@
import { digestCanonicalValue } from "../canonicalDigest.js";
import type { JsonValue } from "../jsonValue.js";
import {
changesState,
isObject,
isParam,
isStateWrite,
QUERY_SEGMENTS,
segmentsOf,
type StateRecordFacts,
type StateToolFacts,
} from "./environmentStateRules.js";
import { isAbsence } from "./environmentStateWriteChecks.js";
import { prefixRelated } from "./environmentStateWriteRules.js";
import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js";
/** The words of a segment or operation name: split on `_`, `-`, `.` and camelCase, in lower case. */
const words = (text: string): string[] =>
text
.split(/[_\-.\s]+/u)
.flatMap((part) => part.split(/(?<=[a-z0-9])(?=[A-Z])/u))
.map((word) => word.toLowerCase())
.filter((word) => word.length > 0);
/** Path words that name a lookup rather than a change. */
const QUERY_WORDS = new Set([
...[...QUERY_SEGMENTS].flatMap(words),
"ids",
"selection",
"discover",
]);
/** Trailing words that only qualify what a segment names, so the word before them decides. */
const NEUTRAL_WORDS = new Set(["data", "info", "results"]);
/** Whole path segments that renew or open a session rather than change state. */
const SESSION_SEGMENTS = new Set([
"auth-refresh",
"auth-with-password",
"login",
"logout",
"refresh",
"token",
]);
const READ_OPERATION_WORDS = new Set([
"list",
"get",
"count",
"aggregate",
"info",
"search",
"find",
"read",
"query",
"schema",
"describe",
"fetch",
"lookup",
]);
const WRITE_OPERATION_WORDS = new Set([
"create",
"insert",
"update",
"modify",
"replace",
"delete",
"remove",
"upsert",
"add",
"set",
"save",
"patch",
"put",
"post",
"write",
"move",
"copy",
"duplicate",
"import",
"run",
"execute",
"trigger",
"send",
"reset",
"clear",
"archive",
"restore",
]);
/** Operation names that only wrap the operations a request body names. */
const OPERATION_CONTAINERS = new Set(["batch", "bulk", "multi"]);
const OPERATION_PARAMETERS = ["operation", "op", "action"];
/** What deciding read-only calls reads besides the tools and the timeline. */
export interface QueryCallIo {
/** Parse one response body and hand it to `use` while its parse is charged; unusable bodies arrive undefined. */
readonly withBody: <T>(
ordinal: number,
use: (body: JsonValue | undefined) => T,
) => T;
readonly request: (ordinal: number) => JsonValue | undefined;
readonly retain: (bytes: number) => void;
}
/** Each response body's canonical digest, taken once while its parse is charged and charged itself once kept; undefined when unusable. */
const responseDigests = (
io: QueryCallIo,
): ((ordinal: number) => string | undefined) => {
const digests = new Map<number, string | undefined>();
return (ordinal) => {
if (digests.has(ordinal)) return digests.get(ordinal);
const digest = io.withBody(ordinal, (body) =>
body === undefined
? undefined
: digestCanonicalValue(body, "Environment state"),
);
io.retain(ENTRY_BYTES + textBytes(digest?.length ?? 0));
digests.set(ordinal, digest);
return digest;
};
};
const succeeded = (record: StateRecordFacts): boolean =>
record.status !== null && record.status >= 200 && record.status < 300;
/** A call that may have changed state without a 2xx answer saying what it did: redirected or unanswered. */
const unresolved = (record: StateRecordFacts): boolean =>
isStateWrite(record) && !succeeded(record);
/** Whether a name is an operation's: a string that is no container wrapping others. */
const isOperation = (name: JsonValue | undefined): name is string =>
typeof name === "string" && !OPERATION_CONTAINERS.has(name.toLowerCase());
/** The names among these that are operations rather than containers wrapping them. */
const realOperations = (names: readonly (JsonValue | undefined)[]): string[] =>
names.filter(isOperation);
/**
* The operations an RPC call names: its body's `operation` or `operations[].operation`, else its
* query's, containers aside. Undefined when `operations` is no list or one of its entries names no
* operation of its own: no string, or another container.
*/
const operationsOf = (
record: StateRecordFacts,
body: JsonValue | undefined,
): string[] | undefined => {
const listed = isObject(body) ? body.operations : undefined;
const entries = Array.isArray(listed)
? listed.map((entry) => (isObject(entry) ? entry.operation : undefined))
: [];
if (
(listed !== undefined && !Array.isArray(listed)) ||
!entries.every(isOperation)
)
return undefined;
const named = isObject(body)
? realOperations([body.operation, ...entries])
: [];
if (named.length > 0) return named;
const query = new URLSearchParams(record.query ?? "");
return realOperations(
OPERATION_PARAMETERS.flatMap((name) => query.getAll(name)),
);
};
/** An operation that reads: it names a read word and no write word. */
const readsOnly = (operation: string): boolean => {
const named = words(operation);
return (
named.some((word) => READ_OPERATION_WORDS.has(word)) &&
!named.some((word) => WRITE_OPERATION_WORDS.has(word))
);
};
/**
* POST tools whose recorded calls read state rather than change it, by the strongest evidence that
* applies: a response repeating the latest read of the same path finished before it, with no write
* between, an RPC tunnel naming only read operations, a last path word naming a lookup, or a session
* segment. A tool with a redirected or unanswered call stays a write, as does one a read line its
* path relates to changes across or finds gone after, no write but the other read-only tools' calls between.
*/
export const readOnlyTools = (
tools: readonly StateToolFacts[],
records: readonly StateRecordFacts[],
io: QueryCallIo,
): Set<string> => {
const position = new Map(
records.map((record, index) => [record.ordinal, index]),
);
const response = responseDigests(io);
/** The timeline indexes of the nearest state writes either side of an index, the skipped calls aside: -1 and the length when none, found in one pass each way. */
const nearestWrites = (
skip: ReadonlySet<number>,
): Pick<Timeline, "writes" | "isWrite"> => {
const writes = records.map(
(record) => !skip.has(record.ordinal) && isStateWrite(record),
);
const before: number[] = [];
const after: number[] = Array.from({ length: records.length });
for (let index = 0, last = -1; index < records.length; index += 1) {
before.push(last);
if (writes[index]) last = index;
}
for (
let index = records.length - 1, next = records.length;
index >= 0;
index -= 1
) {
after[index] = next;
if (writes[index]) next = index;
}
return {
writes: (at) => ({ before: before[at]!, after: after[at]! }),
isWrite: (index) => writes[index]!,
};
};
const candidates = new Map<string, readonly StateRecordFacts[]>();
for (const tool of tools) {
if (
tool.method !== "POST" ||
!changesState(tool.method, tool.path_template)
)
continue;
const calls = tool.record_ordinals
.map((ordinal) => records[position.get(ordinal) ?? -1])
.filter((record): record is StateRecordFacts => record !== undefined);
if (!calls.some(succeeded) || calls.some(unresolved)) continue;
const timeline = {
position,
...nearestWrites(new Set(tool.record_ordinals)),
response,
};
if (
repeatsRead(calls, records, timeline) ||
namesReadOperations(calls, io) ||
namesLookup(tool.path_template)
)
candidates.set(tool.name, calls);
}
// No candidate's call explains a read change across another's, so drop vetoed candidates until none is.
for (let vetoed = true; vetoed;) {
const timeline = {
position,
...nearestWrites(
new Set([...candidates.values()].flat().map((call) => call.ordinal)),
),
response,
};
vetoed = false;
for (const [name, calls] of candidates)
if (changesReadsAcross(calls, records, timeline)) {
candidates.delete(name);
vetoed = true;
}
}
return new Set(candidates.keys());
};
interface Timeline {
readonly position: ReadonlyMap<number, number>;
/** The nearest other writes either side of a timeline index. */
readonly writes: (at: number) => {
readonly before: number;
readonly after: number;
};
/** Whether the record at a timeline index is another write. */
readonly isWrite: (index: number) => boolean;
/** A response body's canonical digest; undefined when unusable. */
readonly response: (ordinal: number) => string | undefined;
}
/** Of these reads, the one that finished last before `call` began, a later one on a tie; records come in start order. */
const lastFinishedBefore = (
reads: readonly StateRecordFacts[],
call: StateRecordFacts,
): StateRecordFacts | undefined => {
let latest: StateRecordFacts | undefined;
for (const read of reads)
if (
read.end <= call.start &&
(latest === undefined || read.end >= latest.end)
)
latest = read;
return latest;
};
/** Every successful call answers as the successful GET of its path that finished last before it began, of those since the last other write, did. */
const repeatsRead = (
calls: readonly StateRecordFacts[],
records: readonly StateRecordFacts[],
{ position, isWrite, response }: Timeline,
): boolean => {
const answering = calls.filter(succeeded);
const reads = lastReads(answering, records, position, isWrite);
return answering.every((call) => {
const read = reads.get(call.ordinal);
if (read === undefined) return false;
const answered = response(call.ordinal);
return answered !== undefined && answered === response(read.ordinal);
});
};
/**
* Each call's successful GET of its path that finished last before it began, of those since the last
* other write, in one sweep: a path's reads wait by end time until a call's start passes them.
*/
const lastReads = (
calls: readonly StateRecordFacts[],
records: readonly StateRecordFacts[],
position: ReadonlyMap<number, number>,
isWrite: (index: number) => boolean,
): Map<number, StateRecordFacts> => {
const paths = new Set(calls.map((call) => call.path));
const callAt = new Map(
calls.map((call) => [position.get(call.ordinal)!, call]),
);
const lastCall = calls.reduce(
(latest, call) => Math.max(latest, position.get(call.ordinal)!),
-1,
);
const waiting = new Map<string, ReadsByEnd>();
const latest = new Map<string, StateRecordFacts>();
const found = new Map<number, StateRecordFacts>();
for (let index = 0; index <= lastCall; index += 1) {
const record = records[index]!;
const call = callAt.get(index);
if (call !== undefined) {
const ready = waiting.get(call.path);
for (
let read = ready?.popEndedBy(call.start);
read !== undefined;
read = ready?.popEndedBy(call.start)
)
latest.set(call.path, read);
const read = latest.get(call.path);
if (read !== undefined) found.set(call.ordinal, read);
} else if (isWrite(index)) {
waiting.clear();
latest.clear();
} else if (
record.method === "GET" &&
succeeded(record) &&
paths.has(record.path)
) {
const heap = waiting.get(record.path) ?? new ReadsByEnd(position);
heap.push(record);
waiting.set(record.path, heap);
}
}
return found;
};
/** Reads in a binary heap by end time, then timeline position, so the one that finished first comes out first. */
class ReadsByEnd {
readonly #reads: StateRecordFacts[] = [];
constructor(private readonly position: ReadonlyMap<number, number>) {}
#before(left: StateRecordFacts, right: StateRecordFacts): boolean {
return (
left.end < right.end ||
(left.end === right.end &&
this.position.get(left.ordinal)! < this.position.get(right.ordinal)!)
);
}
push(read: StateRecordFacts): void {
const reads = this.#reads;
reads.push(read);
for (let at = reads.length - 1; at > 0;) {
const parent = (at - 1) >>> 1;
if (!this.#before(reads[at]!, reads[parent]!)) break;
[reads[at], reads[parent]] = [reads[parent]!, reads[at]!];
at = parent;
}
}
/** The read that finished first, when it finished by `time`. */
popEndedBy(time: number): StateRecordFacts | undefined {
const reads = this.#reads;
const first = reads[0];
if (first === undefined || first.end > time) return undefined;
const last = reads.pop()!;
if (reads.length > 0) {
reads[0] = last;
for (let at = 0; ;) {
const [left, right] = [at * 2 + 1, at * 2 + 2];
let next = at;
if (left < reads.length && this.#before(reads[left]!, reads[next]!))
next = left;
if (right < reads.length && this.#before(reads[right]!, reads[next]!))
next = right;
if (next === at) break;
[reads[at], reads[next]] = [reads[next]!, reads[at]!];
at = next;
}
}
return first;
}
}
/** Every call names only read operations. */
const namesReadOperations = (
calls: readonly StateRecordFacts[],
io: QueryCallIo,
): boolean =>
calls.every((call) => {
const operations = operationsOf(call, io.request(call.ordinal));
return (
operations !== undefined &&
operations.length > 0 &&
operations.every(readsOnly)
);
});
/** The path's last literal segment is a session segment, or names a lookup by its last meaningful word and no write word. */
const namesLookup = (template: string): boolean => {
const last = segmentsOf(template).findLast((segment) => !isParam(segment));
if (last === undefined) return false;
return (
SESSION_SEGMENTS.has(last.toLowerCase()) ||
(QUERY_WORDS.has(lastMeaningfulWord(last) ?? "") &&
!words(last).some((word) => WRITE_OPERATION_WORDS.has(word)))
);
};
/** A segment's last word that is not a trailing neutral word: `selection_data` names a selection, `query-history` a history. */
const lastMeaningfulWord = (segment: string): string | undefined =>
words(segment).findLast((word) => !NEUTRAL_WORDS.has(word));
/**
* A GET line whose path a call's path relates to reads differently, or finds its entity present on one
* side only, just before and just after it, with no other write between: the read that finished last
* before it began and the first begun after it ended, since one overlapping it may show either state.
* Calls in one write-free stretch under one path share a pass over it, and only lines whose reads
* there disagree can change across a call.
*/
const changesReadsAcross = (
calls: readonly StateRecordFacts[],
records: readonly StateRecordFacts[],
{ position, writes, response }: Timeline,
): boolean => {
const stretches = new Map<
string,
{ from: number; to: number; path: string; calls: StateRecordFacts[] }
>();
for (const call of calls.filter(succeeded)) {
const { before, after } = writes(position.get(call.ordinal)!);
const key = JSON.stringify([before, after, call.path]);
const stretch = stretches.get(key);
if (stretch === undefined)
stretches.set(key, {
from: before,
to: after,
path: call.path,
calls: [call],
});
else stretch.calls.push(call);
}
for (const stretch of stretches.values())
if (changesInStretch(stretch, records, { position, response })) return true;
return false;
};
/** Whether a related line changes across one of these calls, all between the same two writes under one path. */
const changesInStretch = (
stretch: {
readonly from: number;
readonly to: number;
readonly path: string;
readonly calls: readonly StateRecordFacts[];
},
records: readonly StateRecordFacts[],
{ position, response }: Pick<Timeline, "position" | "response">,
): boolean => {
const shown = (record: StateRecordFacts): string | symbol =>
succeeded(record) ? (response(record.ordinal) ?? UNKNOWN) : ABSENT;
const lines = new Map<string, StateRecordFacts[]>();
for (let index = stretch.from + 1; index < stretch.to; index += 1) {
const record = records[index]!;
if (
record.method !== "GET" ||
!(succeeded(record) || isAbsence(record.status)) ||
!prefixRelated(stretch.path, record.path)
)
continue;
const line = `${record.path}${record.query ?? ""}`;
const reads = lines.get(line);
if (reads === undefined) lines.set(line, [record]);
else reads.push(record);
}
// A line whose reads all show one thing cannot change across any call.
const changing = [...lines.values()].filter((reads) => {
const first = shown(reads[0]!);
return reads.some((read) => shown(read) !== first);
});
return stretch.calls.some((call) => {
const at = position.get(call.ordinal)!;
return changing.some((reads) => {
const earlier = lastFinishedBefore(
reads.filter((read) => position.get(read.ordinal)! < at),
call,
);
const later = reads.find(
(read) => position.get(read.ordinal)! > at && read.start >= call.end,
);
if (earlier === undefined || later === undefined) return false;
const [was, now] = [shown(earlier), shown(later)];
// Presence on one side only is a change even when a body is unusable.
if ((was === ABSENT) !== (now === ABSENT)) return true;
return was !== UNKNOWN && now !== UNKNOWN && was !== now;
});
});
};
/** A read that found its entity gone, and one whose body is unusable. */
const ABSENT = Symbol("absent");
const UNKNOWN = Symbol("unknown");
/** The timeline's state writes, less the calls of tools whose calls only read, and those tools. */
export const stateWritesOf = (
tools: readonly StateToolFacts[],
records: readonly StateRecordFacts[],
io: QueryCallIo,
): {
readonly readOnly: ReadonlySet<string>;
readonly writes: readonly StateRecordFacts[];
} => {
const readOnly = readOnlyTools(tools, records, io);
const reads = new Set(
tools
.filter((tool) => readOnly.has(tool.name))
.flatMap((tool) => tool.record_ordinals),
);
return {
readOnly,
writes: records.filter(
(record) => isStateWrite(record) && !reads.has(record.ordinal),
),
};
};
src/domain/analysisView/environmentStateRewrites.ts
+48
View File
@@ -0,0 +1,48 @@
import type { StateRecordFacts } from "./environmentStateRules.js";
/** Two recordings of a type that disagree across a write. */
export interface RewriteSpan {
readonly type: string;
readonly earlier: number;
readonly later: number;
}
/**
* Per type whose reads disagree across writes, the earliest write within any such span: one that
* began before the later recording ended and ended after the earlier began, in timeline order.
*/
export const rewriteCuts = (
spans: readonly RewriteSpan[],
writes: readonly StateRecordFacts[],
byOrdinal: ReadonlyMap<number, StateRecordFacts>,
): Map<string, StateRecordFacts> => {
// The latest end among each prefix of writes, so the first write ending after a time is found by bisection.
const reach: number[] = [];
for (const write of writes)
reach.push(Math.max(reach.at(-1) ?? Number.NEGATIVE_INFINITY, write.end));
const starts = writes.map((write) => write.start);
/** The first index whose value passes `test`, which once true stays true. */
const first = (
values: readonly number[],
test: (value: number) => boolean,
) => {
let [low, high] = [0, values.length];
while (low < high) {
const middle = (low + high) >>> 1;
if (test(values[middle]!)) high = middle;
else low = middle + 1;
}
return low;
};
const cuts = new Map<string, number>();
for (const span of spans) {
const earlier = byOrdinal.get(span.earlier);
const later = byOrdinal.get(span.later);
if (earlier === undefined || later === undefined) continue;
// Writes come in start order: those begun before the later recording ended are a prefix.
const index = first(reach, (end) => end > earlier.start);
if (index < first(starts, (start) => start >= later.end))
cuts.set(span.type, Math.min(index, cuts.get(span.type) ?? index));
}
return new Map([...cuts].map(([type, index]) => [type, writes[index]!]));
};
src/domain/analysisView/environmentStateRules.ts
+39 -15
View File
@@ -44,6 +44,8 @@ export interface StateBudget {
/** Whether a record's response had no body. */
readonly emptyResponse: (ordinal: number) => boolean;
readonly retain: (bytes: number) => boolean;
/** Mark the retained bytes; the returned call releases what was retained since, once none of it is held. */
readonly checkpoint: () => () => void;
/** Drop the temporary work of the body just read. */
readonly release: () => void;
}
@@ -107,12 +109,26 @@ interface EntityViewPlan {
readonly clock_formats?: Readonly<Record<string, ClockFormat>> | undefined;
}
export type StateItemView = EntityViewPlan & {
readonly shape: "item";
readonly envelope_ordinal: number;
/** Recorded reads that asked for some fields only and did not return the item, which replay answers. */
/** A query key that picks out replayed reads: a parameter, a parameter with one value, or a parameter left out. */
export interface ReplayKey {
readonly name: string;
readonly value?: string | undefined;
readonly absent?: true | undefined;
}
/** Recorded reads a view leaves to replay, and the query keys that pick calls of them out. */
interface ReplayedReads {
/** Reads that asked for some fields only, or show entities in another representation than the checked lines. */
readonly projected_record_ordinals?: number[] | undefined;
};
/** A call whose query matches one of these replays, a field selection by its values; absent on views that only left out projections. */
readonly projected_query_keys?: ReplayKey[] | undefined;
}
export type StateItemView = EntityViewPlan &
ReplayedReads & {
readonly shape: "item";
readonly envelope_ordinal: number;
};
/** A query parameter that asks for some fields only (`fields`, `fields[...]`, `select`, `$select`). */
export const isProjectionParameter = (name: string): boolean => {
@@ -130,15 +146,23 @@ export const projectsFields = (query: string | undefined): boolean =>
query !== undefined &&
[...new URLSearchParams(query).keys()].some(isProjectionParameter);
export type StateListView = EntityViewPlan & {
readonly shape: "list";
readonly membership: "all" | "recorded";
/** With recorded membership, the checked lines that each list their whole collection. */
readonly all_record_ordinals?: number[] | undefined;
readonly counters: string[];
/** The query the view's other lines follow, when one explains any of them. */
readonly query?: ListQuery | undefined;
};
/** Whether a recorded query asks for some fields only: a projection parameter with a value other than `*`, which asks for all. */
export const selectsSomeFields = (query: string | undefined): boolean =>
query !== undefined &&
[...new URLSearchParams(query)].some(
([name, value]) => isProjectionParameter(name) && value !== "*",
);
export type StateListView = EntityViewPlan &
ReplayedReads & {
readonly shape: "list";
readonly membership: "all" | "recorded";
/** With recorded membership, the checked lines that each list their whole collection. */
readonly all_record_ordinals?: number[] | undefined;
readonly counters: string[];
/** The query the view's other lines follow, when one explains any of them. */
readonly query?: ListQuery | undefined;
};
/** Whether a list view answers on a checked line with every entity of the collection. */
export const listsAll = (view: StateListView, ordinal: number): boolean =>
@@ -177,7 +201,7 @@ export interface ExtractedState {
const WRITE_METHODS = new Set(["POST", "PUT", "PATCH", "DELETE"]);
/** A POST whose path names one of these segments queries state rather than changing it. */
const QUERY_SEGMENTS = new Set([
export const QUERY_SEGMENTS = new Set([
"search",
"query",
"list",
src/domain/analysisView/environmentStateVariants.ts
+535
View File
@@ -0,0 +1,535 @@
import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js";
import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js";
import type { LateLine, StatePlan } from "./environmentStateGate.js";
import type { ObservedEntity } from "./environmentStateMerge.js";
import {
isProjectionParameter,
projectsFields,
type ReplayKey,
type StateItemView,
type StateListView,
type StateRecordFacts,
} from "./environmentStateRules.js";
/** Query names that page a list rather than choose how it shows its entities. */
const PAGING = new Set([
"limit",
"offset",
"page",
"per_page",
"perpage",
"page_size",
"pagesize",
"size",
"start",
"_start",
"_end",
"cursor",
"pagenumber",
"pageindex",
"skip",
"take",
"top",
"$top",
"$skip",
]);
/** What one recorded line's objects showed: each entity's member names, and the members any and every object carried. */
export interface LineShape {
readonly entities: ReadonlyMap<string, string>;
readonly carried: ReadonlySet<string>;
readonly common: ReadonlySet<string> | undefined;
}
/** Each name a query names, with every value it took. */
export const queryValues = (
query: string | null | undefined,
): Map<string, string[]> => {
const values = new Map<string, string[]>();
for (const [name, value] of new URLSearchParams(query ?? ""))
values.set(name, [...(values.get(name) ?? []), value]);
return values;
};
/** Whether a query names a parameter that may change how a line shows its entities, paging and field selections aside. */
export const shapesOtherwise = (query: string | null | undefined): boolean =>
shapingNames(query).some((name) => !isProjectionParameter(name));
/** The query names that may change how a line shows its entities: all but paging. */
const shapingNames = (query: string | null | undefined): string[] =>
[...queryValues(query).keys()].filter(
(name) => !PAGING.has(name.toLowerCase()),
);
/** The shape each observed recording showed of its entities, by ordinal, each part counted against the budget before it is kept. */
export const lineShapes = (
entities: Iterable<ObservedEntity>,
retain: (bytes: number) => void,
): Map<number, LineShape> => {
const shapes = new Map<
number,
{
entities: Map<string, string>;
carried: Set<string>;
common: Set<string> | undefined;
}
>();
retain(ENTRY_BYTES);
for (const entity of entities)
for (const observation of entity.observations) {
const members = Object.keys(observation.object);
let shape = shapes.get(observation.ordinal);
if (shape === undefined) {
// The shape's own entry and containers, with room for the members every object carries.
retain(ENTRY_BYTES * (4 + members.length));
shape = { entities: new Map(), carried: new Set(), common: undefined };
shapes.set(observation.ordinal, shape);
}
const key = JSON.stringify([entity.type, entity.collection, entity.id]);
// A JSON array, since a member name may hold any separator.
const signature = JSON.stringify([...members].sort(compare));
if (!shape.entities.has(key))
retain(ENTRY_BYTES + textBytes(key.length + signature.length));
shape.entities.set(key, signature);
for (const member of members)
if (!shape.carried.has(member)) {
retain(ENTRY_BYTES);
shape.carried.add(member);
}
shape.common =
shape.common === undefined
? new Set(members)
: new Set(members.filter((member) => shape!.common!.has(member)));
}
return shapes;
};
const EMPTY: LineShape = {
entities: new Map(),
carried: new Set(),
common: undefined,
};
/** Line pairs and entity lookups the variant pass may make; a plan whose lines need more keeps every line checked. */
const VARIANT_WORK_LIMIT = 1 << 24;
/** For each entity some lines show, those lines by the members they show it with. */
type LinesByEntity = ReadonlyMap<
string,
ReadonlyMap<string, readonly StateRecordFacts[]>
>;
/** The lines by the entities they show, an entry for each entity each line shows charged before it is kept. */
const linesByEntity = (
lines: readonly StateRecordFacts[],
shape: (record: StateRecordFacts) => LineShape,
retain: (bytes: number) => void,
): LinesByEntity => {
const shown = new Map<string, Map<string, StateRecordFacts[]>>();
for (const line of lines)
for (const [key, members] of shape(line).entities) {
retain(ENTRY_BYTES);
const showing = shown.get(key) ?? new Map<string, StateRecordFacts[]>();
shown.set(key, showing);
const group = showing.get(members);
if (group === undefined) showing.set(members, [line]);
else group.push(line);
}
return shown;
};
/** The lines that show one of a line's entities with other members. */
const conflictingLines = (
mine: LineShape,
shown: LinesByEntity,
): Set<StateRecordFacts> => {
const conflicting = new Set<StateRecordFacts>();
for (const [key, members] of mine.entities)
for (const [others, lines] of shown.get(key)!)
if (others !== members) for (const line of lines) conflicting.add(line);
return conflicting;
};
/** What lines compared later are compared with: the shaping parameter names they named, the members their objects carried, and those every one carried. */
interface LineTotals {
readonly names: Set<string>;
readonly carried: Set<string>;
common: Set<string> | undefined;
}
/** Totals over no lines yet. */
const noLines = (): LineTotals => ({
names: new Set(),
carried: new Set(),
common: undefined,
});
/** Count one more line into the totals. */
const addLine = (
totals: LineTotals,
query: string | null | undefined,
shape: Pick<LineShape, "carried" | "common">,
): void => {
for (const name of shapingNames(query)) totals.names.add(name);
for (const member of shape.carried) totals.carried.add(member);
const { common } = shape;
if (common !== undefined)
totals.common =
totals.common === undefined
? new Set(common)
: new Set([...totals.common].filter((member) => common.has(member)));
};
/** Whether a query names a shaping parameter, paging aside, that none of the counted lines named. */
const namesNewParameter = (
query: string | null | undefined,
totals: LineTotals,
): boolean => shapingNames(query).some((name) => !totals.names.has(name));
/** Whether a line's objects carry a member the counted lines' never did, or lack one theirs always carried. */
const showsOtherMembers = (
shape: Pick<LineShape, "carried" | "common">,
totals: LineTotals,
): boolean => {
const { common } = shape;
return (
[...shape.carried].some((member) => !totals.carried.has(member)) ||
(common !== undefined &&
totals.common !== undefined &&
[...totals.common].some((member) => !common.has(member)))
);
};
/** The base lines so far, with their totals. */
interface BaseLines extends LineTotals {
readonly lines: Set<StateRecordFacts>;
}
/**
* The latest lines that show the plan's entities in another representation than its base lines, as
* an expanding or reducing query does; none when comparing every pair of lines would exceed the work limit.
*/
export const variantLines = (
latest: readonly StateRecordFacts[],
shapeOf: (record: StateRecordFacts) => LineShape | undefined,
retain: (bytes: number) => void = () => undefined,
): StateRecordFacts[] => {
if (latest.length < 2) return [];
const shapes = new Map(
latest.map((record) => [record, shapeOf(record) ?? EMPTY]),
);
const entities = [...shapes.values()].reduce(
(sum, line) => sum + line.entities.size,
0,
);
// Comparing every pair of lines entity by entity bounds comparing only the lines that share an entity.
if (latest.length * (latest.length + entities) > VARIANT_WORK_LIMIT)
return [];
const shape = (record: StateRecordFacts): LineShape => shapes.get(record)!;
const shown = linesByEntity(latest, shape, retain);
const ranks = new Map(
latest.map((record) => {
const names = shapingNames(record.query).length;
const agreeing =
latest.length - 1 - conflictingLines(shape(record), shown).size;
return [
record,
[
names === 0 ? 0 : 1,
projectsFields(record.query ?? undefined) ? 1 : 0,
-agreeing,
names,
shape(record).carried.size,
],
];
}),
);
// Base lines first: no shaping query, then no projection, then the line most others agree with.
const ordered = [...latest].sort((left, right) => {
const [a, b] = [ranks.get(left)!, ranks.get(right)!];
const differs = a.findIndex((value, index) => value !== b[index]);
return differs === -1 ? 0 : a[differs]! - b[differs]!;
});
const base: BaseLines = { ...noLines(), lines: new Set() };
const variants: StateRecordFacts[] = [];
for (const record of ordered) {
if (base.lines.size > 0 && isVariant(record, base, shape(record), shown))
variants.push(record);
else {
base.lines.add(record);
addLine(base, record.query, shape(record));
}
}
return variants;
};
/** Whether a line shows a base line's entity with other members, or names a new shaping parameter while its objects carry or lack members the base lines do not. */
const isVariant = (
record: StateRecordFacts,
base: BaseLines,
mine: LineShape,
shown: LinesByEntity,
): boolean =>
[...conflictingLines(mine, shown)].some((line) => base.lines.has(line)) ||
(namesNewParameter(record.query, base) && showsOtherMembers(mine, base));
/** What replay keys are found against: every value each base line gave a parameter, and the parameters every base line names. */
interface KeyBase {
readonly values: ReadonlyMap<string, ReadonlySet<string>>;
readonly everywhere: readonly string[];
}
/** Parse the base lines' queries once, for any number of replayed lines. */
const keyBase = (base: readonly (string | null | undefined)[]): KeyBase => {
const baseValues = base.map(queryValues);
const values = new Map<string, Set<string>>();
for (const line of baseValues)
for (const [name, given] of line) {
const seen = values.get(name) ?? new Set<string>();
values.set(name, seen);
for (const value of given) seen.add(value);
}
const everywhere = [...values.keys()].filter(
(name) =>
!PAGING.has(name.toLowerCase()) &&
baseValues.every((line) => line.has(name)),
);
return { values, everywhere };
};
/** The keys that tell replayed lines from base lines: a parameter or value only replayed lines give, or a parameter every base line names that one leaves out. */
export const replayKeys = (
base: readonly (string | null | undefined)[],
replayed: readonly (string | null | undefined)[],
retain: (bytes: number) => void = () => undefined,
): ReplayKey[] => keysAgainst(keyBase(base), replayed, retain);
/** The replay keys of replayed lines against base lines parsed once. */
const keysAgainst = (
{ values: baseValues, everywhere }: KeyBase,
replayed: readonly (string | null | undefined)[],
retain: (bytes: number) => void,
): ReplayKey[] => {
const keys = new Map<string, ReplayKey>();
const add = (key: ReplayKey): void => {
const id = JSON.stringify([
key.name,
key.value ?? null,
key.absent ?? false,
]);
if (keys.has(id)) return;
// Each key is kept with its text, so it is charged before it is.
retain(ENTRY_BYTES * 2 + textBytes(id.length));
keys.set(id, key);
};
for (const query of replayed) {
const values = queryValues(query);
for (const [name, given] of values) {
if (PAGING.has(name.toLowerCase())) continue;
const seen = baseValues.get(name);
const unseen = given.filter((value) => seen?.has(value) !== true);
if (seen === undefined) add({ name });
else for (const value of unseen) add({ name, value });
}
for (const name of everywhere)
if (!values.has(name)) add({ name, absent: true });
}
return [...keys.values()].sort(
(left, right) =>
compare(left.name, right.name) ||
compare(left.value ?? "", right.value ?? "") ||
Number(left.absent ?? false) - Number(right.absent ?? false),
);
};
/** Whether a call's query names a replay key: the parameter, the parameter with that value, or leaves out an absent key's parameter. */
const matchesReplayKey = (
keys: readonly ReplayKey[],
call: ReadonlyMap<string, readonly string[]>,
): boolean =>
keys.some((key) =>
key.absent === true
? !call.has(key.name)
: key.value === undefined
? call.has(key.name)
: call.get(key.name)?.includes(key.value) === true,
);
/** Whether a call asks for a selection of fields the checked lines do not show exactly: a projection value a replayed line gave, or one no checked line gave other than `*`. */
const selectsOtherFields = (
call: ReadonlyMap<string, readonly string[]>,
checked: readonly (string | null | undefined)[],
replayed: readonly (string | null | undefined)[],
): boolean => {
const [base, other] = [checked.map(queryValues), replayed.map(queryValues)];
const gave = (lines: Map<string, string[]>[], name: string, value: string) =>
lines.some((values) => values.get(name)?.includes(value) === true);
return [...call].some(
([name, values]) =>
isProjectionParameter(name) &&
values.some(
(value) =>
gave(other, name, value) ||
(value !== "*" && !gave(base, name, value)),
),
);
};
/** Why serving replays a read call rather than answering it from the state (`projection` for some fields only, `variant` for another representation); the gate skips the same reads. */
export const replayReason = (
view: Pick<
StateItemView | StateListView,
| "checked_record_ordinals"
| "projected_record_ordinals"
| "projected_query_keys"
>,
call: ReadonlyMap<string, readonly string[]>,
queryOf: (ordinal: number) => string | null | undefined,
): "projection" | "variant" | undefined => {
if (view.projected_record_ordinals === undefined) return undefined;
// A field selection is tested on its own: a variant key cannot name every value the checked lines do not show.
if (
[...call.keys()].some(isProjectionParameter) &&
selectsOtherFields(
call,
view.checked_record_ordinals.map(queryOf),
view.projected_record_ordinals.map(queryOf),
)
)
return "projection";
const matched = (view.projected_query_keys ?? []).filter(
(key) =>
(key.absent === true || !isProjectionParameter(key.name)) &&
matchesReplayKey([key], call),
);
if (matched.length === 0) return undefined;
return matched.some((key) => isProjectionParameter(key.name))
? "projection"
: "variant";
};
const requestLine = (record: StateRecordFacts): string =>
record.path + (record.query ?? "");
/** The found lines a replay key tells from the other latest lines, which are parsed once for all of them. */
const keyedVariants = (
latest: readonly StateRecordFacts[],
found: readonly StateRecordFacts[],
): StateRecordFacts[] => {
if (found.length === 0) return [];
const variant = new Set(found);
const base = keyBase(
latest.filter((record) => !variant.has(record)).map((line) => line.query),
);
return found.filter(
(record) => keysAgainst(base, [record.query], () => undefined).length > 0,
);
};
/** The plan with the variants a key picks out moved to its replayed reads and their recordings dropped, or undefined when it replays nothing. */
export const withoutVariants = (
plan: StatePlan,
facts: {
readonly shapeOf: (record: StateRecordFacts) => LineShape | undefined;
readonly queryOf: (ordinal: number) => string | null | undefined;
readonly position: (ordinal: number) => number;
readonly retain: (bytes: number) => void;
readonly checkpoint: () => () => void;
},
):
| { readonly plan: StatePlan; readonly dropped: ReadonlySet<number> }
| undefined => {
// The lines indexed by entity are dropped once compared, so what they were charged is released.
const release = facts.checkpoint();
const found = variantLines(plan.latest, facts.shapeOf, facts.retain);
release();
const variants = keyedVariants(plan.latest, found);
const lines = new Set(variants.map(requestLine));
const latest = plan.latest.filter(
(record) => !lines.has(requestLine(record)),
);
const projected = [
...(plan.projected ?? []),
...lateReplays(plan.late ?? [], latest, facts.shapeOf),
];
if (variants.length === 0 && projected.length === 0) return undefined;
const replayed = [
...projected,
...variants.map((record) => record.ordinal),
].sort((left, right) => facts.position(left) - facts.position(right));
const dropped = new Set(
plan.recordings
.filter((record) => lines.has(requestLine(record)))
.map((record) => record.ordinal),
);
return {
plan: {
...plan,
recordings: plan.recordings.filter(
(record) => !dropped.has(record.ordinal),
),
latest,
projected: replayed,
projectedKeys: replayKeys(
latest.map((record) => record.query),
replayed.map(facts.queryOf),
facts.retain,
),
},
dropped,
};
};
/**
* The later item lines serving replays, decided against the lines left: a selection of some fields
* that leaves out members they always show, or a line naming a shaping parameter none of them names
* while its object carries or lacks members theirs do not.
*/
const lateReplays = (
late: readonly LateLine[],
base: readonly StateRecordFacts[],
shapeOf: (record: StateRecordFacts) => LineShape | undefined,
): number[] => {
if (late.length === 0) return [];
const totals = noLines();
for (const record of base)
addLine(totals, record.query, shapeOf(record) ?? EMPTY);
return late.flatMap(({ record, members, projecting }) =>
(projecting ||
shapingNames(record.query).some(
(name) => !isProjectionParameter(name) && !totals.names.has(name),
)) &&
showsOtherMembers({ carried: members, common: members }, totals)
? [record.ordinal]
: [],
);
};
/** Forget the observations of dropped recordings, and entities only they showed; an entity is first sighted where its kept observations show it first. */
export const dropObservations = (
entities: Map<string, ObservedEntity>,
dropped: ReadonlySet<number>,
): void => {
for (const [key, entity] of entities) {
const kept = entity.observations.filter(
(observation) => !dropped.has(observation.ordinal),
);
if (kept.length === entity.observations.length) continue;
if (kept.length === 0) {
entities.delete(key);
continue;
}
const first = kept.reduce((earliest, observation) =>
observation.position < earliest.position ||
(observation.position === earliest.position &&
observation.index < earliest.index)
? observation
: earliest,
);
entities.set(key, {
...entity,
order: [first.position, first.index],
observations: kept,
});
}
};
src/domain/analysisView/environmentStateView.test.ts
+431 -37
View File
@@ -20,14 +20,25 @@ import {
type AnalysisViewResult,
} from "./analysisView.js";
import { EnvironmentStateAccumulator } from "./EnvironmentStateAccumulator.js";
import type { ObservedEntity } from "./environmentStateMerge.js";
import {
at,
identityCandidates,
replaceAt,
renderStateList,
type JsonObject,
type StateRecordFacts,
} from "./environmentStateRules.js";
import { environmentStateFields } from "./environmentStateView.js";
import {
lineShapes,
replayKeys,
variantLines,
withoutVariants,
type LineShape,
} from "./environmentStateVariants.js";
import { rewriteCuts } from "./environmentStateRewrites.js";
import { ENTRY_BYTES } from "./webNetworkRouteTemplates.js";
type State = Extract<AnalysisViewResult, { kind: "environment-state" }>;
const ALL_RULES = [
@@ -210,7 +221,7 @@ it("keeps the latest value of a field that changed with no related write and nam
);
});
it("leaves a type recorded when a value changed across a write under another root", () => {
it("ends a type's state at a write under another root its value changed across", () => {
const state = stateOf(
captureOf([
["GET", "/api/search", [{ id: 1, title: "Granite Ledger" }]],
@@ -218,16 +229,16 @@ it("leaves a type recorded when a value changed across a write under another roo
["GET", "/api/search", [{ id: 1, title: "Saltmarsh Depot" }]],
]),
);
expect(state.types).toEqual([]);
expect(state.entities).toEqual([]);
expect(viewOf(state, "get_search")).toEqual({
tool: "get_search",
shape: "recorded",
reason: "fields-differ",
});
expect(state.types).toMatchObject([
{ collection_template: "/api/search", init_before_ordinal: 1 },
]);
expect(state.entities).toMatchObject([
{ id: "1", fields: { title: "Granite Ledger" } },
]);
expect(viewOf(state, "get_search")).toMatchObject({ shape: "list" });
});
it("leaves a type recorded when a value changed across a write already in flight", () => {
it("leaves out of a type's state a read made while the write its value changed across was in flight", () => {
const exchange = ([method, path, seconds, time, title]: readonly [
string,
string,
@@ -268,7 +279,7 @@ it("leaves a type recorded when a value changed across a write already in flight
expect(viewOf(state, "get_search")).toEqual({
tool: "get_search",
shape: "recorded",
reason: "fields-differ",
reason: "no-init-recording",
});
});
@@ -789,6 +800,391 @@ it("refuses when extraction would exceed the projection budget", () => {
});
});
it("counts the line shapes it builds against the budget as it builds them", () => {
const members = Object.fromEntries(
Array.from({ length: 40 }, (_, index) => [
`member_with_a_long_name_${String(index)}`,
index,
]),
);
const entities: ObservedEntity[] = Array.from({ length: 50 }, (_, id) => ({
type: "/api/items",
collection: "/api/items",
id: String(id),
order: [0, id],
observations: [1, 2].map((ordinal) => ({
position: ordinal,
index: id,
ordinal,
tool: "get_items",
item: false,
object: members,
})),
}));
let retained = 0;
lineShapes(entities, (bytes) => {
retained += bytes;
});
// Each of the hundred observations keeps its entity's key and the names of its members.
expect(retained).toBeGreaterThan(100 * Object.keys(members).join(",").length);
});
it("leaves every line checked rather than compare more lines than the variant pass allows", () => {
const lines = Array.from({ length: 3000 }, (_, ordinal) => ({
ordinal,
method: "GET",
path: "/api/items",
query: `?view=${String(ordinal)}`,
status: 200,
start: ordinal,
end: ordinal,
}));
const shape = {
entities: new Map(
Array.from({ length: 4 }, (_, id) => [String(id), "id,name"]),
),
carried: new Set(["id", "name"]),
common: new Set(["id", "name"]),
};
let looked = 0;
expect(
variantLines(lines, () => {
looked += 1;
return shape;
}),
).toEqual([]);
// Comparing every pair would look each line's shape up thousands of times.
expect(looked).toBeLessThan(10 * lines.length);
});
it("tells apart member names that hold the separator, so a line spelling them otherwise is a variant", () => {
const line = (ordinal: number, query: string) => ({
ordinal,
method: "GET",
path: "/api/items/1",
query,
status: 200,
start: ordinal,
end: ordinal,
});
const observed = (ordinal: number, object: JsonObject) => ({
position: ordinal,
index: 0,
ordinal,
tool: "get_items_by_id",
item: true,
object,
});
const shapes = lineShapes(
[
{
type: "/api/items",
collection: "/api/items",
id: "1",
order: [1, 0],
observations: [
observed(1, { id: 1, a: 1, b: 2 }),
observed(2, { id: 1, "a,b": 3 }),
],
},
],
() => undefined,
);
expect(
variantLines([line(1, "?view=x"), line(2, "?view=y")], (record) =>
shapes.get(record.ordinal),
),
).toHaveLength(1);
});
it("leaves every line checked when comparing many mostly empty lines would exceed the variant pass's work", () => {
let reads = 0;
const lines = Array.from({ length: 4200 }, (_, ordinal) => {
const record = {
ordinal,
method: "GET",
path: "/api/items",
status: 200,
start: ordinal,
end: ordinal,
};
return Object.defineProperty(record, "query", {
enumerable: true,
get: () => {
reads += 1;
return `?page=${String(ordinal)}`;
},
}) as typeof record & { readonly query: string };
});
const empty = {
entities: new Map(),
carried: new Set<string>(),
common: undefined,
};
const shown = {
entities: new Map([
["1", "id"],
["2", "id"],
]),
carried: new Set(["id"]),
common: new Set(["id"]),
};
expect(
variantLines(lines, (record) => (record.ordinal === 0 ? shown : empty)),
).toEqual([]);
// Comparing every pair would read each line's query thousands of times.
expect(reads).toBeLessThan(10 * lines.length);
});
/** Item lines reading each of `items` items plainly, then each again with `?expand=shelf`; each shape's entities count the walks over them. */
const expandedItemLines = (items: number) => {
let walks = 0;
const lines: StateRecordFacts[] = Array.from(
{ length: items * 2 },
(_, ordinal) => ({
ordinal,
method: "GET",
path: `/api/items/${String(ordinal % items)}`,
query: ordinal < items ? undefined : "?expand=shelf",
status: 200,
start: ordinal,
end: ordinal,
}),
);
const shapes = new Map(
lines.map((line): [StateRecordFacts, LineShape] => {
const members =
line.query === undefined ? ["id", "name"] : ["id", "name", "shelf"];
const entities = new Map([
[
JSON.stringify([
"/api/items",
"/api/items",
String(line.ordinal % items),
]),
JSON.stringify(members),
],
]);
const walk = entities[Symbol.iterator].bind(entities);
entities[Symbol.iterator] = () => {
walks += 1;
return walk();
};
const carried = new Set(members);
return [line, { entities, carried, common: carried }];
}),
);
return {
lines,
shapeOf: (record: StateRecordFacts) => shapes.get(record),
walks: () => walks,
};
};
it("compares lines through the entities they show, so items read plainly and expanded are not compared pair by pair", () => {
const { lines, shapeOf, walks } = expandedItemLines(1000);
expect(variantLines(lines, shapeOf)).toEqual(lines.slice(1000));
// Comparing every pair would walk each line's entities thousands of times.
expect(walks()).toBeLessThan(10 * lines.length);
});
it("reads each base line's query a fixed number of times however many variants it keys", () => {
const items = 200;
const { lines, shapeOf } = expandedItemLines(items);
let reads = 0;
for (const line of lines.slice(0, items)) {
const { query } = line;
Object.defineProperty(line, "query", {
enumerable: true,
get: () => ((reads += 1), query),
});
}
const tool = {
name: "get_items_by_id",
method: "GET",
path_template: "/api/items/{id}",
record_ordinals: lines.map((line) => line.ordinal),
};
const plan = {
tool,
item: true,
type: "/api/items",
pointer: "",
recordings: lines,
latest: lines,
};
const split = withoutVariants(plan, {
shapeOf,
queryOf: (ordinal) => lines[ordinal]?.query,
position: (ordinal) => ordinal,
retain: () => undefined,
checkpoint: () => () => undefined,
});
expect(split?.plan.projected).toHaveLength(items);
// Keying each variant against the base lines anew reads every base line's query once per variant.
expect(reads).toBeLessThan(10 * items);
});
it("counts what it holds to compare lines by the entities they show against the budget", () => {
const { lines, shapeOf } = expandedItemLines(100);
let retained = 0;
variantLines(lines, shapeOf, (bytes) => {
retained += bytes;
});
// An entry for each entity each line shows.
expect(retained).toBeGreaterThanOrEqual(lines.length * ENTRY_BYTES);
});
/** The smallest allowance, to a KiB, a capture's state projects within. */
const allowance = (parent: AnalysisViewParent): number => {
const fits = (bytes: number): boolean =>
projectAnalysisView(
parent,
{ kind: "environment-state", template_rules: [...ALL_RULES] },
{ state: new EnvironmentStateAccumulator(bytes, [...ALL_RULES]) },
).ok;
let [low, high] = [0, 8 * 1024];
while (high - low > 1) {
const middle = Math.floor((low + high) / 2);
if (fits(middle * 1024)) high = middle;
else low = middle;
}
return high * 1024;
};
it("counts the member names of later item lines against the budget before keeping them", () => {
const reads = 20;
// Each later read of an item names members its initial reads never did, `width` characters long.
const capture = (width: number) =>
captureOf([
["GET", "/api/items/1", { id: 1, name: "One" }],
["POST", "/api/items", { id: 2, name: "Two" }, 201, { name: "Two" }],
...Array.from({ length: reads }, (_, read): TimedCaptureRow => [
"GET",
`/api/items/1?expand=meta&view=${String(read)}`,
{
id: 1,
name: "One",
...Object.fromEntries(
Array.from({ length: 5 }, (_, member) => [
`${String(read)}_${String(member)}_`.padEnd(width, "m"),
1,
]),
),
},
]),
]);
expect(
allowance(capture(400)) - allowance(capture(10)),
).toBeGreaterThanOrEqual((reads * 5 * 390 * 14) / 2);
});
it("counts each change of a line's membership across a write against the budget", () => {
const tags = 64;
// Each tag's list is read three times across two writes; the third read shows the first book again or keeps the second.
const capture = (third: number) =>
captureOf([
...Array.from({ length: tags }, (_, tag): TimedCaptureRow => [
"GET",
`/api/books?tag=${String(tag)}`,
[{ id: 1 }],
]),
["POST", "/api/view-log", { ok: true }, 200, { tag: 0 }],
...Array.from({ length: tags }, (_, tag): TimedCaptureRow => [
"GET",
`/api/books?tag=${String(tag)}`,
[{ id: 2 }],
]),
["POST", "/api/view-log", { ok: true }, 200, { tag: 1 }],
...Array.from({ length: tags }, (_, tag): TimedCaptureRow => [
"GET",
`/api/books?tag=${String(tag)}`,
[{ id: third }],
]),
]);
// Both captures keep the same entities and observations; only the second change of each line differs.
expect(allowance(capture(1)) - allowance(capture(2))).toBeGreaterThanOrEqual(
tags * 256,
);
});
it("counts each replay key it keeps against the budget", () => {
let retained = 0;
const names = Array.from(
{ length: 500 },
(_, index) => `view_${String(index)}`,
);
const keys = replayKeys(
["?view=plain"],
[`?${names.map((name) => `${name}=1`).join("&")}`],
(bytes) => {
retained += bytes;
},
);
expect(keys.length).toBeGreaterThan(names.length);
expect(retained).toBeGreaterThanOrEqual(keys.length * 128);
});
it("finds each span's earliest write without scanning the writes before it", () => {
let reads = 0;
const timed = (ordinal: number, start: number, end: number) => {
const record = {
ordinal,
method: "POST",
path: "/api/log",
query: undefined,
status: 200,
};
return Object.defineProperties(record, {
start: { enumerable: true, get: () => ((reads += 1), start) },
end: { enumerable: true, get: () => ((reads += 1), end) },
}) as typeof record & { readonly start: number; readonly end: number };
};
const count = 2000;
const writes = Array.from({ length: count }, (_, index) =>
timed(index, index * 10, index * 10 + 1),
);
const reads_ = new Map(
Array.from({ length: count }, (_, index) => {
const ordinal = count + index;
return [
ordinal,
timed(ordinal, index * 10 + 5, index * 10 + 12),
] as const;
}),
);
const spans = [...reads_.keys()].map((earlier, index) => ({
type: `/api/type_${String(index)}`,
earlier,
later: earlier,
}));
reads = 0;
const cuts = rewriteCuts(spans, writes, reads_);
expect(cuts.size).toBe(count - 1);
expect(reads).toBeLessThan(200 * count);
});
it("releases a planning pass a rewrite cut discards before extracting the state again", () => {
const recipe = (id: number, views: number) => ({
id,
name: `Recipe ${String(id)} simmered slowly with a long descriptive title`,
views,
});
const shelf = Array.from({ length: 400 }, (_, index) => recipe(index + 1, 1));
// Logging a view bumps the counter only when `views` changes, which cuts the type's reads at the log.
const capture = (views: number) =>
captureOf([
["GET", "/api/recipes", shelf],
["GET", "/api/recipes/1", recipe(1, 1)],
["POST", "/api/view-log", { ok: true }, 200, { recipe: 1 }],
["GET", "/api/recipes/1", recipe(1, views)],
]);
expect(stateOf(capture(2)).types).toMatchObject([
{ collection_template: "/api/recipes", init_before_ordinal: 2 },
]);
expect(allowance(capture(2))).toBeLessThan(allowance(capture(1)) + 16 * 1024);
});
it("ends every type's state at a write whose URL was redacted, and says so", () => {
const parent = captureOf([
["GET", "/api/items/1", { id: 1, title: "Granite Ledger" }],
@@ -872,7 +1268,7 @@ it("does not take a later recording that is itself an entity as the list", () =>
expect(state.entities).toEqual([]);
});
it("leaves a type recorded when two of its tools read different values across a write", () => {
it("ends a type's state at a write two of its tools read different values across", () => {
const state = stateOf(
captureOf([
["GET", "/api/search", [{ id: 1, title: "Granite Ledger" }]],
@@ -880,16 +1276,17 @@ it("leaves a type recorded when two of its tools read different values across a
["GET", "/api/search/1", { id: 1, title: "Saltmarsh Depot" }],
]),
);
expect(state.types).toEqual([]);
expect(state.entities).toEqual([]);
expect(state.types).toMatchObject([
{ collection_template: "/api/search", init_before_ordinal: 1 },
]);
expect(state.entities).toMatchObject([
{ id: "1", fields: { title: "Granite Ledger" } },
]);
expect(
state.views.map((view) =>
view.shape === "recorded" ? [view.tool, view.reason] : view.tool,
),
).toEqual([
["get_search", "fields-differ"],
["get_search_by_id", "fields-differ"],
]);
).toEqual(["get_search", ["get_search_by_id", "no-init-recording"]]);
});
it.each(["old", "new"])(
@@ -904,14 +1301,9 @@ it.each(["old", "new"])(
]),
);
if (title === "new") {
expect(state.types).toEqual([]);
expect(state.entities).toEqual([]);
expect(
state.views.every(
(view) =>
view.shape === "recorded" && view.reason === "fields-differ",
),
).toBe(true);
expect(state.types).toMatchObject([{ init_before_ordinal: 1 }]);
expect(state.entities[0]?.fields.title).toBe("old");
expect(viewOf(state, "get_search")).toMatchObject({ shape: "list" });
} else {
expect(viewOf(state, "get_search_by_id")).toMatchObject({
shape: "item",
@@ -1071,7 +1463,7 @@ it.each([
[[{ id: 1 }], []],
[[{ id: 1 }], [{ id: 2 }]],
] as const)(
"keeps changed list membership across another root's write recorded (%j -> %j)",
"ends a list's state at another root's write its membership changed across (%j -> %j)",
(before, after) => {
const state = stateOf(
captureOf([
@@ -1080,12 +1472,15 @@ it.each([
["GET", "/api/search", [...after]],
]),
);
expect(state.types).toEqual([]);
expect(state.entities).toEqual([]);
expect(viewOf(state, "get_search")).toEqual({
tool: "get_search",
shape: "recorded",
reason: "fields-differ",
// An empty initial list proves no identity, so it is held as a document.
expect(state.types.map((type) => type.init_before_ordinal)).toEqual(
before.length > 0 ? [1] : [],
);
expect(state.entities.map((entity) => entity.id)).toEqual(
before.map((item) => String(item.id)),
);
expect(viewOf(state, "get_search")).toMatchObject({
shape: before.length > 0 ? "list" : "document",
});
},
);
@@ -1207,7 +1602,7 @@ it.each([
[{ id: 1 }, { id: 1, title: "Created" }],
[{ id: 1, title: "Original" }, { id: 1 }],
])(
"keeps fields added or removed across another root's write recorded (%j -> %j)",
"ends a type's state at another root's write it gained or lost fields across (%j -> %j)",
(before, after) => {
const state = stateOf(
captureOf([
@@ -1216,11 +1611,10 @@ it.each([
["GET", "/api/search/1", after],
]),
);
expect(state.types).toEqual([]);
expect(state.entities).toEqual([]);
expect(state.types).toMatchObject([{ init_before_ordinal: 1 }]);
expect(state.entities).toMatchObject([{ fields: before }]);
expect(viewOf(state, "get_search_by_id")).toMatchObject({
shape: "recorded",
reason: "fields-differ",
shape: "item",
});
},
);
src/domain/analysisView/environmentStateView.ts
+22 -6
View File
@@ -120,6 +120,26 @@ const entityViewFields = {
"How the view spells clock fields it always spells otherwise than the entities' stored values; a stored value the view never recorded is spelled this way.",
),
};
const replayedReadFields = {
projected_record_ordinals: z
.array(z.number().int().nonnegative())
.optional()
.describe(
"Recorded reads left to replay: they asked for some fields only (`fields`, `select`), or show the entities in another representation than the checked lines (an augmenting or reducing query). Replayed calls do not show served writes.",
),
projected_query_keys: z
.array(
z.strictObject({
name: z.string(),
value: z.string().optional(),
absent: z.literal(true).optional(),
}),
)
.optional()
.describe(
"Query keys that pick calls of the replayed reads out: a parameter only they name, a value only they give a parameter, or (absent) a parameter every checked line names and they leave out. A call matching any key replays; a projection parameter's value is tested on its own: a call replays unless every value it gives is `*` or one a checked line gave, and no replayed read gave it.",
),
};
const viewSchema = z.discriminatedUnion("shape", [
z.strictObject({
...entityViewFields,
@@ -131,12 +151,7 @@ const viewSchema = z.discriminatedUnion("shape", [
.describe(
"Record whose response, with the entity object replaced, every item response shares.",
),
projected_record_ordinals: z
.array(z.number().int().nonnegative())
.optional()
.describe(
"Recorded reads that asked for some fields only (`fields`, `select`) and did not return the item; calls naming a projection replay.",
),
...replayedReadFields,
}),
z.strictObject({
...entityViewFields,
@@ -160,6 +175,7 @@ const viewSchema = z.discriminatedUnion("shape", [
.describe(
"How the view's other checked lines select and order entities, when filters, a search, a sort or paging explain them; absent when none does.",
),
...replayedReadFields,
}),
z.strictObject({
tool: z.string(),
src/domain/analysisView/environmentStateWriteAdoption.ts
+112
View File
@@ -0,0 +1,112 @@
import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js";
import {
at,
isObject,
stateValueRetentionBytes,
scalarText,
type StateEntity,
type StateRecordFacts,
type StateView,
} from "./environmentStateRules.js";
import {
collectionTarget,
entityTarget,
itemTarget,
succeeded,
type SimulatedState,
} from "./environmentStateWriteChecks.js";
import type { WriteIo } from "./environmentStateWriteEffects.js";
/**
* Entities a write no built plan models made, such as rows an import or a nested create added: the
* copy takes one when a successful item read first shows it, rather than failing the read. A call
* of a built plan that serving would not apply keeps its evidence.
*/
export class UnseenEntities {
/** Targets of the entities the copy has held: the initial ones and any a modelled write or an adoption touched. */
readonly #held: Set<string>;
/** Timeline indexes of writes no built plan models, and of redirected or unanswered ones. */
readonly #unplannedAt: number[] = [];
constructor(
initial: readonly StateEntity[],
private readonly facts: {
readonly state: SimulatedState;
readonly io: Pick<WriteIo, "withBody" | "retain">;
readonly lastCheck: ReadonlyMap<string, number>;
readonly idField: (type: string) => string | undefined;
},
) {
this.#held = new Set(
initial.map((entity) =>
entityTarget(entity.type, entity.collection, entity.id),
),
);
}
/** Note an entity a modelled write created, changed or removed, which no read takes back. */
touched(entity: Pick<StateEntity, "type" | "collection" | "id">): void {
const key = entityTarget(entity.type, entity.collection, entity.id);
if (this.#held.has(key)) return;
this.facts.io.retain(ENTRY_BYTES + textBytes(key.length));
this.#held.add(key);
}
/** Note a write no built plan models, or one redirected or unanswered. */
unplanned(index: number): void {
this.facts.io.retain(ENTRY_BYTES);
this.#unplannedAt.push(index);
}
/**
* Put into the copy the entity a successful item read shows that the copy never held, when an
* unplanned write ran since the entity or its collection was last checked; the entity taken, if any.
*/
adopt(
view: Extract<StateView, { shape: "item" }>,
read: { readonly record: StateRecordFacts; readonly index: number },
): StateEntity | undefined {
const { record, index } = read;
const { state, lastCheck } = this.facts;
// Only an unplanned write before the read can have made what it shows, so its target is found only then.
if (!succeeded(record) || !this.#unplannedAt.some((write) => write < index))
return undefined;
const { collection, id } = itemTarget(state, view.type, record);
const key = entityTarget(view.type, collection, id);
if (this.#held.has(key) || state.entity(view.type, collection, id))
return undefined;
const since = Math.max(
lastCheck.get(key) ?? -1,
lastCheck.get(collectionTarget(view.type, collection)) ?? -1,
);
if (!this.#unplannedAt.some((write) => write > since && write < index))
return undefined;
const object = this.facts.io.withBody(record.ordinal, (body) =>
body === undefined || body === null ? undefined : at(body, view.pointer),
);
const idField = this.facts.idField(view.type);
if (
!isObject(object) ||
idField === undefined ||
scalarText(object[idField]) !== id
)
return undefined;
const entity = {
type: view.type,
collection,
id,
fields: object,
view_fields: [],
record_ordinals: [],
};
// The copy keeps the read's object, which the body's own reservation no longer covers.
this.facts.io.retain(
ENTRY_BYTES * 4 +
textBytes(key.length) +
stateValueRetentionBytes(object),
);
state.put(entity);
this.#held.add(key);
return entity;
}
}
src/domain/analysisView/environmentStateWriteChecks.ts
+5 -4
View File
@@ -240,12 +240,13 @@ const matchingTarget = (
readonly ids: readonly string[];
readonly written: string;
},
collections: readonly string[],
/** Called only when the path's own collection holds none of its ids, since listing walks the type's entities. */
collections: () => readonly string[],
held: (collection: string, id: string) => boolean,
): { readonly collection: string; readonly id: string } | undefined => {
for (const id of path.ids)
if (held(path.written, id)) return { collection: path.written, id };
const ranked = [...new Set(collections)]
const ranked = [...new Set(collections())]
.flatMap((collection) => {
const verbatim = segmentMatch(segmentsOf(collection), path.read);
return verbatim === undefined ? [] : [{ collection, verbatim }];
@@ -278,12 +279,12 @@ export const itemTarget = (
return (
matchingTarget(
path,
state.ofType(type).map((entity) => entity.collection),
() => state.ofType(type).map((entity) => entity.collection),
(collection, id) => state.entity(type, collection, id) !== undefined,
) ??
matchingTarget(
path,
removed.map((target) => target.collection),
() => removed.map((target) => target.collection),
(collection, id) =>
removed.some(
(target) => target.collection === collection && target.id === id,
src/domain/analysisView/environmentStateWriteCreate.ts
+51 -13
View File
@@ -5,7 +5,6 @@ import {
collectionPathOf,
isObject,
isParam,
isStateWrite,
normalizedPath,
scalarText,
segmentsOf,
@@ -15,6 +14,11 @@ import {
type SuppliedSegment,
} from "./environmentStateRules.js";
import { createdFields } from "./environmentStateWriteApply.js";
import {
recordedPathValues,
respondedIds,
sameLinkedSource,
} from "./environmentStateWriteLinks.js";
import {
collectionTarget,
entityTarget,
@@ -71,7 +75,10 @@ const respondedId = (
return members.length === 1 ? scalarText(members[0]) : undefined;
};
/** The written entity a create's response or a later read shows. */
/**
* The written entity a create's response or a later read shows. A linked create's read must name an
* id its response returned, as the read that linked it did, envelopes included.
*/
const createdObservation = (
context: EffectContext,
step: WriteStep,
@@ -79,6 +86,7 @@ const createdObservation = (
readonly type: string;
readonly collection: string;
readonly idField: string;
readonly linked: boolean;
},
): Observed | undefined => {
const held = (id: string) =>
@@ -87,17 +95,29 @@ const createdObservation = (
newEntityIn(step.record.ordinal, body, target.idField, held),
);
if (own !== undefined) return own;
// A response that names the created id alone ties the later read to that id.
const id = context.io.withBody(step.record.ordinal, (body) =>
respondedId(body, target.idField),
);
const { id, ids } = context.io.withBody(step.record.ordinal, (body) => {
const returned = target.linked ? respondedIds(body) : [];
// A response that names the created id alone ties the later read to that id.
const named =
respondedId(body, target.idField) ??
(returned.length === 1 ? returned[0] : undefined);
return { id: named, ids: returned };
});
const later = laterObservation(searchFrom(context, step), {
...target,
type: target.type,
collection: target.collection,
idField: target.idField,
...(id === undefined ? {} : { id }),
held,
isWrite: isStateWrite,
isWrite: context.isWrite,
});
if (later !== undefined) context.observed(later.ordinal);
// A read named by the id is tied already; otherwise a linked create's read must name a returned id.
if (
later === undefined ||
(target.linked && id === undefined && !ids.includes(later.id))
)
return undefined;
context.observed(later.ordinal);
return later;
};
@@ -155,7 +175,8 @@ const buildCreate = (
),
body: explained.body,
rules: explained.rules,
collection: explained.tied ? evidence.collection : null,
collection:
explained.tied || draft.linked !== undefined ? evidence.collection : null,
insert: draft.candidates[0]!,
});
draft.observed = evidence.observed.object;
@@ -302,6 +323,20 @@ const servedCollection = (
: selectedCollection(segments, createCandidates(context, type));
};
/** Whether a linked create's call sends the source path values its exemplar did, the only ones its link holds for. */
const fromLinkedSource = (
context: EffectContext,
draft: WriteDraft,
step: WriteStep,
): boolean =>
sameLinkedSource(
step.tool.path_template,
context.records.find(
(record) => record.ordinal === draft.plan.exemplar_ordinal,
)?.path,
recordedPathValues(step.tool.path_template, step.record.path),
);
/** Add a created entity to the state at the plan's insert position, or last before the plan has one. */
const insertCreated = (
context: EffectContext,
@@ -331,11 +366,13 @@ export const applyCreate = (
const idField = context.idField(type);
const spelled = normalizedPath(step.record.path);
// Serving writes a create into the collection it chooses for the path, and a tied plan refuses any other, so a create there is not counted.
const collection = servedCollection(context, step, type) ?? spelled;
const collection =
draft.linked ?? servedCollection(context, step, type) ?? spelled;
// A linked plan's collection holds only for its exemplar's source path, so serving refuses any other.
if (
built(draft) &&
draft.plan.collection !== null &&
draft.plan.collection !== collection
((draft.plan.collection !== null && draft.plan.collection !== collection) ||
(draft.linked !== undefined && !fromLinkedSource(context, draft, step)))
)
return UNKNOWN;
const body = requestObject(context, step.record);
@@ -343,6 +380,7 @@ export const applyCreate = (
type,
collection,
idField,
linked: draft.linked !== undefined,
});
if (observed === undefined) {
fail(draft, step, body === undefined ? "body-unusable" : "no-observation");
src/domain/analysisView/environmentStateWriteEffects.ts
+9 -2
View File
@@ -3,7 +3,6 @@ import type { JsonValue } from "../jsonValue.js";
import { ok } from "../result.js";
import {
isObject,
isStateWrite,
locateItem,
at,
same,
@@ -68,6 +67,8 @@ export interface WriteDraft {
readonly updates: UpdateCall[];
/** The last value each sequence took, by field and, for a collection's sequence, collection; the next call must follow it. */
readonly sequences: Map<string, number>;
/** The collection a create writes into when its own path names no state type: the one its returned id was read under. */
readonly linked?: string | undefined;
}
/** A write observation plus the fields its selected read view actually covers, if any. */
@@ -84,12 +85,15 @@ export interface WriteIo {
readonly requestBody: (ordinal: number) => JsonValue | undefined;
readonly emptyResponse: (ordinal: number) => boolean;
readonly retain: (bytes: number) => void;
readonly queryOf: (ordinal: number) => string | null | undefined;
}
/** What the write gate reads: the manifest tools, the timeline, the read views and the initial state. */
export interface WriteSimulationInput {
readonly tools: readonly StateToolFacts[];
readonly records: readonly StateRecordFacts[];
/** POST tools whose calls read state rather than change it. */
readonly readOnly: ReadonlySet<string>;
readonly timed: boolean;
readonly views: ReadonlyMap<string, StateView>;
readonly types: readonly StateType[];
@@ -105,6 +109,8 @@ export interface EffectContext {
readonly io: WriteIo;
readonly records: readonly StateRecordFacts[];
readonly timed: boolean;
/** Whether a record may change state: a write the capture does not show reading only. */
readonly isWrite: (record: StateRecordFacts) => boolean;
readonly viewOf: (ordinal: number) => StateView | undefined;
readonly idField: (type: string) => string;
/** Mark a later read as a create's or update's observation, so it does not check that write. */
@@ -214,6 +220,7 @@ export const searchFrom = (context: EffectContext, step: WriteStep) => ({
...(context.timed ? { after: step.record.end } : {}),
viewOf: context.viewOf,
withBody: context.io.withBody,
queryOf: context.io.queryOf,
});
/** The entity an update's response shows, located by its identity, else a later item read of the target the write resolved. */
@@ -256,7 +263,7 @@ const updatedObservation = (
idField: entity.idField,
id: entity.id,
held: () => true,
isWrite: isStateWrite,
isWrite: context.isWrite,
sameItem: (record) => {
const found = itemTarget(context.state, entity.type, record);
return found.collection === entity.collection && found.id === entity.id;
src/domain/analysisView/environmentStateWriteLinks.test.ts
+331
View File
@@ -0,0 +1,331 @@
import { describe, expect, it } from "vitest";
import type { JsonValue } from "../jsonValue.js";
import type { StateRecordFacts } from "./environmentStateRules.js";
import { creationLimit } from "./environmentStateWriteLinks.js";
const record = (
ordinal: number,
method: string,
path: string,
[start, end] = [ordinal, ordinal],
): StateRecordFacts => ({
ordinal,
method,
path,
query: undefined,
status: 200,
start,
end,
});
/** The creation limit of `/api/projects` over list reads and a POST returning `created`, in ordinal order. */
const projectsLimit = (
reads: readonly StateRecordFacts[],
post: StateRecordFacts,
bodies: ReadonlyMap<number, JsonValue>,
{
created,
retain = () => undefined,
}: {
readonly created: readonly string[];
readonly retain?: (bytes: number) => void;
},
) =>
creationLimit(
[{ record: post, ids: created }],
{
type: "/api/projects",
tools: [
{
name: "get_projects",
method: "GET",
path_template: "/api/projects",
record_ordinals: reads.map((read) => read.ordinal),
},
],
records: new Map([...reads, post].map((at) => [at.ordinal, at])),
position: (at) => at.ordinal,
withBody: (ordinal, use) => use(bodies.get(ordinal)),
retain,
},
undefined,
);
describe("creation limits", () => {
it("reads the ids of a list body longer than a call's arguments can spread", () => {
// A million elements overflow the argument limit of a spread, here and in the test workers' larger stack.
const [list, post, later] = [
record(0, "GET", "/api/metrics"),
record(1, "POST", "/api/imports"),
record(2, "GET", "/api/metrics"),
];
const bodies = new Map<number, JsonValue>([
[0, Array.from({ length: 1_000_000 }, () => 0)],
[2, [{ id: "1003" }]],
]);
const limit = creationLimit(
[{ record: post, ids: ["1003"] }],
{
type: "/api/metrics",
tools: [
{
name: "get_metrics",
method: "GET",
path_template: "/api/metrics",
record_ordinals: [0, 2],
},
],
records: new Map([list, post, later].map((at) => [at.ordinal, at])),
position: (at) => at.ordinal,
withBody: (ordinal, use) => use(bodies.get(ordinal)),
retain: () => undefined,
},
undefined,
);
expect(limit).toBe(post);
});
it("counts a read still running when the POST began as showing what it made", () => {
const earlier = record(0, "GET", "/api/projects", [0, 5]);
const post = record(2, "POST", "/api/imports", [20, 25]);
const bodies = new Map<number, JsonValue>([
[0, [{ id: "proj-0001" }]],
[1, [{ id: "proj-0001" }, { id: "proj-0002" }]],
]);
// The second list began before the POST but ended after it began, so it may show the import.
const overlapping = record(1, "GET", "/api/projects", [10, 40]);
expect(
projectsLimit([earlier, overlapping], post, bodies, {
created: ["proj-0002"],
}),
).toBe(post);
const finished = record(1, "GET", "/api/projects", [10, 15]);
expect(
projectsLimit([earlier, finished], post, bodies, {
created: ["proj-0002"],
}),
).toBeUndefined();
});
it("reads a list's ids from a suffix-named field when no identity name identifies its entities", () => {
const [list, post, later] = [
record(0, "GET", "/api/projects"),
record(1, "POST", "/api/imports"),
record(2, "GET", "/api/projects"),
];
const bodies = new Map<number, JsonValue>([
[0, [{ projectId: "proj-0001", name: "Alpha" }]],
[
2,
[
{ projectId: "proj-0001", name: "Alpha" },
{ projectId: "proj-0002", name: "Beta" },
],
],
]);
expect(
projectsLimit([list, later], post, bodies, { created: ["proj-0002"] }),
).toBe(post);
});
it("reads no suffix-named field as a list's ids when an id field already identifies its entities", () => {
const [list, post, later] = [
record(0, "GET", "/api/projects"),
record(1, "POST", "/api/users/user-0003/activate"),
record(2, "GET", "/api/projects"),
];
const project = (id: string, ownerId: string) => ({ id, ownerId });
const bodies = new Map<number, JsonValue>([
[
0,
[project("proj-0001", "user-0001"), project("proj-0002", "user-0002")],
],
[
2,
[project("proj-0001", "user-0003"), project("proj-0002", "user-0002")],
],
]);
// The owner the user call returned is no project the call made.
expect(
projectsLimit([list, later], post, bodies, { created: ["user-0003"] }),
).toBeUndefined();
});
it("counts no id an item read answered with JSON null, the absence the gate accepts, as shown", () => {
const item = record(0, "GET", "/api/projects/proj-0002");
const post = record(1, "POST", "/api/imports");
const later = record(2, "GET", "/api/projects");
const bodies = new Map<number, JsonValue>([
[0, null],
[2, [{ id: "proj-0001" }, { id: "proj-0002" }]],
]);
const limit = creationLimit(
[{ record: post, ids: ["proj-0002"] }],
{
type: "/api/projects",
tools: [
{
name: "get_projects_by_id",
method: "GET",
path_template: "/api/projects/{id}",
record_ordinals: [0],
},
{
name: "get_projects",
method: "GET",
path_template: "/api/projects",
record_ordinals: [2],
},
],
records: new Map([item, post, later].map((at) => [at.ordinal, at])),
position: (at) => at.ordinal,
withBody: (ordinal, use) => use(bodies.get(ordinal)),
retain: () => undefined,
},
undefined,
);
expect(limit).toBe(post);
});
it("reads the ids a list shows from its listed entities, not the entities they nest", () => {
const [list, post, later] = [
record(0, "GET", "/api/projects"),
record(1, "POST", "/api/users"),
record(2, "GET", "/api/projects"),
];
const project = (owner: string) => ({
id: "proj-0001",
owner: { id: owner },
});
const bodies = new Map<number, JsonValue>([
[0, [project("user-0001")]],
[2, { projects: [project("user-0009")] }],
]);
expect(
projectsLimit([list, later], post, bodies, { created: ["user-0009"] }),
).toBeUndefined();
});
});
describe("the ids creation limits keep", () => {
it("keeps and charges only the shown ids a creation returned", () => {
let retained = 0;
const lists = 50;
const reads = Array.from({ length: lists }, (_, index) =>
record(index < lists / 2 ? index : index + 1, "GET", "/api/projects"),
);
const post = record(lists / 2, "POST", "/api/imports");
const rows = (extra: number) =>
Array.from({ length: 1000 }, (_, row) => ({
id: `proj-${String(row + extra).padStart(5, "0")}`,
}));
const bodies = new Map<number, JsonValue>(
reads.map((read) => [
read.ordinal,
read.ordinal > post.ordinal
? [...rows(0), { id: "proj-made" }]
: rows(0),
]),
);
expect(
projectsLimit(reads, post, bodies, {
created: ["proj-made"],
retain: (bytes) => {
retained += bytes;
},
}),
).toBe(post);
// One kept id per later list, not a thousand per list.
expect(retained).toBeGreaterThan(0);
expect(retained).toBeLessThan(lists * 1000 * 16);
});
});
describe("the reads creation limits parse", () => {
/**
* The creation limit of `/api/projects` over list reads, a POST returning `proj-0002` that later
* lists show, and the write that ends the initial reads; with the reads parsed, and those whose
* body was read after its parse ended.
*/
const parsedLimit = (
reads: readonly StateRecordFacts[],
post: StateRecordFacts,
until: StateRecordFacts,
) => {
const parsed: number[] = [];
const outside = new Set<number>();
let parsing = false;
const withBody = <T>(
ordinal: number,
use: (body: JsonValue | undefined) => T,
): T => {
parsed.push(ordinal);
const body: JsonValue[] =
ordinal < post.ordinal
? [{ id: "proj-0001" }]
: [{ id: "proj-0001" }, { id: "proj-0002" }];
parsing = true;
try {
return use(
new Proxy(body, {
get: (target, key, receiver) => {
if (!parsing) outside.add(ordinal);
return Reflect.get(target, key, receiver) as unknown;
},
}),
);
} finally {
parsing = false;
}
};
const limit = creationLimit(
[{ record: post, ids: ["proj-0002"] }],
{
type: "/api/projects",
tools: [
{
name: "get_projects",
method: "GET",
path_template: "/api/projects",
record_ordinals: reads.map((read) => read.ordinal),
},
],
records: new Map([...reads, post, until].map((at) => [at.ordinal, at])),
position: (at) => at.ordinal,
withBody,
retain: () => undefined,
},
until,
);
return { limit, parsed, outside };
};
it("parses no read at or after the write that ends the initial reads, since no id it shows can be fresh", () => {
const reads = [0, 2, 4, 5].map((ordinal) =>
record(ordinal, "GET", "/api/projects"),
);
const post = record(1, "POST", "/api/imports");
const { limit, parsed } = parsedLimit(
reads,
post,
record(3, "PUT", "/api/projects/proj-0001"),
);
expect(limit).toBe(post);
expect(parsed).toEqual([0, 2]);
});
it("reads the ids a body shows only while its parse is charged", () => {
const reads = [0, 2].map((ordinal) =>
record(ordinal, "GET", "/api/projects"),
);
const post = record(1, "POST", "/api/imports");
const { limit, outside } = parsedLimit(
reads,
post,
record(3, "PUT", "/api/projects/proj-0001"),
);
expect(limit).toBe(post);
expect([...outside]).toEqual([]);
});
});
src/domain/analysisView/environmentStateWriteLinks.ts
+451
View File
@@ -0,0 +1,451 @@
import type { JsonValue } from "../jsonValue.js";
import {
collectionPathOf,
IDENTITY_NAMES,
identityCandidates,
isObject,
isParam,
lastSegmentValue,
listItems,
locateItem,
locateList,
normalizedPath,
pathSegmentMatches,
scalarText,
segmentsOf,
type StateEntity,
type StateRecordFacts,
type StateToolFacts,
type StateView,
} from "./environmentStateRules.js";
import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js";
import {
pathValuesOf,
type WriteEffect,
} from "./environmentStateWriteRules.js";
/** The shortest id that names one entity rather than counting one, or naming one of any type. */
const MIN_ID_LENGTH = 4;
/** Identity names that describe an entity rather than number it, so its own suffix-named id may sit beside them. */
const DESCRIPTIVE_NAMES: ReadonlySet<string> = new Set(["key", "slug", "name"]);
const succeeded = (record: StateRecordFacts): boolean =>
record.status !== null && record.status >= 200 && record.status < 300;
/** An object's identity-named scalar members. */
const identityValues = (object: JsonValue | undefined): string[] =>
isObject(object)
? IDENTITY_NAMES.flatMap((name) => {
const value = scalarText(object[name]);
return value === undefined ? [] : [value];
})
: [];
/**
* The ids a write's response names for an entity it made: an identity-named member of the body or of
* an object member (an envelope) at any length, or a scalar body or sole scalar member long enough to name one.
*/
export const respondedIds = (body: JsonValue | undefined): string[] => {
if (body === undefined || body === null) return [];
const named = (value: JsonValue | undefined): string[] => {
const text = scalarText(value);
return text !== undefined && text.length >= MIN_ID_LENGTH ? [text] : [];
};
if (!isObject(body)) return named(body);
const members = Object.values(body);
return [
...new Set([
...identityValues(body),
...(members.length === 1 ? named(members[0]) : []),
...members.flatMap(identityValues),
]),
];
};
/** The identity-named scalar values a body shows at any depth. */
const identitiesIn = (root: JsonValue | undefined): Set<string> => {
const shown = new Set<string>();
const pending: JsonValue[] = root === undefined ? [] : [root];
for (let next = pending.pop(); next !== undefined; next = pending.pop()) {
// One push per child: spreading a large array would exceed the engine's argument limit.
if (Array.isArray(next)) for (const child of next) pending.push(child);
else if (isObject(next)) {
for (const id of identityValues(next)) shown.add(id);
for (const child of Object.values(next)) pending.push(child);
}
}
return shown;
};
/**
* The ids a list read shows: its listed entities' own identity-named members and, across two or
* more of them when no id-named field identifies them, the first suffix-named field (`projectId`)
* distinct on each, so a foreign key beside an `id` is no id of theirs; any id it shows at any depth
* when it lists none.
*/
const listedIds = (body: JsonValue | undefined): Set<string> => {
const pointer = body === undefined ? undefined : locateList(body, undefined);
const items =
body === undefined || pointer === undefined
? undefined
: listItems(body, pointer);
if (items === undefined) return identitiesIn(body);
const candidates = items.length < 2 ? [] : identityCandidates(items);
const suffixed = candidates.some(
(name) => IDENTITY_NAMES.includes(name) && !DESCRIPTIVE_NAMES.has(name),
)
? undefined
: candidates.find((name) => !IDENTITY_NAMES.includes(name));
return new Set([
...items.flatMap(identityValues),
...(suffixed === undefined
? []
: items.flatMap((item) => scalarText(item[suffixed]) ?? [])),
]);
};
/** Whether an item read's body shows the entity its path names, not an absence such as JSON null. */
const showsItem = (body: JsonValue | undefined, id: string): boolean =>
body !== undefined && body !== null && locateItem(body, id) !== undefined;
/** The ids a read shows: its listed entities' for a list read, or its path's id for an item read that shows the item. */
const shownIds = (
record: StateRecordFacts,
item: boolean,
body: JsonValue | undefined,
): Set<string> => {
if (!item) return listedIds(body);
const id = lastSegmentValue(record.path);
return showsItem(body, id) ? new Set([id]) : new Set();
};
/** The shown ids a creation returned, each charged before it is kept. */
const keptIds = (
shown: ReadonlySet<string>,
returned: ReadonlySet<string>,
retain: (bytes: number) => void,
): Set<string> => {
const kept = new Set<string>();
for (const id of shown)
if (returned.has(id)) {
retain(ENTRY_BYTES + textBytes(id.length));
kept.add(id);
}
retain(ENTRY_BYTES);
return kept;
};
/** One read of a type: where it ran, when it finished, whether it reads one item, and the ids it shows. */
interface ShownRead {
readonly at: number;
readonly end: number;
readonly item: boolean;
readonly ids: ReadonlySet<string>;
}
/**
* The ids a POST at `post.at` made that a type's reads show only after it began, before `until`: a
* read finished before the POST began shows what was there, and one still running may show its
* effect. A short id may be any type's, so it counts only when the type was read before the POST
* and an item read names it after.
*/
const freshIds = (
ids: readonly string[],
shown: readonly ShownRead[],
post: { readonly at: number; readonly start: number },
until: number,
): string[] => {
const before = (read: ShownRead): boolean =>
read.at < post.at && read.end <= post.start;
const seen = shown.some(before);
return ids.filter((id) => {
const short = id.length < MIN_ID_LENGTH;
return (
(!short || seen) &&
!shown.some((read) => before(read) && read.ids.has(id)) &&
shown.some(
(read) =>
!before(read) &&
read.at < until &&
(!short || read.item) &&
read.ids.has(id),
)
);
});
};
/** A successful POST and the ids its response named. */
export interface CreatedIds {
readonly record: StateRecordFacts;
readonly ids: readonly string[];
}
/** The successful POSTs among a timeline's writes, with the ids their responses named. */
export const createdIdsOf = (
writes: readonly StateRecordFacts[],
idsOf: (ordinal: number) => string[],
): CreatedIds[] =>
writes.flatMap((record) => {
if (record.method !== "POST" || !succeeded(record)) return [];
const ids = idsOf(record.ordinal);
return ids.length === 0 ? [] : [{ record, ids }];
});
/** The state type a GET tool reads and whether it reads one item: its path less a trailing parameter. */
export const readType = (
template: string,
): { readonly item: boolean; readonly type: string } => {
const segments = segmentsOf(template);
const item = segments.length > 0 && isParam(segments.at(-1)!);
return {
item,
type: item
? `/${segments.slice(0, -1).join("/")}`
: normalizedPath(template),
};
};
/**
* The earliest POST that made an entity a type's reads first show after it, before `until`: such a
* POST changes the type although its path is elsewhere, so it bounds the type's initial recordings.
*/
export const creationLimit = (
creations: readonly CreatedIds[],
reads: {
readonly type: string;
readonly tools: readonly StateToolFacts[];
readonly records: ReadonlyMap<number, StateRecordFacts>;
readonly position: (record: StateRecordFacts) => number;
readonly withBody: <T>(
ordinal: number,
use: (body: JsonValue | undefined) => T,
) => T;
readonly retain: (bytes: number) => void;
},
until: StateRecordFacts | undefined,
): StateRecordFacts | undefined => {
if (creations.length === 0) return undefined;
// Only the ids some creation returned can be fresh, so each read keeps those alone, charged.
const returned = new Set(creations.flatMap((creation) => creation.ids));
const end = until === undefined ? Infinity : reads.position(until);
const readers = reads.tools.flatMap((tool) => {
const read = readType(tool.path_template);
return tool.method === "GET" && read.type === reads.type
? [{ tool, item: read.item }]
: [];
});
// A read at or after `end` is no counted creation's read before or after it, so it is not parsed.
const shown: ShownRead[] = readers.flatMap(({ tool, item }) =>
tool.record_ordinals.flatMap((ordinal) => {
const record = reads.records.get(ordinal);
if (record === undefined || !succeeded(record)) return [];
const at = reads.position(record);
if (at >= end) return [];
const ids = reads.withBody(ordinal, (body) =>
shownIds(record, item, body),
);
return [
{
at,
end: record.end,
item,
ids: keptIds(ids, returned, reads.retain),
},
];
}),
);
for (const creation of [...creations].sort(
(left, right) => reads.position(left.record) - reads.position(right.record),
)) {
const at = reads.position(creation.record);
if (at >= end) break;
const post = { at, start: creation.record.start };
if (freshIds(creation.ids, shown, post, end).length > 0)
return creation.record;
}
return undefined;
};
/**
* The state type and collection a POST whose path names none creates into: that of the first later
* successful item read naming an id its first successful call returned, before any other tool's
* write, when neither the initial state nor any read finished before the call of that id, its
* type's lists or its collection showed that id. A short id may be any type's, so it links only to a type read before the call.
*/
export const linkedCreate = (input: {
readonly tool: StateToolFacts;
readonly records: readonly StateRecordFacts[];
readonly isWrite: (record: StateRecordFacts) => boolean;
readonly viewOf: (ordinal: number) => StateView | undefined;
readonly entities: readonly StateEntity[];
readonly respondedIds: (ordinal: number) => readonly string[];
readonly shownIds: (ordinal: number) => ReadonlySet<string>;
readonly showsItem: (ordinal: number, id: string) => boolean;
}): { readonly type: string; readonly collection: string } | undefined => {
const { tool, records } = input;
const own = new Set(tool.record_ordinals);
const first = records.findIndex(
(record) => own.has(record.ordinal) && succeeded(record),
);
if (first === -1) return undefined;
const post = records[first]!;
const ids = new Set(input.respondedIds(post.ordinal));
if (ids.size === 0) return undefined;
// Only a read finished before the call began shows what was there before it, as `freshIds` decides.
const before = records
.slice(0, first)
.filter((record) => record.end <= post.start);
for (const record of records.slice(first + 1)) {
// Another write may have made what a later read shows, so the search ends there.
if (!own.has(record.ordinal) && input.isWrite(record)) return undefined;
const view = input.viewOf(record.ordinal);
// A read naming the id but showing no entity, such as JSON null, says nothing of where the call created.
if (
record.method !== "GET" ||
!succeeded(record) ||
view?.shape !== "item" ||
!ids.has(lastSegmentValue(record.path)) ||
!input.showsItem(record.ordinal, lastSegmentValue(record.path))
)
continue;
const id = lastSegmentValue(record.path);
const collection = collectionPathOf(record, true);
// An earlier read of the id under another type or collection names another entity.
const readBefore = before.some((earlier) => {
const read = input.viewOf(earlier.ordinal);
return (
earlier.method === "GET" &&
succeeded(earlier) &&
lastSegmentValue(earlier.path) === id &&
((read?.shape === "item" && read.type === view.type) ||
normalizedPath(collectionPathOf(earlier, true)) ===
normalizedPath(collection)) &&
input.showsItem(earlier.ordinal, id)
);
});
const held =
readBefore ||
input.entities.some(
(entity) => entity.type === view.type && entity.id === id,
) ||
before.some((earlier) => {
const read = input.viewOf(earlier.ordinal);
return (
earlier.method === "GET" &&
succeeded(earlier) &&
((read?.shape === "list" && read.type === view.type) ||
normalizedPath(earlier.path) === normalizedPath(collection)) &&
input.shownIds(earlier.ordinal).has(id)
);
});
const unseen =
id.length < MIN_ID_LENGTH &&
!before.some((earlier) => {
const read = input.viewOf(earlier.ordinal);
return (
earlier.method === "GET" &&
succeeded(earlier) &&
(read?.shape === "item" || read?.shape === "list") &&
read.type === view.type
);
});
return held || unseen ? undefined : { type: view.type, collection };
}
return undefined;
};
/** The collection a create plan writes into when its tool's own path names no state type: the one its returned id was read under. */
export const linkedCollection = (
plan: {
readonly effect: string;
readonly target: string | null;
readonly collection: string | null;
},
template: string,
): string | undefined =>
plan.effect === "create" &&
plan.target !== null &&
plan.collection !== null &&
plan.target !== normalizedPath(template)
? plan.collection
: undefined;
/**
* The values a recorded path gives its template's parameters as spelled, undecoded, by segment
* position, since a template may name two parameters alike; undefined when it does not fit.
*/
export const recordedPathValues = (
template: string,
path: string,
): Map<number, string> | undefined => {
if (pathValuesOf(template, path) === undefined) return undefined;
const segments = segmentsOf(path);
return new Map(
segmentsOf(template).flatMap((segment, index) =>
isParam(segment) ? [[index, segments[index]!] as const] : [],
),
);
};
/**
* Whether a linked create's call sends the path values its exemplar did, each as recorded or in the
* other spelling once percent-decoded: the read that linked its target followed only that call. The
* call's values come as spelled, so neither side is decoded twice.
*/
export const sameLinkedSource = (
template: string,
exemplar: string | undefined,
call: ReadonlyMap<number, string> | undefined,
): boolean => {
const recorded =
exemplar === undefined ? undefined : recordedPathValues(template, exemplar);
if (recorded === undefined || call === undefined) return false;
return (
recorded.size === call.size &&
[...recorded].every(([index, value]) => {
const given = call.get(index);
return given !== undefined && pathSegmentMatches(value, given);
})
);
};
/** A write's effect and target, where a POST its path gives no target becomes a create into the type its returned id is first read under. */
export const linkedTarget = (
tool: StateToolFacts,
classified: { readonly effect: WriteEffect; readonly target: string | null },
input: {
readonly records: readonly StateRecordFacts[];
readonly isWrite: (record: StateRecordFacts) => boolean;
readonly viewOf: (ordinal: number) => StateView | undefined;
readonly entities: readonly StateEntity[];
readonly io: {
readonly withBody: <T>(
ordinal: number,
use: (body: JsonValue | undefined) => T,
) => T;
};
},
): {
readonly effect: WriteEffect;
readonly target: string | null;
readonly collection?: string;
} => {
const link =
classified.effect === "none" && tool.method === "POST"
? linkedCreate({
tool,
records: input.records,
isWrite: input.isWrite,
viewOf: input.viewOf,
entities: input.entities,
respondedIds: (ordinal) => input.io.withBody(ordinal, respondedIds),
shownIds: (ordinal) => input.io.withBody(ordinal, listedIds),
showsItem: (ordinal, id) =>
input.io.withBody(ordinal, (body) => showsItem(body, id)),
})
: undefined;
return link === undefined
? classified
: { effect: "create", target: link.type, collection: link.collection };
};
src/domain/analysisView/environmentStateWriteObserve.ts
+9 -9
View File
@@ -14,6 +14,7 @@ import {
type StateRecordFacts,
type StateView,
} from "./environmentStateRules.js";
import { queryValues, replayReason } from "./environmentStateVariants.js";
/** Where an observed entity object was found: the record and a JSON pointer into its response. */
export interface Observed {
@@ -92,23 +93,22 @@ export interface ObservationSearch {
ordinal: number,
use: (body: JsonValue | undefined) => T,
) => T;
readonly queryOf: (ordinal: number) => string | null | undefined;
}
/** Whether a read asked for some fields only and shows no whole entity: an item view replays such reads, or the read leaves out a field its view requires. */
/** Whether a successful read neither checks the state nor observes a write: serving replays it, or it asked for some fields only and shows no whole entity. */
export const projectedRead = (
view: StateView,
record: StateRecordFacts,
io: Pick<ObservationSearch, "withBody">,
io: Pick<ObservationSearch, "withBody" | "queryOf">,
): boolean => {
if (
(view.shape !== "item" && view.shape !== "list") ||
!projectsFields(record.query)
)
return false;
// An absence or error holds no entity, so it is never a partial one.
if (view.shape !== "item" && view.shape !== "list") return false;
// An absence or error holds no entity, so it is never a partial one and stays evidence.
if (!succeeded(record)) return false;
if (view.shape === "item" && view.projected_record_ordinals !== undefined)
if (replayReason(view, queryValues(record.query), io.queryOf) !== undefined)
return true;
// Serving answers such a selection with the whole entity, which part of one can neither confirm nor check.
if (!projectsFields(record.query)) return false;
return io.withBody(record.ordinal, (body) => {
const objects =
body === undefined || body === null
src/domain/analysisView/environmentStateWrites.test.ts
+679
View File
@@ -12,6 +12,7 @@ import {
BOOK_ROWS,
BOOK_WRITE_CLOCK,
DETAILED_LIST_ROWS,
EXPANDED_LIST_ROWS,
FIRST_BOOK,
OMITTED_FIELD_ROWS,
PROJECTED_ROWS,
@@ -31,6 +32,7 @@ import {
bookClock,
detailedBook,
environmentStateViewOf,
expandedBook,
rawCaptureEvidence,
renamedBook,
type Book,
@@ -215,6 +217,83 @@ describe("wider write coverage", () => {
});
});
it("leaves a list line that expands its books to replay, serving the create the plain list confirms", () => {
const state = stateOf(EXPANDED_LIST_ROWS);
expect(state.views.find((view) => view.tool === "get_books")).toMatchObject(
{
shape: "list",
checked_record_ordinals: [0],
projected_record_ordinals: [1],
projected_query_keys: [{ name: "expand" }],
},
);
expect(planOf(state, "post_books")).toMatchObject({
served: true,
checked_record_ordinals: [5],
});
});
it("leaves a list line that asks for some fields only to replay", () => {
const view = stateOf([
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
[
"GET",
"/api/books?fields=id,name",
[
{ id: 1, name: "First Book" },
{ id: 2, name: "Second Book" },
],
],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/2", SECOND_BOOK],
]).views.find((candidate) => candidate.tool === "get_books");
expect(view).toMatchObject({
shape: "list",
checked_record_ordinals: [0],
projected_record_ordinals: [1],
projected_query_keys: [{ name: "fields" }],
});
});
it("tells a line that adds a parameter apart by the members its books show, not by the parameters it shares", () => {
const view = stateOf([
["GET", "/api/books?sort=name&limit=1", [FIRST_BOOK]],
["GET", "/api/books?sort=name&q=first", [FIRST_BOOK]],
[
"GET",
"/api/books?sort=name&with=stats&offset=1",
[{ ...SECOND_BOOK, stats: { reads: 4 } }],
],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/2", SECOND_BOOK],
]).views.find((candidate) => candidate.tool === "get_books");
expect(view).toMatchObject({
shape: "list",
checked_record_ordinals: [0, 1],
projected_record_ordinals: [2],
projected_query_keys: [{ name: "with" }],
});
});
it("confirms a delete by a missing book's read on a line that expands its shelf", () => {
const state = stateOf([
...BOOK_INIT_ROWS,
["GET", "/api/books/1?expand=shelf", expandedBook(FIRST_BOOK)],
["DELETE", "/api/books/2", "", 204],
["GET", "/api/books/2?expand=shelf", BOOK_NOT_FOUND, 404],
]);
expect(
state.views.find((view) => view.tool === "get_books_by_id"),
).toMatchObject({
projected_record_ordinals: [3],
projected_query_keys: [{ name: "expand" }],
});
expect(planOf(state, "delete_books_by_id")).toMatchObject({
served: true,
checked_record_ordinals: [5],
});
});
it("serves creates whose number follows the largest held one in its digit groups", () => {
expect(planOf(stateOf(ASSET_ROWS), "post_books")).toMatchObject({
served: true,
@@ -1259,3 +1338,603 @@ describe("what a list query read shows of writes", () => {
});
});
});
describe("calls the capture shows reading only", () => {
const LOOKUP: TimedCaptureRow = [
"POST",
"/api/books/ids",
[FIRST_BOOK, SECOND_BOOK],
200,
{ ids: [1, 2] },
];
it("replays a lookup POST and plans no write for it", () => {
const state = stateOf([...BOOK_INIT_ROWS, LOOKUP]);
expect(state.writes.map((plan) => plan.tool)).not.toContain(
"post_books_ids",
);
expect(state.views).toContainEqual({
tool: "post_books_ids",
shape: "recorded",
reason: "query-body",
});
});
it("keeps a read after a lookup POST among the initial recordings", () => {
// The lookup changes nothing, so the second book's only read, after it, still seeds the state.
const state = stateOf([
["GET", "/api/books/1", FIRST_BOOK],
LOOKUP,
["GET", "/api/books/2", SECOND_BOOK],
]);
expect(state.entities.map((entity) => entity.id).sort()).toEqual([
"1",
"2",
]);
});
});
describe("types whose reads disagree across a write", () => {
const recipe = (id: number, name: string, views: number) => ({
id,
name,
views,
});
it("plans a type its reads disagree on across an unrelated write from the reads before it, and serves its create", () => {
const state = stateOf([
["GET", "/api/recipes", [recipe(1, "Soup", 1), recipe(2, "Stew", 4)]],
["GET", "/api/recipes/1", recipe(1, "Soup", 1)],
// Logging a view bumps the counter, so the reads after it disagree with those before.
["POST", "/api/view-log", { ok: true }, 200, { recipe: 1 }],
["GET", "/api/recipes/1", recipe(1, "Soup", 2)],
["POST", "/api/recipes", recipe(3, "Broth", 0), 201, { name: "Broth" }],
["GET", "/api/recipes/3", recipe(3, "Broth", 0)],
]);
expect(state.types).toMatchObject([
{ collection_template: "/api/recipes", init_before_ordinal: 2 },
]);
expect(planOf(state, "post_recipes")).toMatchObject({
served: true,
checked_record_ordinals: [5],
});
});
it("ends a type's state at the earlier of a write its reads disagree across and a create that made one of its entities", () => {
const rows = (disagreeFirst: boolean): TimedCaptureRow[] => {
const disagreement: TimedCaptureRow[] = [
["POST", "/api/view-log", { ok: true }, 200, { recipe: 1 }],
["GET", "/api/recipes/1", recipe(1, "Soup", 2)],
];
const creation: TimedCaptureRow[] = [
["POST", "/api/imports", { id: "rcp-0042" }, 201, { name: "Broth" }],
["GET", "/api/recipes", [recipe(1, "Soup", 1), { id: "rcp-0042" }]],
];
return [
["GET", "/api/recipes", [recipe(1, "Soup", 1)]],
["GET", "/api/recipes/1", recipe(1, "Soup", 1)],
...(disagreeFirst
? [...disagreement, ...creation]
: [...creation, ...disagreement]),
];
};
expect(stateOf(rows(true)).types).toMatchObject([
{ init_before_ordinal: 2 },
]);
expect(stateOf(rows(false)).types).toMatchObject([
{ init_before_ordinal: 2 },
]);
});
});
const note = (id: number, title: string) => ({ id, title });
const A = note(1001, "Alpha");
const B = note(1002, "Beta");
const C = note(1003, "Gamma");
const notesOf = (state: StateView) =>
state.entities
.filter((entity) => entity.type === "/api/notes")
.map((entity) => entity.id);
/** Two notes read, the given exchanges, then a POST elsewhere whose returned id the notes are read under. */
const published = (before: readonly TimedCaptureRow[]): TimedCaptureRow[] => [
["GET", "/api/notes", [A, B]],
["GET", "/api/notes/1001", A],
["GET", "/api/notes/1002", B],
...before,
["POST", "/api/drafts/publish", { id: 1003 }, 201, { title: "Gamma" }],
["GET", "/api/notes/1003", C],
["GET", "/api/notes", [A, B, C]],
];
describe("creates read under another path", () => {
it("plans a POST to another path as a create into the type its returned id is first read under", () => {
const state = stateOf(published([]));
expect(planOf(state, "post_drafts_publish")).toMatchObject({
effect: "create",
target: "/api/notes",
collection: "/api/notes",
});
expect(notesOf(state)).toEqual(["1001", "1002"]);
});
it("ends a type's initial reads at a POST elsewhere that made one of its entities", () => {
const state = stateOf([
["GET", "/api/notes", [A, B]],
["POST", "/api/imports", { id: 1003 }, 201, { title: "Gamma" }],
["GET", "/api/notes", [A, B, C]],
]);
expect(notesOf(state)).toEqual(["1001", "1002"]);
expect(
state.types.find((type) => type.collection_template === "/api/notes"),
).toMatchObject({ init_before_ordinal: 1 });
});
it("links no create whose returned id a read showed before it", () => {
const shownBefore = published([
[
"PATCH",
"/api/notes/1001",
note(1001, "Alpha 2"),
200,
{ title: "Alpha 2" },
],
["GET", "/api/notes/1003", C],
]);
expect(planOf(stateOf(shownBefore), "post_drafts_publish")).toMatchObject({
effect: "none",
reason: "no-state-target",
});
expect(planOf(stateOf(published([])), "post_drafts_publish")).toMatchObject(
{ effect: "create" },
);
});
/** A create linked under project 11, then a call for project 12, whose note a later item read and delete show. */
const otherProject = (): TimedCaptureRow[] => {
const D = note(1004, "Delta");
return [
["GET", "/api/projects/11/notes", [A, B]],
["GET", "/api/projects/11/notes/1001", A],
["GET", "/api/projects/11/notes/1002", B],
[
"POST",
"/api/projects/11/publish",
{ id: 1003 },
201,
{ title: "Gamma" },
],
["GET", "/api/projects/11/notes/1003", C],
["GET", "/api/projects/11/notes", [A, B, C]],
["POST", "/api/projects/12/publish", D, 201, { title: "Delta" }],
["GET", "/api/projects/12/notes/1004", D],
["GET", "/api/projects/11/notes", [A, B, C]],
["DELETE", "/api/projects/12/notes/1004", "", 204],
["GET", "/api/projects/12/notes/1004", { error: "Not found" }, 404],
];
};
it("puts no linked create's call that sends other path arguments into the linked collection", () => {
// Project 12's note never reaches project 11, so the later read of project 11 confirms the create.
expect(
planOf(stateOf(otherProject()), "post_projects_publish_by_id"),
).toMatchObject({
served: true,
collection: "/api/projects/11/notes",
checked_record_ordinals: [5],
contradicted_by: null,
});
});
it("takes the entity such a call made when a read first shows it", () => {
expect(
planOf(stateOf(otherProject()), "delete_projects_notes_by_id_id_2"),
).toMatchObject({ served: true, checked_record_ordinals: [10] });
});
it("links a create whose response names a short id under an identity member", () => {
const [one, two, three] = [
note(1, "Alpha"),
note(2, "Beta"),
note(3, "Gamma"),
];
const rows = (response: JsonValue): TimedCaptureRow[] => [
["GET", "/api/notes", [one, two]],
["GET", "/api/notes/1", one],
["GET", "/api/notes/2", two],
["POST", "/api/drafts/publish", response, 201, { title: "Gamma" }],
["GET", "/api/notes/3", three],
["GET", "/api/notes", [one, two, three]],
];
expect(
planOf(stateOf(rows({ id: 3 })), "post_drafts_publish"),
).toMatchObject({ effect: "create", target: "/api/notes", served: true });
// A bare or unnamed short scalar may count rather than name, so it links nothing.
for (const response of [3, { created: 3 }])
expect(
planOf(stateOf(rows(response)), "post_drafts_publish"),
).toMatchObject({ effect: "none", reason: "no-state-target" });
});
it("links a short id to no type that no read showed before the call", () => {
// Another type's id 1 may be the one the call returned, so the notes keep their initial reads.
const [one, two] = [note(1, "Alpha"), note(2, "Beta")];
const state = stateOf([
["POST", "/api/imports", { id: 1 }, 201, { source: "csv" }],
["GET", "/api/notes/1", one],
["GET", "/api/notes", [one, two]],
]);
expect(planOf(state, "post_imports")).toMatchObject({
effect: "none",
reason: "no-state-target",
});
expect(notesOf(state)).toEqual(["1", "2"]);
});
it("links no create whose returned id the initial state holds", () => {
const held = stateOf([
["GET", "/api/notes", [A, B, C]],
["GET", "/api/notes/1001", A],
["POST", "/api/drafts/publish", { id: 1003 }, 201, { title: "Gamma" }],
["GET", "/api/notes/1003", C],
]);
expect(planOf(held, "post_drafts_publish")).toMatchObject({
effect: "none",
reason: "no-state-target",
});
expect(notesOf(held)).toEqual(["1001", "1002", "1003"]);
expect(planOf(stateOf(published([])), "post_drafts_publish")).toMatchObject(
{ effect: "create" },
);
});
});
describe("creates a later write may explain", () => {
it("links no create past a later write, which may have made what the read shows", () => {
const drafted = (publish: boolean): TimedCaptureRow[] => [
["GET", "/api/notes", [A, B]],
["GET", "/api/notes/1001", A],
["GET", "/api/notes/1002", B],
["POST", "/api/drafts", C, 201, { title: "Gamma" }],
...(publish
? ([
[
"POST",
"/api/drafts/1003/publish",
{ id: 1003, status: "published" },
201,
],
] as TimedCaptureRow[])
: []),
["GET", "/api/notes/1003", C],
["GET", "/api/notes", [A, B, C]],
];
const state = stateOf(drafted(true));
// The publish call made the note, so the draft's POST creates none.
expect(planOf(state, "post_drafts")).toMatchObject({
effect: "none",
reason: "no-state-target",
});
expect(
state.writes.find((plan) => plan.tool !== "post_drafts"),
).toMatchObject({ effect: "create", target: "/api/notes" });
expect(planOf(stateOf(drafted(false)), "post_drafts")).toMatchObject({
effect: "create",
target: "/api/notes",
});
});
});
describe("state order after a variant line is dropped", () => {
it("orders entities by the lines left, so a plain list listing them in another order lists them all", () => {
const view = stateOf([
[
"GET",
"/api/books?expand=shelf",
[expandedBook(SECOND_BOOK), expandedBook(FIRST_BOOK)],
],
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
]).views.find((candidate) => candidate.tool === "get_books");
expect(view).toMatchObject({
shape: "list",
membership: "all",
projected_record_ordinals: [0],
});
});
});
describe("item lines first recorded after a write", () => {
it("replays an item line that shows another representation, though first recorded after a write", () => {
const created = book(3, "Third Book", bookClock(4));
const state = stateOf([
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/2", SECOND_BOOK],
["POST", "/api/books", created, 201, { name: "Third Book" }],
["GET", "/api/books/3", created],
["GET", "/api/books/1?expand=shelf", expandedBook(FIRST_BOOK)],
]);
expect(
state.views.find((view) => view.tool === "get_books_by_id"),
).toMatchObject({
projected_record_ordinals: [5],
projected_query_keys: [{ name: "expand" }],
});
});
});
describe("item lines first recorded after a write, against the lines left", () => {
const created = book(3, "Third Book", bookClock(4));
const lines = (late: TimedCaptureRow): TimedCaptureRow[] => [
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/2", SECOND_BOOK],
["GET", "/api/books/1?expand=shelf", expandedBook(FIRST_BOOK)],
["POST", "/api/books", created, 201, { name: "Third Book" }],
["GET", "/api/books/3", created],
late,
];
const itemView = (rows: TimedCaptureRow[]) =>
stateOf(rows).views.find((view) => view.tool === "get_books_by_id");
it("compares a later line with the lines left after variants are dropped, not the dropped ones", () => {
// The early `expand` line is dropped as a variant, so `include` shows a shelf the lines left never show.
expect(
itemView(
lines(["GET", "/api/books/2?include=shelf", expandedBook(SECOND_BOOK)]),
),
).toMatchObject({
projected_record_ordinals: [3, 6],
projected_query_keys: expect.arrayContaining([{ name: "include" }]),
});
});
it("replays a later selection of some fields that keeps the identity but leaves members out", () => {
expect(
itemView(
lines([
"GET",
"/api/books/2?fields=id,name",
{ id: 2, name: "Second Book" },
]),
),
).toMatchObject({
projected_record_ordinals: [3, 6],
projected_query_keys: expect.arrayContaining([{ name: "fields" }]),
});
});
});
describe("reads overlapping a linked create", () => {
it("links a create whose id only a list still running when it began showed before", () => {
const state = stateOf([
["GET", "/api/notes", [A, B]],
["GET", "/api/notes/1001", A],
["GET", "/api/notes/1002", B],
// Begun before the publish and finished after it, so it may show what the publish made.
["GET", "/api/notes", [A, B, C], 200, undefined, 2500],
["POST", "/api/drafts/publish", { id: 1003 }, 201, { title: "Gamma" }],
["GET", "/api/notes/1003", C],
["GET", "/api/notes", [A, B, C]],
]);
expect(planOf(state, "post_drafts_publish")).toMatchObject({
effect: "create",
target: "/api/notes",
});
expect(notesOf(state)).toEqual(["1001", "1002"]);
});
});
describe("earlier reads of the returned id under other types", () => {
it("links a create whose id an earlier read of another type named", () => {
const user = { id: 1003, login: "gamma" };
const state = stateOf([
["GET", "/api/users/1003", user],
["GET", "/api/notes", [A, B]],
["GET", "/api/notes/1001", A],
["GET", "/api/notes/1002", B],
["POST", "/api/drafts/publish", { id: 1003 }, 201, { title: "Gamma" }],
["GET", "/api/notes/1003", C],
["GET", "/api/notes", [A, B, C]],
]);
expect(planOf(state, "post_drafts_publish")).toMatchObject({
effect: "create",
target: "/api/notes",
});
});
});
describe("item reads that show no entity", () => {
it("links a create past a read of its id that answered JSON null under another type", () => {
const user = (id: number, login: string) => ({ id, login });
const state = stateOf([
["GET", "/api/notes", [A, B]],
["GET", "/api/notes/1001", A],
["GET", "/api/notes/1002", B],
["GET", "/api/users/1001", user(1001, "alpha")],
// Ends the users' initial reads, so the null read below plans nothing.
[
"PATCH",
"/api/users/1001",
user(1001, "alpha2"),
200,
{ login: "alpha2" },
],
["POST", "/api/drafts/publish", { id: 1003 }, 201, { title: "Gamma" }],
["GET", "/api/users/1003", null],
["GET", "/api/notes/1003", C],
["GET", "/api/notes", [A, B, C]],
]);
expect(planOf(state, "post_drafts_publish")).toMatchObject({
effect: "create",
target: "/api/notes",
});
});
});
describe("what a linked create observes", () => {
it("observes a linked create only in a read of an id its response named, inside an envelope too", () => {
const D = note(1004, "Delta");
const state = stateOf([
["GET", "/api/notes", [A, B]],
["GET", "/api/notes/1001", A],
["GET", "/api/notes/1002", B],
[
"POST",
"/api/drafts/publish",
{ data: { id: 1003 } },
201,
{ title: "Gamma" },
],
// Another note the state never held, which the publish did not make.
["GET", "/api/notes/1004", D],
["GET", "/api/notes/1003", C],
]);
expect(planOf(state, "post_drafts_publish")).toMatchObject({
effect: "create",
target: "/api/notes",
observation: { ordinal: 5 },
});
});
});
describe("creates whose returned id an earlier read showed", () => {
it("links no create whose returned id a list of its collection showed before it", () => {
const listed = (shown: JsonValue[]): TimedCaptureRow[] =>
published([
[
"PATCH",
"/api/notes/1001",
note(1001, "Alpha 2"),
200,
{ title: "Alpha 2" },
],
["GET", "/api/notes?include=archived", shown],
]);
const alpha = note(1001, "Alpha 2");
expect(
planOf(stateOf(listed([alpha, B, C])), "post_drafts_publish"),
).toMatchObject({ effect: "none", reason: "no-state-target" });
expect(
planOf(stateOf(listed([alpha, B])), "post_drafts_publish"),
).toMatchObject({ effect: "create", target: "/api/notes" });
});
});
describe("lines left to replay", () => {
it("plans a list past a line that asks for some fields only and leaves the proved id out", () => {
const created = book(3, "Third Book", bookClock(4));
const state = stateOf([
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
[
"GET",
"/api/books?fields=name",
[{ name: "First Book" }, { name: "Second Book" }],
],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/2", SECOND_BOOK],
["POST", "/api/books", created, 201, { name: "Third Book" }],
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK, created]],
]);
expect(state.views.find((view) => view.tool === "get_books")).toMatchObject(
{
shape: "list",
checked_record_ordinals: [0],
projected_record_ordinals: [1],
projected_query_keys: [{ name: "fields" }],
},
);
expect(planOf(state, "post_books")).toMatchObject({
served: true,
checked_record_ordinals: [5],
});
});
it("checks a list line asking for every field beside the plain one, as serving answers both from the state", () => {
const created = book(3, "Third Book", bookClock(4));
const state = stateOf([
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
["GET", "/api/books?fields=*", [FIRST_BOOK, SECOND_BOOK]],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/2", SECOND_BOOK],
["POST", "/api/books", created, 201, { name: "Third Book" }],
["GET", "/api/books?fields=*", [FIRST_BOOK, SECOND_BOOK, created]],
]);
const view = state.views.find(
(candidate) => candidate.tool === "get_books",
);
expect(view).toMatchObject({
shape: "list",
checked_record_ordinals: [0, 1],
});
expect(view).not.toHaveProperty("projected_record_ordinals");
expect(planOf(state, "post_books")).toMatchObject({
served: true,
checked_record_ordinals: [5],
});
});
it("keeps a list's whole line when only a replayed line's membership changed across a write", () => {
// The expanded line lists shelved books only; the whole list still shows both after the export.
const view = stateOf([
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/2", SECOND_BOOK],
[
"GET",
"/api/books?expand=shelf",
[FIRST_BOOK, SECOND_BOOK].map(expandedBook),
],
["POST", "/api/exports", { queued: true }, 202, { format: "csv" }],
["GET", "/api/books?expand=shelf", [expandedBook(FIRST_BOOK)]],
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
]).views.find((candidate) => candidate.tool === "get_books");
expect(view).toMatchObject({
shape: "list",
checked_record_ordinals: [5],
projected_record_ordinals: [4],
projected_query_keys: [{ name: "expand" }],
});
});
});
describe("field selections the gate reads as serving answers them", () => {
it("approves no create from a read serving replays, a selection no checked line gave", () => {
const state = stateOf([
["GET", "/api/books/1?fields=*", FIRST_BOOK],
["GET", "/api/books/2?fields=*", SECOND_BOOK],
["GET", "/api/books/2?fields=name", { name: "Second Book" }],
["POST", "/api/books", THIRD_BOOK, 201, { name: "Third Book" }],
[
"GET",
"/api/books/3?fields=id,name,slug,created_at,updated_at",
THIRD_BOOK,
],
]);
expect(
state.views.find((view) => view.tool === "get_books_by_id"),
).toMatchObject({
projected_query_keys: [{ name: "fields", value: "name" }],
});
expect(planOf(state, "post_books")).toMatchObject({
served: false,
reason: "unconfirmed",
});
});
it("checks an update against a read of every field, which serving answers from the state", () => {
const state = stateOf([
...BOOK_INIT_ROWS,
["GET", "/api/books/2?fields=name", { name: "Second Book" }],
["PATCH", "/api/books/1", RENAMED_BOOK, 200, { name: "Renamed Book" }],
["GET", "/api/books/1?fields=*", FIRST_BOOK],
["GET", "/api/books/1", RENAMED_BOOK],
]);
expect(
state.views.find((view) => view.tool === "get_books_by_id"),
).toMatchObject({ projected_query_keys: [{ name: "fields" }] });
expect(planOf(state, "patch_books_by_id")).toMatchObject({
served: false,
reason: "contradicted",
contradicted_by: 5,
});
});
});
tests/fixtures/analysisView.ts
Vendored
+1 -1
View File
@@ -320,7 +320,7 @@ export type TimedCaptureRow = readonly [
path: string,
body?: JsonValue,
status?: number,
request?: JsonValue,
request?: JsonValue | undefined,
time?: number,
];
tests/fixtures/environmentStateWrites.ts
Vendored
+20
View File
@@ -126,6 +126,26 @@ export const PROJECTED_ROWS: readonly TimedCaptureRow[] = [
["GET", "/api/books/2?fields=name", { name: "Second Book" }],
];
/** A book as a list asked to expand its shelf shows it. */
export const expandedBook = (entity: Book): Book => ({
...entity,
shelf: { id: 7, name: "Main Hall" },
});
/** The list read plainly and with its shelves expanded, then a create the plain list shows. */
export const EXPANDED_LIST_ROWS: readonly TimedCaptureRow[] = [
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK]],
[
"GET",
"/api/books?expand=shelf",
[FIRST_BOOK, SECOND_BOOK].map(expandedBook),
],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/2", SECOND_BOOK],
["POST", "/api/books", THIRD_BOOK_AT_4, 201, { name: "Third Book" }],
["GET", "/api/books", [FIRST_BOOK, SECOND_BOOK, THIRD_BOOK_AT_4]],
];
/** A book numbered as an asset, in digit groups. */
export const assetBook = (entity: Book, asset: string): Book => ({
...entity,