mirror of
https://github.com/morluto/rea.git
synced 2026-11-03 05:16:16 +00:00
feat(environment): extract the initial state a capture's reads observed
Add the environment-state analysis view. From the parents the environment manifest accepts, it extracts the data the capture's reads observed before the first related write: entity types with an identity field (proved by a path parameter or inferred from an identity name), entities with their recorded values and provenance, documents for written non-entity paths, and a render plan per read tool that is kept only when it reproduces the tool's latest initial recordings from the state; every other read tool is a recorded view with the reason. The Evidence is always inferred. Response-body decoding moves to a domain owner shared with replay, exact JSON parsing moves to the owner of the exact-number policy, the manifest and state views share parent admission and record grouping, and projectAnalysisView takes its accumulators as one object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
bf1eeee94f
commit
ad439bd2f2
39 files changed
+2799
-221
No files matched your search
@@ -77,3 +77,9 @@ body shapes and record pointers for an agent environment. Opt into
|
||||
`template_rules` only when recorded identifiers need them; every template,
|
||||
parameter and merged sibling is an inference, no header, cookie or body value is
|
||||
copied, and the three stated unknowns stay unresolved.
|
||||
|
||||
Use `view: {kind: "environment-state"}` with the manifest's `template_rules` to
|
||||
extract the data those reads observed instead: entity types and identity
|
||||
fields, entities with recorded values, documents, and per-tool render plans
|
||||
checked against the recordings. The state comes only from reads before the
|
||||
first related write; types, identity and list membership are inferences.
|
||||
@@ -213,6 +213,18 @@ That same warning prefixes every tool description in the standalone card, so
|
||||
card discovery carries it before an MCP client initializes. It labels the
|
||||
captured text without changing the tool names or schema fields clients use.
|
||||
|
||||
## The state a capture observed
|
||||
|
||||
Replay answers each call from one recording, so the answers reflect whichever
|
||||
moment of the capture recorded them. The
|
||||
[environment state](web-network-captures.md#environment-state) view extracts
|
||||
instead the data the capture's reads observed before the first related write:
|
||||
entity types with an identity field, entities with their recorded values,
|
||||
documents for written paths, and a render plan per read tool, kept only where
|
||||
it reproduces that tool's recordings from the state. It is the starting state
|
||||
an environment can hold, load and report; serving from it is a later increment
|
||||
and does not change replay.
|
||||
|
||||
## Serve a manifest
|
||||
|
||||
```sh
|
||||
|
||||
@@ -225,6 +225,13 @@ names are inferences; the Evidence is inferred whenever any tool or excluded gro
|
||||
is templated, and derived otherwise.
|
||||
See [environment manifests](web-network-captures.md#environment-manifest).
|
||||
|
||||
`inspect_analysis_view` also accepts `environment-state` for the same parents,
|
||||
extracting the initial state the recorded reads observed: entity types with an
|
||||
identity field, entities, documents for written non-entity paths, and a render
|
||||
plan per read tool that reproduces its recordings from the state or names why it
|
||||
cannot. Types and identity are inferences, so the Evidence is always inferred.
|
||||
See [environment state](web-network-captures.md#environment-state).
|
||||
|
||||
`inspect_analysis_view` projects a caller-selected view of already completed
|
||||
`inspect_binary_layout`, `analyze_javascript_application`, or `analyze_function`
|
||||
Evidence. Native views select procedure, pseudocode, assembly, basic blocks,
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
# Implementation plan
|
||||
|
||||
1. Move response-body decoding from the environment source into
|
||||
`recordedResponseBody.ts` with the caller's byte limit, and exact JSON
|
||||
parsing into `exactJsonNumber.ts`, so replay and state extraction share one
|
||||
owner each.
|
||||
2. Add `environmentStateRules.ts` as pure helpers: write classification, API
|
||||
bases and roots, entity location, identity names, JSON pointers of depth one,
|
||||
value bytes and the entity and list renderers.
|
||||
3. Add `EnvironmentStateExtraction.ts` for planning item and list tools,
|
||||
observing entities and assembling the result, `environmentStateMerge.ts` for
|
||||
merging observations, and `environmentStateGate.ts` for reproducing each
|
||||
plan's recordings; every body is parsed through a budget callback and
|
||||
released before the next.
|
||||
4. Add `EnvironmentStateAccumulator.ts`, extending the manifest accumulator so
|
||||
grouping is shared, retaining record lines and references to the admitted
|
||||
records, and charging bodies and kept values.
|
||||
5. Add `environmentStateView.ts` with the request schema (sharing the
|
||||
manifest's `template_rules` schema), the result fields and the projector;
|
||||
share capture admission and record grouping with the manifest view; register
|
||||
the kind and pass accumulators to `projectAnalysisView` as one object;
|
||||
always seal the result inferred.
|
||||
6. Cover domain, service, MCP, CLI, catalog and acceptance behavior; document
|
||||
the view in the capture guide, the agent environments guide, the contracts
|
||||
reference and the authored skill.
|
||||
|
||||
No new tool, provider, side effect or network activity is introduced. The state
|
||||
is a projection of already retained Evidence; serving it is deferred.
|
||||
@@ -0,0 +1,66 @@
|
||||
# Environment state view from captures
|
||||
|
||||
Issue: https://github.com/morluto/rea/issues/4933
|
||||
|
||||
## Outcome
|
||||
|
||||
`inspect_analysis_view` accepts `view: {"kind": "environment-state"}` for the
|
||||
parents the environment manifest accepts and extracts the initial state the
|
||||
capture's reads observed: entity types with an identity field, entities with
|
||||
their recorded values, documents for written paths whose reads are not
|
||||
entities, and a render plan per read tool that reproduces the tool's latest
|
||||
initial recordings from the state, or a recorded view with the reason it
|
||||
cannot. This increment adds the view kind only; serving from the state, loading
|
||||
another state through the AgentEnv data plane and modelling write effects are
|
||||
later increments.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Accept `inspect_web_network_capture` HAR parents and
|
||||
`capture_browser_scenario` parents that selected response bodies, with the
|
||||
manifest's admission: the same format, subject and sensitive-value refusals.
|
||||
- Group records exactly as the manifest does for the same `template_rules`, so
|
||||
tool names match the manifest the state accompanies.
|
||||
- Order primary-origin records by recorded start time, or by record order when
|
||||
any start time is missing, and say which. Bound each type's state by the first
|
||||
write under its root (API base, version segments, one segment); a query
|
||||
`POST` is not a write, and a read overlapping the write is left out.
|
||||
- Prove an item type's identity field from the requested path value its
|
||||
responses repeat; otherwise infer it from a fixed list of identity names that
|
||||
every listed item carries distinctly. Never treat an envelope that is itself
|
||||
an entity as a list.
|
||||
- Merge observations into one entity per collection and identity: item views'
|
||||
latest values win over list views', differing views keep their own values,
|
||||
and a value that changed with no related write is volatile and keeps the
|
||||
latest.
|
||||
- Keep an item or list plan only when it reproduces each request line's latest
|
||||
initial recording from the state, comparing JSON values without regard to
|
||||
member order; otherwise record the tool with a reason. Parse bodies only when
|
||||
JSON-like, UTF-8, at most 256 KiB, exactly representable and free of
|
||||
`__proto__` members.
|
||||
- Mark the Evidence inferred; state unknowns for authorization, response
|
||||
semantics and write effects.
|
||||
- Charge the manifest's admission and grouping, every record line, every body
|
||||
as temporary bytes before decoding and parsing, and every kept value, within
|
||||
the existing 96 MiB budget.
|
||||
- Read response bodies through one domain owner shared with replay, and parse
|
||||
exact JSON through the owner of the exact-number policy.
|
||||
|
||||
## Acceptance and limits
|
||||
|
||||
The Grafana-shaped HAR fixture with distinct start times drives domain,
|
||||
service, MCP, CLI and acceptance coverage: folders merge their detail and list
|
||||
views under a proved `uid`, the dashboard entity is read from its `/dashboard`
|
||||
wrapper, and the annotation list after the annotation write has no initial
|
||||
recording. Synthetic captures cover write bounds, volatile fields, complete
|
||||
lists with counters, each recorded reason and documents; the scenario fixture
|
||||
covers record order and the response-body requirement. The view performs no
|
||||
network activity and serves nothing. Values copied outside entity objects,
|
||||
derived values and cross-type effects are not tied to the state.
|
||||
|
||||
## Deferred issue criteria
|
||||
|
||||
Serving tool calls from the state, `data/reset`, `data/add` and `data/get` on
|
||||
the environment data plane, exporting the state with a bundle, and write
|
||||
effects are later increments of #4933. This PR must reference rather than
|
||||
close the issue.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Tasks
|
||||
|
||||
- [x] Record the view contract, requirements and deferred increments.
|
||||
- [x] Move body decoding and exact JSON parsing to their domain owners without
|
||||
changing replay.
|
||||
- [x] Implement the extraction rules, merge, gate and accumulator.
|
||||
- [x] Register the `environment-state` kind, result schema, service wiring and
|
||||
confidence rule.
|
||||
- [x] Add the timed Grafana-shaped fixture and synthetic captures.
|
||||
- [x] Add domain, service, MCP, CLI, catalog and acceptance coverage.
|
||||
- [x] Document the view in the guides, contracts reference and skill.
|
||||
- [x] Run static, formatting, duplication, documentation and test gates.
|
||||
@@ -0,0 +1,39 @@
|
||||
# Verification
|
||||
|
||||
Verified base: bf1eeee9 (fetched main).
|
||||
|
||||
Executed on the implementation worktree after `npm ci` and `npm run
|
||||
build:cached`; every check below ran as written in this pass, after the last
|
||||
source change.
|
||||
|
||||
- `npm run check:fast`: typecheck and lint passed.
|
||||
- `npm run format:check`: passed.
|
||||
- `npx knip`: passed (exit 0).
|
||||
- `npm run jscpd`: 3.40% duplicated lines over 3331 files, under the 3.5%
|
||||
gate; no clone involves a file this change adds.
|
||||
- `npm run scan:todos`: passed.
|
||||
- `npm run docs:check`: the built documentation checks and the README
|
||||
consistency conformance test (5 tests) passed.
|
||||
- `npx vitest run --project domain --project services`: 338 files, 5830 tests
|
||||
passed, 5 pre-existing skips.
|
||||
- `npm run test:focused --` the analysis-view MCP and CLI boundaries, the
|
||||
environment serve and export CLI boundaries, the environment serve MCP
|
||||
boundary, the compact input-schema MCP budget, the analysis-detail and
|
||||
advertised-input-example contracts and the qlog acceptance test: 9 files, 68
|
||||
tests passed, 1 pre-existing skip.
|
||||
|
||||
Real captures (outside the test suite): the built view ran through the
|
||||
application service on 44 recorded captures (39 HAR, 5 browser scenario) with
|
||||
all three template rules, in 7 to 46 ms each. 42 produced a state: 568 types,
|
||||
2,213 entities, 2 documents, and 710 of 1,258 read tools state-backed (211
|
||||
item, 497 list, 2 document views). The other read tools are recorded views:
|
||||
306 not entities (singletons, configuration and computed views), 129 without
|
||||
an identity field, 32 with unusable bodies (most are Nomad bodies with
|
||||
integers beyond exact native conversion), 31 lists and 7 items that do not
|
||||
reproduce, 26 without an initial recording, 12 query `POST`s and 5 documents
|
||||
read under several queries. The two browser-scenario captures that did not
|
||||
produce a state are refused inline with `resource_constraint`, as the
|
||||
environment manifest view refuses them on unchanged main.
|
||||
|
||||
Not covered here: serving tool calls from the state, the AgentEnv data plane
|
||||
and write effects, which are later increments.
|
||||
@@ -419,6 +419,93 @@ recorded body is charged as temporary bytes before it is parsed. Retained state
|
||||
temporary work must fit together within the 96 MiB projection representation
|
||||
budget; exceeding it returns `resource_constraint` without a partial manifest.
|
||||
|
||||
## Environment state
|
||||
|
||||
Pass the same parent to `inspect_analysis_view` with
|
||||
`view: {"kind":"environment-state"}` to extract the initial state the
|
||||
capture's reads observed, and a render plan per read tool that reproduces those
|
||||
reads from it. Where the environment manifest names an application's tools, the
|
||||
state holds its data: entities by type and identity, plus documents for written
|
||||
paths whose reads are not entities. Use the `template_rules` of the manifest the
|
||||
state accompanies so tool names match; the view groups records the same way.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "inspect_analysis_view",
|
||||
"arguments": {
|
||||
"source": { "kind": "retained-evidence", "evidence_id": "ev_..." },
|
||||
"view": {
|
||||
"kind": "environment-state",
|
||||
"template_rules": [
|
||||
"after-uid-literal",
|
||||
"opaque-token",
|
||||
"sibling-variance"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Records on the primary origin are ordered by recorded start time, or by record
|
||||
order when any start time is missing (`timeline`), as for a
|
||||
`capture_browser_scenario` parent. A `POST`, `PUT`, `PATCH` or `DELETE` with a
|
||||
2xx or 3xx status, or no status, may change state, unless it is a `POST` whose
|
||||
path has a segment such as `search`, `query`, `list`, `find`, `fuzzy`,
|
||||
`count` or `stats`, which queries state instead. A type's state comes from its
|
||||
successful reads that finished before the first such write under the same root:
|
||||
the path's API base (the literal segments every tool sharing its first segment
|
||||
has), any `v1`-style version segments, then one more segment. A read that
|
||||
overlaps that write in time is left out.
|
||||
|
||||
Each `GET` tool whose template ends in a parameter is an item tool, and its
|
||||
type's `collection_template` is the template without that parameter; any other
|
||||
`GET` tool is a list tool of its own template. An item tool proves its type's
|
||||
`id_field` when every initial response carries the requested path value in the
|
||||
same field, at the root or in one wrapper member (`id_basis: "path-parameter"`).
|
||||
Otherwise the field is the first of `id`, `uid`, `uuid`, `_id`, `ID`, `UID`,
|
||||
`Id`, `guid`, `key`, `slug` and `name` that every listed item carries as a
|
||||
distinct string or number (`id_basis: "field-name"`). A list is the response
|
||||
array, or the one array of objects in an envelope that is not itself an entity.
|
||||
|
||||
`entities` merge every initial observation of one identity within one
|
||||
collection path: an item view's latest values win over a list view's, a view
|
||||
whose latest values differ keeps them in `view_fields`, and a field whose value
|
||||
changed between one tool's observations with no related write between is named
|
||||
in the type's `volatile_fields`, keeping the latest value. Entities are ordered
|
||||
by type, collection and first sighting; that order is a list's state order.
|
||||
Every entity cites its `record_ordinals`.
|
||||
|
||||
`views` lists one plan per read tool in tool order:
|
||||
|
||||
| Shape | Plan |
|
||||
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `item` | The entity object at `pointer` (the root or one member), the ordered `fields` the view shows and the `required_fields` every recording had, and `envelope_ordinal`: the recording whose response, with the entity replaced, every item response shares |
|
||||
| `list` | The array at `pointer`, its `fields`, `membership` (`all`: every entity of the collection in state order, when each recording listed exactly that and had no query; `recorded`: the entities each recording listed) and `counters` that equal the number listed |
|
||||
| `document` | The path's latest initial value, held in `documents` |
|
||||
| `recorded` | A `reason` the tool replays its recordings instead |
|
||||
|
||||
An item or list plan is kept only when it reproduces the latest initial
|
||||
recording of each request line from the state, comparing JSON values without
|
||||
regard to member order; a volatile field compares its recorded value. Each
|
||||
plan lists the `checked_record_ordinals`. Otherwise the tool is a recorded view
|
||||
with reason `envelope-varies`, `fields-differ` or `list-differs`; the other
|
||||
reasons are `query-body` (a query `POST`), `no-init-recording`,
|
||||
`body-unusable`, `not-entity`, `no-identity` and `query-varies` (a document
|
||||
read under several queries). A body is usable when its content type is
|
||||
JSON-like, it is UTF-8 text of at most 256 KiB, every number survives native
|
||||
conversion exactly, and no member is named `__proto__`.
|
||||
|
||||
Values in the state are recorded data, so the view refuses the same parents as
|
||||
the manifest, including those inspected with `sensitive_values`, and a
|
||||
`capture_browser_scenario` parent must have selected response bodies. Types and
|
||||
identity are inferences, so the Evidence is always inferred. `unknowns` states
|
||||
that the state establishes no authorization model, no way the application
|
||||
computes responses or derived values, and no effects of writes. The state
|
||||
accumulator charges the manifest's admission and grouping, every record line,
|
||||
each body as temporary bytes before it is decoded and parsed, and every value
|
||||
it keeps, within the same 96 MiB budget. The view performs no network activity
|
||||
and serves nothing.
|
||||
|
||||
## Resource and lifecycle boundaries
|
||||
|
||||
Complete evidence has a 32 MiB input, 96 MiB decoder reply and 64-level nesting
|
||||
|
||||
@@ -448,7 +448,7 @@ export const SCOPES = {
|
||||
"src/server/http/**",
|
||||
"src/{cli,cliCommandNames,cli/environmentCommands}.ts",
|
||||
"src/config/{environmentName,mcpHttpEndpoint}.ts",
|
||||
"src/domain/analysisView/{environmentManifest*,EnvironmentManifest*,scenarioCaptureRecords,WebNetworkProjectionAccumulator,analysisViewDataAdmission}.ts",
|
||||
"src/domain/analysisView/{environmentManifest*,EnvironmentManifest*,environmentState*,EnvironmentState*,recordedResponseBody,scenarioCaptureRecords,WebNetworkProjectionAccumulator,analysisViewDataAdmission}.ts",
|
||||
"src/process/{OwnedCommand,ProviderProcess,PrivateRuntimeRoot}.ts",
|
||||
"src/process/ProcessOwnership*.ts",
|
||||
"tests/fixtures/historicalHar.ts",
|
||||
|
||||
@@ -3,6 +3,7 @@ import {
|
||||
analysisViewCaptureEvidence,
|
||||
analysisViewObjectDenseCapture,
|
||||
environmentManifestCaptureFixture,
|
||||
environmentStateCaptureFixture,
|
||||
} from "../../../tests/fixtures/analysisView.js";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
@@ -450,6 +451,7 @@ it("refuses an object-dense borrowed capture before making a full schema clone",
|
||||
for (const view of [
|
||||
{ kind: "api-surface", limit: 1 },
|
||||
{ kind: "environment-manifest" },
|
||||
{ kind: "environment-state" },
|
||||
] as const) {
|
||||
const result = inspectAnalysisView(
|
||||
{
|
||||
@@ -522,42 +524,75 @@ it("seals an inferred environment manifest and a derived manifest without templa
|
||||
});
|
||||
});
|
||||
|
||||
it.each(["summary", "capture", "api-surface", "environment-manifest"] as const)(
|
||||
"rejects inline accessors before parsing a %s view",
|
||||
(kind) => {
|
||||
let accessed = false;
|
||||
const evidence = {
|
||||
get normalized_result() {
|
||||
accessed = true;
|
||||
throw new Error("Evidence must be admitted before parsing");
|
||||
},
|
||||
};
|
||||
const input = {
|
||||
source: { kind: "inline", evidence },
|
||||
view: { kind },
|
||||
};
|
||||
const parsed = inspectAnalysisViewInputSchema.parse(input);
|
||||
if (parsed.source.kind !== "inline")
|
||||
throw new Error("Expected inline input");
|
||||
expect(parsed.source.evidence).toBe(evidence);
|
||||
expect(inspectAnalysisView(input)).toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
_tag: "AnalysisInputError",
|
||||
issues: [{ path: ["source", "evidence", "normalized_result"] }],
|
||||
},
|
||||
});
|
||||
expect(accessed).toBe(false);
|
||||
expect(
|
||||
inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: analysisViewCaptureEvidence() },
|
||||
view: { kind },
|
||||
}).ok,
|
||||
).toBe(true);
|
||||
},
|
||||
);
|
||||
it("seals an inferred environment state whose values are the recorded ones", () => {
|
||||
const parent = analysisViewCaptureEvidence(environmentStateCaptureFixture());
|
||||
const sealed = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: parent },
|
||||
view: { kind: "environment-state", template_rules: ["opaque-token"] },
|
||||
});
|
||||
if (!sealed.ok) throw sealed.error;
|
||||
expect(sealed.value.confidence).toBe("inferred");
|
||||
expect(sealed.value.authority).toBe(parent.authority);
|
||||
expect(sealed.value.evidence_links).toEqual([parent.evidence_id]);
|
||||
expect(sealed.value.parameters).toMatchObject({
|
||||
view: { kind: "environment-state", template_rules: ["opaque-token"] },
|
||||
});
|
||||
expect(sealed.value.normalized_result).toMatchObject({
|
||||
kind: "environment-state",
|
||||
name: "grafana-example",
|
||||
parent_evidence_id: parent.evidence_id,
|
||||
timeline: "recorded-start-time",
|
||||
});
|
||||
expect(JSON.stringify(sealed.value.normalized_result)).toContain(
|
||||
"probe folder",
|
||||
);
|
||||
expect(parseEvidence(sealed.value)).toEqual(sealed.value);
|
||||
});
|
||||
|
||||
it.each(["summary", "capture", "api-surface", "environment-manifest"] as const)(
|
||||
it.each([
|
||||
"summary",
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
"environment-state",
|
||||
] as const)("rejects inline accessors before parsing a %s view", (kind) => {
|
||||
let accessed = false;
|
||||
const evidence = {
|
||||
get normalized_result() {
|
||||
accessed = true;
|
||||
throw new Error("Evidence must be admitted before parsing");
|
||||
},
|
||||
};
|
||||
const input = {
|
||||
source: { kind: "inline", evidence },
|
||||
view: { kind },
|
||||
};
|
||||
const parsed = inspectAnalysisViewInputSchema.parse(input);
|
||||
if (parsed.source.kind !== "inline") throw new Error("Expected inline input");
|
||||
expect(parsed.source.evidence).toBe(evidence);
|
||||
expect(inspectAnalysisView(input)).toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
_tag: "AnalysisInputError",
|
||||
issues: [{ path: ["source", "evidence", "normalized_result"] }],
|
||||
},
|
||||
});
|
||||
expect(accessed).toBe(false);
|
||||
expect(
|
||||
inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: analysisViewCaptureEvidence() },
|
||||
view: { kind },
|
||||
}).ok,
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it.each([
|
||||
"summary",
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
"environment-state",
|
||||
] as const)(
|
||||
"rejects nested, hidden and inherited inline accessors for %s",
|
||||
(kind) => {
|
||||
const parent = analysisViewCaptureEvidence();
|
||||
@@ -639,6 +674,7 @@ it.each([
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
"environment-state",
|
||||
] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: { ...parent, raw_result: array } },
|
||||
@@ -675,6 +711,7 @@ it("rejects callable array overrides and subclass species before parsing", () =>
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
"environment-state",
|
||||
] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: { ...parent, raw_result: array } },
|
||||
@@ -688,26 +725,29 @@ it("rejects callable array overrides and subclass species before parsing", () =>
|
||||
expect(invoked).toBe(false);
|
||||
});
|
||||
|
||||
it.each(["summary", "capture", "api-surface", "environment-manifest"] as const)(
|
||||
"preserves ordinary inline Evidence for %s",
|
||||
(kind) => {
|
||||
const parent = analysisViewCaptureEvidence();
|
||||
const before = JSON.stringify(parent);
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: parent },
|
||||
view: { kind },
|
||||
});
|
||||
if (!result.ok) throw result.error;
|
||||
expect(parseEvidence(result.value)).toEqual(result.value);
|
||||
expect(result.value.evidence_links).toEqual([parent.evidence_id]);
|
||||
expect(result.value.normalized_result).toMatchObject({
|
||||
kind,
|
||||
parent_evidence_id: parent.evidence_id,
|
||||
});
|
||||
expect(JSON.stringify(parent)).toBe(before);
|
||||
expect(Object.isFrozen(parent)).toBe(false);
|
||||
},
|
||||
);
|
||||
it.each([
|
||||
"summary",
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
"environment-state",
|
||||
] as const)("preserves ordinary inline Evidence for %s", (kind) => {
|
||||
const parent = analysisViewCaptureEvidence();
|
||||
const before = JSON.stringify(parent);
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: parent },
|
||||
view: { kind },
|
||||
});
|
||||
if (!result.ok) throw result.error;
|
||||
expect(parseEvidence(result.value)).toEqual(result.value);
|
||||
expect(result.value.evidence_links).toEqual([parent.evidence_id]);
|
||||
expect(result.value.normalized_result).toMatchObject({
|
||||
kind,
|
||||
parent_evidence_id: parent.evidence_id,
|
||||
});
|
||||
expect(JSON.stringify(parent)).toBe(before);
|
||||
expect(Object.isFrozen(parent)).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects executable constructor values before schema plain-object checks", () => {
|
||||
const parent = analysisViewCaptureEvidence();
|
||||
@@ -724,6 +764,7 @@ it("rejects executable constructor values before schema plain-object checks", ()
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
"environment-state",
|
||||
] as const) {
|
||||
expect(
|
||||
inspectAnalysisView({
|
||||
@@ -757,6 +798,7 @@ it("rejects inline proxy values before reflection or parsing can run traps", ()
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
"environment-state",
|
||||
] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: { ...parent, raw_result: value } },
|
||||
@@ -801,6 +843,7 @@ it("keeps ignored array metadata inert through parsing and projection", () => {
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
"environment-state",
|
||||
] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: parent },
|
||||
@@ -848,7 +891,11 @@ it("admits raw_result and parameters before authenticating a portable capture",
|
||||
const dense = analysisViewObjectDenseCapture();
|
||||
const parent = analysisViewCaptureEvidence();
|
||||
for (const field of ["raw_result", "parameters"] as const)
|
||||
for (const kind of ["api-surface", "environment-manifest"] as const) {
|
||||
for (const kind of [
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
"environment-state",
|
||||
] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: { ...parent, [field]: dense } },
|
||||
view: { kind },
|
||||
|
||||
@@ -28,6 +28,7 @@ import {
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import { WebNetworkApiSurfaceAccumulator } from "../../domain/analysisView/WebNetworkApiSurfaceAccumulator.js";
|
||||
import { EnvironmentManifestAccumulator } from "../../domain/analysisView/EnvironmentManifestAccumulator.js";
|
||||
import { EnvironmentStateAccumulator } from "../../domain/analysisView/EnvironmentStateAccumulator.js";
|
||||
import { admitAnalysisViewData } from "../../domain/analysisView/analysisViewDataAdmission.js";
|
||||
|
||||
/** Authenticate a selected-view request and project already completed Evidence. */
|
||||
@@ -68,7 +69,12 @@ export const inspectAnalysisViewValidated = (
|
||||
input.view.template_rules,
|
||||
)
|
||||
: undefined;
|
||||
const accumulator = apiSurfaceAccumulator ?? manifestAccumulator;
|
||||
const stateAccumulator =
|
||||
input.view.kind === "environment-state"
|
||||
? new EnvironmentStateAccumulator(undefined, input.view.template_rules)
|
||||
: undefined;
|
||||
const accumulator =
|
||||
apiSurfaceAccumulator ?? manifestAccumulator ?? stateAccumulator;
|
||||
if (input.source.kind === "inline") {
|
||||
const admitted =
|
||||
accumulator === undefined
|
||||
@@ -100,8 +106,11 @@ export const inspectAnalysisViewValidated = (
|
||||
},
|
||||
},
|
||||
input.view,
|
||||
apiSurfaceAccumulator,
|
||||
manifestAccumulator,
|
||||
{
|
||||
apiSurface: apiSurfaceAccumulator,
|
||||
manifest: manifestAccumulator,
|
||||
state: stateAccumulator,
|
||||
},
|
||||
);
|
||||
if (!projected.ok) return projected;
|
||||
if (
|
||||
@@ -234,10 +243,12 @@ const createAnalysisViewEvidence = (
|
||||
result: jsonValueSchema.parse(result),
|
||||
rawResult: null,
|
||||
confidence:
|
||||
// Grouping is whole-capture, so any templated group or parameter makes the result inferred.
|
||||
(result.kind === "api-surface" ||
|
||||
// Grouping is whole-capture, so any templated group or parameter makes the result inferred;
|
||||
// entity types and identity in an extracted state always are.
|
||||
result.kind === "environment-state" ||
|
||||
((result.kind === "api-surface" ||
|
||||
result.kind === "environment-manifest") &&
|
||||
result.templated_groups > 0
|
||||
result.templated_groups > 0)
|
||||
? "inferred"
|
||||
: "derived",
|
||||
authority: parent.authority,
|
||||
|
||||
@@ -2,13 +2,12 @@ import { isJsonLike } from "../../domain/analysisView/EnvironmentManifestAccumul
|
||||
import { mediaTypeEssence } from "../../domain/analysisView/webNetworkRouteTemplates.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { AnalysisInputError } from "../../domain/analysisErrorCore.js";
|
||||
import { isExactJsonNumber } from "../../domain/exactJsonNumber.js";
|
||||
import { jsonValueSchema, type JsonValue } from "../../domain/jsonValue.js";
|
||||
import { parseExactJson } from "../../domain/exactJsonNumber.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import {
|
||||
iterateUrlQueryParameters,
|
||||
recordedPathSegments,
|
||||
} from "../../domain/recordedTransportUrl.js";
|
||||
import { safeParseJson } from "../../domain/safeJson.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import { environmentToolArguments } from "./environmentTools.js";
|
||||
import {
|
||||
@@ -190,22 +189,8 @@ const replayBody = (
|
||||
exchange.contentType !== null &&
|
||||
isJsonLike(mediaTypeEssence(exchange.contentType))
|
||||
) {
|
||||
const parsed = safeParseJson(text, (_key, value, context) => {
|
||||
// The native parser supplies each primitive's original spelling. Share
|
||||
// HAR's exact-number policy rather than accepting an already-rounded ID.
|
||||
if (
|
||||
typeof value === "number" &&
|
||||
!isExactJsonNumber(value, context?.source)
|
||||
)
|
||||
throw new RangeError(
|
||||
"JSON numeric conversion would lose recorded data",
|
||||
);
|
||||
return value;
|
||||
});
|
||||
const value = parsed.ok
|
||||
? jsonValueSchema.safeParse(parsed.value)
|
||||
: undefined;
|
||||
if (value?.success === true) return { body: value.data, truncated: false };
|
||||
const value = parseExactJson(text);
|
||||
if (value !== undefined) return { body: value, truncated: false };
|
||||
}
|
||||
return { body: text, truncated: false };
|
||||
};
|
||||
|
||||
@@ -69,7 +69,7 @@ describe("environment manifests from capture_browser_scenario parents", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("names the one view a scenario capture supports when another is requested", () => {
|
||||
it("names the views a scenario capture supports when another is requested", () => {
|
||||
const refused = inspectAnalysisView({
|
||||
source: {
|
||||
kind: "inline",
|
||||
@@ -81,7 +81,7 @@ describe("environment manifests from capture_browser_scenario parents", () => {
|
||||
expect(projectAnalysisError(refused.error)).toMatchObject({
|
||||
code: "unsupported_target",
|
||||
message: expect.stringContaining(
|
||||
"supports only the environment-manifest view",
|
||||
"supports only the environment-manifest and environment-state views",
|
||||
),
|
||||
});
|
||||
});
|
||||
|
||||
@@ -5,6 +5,10 @@ import {
|
||||
type AnalysisViewResult,
|
||||
} from "../../domain/analysisView/analysisView.js";
|
||||
import { scenarioCaptureRecords } from "../../domain/analysisView/scenarioCaptureRecords.js";
|
||||
import {
|
||||
readRecordedResponseBody,
|
||||
type RecordedResponseBody,
|
||||
} from "../../domain/analysisView/recordedResponseBody.js";
|
||||
import { readRecordedRequestLine } from "../../domain/analysisView/webNetworkRouteTemplates.js";
|
||||
import { browserScenarioCaptureSchema } from "../../domain/browserScenarioCapture.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
@@ -30,28 +34,19 @@ export type EnvironmentManifest = Extract<
|
||||
>;
|
||||
export type EnvironmentManifestTool = EnvironmentManifest["tools"][number];
|
||||
|
||||
/** One retained parent record reduced to what a replay returns. */
|
||||
export interface RecordedExchange {
|
||||
/** One retained parent record reduced to what a replay returns; a cut body is reported truncated. */
|
||||
export interface RecordedExchange extends RecordedResponseBody {
|
||||
readonly ordinal: number;
|
||||
readonly path: string;
|
||||
/** WHATWG query spelling including `?`, or null when the request had none. */
|
||||
readonly query: string | null;
|
||||
readonly status: number | null;
|
||||
readonly contentType: string | null;
|
||||
/** Producer-decoded response text, or null when the record retained none. */
|
||||
readonly bodyText: string | null;
|
||||
/** Set when the retained bytes are not UTF-8, so `bodyText` holds their base64 rather than a lossy decoding. */
|
||||
readonly bodyEncoding?: "base64";
|
||||
/** Set when only a prefix of a large base64 body was decoded; replay reports it truncated. */
|
||||
readonly bodyCut?: true;
|
||||
}
|
||||
|
||||
/** Replayed bodies larger than this are cut at a UTF-8 boundary and flagged. */
|
||||
export const REPLAY_BODY_LIMIT_BYTES = 256 * 1024;
|
||||
|
||||
/** Base64 characters that decode to just over the replay limit, so a cut prefix still fills it. */
|
||||
const BASE64_PREFIX_CHARS = Math.ceil((REPLAY_BODY_LIMIT_BYTES + 4) / 3) * 4;
|
||||
|
||||
/** A HAR inspected by inspect_web_network_capture, or a session recorded by capture_browser_scenario. */
|
||||
export type EnvironmentCaptureKind = "web-network-capture" | "browser-scenario";
|
||||
|
||||
@@ -107,52 +102,6 @@ const parseEvidenceFile = (
|
||||
}
|
||||
};
|
||||
|
||||
const contentTextSchema = z.object({
|
||||
response: z.object({
|
||||
content: z.object({
|
||||
text: z.unknown().optional(),
|
||||
encoding: z.unknown().optional(),
|
||||
}),
|
||||
}),
|
||||
});
|
||||
|
||||
/** Strict, and keeping a leading byte-order mark, so replay returns the recorded text unchanged. */
|
||||
const strictUtf8 = (): TextDecoder =>
|
||||
new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
|
||||
|
||||
/** The retained response text of one record, decoding HAR base64 content when the decoder kept it as valid UTF-8. */
|
||||
const recordedBody = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
): Pick<RecordedExchange, "bodyText" | "bodyEncoding"> => {
|
||||
const content = contentTextSchema.safeParse(record.reported);
|
||||
if (!content.success) return { bodyText: null };
|
||||
const { text, encoding } = content.data.response.content;
|
||||
if (typeof text !== "string") return { bodyText: null };
|
||||
if (encoding !== "base64") return { bodyText: text };
|
||||
const binary = record.binary_fields.find(
|
||||
(field) =>
|
||||
field.pointer === "/response/content/text" && field.state === "retained",
|
||||
);
|
||||
if (binary === undefined || binary.content_base64 === null)
|
||||
return { bodyText: null };
|
||||
// Replay returns at most REPLAY_BODY_LIMIT_BYTES, so a large body is decoded only that far.
|
||||
const cut = binary.content_base64.length > BASE64_PREFIX_CHARS;
|
||||
const encoded = cut
|
||||
? binary.content_base64.slice(0, BASE64_PREFIX_CHARS)
|
||||
: binary.content_base64;
|
||||
const flags = cut ? { bodyCut: true as const } : {};
|
||||
try {
|
||||
// A cut prefix may end inside a character; streaming leaves that tail undecoded.
|
||||
const text = strictUtf8().decode(Buffer.from(encoded, "base64"), {
|
||||
stream: cut,
|
||||
});
|
||||
return { bodyText: text, ...flags };
|
||||
} catch {
|
||||
// Binary or non-UTF-8 bytes: keep them exactly, as base64, instead of inventing replacement characters.
|
||||
return { bodyText: encoded, bodyEncoding: "base64", ...flags };
|
||||
}
|
||||
};
|
||||
|
||||
const recordedExchange = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
): RecordedExchange | undefined => {
|
||||
@@ -164,7 +113,7 @@ const recordedExchange = (
|
||||
query: line.query === undefined || line.query === "" ? null : line.query,
|
||||
status: line.status,
|
||||
contentType: line.rawMime === "" ? null : line.rawMime,
|
||||
...recordedBody(record),
|
||||
...readRecordedResponseBody(record, REPLAY_BODY_LIMIT_BYTES),
|
||||
};
|
||||
};
|
||||
|
||||
|
||||
@@ -38,6 +38,7 @@ it("advertises only that operation's summary from analysis detail outputs", () =
|
||||
);
|
||||
expect(schema).not.toContain("api-surface");
|
||||
expect(schema).not.toContain("environment-manifest");
|
||||
expect(schema).not.toContain("environment-state");
|
||||
expect(schema).not.toContain("inspect_web_network_capture");
|
||||
}
|
||||
const viewSchema = JSON.stringify(
|
||||
@@ -45,6 +46,7 @@ it("advertises only that operation's summary from analysis detail outputs", () =
|
||||
);
|
||||
expect(viewSchema).toContain("api-surface");
|
||||
expect(viewSchema).toContain("environment-manifest");
|
||||
expect(viewSchema).toContain("environment-state");
|
||||
|
||||
const layout = projectSummary(analysisViewLayoutEvidence());
|
||||
const javascript = projectSummary(analysisViewJavaScriptEvidence());
|
||||
|
||||
@@ -17,7 +17,7 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [
|
||||
...toolContractMetadata("inspect_analysis_view"),
|
||||
kind: "application",
|
||||
description:
|
||||
'Inspect selected views of completed binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, decode_interface_builder, inspect_evm_interface, inspect_managed_artifact, inspect_managed_members, inspect_android_package, inspect_dart_aot or inspect_web_network_capture Evidence without repeating analysis. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. For a retained record use source: {kind: "retained-evidence", evidence_id: "<parent_evidence_id>"}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: <complete Evidence>}. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Typed module references use view {kind: "javascript", collection: "modules", projection: "references", where: {path: "main.js", kind: "javascript-module"}, offset: 0, limit: 64}; where matches exact retained strings (null selects unavailable paths). Rows include canonical identity, retained source-range count and observation support states. Source coverage and selection remain explicit; exhausting matching rows does not prove runtime loading or complete analysis. For exact file/native facts and processing gaps, use view {kind: "javascript-facts", where: {paths: ["package.json"]}, property_keys: ["name", "operation", "analysis_status"]}. Rows page original observations with canonical identities and Evidence; missing properties remain omitted. Paths match recorded artifact-path/source-range locations; optional kinds filter nodes. To discover field names without values, opt into include_available_property_keys: true with property_keys: []. Canonical identity and Evidence are separate from properties; keep each node_id/observation_id pair together when composing results. Inventory uses view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json"], offset: 0, limit: 64}. Inventory views page nodes, occurrences, edges or integrity contradictions; occurrence logical_paths matches exact recordeLine truncated
|
||||
'Inspect selected views of completed binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, decode_interface_builder, inspect_evm_interface, inspect_managed_artifact, inspect_managed_members, inspect_android_package, inspect_dart_aot or inspect_web_network_capture Evidence without repeating analysis. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for written non-entity paths, and a render plan per read tool: item and list views reproduce their latest recordings from the state, and every other read tool is a recorded view with a reason. Types and identity are inferences; values are recorded data, so it refuses parents inspected with sensitive values, and a scenario parent must have selected response bodies. For a retained record use source: {kind: "retained-evidence", evidence_id: "<parent_evidence_id>"}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: <complete Evidence>}. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Typed module references use view {kind: "javascript", collection: "modules", projection: "references", where: {path: "main.js", kind: "javascript-module"}, offset: 0, limit: 64}; where matches exact retained strings (null selects unavailable paths). Rows include canonical identity, retained source-range count and observation support states. Source coverage and selection remain explicit; exhausting matching rows does not prove runtime loading or complete analysis. For exact file/native facts and
Line truncated
|
||||
inputSchema: inspectAnalysisViewInputSchema,
|
||||
outputSchema: evidenceResultOf(analysisViewResultSchema),
|
||||
examples: [
|
||||
|
||||
@@ -195,11 +195,17 @@ export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumula
|
||||
maximumBytes = WEB_NETWORK_CAPTURE_LIMITS.outputBytes,
|
||||
readonly includeQueryParameterNames = true,
|
||||
rules: readonly TemplateRule[] = [],
|
||||
budgetSubject = "Environment-manifest",
|
||||
) {
|
||||
super(maximumBytes, "Environment-manifest");
|
||||
super(maximumBytes, budgetSubject);
|
||||
this.#rules = new Set(rules);
|
||||
}
|
||||
|
||||
/** Whether the finished projection publishes body shapes, and so states their limitation. */
|
||||
protected get reportsBodyShapes(): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
add(record: WebNetworkCaptureRecord): Result<void, AnalysisError> {
|
||||
const limitations = this.addLimitations(record.limitations);
|
||||
if (!limitations.ok) return limitations;
|
||||
@@ -484,7 +490,8 @@ export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumula
|
||||
compare(left.reason, right.reason),
|
||||
);
|
||||
const limitations = this.addLimitations([
|
||||
...(tools.some((tool) => tool.body_shape !== null)
|
||||
...(this.reportsBodyShapes &&
|
||||
tools.some((tool) => tool.body_shape !== null)
|
||||
? [
|
||||
"Body shapes list top-level JSON key types from one recorded request each; they do not establish required fields or a complete schema.",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
import { z } from "zod";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { parseExactJson } from "../exactJsonNumber.js";
|
||||
import { ok, type Result } from "../result.js";
|
||||
import {
|
||||
WEB_NETWORK_CAPTURE_LIMITS,
|
||||
type WebNetworkCaptureRecord,
|
||||
} from "../webNetworkCapture.js";
|
||||
import type { UnsignedAnalysisView } from "./analysisView.js";
|
||||
import {
|
||||
EnvironmentManifestAccumulator,
|
||||
isJsonLike,
|
||||
} from "./EnvironmentManifestAccumulator.js";
|
||||
import { EnvironmentStateExtraction } from "./EnvironmentStateExtraction.js";
|
||||
import type { TemplateRule } from "./environmentManifestRules.js";
|
||||
import type {
|
||||
ParsedBody,
|
||||
StateRecordFacts,
|
||||
StateTimeline,
|
||||
} from "./environmentStateRules.js";
|
||||
import {
|
||||
base64PrefixChars,
|
||||
readRecordedResponseBody,
|
||||
} from "./recordedResponseBody.js";
|
||||
import {
|
||||
ENTRY_BYTES,
|
||||
RECORD_SCRATCH_BYTES,
|
||||
mediaTypeEssence,
|
||||
readRecordedRequestLine,
|
||||
textBytes,
|
||||
} from "./webNetworkRouteTemplates.js";
|
||||
|
||||
type StateView = Extract<UnsignedAnalysisView, { kind: "environment-state" }>;
|
||||
type StateFields = Pick<
|
||||
StateView,
|
||||
| "origin"
|
||||
| "name"
|
||||
| "timeline"
|
||||
| "types"
|
||||
| "entities"
|
||||
| "documents"
|
||||
| "views"
|
||||
| "unknowns"
|
||||
| "coverage"
|
||||
| "limitations"
|
||||
>;
|
||||
|
||||
/** Bodies beyond this many UTF-8 bytes are not parsed into state; their views stay recorded. */
|
||||
export const STATE_BODY_LIMIT_BYTES = 256 * 1024;
|
||||
|
||||
const timingSchema = z.object({
|
||||
startedDateTime: z.string(),
|
||||
time: z.number().nonnegative(),
|
||||
});
|
||||
const contentSchema = z.object({
|
||||
response: z.object({ content: z.object({ text: z.string() }) }),
|
||||
});
|
||||
|
||||
interface RecordEntry {
|
||||
readonly facts: StateRecordFacts;
|
||||
readonly origin: string;
|
||||
readonly contentType: string;
|
||||
readonly timed: boolean;
|
||||
readonly record: WebNetworkCaptureRecord;
|
||||
}
|
||||
|
||||
const UNKNOWNS: StateView["unknowns"] = [
|
||||
{
|
||||
kind: "authorization-model",
|
||||
statement:
|
||||
"Authorization headers and cookies are structurally excluded from captures; the state states no authorization model.",
|
||||
},
|
||||
{
|
||||
kind: "response-semantics",
|
||||
statement:
|
||||
"Views reproduce their recorded responses from the state; they do not establish how the application computes responses, derived values or values copied outside entity objects.",
|
||||
},
|
||||
{
|
||||
kind: "state-transition",
|
||||
statement:
|
||||
"The state is one snapshot from before the first related recorded write; it states no effects of writes.",
|
||||
},
|
||||
];
|
||||
|
||||
/** The manifest's grouping plus the record facts and bodies the init state is extracted from. */
|
||||
export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator {
|
||||
readonly #entries = new Map<number, RecordEntry>();
|
||||
|
||||
constructor(
|
||||
maximumBytes = WEB_NETWORK_CAPTURE_LIMITS.outputBytes,
|
||||
rules: readonly TemplateRule[] = [],
|
||||
) {
|
||||
super(maximumBytes, true, rules, "Environment-state");
|
||||
}
|
||||
|
||||
/** The state publishes no body shapes. */
|
||||
protected override get reportsBodyShapes(): boolean {
|
||||
return false;
|
||||
}
|
||||
|
||||
override add(record: WebNetworkCaptureRecord): Result<void, AnalysisError> {
|
||||
const grouped = super.add(record);
|
||||
if (!grouped.ok) return grouped;
|
||||
try {
|
||||
const line = readRecordedRequestLine(record.reported, true, this.scratch);
|
||||
if (line === null) return this.overBudget();
|
||||
if (line.kind === "excluded") return ok(undefined);
|
||||
const timing = timingSchema.safeParse(record.reported);
|
||||
const start = timing.success
|
||||
? Date.parse(timing.data.startedDateTime)
|
||||
: Number.NaN;
|
||||
// The entry keeps the line's strings and a reference to the already admitted record.
|
||||
if (
|
||||
!this.retain(
|
||||
ENTRY_BYTES * 8 +
|
||||
textBytes(
|
||||
line.method.length +
|
||||
line.origin.length +
|
||||
line.path.length +
|
||||
(line.query?.length ?? 0) +
|
||||
line.rawMime.length,
|
||||
),
|
||||
)
|
||||
)
|
||||
return this.overBudget();
|
||||
const timed = timing.success && Number.isFinite(start);
|
||||
this.#entries.set(record.ordinal, {
|
||||
facts: {
|
||||
ordinal: record.ordinal,
|
||||
method: line.method,
|
||||
path: line.path,
|
||||
query: line.query === "" ? undefined : line.query,
|
||||
status: line.status,
|
||||
start: timed ? start : record.ordinal,
|
||||
end: timed ? start + timing.data.time : record.ordinal,
|
||||
},
|
||||
origin: line.origin,
|
||||
contentType: line.rawMime,
|
||||
timed,
|
||||
record,
|
||||
});
|
||||
return ok(undefined);
|
||||
} finally {
|
||||
this.releaseTemporary();
|
||||
}
|
||||
}
|
||||
|
||||
/** Parse one retained response body as exact JSON, charging its decoding and tree as temporary work. */
|
||||
#body(ordinal: number): ParsedBody | null {
|
||||
const entry = this.#entries.get(ordinal);
|
||||
if (entry === undefined || !isJsonLike(mediaTypeEssence(entry.contentType)))
|
||||
return { kind: "unusable" };
|
||||
const content = contentSchema.safeParse(entry.record.reported);
|
||||
if (!content.success) return { kind: "unusable" };
|
||||
// Larger bodies are never parsed; within the limit, decoding copies at most the retained text and the tree is linear in it.
|
||||
const length = content.data.response.content.text.length;
|
||||
if (length > base64PrefixChars(STATE_BODY_LIMIT_BYTES))
|
||||
return { kind: "unusable" };
|
||||
if (!this.temporary(RECORD_SCRATCH_BYTES + textBytes(length) * 4))
|
||||
return null;
|
||||
const body = readRecordedResponseBody(entry.record, STATE_BODY_LIMIT_BYTES);
|
||||
if (
|
||||
body.bodyText === null ||
|
||||
body.bodyEncoding === "base64" ||
|
||||
body.bodyCut === true ||
|
||||
Buffer.byteLength(body.bodyText, "utf8") > STATE_BODY_LIMIT_BYTES
|
||||
)
|
||||
return { kind: "unusable" };
|
||||
const value = parseExactJson(body.bodyText);
|
||||
return value === undefined ? { kind: "unusable" } : { kind: "json", value };
|
||||
}
|
||||
|
||||
/** Project the init state of the manifest's primary origin; records keep timeline order. */
|
||||
finishState(): Result<StateFields, AnalysisError> {
|
||||
const manifest = this.finish();
|
||||
if (!manifest.ok) return manifest;
|
||||
const entries = [...this.#entries.values()].filter(
|
||||
(entry) => entry.origin === manifest.value.origin,
|
||||
);
|
||||
const timeline: StateTimeline =
|
||||
entries.length > 0 && entries.every((entry) => entry.timed)
|
||||
? "recorded-start-time"
|
||||
: "record-order";
|
||||
const records = entries
|
||||
.map((entry) => entry.facts)
|
||||
.sort((left, right) =>
|
||||
timeline === "record-order"
|
||||
? left.ordinal - right.ordinal
|
||||
: left.start - right.start || left.ordinal - right.ordinal,
|
||||
)
|
||||
.map((facts) =>
|
||||
timeline === "record-order"
|
||||
? { ...facts, start: facts.ordinal, end: facts.ordinal }
|
||||
: facts,
|
||||
);
|
||||
const state = new EnvironmentStateExtraction({
|
||||
tools: manifest.value.tools,
|
||||
records,
|
||||
timeline,
|
||||
budget: {
|
||||
body: (ordinal) => this.#body(ordinal),
|
||||
retain: (bytes) => this.retain(bytes),
|
||||
release: () => this.releaseTemporary(),
|
||||
},
|
||||
}).run();
|
||||
if (state === null) return this.overBudget();
|
||||
const limitations = this.addLimitations([
|
||||
...(timeline === "record-order"
|
||||
? [
|
||||
"Some records carry no start time, so the state's timeline is record order and overlapping requests are not detected.",
|
||||
]
|
||||
: []),
|
||||
...(state.views.some((view) => view.shape === "recorded")
|
||||
? [
|
||||
"Tools whose recordings do not reproduce from the extracted state are listed as recorded views with the reason; they replay their recordings.",
|
||||
]
|
||||
: []),
|
||||
]);
|
||||
if (!limitations.ok) return limitations;
|
||||
return ok({
|
||||
origin: manifest.value.origin,
|
||||
name: manifest.value.name,
|
||||
timeline,
|
||||
...state,
|
||||
unknowns: UNKNOWNS.map((unknown) => ({ ...unknown })),
|
||||
coverage: manifest.value.coverage,
|
||||
limitations: this.sortedLimitations(),
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,500 @@
|
||||
import type { JsonValue } from "../jsonValue.js";
|
||||
import {
|
||||
compareUnicodeCodePoints as compare,
|
||||
compositeKey,
|
||||
} from "../unicodeCodePointOrder.js";
|
||||
import {
|
||||
IDENTITY_NAMES,
|
||||
at,
|
||||
collectionPathOf,
|
||||
identifies,
|
||||
isObject,
|
||||
isParam,
|
||||
isQueryPost,
|
||||
isStateWrite,
|
||||
jsonBytes,
|
||||
hasPrototypeKey,
|
||||
lastSegmentValue,
|
||||
listItems,
|
||||
normalizedPath,
|
||||
locateItem,
|
||||
locateList,
|
||||
rootOf,
|
||||
rootsRelated,
|
||||
same,
|
||||
scalarText,
|
||||
segmentsOf,
|
||||
apiBaseLengths,
|
||||
type ExtractedState,
|
||||
type JsonObject,
|
||||
type RecordedViewReason,
|
||||
type StateBudget,
|
||||
type StateDocument,
|
||||
type StateEntity,
|
||||
type StateRecordFacts,
|
||||
type StateTimeline,
|
||||
type StateToolFacts,
|
||||
type StateType,
|
||||
type StateView,
|
||||
} from "./environmentStateRules.js";
|
||||
import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js";
|
||||
import {
|
||||
compareObservedEntities,
|
||||
mergeObservedEntity,
|
||||
type ObservedEntity,
|
||||
} from "./environmentStateMerge.js";
|
||||
import {
|
||||
checkItemPlan,
|
||||
checkListPlan,
|
||||
type StateGateContext,
|
||||
type StatePlan,
|
||||
} from "./environmentStateGate.js";
|
||||
|
||||
class StateBudgetExhausted extends Error {}
|
||||
|
||||
type Candidate = Pick<StatePlan, "tool" | "item" | "type">;
|
||||
|
||||
const entityKey = (type: string, collection: string, id: string): string =>
|
||||
compositeKey([type, collection, id]);
|
||||
|
||||
const requestLine = (record: StateRecordFacts): string =>
|
||||
record.path + (record.query ?? "");
|
||||
|
||||
/** Derive the init state and render plans from manifest tools and timeline-ordered records. */
|
||||
export class EnvironmentStateExtraction {
|
||||
readonly #tools: readonly StateToolFacts[];
|
||||
readonly #timeline: StateTimeline;
|
||||
readonly #budget: StateBudget;
|
||||
readonly #position: ReadonlyMap<number, number>;
|
||||
readonly #byOrdinal: ReadonlyMap<number, StateRecordFacts>;
|
||||
readonly #writes: readonly StateRecordFacts[];
|
||||
readonly #bases: ReadonlyMap<string, number>;
|
||||
readonly #identity = new Map<
|
||||
string,
|
||||
{ readonly field: string; readonly basis: StateType["id_basis"] }
|
||||
>();
|
||||
readonly #views = new Map<string, StateView>();
|
||||
readonly #plans: StatePlan[] = [];
|
||||
readonly #entities = new Map<string, ObservedEntity>();
|
||||
readonly #volatile = new Map<string, Set<string>>();
|
||||
readonly #merged = new Map<string, StateEntity>();
|
||||
|
||||
constructor(input: {
|
||||
readonly tools: readonly StateToolFacts[];
|
||||
readonly records: readonly StateRecordFacts[];
|
||||
readonly timeline: StateTimeline;
|
||||
readonly budget: StateBudget;
|
||||
}) {
|
||||
this.#tools = input.tools;
|
||||
this.#timeline = input.timeline;
|
||||
this.#budget = input.budget;
|
||||
this.#position = new Map(
|
||||
input.records.map((record, index) => [record.ordinal, index]),
|
||||
);
|
||||
this.#byOrdinal = new Map(
|
||||
input.records.map((record) => [record.ordinal, record]),
|
||||
);
|
||||
this.#writes = input.records.filter(isStateWrite);
|
||||
this.#bases = apiBaseLengths(input.tools.map((tool) => tool.path_template));
|
||||
}
|
||||
|
||||
/** The extracted state, or null when the representation budget is exhausted. */
|
||||
run(): ExtractedState | null {
|
||||
try {
|
||||
const candidates = this.#candidates();
|
||||
for (const candidate of candidates.filter((entry) => entry.item))
|
||||
this.#planItem(candidate);
|
||||
for (const candidate of candidates.filter((entry) => !entry.item))
|
||||
this.#planList(candidate);
|
||||
for (const plan of this.#plans) this.#observePlan(plan);
|
||||
this.#merge();
|
||||
const gate = this.#gateContext();
|
||||
for (const plan of this.#plans)
|
||||
this.#views.set(
|
||||
plan.tool.name,
|
||||
plan.item ? checkItemPlan(gate, plan) : checkListPlan(gate, plan),
|
||||
);
|
||||
const documents = this.#documents(candidates);
|
||||
for (const { tool } of candidates)
|
||||
if (!this.#views.has(tool.name)) this.#recorded(tool, "not-entity");
|
||||
return this.#result(documents);
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof StateBudgetExhausted) return null;
|
||||
throw cause;
|
||||
}
|
||||
}
|
||||
|
||||
#retain(bytes: number): void {
|
||||
if (!this.#budget.retain(bytes)) throw new StateBudgetExhausted();
|
||||
}
|
||||
|
||||
/** Parse one body and hand it to `use` before its temporary work is dropped; unusable bodies arrive undefined. */
|
||||
#withBody<T>(ordinal: number, use: (body: JsonValue | undefined) => T): T {
|
||||
const parsed = this.#budget.body(ordinal);
|
||||
if (parsed === null) throw new StateBudgetExhausted();
|
||||
try {
|
||||
return use(
|
||||
parsed.kind === "json" && !hasPrototypeKey(parsed.value)
|
||||
? parsed.value
|
||||
: undefined,
|
||||
);
|
||||
} finally {
|
||||
this.#budget.release();
|
||||
}
|
||||
}
|
||||
|
||||
#recorded(tool: StateToolFacts, reason: RecordedViewReason): void {
|
||||
this.#views.set(tool.name, { tool: tool.name, shape: "recorded", reason });
|
||||
}
|
||||
|
||||
/** GET tools may render from state; query POSTs stay recorded. */
|
||||
#candidates(): Candidate[] {
|
||||
const candidates: Candidate[] = [];
|
||||
for (const tool of this.#tools) {
|
||||
if (isQueryPost(tool.method, tool.path_template))
|
||||
this.#recorded(tool, "query-body");
|
||||
if (tool.method !== "GET") continue;
|
||||
const segments = segmentsOf(tool.path_template);
|
||||
const item = segments.length > 0 && isParam(segments.at(-1)!);
|
||||
candidates.push({
|
||||
tool,
|
||||
item,
|
||||
type: item
|
||||
? `/${segments.slice(0, -1).join("/")}`
|
||||
: normalizedPath(tool.path_template),
|
||||
});
|
||||
}
|
||||
return candidates;
|
||||
}
|
||||
|
||||
/** The first write whose root relates to the template's bounds its init recordings. */
|
||||
#initLimit(template: string): StateRecordFacts | undefined {
|
||||
const root = rootOf(template, this.#bases);
|
||||
return this.#writes.find((write) =>
|
||||
rootsRelated(root, rootOf(write.path, this.#bases)),
|
||||
);
|
||||
}
|
||||
|
||||
/** Successful recordings that finished before the first related write began. */
|
||||
#initRecordings(tool: StateToolFacts, template: string): StateRecordFacts[] {
|
||||
const limit = this.#initLimit(template);
|
||||
const before = (record: StateRecordFacts): boolean =>
|
||||
limit === undefined ||
|
||||
(this.#position.get(record.ordinal)! <
|
||||
this.#position.get(limit.ordinal)! &&
|
||||
(this.#timeline === "record-order" || record.end <= limit.start));
|
||||
return tool.record_ordinals
|
||||
.map((ordinal) => this.#byOrdinal.get(ordinal))
|
||||
.filter(
|
||||
(record): record is StateRecordFacts =>
|
||||
record !== undefined &&
|
||||
record.status !== null &&
|
||||
record.status >= 200 &&
|
||||
record.status < 300 &&
|
||||
before(record),
|
||||
)
|
||||
.sort(
|
||||
(left, right) =>
|
||||
this.#position.get(left.ordinal)! -
|
||||
this.#position.get(right.ordinal)!,
|
||||
);
|
||||
}
|
||||
|
||||
#latestPerLine(recordings: readonly StateRecordFacts[]): StateRecordFacts[] {
|
||||
const latest = new Map<string, StateRecordFacts>();
|
||||
for (const record of recordings) {
|
||||
latest.delete(requestLine(record));
|
||||
latest.set(requestLine(record), record);
|
||||
}
|
||||
return [...latest.values()];
|
||||
}
|
||||
|
||||
/** A candidate's initial recordings, or undefined after recording that it has none. */
|
||||
#initRecordingsOf(candidate: Candidate): StateRecordFacts[] | undefined {
|
||||
const recordings = this.#initRecordings(candidate.tool, candidate.type);
|
||||
if (recordings.length > 0) return recordings;
|
||||
this.#recorded(candidate.tool, "no-init-recording");
|
||||
return undefined;
|
||||
}
|
||||
|
||||
#addPlan(
|
||||
candidate: Candidate,
|
||||
pointer: string,
|
||||
recordings: readonly StateRecordFacts[],
|
||||
): void {
|
||||
this.#plans.push({
|
||||
...candidate,
|
||||
pointer,
|
||||
recordings,
|
||||
latest: this.#latestPerLine(recordings),
|
||||
});
|
||||
}
|
||||
|
||||
/** Item tools prove a type's identity field by the path value their bodies repeat. */
|
||||
#planItem(candidate: Candidate): void {
|
||||
const { tool, type } = candidate;
|
||||
const recordings = this.#initRecordingsOf(candidate);
|
||||
if (recordings === undefined) return;
|
||||
let pointer: string | undefined;
|
||||
let keys: string[] | undefined;
|
||||
for (const record of recordings) {
|
||||
const located = this.#withBody(record.ordinal, (body) =>
|
||||
body === undefined
|
||||
? null
|
||||
: locateItem(body, lastSegmentValue(record.path)),
|
||||
);
|
||||
if (located === null) {
|
||||
this.#recorded(tool, "body-unusable");
|
||||
return;
|
||||
}
|
||||
if (
|
||||
located === undefined ||
|
||||
(pointer ?? located.pointer) !== located.pointer
|
||||
) {
|
||||
this.#recorded(tool, "no-identity");
|
||||
return;
|
||||
}
|
||||
pointer = located.pointer;
|
||||
keys = (keys ?? located.keys).filter((key) => located.keys.includes(key));
|
||||
}
|
||||
const field =
|
||||
IDENTITY_NAMES.find((name) => keys?.includes(name) === true) ?? keys?.[0];
|
||||
const known = this.#identity.get(type);
|
||||
if (
|
||||
field === undefined ||
|
||||
pointer === undefined ||
|
||||
(known !== undefined && known.field !== field)
|
||||
) {
|
||||
this.#recorded(tool, "no-identity");
|
||||
return;
|
||||
}
|
||||
this.#identity.set(type, { field, basis: "path-parameter" });
|
||||
this.#addPlan(candidate, pointer, recordings);
|
||||
}
|
||||
|
||||
/** List tools use the proved field, else the first identity name every non-empty recording's items carry. */
|
||||
#planList(candidate: Candidate): void {
|
||||
const { tool, type } = candidate;
|
||||
const recordings = this.#initRecordingsOf(candidate);
|
||||
if (recordings === undefined) return;
|
||||
const proved = this.#identity.get(type)?.field;
|
||||
let pointer: string | undefined;
|
||||
let names: readonly string[] | undefined;
|
||||
for (const record of recordings) {
|
||||
const found = this.#withBody(record.ordinal, (body) => {
|
||||
if (body === undefined) return null;
|
||||
const located = pointer ?? locateList(body, proved);
|
||||
const items =
|
||||
located === undefined ? undefined : listItems(body, located);
|
||||
if (items === undefined) return undefined;
|
||||
pointer = located;
|
||||
return items.length === 0
|
||||
? (names ?? (proved === undefined ? IDENTITY_NAMES : [proved]))
|
||||
: (proved === undefined ? IDENTITY_NAMES : [proved]).filter((name) =>
|
||||
identifies(items, name),
|
||||
);
|
||||
});
|
||||
if (found === null || found === undefined) {
|
||||
this.#recorded(tool, found === null ? "body-unusable" : "not-entity");
|
||||
return;
|
||||
}
|
||||
names = (names ?? found).filter((name) => found.includes(name));
|
||||
}
|
||||
const field = names?.[0];
|
||||
if (field === undefined || pointer === undefined) {
|
||||
this.#recorded(tool, "no-identity");
|
||||
return;
|
||||
}
|
||||
if (!this.#identity.has(type))
|
||||
this.#identity.set(type, { field, basis: "field-name" });
|
||||
this.#addPlan(candidate, pointer, recordings);
|
||||
}
|
||||
|
||||
#observePlan(plan: StatePlan): void {
|
||||
for (const record of plan.recordings)
|
||||
this.#withBody(record.ordinal, (body) => {
|
||||
const located = body === undefined ? undefined : at(body, plan.pointer);
|
||||
const objects = plan.item
|
||||
? isObject(located)
|
||||
? [located]
|
||||
: []
|
||||
: Array.isArray(located)
|
||||
? located.filter(isObject)
|
||||
: [];
|
||||
objects.forEach((object, index) =>
|
||||
this.#observe(plan, record, object, index),
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/** Record one observed entity object; a value that changes between one tool's own observations is volatile. */
|
||||
#observe(
|
||||
plan: StatePlan,
|
||||
record: StateRecordFacts,
|
||||
object: JsonObject,
|
||||
index: number,
|
||||
): void {
|
||||
const field = this.#identity.get(plan.type)!.field;
|
||||
const id = scalarText(object[field]);
|
||||
if (id === undefined) return;
|
||||
const collection = collectionPathOf(record, plan.item);
|
||||
const key = entityKey(plan.type, collection, id);
|
||||
let entity = this.#entities.get(key);
|
||||
if (entity === undefined) {
|
||||
this.#retain(ENTRY_BYTES * 4 + textBytes(key.length));
|
||||
entity = {
|
||||
type: plan.type,
|
||||
collection,
|
||||
id,
|
||||
order: [this.#position.get(record.ordinal)!, index],
|
||||
observations: [],
|
||||
};
|
||||
this.#entities.set(key, entity);
|
||||
}
|
||||
const earlier = entity.observations.findLast(
|
||||
(observation) => observation.tool === plan.tool.name,
|
||||
);
|
||||
for (const name of Object.keys(object))
|
||||
if (
|
||||
earlier !== undefined &&
|
||||
Object.hasOwn(earlier.object, name) &&
|
||||
!same(earlier.object[name], object[name])
|
||||
) {
|
||||
const fields = this.#volatile.get(plan.type) ?? new Set<string>();
|
||||
this.#volatile.set(plan.type, fields.add(name));
|
||||
}
|
||||
this.#retain(ENTRY_BYTES + jsonBytes(object));
|
||||
entity.observations.push({
|
||||
position: this.#position.get(record.ordinal)!,
|
||||
ordinal: record.ordinal,
|
||||
tool: plan.tool.name,
|
||||
item: plan.item,
|
||||
object,
|
||||
});
|
||||
}
|
||||
|
||||
#merge(): void {
|
||||
for (const entity of [...this.#entities.values()].sort(
|
||||
compareObservedEntities,
|
||||
))
|
||||
this.#merged.set(
|
||||
entityKey(entity.type, entity.collection, entity.id),
|
||||
mergeObservedEntity(entity),
|
||||
);
|
||||
}
|
||||
|
||||
#gateContext(): StateGateContext {
|
||||
return {
|
||||
withBody: (ordinal, use) => this.#withBody(ordinal, use),
|
||||
retain: (bytes) => this.#retain(bytes),
|
||||
entity: (type, collection, id) =>
|
||||
this.#merged.get(entityKey(type, collection, id)),
|
||||
inCollection: (type, collection) => this.#inCollection(type, collection),
|
||||
volatileOf: (type) => this.#volatileOf(type),
|
||||
idField: (type) => this.#identity.get(type)!.field,
|
||||
};
|
||||
}
|
||||
|
||||
#volatileOf(type: string): ReadonlySet<string> {
|
||||
return this.#volatile.get(type) ?? new Set<string>();
|
||||
}
|
||||
|
||||
#inCollection(type: string, collection: string): StateEntity[] {
|
||||
return [...this.#merged.values()].filter(
|
||||
(entity) => entity.type === type && entity.collection === collection,
|
||||
);
|
||||
}
|
||||
|
||||
/** A written path whose read is not an entity holds one document value: its latest init recording. */
|
||||
#documents(candidates: readonly Candidate[]): StateDocument[] {
|
||||
const written = new Set(
|
||||
this.#tools
|
||||
.filter(
|
||||
(tool) =>
|
||||
["POST", "PUT", "PATCH"].includes(tool.method) &&
|
||||
!isQueryPost(tool.method, tool.path_template),
|
||||
)
|
||||
.map((tool) => tool.path_template),
|
||||
);
|
||||
const documents: StateDocument[] = [];
|
||||
for (const { tool, item } of candidates) {
|
||||
const current = this.#views.get(tool.name);
|
||||
const entityless =
|
||||
current === undefined ||
|
||||
(current.shape === "recorded" &&
|
||||
(current.reason === "not-entity" ||
|
||||
current.reason === "no-identity"));
|
||||
if (item || !entityless || !written.has(tool.path_template)) continue;
|
||||
const latest = this.#latestPerLine(
|
||||
this.#initRecordings(tool, tool.path_template),
|
||||
);
|
||||
if (latest.length !== 1) {
|
||||
if (latest.length > 1) this.#recorded(tool, "query-varies");
|
||||
continue;
|
||||
}
|
||||
const record = latest[0]!;
|
||||
const value = this.#withBody(record.ordinal, (body) => body);
|
||||
if (value === undefined) {
|
||||
this.#recorded(tool, "body-unusable");
|
||||
continue;
|
||||
}
|
||||
this.#retain(
|
||||
ENTRY_BYTES + jsonBytes(value) + textBytes(record.path.length),
|
||||
);
|
||||
documents.push({
|
||||
tool: tool.name,
|
||||
path: record.path,
|
||||
value,
|
||||
record_ordinal: record.ordinal,
|
||||
});
|
||||
this.#views.set(tool.name, {
|
||||
tool: tool.name,
|
||||
shape: "document",
|
||||
checked_record_ordinals: [record.ordinal],
|
||||
});
|
||||
}
|
||||
return documents.sort((left, right) => compare(left.tool, right.tool));
|
||||
}
|
||||
|
||||
/** Types and entities that at least one state-backed view renders; view values only for those views. */
|
||||
#result(documents: StateDocument[]): ExtractedState {
|
||||
const backed = new Set(
|
||||
[...this.#views.values()].flatMap((view) =>
|
||||
view.shape === "item" || view.shape === "list" ? [view.tool] : [],
|
||||
),
|
||||
);
|
||||
const types = [...this.#identity.entries()]
|
||||
.filter(([type]) =>
|
||||
this.#plans.some(
|
||||
(plan) => plan.type === type && backed.has(plan.tool.name),
|
||||
),
|
||||
)
|
||||
.map(([type, { field, basis }]): StateType => ({
|
||||
collection_template: type,
|
||||
id_field: field,
|
||||
id_basis: basis,
|
||||
init_before_ordinal: this.#initLimit(type)?.ordinal ?? null,
|
||||
volatile_fields: [...this.#volatileOf(type)].sort(compare),
|
||||
}))
|
||||
.sort((left, right) =>
|
||||
compare(left.collection_template, right.collection_template),
|
||||
);
|
||||
const typed = new Set(types.map((type) => type.collection_template));
|
||||
return {
|
||||
types,
|
||||
entities: [...this.#merged.values()]
|
||||
.filter((entity) => typed.has(entity.type))
|
||||
.map((entity) => ({
|
||||
...entity,
|
||||
view_fields: entity.view_fields.filter((view) =>
|
||||
backed.has(view.tool),
|
||||
),
|
||||
})),
|
||||
documents,
|
||||
views: this.#tools.flatMap((tool) => {
|
||||
const view = this.#views.get(tool.name);
|
||||
return view === undefined ? [] : [view];
|
||||
}),
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -100,7 +100,7 @@ export abstract class WebNetworkProjectionAccumulator {
|
||||
/** Authenticate and gate a retained HAR parent before a grouping view reads any record. */
|
||||
export const admitHarCaptureParent = (
|
||||
parent: AnalysisViewParent,
|
||||
viewKind: "api-surface" | "environment-manifest",
|
||||
viewKind: "api-surface" | "environment-manifest" | "environment-state",
|
||||
accumulator: WebNetworkProjectionAccumulator,
|
||||
): Result<WebNetworkCapture, AnalysisError> => {
|
||||
const unsupported = (reason: string) =>
|
||||
@@ -113,7 +113,7 @@ export const admitHarCaptureParent = (
|
||||
);
|
||||
if (parent.operation !== "inspect_web_network_capture")
|
||||
return unsupported(
|
||||
`View ${viewKind} is unsupported for parent operation ${parent.operation}; ${viewKind} requires ${viewKind === "environment-manifest" ? "inspect_web_network_capture or capture_browser_scenario" : "inspect_web_network_capture"}.`,
|
||||
`View ${viewKind} is unsupported for parent operation ${parent.operation}; ${viewKind} requires ${viewKind === "api-surface" ? "inspect_web_network_capture" : "inspect_web_network_capture or capture_browser_scenario"}.`,
|
||||
);
|
||||
const admitted = accumulator.admitCaptureClone(parent.normalizedResult);
|
||||
if (!admitted.ok) return admitted;
|
||||
@@ -133,11 +133,11 @@ export const admitHarCaptureParent = (
|
||||
return unsupported(
|
||||
`${viewKind} does not support capture format ${parsed.data.format}.`,
|
||||
);
|
||||
// The manifest cites the report path and digest, so its parent's subject must be exactly them;
|
||||
// Environment views cite the report path and digest, so their parent's subject must be exactly them;
|
||||
// the api-surface view keeps the admission its existing callers rely on.
|
||||
const { artifact } = parsed.data;
|
||||
if (
|
||||
viewKind === "environment-manifest" &&
|
||||
viewKind !== "api-surface" &&
|
||||
artifact.sha256 !== null &&
|
||||
(parent.subject == null ||
|
||||
parent.subject.local_path !== artifact.path ||
|
||||
|
||||
@@ -100,11 +100,17 @@ import {
|
||||
} from "./webNetworkCaptureView.js";
|
||||
import type { WebNetworkApiSurfaceAccumulator } from "./WebNetworkApiSurfaceAccumulator.js";
|
||||
import type { EnvironmentManifestAccumulator } from "./EnvironmentManifestAccumulator.js";
|
||||
import type { EnvironmentStateAccumulator } from "./EnvironmentStateAccumulator.js";
|
||||
import {
|
||||
projectEnvironmentManifestView,
|
||||
environmentManifestFields,
|
||||
environmentManifestViewRequestSchema,
|
||||
} from "./environmentManifestView.js";
|
||||
import {
|
||||
environmentStateFields,
|
||||
environmentStateViewRequestSchema,
|
||||
projectEnvironmentStateView,
|
||||
} from "./environmentStateView.js";
|
||||
import {
|
||||
pcapViewRequestSchema,
|
||||
pcapViewPageSchema,
|
||||
@@ -249,6 +255,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
|
||||
androidViewRequestSchema,
|
||||
apiSurfaceViewSchema,
|
||||
environmentManifestViewRequestSchema,
|
||||
environmentStateViewRequestSchema,
|
||||
captureViewRequestSchema,
|
||||
qlogViewRequestSchema,
|
||||
pcapViewRequestSchema,
|
||||
@@ -528,6 +535,16 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
|
||||
]),
|
||||
view: environmentManifestViewRequestSchema,
|
||||
}),
|
||||
z.strictObject({
|
||||
...viewResultBase,
|
||||
...environmentStateFields,
|
||||
kind: z.literal("environment-state"),
|
||||
parent_operation: z.enum([
|
||||
"inspect_web_network_capture",
|
||||
"capture_browser_scenario",
|
||||
]),
|
||||
view: environmentStateViewRequestSchema,
|
||||
}),
|
||||
z.strictObject({
|
||||
...summaryViewResultBase,
|
||||
summary: z.union([
|
||||
@@ -719,6 +736,10 @@ export type UnsignedAnalysisView =
|
||||
Extract<AnalysisViewResult, { readonly kind: "environment-manifest" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "environment-state" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "summary" }>,
|
||||
"view_digest"
|
||||
@@ -816,7 +837,7 @@ const unsupportedParent = (
|
||||
INSPECT_ANALYSIS_VIEW_OPERATION,
|
||||
parentArtifactPath(parent),
|
||||
parent.operation === "capture_browser_scenario"
|
||||
? `Parent Evidence operation "capture_browser_scenario" supports only the environment-manifest view.`
|
||||
? `Parent Evidence operation "capture_browser_scenario" supports only the environment-manifest and environment-state views.`
|
||||
: `Parent Evidence operation ${JSON.stringify(parent.operation)} does not support analysis views. Use completed binary layout, JavaScript application, native function, artifact inventory/inspection, WASM, Mach-O, Interface Builder, EVM, managed artifact or member, Android package, Dart AOT, or web network capture Evidence.`,
|
||||
);
|
||||
|
||||
@@ -856,12 +877,47 @@ const retainedViewProjectors = new Map<string, typeof projectWasmArtifactView>([
|
||||
["inventory_artifact", projectArtifactInventoryView],
|
||||
]);
|
||||
|
||||
/** Budgets a caller created before admitting the source; each grouping view uses its own. */
|
||||
export interface AnalysisViewAccumulators {
|
||||
readonly apiSurface?: WebNetworkApiSurfaceAccumulator | undefined;
|
||||
readonly manifest?: EnvironmentManifestAccumulator | undefined;
|
||||
readonly state?: EnvironmentStateAccumulator | undefined;
|
||||
}
|
||||
|
||||
type GroupingViewRequest = Extract<
|
||||
AnalysisViewRequest,
|
||||
{ kind: "api-surface" | "environment-manifest" | "environment-state" }
|
||||
>;
|
||||
|
||||
const isGroupingView = (
|
||||
view: AnalysisViewRequest,
|
||||
): view is GroupingViewRequest =>
|
||||
view.kind === "api-surface" ||
|
||||
view.kind === "environment-manifest" ||
|
||||
view.kind === "environment-state";
|
||||
|
||||
/** Views that group a capture's records, each under the budget its caller created. */
|
||||
const projectGroupingView = (
|
||||
parent: AnalysisViewParent,
|
||||
view: GroupingViewRequest,
|
||||
accumulators: AnalysisViewAccumulators,
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
if (view.kind === "api-surface")
|
||||
return projectWebNetworkApiSurfaceView(
|
||||
parent,
|
||||
view,
|
||||
accumulators.apiSurface,
|
||||
);
|
||||
if (view.kind === "environment-manifest")
|
||||
return projectEnvironmentManifestView(parent, view, accumulators.manifest);
|
||||
return projectEnvironmentStateView(parent, view, accumulators.state);
|
||||
};
|
||||
|
||||
/** Project one selected view from authenticated parent analysis Evidence. */
|
||||
export const projectAnalysisView = (
|
||||
parent: AnalysisViewParent,
|
||||
view: AnalysisViewRequest,
|
||||
apiSurfaceAccumulator?: WebNetworkApiSurfaceAccumulator,
|
||||
manifestAccumulator?: EnvironmentManifestAccumulator,
|
||||
accumulators: AnalysisViewAccumulators = {},
|
||||
): Result<AnalysisViewResult, AnalysisError> => {
|
||||
const digest = parentDigest(parent.evidenceId);
|
||||
if (!digestSchema.safeParse(digest).success)
|
||||
@@ -874,21 +930,9 @@ export const projectAnalysisView = (
|
||||
},
|
||||
]),
|
||||
);
|
||||
if (view.kind === "api-surface") {
|
||||
const projected = projectWebNetworkApiSurfaceView(
|
||||
parent,
|
||||
view,
|
||||
apiSurfaceAccumulator,
|
||||
);
|
||||
return projected.ok ? ok(sealAnalysisView(projected.value)) : projected;
|
||||
}
|
||||
if (view.kind === "environment-manifest") {
|
||||
const projected = projectEnvironmentManifestView(
|
||||
parent,
|
||||
view,
|
||||
manifestAccumulator,
|
||||
);
|
||||
return projected.ok ? ok(sealAnalysisView(projected.value)) : projected;
|
||||
if (isGroupingView(view)) {
|
||||
const grouped = projectGroupingView(parent, view, accumulators);
|
||||
return grouped.ok ? ok(sealAnalysisView(grouped.value)) : grouped;
|
||||
}
|
||||
if (parent.operation === "inspect_web_network_capture") {
|
||||
const projected = projectWebNetworkCaptureView(parent, view);
|
||||
|
||||
@@ -159,7 +159,7 @@ export const projectBinaryLayoutView = (
|
||||
layout: BinaryLayout,
|
||||
view: Exclude<
|
||||
AnalysisViewRequest,
|
||||
{ kind: "api-surface" | "environment-manifest" }
|
||||
{ kind: "api-surface" | "environment-manifest" | "environment-state" }
|
||||
>,
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
const shared = parentFields(parent, layout);
|
||||
|
||||
@@ -10,23 +10,28 @@ import type {
|
||||
UnsignedAnalysisView,
|
||||
} from "./analysisView.js";
|
||||
import { EnvironmentManifestAccumulator } from "./EnvironmentManifestAccumulator.js";
|
||||
import { TEMPLATE_RULES } from "./environmentManifestRules.js";
|
||||
import {
|
||||
TEMPLATE_RULES,
|
||||
type TemplateRule,
|
||||
} from "./environmentManifestRules.js";
|
||||
import { admitHarCaptureParent } from "./WebNetworkProjectionAccumulator.js";
|
||||
import { admitScenarioCaptureParent } from "./scenarioCaptureRecords.js";
|
||||
import { ROUTE_RULES } from "./webNetworkRouteTemplates.js";
|
||||
|
||||
/** Opt-in template rules, each listed at most once; shared by every view that groups tools. */
|
||||
export const templateRulesSchema = z
|
||||
.array(z.enum(TEMPLATE_RULES))
|
||||
.refine((rules) => new Set(rules).size === rules.length, {
|
||||
message: "template_rules must not repeat a rule",
|
||||
})
|
||||
.meta({ uniqueItems: true })
|
||||
.default([]);
|
||||
|
||||
export const environmentManifestViewRequestSchema = z.strictObject({
|
||||
kind: z.literal("environment-manifest"),
|
||||
template_rules: z
|
||||
.array(z.enum(TEMPLATE_RULES))
|
||||
.refine((rules) => new Set(rules).size === rules.length, {
|
||||
message: "template_rules must not repeat a rule",
|
||||
})
|
||||
.meta({ uniqueItems: true })
|
||||
.default([])
|
||||
.describe(
|
||||
"Opt-in template rules applied after the four format rules, each listed at most once: after-uid-literal templates the segment following a literal uid, opaque-token templates long mixed alphanumeric segments, sibling-variance merges groups that differ in one identifier-like segment. Literal sibling folding into existing opt-in parameters also requires sibling-variance. Merged parameters are inferences.",
|
||||
),
|
||||
template_rules: templateRulesSchema.describe(
|
||||
"Opt-in template rules applied after the four format rules, each listed at most once: after-uid-literal templates the segment following a literal uid, opaque-token templates long mixed alphanumeric segments, sibling-variance merges groups that differ in one identifier-like segment. Literal sibling folding into existing opt-in parameters also requires sibling-variance. Merged parameters are inferences.",
|
||||
),
|
||||
include_query_parameter_names: z
|
||||
.boolean()
|
||||
.default(true)
|
||||
@@ -122,7 +127,7 @@ export const environmentManifestFields = {
|
||||
};
|
||||
|
||||
/** The records, coverage and identity a manifest projects from, for either parent kind. */
|
||||
interface CaptureParent {
|
||||
export interface CaptureParent {
|
||||
readonly operation:
|
||||
| "inspect_web_network_capture"
|
||||
| "capture_browser_scenario";
|
||||
@@ -135,10 +140,11 @@ interface CaptureParent {
|
||||
/** HAR captures keep their own gate; a browser scenario has no captured file, so its artifact is empty. */
|
||||
const admitCaptureParent = (
|
||||
parent: AnalysisViewParent,
|
||||
viewKind: "environment-manifest" | "environment-state",
|
||||
accumulator: EnvironmentManifestAccumulator,
|
||||
): Result<CaptureParent, AnalysisError> => {
|
||||
if (parent.operation === "capture_browser_scenario") {
|
||||
const scenario = admitScenarioCaptureParent(parent, accumulator);
|
||||
const scenario = admitScenarioCaptureParent(parent, viewKind, accumulator);
|
||||
return scenario.ok
|
||||
? ok({
|
||||
operation: "capture_browser_scenario",
|
||||
@@ -149,16 +155,64 @@ const admitCaptureParent = (
|
||||
})
|
||||
: scenario;
|
||||
}
|
||||
const har = admitHarCaptureParent(
|
||||
parent,
|
||||
"environment-manifest",
|
||||
accumulator,
|
||||
);
|
||||
const har = admitHarCaptureParent(parent, viewKind, accumulator);
|
||||
return har.ok
|
||||
? ok({ operation: "inspect_web_network_capture", ...har.value })
|
||||
: har;
|
||||
};
|
||||
|
||||
/** Admit the parent, the shared and view-specific limitations, then group every retained record. */
|
||||
export const groupCaptureRecords = (
|
||||
parent: AnalysisViewParent,
|
||||
viewKind: "environment-manifest" | "environment-state",
|
||||
rules: readonly TemplateRule[],
|
||||
target: {
|
||||
readonly accumulator: EnvironmentManifestAccumulator;
|
||||
readonly noun: "manifest" | "state";
|
||||
readonly limitations: readonly string[];
|
||||
},
|
||||
): Result<CaptureParent, AnalysisError> => {
|
||||
const admitted = admitCaptureParent(parent, viewKind, target.accumulator);
|
||||
if (!admitted.ok) return admitted;
|
||||
const added = addCaptureRecords(parent, admitted.value, rules, target);
|
||||
return added.ok ? admitted : added;
|
||||
};
|
||||
|
||||
/** Admit the shared and view-specific limitations, then group every retained record. */
|
||||
const addCaptureRecords = (
|
||||
parent: AnalysisViewParent,
|
||||
capture: CaptureParent,
|
||||
rules: readonly TemplateRule[],
|
||||
target: {
|
||||
readonly accumulator: EnvironmentManifestAccumulator;
|
||||
readonly noun: "manifest" | "state";
|
||||
readonly limitations: readonly string[];
|
||||
},
|
||||
): Result<void, AnalysisError> => {
|
||||
const limitations = target.accumulator.addLimitations([
|
||||
...parent.limitations,
|
||||
...capture.limitations,
|
||||
"Tool templates, parameters and names are inferences from recorded request lines; they do not establish handler identity, URL-space schemas or causality.",
|
||||
...target.limitations,
|
||||
...(rules.length > 0
|
||||
? [
|
||||
`Opt-in template rules ${uniqueSorted(rules).join(", ")} merge recorded literals by shape; merged parameters may conflate distinct resources.`,
|
||||
]
|
||||
: []),
|
||||
...(capture.records.length < capture.total_records
|
||||
? [
|
||||
`Parent contains an ordinal subset; the ${target.noun} covers only retained records.`,
|
||||
]
|
||||
: []),
|
||||
]);
|
||||
if (!limitations.ok) return limitations;
|
||||
for (const record of capture.records) {
|
||||
const added = target.accumulator.add(record);
|
||||
if (!added.ok) return added;
|
||||
}
|
||||
return ok(undefined);
|
||||
};
|
||||
|
||||
/** Project retained request lines, from a HAR or a browser scenario, into a replayable tool surface; templates are inferences. */
|
||||
export const projectEnvironmentManifestView = (
|
||||
parent: AnalysisViewParent,
|
||||
@@ -179,34 +233,17 @@ export const projectEnvironmentManifestView = (
|
||||
`View ${view.kind} is unsupported for parent operation ${parent.operation}; environment-manifest requires inspect_web_network_capture or capture_browser_scenario.`,
|
||||
),
|
||||
);
|
||||
const admitted = admitCaptureParent(parent, accumulator);
|
||||
if (!admitted.ok) return admitted;
|
||||
const capture = admitted.value;
|
||||
const limitations = accumulator.addLimitations([
|
||||
...parent.limitations,
|
||||
...capture.limitations,
|
||||
"Tool templates, parameters and names are inferences from recorded request lines; they do not establish handler identity, URL-space schemas or causality.",
|
||||
...(view.include_query_parameter_names
|
||||
const grouped = groupCaptureRecords(parent, view.kind, view.template_rules, {
|
||||
accumulator,
|
||||
noun: "manifest",
|
||||
limitations: view.include_query_parameter_names
|
||||
? [
|
||||
"Observed query parameter names do not establish required parameters, value types or a complete API schema.",
|
||||
]
|
||||
: []),
|
||||
...(view.template_rules.length > 0
|
||||
? [
|
||||
`Opt-in template rules ${uniqueSorted(view.template_rules).join(", ")} merge recorded literals by shape; merged parameters may conflate distinct resources.`,
|
||||
]
|
||||
: []),
|
||||
...(capture.records.length < capture.total_records
|
||||
? [
|
||||
"Parent contains an ordinal subset; the manifest covers only retained records.",
|
||||
]
|
||||
: []),
|
||||
]);
|
||||
if (!limitations.ok) return limitations;
|
||||
for (const record of capture.records) {
|
||||
const added = accumulator.add(record);
|
||||
if (!added.ok) return added;
|
||||
}
|
||||
: [],
|
||||
});
|
||||
if (!grouped.ok) return grouped;
|
||||
const capture = grouped.value;
|
||||
const manifest = accumulator.finish();
|
||||
if (!manifest.ok) return manifest;
|
||||
return ok({
|
||||
|
||||
@@ -0,0 +1,258 @@
|
||||
import { canonicalJson } from "../comparisonSemantics.js";
|
||||
import type { JsonValue } from "../jsonValue.js";
|
||||
import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js";
|
||||
import {
|
||||
at,
|
||||
collectionPathOf,
|
||||
isObject,
|
||||
lastSegmentValue,
|
||||
listItems,
|
||||
renderStateEntity,
|
||||
renderStateList,
|
||||
replaceAt,
|
||||
same,
|
||||
scalarText,
|
||||
type JsonObject,
|
||||
type RecordedViewReason,
|
||||
type StateEntity,
|
||||
type StateRecordFacts,
|
||||
type StateToolFacts,
|
||||
type StateView,
|
||||
} from "./environmentStateRules.js";
|
||||
import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js";
|
||||
|
||||
/** A read tool whose recordings locate entities of one type at one pointer. */
|
||||
export interface StatePlan {
|
||||
readonly tool: StateToolFacts;
|
||||
readonly item: boolean;
|
||||
readonly type: string;
|
||||
readonly pointer: string;
|
||||
/** Successful init recordings in timeline order. */
|
||||
readonly recordings: readonly StateRecordFacts[];
|
||||
/** The latest of them per request line, in timeline order. */
|
||||
readonly latest: readonly StateRecordFacts[];
|
||||
}
|
||||
|
||||
/** What the gate reads from an extraction; `withBody` drops each body's temporary work after `use`. */
|
||||
export interface StateGateContext {
|
||||
readonly withBody: <T>(
|
||||
ordinal: number,
|
||||
use: (body: JsonValue | undefined) => T,
|
||||
) => T;
|
||||
readonly retain: (bytes: number) => void;
|
||||
readonly entity: (
|
||||
type: string,
|
||||
collection: string,
|
||||
id: string,
|
||||
) => StateEntity | undefined;
|
||||
readonly inCollection: (type: string, collection: string) => StateEntity[];
|
||||
readonly volatileOf: (type: string) => ReadonlySet<string>;
|
||||
readonly idField: (type: string) => string;
|
||||
}
|
||||
|
||||
/** Ordered key union and always-present keys, grown one object at a time. */
|
||||
class FieldUnion {
|
||||
readonly fields: string[] = [];
|
||||
#required: Set<string> | undefined;
|
||||
|
||||
add(object: JsonObject): void {
|
||||
for (const key of Object.keys(object))
|
||||
if (!this.fields.includes(key)) this.fields.push(key);
|
||||
const keys = new Set(Object.keys(object));
|
||||
this.#required =
|
||||
this.#required === undefined
|
||||
? keys
|
||||
: new Set([...this.#required].filter((key) => keys.has(key)));
|
||||
}
|
||||
|
||||
get required(): string[] {
|
||||
return this.fields.filter((key) => this.#required?.has(key) === true);
|
||||
}
|
||||
|
||||
/** Retained bytes of the field lists a plan keeps. */
|
||||
get bytes(): number {
|
||||
return this.fields.reduce(
|
||||
(sum, name) => sum + ENTRY_BYTES * 2 + textBytes(name.length) * 2,
|
||||
ENTRY_BYTES * 2,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const recorded = (plan: StatePlan, reason: RecordedViewReason): StateView => ({
|
||||
tool: plan.tool.name,
|
||||
shape: "recorded",
|
||||
reason,
|
||||
});
|
||||
|
||||
/** Why one latest item recording does not render from the shared envelope and the entity's state, if it does not. */
|
||||
const itemMismatch = (
|
||||
context: StateGateContext,
|
||||
plan: StatePlan,
|
||||
check: {
|
||||
readonly view: { readonly tool: string; readonly fields: string[] };
|
||||
readonly envelope: { skeleton?: string };
|
||||
},
|
||||
record: StateRecordFacts,
|
||||
): RecordedViewReason | undefined => {
|
||||
const { view, envelope } = check;
|
||||
const entity = context.entity(
|
||||
plan.type,
|
||||
collectionPathOf(record, true),
|
||||
lastSegmentValue(record.path),
|
||||
);
|
||||
return context.withBody(record.ordinal, (body) => {
|
||||
const object = body === undefined ? undefined : at(body, plan.pointer);
|
||||
if (entity === undefined || body === undefined || !isObject(object))
|
||||
return "fields-differ";
|
||||
const skeleton = canonicalJson(
|
||||
replaceAt(body, plan.pointer, null),
|
||||
"Environment state",
|
||||
);
|
||||
if (envelope.skeleton === undefined) {
|
||||
context.retain(textBytes(skeleton.length));
|
||||
envelope.skeleton = skeleton;
|
||||
}
|
||||
if (skeleton !== envelope.skeleton) return "envelope-varies";
|
||||
const shown = renderStateEntity(
|
||||
entity,
|
||||
view,
|
||||
context.volatileOf(plan.type),
|
||||
object,
|
||||
);
|
||||
return same(shown, object) ? undefined : "fields-differ";
|
||||
});
|
||||
};
|
||||
|
||||
/** An item plan must reproduce each latest recording from the first one's envelope and the entity's state. */
|
||||
export const checkItemPlan = (
|
||||
context: StateGateContext,
|
||||
plan: StatePlan,
|
||||
): StateView => {
|
||||
const union = new FieldUnion();
|
||||
for (const record of plan.latest)
|
||||
context.withBody(record.ordinal, (body) => {
|
||||
const object = body === undefined ? undefined : at(body, plan.pointer);
|
||||
if (isObject(object)) union.add(object);
|
||||
});
|
||||
context.retain(union.bytes);
|
||||
const view = { tool: plan.tool.name, fields: union.fields };
|
||||
const envelope: { skeleton?: string } = {};
|
||||
for (const record of plan.latest) {
|
||||
const reason = itemMismatch(context, plan, { view, envelope }, record);
|
||||
if (reason !== undefined) return recorded(plan, reason);
|
||||
}
|
||||
return {
|
||||
tool: plan.tool.name,
|
||||
shape: "item",
|
||||
type: plan.type,
|
||||
pointer: plan.pointer,
|
||||
fields: union.fields,
|
||||
required_fields: union.required,
|
||||
envelope_ordinal: plan.latest[0]!.ordinal,
|
||||
checked_record_ordinals: plan.latest.map((record) => record.ordinal),
|
||||
};
|
||||
};
|
||||
|
||||
/** Whether every latest recording lists its whole collection in state order, and which counters equal the item count. */
|
||||
const listFacts = (
|
||||
context: StateGateContext,
|
||||
plan: StatePlan,
|
||||
union: FieldUnion,
|
||||
): { readonly all: boolean; readonly counters: string[] } => {
|
||||
const field = context.idField(plan.type);
|
||||
let all = true;
|
||||
let counters: string[] | undefined;
|
||||
for (const record of plan.latest)
|
||||
context.withBody(record.ordinal, (body) => {
|
||||
const items =
|
||||
body === undefined ? undefined : listItems(body, plan.pointer);
|
||||
if (body === undefined || items === undefined) return;
|
||||
items.forEach((item) => union.add(item));
|
||||
const ids = items.map((item) => scalarText(item[field]) ?? "");
|
||||
const expected = context
|
||||
.inCollection(plan.type, collectionPathOf(record, false))
|
||||
.map((entity) => entity.id);
|
||||
if (record.query !== undefined || !same(ids, expected)) all = false;
|
||||
const parent = plan.pointer === "" || !isObject(body) ? {} : body;
|
||||
const names = Object.keys(parent).filter(
|
||||
(name) => /total|count/iu.test(name) && parent[name] === items.length,
|
||||
);
|
||||
counters = (counters ?? names).filter((name) => names.includes(name));
|
||||
});
|
||||
return { all, counters: (counters ?? []).sort(compare) };
|
||||
};
|
||||
|
||||
/** Whether one latest list recording renders from its own envelope and the listed entities' state. */
|
||||
const listReproduces = (
|
||||
context: StateGateContext,
|
||||
plan: StatePlan,
|
||||
view: {
|
||||
readonly tool: string;
|
||||
readonly fields: string[];
|
||||
readonly pointer: string;
|
||||
readonly counters: string[];
|
||||
readonly all: boolean;
|
||||
},
|
||||
record: StateRecordFacts,
|
||||
): boolean => {
|
||||
const field = context.idField(plan.type);
|
||||
return context.withBody(record.ordinal, (body) => {
|
||||
const items =
|
||||
body === undefined ? undefined : listItems(body, plan.pointer);
|
||||
if (body === undefined || items === undefined) return false;
|
||||
const observed = new Map(
|
||||
items.map((item) => [scalarText(item[field]) ?? "", item]),
|
||||
);
|
||||
const members = view.all
|
||||
? context.inCollection(plan.type, collectionPathOf(record, false))
|
||||
: items.flatMap((item) => {
|
||||
const entity = context.entity(
|
||||
plan.type,
|
||||
collectionPathOf(record, false),
|
||||
scalarText(item[field]) ?? "",
|
||||
);
|
||||
return entity === undefined ? [] : [entity];
|
||||
});
|
||||
const rendered = members.map((entity) =>
|
||||
renderStateEntity(
|
||||
entity,
|
||||
view,
|
||||
context.volatileOf(plan.type),
|
||||
observed.get(entity.id),
|
||||
),
|
||||
);
|
||||
return same(renderStateList(view, body, rendered), body);
|
||||
});
|
||||
};
|
||||
|
||||
/** A list plan must reproduce each latest recording from its own envelope and the listed entities' state. */
|
||||
export const checkListPlan = (
|
||||
context: StateGateContext,
|
||||
plan: StatePlan,
|
||||
): StateView => {
|
||||
const union = new FieldUnion();
|
||||
const { all, counters } = listFacts(context, plan, union);
|
||||
context.retain(union.bytes);
|
||||
const view = {
|
||||
tool: plan.tool.name,
|
||||
fields: union.fields,
|
||||
pointer: plan.pointer,
|
||||
counters,
|
||||
all,
|
||||
};
|
||||
if (
|
||||
!plan.latest.every((record) => listReproduces(context, plan, view, record))
|
||||
)
|
||||
return recorded(plan, "list-differs");
|
||||
return {
|
||||
tool: plan.tool.name,
|
||||
shape: "list",
|
||||
type: plan.type,
|
||||
pointer: plan.pointer,
|
||||
fields: union.fields,
|
||||
required_fields: union.required,
|
||||
membership: all ? "all" : "recorded",
|
||||
counters,
|
||||
checked_record_ordinals: plan.latest.map((record) => record.ordinal),
|
||||
};
|
||||
};
|
||||
@@ -0,0 +1,71 @@
|
||||
import type { JsonValue } from "../jsonValue.js";
|
||||
import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js";
|
||||
import {
|
||||
same,
|
||||
type JsonObject,
|
||||
type StateEntity,
|
||||
} from "./environmentStateRules.js";
|
||||
|
||||
/** One entity object as one tool recorded it. */
|
||||
export interface StateObservation {
|
||||
readonly position: number;
|
||||
readonly ordinal: number;
|
||||
readonly tool: string;
|
||||
readonly item: boolean;
|
||||
readonly object: JsonObject;
|
||||
}
|
||||
|
||||
/** Every init observation of one entity, with where it was first seen. */
|
||||
export interface ObservedEntity {
|
||||
readonly type: string;
|
||||
readonly collection: string;
|
||||
readonly id: string;
|
||||
/** Timeline position of the first observation, then its index within that body. */
|
||||
readonly order: readonly [number, number];
|
||||
readonly observations: StateObservation[];
|
||||
}
|
||||
|
||||
/** Collection order: type, collection, then first sighting. */
|
||||
export const compareObservedEntities = (
|
||||
left: ObservedEntity,
|
||||
right: ObservedEntity,
|
||||
): number =>
|
||||
compare(left.type, right.type) ||
|
||||
compare(left.collection, right.collection) ||
|
||||
left.order[0] - right.order[0] ||
|
||||
left.order[1] - right.order[1];
|
||||
|
||||
/** One entity: item views' latest values win over list views', and a view whose latest values differ keeps them. */
|
||||
export const mergeObservedEntity = (entity: ObservedEntity): StateEntity => {
|
||||
const latest = new Map<string, StateObservation>();
|
||||
for (const observation of entity.observations)
|
||||
latest.set(observation.tool, observation);
|
||||
const fields = new Map<string, JsonValue>();
|
||||
const ranked = [...latest.values()].sort(
|
||||
(left, right) =>
|
||||
Number(left.item) - Number(right.item) || left.position - right.position,
|
||||
);
|
||||
for (const observation of ranked)
|
||||
for (const [name, value] of Object.entries(observation.object))
|
||||
fields.set(name, value);
|
||||
return {
|
||||
type: entity.type,
|
||||
collection: entity.collection,
|
||||
id: entity.id,
|
||||
fields: Object.fromEntries(fields),
|
||||
view_fields: [...latest.values()]
|
||||
.map((observation) => ({
|
||||
tool: observation.tool,
|
||||
fields: Object.fromEntries(
|
||||
Object.entries(observation.object).filter(
|
||||
([name, value]) => !same(fields.get(name), value),
|
||||
),
|
||||
),
|
||||
}))
|
||||
.filter((view) => Object.keys(view.fields).length > 0)
|
||||
.sort((left, right) => compare(left.tool, right.tool)),
|
||||
record_ordinals: [
|
||||
...new Set(entity.observations.map((observation) => observation.ordinal)),
|
||||
],
|
||||
};
|
||||
};
|
||||
@@ -0,0 +1,415 @@
|
||||
import { canonicalJson } from "../comparisonSemantics.js";
|
||||
import type { JsonValue } from "../jsonValue.js";
|
||||
import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js";
|
||||
|
||||
/** How records are ordered: by recorded start time, or by record order when any start time is missing. */
|
||||
export type StateTimeline = "recorded-start-time" | "record-order";
|
||||
|
||||
/** What extraction reads of one primary-origin record; times are timeline coordinates. */
|
||||
export interface StateRecordFacts {
|
||||
readonly ordinal: number;
|
||||
readonly method: string;
|
||||
readonly path: string;
|
||||
/** WHATWG query spelling including `?`, or undefined when the request had none. */
|
||||
readonly query: string | undefined;
|
||||
readonly status: number | null;
|
||||
readonly start: number;
|
||||
readonly end: number;
|
||||
}
|
||||
|
||||
/** The manifest tool facts extraction uses. */
|
||||
export interface StateToolFacts {
|
||||
readonly name: string;
|
||||
readonly method: string;
|
||||
readonly path_template: string;
|
||||
readonly record_ordinals: readonly number[];
|
||||
}
|
||||
|
||||
/** A record's response body as JSON, or unusable when absent, cut, not JSON or not exactly representable. */
|
||||
export type ParsedBody =
|
||||
| { readonly kind: "json"; readonly value: JsonValue }
|
||||
| { readonly kind: "unusable" };
|
||||
|
||||
/** Budgeted access the accumulator lends extraction; null and false mean the budget is exhausted. */
|
||||
export interface StateBudget {
|
||||
readonly body: (ordinal: number) => ParsedBody | null;
|
||||
readonly retain: (bytes: number) => boolean;
|
||||
/** Drop the temporary work of the body just read. */
|
||||
readonly release: () => void;
|
||||
}
|
||||
|
||||
export type RecordedViewReason =
|
||||
| "query-body"
|
||||
| "no-init-recording"
|
||||
| "body-unusable"
|
||||
| "not-entity"
|
||||
| "no-identity"
|
||||
| "query-varies"
|
||||
| "fields-differ"
|
||||
| "envelope-varies"
|
||||
| "list-differs";
|
||||
|
||||
export interface StateType {
|
||||
readonly collection_template: string;
|
||||
readonly id_field: string;
|
||||
readonly id_basis: "path-parameter" | "field-name";
|
||||
readonly init_before_ordinal: number | null;
|
||||
readonly volatile_fields: string[];
|
||||
}
|
||||
|
||||
export type JsonObject = Record<string, JsonValue>;
|
||||
|
||||
export interface StateEntity {
|
||||
readonly type: string;
|
||||
readonly collection: string;
|
||||
readonly id: string;
|
||||
readonly fields: JsonObject;
|
||||
readonly view_fields: {
|
||||
readonly tool: string;
|
||||
readonly fields: JsonObject;
|
||||
}[];
|
||||
readonly record_ordinals: number[];
|
||||
}
|
||||
|
||||
export interface StateDocument {
|
||||
readonly tool: string;
|
||||
readonly path: string;
|
||||
readonly value: JsonValue;
|
||||
readonly record_ordinal: number;
|
||||
}
|
||||
|
||||
interface EntityViewPlan {
|
||||
readonly tool: string;
|
||||
readonly type: string;
|
||||
readonly pointer: string;
|
||||
readonly fields: string[];
|
||||
readonly required_fields: string[];
|
||||
readonly checked_record_ordinals: number[];
|
||||
}
|
||||
|
||||
export type StateItemView = EntityViewPlan & {
|
||||
readonly shape: "item";
|
||||
readonly envelope_ordinal: number;
|
||||
};
|
||||
|
||||
export type StateListView = EntityViewPlan & {
|
||||
readonly shape: "list";
|
||||
readonly membership: "all" | "recorded";
|
||||
readonly counters: string[];
|
||||
};
|
||||
|
||||
export type StateView =
|
||||
| StateItemView
|
||||
| StateListView
|
||||
| {
|
||||
readonly tool: string;
|
||||
readonly shape: "document";
|
||||
readonly checked_record_ordinals: number[];
|
||||
}
|
||||
| {
|
||||
readonly tool: string;
|
||||
readonly shape: "recorded";
|
||||
readonly reason: RecordedViewReason;
|
||||
};
|
||||
|
||||
export interface ExtractedState {
|
||||
readonly types: StateType[];
|
||||
readonly entities: StateEntity[];
|
||||
readonly documents: StateDocument[];
|
||||
readonly views: StateView[];
|
||||
}
|
||||
|
||||
const WRITE_METHODS = new Set(["POST", "PUT", "PATCH", "DELETE"]);
|
||||
/** A POST whose path names one of these segments queries state rather than changing it. */
|
||||
const QUERY_SEGMENTS = new Set([
|
||||
"search",
|
||||
"query",
|
||||
"list",
|
||||
"find",
|
||||
"fuzzy",
|
||||
"lookup",
|
||||
"count",
|
||||
"aggregate",
|
||||
"stats",
|
||||
"_search",
|
||||
"_count",
|
||||
"_msearch",
|
||||
]);
|
||||
/** Identity field names tried, in order, when no path parameter proves one. */
|
||||
export const IDENTITY_NAMES = [
|
||||
"id",
|
||||
"uid",
|
||||
"uuid",
|
||||
"_id",
|
||||
"ID",
|
||||
"UID",
|
||||
"Id",
|
||||
"guid",
|
||||
"key",
|
||||
"slug",
|
||||
"name",
|
||||
];
|
||||
const VERSION_SEGMENT = /^v\d+(?:\.\d+)*$/u;
|
||||
|
||||
export const segmentsOf = (path: string): string[] =>
|
||||
path.split("/").filter((segment) => segment.length > 0);
|
||||
export const isParam = (segment: string): boolean =>
|
||||
segment.startsWith("{") && segment.endsWith("}");
|
||||
export const isObject = (value: JsonValue | undefined): value is JsonObject =>
|
||||
typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
export const scalarText = (value: JsonValue | undefined): string | undefined =>
|
||||
typeof value === "string"
|
||||
? value
|
||||
: typeof value === "number"
|
||||
? String(value)
|
||||
: undefined;
|
||||
/** Equal JSON values; object member order carries no meaning. */
|
||||
export const same = (
|
||||
left: JsonValue | undefined,
|
||||
right: JsonValue | undefined,
|
||||
): boolean =>
|
||||
left === undefined || right === undefined
|
||||
? left === right
|
||||
: canonicalJson(left, "Environment state") ===
|
||||
canonicalJson(right, "Environment state");
|
||||
|
||||
/** The decoded final path segment, which an item request names its entity by. */
|
||||
export const lastSegmentValue = (path: string): string => {
|
||||
const segment = segmentsOf(path).at(-1) ?? "";
|
||||
try {
|
||||
return decodeURIComponent(segment);
|
||||
} catch {
|
||||
return segment;
|
||||
}
|
||||
};
|
||||
|
||||
/** A path's non-empty segments, so a trailing slash names the same collection. */
|
||||
export const normalizedPath = (path: string): string =>
|
||||
`/${segmentsOf(path).join("/")}`;
|
||||
|
||||
/** An item record's collection is its path without the identity segment; a list record's is its path. */
|
||||
export const collectionPathOf = (
|
||||
record: StateRecordFacts,
|
||||
item: boolean,
|
||||
): string =>
|
||||
item
|
||||
? `/${segmentsOf(record.path).slice(0, -1).join("/")}`
|
||||
: normalizedPath(record.path);
|
||||
|
||||
/** A POST whose path names a query segment observes state rather than changing it. */
|
||||
export const isQueryPost = (method: string, path: string): boolean =>
|
||||
method === "POST" &&
|
||||
segmentsOf(path).some((segment) => QUERY_SEGMENTS.has(segment.toLowerCase()));
|
||||
|
||||
/** Whether a record may change state: a successful or unanswered write that is not a query POST. */
|
||||
export const isStateWrite = (record: StateRecordFacts): boolean =>
|
||||
WRITE_METHODS.has(record.method) &&
|
||||
!isQueryPost(record.method, record.path) &&
|
||||
(record.status === null || (record.status >= 200 && record.status < 400));
|
||||
|
||||
/** Literal leading segments shared by every template with the same first segment: each path's API base. */
|
||||
export const apiBaseLengths = (
|
||||
templates: readonly string[],
|
||||
): ReadonlyMap<string, number> => {
|
||||
const groups = new Map<string, string[][]>();
|
||||
for (const template of templates) {
|
||||
const segments = segmentsOf(template);
|
||||
const group = groups.get(segments[0] ?? "") ?? [];
|
||||
groups.set(segments[0] ?? "", [...group, segments]);
|
||||
}
|
||||
return new Map(
|
||||
[...groups.entries()].map(([first, members]) => {
|
||||
const head = members[0] ?? [];
|
||||
let length = 0;
|
||||
while (
|
||||
length < head.length &&
|
||||
!isParam(head[length]!) &&
|
||||
members.every((segments) => segments[length] === head[length])
|
||||
)
|
||||
length += 1;
|
||||
return [first, length];
|
||||
}),
|
||||
);
|
||||
};
|
||||
|
||||
/** The path's API base, then any version segments, then one segment: the scope a write may affect. */
|
||||
export const rootOf = (
|
||||
path: string,
|
||||
bases: ReadonlyMap<string, number>,
|
||||
): string[] => {
|
||||
const segments = segmentsOf(path);
|
||||
let end = Math.min(bases.get(segments[0] ?? "") ?? 0, segments.length);
|
||||
while (end < segments.length && VERSION_SEGMENT.test(segments[end]!))
|
||||
end += 1;
|
||||
return segments.slice(0, end + 1);
|
||||
};
|
||||
|
||||
/** Roots relate when they agree on their common length; template parameters match any segment. */
|
||||
export const rootsRelated = (
|
||||
left: readonly string[],
|
||||
right: readonly string[],
|
||||
): boolean =>
|
||||
left
|
||||
.slice(0, right.length)
|
||||
.every(
|
||||
(segment, index) =>
|
||||
segment === right[index] || isParam(segment) || isParam(right[index]!),
|
||||
);
|
||||
|
||||
/** Value bytes as retained state, charged before a value is kept. */
|
||||
export const jsonBytes = (value: JsonValue): number => {
|
||||
if (typeof value === "string") return textBytes(value.length);
|
||||
if (Array.isArray(value))
|
||||
return value.reduce<number>(
|
||||
(sum, item) => sum + jsonBytes(item),
|
||||
ENTRY_BYTES,
|
||||
);
|
||||
if (isObject(value))
|
||||
return Object.entries(value).reduce<number>(
|
||||
(sum, [name, item]) => sum + textBytes(name.length) + jsonBytes(item),
|
||||
ENTRY_BYTES,
|
||||
);
|
||||
return ENTRY_BYTES;
|
||||
};
|
||||
|
||||
/** JSON parsers keep a `__proto__` member that plain object construction would turn into a prototype. */
|
||||
export const hasPrototypeKey = (value: JsonValue): boolean =>
|
||||
Array.isArray(value)
|
||||
? value.some(hasPrototypeKey)
|
||||
: isObject(value) &&
|
||||
Object.entries(value).some(
|
||||
([name, item]) => name === "__proto__" || hasPrototypeKey(item),
|
||||
);
|
||||
|
||||
export const pointerTo = (name: string): string =>
|
||||
`/${name.replaceAll("~", "~0").replaceAll("/", "~1")}`;
|
||||
const pointerName = (pointer: string): string =>
|
||||
pointer.slice(1).replaceAll("~1", "/").replaceAll("~0", "~");
|
||||
|
||||
/** The value at a pointer of depth at most one. */
|
||||
export const at = (body: JsonValue, pointer: string): JsonValue | undefined => {
|
||||
if (pointer === "") return body;
|
||||
if (!isObject(body)) return undefined;
|
||||
const name = pointerName(pointer);
|
||||
return Object.hasOwn(body, name) ? body[name] : undefined;
|
||||
};
|
||||
|
||||
/** A copy of `body` with the value at a pointer of depth at most one replaced. */
|
||||
export const replaceAt = (
|
||||
body: JsonValue,
|
||||
pointer: string,
|
||||
value: JsonValue,
|
||||
): JsonValue => {
|
||||
if (pointer === "" || !isObject(body)) return value;
|
||||
const name = pointerName(pointer);
|
||||
return Object.fromEntries(
|
||||
Object.entries(body).map(([key, item]) => [
|
||||
key,
|
||||
key === name ? value : item,
|
||||
]),
|
||||
);
|
||||
};
|
||||
|
||||
const matchingKeys = (object: JsonObject, value: string): string[] =>
|
||||
Object.keys(object).filter((key) => scalarText(object[key]) === value);
|
||||
|
||||
/** The entity object an item response carries, at the root or in one wrapper property, with the keys repeating its path value. */
|
||||
export const locateItem = (
|
||||
body: JsonValue,
|
||||
value: string,
|
||||
): { readonly pointer: string; readonly keys: string[] } | undefined => {
|
||||
if (!isObject(body)) return undefined;
|
||||
const root = matchingKeys(body, value);
|
||||
if (root.length > 0) return { pointer: "", keys: root };
|
||||
const wrapped = Object.entries(body).flatMap(([name, item]) => {
|
||||
if (!isObject(item)) return [];
|
||||
const keys = matchingKeys(item, value);
|
||||
return keys.length > 0 ? [{ pointer: pointerTo(name), keys }] : [];
|
||||
});
|
||||
return wrapped.length === 1 ? wrapped[0] : undefined;
|
||||
};
|
||||
|
||||
/** Whether every item carries `name` as a distinct string or number. */
|
||||
export const identifies = (
|
||||
items: readonly JsonObject[],
|
||||
name: string,
|
||||
): boolean => {
|
||||
const values = items.map((item) => scalarText(item[name]));
|
||||
return (
|
||||
values.every((value) => value !== undefined) &&
|
||||
new Set(values).size === values.length
|
||||
);
|
||||
};
|
||||
|
||||
/** The array of entity objects a list response carries, at the root or in one property of an envelope that is not itself an entity. */
|
||||
export const locateList = (
|
||||
body: JsonValue,
|
||||
idField: string | undefined,
|
||||
): string | undefined => {
|
||||
if (Array.isArray(body)) return body.every(isObject) ? "" : undefined;
|
||||
if (
|
||||
!isObject(body) ||
|
||||
IDENTITY_NAMES.some((name) => scalarText(body[name]) !== undefined)
|
||||
)
|
||||
return undefined;
|
||||
const arrays = Object.entries(body).filter(
|
||||
(entry): entry is [string, JsonObject[]] =>
|
||||
Array.isArray(entry[1]) && entry[1].every(isObject),
|
||||
);
|
||||
const carrying = arrays.filter(
|
||||
([, items]) =>
|
||||
items.length > 0 &&
|
||||
(idField === undefined
|
||||
? IDENTITY_NAMES.some((name) => identifies(items, name))
|
||||
: identifies(items, idField)),
|
||||
);
|
||||
const chosen =
|
||||
carrying.length === 1
|
||||
? carrying[0]
|
||||
: carrying.length === 0 && arrays.length === 1
|
||||
? arrays[0]
|
||||
: undefined;
|
||||
return chosen === undefined ? undefined : pointerTo(chosen[0]);
|
||||
};
|
||||
|
||||
export const listItems = (
|
||||
body: JsonValue,
|
||||
pointer: string,
|
||||
): JsonObject[] | undefined => {
|
||||
const items = at(body, pointer);
|
||||
return Array.isArray(items) && items.every(isObject) ? items : undefined;
|
||||
};
|
||||
|
||||
/** One entity as a view shows it: the view's fields it has, view-specific values first; volatile fields take `observed` values when given. */
|
||||
export const renderStateEntity = (
|
||||
entity: StateEntity,
|
||||
view: Pick<EntityViewPlan, "tool" | "fields">,
|
||||
volatileFields: ReadonlySet<string>,
|
||||
observed?: JsonObject,
|
||||
): JsonObject => {
|
||||
const own =
|
||||
entity.view_fields.find((candidate) => candidate.tool === view.tool)
|
||||
?.fields ?? {};
|
||||
return Object.fromEntries(
|
||||
view.fields.flatMap((name): [string, JsonValue][] => {
|
||||
const source =
|
||||
observed !== undefined && volatileFields.has(name)
|
||||
? observed
|
||||
: Object.hasOwn(own, name)
|
||||
? own
|
||||
: entity.fields;
|
||||
return Object.hasOwn(source, name) ? [[name, source[name]!]] : [];
|
||||
}),
|
||||
);
|
||||
};
|
||||
|
||||
/** A list response rendered from `base`: its array replaced by `items` and its counters set to their number. */
|
||||
export const renderStateList = (
|
||||
view: Pick<StateListView, "pointer" | "counters">,
|
||||
base: JsonValue,
|
||||
items: readonly JsonObject[],
|
||||
): JsonValue =>
|
||||
view.counters.reduce(
|
||||
(body, name) => replaceAt(body, pointerTo(name), items.length),
|
||||
replaceAt(base, view.pointer, [...items]),
|
||||
);
|
||||
@@ -0,0 +1,351 @@
|
||||
import { expect, it } from "vitest";
|
||||
import {
|
||||
analysisViewCaptureEvidence,
|
||||
environmentManifestCaptureFixture,
|
||||
environmentStateCaptureFixture,
|
||||
} from "../../../tests/fixtures/analysisView.js";
|
||||
import {
|
||||
SCENARIO_EXCHANGES,
|
||||
scenarioCaptureEvidence,
|
||||
scenarioCaptureFixture,
|
||||
} from "../../../tests/fixtures/browserScenarioCapture.js";
|
||||
import type { Evidence } from "../evidence.js";
|
||||
import type { JsonValue } from "../jsonValue.js";
|
||||
import {
|
||||
projectAnalysisView,
|
||||
type AnalysisViewParent,
|
||||
type AnalysisViewResult,
|
||||
} from "./analysisView.js";
|
||||
import { EnvironmentStateAccumulator } from "./EnvironmentStateAccumulator.js";
|
||||
|
||||
type State = Extract<AnalysisViewResult, { kind: "environment-state" }>;
|
||||
const ALL_RULES = [
|
||||
"after-uid-literal",
|
||||
"opaque-token",
|
||||
"sibling-variance",
|
||||
] as const;
|
||||
|
||||
const parentOf = (
|
||||
evidence: Evidence = analysisViewCaptureEvidence(
|
||||
environmentStateCaptureFixture(),
|
||||
),
|
||||
): AnalysisViewParent => ({
|
||||
evidenceId: evidence.evidence_id,
|
||||
operation: evidence.operation,
|
||||
normalizedResult: evidence.normalized_result,
|
||||
subject: evidence.subject,
|
||||
limitations: [],
|
||||
parameters: evidence.parameters,
|
||||
});
|
||||
|
||||
const stateOf = (parent: AnalysisViewParent = parentOf()): State => {
|
||||
const result = projectAnalysisView(parent, {
|
||||
kind: "environment-state",
|
||||
template_rules: [...ALL_RULES],
|
||||
});
|
||||
if (!result.ok || result.value.kind !== "environment-state")
|
||||
throw new Error("Expected an environment state");
|
||||
return result.value;
|
||||
};
|
||||
|
||||
type Row = readonly [
|
||||
method: string,
|
||||
path: string,
|
||||
body?: JsonValue,
|
||||
status?: number,
|
||||
];
|
||||
|
||||
/** HAR entries one second apart on one origin; a string body is recorded as written. */
|
||||
const captureOf = (rows: readonly Row[]): AnalysisViewParent =>
|
||||
parentOf(
|
||||
analysisViewCaptureEvidence(
|
||||
environmentManifestCaptureFixture(
|
||||
rows.map(([method, path, body = null, status = 200], index) => ({
|
||||
startedDateTime: new Date(
|
||||
Date.parse("2026-10-17T18:00:00.000Z") + index * 1000,
|
||||
).toISOString(),
|
||||
time: 5,
|
||||
request: { method, url: `http://app.example${path}` },
|
||||
response: {
|
||||
status,
|
||||
content: {
|
||||
mimeType: "application/json",
|
||||
text: typeof body === "string" ? body : JSON.stringify(body),
|
||||
},
|
||||
},
|
||||
})),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
const viewOf = (state: State, tool: string) =>
|
||||
state.views.find((view) => view.tool === tool);
|
||||
|
||||
it("extracts the state the Grafana-shaped capture's reads observed", () => {
|
||||
const state = stateOf();
|
||||
expect(state.timeline).toBe("recorded-start-time");
|
||||
expect(state.types).toContainEqual({
|
||||
collection_template: "/api/folders",
|
||||
id_field: "uid",
|
||||
id_basis: "path-parameter",
|
||||
init_before_ordinal: 20,
|
||||
volatile_fields: [],
|
||||
});
|
||||
expect(state.types).toContainEqual({
|
||||
collection_template: "/api/search",
|
||||
id_field: "id",
|
||||
id_basis: "field-name",
|
||||
init_before_ordinal: null,
|
||||
volatile_fields: [],
|
||||
});
|
||||
expect(
|
||||
state.entities.filter((entity) => entity.type === "/api/folders"),
|
||||
).toEqual([
|
||||
{
|
||||
type: "/api/folders",
|
||||
collection: "/api/folders",
|
||||
id: "fg1jo91e57e2ob",
|
||||
fields: { id: 5, uid: "fg1jo91e57e2ob", title: "probe folder" },
|
||||
view_fields: [],
|
||||
record_ordinals: [12, 14],
|
||||
},
|
||||
{
|
||||
type: "/api/folders",
|
||||
collection: "/api/folders",
|
||||
id: "general",
|
||||
fields: { id: 0, uid: "general", title: "Dashboards" },
|
||||
view_fields: [],
|
||||
record_ordinals: [13],
|
||||
},
|
||||
]);
|
||||
expect(viewOf(state, "get_folders")).toMatchObject({
|
||||
shape: "list",
|
||||
pointer: "",
|
||||
membership: "recorded",
|
||||
});
|
||||
expect(viewOf(state, "get_folders_by_uid")).toMatchObject({
|
||||
shape: "item",
|
||||
envelope_ordinal: 12,
|
||||
checked_record_ordinals: [12, 13],
|
||||
});
|
||||
expect(viewOf(state, "get_dashboards_by_uid")).toMatchObject({
|
||||
shape: "item",
|
||||
pointer: "/dashboard",
|
||||
});
|
||||
expect(viewOf(state, "get_annotations")).toEqual({
|
||||
tool: "get_annotations",
|
||||
shape: "recorded",
|
||||
reason: "no-init-recording",
|
||||
});
|
||||
expect(state.documents).toEqual([]);
|
||||
expect(state.unknowns.map((unknown) => unknown.kind)).toEqual([
|
||||
"authorization-model",
|
||||
"response-semantics",
|
||||
"state-transition",
|
||||
]);
|
||||
expect(
|
||||
state.limitations.filter((limitation) =>
|
||||
limitation.startsWith("Body shapes"),
|
||||
),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it("ends a type's state at the first write under its root, not at unrelated writes", () => {
|
||||
const state = stateOf(
|
||||
captureOf([
|
||||
["GET", "/api/items/1", { id: 1, title: "Granite Ledger" }],
|
||||
["PUT", "/api/notes/9", { id: 9 }],
|
||||
["GET", "/api/items/1", { id: 1, title: "Granite Ledger" }],
|
||||
["PUT", "/api/items/1", { id: 1, title: "Saltmarsh Depot" }],
|
||||
["GET", "/api/items/1", { id: 1, title: "Saltmarsh Depot" }],
|
||||
]),
|
||||
);
|
||||
expect(state.types).toEqual([
|
||||
{
|
||||
collection_template: "/api/items",
|
||||
id_field: "id",
|
||||
id_basis: "path-parameter",
|
||||
init_before_ordinal: 3,
|
||||
volatile_fields: [],
|
||||
},
|
||||
]);
|
||||
expect(state.entities).toMatchObject([
|
||||
{ id: "1", fields: { title: "Granite Ledger" }, record_ordinals: [0, 2] },
|
||||
]);
|
||||
expect(state.views).toContainEqual(
|
||||
expect.objectContaining({ shape: "item", checked_record_ordinals: [2] }),
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps the latest value of a field that changed with no related write and names it volatile", () => {
|
||||
const state = stateOf(
|
||||
captureOf([
|
||||
["GET", "/api/items/1", { id: 1, title: "Lighthouse Rota", seen: 10 }],
|
||||
["GET", "/api/items/1", { id: 1, title: "Lighthouse Rota", seen: 11 }],
|
||||
["GET", "/api/items/2", { id: 2, title: "Blue Heron", seen: 3 }],
|
||||
]),
|
||||
);
|
||||
expect(state.types[0]).toMatchObject({ volatile_fields: ["seen"] });
|
||||
expect(state.entities[0]).toMatchObject({
|
||||
id: "1",
|
||||
fields: { seen: 11 },
|
||||
});
|
||||
expect(state.views).toContainEqual(
|
||||
expect.objectContaining({
|
||||
shape: "item",
|
||||
checked_record_ordinals: [1, 2],
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("lists the whole collection with its counter only when every recording did", () => {
|
||||
const rows: Row[] = [
|
||||
[
|
||||
"GET",
|
||||
"/api/items",
|
||||
{
|
||||
total: 2,
|
||||
items: [
|
||||
{ id: 1, title: "Ada Finch" },
|
||||
{ id: 2, title: "Tomas Vey" },
|
||||
],
|
||||
},
|
||||
],
|
||||
["GET", "/api/items/1", { title: "Ada Finch", id: 1, notes: "first" }],
|
||||
["GET", "/api/items/2", { id: 2, notes: "second", title: "Tomas Vey" }],
|
||||
];
|
||||
const whole = stateOf(captureOf(rows));
|
||||
expect(whole.views).toContainEqual(
|
||||
expect.objectContaining({
|
||||
shape: "list",
|
||||
pointer: "/items",
|
||||
fields: ["id", "title"],
|
||||
membership: "all",
|
||||
counters: ["total"],
|
||||
}),
|
||||
);
|
||||
expect(whole.entities.map((entity) => entity.fields)).toEqual([
|
||||
{ id: 1, title: "Ada Finch", notes: "first" },
|
||||
{ id: 2, notes: "second", title: "Tomas Vey" },
|
||||
]);
|
||||
const filtered = stateOf(
|
||||
captureOf([
|
||||
...rows,
|
||||
[
|
||||
"GET",
|
||||
"/api/items?title=Ada",
|
||||
{ total: 1, items: [{ id: 1, title: "Ada Finch" }] },
|
||||
],
|
||||
]),
|
||||
);
|
||||
expect(filtered.views).toContainEqual(
|
||||
expect.objectContaining({ shape: "list", membership: "recorded" }),
|
||||
);
|
||||
});
|
||||
|
||||
it("leaves recorded the views it cannot reproduce, with the reason", () => {
|
||||
const state = stateOf(
|
||||
captureOf([
|
||||
[
|
||||
"GET",
|
||||
"/api/boards/1",
|
||||
{ meta: { viewer: "Ada Finch" }, board: { id: 1, name: "Rota" } },
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"/api/boards/2",
|
||||
{ meta: { viewer: "Mina Okafor" }, board: { id: 2, name: "Depot" } },
|
||||
],
|
||||
["GET", "/api/ledgers/1", '{"id":1,"total":12345678901234567890}'],
|
||||
["POST", "/api/boards/search", [{ id: 1, name: "Rota" }]],
|
||||
]),
|
||||
);
|
||||
expect(
|
||||
state.views.map((view) =>
|
||||
view.shape === "recorded" ? [view.tool, view.reason] : view.tool,
|
||||
),
|
||||
).toEqual([
|
||||
["post_boards_search", "query-body"],
|
||||
["get_boards_by_id", "envelope-varies"],
|
||||
["get_ledgers_by_id", "body-unusable"],
|
||||
]);
|
||||
expect(state.types).toEqual([]);
|
||||
expect(state.entities).toEqual([]);
|
||||
});
|
||||
|
||||
it("holds a written path's non-entity read as a document", () => {
|
||||
const state = stateOf(
|
||||
captureOf([
|
||||
["GET", "/api/settings", { theme: "dark", density: "compact" }],
|
||||
["PUT", "/api/settings", { theme: "light", density: "compact" }],
|
||||
["GET", "/api/settings", { theme: "light", density: "compact" }],
|
||||
]),
|
||||
);
|
||||
expect(state.documents).toEqual([
|
||||
{
|
||||
tool: "get_settings",
|
||||
path: "/api/settings",
|
||||
value: { theme: "dark", density: "compact" },
|
||||
record_ordinal: 0,
|
||||
},
|
||||
]);
|
||||
expect(viewOf(state, "get_settings")).toEqual({
|
||||
tool: "get_settings",
|
||||
shape: "document",
|
||||
checked_record_ordinals: [0],
|
||||
});
|
||||
});
|
||||
|
||||
it("orders a scenario capture by request order and needs its response bodies", () => {
|
||||
const state = stateOf(parentOf(scenarioCaptureEvidence()));
|
||||
expect(state.timeline).toBe("record-order");
|
||||
expect(state.types).toEqual([
|
||||
{
|
||||
collection_template: "/api/users",
|
||||
id_field: "id",
|
||||
id_basis: "path-parameter",
|
||||
init_before_ordinal: null,
|
||||
volatile_fields: [],
|
||||
},
|
||||
]);
|
||||
expect(state.entities.map((entity) => entity.fields)).toEqual([
|
||||
{ id: 2, login: "probe" },
|
||||
{ id: 3, login: "viewer" },
|
||||
]);
|
||||
const headersOnly = scenarioCaptureEvidence(
|
||||
scenarioCaptureFixture(SCENARIO_EXCHANGES, {
|
||||
request_body: true,
|
||||
response_body: false,
|
||||
header_values: true,
|
||||
}),
|
||||
);
|
||||
expect(
|
||||
projectAnalysisView(parentOf(headersOnly), {
|
||||
kind: "environment-state",
|
||||
template_rules: [],
|
||||
}),
|
||||
).toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
_tag: "AnalysisUnsupportedTargetError",
|
||||
message: expect.stringContaining("response_body selected"),
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses when extraction would exceed the projection budget", () => {
|
||||
expect(
|
||||
projectAnalysisView(
|
||||
parentOf(),
|
||||
{ kind: "environment-state", template_rules: [...ALL_RULES] },
|
||||
{ state: new EnvironmentStateAccumulator(160 * 1024, [...ALL_RULES]) },
|
||||
),
|
||||
).toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
_tag: "AnalysisResourceConstraintError",
|
||||
resource: "memory",
|
||||
message: expect.stringContaining("Environment-state"),
|
||||
},
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,188 @@
|
||||
import { z } from "zod";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { jsonValueSchema } from "../jsonValue.js";
|
||||
import { ok, type Result } from "../result.js";
|
||||
import type {
|
||||
AnalysisViewParent,
|
||||
AnalysisViewRequest,
|
||||
UnsignedAnalysisView,
|
||||
} from "./analysisView.js";
|
||||
import { EnvironmentStateAccumulator } from "./EnvironmentStateAccumulator.js";
|
||||
import {
|
||||
groupCaptureRecords,
|
||||
templateRulesSchema,
|
||||
} from "./environmentManifestView.js";
|
||||
|
||||
export const environmentStateViewRequestSchema = z.strictObject({
|
||||
kind: z.literal("environment-state"),
|
||||
template_rules: templateRulesSchema.describe(
|
||||
"Opt-in template rules as for environment-manifest, each listed at most once. Use the rules of the manifest the state accompanies so tool names match.",
|
||||
),
|
||||
});
|
||||
|
||||
const jsonObjectFieldsSchema = z.record(z.string(), jsonValueSchema);
|
||||
const checkedSchema = z
|
||||
.array(z.number().int().nonnegative())
|
||||
.describe(
|
||||
"Latest init record per request line that the view reproduced from the state.",
|
||||
);
|
||||
const entityViewFields = {
|
||||
tool: z.string(),
|
||||
type: z.string(),
|
||||
pointer: z
|
||||
.string()
|
||||
.describe(
|
||||
"JSON pointer, of depth at most one, to the entity object or array within the response.",
|
||||
),
|
||||
fields: z
|
||||
.array(z.string())
|
||||
.describe("Ordered fields the view shows of each entity."),
|
||||
required_fields: z.array(z.string()),
|
||||
checked_record_ordinals: checkedSchema,
|
||||
};
|
||||
const viewSchema = z.discriminatedUnion("shape", [
|
||||
z.strictObject({
|
||||
...entityViewFields,
|
||||
shape: z.literal("item"),
|
||||
envelope_ordinal: z
|
||||
.number()
|
||||
.int()
|
||||
.nonnegative()
|
||||
.describe(
|
||||
"Record whose response, with the entity object replaced, every item response shares.",
|
||||
),
|
||||
}),
|
||||
z.strictObject({
|
||||
...entityViewFields,
|
||||
shape: z.literal("list"),
|
||||
membership: z
|
||||
.enum(["all", "recorded"])
|
||||
.describe(
|
||||
"all: every entity of the collection in state order; recorded: the entities each recorded response listed.",
|
||||
),
|
||||
counters: z
|
||||
.array(z.string())
|
||||
.describe("Envelope members that count the listed entities."),
|
||||
}),
|
||||
z.strictObject({
|
||||
tool: z.string(),
|
||||
shape: z.literal("document"),
|
||||
checked_record_ordinals: checkedSchema,
|
||||
}),
|
||||
z.strictObject({
|
||||
tool: z.string(),
|
||||
shape: z.literal("recorded"),
|
||||
reason: z.enum([
|
||||
"query-body",
|
||||
"no-init-recording",
|
||||
"body-unusable",
|
||||
"not-entity",
|
||||
"no-identity",
|
||||
"query-varies",
|
||||
"fields-differ",
|
||||
"envelope-varies",
|
||||
"list-differs",
|
||||
]),
|
||||
}),
|
||||
]);
|
||||
|
||||
export const environmentStateFields = {
|
||||
origin: z
|
||||
.string()
|
||||
.describe("Primary origin, as the environment manifest selects it."),
|
||||
name: z.string(),
|
||||
timeline: z
|
||||
.enum(["recorded-start-time", "record-order"])
|
||||
.describe(
|
||||
"Records are ordered by recorded start time, or by record order when any start time is missing.",
|
||||
),
|
||||
types: z.array(
|
||||
z.strictObject({
|
||||
collection_template: z.string(),
|
||||
id_field: z.string(),
|
||||
id_basis: z
|
||||
.enum(["path-parameter", "field-name"])
|
||||
.describe(
|
||||
"path-parameter: item responses repeat the requested path value in this field; field-name: an inferred identity name every listed item carries distinctly.",
|
||||
),
|
||||
init_before_ordinal: z
|
||||
.number()
|
||||
.int()
|
||||
.nonnegative()
|
||||
.nullable()
|
||||
.describe(
|
||||
"First recorded write whose path root relates to the type; state comes from records that finished before it.",
|
||||
),
|
||||
volatile_fields: z
|
||||
.array(z.string())
|
||||
.describe(
|
||||
"Fields whose recorded values changed with no related write between; the state keeps the latest.",
|
||||
),
|
||||
}),
|
||||
),
|
||||
entities: z
|
||||
.array(
|
||||
z.strictObject({
|
||||
type: z.string(),
|
||||
collection: z.string(),
|
||||
id: z.string(),
|
||||
fields: jsonObjectFieldsSchema,
|
||||
view_fields: z.array(
|
||||
z.strictObject({ tool: z.string(), fields: jsonObjectFieldsSchema }),
|
||||
),
|
||||
record_ordinals: z.array(z.number().int().nonnegative()),
|
||||
}),
|
||||
)
|
||||
.describe(
|
||||
"Entities in collection order: type, collection, then first sighting.",
|
||||
),
|
||||
documents: z.array(
|
||||
z.strictObject({
|
||||
tool: z.string(),
|
||||
path: z.string(),
|
||||
value: jsonValueSchema,
|
||||
record_ordinal: z.number().int().nonnegative(),
|
||||
}),
|
||||
),
|
||||
views: z.array(viewSchema),
|
||||
unknowns: z.array(
|
||||
z.strictObject({
|
||||
kind: z.enum([
|
||||
"authorization-model",
|
||||
"response-semantics",
|
||||
"state-transition",
|
||||
]),
|
||||
statement: z.string(),
|
||||
}),
|
||||
),
|
||||
};
|
||||
|
||||
/** Extract the init state a capture's reads observed and each read tool's render plan; types and identity are inferences. */
|
||||
export const projectEnvironmentStateView = (
|
||||
parent: AnalysisViewParent,
|
||||
view: Extract<AnalysisViewRequest, { kind: "environment-state" }>,
|
||||
accumulator = new EnvironmentStateAccumulator(undefined, view.template_rules),
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
const grouped = groupCaptureRecords(parent, view.kind, view.template_rules, {
|
||||
accumulator,
|
||||
noun: "state",
|
||||
limitations: [
|
||||
"Entity types, identity fields, list membership and counters are inferences from recorded response bodies; the state does not establish server-side schemas, derived values or values copied outside entity objects.",
|
||||
],
|
||||
});
|
||||
if (!grouped.ok) return grouped;
|
||||
const state = accumulator.finishState();
|
||||
if (!state.ok) return state;
|
||||
return ok({
|
||||
kind: "environment-state",
|
||||
view,
|
||||
parent_evidence_id: parent.evidenceId,
|
||||
parent_operation: grouped.value.operation,
|
||||
parent_digest: parent.evidenceId.slice(3),
|
||||
artifact: {
|
||||
path: grouped.value.artifact.path,
|
||||
sha256: grouped.value.artifact.sha256,
|
||||
},
|
||||
...state.value,
|
||||
});
|
||||
};
|
||||
@@ -309,7 +309,7 @@ export const projectJavaScriptApplicationView = (
|
||||
analysis: JavaScriptApplicationAnalysisResult,
|
||||
view: Exclude<
|
||||
AnalysisViewRequest,
|
||||
{ kind: "api-surface" | "environment-manifest" }
|
||||
{ kind: "api-surface" | "environment-manifest" | "environment-state" }
|
||||
>,
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
if (
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
import { z } from "zod";
|
||||
import type { WebNetworkCaptureRecord } from "../webNetworkCapture.js";
|
||||
|
||||
/** One record's retained response text, as replay and state extraction both read it. */
|
||||
export interface RecordedResponseBody {
|
||||
/** Producer-decoded response text, or null when the record retained none. */
|
||||
readonly bodyText: string | null;
|
||||
/** Set when the retained bytes are not UTF-8, so `bodyText` holds their base64 rather than a lossy decoding. */
|
||||
readonly bodyEncoding?: "base64";
|
||||
/** Set when only a prefix of a large base64 body was decoded. */
|
||||
readonly bodyCut?: true;
|
||||
}
|
||||
|
||||
const contentTextSchema = z.object({
|
||||
response: z.object({
|
||||
content: z.object({
|
||||
text: z.unknown().optional(),
|
||||
encoding: z.unknown().optional(),
|
||||
}),
|
||||
}),
|
||||
});
|
||||
|
||||
/** Strict, and keeping a leading byte-order mark, so callers see the recorded text unchanged. */
|
||||
const strictUtf8 = (): TextDecoder =>
|
||||
new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
|
||||
|
||||
/** Base64 characters that decode to just over `limitBytes`, so a cut prefix still fills it. */
|
||||
export const base64PrefixChars = (limitBytes: number): number =>
|
||||
Math.ceil((limitBytes + 4) / 3) * 4;
|
||||
|
||||
/** The retained response text of one record, decoding HAR base64 content when the decoder kept it as valid UTF-8; base64 beyond `limitBytes` is decoded only that far. */
|
||||
export const readRecordedResponseBody = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
limitBytes: number,
|
||||
): RecordedResponseBody => {
|
||||
const content = contentTextSchema.safeParse(record.reported);
|
||||
if (!content.success) return { bodyText: null };
|
||||
const { text, encoding } = content.data.response.content;
|
||||
if (typeof text !== "string") return { bodyText: null };
|
||||
if (encoding !== "base64") return { bodyText: text };
|
||||
const binary = record.binary_fields.find(
|
||||
(field) =>
|
||||
field.pointer === "/response/content/text" && field.state === "retained",
|
||||
);
|
||||
if (binary === undefined || binary.content_base64 === null)
|
||||
return { bodyText: null };
|
||||
const prefixChars = base64PrefixChars(limitBytes);
|
||||
const cut = binary.content_base64.length > prefixChars;
|
||||
const encoded = cut
|
||||
? binary.content_base64.slice(0, prefixChars)
|
||||
: binary.content_base64;
|
||||
const flags = cut ? { bodyCut: true as const } : {};
|
||||
try {
|
||||
// A cut prefix may end inside a character; streaming leaves that tail undecoded.
|
||||
const text = strictUtf8().decode(Buffer.from(encoded, "base64"), {
|
||||
stream: cut,
|
||||
});
|
||||
return { bodyText: text, ...flags };
|
||||
} catch {
|
||||
// Binary or non-UTF-8 bytes: keep them exactly, as base64, instead of inventing replacement characters.
|
||||
return { bodyText: encoded, bodyEncoding: "base64", ...flags };
|
||||
}
|
||||
};
|
||||
@@ -158,6 +158,7 @@ it("reserves retained request-body expansion before schema decoding or projectio
|
||||
normalizedResult: capture,
|
||||
limitations: [],
|
||||
},
|
||||
"environment-manifest",
|
||||
new EnvironmentManifestAccumulator(128 * 1024),
|
||||
);
|
||||
expect(projected).toMatchObject({
|
||||
|
||||
@@ -262,9 +262,10 @@ export const scenarioCaptureRecords = (
|
||||
};
|
||||
};
|
||||
|
||||
/** Admit a capture_browser_scenario parent whose network content identifies response media types. */
|
||||
/** Admit a capture_browser_scenario parent whose network content identifies response media types; state also needs bodies. */
|
||||
export const admitScenarioCaptureParent = (
|
||||
parent: AnalysisViewParent,
|
||||
viewKind: "environment-manifest" | "environment-state",
|
||||
accumulator: WebNetworkProjectionAccumulator,
|
||||
): Result<ScenarioCaptureRecords, AnalysisError> => {
|
||||
// Body validation decodes base64, and projection constructs UTF-8 request text.
|
||||
@@ -288,13 +289,19 @@ export const admitScenarioCaptureParent = (
|
||||
]),
|
||||
);
|
||||
const content = parsed.data.scenario.network_content;
|
||||
// Tools are chosen by response media type, which only bodies or header values carry.
|
||||
if (!content.response_body && !content.header_values)
|
||||
// Tools are chosen by response media type, which only bodies or header values carry; state is read from bodies.
|
||||
if (
|
||||
viewKind === "environment-state"
|
||||
? !content.response_body
|
||||
: !content.response_body && !content.header_values
|
||||
)
|
||||
return err(
|
||||
new AnalysisUnsupportedTargetError(
|
||||
"inspect_analysis_view",
|
||||
parent.evidenceId,
|
||||
"environment-manifest needs a capture_browser_scenario capture with capture.network.response_body or header_values selected.",
|
||||
viewKind === "environment-state"
|
||||
? "environment-state needs a capture_browser_scenario capture with capture.network.response_body selected."
|
||||
: "environment-manifest needs a capture_browser_scenario capture with capture.network.response_body or header_values selected.",
|
||||
),
|
||||
);
|
||||
return ok(scenarioCaptureRecords(parsed.data, "manifest"));
|
||||
|
||||
@@ -437,7 +437,7 @@ export const projectWebNetworkCaptureView = (
|
||||
path: ["view"],
|
||||
reason: "invalid_value",
|
||||
message:
|
||||
"Historical captures support summary, capture record pages, qlog fact pages and admitted api-surface or environment-manifest views.",
|
||||
"Historical captures support summary, capture record pages, qlog fact pages and admitted api-surface, environment-manifest or environment-state views.",
|
||||
},
|
||||
]),
|
||||
);
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import { isSafeNumber } from "lossless-json";
|
||||
import { jsonValueSchema, type JsonValue } from "./jsonValue.js";
|
||||
import { safeParseJson } from "./safeJson.js";
|
||||
|
||||
/** Whether native JSON conversion and serialization preserve the recorded number. */
|
||||
export const isExactJsonNumber = (
|
||||
@@ -6,3 +8,17 @@ export const isExactJsonNumber = (
|
||||
source: string | undefined,
|
||||
): boolean =>
|
||||
source !== undefined && isSafeNumber(source) && !Object.is(value, -0);
|
||||
|
||||
/** Parse JSON text whose every number survives native conversion exactly; undefined otherwise. */
|
||||
export const parseExactJson = (text: string): JsonValue | undefined => {
|
||||
const parsed = safeParseJson(text, (_key, value, context) => {
|
||||
// The native parser supplies each primitive's original spelling. Share
|
||||
// HAR's exact-number policy rather than accepting an already-rounded ID.
|
||||
if (typeof value === "number" && !isExactJsonNumber(value, context?.source))
|
||||
throw new RangeError("JSON numeric conversion would lose recorded data");
|
||||
return value;
|
||||
});
|
||||
if (!parsed.ok) return undefined;
|
||||
const value = jsonValueSchema.safeParse(parsed.value);
|
||||
return value.success ? value.data : undefined;
|
||||
};
|
||||
@@ -302,7 +302,11 @@ it("runs built CLI and real stdio SDK with exact evidence parity, selection, ret
|
||||
covered_ranges: [{ start: "10", end: "15" }],
|
||||
});
|
||||
// An unsupported view over an existing retained parent proves lookup without pretending qlog is HAR.
|
||||
for (const kind of ["api-surface", "environment-manifest"] as const) {
|
||||
for (const kind of [
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
"environment-state",
|
||||
] as const) {
|
||||
const retained = await call("inspect_analysis_view", {
|
||||
source: { kind: "retained-evidence", evidence_id: mcp.evidence_id },
|
||||
view: { kind },
|
||||
|
||||
@@ -22,6 +22,7 @@ import {
|
||||
analysisViewJavaScriptAnalysisWithSource,
|
||||
analysisViewNativeEvidence,
|
||||
environmentManifestCaptureFixture,
|
||||
environmentStateCaptureFixture,
|
||||
} from "../../fixtures/analysisView.js";
|
||||
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
|
||||
import { ghidraFunctionDossier } from "../../../src/domain/ghidraValues.fixture.js";
|
||||
@@ -618,6 +619,52 @@ cliTest(
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"extracts an environment state through the compiled CLI with recorded values",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-environment-state-cli-");
|
||||
const path = join(root, "view.json");
|
||||
await writeFile(
|
||||
path,
|
||||
JSON.stringify({
|
||||
source: {
|
||||
kind: "inline",
|
||||
evidence: analysisViewCaptureEvidence(
|
||||
environmentStateCaptureFixture(),
|
||||
),
|
||||
},
|
||||
view: {
|
||||
kind: "environment-state",
|
||||
template_rules: [
|
||||
"after-uid-literal",
|
||||
"opaque-token",
|
||||
"sibling-variance",
|
||||
],
|
||||
},
|
||||
}),
|
||||
);
|
||||
const result = await cli.run({
|
||||
arguments: ["inspect-analysis-view", path, "--json"],
|
||||
});
|
||||
expect(result.exitCode, result.stdout).toBe(0);
|
||||
const evidence = parseEvidence(result.json);
|
||||
expect(evidence.confidence).toBe("inferred");
|
||||
expect(evidence.normalized_result).toMatchObject({
|
||||
kind: "environment-state",
|
||||
origin: "http://grafana.example",
|
||||
timeline: "recorded-start-time",
|
||||
types: expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
collection_template: "/api/folders",
|
||||
id_field: "uid",
|
||||
}),
|
||||
]),
|
||||
});
|
||||
for (const value of ["probe folder", "probe@example.test"])
|
||||
expect(result.stdout).toContain(value);
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"decodes a Grafana-shaped HAR and projects its environment manifest through the compiled workflows",
|
||||
async ({ cli }) => {
|
||||
|
||||
@@ -43,6 +43,7 @@ import {
|
||||
analysisViewObjectDenseCapture,
|
||||
analysisViewNativeEvidence,
|
||||
environmentManifestCaptureFixture,
|
||||
environmentStateCaptureFixture,
|
||||
} from "../../fixtures/analysisView.js";
|
||||
import { createTestBinarySession } from "../../fixtures/binarySession.js";
|
||||
import { BINARY_LAYOUT_TEST_PROVIDER } from "../../fixtures/binaryDiagnostics/layout.js";
|
||||
@@ -1167,6 +1168,72 @@ it.each(PUBLISHED_EXAMPLES)(
|
||||
},
|
||||
);
|
||||
|
||||
it("extracts an environment state identically from retained and portable MCP Evidence", async () => {
|
||||
const { client, session } = await connect();
|
||||
const { ajv, inputSchema, outputSchema } = await advertisedMcpToolSchemas(
|
||||
client,
|
||||
"inspect_analysis_view",
|
||||
);
|
||||
const capture = environmentStateCaptureFixture();
|
||||
const parent = analysisViewCaptureEvidence(capture);
|
||||
expect(session.recordEvidence(parent).ok).toBe(true);
|
||||
const view = {
|
||||
kind: "environment-state",
|
||||
template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"],
|
||||
};
|
||||
const outputs = [];
|
||||
for (const source of [
|
||||
{ kind: "inline", evidence: parent },
|
||||
{ kind: "retained-evidence", evidence_id: parent.evidence_id },
|
||||
]) {
|
||||
expect(ajv.validate(inputSchema, { source, view })).toBe(true);
|
||||
const response = await client.callTool({
|
||||
name: "inspect_analysis_view",
|
||||
arguments: { source, view },
|
||||
});
|
||||
expect(response.isError).not.toBe(true);
|
||||
expect(
|
||||
ajv.validate(outputSchema, response.structuredContent),
|
||||
JSON.stringify(ajv.errors),
|
||||
).toBe(true);
|
||||
const evidence = parseEvidence(response.structuredContent);
|
||||
expect(evidence.confidence).toBe("inferred");
|
||||
expect(evidence.normalized_result).toMatchObject({
|
||||
kind: "environment-state",
|
||||
views: expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
tool: "get_folders_by_uid",
|
||||
shape: "item",
|
||||
}),
|
||||
]),
|
||||
});
|
||||
expect(session.evidenceById(evidence.evidence_id)).toEqual(evidence);
|
||||
outputs.push(evidence.normalized_result);
|
||||
}
|
||||
expect(outputs[0]).toEqual(outputs[1]);
|
||||
const hidden = createEvidence(
|
||||
{
|
||||
path: capture.artifact.path,
|
||||
sha256: capture.artifact.sha256!,
|
||||
format: "file",
|
||||
},
|
||||
{ id: "fixture", name: "fixture", version: "1" },
|
||||
{
|
||||
operation: "inspect_web_network_capture",
|
||||
parameters: { sensitive_value_count: 1 },
|
||||
result: capture,
|
||||
confidence: "observed",
|
||||
},
|
||||
);
|
||||
const refused = await client.callTool({
|
||||
name: "inspect_analysis_view",
|
||||
arguments: { source: { kind: "inline", evidence: hidden }, view },
|
||||
});
|
||||
expect(parseMcpToolError(refused).error.code).toBe("unsupported_target");
|
||||
expect(JSON.stringify(refused)).not.toContain("probe folder");
|
||||
await client.ping();
|
||||
});
|
||||
|
||||
it("projects an environment manifest identically from retained and portable MCP Evidence", async () => {
|
||||
const { client, session } = await connect();
|
||||
const { ajv, inputSchema, outputSchema } = await advertisedMcpToolSchemas(
|
||||
|
||||
Vendored
+13
-3
@@ -1,4 +1,9 @@
|
||||
import { apiSurfaceHar, environmentManifestHar } from "./historicalHar.js";
|
||||
import type { JsonValue } from "../../src/domain/jsonValue.js";
|
||||
import {
|
||||
apiSurfaceHar,
|
||||
environmentManifestHar,
|
||||
environmentStateHar,
|
||||
} from "./historicalHar.js";
|
||||
import { webNetworkCaptureSchema } from "../../src/domain/webNetworkCapture.js";
|
||||
import { BINARY_LAYOUT_TEST_PROVIDER } from "./binaryDiagnostics/layout.js";
|
||||
import { binaryLayoutFixture } from "./binaryDiagnostics/layout.js";
|
||||
@@ -290,9 +295,10 @@ export const analysisViewObjectDenseCapture = (headerCount = 100_000) => {
|
||||
};
|
||||
|
||||
/** Capture parent decoded from the Grafana-shaped HAR with producer ordinals intact. */
|
||||
export const environmentManifestCaptureFixture = () => {
|
||||
export const environmentManifestCaptureFixture = (
|
||||
entries: readonly JsonValue[] = environmentManifestHar().log.entries,
|
||||
) => {
|
||||
const capture = analysisViewCaptureFixture();
|
||||
const entries = environmentManifestHar().log.entries;
|
||||
capture.artifact.path = "/fixtures/environment-manifest.har";
|
||||
capture.total_records = entries.length;
|
||||
capture.records = entries.map((reported, ordinal) => ({
|
||||
@@ -303,3 +309,7 @@ export const environmentManifestCaptureFixture = () => {
|
||||
}));
|
||||
return capture;
|
||||
};
|
||||
|
||||
/** The same capture with distinct start times, as the environment-state view orders it. */
|
||||
export const environmentStateCaptureFixture = () =>
|
||||
environmentManifestCaptureFixture(environmentStateHar().log.entries);
|
||||
Vendored
+11
@@ -309,3 +309,14 @@ export const environmentManifestHar = () => ({
|
||||
),
|
||||
},
|
||||
});
|
||||
|
||||
/** The Grafana-shaped HAR with one second between requests, so its timeline orders reads before the writes that follow them. */
|
||||
export const environmentStateHar = () => {
|
||||
const har = environmentManifestHar();
|
||||
const start = Date.parse("2026-10-17T18:00:00.000Z");
|
||||
har.log.entries = har.log.entries.map((entry, index) => ({
|
||||
...entry,
|
||||
startedDateTime: new Date(start + index * 1000).toISOString(),
|
||||
}));
|
||||
return har;
|
||||
};
|
||||
Reference in new issue
Block a user