feat(snapshots)!: persist the target identity as the snapshot target (#11551)

* feat(snapshots)!: persist the target identity as the snapshot target

The snapshot target is now `{ kind, identity }`, where `identity` is the
FileTargetIdentity that `binaryTargetIdentity` produces and `kind` is the
binary kind query IDs commit to. The per-format target projection and
`snapshotTargetIdentity` are deleted; the schema checks the format's
selection, architecture and kind instead of keeping one union arm per format.

Entry execution subjects use the Evidence subject shape (file or explicit
unknown), so a snapshot entry commits to the same identity, including the
selection, as its Evidence record. Workflow entries take the subject of the
workflow's own Evidence. Replay no longer re-derives the selection from the
open target.

`FileTargetSelection` gains `universal-slice` (`offset`, `size`, and the FAT
`cpu_type`/`cpu_subtype` when target admission supplied the FAT table, else
null). `binaryTargetIdentity` produces it for `mach-o-universal` targets, so
Evidence and snapshots for different slices of one container no longer share
an identity.

Refs #11417

BREAKING CHANGE: analysis snapshot files change shape: `target` is
`{ kind, identity: { kind: "file", sha256, format, architecture, selection } }`
and entry `execution.subject` is the Evidence subject
(`{ identity, name, local_path }`, never null). Snapshot binding mismatch
details report `target.snapshot`/`target.selected` in the same shape. Evidence
IDs of universal Mach-O targets change because their subject identity now
carries the selected slice. Recreate saved snapshots and Evidence bundles of
universal Mach-O targets.

* docs(evidence): name the universal Mach-O slice selection in the subject contract

* fix(snapshots): read the snapshot target identity in the Ghidra verification scripts

The real Ghidra snapshot checks (lifecycle, RISC-V, MIPS, PowerPC) read
snapshot.target.sha256 and .architecture, which moved under
snapshot.target.identity, so they would fail after saving a valid snapshot.

* feat(evidence)!: name live browser page Evidence by its page URL (#11559)

Browser page observations (inspect_web_page, analyze_web_bundle,
observe_web_session, discover_webmcp_tools, capture_web_screenshot,
observe_web_execution, inspect_web_event_listeners) named an unknown
(`not-observed`) subject although the provider reports the page URL.

`TargetIdentity` gains a `url` kind: the reported URL with userinfo removed
and its origin (`null` for an opaque origin). The schema refuses a URL that
carries userinfo or an origin the URL does not have.
`browserUrlTargetIdentity` in the browser URL owner builds it from the
provider's reported URL; a withheld or unparseable URL stays unknown
(`not-observed`). Target lists and comparisons still name no single
document. Analysis views keep their parent's URL subject, as they already
keep a file subject.

BREAKING CHANGE: Evidence IDs of live browser page observations and their
analysis views change, and their subject is
`{ "kind": "url", "url", "origin" }` with `name` and `local_path` null.
Recreate stored Evidence for these tools. This lands inside Evidence format
revision 2 under the release hold (#11417); no `evidence_format` bump.
This commit is contained in:
morluto authored and GitHub committed 2026-11-16 07:04:34 +08:00
1 parent f6df353002
commit ff0176256a
30 files changed
+678 -396

No files matched your search

+9 -4
View File
@@ -1023,12 +1023,17 @@ it, or with another revision, as an integrity mismatch whose recovery is to
recreate the observations with the current release; re-exporting an earlier
bundle does not update its format or identities. Revision 2 names the subject
by its target identity, `subject.identity` (`kind: "file"`, `sha256`, `format`,
`architecture` and `selection`: the raw or DOL entry point or the ESP flash
offsets that select what was analysed, otherwise `null`), and the Evidence ID
`architecture` and `selection`: the raw or DOL entry point, the ESP flash
offsets, or the universal Mach-O slice (`offset`, `size`, and the FAT
`cpu_type` and `cpu_subtype`, `null` when the FAT table was not admitted) that
select what was analysed, otherwise `null`), and the Evidence ID
commits to that identity; `subject.name` and `subject.local_path` are display
facts outside the ID. The subject has no separate `digest`, `format` or
`architecture` members; read them from `subject.identity`. Every record has a
subject: one whose target REA did not identify names
`architecture` members; read them from `subject.identity`. A live browser page
observation names the page's reported URL,
`{ "kind": "url", "url": …, "origin": … }`, with userinfo removed and `origin`
`null` for an opaque origin; its `name` and `local_path` are `null`. Every
record has a subject: one whose target REA did not identify names
`{ "kind": "unknown", "reason": … }`, where the reason is
`derived-from-linked-evidence` (comparisons and workflows, whose sources are
committed through `evidence_links`), `not-observed` or `unsupported-subject`,
@@ -96,7 +96,7 @@ export async function verifyLegacyGhidraReferenceSnapshot(
analysisProfile: legacyProfile,
limitations: observed.limitations,
locations: observed.locations,
subject: nativeTarget,
subject: evidence.subject,
},
}),
],
@@ -70,7 +70,7 @@ export async function verifyLegacyValueScopeSnapshot(
analysisProfile: legacyProfile,
limitations: observed.limitations,
locations: observed.locations,
subject: nativeTarget,
subject: evidence.subject,
},
}),
],
@@ -52,7 +52,7 @@ export async function verifyGhidraSnapshotLifecycle(
const snapshot = parseAnalysisSnapshot(
JSON.parse(snapshotBytes.toString("utf8")),
);
assert.equal(snapshot.target.sha256, target.sha256);
assert.equal(snapshot.target.identity.sha256, target.sha256);
assert.notEqual(original.procedure.name, edited.procedure.name);
const runId = (await call("binary_session")).analysis_run.run_id;
const api = await call("inspect_native_api", { procedure: entry });
+2 -2
View File
@@ -347,7 +347,7 @@ try {
const snapshot = parseAnalysisSnapshot(
JSON.parse(await readFile(snapshotPath, "utf8")),
);
assert.equal(snapshot.target.architecture, "mips");
assert.equal(snapshot.target.identity.architecture, "mips");
const parameters = snapshot.binding.analysis_profile.parameters;
assert.equal(
parameters.language_id,
@@ -660,7 +660,7 @@ try {
const snapshot = parseAnalysisSnapshot(
JSON.parse(await readFile(snapshotPath, "utf8")),
);
assert.equal(snapshot.target.architecture, "mips");
assert.equal(snapshot.target.identity.architecture, "mips");
assert.deepEqual(
snapshot.binding.analysis_profile.parameters.mips_elf,
expectedMipsElfProfile(independent.mips),
+1 -1
View File
@@ -234,7 +234,7 @@ try {
const snapshot = parseAnalysisSnapshot(
JSON.parse(await readFile(snapshotPath, "utf8")),
);
assert.equal(snapshot.target.architecture, "powerpc");
assert.equal(snapshot.target.identity.architecture, "powerpc");
assert.equal(
snapshot.binding.analysis_profile.parameters.language_id,
"PowerPC:BE:32:default",
+2 -2
View File
@@ -368,8 +368,8 @@ try {
const snapshot = parseAnalysisSnapshot(
JSON.parse(await readFile(snapshotPath, "utf8")),
);
assert.equal(snapshot.target.architecture, "riscv");
assert.equal(snapshot.target.sha256, fixture.sha256);
assert.equal(snapshot.target.identity.architecture, "riscv");
assert.equal(snapshot.target.identity.sha256, fixture.sha256);
assertProfile(snapshot.binding.analysis_profile);
for (const record of snapshot.evidence_bundle.records)
assertEvidence(parseEvidence(record));
+47 -3
View File
@@ -1,4 +1,7 @@
import { unknownTarget } from "../domain/targetIdentity.js";
import {
unknownTarget,
type NonFileTargetIdentity,
} from "../domain/targetIdentity.js";
import { isDeepStrictEqual } from "node:util";
import type { ProviderIdentity } from "../ports/AnalysisProvider.js";
import {
@@ -6,7 +9,10 @@ import {
type Evidence,
type EvidenceObservation,
} from "../domain/evidence.js";
import { sanitizeBrowserUrl } from "../domain/browserObservation.js";
import {
browserUrlTargetIdentity,
sanitizeBrowserUrl,
} from "../domain/browserObservation.js";
import type {
InspectWebPageInput,
ListBrowserTargetsInput,
@@ -96,7 +102,7 @@ export const createBrowserEvidence = (
const comparison =
operation === "compare_web_captures" ||
operation === "compare_web_screenshots";
return createEvidence(unknownTarget("not-observed"), provider, {
return createEvidence(browserSubject(operation, observed), provider, {
predicateType: browserPredicate(operation),
operation,
parameters: {
@@ -135,6 +141,44 @@ export const createBrowserEvidence = (
});
};
/**
* The document one page observation is about: the URL the provider reported
* for it. Target lists and comparisons observe no single document.
*/
const browserSubject = (
operation: BrowserEvidenceOperation,
observed: Exclude<
BrowserEvidenceResult,
WebBundleAcquisitionPartialObservation
>,
): NonFileTargetIdentity => {
switch (operation) {
case "list_browser_targets":
case "compare_web_captures":
case "compare_web_screenshots":
return unknownTarget("not-observed");
case "analyze_web_bundle":
// A completed analysis reports its captured page; an acquisition that
// stopped before analysis reports the page inspection itself.
if ("capture" in observed && "target_url" in observed.capture)
return browserUrlTargetIdentity(observed.capture.target_url);
if ("target" in observed && "url" in observed.target)
return browserUrlTargetIdentity(observed.target.url);
break;
case "observe_web_session":
if ("target" in observed && "initial_url" in observed.target)
return browserUrlTargetIdentity(observed.target.initial_url);
break;
case "inspect_web_page":
case "discover_webmcp_tools":
case "capture_web_screenshot":
if ("target" in observed && "url" in observed.target)
return browserUrlTargetIdentity(observed.target.url);
break;
}
throw new Error(`Browser ${operation} result does not report its document`);
};
const browserParameters = (
input: BrowserEvidenceInput,
result: BrowserEvidenceResult,
+14 -10
View File
@@ -1,4 +1,4 @@
import { unknownTarget } from "../domain/targetIdentity.js";
import { browserUrlTargetIdentity } from "../domain/browserObservation.js";
import { z } from "zod";
import type { ExecutionOptions } from "../ports/AnalysisProvider.js";
import type { WebRuntimePort } from "../ports/WebRuntimePort.js";
@@ -156,15 +156,19 @@ export class WebRuntimeService {
),
);
return completeProviderEvidence(
createEvidence(unknownTarget("not-observed"), this.provider.identity(), {
predicateType: `rea.${operation.replaceAll("_", "-")}`,
operation,
parameters: input,
result,
confidence: "observed",
authority: "external-service",
limitations: result.limitations,
}),
createEvidence(
browserUrlTargetIdentity(result.target.initial_url),
this.provider.identity(),
{
predicateType: `rea.${operation.replaceAll("_", "-")}`,
operation,
parameters: input,
result,
confidence: "observed",
authority: "external-service",
limitations: result.limitations,
},
),
options?.signal,
);
}
+1 -1
View File
@@ -484,7 +484,7 @@ export const workflowSnapshotRecord = (
analysisProfile: analysisProfileSchema.parse(evidence.analysis_profile),
limitations: evidence.limitations,
locations: evidence.locations,
subject: evidenceSubjectTarget(evidence.subject) ?? null,
subject: evidence.subject,
},
};
};
@@ -28,6 +28,7 @@ import {
createEvidence,
evidenceFileTarget,
evidenceSubjectDisplay,
evidenceSubjectInput,
evidenceSubjectTarget,
rebindEvidenceToTargetPath,
type Evidence,
@@ -373,7 +374,12 @@ const createAnalysisViewEvidence = (
parent: Evidence,
result: AnalysisViewResult,
): Evidence => {
const target = evidenceSubjectTarget(parent.subject);
// A view names the target its parent observed (a file or a page URL); a
// parent that names no target is committed through the view's link.
const subject =
parent.subject.identity.kind === "unknown"
? unknownTarget("derived-from-linked-evidence")
: evidenceSubjectInput(parent.subject);
const nativeFunctionView =
result.kind === "native" ||
(result.kind === "summary" &&
@@ -387,7 +393,7 @@ const createAnalysisViewEvidence = (
(result.kind === "summary" &&
isNativeObservationOperation(result.parent_operation));
return createEvidence(
target ?? unknownTarget("derived-from-linked-evidence"),
subject,
ANALYSIS_VIEW_PROVIDER,
{
predicateType: "rea.analysis-view",
@@ -8,8 +8,13 @@ import {
ANALYSIS_SNAPSHOT_PROFILE,
ANALYSIS_SNAPSHOT_PROVIDER,
ANALYSIS_SNAPSHOT_TARGET,
snapshotExecution,
} from "../../domain/analysisSnapshot.fixture.js";
import { createEvidence, immutableEvidence } from "../../domain/evidence.js";
import {
createEvidence,
evidenceSubject,
immutableEvidence,
} from "../../domain/evidence.js";
import { createAnalysisExecutionEvidence } from "../../domain/analysisObservation.js";
import {
createEvidenceBundle,
@@ -37,6 +42,7 @@ import {
isSourceEvidenceRetainable,
} from "./AnalysisSnapshotCache.js";
import { EvidenceLedger } from "../investigation/EvidenceLedger.js";
import { nativeCallGraphSchema } from "../../domain/native/nativeCallGraph.js";
it.each(["provider", "workflow"] as const)(
@@ -73,7 +79,11 @@ it.each(["provider", "workflow"] as const)(
execution: { ...execution, analysisProfile },
};
if (kind === "provider") cache.record(input);
else cache.recordWorkflow(input);
else
cache.recordWorkflow({
...input,
execution: snapshotExecution(input.execution),
});
const entries = () =>
kind === "provider" ? cache.entries() : cache.workflowEntries();
const expected = entries();
@@ -123,10 +133,12 @@ it("exports independent nested query, workflow and Evidence payloads", () => {
operation: "analyze_function",
parameters,
execution: {
...createAnalysisExecution(result, REA_WORKFLOW_PROVIDER, {
subject: target,
rawResult: result,
}),
...snapshotExecution(
createAnalysisExecution(result, REA_WORKFLOW_PROVIDER, {
subject: target,
rawResult: result,
}),
),
analysisProfile: workflowProfile,
},
});
@@ -226,7 +238,10 @@ it.each(["provider", "workflow"] as const)(
profile,
operation,
parameters,
execution: { ...execution, analysisProfile: evidenceProfile },
execution: snapshotExecution({
...execution,
analysisProfile: evidenceProfile,
}),
});
expect(ledger.record(evidence).ok).toBe(true);
const before = cache.export(target, profile, ledger.export());
@@ -285,14 +300,14 @@ it.each(["provider", "workflow"] as const)(
if (!after.ok) throw after.error;
const entries =
kind === "provider" ? after.value.entries : after.value.workflow_entries;
expect(entries[0]?.execution.subject?.path).toBe("/relocated/fixture");
expect(entries[0]?.execution.subject.local_path).toBe("/relocated/fixture");
expect(entries[0]?.execution.result).toBe("original");
expect(after.value.evidence_bundle.records).toContainEqual(relocated);
const originalEntries =
kind === "provider"
? before.value.entries
: before.value.workflow_entries;
expect(originalEntries[0]?.execution.subject?.path).toBe(target.path);
expect(originalEntries[0]?.execution.subject.local_path).toBe(target.path);
},
);
@@ -327,7 +342,11 @@ it.each(["provider", "workflow"] as const)(
};
if (kind === "provider")
cache.record({ ...input, operation: "procedure_info" });
else cache.recordWorkflow(input);
else
cache.recordWorkflow({
...input,
execution: snapshotExecution(input.execution),
});
expect(
ledger.record(
createEvidence(subject ?? unknownTarget("not-observed"), provider, {
@@ -356,7 +375,9 @@ it.each(["provider", "workflow"] as const)(
const saved =
kind === "provider" ? snapshot.entries : snapshot.workflow_entries;
expect(saved).toHaveLength(1);
expect(saved[0]?.execution.subject?.sha256).toBe(target.sha256);
expect(saved[0]?.execution.subject.identity).toMatchObject({
sha256: target.sha256,
});
expect(snapshot.evidence_bundle.records).toHaveLength(1);
expect(
kind === "provider" ? cache.entries() : cache.workflowEntries(),
@@ -402,9 +423,11 @@ it("counts newly retained direct and workflow bindings when staging", () => {
operation: "trace_feature",
parameters: { query: "feature" },
execution: {
...createAnalysisExecution("workflow", REA_WORKFLOW_PROVIDER, {
subject: target,
}),
...snapshotExecution(
createAnalysisExecution("workflow", REA_WORKFLOW_PROVIDER, {
subject: target,
}),
),
analysisProfile: workflowProfile,
},
});
@@ -532,11 +555,13 @@ it("binds an omitted subject to the selected target in snapshot replay", () => {
]),
);
if (!snapshot.ok) throw snapshot.error;
expect(snapshot.value.entries[0]?.execution.subject?.path).toBe(target.path);
expect(snapshot.value.entries[0]?.execution.subject.local_path).toBe(
target.path,
);
const reparsed = parseAnalysisSnapshot(
JSON.parse(serializeAnalysisSnapshot(snapshot.value)),
);
expect(reparsed.entries[0]?.execution.subject?.path).toBe(target.path);
expect(reparsed.entries[0]?.execution.subject.local_path).toBe(target.path);
cache.clear();
const imported = cache.import(
reparsed,
@@ -583,9 +608,11 @@ describe("analysis snapshot cache capacity", () => {
const target = ANALYSIS_SNAPSHOT_TARGET;
const profile = ANALYSIS_SNAPSHOT_PROFILE;
const execution = {
...createAnalysisExecution("workflow", REA_WORKFLOW_PROVIDER, {
subject: target,
}),
...snapshotExecution(
createAnalysisExecution("workflow", REA_WORKFLOW_PROVIDER, {
subject: target,
}),
),
analysisProfile: workflowAnalysisProfile(profile),
};
const workflowInput = {
@@ -673,7 +700,7 @@ describe("analysis snapshot cache capacity", () => {
cache.recordWorkflow({
...workflowInput,
target: otherTarget,
execution: { ...execution, subject: otherTarget },
execution: { ...execution, subject: evidenceSubject(otherTarget) },
});
expect(cache.entries()).toEqual([]);
expect(cache.workflowEntries()).toHaveLength(1);
@@ -1233,10 +1260,12 @@ it.each([
operation,
parameters,
execution: {
...createAnalysisExecution(oldResult, REA_WORKFLOW_PROVIDER, {
subject: target,
rawResult: oldResult,
}),
...snapshotExecution(
createAnalysisExecution(oldResult, REA_WORKFLOW_PROVIDER, {
subject: target,
rawResult: oldResult,
}),
),
analysisProfile: oldProfile,
},
});
@@ -1316,9 +1345,11 @@ it.each([1, 2])(
limitations: [],
};
const execution = {
...createAnalysisExecution(oldResult, REA_WORKFLOW_PROVIDER, {
subject: target,
}),
...snapshotExecution(
createAnalysisExecution(oldResult, REA_WORKFLOW_PROVIDER, {
subject: target,
}),
),
analysisProfile: oldProfile,
};
cache.recordWorkflow({ target, profile, operation, parameters, execution });
+38 -53
View File
@@ -10,10 +10,17 @@ import {
import { NoBinaryOpenError } from "../../domain/configurationErrors.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import {
evidenceSubject,
evidenceSubjectDisplay,
evidenceSubjectTarget,
rebindEvidenceSubjectTargetPath,
type Evidence,
evidenceSubjectDisplay,
type EvidenceSubjectTarget,
} from "../../domain/evidence.js";
import {
sameTargetIdentity,
unknownTarget,
} from "../../domain/targetIdentity.js";
import { err, ok, type Result } from "../../domain/result.js";
import {
analysisQueryId,
@@ -25,7 +32,6 @@ import {
snapshotBindingMismatch,
snapshotMatchesProfile,
snapshotTarget,
snapshotTargetIdentity,
parseAnalysisSnapshot,
immutableAnalysisSnapshot,
type AnalysisSnapshot,
@@ -49,10 +55,6 @@ import {
snapshotReplayBinding,
type SnapshotReplayBinding,
} from "./AnalysisSnapshotReplayBinding.js";
import {
binaryTargetSelection,
sameTargetIdentity,
} from "../../domain/targetIdentity.js";
import { createInvestigationEvidenceClosureChecker } from "../investigation/InvestigationEvidence.js";
const RELATIONSHIP_OPERATIONS: ReadonlySet<AnalysisOperation> = new Set([
@@ -96,6 +98,12 @@ const DOCUMENT_SCOPED_OPERATIONS: ReadonlySet<string> = new Set(
/** Maximum duplicated result bindings retained across direct and workflow queries. */
const SNAPSHOT_CACHE_ENTRY_CEILING = 10_000;
/** A file target without its recorded display name, which follows the replayed path. */
const withoutDisplayName = ({
name: _name,
...target
}: EvidenceSubjectTarget): EvidenceSubjectTarget => target;
const replayDocumentSelected = (
operation: AnalysisOperation,
parameters: Readonly<Record<string, JsonValue>>,
@@ -232,10 +240,7 @@ export class AnalysisSnapshotCache {
snapshot = immutableAnalysisSnapshot(snapshot);
if (
(this.#target !== undefined &&
!sameTargetIdentity(
snapshotTargetIdentity(this.#target),
snapshotTargetIdentity(snapshot.target),
)) ||
!sameTargetIdentity(this.#target.identity, snapshot.target.identity)) ||
(this.#binding !== undefined &&
!snapshotMatchesProfile(
this.#binding,
@@ -358,35 +363,31 @@ export class AnalysisSnapshotCache {
previous.parameters,
);
const entry = this.#entries.get(queryId);
const entryIdentity = entry?.execution.subject.identity;
if (
entry !== undefined &&
entry.execution.subject !== null &&
entryIdentity?.kind === "file" &&
evidenceMatchesEntry(previous, entry, header)
)
this.#entries.set(queryId, {
...entry,
execution: {
...entry.execution,
subject: {
...entry.execution.subject,
path: display.local_path,
},
subject: { identity: entryIdentity, ...display },
},
});
const workflow = this.#workflowEntries.get(queryId);
const workflowIdentity = workflow?.execution.subject.identity;
if (
workflow !== undefined &&
workflow.execution.subject !== null &&
workflowIdentity?.kind === "file" &&
evidenceMatchesWorkflowEntry(previous, workflow, header)
)
this.#workflowEntries.set(queryId, {
...workflow,
execution: {
...workflow.execution,
subject: {
...workflow.execution.subject,
path: display.local_path,
},
subject: { identity: workflowIdentity, ...display },
},
});
}
@@ -401,18 +402,21 @@ export class AnalysisSnapshotCache {
if (target === undefined || profile === undefined)
return err(new NoBinaryOpenError());
try {
const snapshotTargetIdentity = snapshotTarget(target);
const persistedTarget = snapshotTarget(target);
const namesTarget = (subject: Evidence["subject"]): boolean =>
subject.identity.kind === "file" &&
subject.identity.sha256 === target.sha256;
const binding = snapshotBinding(profile);
// In-memory replay can preserve unbound and child-artifact observations.
// A persisted target snapshot can authenticate only rows whose Evidence
// has that target subject; the record owner applies the same scope to its bundle.
const entries = [...this.#entries.values()]
.filter((entry) => entry.execution.subject?.sha256 === target.sha256)
.filter((entry) => namesTarget(entry.execution.subject))
.sort((left, right) =>
compareUnicodeCodePoints(left.query_id, right.query_id),
);
const workflows = [...this.#workflowEntries.values()]
.filter((entry) => entry.execution.subject?.sha256 === target.sha256)
.filter((entry) => namesTarget(entry.execution.subject))
.sort((left, right) =>
compareUnicodeCodePoints(left.query_id, right.query_id),
);
@@ -420,7 +424,7 @@ export class AnalysisSnapshotCache {
// payloads here only adds another full materialization.
return ok(
parseAnalysisSnapshot({
target: snapshotTargetIdentity,
target: persistedTarget,
binding,
entries,
workflow_entries: workflows,
@@ -507,7 +511,7 @@ export class AnalysisSnapshotCache {
readonly limitations: readonly string[];
readonly relationshipCoverage?: AnalysisExecution["relationshipCoverage"];
readonly locations: AnalysisExecution["locations"];
readonly subject: NonNullable<AnalysisExecution["subject"]> | null;
readonly subject: Evidence["subject"];
};
}): void {
this.select(input.target, input.profile);
@@ -556,27 +560,14 @@ export class AnalysisSnapshotCache {
this.#replayBindings.set(queryId, restored);
binding = restored;
}
const cachedSubject = cached.execution.subject;
// The query ID commits to the snapshot target, including its selection,
// so a cached subject naming the target's bytes made the same selection.
// Snapshot subjects do not persist it yet (ADR target-identity slice 2d).
const selection =
cachedSubject?.sha256 === target.sha256
? binaryTargetSelection(target)
: null;
// Replay names the file at the current path; its display name follows
// that path, as the live observation's would.
const cachedTarget = evidenceSubjectTarget(cached.execution.subject);
const subject =
cachedSubject == null
cachedTarget === undefined
? null
: rebindEvidenceSubjectTargetPath(
{
path: cachedSubject.path,
sha256: cachedSubject.sha256,
format: cachedSubject.format,
...(cachedSubject.architecture === null
? {}
: { architecture: cachedSubject.architecture }),
...(selection === null ? {} : { selection }),
},
withoutDisplayName(cachedTarget),
target,
);
const execution: AnalysisExecution = structuredClone({
@@ -656,17 +647,11 @@ export class AnalysisSnapshotCache {
parameters,
execution: {
...execution,
subject:
subject: evidenceSubject(
execution.subject === undefined
? {
path: target.path,
sha256: target.sha256,
format: target.format,
...(target.architecture === undefined
? {}
: { architecture: target.architecture }),
}
: execution.subject,
? target
: (execution.subject ?? unknownTarget("not-observed")),
),
},
});
this.#entries.set(entry.query_id, entry);
@@ -172,7 +172,10 @@ describe("target format vocabularies", () => {
analysisSnapshotSchema.shape.target.parse(
roundTrip(snapshotTarget(target)),
),
).toMatchObject({ format: target.format, kind: target.kind });
).toMatchObject({
kind: target.kind,
identity: { format: target.format },
});
expect(binaryTargetSummary(target)).toMatchObject({
format: target.format,
kind: target.kind,
+12
View File
@@ -1,5 +1,7 @@
import { createAnalysisProfile } from "./analysisProfile.js";
import type { BinaryTarget } from "./binaryTargetTypes.js";
import { evidenceSubject, type EvidenceSubjectTarget } from "./evidence.js";
import { unknownTarget } from "./targetIdentity.js";
/** Immutable target shared by analysis-snapshot contract tests. */
export const ANALYSIS_SNAPSHOT_TARGET: BinaryTarget = {
@@ -23,3 +25,13 @@ export const ANALYSIS_SNAPSHOT_PROFILE = createAnalysisProfile(
ANALYSIS_SNAPSHOT_PROVIDER,
{ loader: "mach-o-arm64" },
);
/** A provider execution as a snapshot entry records it: with its Evidence subject. */
export const snapshotExecution = <
Execution extends { readonly subject?: EvidenceSubjectTarget | null },
>(
execution: Execution,
) => ({
...execution,
subject: evidenceSubject(execution.subject ?? unknownTarget("not-observed")),
});
+57 -51
View File
@@ -16,7 +16,8 @@ import {
ANALYSIS_SNAPSHOT_PROVIDER,
ANALYSIS_SNAPSHOT_TARGET,
} from "./analysisSnapshot.fixture.js";
import { createEvidence } from "./evidence.js";
import { createEvidence, evidenceSubject } from "./evidence.js";
import { unknownTarget } from "./targetIdentity.js";
import { createEvidenceBundle } from "./evidenceBundle.js";
import {
RAW_ONLY_ARCHITECTURES,
@@ -46,12 +47,7 @@ const snapshotWithHistoricalEvidence = () => {
limitations: [],
relationship_coverage: null,
locations: [],
subject: {
path: ANALYSIS_SNAPSHOT_TARGET.path,
sha256: ANALYSIS_SNAPSHOT_TARGET.sha256,
format: ANALYSIS_SNAPSHOT_TARGET.format,
architecture: ANALYSIS_SNAPSHOT_TARGET.architecture ?? null,
},
subject: current.subject,
},
};
return {
@@ -121,22 +117,25 @@ describe("analysis snapshot target identity", () => {
it.each(RAW_ONLY_ARCHITECTURES)(
"keeps %s confined to executable raw snapshots",
(architecture) => {
const target = {
const identity = {
kind: "file",
sha256: ANALYSIS_SNAPSHOT_TARGET.sha256,
kind: "executable",
architecture,
};
expect(
analysisSnapshotSchema.shape.target.safeParse({
...target,
format: "pe",
kind: "executable",
identity: { ...identity, format: "pe", selection: null },
}).success,
).toBe(false);
expect(
analysisSnapshotSchema.shape.target.safeParse({
...target,
format: "raw-binary",
entry_point: "0x8000",
kind: "executable",
identity: {
...identity,
format: "raw-binary",
selection: { kind: "entry-point", entry_point: "0x8000" },
},
}).success,
).toBe(true);
},
@@ -177,16 +176,22 @@ describe("analysis snapshot target identity", () => {
workflow_entries: [],
evidence_bundle: createEvidenceBundle([]),
});
expect(parsed.target).toHaveProperty("entry_point", target.entryPoint);
const { entry_point: _entry, ...legacy } = {
...snapshot,
expect(parsed.target.identity.selection).toEqual({
kind: "entry-point",
entry_point: target.entryPoint,
};
});
// A raw target without its entry selection is not one REA can open.
expect(
analysisSnapshotSchema.safeParse({ ...parsed, target: legacy }).success,
analysisSnapshotSchema.safeParse({
...parsed,
target: {
...parsed.target,
identity: { ...parsed.target.identity, selection: null },
},
}).success,
).toBe(false);
expect(snapshotTarget(ANALYSIS_SNAPSHOT_TARGET)).not.toHaveProperty(
"entry_point",
expect(snapshotTarget(ANALYSIS_SNAPSHOT_TARGET).identity.selection).toBe(
null,
);
});
});
@@ -220,8 +225,11 @@ describe("analysis snapshot nested subject contract", () => {
if (entry === undefined) throw new Error("Fixture has no snapshot entry");
const malformedSubject = {
...entry.execution.subject,
format: "pe",
architecture,
identity: {
...entry.execution.subject.identity,
format: "pe",
architecture,
},
};
expect(
analysisSnapshotSchema.safeParse({
@@ -309,18 +317,20 @@ describe("analysis snapshot contract", () => {
limitations: [],
relationship_coverage: null,
locations: [],
subject: target,
subject: evidence.subject,
},
},
],
workflow_entries: [],
evidence_bundle: createEvidenceBundle([evidence]),
});
expect(parsed.target).toMatchObject({
expect(parsed.target.identity).toMatchObject({
format,
architecture,
});
expect(parsed.entries[0]?.execution.subject?.format).toBe(format);
expect(fileIdentity(parsed.entries[0]?.execution.subject).format).toBe(
format,
);
expect(
fileIdentity(parsed.evidence_bundle.records[0]?.subject).format,
).toBe(format);
@@ -329,7 +339,7 @@ describe("analysis snapshot contract", () => {
it("rejects a snapshot subject path containing NUL", () => {
const binding = snapshotBinding(ANALYSIS_SNAPSHOT_PROFILE);
const entry = {
const entry = (localPath: string) => ({
query_id: `query_${"0".repeat(64)}`,
operation: "analyze_function",
parameters: {},
@@ -341,22 +351,28 @@ describe("analysis snapshot contract", () => {
relationship_coverage: null,
locations: [],
subject: {
path: "a\u0000b",
sha256: "a".repeat(64),
format: "elf" as const,
architecture: null,
identity: {
kind: "file",
sha256: "a".repeat(64),
format: "elf",
architecture: null,
selection: null,
},
name: "b",
local_path: localPath,
},
},
};
expect(
});
const parsed = (localPath: string) =>
analysisSnapshotSchema.safeParse({
target: snapshotTarget(ANALYSIS_SNAPSHOT_TARGET),
binding,
entries: [entry],
entries: [entry(localPath)],
workflow_entries: [],
evidence_bundle: createEvidenceBundle([]),
}).success,
).toBe(false);
}).success;
expect(parsed("/fixture/ab")).toBe(true);
expect(parsed("a\u0000b")).toBe(false);
});
it("accepts snapshots with more than ten thousand analysis entries", () => {
@@ -372,7 +388,7 @@ describe("analysis snapshot contract", () => {
limitations: [],
relationship_coverage: null,
locations: [],
subject: null,
subject: evidenceSubject(unknownTarget("not-observed")),
},
};
@@ -631,12 +647,7 @@ it("binds raw Evidence only when its entry point matches the snapshot target", (
limitations: [],
relationship_coverage: null,
locations: [],
subject: {
path: selected.path,
sha256: selected.sha256,
format: "raw-binary",
architecture: "arm",
},
subject: evidenceSubject(selected),
},
},
],
@@ -691,12 +702,7 @@ describe("analysis snapshot Evidence binding", () => {
const parameters = { address: "0x1000", document: "main" };
const result = { name: "main" };
const rawResult = { name: "provider-main" };
const subject = {
path: ANALYSIS_SNAPSHOT_TARGET.path,
sha256: ANALYSIS_SNAPSHOT_TARGET.sha256,
format: ANALYSIS_SNAPSHOT_TARGET.format,
architecture: ANALYSIS_SNAPSHOT_TARGET.architecture ?? null,
};
const subject = evidenceSubject(ANALYSIS_SNAPSHOT_TARGET);
const entry = {
query_id: analysisQueryId(target, binding, "address_name", parameters),
operation: "address_name",
@@ -769,8 +775,8 @@ describe("analysis snapshot Evidence binding", () => {
},
(altered: typeof snapshot) => {
const subject = altered.entries[0]?.execution.subject;
if (subject !== null && subject !== undefined)
subject.path = "/different/path";
if (subject === undefined) throw new TypeError("missing fixture entry");
Reflect.set(subject, "local_path", "/different/path");
},
(altered: typeof snapshot) => {
const entry = altered.entries[0];
+79 -152
View File
@@ -3,17 +3,15 @@ import { createHash } from "node:crypto";
import canonicalize from "canonicalize";
import { z } from "zod";
import { withEvidenceSubjectDeclaration } from "./evidenceSubject.js";
import {
compareUnicodeCodePoints,
compositeKey,
} from "./unicodeCodePointOrder.js";
import { localPathStringSchema } from "./localPath.js";
import {
binaryTargetIdentity,
fileTargetIdentitySchema,
sameTargetIdentity,
type FileTargetIdentity,
type FileTargetSelection,
} from "./targetIdentity.js";
import {
@@ -38,11 +36,10 @@ import {
import {
evidenceFileTarget,
evidenceLocationSchema,
evidenceSubjectTarget,
evidenceSubjectSchema,
rebindEvidenceToTargetPath,
type Evidence,
type EvidenceLocation,
type EvidenceSubjectTarget,
} from "./evidence.js";
import {
jsonObjectSchema,
@@ -52,10 +49,6 @@ import {
import { digestSchema, prefixedDigestSchema } from "./../domain/digests.js";
import { canonicalJsonDigestSteps } from "./canonicalJsonDigestSteps.js";
import { freezeJsonSnapshot } from "./immutableJson.js";
import {
binaryArchitectureSchema,
rawBinaryEntryPointSchema,
} from "./rawBinary.js";
import {
relationshipCoverageSchema,
type RelationshipCoverage,
@@ -101,65 +94,66 @@ const formatSchema = z.enum([
"uimage",
]);
const kindSchema = z.enum(["executable", "database", "archive", "artifact"]);
const subjectBaseSchema = z.object({
path: localPathStringSchema,
sha256: digestSchema,
});
const subjectSchema = withEvidenceSubjectDeclaration(subjectBaseSchema);
const targetIdentitySchema = z.object({
sha256: digestSchema,
kind: kindSchema,
architecture: architectureSchema.nullable(),
});
type SnapshotFormat = z.infer<typeof formatSchema>;
/** The selection each format's open targets make inside their bytes. */
const SNAPSHOT_SELECTION_KINDS: Partial<
Record<SnapshotFormat, FileTargetSelection["kind"]>
> = {
"raw-binary": "entry-point",
"gamecube-dol": "entry-point",
// The caller's inputs select how the dump is read, so they are identity.
"esp-flash-image": "esp-flash",
"mach-o-universal": "universal-slice",
};
const RAW_ONLY = new Set<string>(RAW_ONLY_ARCHITECTURES);
/** Whether a format admits an architecture, as `binaryTargetIdentity` produces it. */
const snapshotArchitectureAllowed = (
format: SnapshotFormat,
architecture: string | null,
): boolean =>
format === "raw-binary"
? architecture !== null
: format === "gamecube-dol"
? architecture === "powerpc"
: format === "esp-flash-image"
? architecture === null
: architecture === null || !RAW_ONLY.has(architecture);
/**
* The persisted target: the path-free identity `binaryTargetIdentity`
* produces, plus the binary kind that query IDs commit to.
*/
const targetSchema = z
.discriminatedUnion("format", [
targetIdentitySchema.extend({
format: z.enum(
formatSchema.options.filter(
(format) =>
format !== "raw-binary" &&
format !== "gamecube-dol" &&
format !== "esp-flash-image",
),
),
architecture: architectureSchema
.exclude([...RAW_ONLY_ARCHITECTURES])
.nullable(),
.strictObject({
kind: kindSchema,
identity: fileTargetIdentitySchema.extend({
format: formatSchema,
architecture: architectureSchema.nullable(),
}),
targetIdentitySchema.extend({
format: z.literal("raw-binary"),
kind: z.literal("executable"),
architecture: binaryArchitectureSchema,
entry_point: rawBinaryEntryPointSchema,
}),
targetIdentitySchema.extend({
format: z.literal("gamecube-dol"),
kind: z.literal("executable"),
architecture: z.literal("powerpc"),
entry_point: rawBinaryEntryPointSchema,
}),
// The caller's inputs select how the dump is read, so they are identity.
targetIdentitySchema.extend({
format: z.literal("esp-flash-image"),
kind: z.literal("archive"),
architecture: z.null(),
flash_offset: z.number().int().min(0).max(0xffff_ffff).nullable(),
partition_table_offset: z
.number()
.int()
.min(0)
.max(0xffff_ffff)
.nullable(),
}),
])
.superRefine((target, context) => {
})
.superRefine(({ kind, identity }, context) => {
// Query IDs commit to the kind, so a kind the format cannot have would
// admit entries that no live lookup or export of this target can reach.
if (target.kind !== BINARY_TARGET_FORMAT_KINDS[target.format])
if (kind !== BINARY_TARGET_FORMAT_KINDS[identity.format])
context.addIssue({
code: "custom",
path: ["kind"],
message: `Snapshot target kind ${target.kind} does not match format ${target.format}`,
message: `Snapshot target kind ${kind} does not match format ${identity.format}`,
});
if (!snapshotArchitectureAllowed(identity.format, identity.architecture))
context.addIssue({
code: "custom",
path: ["identity", "architecture"],
message: `Snapshot target architecture ${String(identity.architecture)} does not apply to format ${identity.format}`,
});
const expected = SNAPSHOT_SELECTION_KINDS[identity.format] ?? null;
if ((identity.selection?.kind ?? null) !== expected)
context.addIssue({
code: "custom",
path: ["identity", "selection"],
message:
expected === null
? `Snapshot target format ${identity.format} makes no selection`
: `Snapshot target format ${identity.format} requires a ${expected} selection`,
});
});
const bindingSchema = z
@@ -191,7 +185,7 @@ const entrySchema = z.object({
limitations: z.array(z.string()),
relationship_coverage: relationshipCoverageSchema.nullable(),
locations: z.array(evidenceLocationSchema),
subject: subjectSchema.nullable(),
subject: evidenceSubjectSchema,
}),
});
const workflowEntrySchema = entrySchema.extend({
@@ -375,7 +369,7 @@ const snapshotContentDigest = (
const metadata = jsonObjectSchema.parse({
limitations: execution.limitations,
locations: execution.locations,
subject: executionSubjectContent(execution.subject ?? undefined),
subject: subjectContent(execution.subject),
relationship_coverage: execution.relationship_coverage,
});
return contentDigest({
@@ -385,25 +379,11 @@ const snapshotContentDigest = (
});
};
/** The execution subject facts a snapshot content digest commits to. */
const executionSubjectContent = (
subject:
| {
readonly path: string;
readonly sha256: string;
readonly format: string;
readonly architecture?: string | null;
}
| undefined,
) =>
subject === undefined
? null
: {
local_path: subject.path,
sha256: subject.sha256,
format: subject.format,
architecture: subject.architecture ?? null,
};
/** The subject facts a snapshot content digest commits to; the display name is not one. */
const subjectContent = (subject: Evidence["subject"]) => ({
identity: subject.identity,
local_path: subject.local_path,
});
const evidenceContentDigest = (evidence: Evidence): string => {
return contentDigest({
@@ -425,7 +405,7 @@ interface SnapshotExecution {
readonly limitations: readonly string[];
readonly relationshipCoverage?: RelationshipCoverage | null | undefined;
readonly locations: readonly EvidenceLocation[];
readonly subject: EvidenceSubjectTarget | null;
readonly subject: Evidence["subject"];
}
type SnapshotExecutionProjection = Pick<
@@ -451,13 +431,7 @@ const snapshotExecutionProjection = (
limitations: execution.limitations,
relationship_coverage: execution.relationshipCoverage ?? null,
locations: execution.locations,
subject:
execution.subject === null
? null
: {
...execution.subject,
architecture: execution.subject.architecture ?? null,
},
subject: execution.subject,
});
const canonicalJson = (value: unknown): string => {
@@ -468,39 +442,12 @@ const canonicalJson = (value: unknown): string => {
return encoded;
};
/** Project a binary into the path-free identity used for cache invalidation. */
/** The persisted snapshot target of an open binary: its target identity and kind. */
export const snapshotTarget = (target: BinaryTarget): AnalysisSnapshotTarget =>
target.format === "raw-binary"
? {
sha256: target.sha256,
kind: target.kind,
architecture: target.architecture,
format: target.format,
entry_point: target.entryPoint,
}
: target.format === "gamecube-dol"
? {
sha256: target.sha256,
kind: target.kind,
architecture: target.architecture,
format: target.format,
entry_point: target.entryPoint,
}
: target.format === "esp-flash-image"
? {
sha256: target.sha256,
kind: target.kind,
architecture: null,
format: target.format,
flash_offset: target.espFlash.flashOffset,
partition_table_offset: target.espFlash.partitionTableOffset,
}
: {
sha256: target.sha256,
kind: target.kind,
architecture: target.architecture ?? null,
format: target.format,
};
targetSchema.parse({
kind: target.kind,
identity: binaryTargetIdentity(target),
});
/** Build the immutable provider/profile binding used by a cached snapshot. */
export const snapshotBinding = (
@@ -511,35 +458,12 @@ export const snapshotBinding = (
analysis_profile: profile,
});
/** The target identity a persisted snapshot target names. */
export const snapshotTargetIdentity = (
target: AnalysisSnapshotTarget,
): FileTargetIdentity => ({
kind: "file",
sha256: target.sha256,
format: target.format,
architecture: target.architecture,
selection:
target.format === "raw-binary" || target.format === "gamecube-dol"
? { kind: "entry-point", entry_point: target.entry_point }
: target.format === "esp-flash-image"
? {
kind: "esp-flash",
flash_offset: target.flash_offset,
partition_table_offset: target.partition_table_offset,
}
: null,
});
/** Compare an open binary with the immutable target identity in a snapshot. */
export const snapshotMatchesTarget = (
snapshot: AnalysisSnapshotTarget,
target: BinaryTarget,
): boolean =>
sameTargetIdentity(
snapshotTargetIdentity(snapshot),
binaryTargetIdentity(target),
);
sameTargetIdentity(snapshot.identity, binaryTargetIdentity(target));
/** Compare a snapshot binding with the selected concrete analysis profile. */
export const snapshotMatchesProfile = (
@@ -824,7 +748,10 @@ const restoreSnapshot = (
providerRows: readonly Extract<PreparedEntry, { kind: "provider" }>[],
workflowRows: readonly Extract<PreparedEntry, { kind: "workflow" }>[],
): AnalysisSnapshot => {
const scopedEvidence = evidenceBundleForTarget(bundle, header.target.sha256);
const scopedEvidence = evidenceBundleForTarget(
bundle,
header.target.identity.sha256,
);
if (
scopedEvidence.records.length !== bundle.records.length ||
scopedEvidence.unknowns.length !== bundle.unknowns.length
@@ -843,7 +770,7 @@ const restoreSnapshot = (
if (
evidence.predicate_type !== "rea.analysis" ||
evidence.authority !== "shipped-artifact" ||
evidenceFileTarget(evidence)?.sha256 !== header.target.sha256 ||
evidenceFileTarget(evidence)?.sha256 !== header.target.identity.sha256 ||
evidence.analysis_profile === null ||
(evidence.confidence !== "observed" && evidence.confidence !== "derived")
)
@@ -977,8 +904,8 @@ const subjectNamesSnapshotTarget = (
target: AnalysisSnapshotTarget,
): boolean => {
const file = evidenceFileTarget(evidence);
if (file === undefined || file.sha256 !== target.sha256) return false;
const expected = snapshotTargetIdentity(target);
const expected = target.identity;
if (file === undefined || file.sha256 !== expected.sha256) return false;
return file.format !== expected.format || sameTargetIdentity(file, expected);
};
@@ -1090,7 +1017,7 @@ const evidenceContentMetadata = (evidence: Evidence) => {
limitations: evidence.limitations,
relationship_coverage: evidence.relationship_coverage,
locations: evidence.locations,
subject: executionSubjectContent(evidenceSubjectTarget(evidence.subject)),
subject: subjectContent(evidence.subject),
};
};
+39
View File
@@ -6,9 +6,11 @@ import {
browserOriginSchema,
recordedBrowserOriginSchema,
recordedBrowserObservationEndpointSchema,
browserUrlTargetIdentity,
sanitizeEndpointCandidate,
sanitizeBrowserUrl,
} from "./browserObservation.js";
import { unknownTarget, urlTargetIdentitySchema } from "./targetIdentity.js";
import { webPageInspectionSchema } from "./browserObservationSchemas.js";
import { createWebPageInspection } from "../../tests/support/browser/webPageInspection.js";
@@ -345,3 +347,40 @@ describe("page inspection producer facts", () => {
);
});
});
describe("browser URL target identity", () => {
it("names the reported page URL without userinfo, keeping its query and fragment", () => {
const identity = browserUrlTargetIdentity(
"https://user:pass@app.example.test/path?token=secret#part",
);
expect(identity).toEqual({
kind: "url",
url: "https://app.example.test/path?token=secret#part",
origin: "https://app.example.test",
});
expect(urlTargetIdentitySchema.parse(identity)).toEqual(identity);
const backslash = browserUrlTargetIdentity(
String.raw`https:\\user:pass@app.example.test/path`,
);
expect(urlTargetIdentitySchema.parse(backslash)).toMatchObject({
origin: "https://app.example.test",
});
expect(JSON.stringify(backslash)).not.toContain("user:pass");
});
it("names an opaque origin as null", () => {
expect(browserUrlTargetIdentity("about:blank")).toEqual({
kind: "url",
url: "about:blank",
origin: null,
});
});
it("does not identify a document whose URL was withheld or is not a URL", () => {
for (const value of ["[unsupported-url]", ""])
expect(browserUrlTargetIdentity(value)).toEqual(
unknownTarget("not-observed"),
);
});
});
+24
View File
@@ -9,6 +9,11 @@ import {
parseRecordedTransportUrl,
parseUrlQueryParameterNames,
} from "./recordedTransportUrl.js";
import {
unknownTarget,
type UnknownTargetIdentity,
type UrlTargetIdentity,
} from "./targetIdentity.js";
const parseExactOrigin = (value: string): string | undefined => {
let url: URL;
@@ -336,6 +341,25 @@ const credentialFreeHref = (parsed: URL): string => {
return parsed.href;
};
/**
* The target identity of the live document URL a browser provider reported:
* the URL without userinfo, and its origin. A value that is not a URL, such
* as the placeholder for a document outside the selected origins, does not
* identify the document.
*/
export const browserUrlTargetIdentity = (
value: string,
): UrlTargetIdentity | UnknownTargetIdentity => {
const observation = observeBrowserUrl(value);
return observation.kind === "parsed"
? {
kind: "url",
url: observation.sanitized.url,
origin: observation.sanitized.origin,
}
: unknownTarget("not-observed");
};
/** Normalize endpoint candidates without discarding local query or fragment data. */
export const sanitizeEndpointCandidate = (value: string): string => {
return sanitizeBrowserUrl(value).url;
+13 -10
View File
@@ -15,8 +15,8 @@ import { nulFreeStringSchema } from "./localPath.js";
import {
binaryTargetIdentity,
fileTargetSelectionSchema,
unknownTargetIdentitySchema,
type UnknownTargetIdentity,
nonFileTargetIdentitySchema,
type NonFileTargetIdentity,
sameTargetIdentity,
type FileTargetIdentity,
type FileTargetSelection,
@@ -63,6 +63,7 @@ const SELECTION_FORMATS: Readonly<
> = {
"entry-point": ["raw-binary", "gamecube-dol"],
"esp-flash": ["esp-flash-image"],
"universal-slice": ["mach-o-universal"],
};
/** The file identity an Evidence subject commits to, with its format-owned declaration. */
const subjectIdentitySchema = withEvidenceSubjectDeclaration(
@@ -88,13 +89,15 @@ const fileSubjectSchema = z.strictObject({
name: z.string().min(1),
local_path: nulFreeStringSchema,
});
/** A subject REA did not identify has no display facts to show. */
const unknownSubjectSchema = z.strictObject({
identity: unknownTargetIdentitySchema,
/** A subject that names no file (a URL, or an unknown target) has no file display facts. */
const nonFileSubjectSchema = z.strictObject({
identity: nonFileTargetIdentitySchema,
name: z.null(),
local_path: z.null(),
});
const subjectSchema = z.union([fileSubjectSchema, unknownSubjectSchema]);
const subjectSchema = z.union([fileSubjectSchema, nonFileSubjectSchema]);
/** The Evidence subject, for persistence formats that embed one. */
export const evidenceSubjectSchema = subjectSchema;
/** Source location attached to an evidence observation. */
/** Source-location schema shared by evidence-bearing persistence formats. */
export const evidenceLocationSchema = z.discriminatedUnion("kind", [
@@ -239,11 +242,11 @@ export interface EvidenceSubjectTarget {
readonly name?: string;
}
/** What a producer names as an Evidence subject: a file target, or an explicit unknown. */
/** What a producer names as an Evidence subject: a file target, or a non-file identity such as a URL or an explicit unknown. */
export type EvidenceSubjectInput =
| EvidenceSubjectTarget
| BinaryTarget
| UnknownTargetIdentity;
| NonFileTargetIdentity;
type FileEvidenceSubject = z.infer<typeof fileSubjectSchema>;
@@ -271,13 +274,13 @@ export const evidenceSubjectTarget = (
: { selection: subject.identity.selection }),
};
/** Name the same subject in a derived observation: the file target, or the same unknown. */
/** Name the same subject in a derived observation: the file target, or the same non-file identity. */
export const evidenceSubjectInput = (
subject: Evidence["subject"],
): EvidenceSubjectInput =>
isFileSubject(subject)
? (evidenceSubjectTarget(subject) as EvidenceSubjectTarget)
: (subject.identity as UnknownTargetIdentity);
: (subject.identity as NonFileTargetIdentity);
/** The one projection of a producer's subject input into its target identity. */
export const evidenceSubjectTargetIdentity = (
+74 -15
View File
@@ -1,10 +1,11 @@
import { describe, expect, it } from "vitest";
import { snapshotTarget, snapshotTargetIdentity } from "./analysisSnapshot.js";
import type { BinaryTarget } from "./binaryTargetTypes.js";
import {
binaryTargetIdentity,
sameTargetIdentity,
targetIdentityKey,
urlTargetIdentitySchema,
type FileTargetIdentity,
} from "./targetIdentity.js";
@@ -42,7 +43,7 @@ describe("target identity key", () => {
entryPoint: "0x8000000",
entryPointSource: "0x08000000",
};
const identity = snapshotTargetIdentity(snapshotTarget(raw));
const identity = binaryTargetIdentity(raw);
expect(identity).toEqual({
kind: "file",
sha256: SHA256,
@@ -50,9 +51,7 @@ describe("target identity key", () => {
architecture: "arm",
selection: { kind: "entry-point", entry_point: "0x8000000" },
});
const moved = snapshotTargetIdentity(
snapshotTarget({ ...raw, entryPoint: "0x9000000" }),
);
const moved = binaryTargetIdentity({ ...raw, entryPoint: "0x9000000" });
expect(sameTargetIdentity(identity, moved)).toBe(false);
const flash: BinaryTarget = {
@@ -62,7 +61,7 @@ describe("target identity key", () => {
format: "esp-flash-image",
espFlash: { selected: true, flashOffset: 0, partitionTableOffset: null },
};
const flashIdentity = snapshotTargetIdentity(snapshotTarget(flash));
const flashIdentity = binaryTargetIdentity(flash);
expect(flashIdentity.selection).toEqual({
kind: "esp-flash",
flash_offset: 0,
@@ -71,22 +70,82 @@ describe("target identity key", () => {
expect(
sameTargetIdentity(
flashIdentity,
snapshotTargetIdentity(
snapshotTarget({
...flash,
espFlash: { ...flash.espFlash, flashOffset: 0x1000 },
}),
),
binaryTargetIdentity({
...flash,
espFlash: { ...flash.espFlash, flashOffset: 0x1000 },
}),
),
).toBe(false);
// The path is display, not identity.
expect(
sameTargetIdentity(
flashIdentity,
snapshotTargetIdentity(
snapshotTarget({ ...flash, path: "/elsewhere" }),
),
binaryTargetIdentity({ ...flash, path: "/elsewhere" }),
),
).toBe(true);
});
it("separates the slices of one universal Mach-O container", () => {
const universal: BinaryTarget = {
path: "/fixture/universal",
sha256: SHA256,
kind: "executable",
format: "mach-o-universal",
architecture: "arm64",
availableArchitectures: ["x86_64", "arm64"],
universalSlice: { offset: 0x8000, size: 0x4000 },
universalSlices: [
{ offset: 0x4000, size: 0x2000, cpuType: 0x0100_0007, cpuSubtype: 3 },
{ offset: 0x8000, size: 0x4000, cpuType: 0x0100_000c, cpuSubtype: 0 },
],
};
const arm64 = binaryTargetIdentity(universal);
expect(arm64.selection).toEqual({
kind: "universal-slice",
offset: 0x8000,
size: 0x4000,
cpu_type: 0x0100_000c,
cpu_subtype: 0,
});
const x86 = binaryTargetIdentity({
...universal,
architecture: "x86_64",
universalSlice: { offset: 0x4000, size: 0x2000 },
});
expect(sameTargetIdentity(arm64, x86)).toBe(false);
// Without the admitted FAT table the slice range still selects, and the
// CPU declaration stays explicitly unknown.
const { universalSlices: _slices, ...unadmitted } = universal;
expect(binaryTargetIdentity(unadmitted).selection).toEqual({
kind: "universal-slice",
offset: 0x8000,
size: 0x4000,
cpu_type: null,
cpu_subtype: null,
});
});
});
describe("URL target identity", () => {
const page = {
kind: "url",
url: "https://app.example.test/orders?id=1",
origin: "https://app.example.test",
} as const;
it("separates two pages of one origin", () => {
expect(urlTargetIdentitySchema.parse(page)).toEqual(page);
expect(
sameTargetIdentity(page, { ...page, url: "https://app.example.test/" }),
).toBe(false);
});
it.each([
["userinfo", { ...page, url: "https://user:pass@app.example.test/" }],
["a URL that does not parse", { ...page, url: "not a url" }],
["an origin the URL does not have", { ...page, origin: "https://x.test" }],
["a null origin for a tuple origin", { ...page, origin: null }],
])("refuses %s", (_case, identity) => {
expect(urlTargetIdentitySchema.safeParse(identity).success).toBe(false);
});
});
+94 -3
View File
@@ -3,11 +3,15 @@ import { createHash } from "node:crypto";
import canonicalize from "canonicalize";
import { z } from "zod";
import type { BinaryTarget } from "./binaryTargetTypes.js";
import type {
BinaryTarget,
UniversalMachOSliceIdentity,
} from "./binaryTargetTypes.js";
import { digestSchema } from "./digests.js";
import { rawBinaryEntryPointSchema } from "./rawBinary.js";
const flashOffsetSchema = z.number().int().min(0).max(0xffff_ffff).nullable();
const fatCpuSchema = z.number().int().min(0).max(0xffff_ffff).nullable();
/** Persisted facts inside a file's bytes that select what was analysed. */
export const fileTargetSelectionSchema = z.discriminatedUnion("kind", [
@@ -20,6 +24,15 @@ export const fileTargetSelectionSchema = z.discriminatedUnion("kind", [
flash_offset: flashOffsetSchema,
partition_table_offset: flashOffsetSchema,
}),
// One slice of a universal Mach-O. The FAT CPU declaration is null when
// target admission did not supply the validated FAT table.
z.strictObject({
kind: z.literal("universal-slice"),
offset: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER),
size: z.number().int().min(1).max(Number.MAX_SAFE_INTEGER),
cpu_type: fatCpuSchema,
cpu_subtype: fatCpuSchema,
}),
]);
/**
@@ -78,11 +91,87 @@ export const unknownTarget = (
reason: UnknownTargetIdentity["reason"],
): UnknownTargetIdentity => ({ kind: "unknown", reason });
/** The origin a URL serializes to, or null for an opaque origin. */
const urlOrigin = (parsed: URL): string | null =>
parsed.origin === "null" ? null : parsed.origin;
/**
* A live document's URL with userinfo removed, and its origin. Userinfo is a
* transport credential, so an identity that carries it is refused rather
* than committed.
*/
export const urlTargetIdentitySchema = z
.strictObject({
kind: z.literal("url"),
url: z.string().min(1),
origin: z.string().min(1).nullable(),
})
.superRefine((identity, context) => {
const parsed = URL.parse(identity.url);
if (parsed === null) {
context.addIssue({
code: "custom",
path: ["url"],
message: "A URL target identity must be a parseable URL",
});
return;
}
if (parsed.username !== "" || parsed.password !== "")
context.addIssue({
code: "custom",
path: ["url"],
message: "A URL target identity must not carry userinfo",
});
if (urlOrigin(parsed) !== identity.origin)
context.addIssue({
code: "custom",
path: ["origin"],
message: "A URL target identity's origin must be its URL's origin",
});
});
/** A live document's URL with userinfo removed, and its origin. */
export interface UrlTargetIdentity {
readonly kind: "url";
readonly url: string;
readonly origin: string | null;
}
/**
* What an observation or cache entry is about. Tagged by `kind`; see
* docs/adr/evidence/target-identity.md for the kinds later slices add.
*/
export type TargetIdentity = FileTargetIdentity | UnknownTargetIdentity;
export type TargetIdentity =
| FileTargetIdentity
| UrlTargetIdentity
| UnknownTargetIdentity;
/** A target identity that names no file, and so has no file display facts. */
export type NonFileTargetIdentity = Exclude<TargetIdentity, FileTargetIdentity>;
/** The identities a subject without file display facts can commit to. */
export const nonFileTargetIdentitySchema = z.union([
unknownTargetIdentitySchema,
urlTargetIdentitySchema,
]);
/** The universal Mach-O slice a target selects, with its FAT CPU declaration when admitted. */
const universalSliceSelection = (
slice: { readonly offset: number; readonly size: number },
declarations: readonly UniversalMachOSliceIdentity[] | undefined,
): FileTargetSelection => {
const declared = declarations?.find(
(candidate) =>
candidate.offset === slice.offset && candidate.size === slice.size,
);
return {
kind: "universal-slice",
offset: slice.offset,
size: slice.size,
cpu_type: declared?.cpuType ?? null,
cpu_subtype: declared?.cpuSubtype ?? null,
};
};
/** The selection an open binary target makes inside its bytes, if any. */
export const binaryTargetSelection = (
@@ -96,7 +185,9 @@ export const binaryTargetSelection = (
flash_offset: target.espFlash.flashOffset,
partition_table_offset: target.espFlash.partitionTableOffset,
}
: null;
: target.format === "mach-o-universal"
? universalSliceSelection(target.universalSlice, target.universalSlices)
: null;
/** The one projection of an open binary target into its target identity. */
export const binaryTargetIdentity = (
@@ -1,4 +1,10 @@
import { readFile, stat, writeFile } from "node:fs/promises";
import {
snapshotExecution,
ANALYSIS_SNAPSHOT_PROFILE as PROFILE,
ANALYSIS_SNAPSHOT_PROVIDER as PROVIDER,
ANALYSIS_SNAPSHOT_TARGET as TARGET,
} from "../../../src/domain/analysisSnapshot.fixture.js";
import { join } from "node:path";
import { describe, expect, it, onTestFinished, vi } from "vitest";
@@ -22,11 +28,6 @@ import {
snapshotTarget,
type AnalysisSnapshot,
} from "../../../src/domain/analysisSnapshot.js";
import {
ANALYSIS_SNAPSHOT_PROFILE as PROFILE,
ANALYSIS_SNAPSHOT_PROVIDER as PROVIDER,
ANALYSIS_SNAPSHOT_TARGET as TARGET,
} from "../../../src/domain/analysisSnapshot.fixture.js";
import { createEvidence } from "../../../src/domain/evidence.js";
import { createEvidenceBundle } from "../../../src/domain/evidenceBundle.js";
import { readJsonObjectFile } from "../../../src/application/JsonInputFile.js";
@@ -44,21 +45,23 @@ const snapshotFixture = (): AnalysisSnapshot => {
binding,
operation: "address_name",
parameters: { address: "0x1000" },
execution: createAnalysisExecution(
{ name: "main", "😀": { "\uE000": "unicode" } },
PROVIDER,
{
analysisProfile: PROFILE,
rawResult: { name: "main", "😀": { "\uE000": "unicode" } },
subject: {
path: TARGET.path,
sha256: TARGET.sha256,
format: TARGET.format,
...(TARGET.architecture === undefined
? {}
: { architecture: TARGET.architecture }),
execution: snapshotExecution(
createAnalysisExecution(
{ name: "main", "😀": { "\uE000": "unicode" } },
PROVIDER,
{
analysisProfile: PROFILE,
rawResult: { name: "main", "😀": { "\uE000": "unicode" } },
subject: {
path: TARGET.path,
sha256: TARGET.sha256,
format: TARGET.format,
...(TARGET.architecture === undefined
? {}
: { architecture: TARGET.architecture }),
},
},
},
),
),
});
const evidence = createEvidence(TARGET, PROVIDER, {
@@ -34,6 +34,7 @@ import {
} from "../../../src/application/binary/AnalysisSnapshotFiles.js";
import {
createEvidence,
evidenceSubject,
parseEvidence,
type Evidence,
} from "../../../src/domain/evidence.js";
@@ -509,11 +510,11 @@ const withAlternateWorkflowProfile = (
const result = current.normalized_result;
const subject = {
path,
sha256: snapshot.target.sha256,
format: snapshot.target.format,
...(snapshot.target.architecture === null
sha256: snapshot.target.identity.sha256,
format: snapshot.target.identity.format,
...(snapshot.target.identity.architecture === null
? {}
: { architecture: snapshot.target.architecture }),
: { architecture: snapshot.target.identity.architecture }),
};
const alternateEvidence = createEvidence(subject, REA_WORKFLOW_PROVIDER, {
operation: current.operation,
@@ -537,7 +538,7 @@ const withAlternateWorkflowProfile = (
analysisProfile: alternateProfile,
limitations: current.limitations,
locations: current.locations,
subject,
subject: alternateEvidence.subject,
},
});
return {
@@ -559,7 +560,7 @@ const withEarlierHistoricalEvidence = (
createEvidence(
{
path,
sha256: snapshot.target.sha256,
sha256: snapshot.target.identity.sha256,
format: "analysis-database",
},
REA_WORKFLOW_PROVIDER,
@@ -723,7 +724,7 @@ describe("snapshot output cleanup lifecycle", () => {
if (!saved.ok) throw saved.error;
const loaded = await readAnalysisSnapshot(path);
if (!loaded.ok) throw loaded.error;
expect(loaded.value.target.sha256).toBe(opened.value.sha256);
expect(loaded.value.target.identity.sha256).toBe(opened.value.sha256);
}
} finally {
const closed = await session.close();
@@ -1538,8 +1539,10 @@ describe("snapshot replay provider identity", () => {
details: {
binding_mismatch: {
target: {
snapshot: { sha256: firstSnapshot.value.target.sha256 },
selected: { sha256: expect.any(String) },
snapshot: {
identity: { sha256: firstSnapshot.value.target.identity.sha256 },
},
selected: { identity: { sha256: expect.any(String) } },
},
profile_comparison: "not_compared",
},
@@ -1772,7 +1775,7 @@ const expectProfilelessWorkflowSnapshotRejections = async (
analysisProfile: mixedProfile,
limitations: [],
locations: [],
subject: null,
subject: evidenceSubject(unknownTarget("not-observed")),
},
}),
],
@@ -1821,20 +1824,7 @@ const expectProfilelessWorkflowSnapshotRejections = async (
originalWorkflow.execution.relationship_coverage,
}),
locations: originalWorkflow.execution.locations,
subject:
originalWorkflow.execution.subject === null
? null
: {
path: originalWorkflow.execution.subject.path,
sha256: originalWorkflow.execution.subject.sha256,
format: originalWorkflow.execution.subject.format,
...(originalWorkflow.execution.subject.architecture === null
? {}
: {
architecture:
originalWorkflow.execution.subject.architecture,
}),
},
subject: originalWorkflow.execution.subject,
},
}),
],
@@ -2059,8 +2049,8 @@ it.each([true, false])(
const historical = createEvidence(
{
path: fixture.path,
sha256: saved.value.target.sha256,
format: saved.value.target.format,
sha256: saved.value.target.identity.sha256,
format: saved.value.target.identity.format,
},
evidence.provider,
{
+33
View File
@@ -442,9 +442,32 @@ it("exposes CLI-equivalent Evidence results and session tools", async () => {
observations: { source_maps: { status: "not_requested" } },
},
});
// A page observation names the page URL the inspection reported; a target
// list names no single document.
const page = {
kind: "url",
url: webPageInspectionSchema.parse(
normalizedResultOf(inspected.structuredContent),
).target.url,
origin: browser.allowedOrigin,
};
expect(page.url).toBe(`${browser.allowedOrigin}/app?token=frame-secret`);
expect(subjectOf(listed.structuredContent)).toEqual({
identity: { kind: "unknown", reason: "not-observed" },
name: null,
local_path: null,
});
for (const result of [inspected, analyzed])
expect(subjectOf(result.structuredContent)).toEqual({
identity: page,
name: null,
local_path: null,
});
await verifySessionAndComparisonTools(connected, browser, inspected);
});
const subjectOf = (content: unknown): unknown => parseEvidence(content).subject;
const verifySessionAndComparisonTools = async (
connected: Awaited<ReturnType<typeof connectBrowser>>,
browser: FakeCdpBrowser,
@@ -550,6 +573,13 @@ const verifySessionAndComparisonTools = async (
},
});
expect(JSON.stringify(malformedComparison.content)).toContain("types");
for (const result of [observedSession, webMcp])
expect(subjectOf(result.structuredContent)).toMatchObject({
identity: { kind: "url", origin: browser.allowedOrigin },
});
expect(subjectOf(compared.structuredContent)).toMatchObject({
identity: { kind: "unknown", reason: "not-observed" },
});
const incompleteComparison = await connected.client.callTool({
name: "compare_web_captures",
arguments: { before: { inspection: capture } },
@@ -577,6 +607,9 @@ const verifyScreenshotTools = async (
},
});
expect(screenshot.isError).not.toBe(true);
expect(subjectOf(screenshot.structuredContent)).toMatchObject({
identity: { kind: "url", origin: browser.allowedOrigin },
});
const screenshotArtifact = artifactOf(screenshot.structuredContent);
const visual = await client.callTool({
name: "compare_web_screenshots",
+6 -2
View File
@@ -1,4 +1,4 @@
import { unknownTarget } from "../../../src/domain/targetIdentity.js";
import { browserUrlTargetIdentity } from "../../../src/domain/browserObservation.js";
import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { afterAll, expect, it } from "vitest";
@@ -19,7 +19,7 @@ afterAll(() => schemaValidation.clear());
const makeParent = () =>
createEvidence(
unknownTarget("not-observed"),
browserUrlTargetIdentity("https://example.test/app"),
{ id: "rea-browser", name: "REA Browser", version: "6.8.0" },
{
operation: "analyze_web_bundle",
@@ -64,6 +64,10 @@ it("projects web bundle summary views over MCP using retained and inline sources
{ toolDefinition: schemas.toolDefinition },
);
expect(summaryCall.isError, JSON.stringify(summaryCall)).not.toBe(true);
// A view names the page its parent observed.
expect(parseEvidence(summaryCall.structuredContent).subject).toEqual(
parent.subject,
);
const summaryResult = webBundleSummaryViewResultSchema.parse(
parseEvidence(summaryCall.structuredContent).normalized_result,
+10
View File
@@ -219,6 +219,16 @@ cliTest(
initial_url: `${browser.allowedOrigin}/app?caller=retained#details`,
},
});
// The observation names the page it was armed on.
expect(evidence.subject).toEqual({
identity: {
kind: "url",
url: `${browser.allowedOrigin}/app?caller=retained#details`,
origin: browser.allowedOrigin,
},
name: null,
local_path: null,
});
expect(session.evidenceById(evidence.evidence_id)).toEqual(evidence);
const cliResult = await cli.run({
arguments: [
@@ -782,7 +782,7 @@ describe("Evidence display metadata refresh", () => {
const refreshed = session.exportAnalysisSnapshot();
if (!refreshed.ok) throw refreshed.error;
if (scope === "active") {
expect(refreshed.value.entries[0]?.execution.subject?.path).toBe(
expect(refreshed.value.entries[0]?.execution.subject.local_path).toBe(
relocated.subject.local_path,
);
expect(refreshed.value.evidence_bundle.records).toContainEqual(
@@ -986,8 +986,10 @@ describe("replay of exact immutable calls", () => {
expect(projectAnalysisError(opened.error).details).toMatchObject({
binding_mismatch: {
target: {
snapshot: { sha256: snapshot.value.target.sha256 },
selected: { sha256: expect.any(String) },
snapshot: {
identity: { sha256: snapshot.value.target.identity.sha256 },
},
selected: { identity: { sha256: expect.any(String) } },
},
},
});
@@ -108,7 +108,7 @@ it.each(cases)(
analysisProfile: profile,
limitations: [],
locations: [],
subject: target,
subject: evidence.subject,
},
});
const snapshot = {
@@ -121,13 +121,13 @@ it.each(cases)(
const restored = parseAnalysisSnapshot(
JSON.parse(serializeAnalysisSnapshot(snapshot)),
);
expect(restored.target.architecture).toBe(architecture);
expect(restored.target.identity.architecture).toBe(architecture);
expect(
fileIdentity(restored.evidence_bundle.records[0]?.subject).architecture,
).toBe(architecture);
expect(restored.entries[0]?.execution.subject?.architecture).toBe(
architecture,
);
expect(
fileIdentity(restored.entries[0]?.execution.subject).architecture,
).toBe(architecture);
expect(
sessionOutputSchemas.open_binary.parse({ result: target }).result
.architecture,
+2 -1
View File
@@ -1,4 +1,5 @@
import { rm, writeFile } from "node:fs/promises";
import { snapshotExecution } from "../../src/domain/analysisSnapshot.fixture.js";
import { join } from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { projectAnalysisError } from "../../src/domain/analysisErrorProjection.js";
@@ -73,7 +74,7 @@ const reobservesExternallyMutableState = async (): Promise<void> => {
binding: snapshotBinding(profile),
operation: "procedure_pseudo_code",
parameters: query,
execution: cached,
execution: snapshotExecution(cached),
}),
);
snapshot.value.evidence_bundle = createEvidenceBundle([