Files
reverse-engineering-anything/scripts/lib/advertised-catalog-summary.mjs
T
morluto bd46f3c1ee test(mcp): compile advertised patterns with the regex engines clients run (#10982)
* test(mcp): compile advertised patterns with the regex engines clients run

List the built package's tools over stdio in every input, output and
catalog profile and compile each distinct advertised regular expression
with V8 (plain, u, v), Go regexp (RE2), C++ RE2, Python re, and Rust regex
and fancy-regex through the Rust jsonschema crate. A refusal names the
engine, its error, and each tool, schema path and profile.

The Python engines are pinned, hash-checked wheels; Go is the host
toolchain with GOTOOLCHAIN=local. A missing engine is reported as not
verified, and REA_REQUIRE_SCHEMA_ENGINES=1 makes it a failure.

The pattern owner now also reports patternProperties keys, which clients
compile as well.

Refs #10830

* test(mcp): run schema engine helpers and listings under the process owner

* test(mcp): load advertised schemas in the validators clients build (#10983)

* test(mcp): load advertised schemas in the validators clients build

Build a validator for every distinct advertised input and output schema
with the Rust jsonschema crate's defaults, which check the declared draft's
meta-schema and formats and compile every subschema. --complete adds Python
jsonschema's check_schema, which jsonschema.validate runs in Python clients;
it takes several seconds, so it is opt-in.

Refusals are grouped by reason, so one shared pattern that breaks many
schemas is reported once with its sites.

Ajv strict mode and OpenAI strict mode are documented as not covered, with
the reasons.

Refs #10830

* ci(mcp): require the client schema engines on every code change (#10984)

* ci(mcp): require the client schema engines on every code change

Run scripts/verify-advertised-schemas.mjs at the end of the build job, after
its artifacts upload, with REA_REQUIRE_SCHEMA_ENGINES=1 so a missing engine
fails rather than passes. Go is the runner's toolchain; the Python engines
install into a virtual environment cached by the requirements file's digest
and interpreter version. Full qualification adds --complete.

Refs #10830

* test(ci): require the client schema engine step in code builds

* test(mcp): record a reviewable summary of the advertised catalog (#10985)

* test(mcp): record a reviewable summary of the advertised catalog

Commit scripts/verify/advertised-schemas/summary.json: each advertised
pattern with the tool schemas that carry it, and a digest of every tool's
schema per input or output profile. The advertised schema check compares it
with the packaged listings it already makes and fails on any difference,
printing lines such as '119 tool schemas gained pattern /.../', so a change
that reaches many tools at once is visible in review.
--update-summary regenerates it.

Refs #10830

* docs(release): check the packed catalog with client engines before publishing (#10986)

* docs(release): check the packed catalog with client engines before publishing

Add a pre-publish step to the release guide: install the packed candidate
into an empty prefix and run verify-advertised-schemas.mjs against it with
--package-root, requiring the RE2 (Go regexp and C++ RE2) and Rust (regex,
fancy-regex and jsonschema) engines. CI checks each commit's build; this
covers the archive that will be published.

Refs #10830

* docs(release): check the exact qualified archive and clean up afterwards

* test(mcp): regenerate the advertised catalog summary from current main
2026-11-15 21:38:23 +08:00

112 lines
3.6 KiB
JavaScript

import { createHash } from "node:crypto";
// A reviewable summary of what the packaged server advertises. A change that
// touches many tools' schemas at once, such as a pattern added to a shared
// path schema (#10830), shows up in review as one line per pattern instead of
// being buried in generated contracts.
const SHOWN_NAMES = 8;
const sorted = (record) =>
Object.fromEntries(
Object.entries(record).sort(([left], [right]) =>
left < right ? -1 : left > right ? 1 : 0,
),
);
/**
* Summarize listings from every profile: each advertised pattern with the
* tool schemas that advertise it, and a digest of each tool's schema per
* input or output profile.
*/
export const summarizeAdvertisedCatalogs = (
catalogs,
advertisedSchemaPatterns,
) => {
const patterns = new Map();
const tools = new Map();
const schemas = catalogs.flatMap(({ profile, tools: listed }) =>
listed.flatMap((tool) =>
[
["input", profile.input],
["output", profile.output],
].flatMap(([kind, selected]) => {
const schema = tool[`${kind}Schema`];
return schema === undefined
? []
: [{ name: tool.name, kind, group: `${kind} ${selected}`, schema }];
}),
),
);
for (const { name, kind, group, schema } of schemas) {
for (const { pattern } of advertisedSchemaPatterns(schema)) {
if (!patterns.has(pattern)) patterns.set(pattern, new Set());
patterns.get(pattern).add(`${name} ${kind}`);
}
if (!tools.has(name)) tools.set(name, {});
tools.get(name)[group] = createHash("sha256")
.update(JSON.stringify(schema))
.digest("hex")
.slice(0, 16);
}
return {
patterns: sorted(
Object.fromEntries(
[...patterns].map(([pattern, sites]) => [pattern, [...sites].sort()]),
),
),
tools: sorted(
Object.fromEntries(
[...tools].map(([name, digests]) => [name, sorted(digests)]),
),
),
};
};
const names = (values) =>
values.length > SHOWN_NAMES
? `${values.slice(0, SHOWN_NAMES).join(", ")} and ${String(values.length - SHOWN_NAMES)} more`
: values.join(", ");
/** Review lines for every difference between two summaries. */
export const describeSummaryChanges = (recorded, current) => {
const lines = [];
const recordedPatterns = recorded?.patterns ?? {};
for (const pattern of new Set([
...Object.keys(recordedPatterns),
...Object.keys(current.patterns),
])) {
const before = new Set(recordedPatterns[pattern] ?? []);
const after = new Set(current.patterns[pattern] ?? []);
const gained = [...after].filter((site) => !before.has(site));
const lost = [...before].filter((site) => !after.has(site));
if (gained.length > 0)
lines.push(
`${String(gained.length)} tool schemas gained pattern /${pattern}/: ${names(gained)}`,
);
if (lost.length > 0)
lines.push(
`${String(lost.length)} tool schemas lost pattern /${pattern}/: ${names(lost)}`,
);
}
const recordedTools = recorded?.tools ?? {};
const changed = new Map();
for (const tool of new Set([
...Object.keys(recordedTools),
...Object.keys(current.tools),
]))
for (const group of new Set([
...Object.keys(recordedTools[tool] ?? {}),
...Object.keys(current.tools[tool] ?? {}),
]))
if (recordedTools[tool]?.[group] !== current.tools[tool]?.[group]) {
if (!changed.has(group)) changed.set(group, []);
changed.get(group).push(tool);
}
for (const [group, changedTools] of changed)
lines.push(
`${String(changedTools.length)} tools changed their ${group} schema: ${names(changedTools)}`,
);
return lines;
};