Files
reverse-engineering-anything/scripts/verify/advertised-schemas/engines.py
T
morluto bd46f3c1ee test(mcp): compile advertised patterns with the regex engines clients run (#10982)
* test(mcp): compile advertised patterns with the regex engines clients run

List the built package's tools over stdio in every input, output and
catalog profile and compile each distinct advertised regular expression
with V8 (plain, u, v), Go regexp (RE2), C++ RE2, Python re, and Rust regex
and fancy-regex through the Rust jsonschema crate. A refusal names the
engine, its error, and each tool, schema path and profile.

The Python engines are pinned, hash-checked wheels; Go is the host
toolchain with GOTOOLCHAIN=local. A missing engine is reported as not
verified, and REA_REQUIRE_SCHEMA_ENGINES=1 makes it a failure.

The pattern owner now also reports patternProperties keys, which clients
compile as well.

Refs #10830

* test(mcp): run schema engine helpers and listings under the process owner

* test(mcp): load advertised schemas in the validators clients build (#10983)

* test(mcp): load advertised schemas in the validators clients build

Build a validator for every distinct advertised input and output schema
with the Rust jsonschema crate's defaults, which check the declared draft's
meta-schema and formats and compile every subschema. --complete adds Python
jsonschema's check_schema, which jsonschema.validate runs in Python clients;
it takes several seconds, so it is opt-in.

Refusals are grouped by reason, so one shared pattern that breaks many
schemas is reported once with its sites.

Ajv strict mode and OpenAI strict mode are documented as not covered, with
the reasons.

Refs #10830

* ci(mcp): require the client schema engines on every code change (#10984)

* ci(mcp): require the client schema engines on every code change

Run scripts/verify-advertised-schemas.mjs at the end of the build job, after
its artifacts upload, with REA_REQUIRE_SCHEMA_ENGINES=1 so a missing engine
fails rather than passes. Go is the runner's toolchain; the Python engines
install into a virtual environment cached by the requirements file's digest
and interpreter version. Full qualification adds --complete.

Refs #10830

* test(ci): require the client schema engine step in code builds

* test(mcp): record a reviewable summary of the advertised catalog (#10985)

* test(mcp): record a reviewable summary of the advertised catalog

Commit scripts/verify/advertised-schemas/summary.json: each advertised
pattern with the tool schemas that carry it, and a digest of every tool's
schema per input or output profile. The advertised schema check compares it
with the packaged listings it already makes and fails on any difference,
printing lines such as '119 tool schemas gained pattern /.../', so a change
that reaches many tools at once is visible in review.
--update-summary regenerates it.

Refs #10830

* docs(release): check the packed catalog with client engines before publishing (#10986)

* docs(release): check the packed catalog with client engines before publishing

Add a pre-publish step to the release guide: install the packed candidate
into an empty prefix and run verify-advertised-schemas.mjs against it with
--package-root, requiring the RE2 (Go regexp and C++ RE2) and Rust (regex,
fancy-regex and jsonschema) engines. CI checks each commit's build; this
covers the archive that will be published.

Refs #10830

* docs(release): check the exact qualified archive and clean up afterwards

* test(mcp): regenerate the advertised catalog summary from current main
2026-11-15 21:38:23 +08:00

148 lines
4.8 KiB
Python

"""Compile advertised schemas with the engines and validators clients use.
Reads ``{"patterns": [...], "schemas": [...], "complete": bool}`` as JSON
from the file named by the first argument and writes one result per engine or
validator as JSON on stdout. Pattern results index ``patterns``; schema results
index ``schemas``. A package that is not installed is reported as unavailable,
never as passing; the caller decides whether that is allowed. ``complete`` adds
the slow Python ``jsonschema`` meta-validation. Run with ``python3 -I`` so
nothing is imported from the working directory.
"""
import json
import re
import sys
import time
from importlib import metadata
def distribution_version(name):
try:
return metadata.version(name)
except metadata.PackageNotFoundError:
return None
def check(engine, kind, version, compile_value, values):
start = time.perf_counter()
failures = []
for index, value in enumerate(values):
try:
compile_value(value)
except Exception as error: # Each engine has its own error type.
failures.append({"index": index, "error": str(error)})
return {
"engine": engine,
"kind": kind,
"version": version,
"elapsed": round((time.perf_counter() - start) * 1000),
"failures": failures,
}
def unavailable(engine, kind, reason):
return {"engine": engine, "kind": kind, "unavailable": reason}
def main():
with open(sys.argv[1], encoding="utf-8") as request_file:
request = json.load(request_file)
patterns = request["patterns"]
schemas = request["schemas"]
results = [
check("python re", "pattern", sys.version.split()[0], re.compile, patterns)
]
try:
import re2
except ImportError as error:
results.append(
unavailable("re2 (C++)", "pattern", f"google-re2 is not installed: {error}")
)
else:
# Default options, as gateways embedding RE2 use them.
results.append(
check(
"re2 (C++)",
"pattern",
distribution_version("google-re2"),
lambda pattern: re2.compile(pattern, re2.Options()),
patterns,
)
)
try:
import jsonschema_rs
except ImportError as error:
reason = f"jsonschema-rs is not installed: {error}"
results.extend(
[
unavailable("rust regex", "pattern", reason),
unavailable("rust fancy-regex", "pattern", reason),
unavailable("rust jsonschema", "schema", reason),
]
)
else:
version = distribution_version("jsonschema-rs")
# The Rust `jsonschema` crate compiles each `pattern` with the selected
# engine after translating ECMA-262 syntax, as Rust clients do.
for engine, options in (
("rust regex", jsonschema_rs.RegexOptions),
("rust fancy-regex", jsonschema_rs.FancyRegexOptions),
):
results.append(
check(
engine,
"pattern",
version,
lambda pattern, options=options: jsonschema_rs.validator_for(
{"type": "string", "pattern": pattern},
validate_formats=False,
pattern_options=options(),
),
patterns,
)
)
# Default construction, as a Rust client builds a validator: the
# declared draft's meta-schema, its formats and every subschema.
results.append(
check(
"rust jsonschema",
"schema",
version,
jsonschema_rs.validator_for,
schemas,
)
)
if request.get("complete"):
try:
import jsonschema
except ImportError as error:
results.append(
unavailable(
"python jsonschema", "schema", f"jsonschema is not installed: {error}"
)
)
else:
# Python MCP clients validate structured results with
# `jsonschema.validate`, which first checks the schema against its
# declared draft's meta-schema, including the `regex` format.
results.append(
check(
"python jsonschema",
"schema",
distribution_version("jsonschema"),
lambda schema: jsonschema.validators.validator_for(
schema
).check_schema(schema),
schemas,
)
)
json.dump({"results": results}, sys.stdout)
if __name__ == "__main__":
main()