Prevent stale /tmp/harness-cost cache files from affecting Sonnet 5, dated,
near-miss, and cache-rate pricing tests by using Date.now() in each session ID.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Review feedback on #2878. stdout is async when it is a pipe, which is exactly
how tests/run-all.js runs these files, and process.exit() does not wait for
pending writes — so exiting that way can drop the Passed:/Failed: lines the
aggregator totals, defeating the previous commit. The sibling
tests/ci/ito-*-skill.test.js files already use process.exitCode.
Still exits 1 on a broken counter (Passed: 4, Failed: 7) and 0 when clean.
Review feedback on #2879. stdout is async when it is a pipe, which is exactly
how tests/run-all.js runs these files, and process.exit() does not wait for
pending writes — so exiting that way can drop the Passed:/Failed: lines the
aggregator totals, defeating the point of printing them. The sibling
tests/ci/ito-*-skill.test.js files already use process.exitCode.
Still exits 1 on drift and 0 when clean.
Review feedback on #2879, citing rules/common/coding-style.md ("Immutability
(CRITICAL): ALWAYS create new objects, NEVER mutate existing ones"). Build one
list of locale entries, then derive the matched and orphaned sets from it with
filter/map, and build the canonical Map the same way.
Behaviour is unchanged and both mutations still fail: reintroducing the drift in
one locale file gives `Passed: 4, Failed: 3`, an orphan gives `Passed: 6,
Failed: 1`, both exit 1.
Review feedback on #2879: a locale file with no counterpart in `agents/` was
skipped, so every other case here silently passed over it. Retiring an agent
would leave its seven translations behind with nothing to compare against and
nothing to report it.
Collect those files while building the list and assert the collection is empty.
Zero today across 199 locale docs, so this changes no current result; planting
one orphan takes the file to `Passed: 6, Failed: 1` and exit 1.
Review feedback on #2878: the static assertions ran at the top level, so a
failure exited the process before the `Passed:`/`Failed:` lines. tests/run-all.js
totals those tokens, so the per-case counts were lost — it still went red via the
non-zero exit, but the granular numbers were not in the totals.
Route every case through the same `runTest()` wrapper the file already used for
the integration cases, and build the case list inside a try so a missing or
renamed `instinct-cli.py` / `start-observer.sh` is a reported failure rather than
a crash.
Reverting the fix now prints `Passed: 4, Failed: 7` and exits 1, naming all seven
broken expectations instead of stopping at the first.
`scripts/ci/validate-agents.js` reads only `agents/`, so the translated copies
under `docs/<locale>/agents/` were never validated against the agent they
describe — and drifted.
Two kinds of drift, both machine-checkable and both wrong in the same
direction (the translations were made from an older revision and never
re-synced):
- **Model tier, 39 files.** Every one names a costlier tier than ships:
haiku -> sonnet, sonnet -> opus. `security-reviewer` reads `opus` in ja-JP
and zh-TW; it ships `sonnet`.
- **Tool set, 15 files.** `security-reviewer` and `database-reviewer` list
`Write` and `Edit` in **all seven locales** — both agents ship read-only
(`Read, Grep, Glob, Bash`). `seo-specialist` in zh-CN adds a `Bash` the
canonical agent does not have. Documenting a reviewer as able to write is
the kind of inaccuracy someone auditing what these agents can touch would
act on.
Sync `model` and the `tools` set to canonical, preserving each locale's
existing list style so the diff is only the values that were wrong. The
`["a", "b"]` vs `a, b` formatting difference is left alone: it is consistent
per locale and carries no meaning.
Add `tests/ci/locale-agent-frontmatter.test.js` so this cannot drift back.
`tools` is compared as a set, not a string, so the style difference stays
legal; prose is not compared at all. The last case pins the specific failure:
a canonical read-only agent may never be documented with Write or Edit.
`.kiro/agents/` is deliberately excluded — it uses a different schema
(`allowedTools: [read, shell]`, no `model`), not a translation of this one.
`start-observer.sh status` globbed `*.yaml`, but the observer prompt tells the
analyzer to write `${INSTINCTS_DIR}/<id>.md` and the loader accepts
`.yaml`, `.yml`, and `.md` (ALLOWED_INSTINCT_EXTENSIONS in
scripts/instinct-cli.py). The one command an operator runs to confirm that
learning works therefore reported `Instincts: 0` on a healthy install — which
is indistinguishable from a silently dead observer, exactly the failure the
status check exists to surface.
Match the loader instead of one of its three extensions, and match how it
enumerates them: `Path.iterdir()` is top level only and `is_file()` skips
directories, so `-maxdepth 1 -type f`; `suffix.lower()` makes the comparison
case-insensitive, so `-iname`. `tr` drops the column padding BSD `wc` emits,
which is why the reported output read `Instincts: 0`.
Verified end to end against the shipped script with 3 `.md`, one `.yaml`, one
`.yml`, one `.YAML`, a `notes.txt`, and a nested `.md`: 1 before, 6 after —
the same six files the loader picks up.
The mcp-health-check preflight probes HTTP MCP servers with a bare GET.
Some Streamable HTTP servers route only POST /mcp and answer a bare GET
with 404 (Paper Desktop 0.5.3 is one). The probe scored that as down and
blocked every tool call for the server indefinitely, since the 30s
backoff just re-probes and re-fails.
A routed HTTP response of any status proves the endpoint is reachable,
which is all this preflight claims to check -- 400/401/403/405/406 are
already treated this way for the same reason. Add 404 to the set and let
the real MCP client validate the endpoint.
Adds a regression test that stands up a POST-only server (404 on GET,
200 on POST /mcp); it fails on the current code and passes with the fix.
The preflight probe in mcp-health-check only ever sent a bare GET to the
server URL. Some Streamable HTTP MCP servers route POST exclusively and
answer any GET with 404 — api.telnyx.com/v2/mcp is one — so the probe
failed permanently against a perfectly healthy server.
404 is not in HEALTHY_HTTP_CODES, so every probe failed, the backoff
compounded to the 10-minute ceiling, and the hook blocked every tool call
for that server before it left the machine while `claude mcp list` still
reported it Connected.
Replay a failed GET as a real JSON-RPC initialize POST and accept that as
proof of life. Whitelisting 404 was the alternative, but it would mask
genuine outages on every other server.
Adds a regression test with a POST-only server that 404s all GETs and
validates the initialize body; it fails without this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cost-estimate.js carries its own copy of the stale Opus/Haiku/Sonnet rate
table already reported in #2574, but grepping every .js/.json/.md file
outside node_modules turns up zero callers besides its own test. It was
added in 940135e alongside the statusline observability hooks and never
wired into any of them.
The maintainer's comment on #2656 named two acceptable outcomes: remove
the unused duplicate, or share one rate source with the live tracker.
cost-tracker.js's own fix (#2574) has not landed yet, so sharing its
table now would import numbers that are still wrong. Removing the dead
file is the smaller, immediately-correct step.
Fixes#2656
Artifacts written by `evolve --generate` are inert: Claude Code (and every
spec-compliant Agent Skills client) injects only `name` + `description` at
startup and will not load an artifact missing them.
Today the generator writes:
- skills: `# {name}` with no frontmatter block at all
- commands: `# {cmd_name}` with no frontmatter block at all
- agents: `model`/`tools` only, no `name`, no `description`
So the whole evolve pipeline terminates in files that can never load. I hit
this on a real install: 12 generated artifacts across two projects, none of
which Claude Code had ever seen.
This adds a `_evolved_description()` helper and emits proper frontmatter for
all three artifact kinds. The description is sanitised for the two things that
break loaders: `: ` in an unquoted scalar (rejected by strict YAML parsers)
and `<`/`>` (system-prompt injection risk).
Adds two tests to tests/scripts/instinct-cli-evolve-generate.test.js. Both
fail against current main and pass with this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Santhi Prakash