`descriptionFromMd` in validate-commands.js splits each frontmatter line on its
first colon — the same forgiving read the skill linter used before the strict
rules landed. So a command whose frontmatter is not valid YAML passes every
check here, and Claude Code parses that frontmatter when the command is loaded.
Same class of gap as the SKILL.md one, on the other set of files: the #494
thread verified all 25 SKILL.md files *and* the command files by hand. This
makes the second half a check too, sharing the rules rather than restating them.
All nine current commands already pass — confirmed independently against ruby's
psych before writing this — so it is a guard, not a repair. That is the same
shape the skills half had, and the same reason: the manual verification was
real, it just could not stay true on its own.
Scoped to `.claude/commands/*.md`. The `.gemini/` and `commands/` directories
are TOML and already go through a real TOML parser, so a malformed one surfaces
as a missing description in the existing check. A test pins that the YAML rules
are NOT applied to them, since a TOML description legitimately carries a colon
and running these rules over it would fail nearly every command.
The test sandbox copies the validator into a temp tree, so it now copies
scripts/lib/skill-lint.js beside it. Without that every existing test fails with
a module-not-found, which is how I found it.
Five new tests. Mutation-tested 2/2: skipping the check, and reporting its
errors without counting them, each fail two. Every validator exits 0 and all six
script suites pass.
Refs #494
`parseFrontmatter` splits each line on its first colon. That is enough to read
`name` and `description`, and it is forgiving by design — but it means a
SKILL.md whose frontmatter is not valid YAML passes every check in the linter.
The hosts that consume these skills do not share that forgiveness. Per Cursor
staff in #494, attaching a skill to a message parses its frontmatter as YAML,
and a parse failure fails the request with a generic server error and takes the
rest of the chat's context with it. So the repo could ship frontmatter that
breaks a host while CI stayed green.
Three shapes pass the current linter and are rejected by a strict parser
(confirmed against ruby's psych, each isolated by mutating only the
frontmatter of a real SKILL.md that otherwise lints clean):
description: Use this: when X colon-space: YAML reads a nested mapping
\tkey: value tab indentation is invalid YAML whitespace
description: "unterminated an unclosed quote
All three now fail with the line number and the fix. Deliberate boundaries:
a colon with no trailing space is untouched, so `https://example.com/a:b`
still passes, and a duplicate key is NOT reported because `safe_load` accepts
one — the rule tracks the parser, not taste.
This was verified once by hand against all 25 skills in the #494 thread. This
makes it a check instead of a memory.
Narrow on purpose: the repo has no package.json and therefore no YAML parser to
depend on, so this is a small set of rules aimed at the classes actually
observed to break a host, not a YAML implementation. No current frontmatter
uses block scalars or multi-line values and the check says so.
All 25 skills still pass with 0 errors, every validator exits 0, and the six
script suites are green (35 pass in skill-lint-test). Mutation-tested 4/4:
removing the call, widening the colon rule to any colon, and dropping either
the tab or the unterminated-quote rule each fail the new tests.
Refs #494
Add platform coverage for all six Node test suites, validators, and routing checks while preserving existing Ubuntu check names. Require both matrix jobs before plugin validation and retain failures from each platform.
#579 moved the code patterns verbatim, but three prose sections (supply chain, privacy, LLM) and the cookie bullet were rewritten rather than moved, and some rules ended up in none of SKILL.md, the reference, or the shared checklist.
Restores them inline: the sameSite value and its CSRF reason, why forced audit fixes are dangerous, the audit signatures commands and "not automatic proof of compromise", the retention TTL, auditable consent, the (2025) on the OWASP LLM list, and "the full system prompt" plus "validate every tool argument" in the LLM rule. Red Flags already audits the system-prompt item, so the rule and its audit agree again.
Same 216 lines, +55 words.
A file under skills/<name>/references/ sits one directory deeper than SKILL.md, so a link to a shared checklist needs ../../../references/ from there. The validator only read SKILL.md, which left that off-by-a-level mistake unguarded in the one place it can now happen (hardening-patterns.md on main, optimization-patterns.md in #580).
The rule is unchanged: only references/*.md links, fenced blocks exempt, paths that do not exist yet ignored. The file set grows to include the markdown files directly inside each skill's references/ directory, and every link resolves from the file that contains it. The failure hint names the three-level prefix when the violation is in a reference file.
Extract the two fs.rmSync calls into clearGradingSlot(base), export it,
and call it from runBehavioral. Rewrite both slot-clearing tests to call
clearGradingSlot instead of inlining fs.rmSync, so each test exercises
the production cleanup path and fails if the call is removed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move stale-file cleanup from persistGradingOutcome (which only ran on
rejected grading) to the top of each eval iteration, before the executor
is invoked. Both ${base}.grading.json and ${base}.grading.raw.txt are
removed with fs.rmSync(p, { force: true }), which ignores a missing file
but still throws on a permissions error.
This closes three gaps at the source:
- The bare catch around unlinkSync swallowed EACCES alongside ENOENT.
- An executor or grader crash skipped persistGradingOutcome entirely,
leaving the previous run's .grading.json intact.
- A successful grading left a stale .grading.raw.txt from a prior
rejected run beside the new .grading.json.
persistGradingOutcome now only writes, never deletes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each .grading.json now carries a run object with executor_model,
grader_model, and timestamp so results can be attributed to a
specific model version and point in time.
The executor model is read from the init event of the stream-json
output that the executor already produces (type "system",
subtype "init", top-level model field). The grader model is recorded
as "unknown": the grader is a bare `claude -p` whose output
parseGrading extracts by regex, so switching it to stream-json would
break parsing, and adding --model is a behaviour change that does
not belong in a recording fix.
RESULTS_DIR is only ever created and never cleared, and the output
path is derived from skill name and eval id alone. When a run's
grading is rejected (parseGrading returns null), the runner writes
.grading.raw.txt but leaves any .grading.json from a previous
successful run in place. A collector reading that directory gets the
old score beside the raw output of the run that actually happened.
Extract persistGradingOutcome so the write-and-cleanup is a testable
unit, and unlink the stale .grading.json before writing the raw file.
Reuse the skill linter's fence parser so inline backticks cannot hide broken references and trailing text cannot close a code fence. Cover both failures with LF and CRLF regression cases, preserving diagnostic line numbers.
SKILL.md loaded ~5.3k tokens on every activation, 30% of it inline
TypeScript examples that an agent only needs when it reaches that code.
docs/skill-anatomy.md already asks for SKILL.md under 500 lines with
reference material in supporting files; this skill was at 524.
Move the OWASP prevention patterns, input-validation and upload code,
SSRF check, destructive-path discussion, dependency-audit decision tree,
rate-limiting and secrets snippets, data-classification table, and LLM
output example verbatim into references/hardening-patterns.md (same
convention as constraint-driven-development/references/floor-guard.md).
SKILL.md keeps the workflow: threat model, three-tier boundaries, and a
condensed "Hardening Controls" section stating each rule with a deep link
to its pattern. Frontmatter, Process, Rationalizations, Red Flags, and
Verification are byte-identical to before.
The skill's own review checklist duplicated references/security-checklist.md;
replace it with a pointer and add its three privacy items to the shared
checklist so nothing is lost.
SKILL.md: 524 -> 216 lines, 4096 -> 2449 words (~-40% tokens per load).
Routing unchanged: run-evals rank-1 stays 100%, no description edits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The round-trip test added in this PR failed on ubuntu-latest while passing
on macOS: after Read → Stop, a protected block containing a backslash came
back altered (byte 109, line 3 of the fixture).
expand_file rebuilds each line with `${line//$placeholder/$block_content}`.
Bash 5.2 enables patsub_replacement by default, which makes the replacement
side of that expansion special: an unquoted `&` expands to the matched text
and backslashes are consumed. Block content is raw code and routinely
contains `&&` and backslashes, so on any bash 5.2 host every protected block
with either character was corrupted when it was expanded back. Bash 3.2 on
macOS has no such option, which is why local runs never showed it.
Turn the option off at the top of the hook; older bashes reject the shopt,
so the error is ignored there. Extend the byte-fidelity fixture with `&&`
and `&` so the ampersand case is guarded alongside the backslash one.
Wire the two regression suites into the workflow as a test-hooks job
so the coverage actually guards the hooks, per review. Deliberately
excludes session-start-test.sh: it currently asserts the pre-#474
payload shape and fails against the envelope the hook now emits;
PR #510 updates that test and can wire it in when it lands.
When a Stop leaves no BLOCK_ placeholder to expand, the hook has to restore
the backup to recover the protected block — which overwrote whatever had
replaced the file. Copy that content into the cache as <id>.recovered first
and name both paths in the warning, so the rewrite is merged back by hand
rather than lost.
Adds a test that the fallback keeps the rewritten content in the cache, and
a minimal lifecycle test (Read, a direct write with no event, Stop) asserting
the write survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Stop branch restores every cached file from its backup, but that
backup is written on PreToolUse Read and refreshed only by the
PostToolUse Edit|Write branch. A change made to a filtered file through
any other route between Read and Stop — a Bash command, a formatter, an
external editor — is overwritten at session end. Reproduced in #295:
Bash wrote const editable = 2 and read it back, Stop left const
editable = 1 on disk, with no crash and no failed tool call.
The hook protects the content of an annotated block, not the file: the
header documents the backup as holding the model's changes, and Edit and
Write already preserve them. So Stop now runs the same placeholder
expansion the Edit|Write branch runs, against the file as it is on disk.
Whatever wrote the file wrote its filtered form with the placeholders
intact, so expanding it restores the protected blocks and keeps the
change, whichever route it came from.
The expansion body moves into an expand_file function that both branches
call, so the fuzzy-match and deleted-block warnings apply at Stop too.
When no cached BLOCK_<hash> is left, the file was rewritten wholesale and
the hook cannot place the block; it falls back to the backup as before,
now with a warning instead of in silence. That residual is documented in
SIMPLIFY-IGNORE.md. With no outside write, expansion reproduces the
backup exactly, so Read→Stop and Read→Edit→Stop are byte-identical.
Test 11 drives the real hook through Read, an Edit event, a write with no
event after it, and Stop; it fails at be4e44a on the two assertions for
the lost write. Suite: 28 passed, 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two hardening changes to parseGrading:
- Once a result's id is validated, overwrite its text with the declared
expectation at that index. The id is the binding, so a grader that
paraphrases an expectation no longer leaks its own wording into the
saved grading report.
- Stop rejecting a grading whose summary.pass_rate disagrees with the
counters. pass_rate is derived, not an independent observation, so a
rounded or mis-divided value is now recomputed from the validated
results instead of discarding the run. The integer counters
(passed/failed/total) remain exact-match checks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
parseGrading accepted any result set whose length matched the expected count, without verifying that each result corresponded to a declared expectation. A grader could return duplicates, omit an expectation, or include results for undeclared expectations and still pass validation.
parseGrading now takes the full expectations array, requires each result to carry an integer id in 1..n matching the numbered expectations in the prompt, rejects duplicates and out-of-range ids, and derives pass_rate from the validated counters. The grader prompt requests the id field and the call site in runBehavioral is updated.
Five tests added: reordered-but-complete results, duplicate ids, out-of-range ids, omitted expectations, untrusted pass_rate.
The plugin's SessionStart hook injected using-agent-skills/SKILL.md as
additionalContext on every session. Claude Code already discovers and
routes skills from their descriptions, so the hook stacks a second
router on the native one — the pattern docs/getting-started.md warns
against ("That creates two routers for the same task"), measured at
~3.5k wasted cache-creation tokens per session in #569.
Delete hooks/hooks.json (the registration; the plugin now ships no
hooks) and keep hooks/session-start.sh as a standalone script for hosts
without native skill routing, matching the fix shape suggested in the
issue. Update the hook header, CONTRIBUTING.md, codex-setup.md, and
gemini-cli-setup.md to match.
`## All 24 Skills` is contradicted by the sentence directly beneath it —
"The pack includes 25 skills total" — and by the section's own contents:
skills listed under the heading 25
directories under skills/ 25
validate-skills --strict 25 skills checked
difference (none)
The section does list every skill, meta-skill included, in its own `Meta`
subsection. So 25 is the honest number for the heading.
The nearby "24 lifecycle skills plus the `using-agent-skills` meta-skill"
phrasing is correct and is left alone — 24 is a real count, just not of
what the heading is counting.
Also retargets the one inbound anchor, which the rename would otherwise
break:
docs/adoption-guide.md:5 ...README.md#all-24-skills -> #all-25-skills
That link is the only reference to the anchor anywhere in the repo
(checked across md/json/ts/js), so the rename is complete.
Gates: validate-skills --strict 25 skills, 0 errors, 0 warnings; the JS
suite 44 pass / 0 fail.
Addy Osmani