Two follow-ups from the review on #600, both already on main before it:
- The diff walk treated any line starting with `---` or `+++` as a file
header. Inside a hunk those lines are content. An added line that starts
with `++` (`++n; // eslint-disable-line no-plusplus`) shows as `+++n; ...`
and was dropped, so the suppression passed. An added line that starts with
`++ ` was taken as a header, so the lines after it were filed under a
made-up path and a removed assertion further down the same test file went
unreported. A removed line starting with `--` never reached the removed
list either. Headers are now read only between a file's `diff` line and
its first `@@`.
- Run from a subfolder, `git ls-files --others` listed only that
subfolder's untracked files, relative to it, while `git diff` covers the
whole tree from the top. An untracked suppression elsewhere in the
repository was missed, and one inside the subfolder was reported under a
different name from the tracked files. The guard now changes to
`git rev-parse --show-toplevel` first, and exits 2 outside a work tree.
The test gains six cases: five fail on main, and the sixth (outside a work
tree) passes on both and covers the new exit-2 path; 28 pass. The test now
also runs in the macOS/Windows matrix.
The reference guard in constraint-driven-development promised to include
untracked files, catch deleted test files and weakened thresholds, and stay
silent when constraints tighten. Run unchanged, it did none of those for
ordinary local git changes:
- `git diff --no-index /dev/null <file>` exits 1 whenever the sides differ,
and the shared git() wrapper turned every nonzero exit into null, so every
untracked file diffed as empty and no added-line rule could see it.
- The filename came from the `+++` header only, so a deletion
(`+++ /dev/null`) sliced to `ev/null` and matched nothing; there was no
deletion rule, and staging did not help.
- A number going down was treated as weaker, which is right for a minimum
(coverage) and inverted for a maximum (size, latency, error budget).
- A removed floor bullet was never reported, although the comment said it was.
The wrapper now keeps stdout on the expected exit 1 and every other failure
exits 2 rather than reading as clean; both diff headers name the file and a
deleted test file is its own finding; each threshold carries a direction read
from the words around it (>=, at least, must not fall; <=, at most, under,
must not grow), a number with no readable direction is reported when it
changes; a removed floor bullet or table row is reported, except an Exceptions
row, whose removal tightens. The three language patterns are unchanged.
scripts/floor-guard-reference-test.js extracts the fenced block and runs it
with real git against fourteen fixtures plus three edge cases; eight of them
fail on the previous reference and all pass now. The workflow runs it beside
the other script tests.
Add platform coverage for all six Node test suites, validators, and routing checks while preserving existing Ubuntu check names. Require both matrix jobs before plugin validation and retain failures from each platform.
Wire the two regression suites into the workflow as a test-hooks job
so the coverage actually guards the hooks, per review. Deliberately
excludes session-start-test.sh: it currently asserts the pre-#474
payload shape and fails against the envelope the hook now emits;
PR #510 updates that test and can wire it in when it lands.
stripFencedCodeBlocks matched fences with a single whole-document regex that
only recognised a column-zero backtick fence closed by a run of exactly the
same length. Three CommonMark-legal forms slipped through and their contents
were linted as prose: tilde fences, fences indented one to three spaces, and
closing fences longer than their opener. This is live on main today —
spec-driven-development's list-nested examples use indented fences, so the
linter has been reading their contents as skill text.
Replace the regex with a line scanner that tracks the opener's marker and
length, closes only on a same-marker run at least as long, treats an
unterminated fence as running to end of file (so sections after it fail
loud), and follows the CommonMark rule that a backtick fence's info string
may not contain backticks. Stripped lines become empty lines so positions
are preserved.
Add unit coverage for each form plus CRLF input, and run
scripts/lib/skill-lint-test.js in CI — it was the only test file the
workflow did not execute.
Revives the approach from #444 by @ujjwalks, which was approved and then
closed after a rebase conflict.
Closes#437
Eleven positive trigger prompts ranked their skill #2-#3 because the
descriptions lacked vocabulary users actually say (follow-up to #351):
- debugging-and-error-recovery: regression vocabulary (worked yesterday,
what broke)
- documentation-and-adrs: 'document an architecture decision', design
reasoning ('documentation' and 'document' don't stem together)
- doubt-driven-development: cross-examining assumptions, stress-testing
a plan for hidden failure modes, high-stakes changes
- git-workflow-and-versioning: splitting a messy working tree into
atomic commits
- incremental-implementation: thin/verifiable slices, next task from a
plan, feature flags
- shipping-and-launch: what needs to be in place before shipping
- spec-driven-development: PRD, objectives and scope
- test-driven-development: red-green-refactor
- using-agent-skills: deciding which workflow applies to the work
- source-driven-development: verify against the official docs
('docs'/'documentation' don't stem together)
Trigger rank-1 rate: 86% -> 100% (76/76), no new collision warnings.
CI floor raised from 80 to 95 per the ratchet policy; README baseline
updated and the stale #351 tracking sentence rephrased as guidance.
- Problem: the CI step skipped the session-start hook test when jq was
missing, turning the whole step into a silent no-op on runners without
jq — the same silently-green shape this PR exists to remove.
- Fix: run hooks/session-start-test.sh unconditionally. The script already
detects jq itself (HAS_JQ) and asserts the hookSpecificOutput envelope on
both code paths, so a jq-less runner now covers the hook's real
jq-missing branch instead of skipping it.
- Verification: test passes with jq present; with jq removed from PATH
(minimal PATH env, command -v jq empty) the hook emits the jq-is-required
envelope and the test still passes; workflow YAML parses clean.
- Problem: session-start-test.sh was not referenced by any workflow, so envelope regressions never failed CI.
- Fix: add a validate-skills step that runs the test and skips when jq is missing.
- Verification: bash hooks/session-start-test.sh → session-start JSON payload OK
The path fix has no regression guard: nothing in CI resolves references/
links, so all 18 were broken while CI stayed green. validate-artifact-
paths.js is scoped to spec/plan/todo artifacts and says in its own header
that it is not a general markdown path linter.
Add validate-reference-links.js, which resolves every `references/*.md`
link in skills/*/SKILL.md against that skill's own directory. This accepts
both conventions in CLAUDE.md: shared checklists reached via
../../references/, and a skill's own colocated references/ directory.
Scope stays narrow on purpose. Skills legitimately name paths that do not
exist yet -- tasks/todo.md, PERF.md, docs/ideas/[idea-name].md -- and a
general markdown linter would fail the build on them. A test pins that.
Proven against the pre-fix tree: 18 error(s), exit 1, matching the 18
links fixed in the previous commit. After the fix: 0 error(s), exit 0.
7 unit tests cover the regression itself, colocated references/,
markdown-link syntax, a renamed target, multiple violations in one skill,
and the non-reference paths that must be ignored. Wired into the
validate-skills job, alongside the other skill-content checks.
Known limitation: fenced code blocks are not stripped, so a SKILL.md that
documents the anti-pattern inside a fence would be flagged. Nothing does
today. Sharing stripFencedCodeBlocks looks right once #444 lands.
Refs #468
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The /spec and /plan commands write spec, plan, and todo artifacts to paths that /build and the spec/plan skills read back. When a producer moves an artifact without updating the consumers, the pipeline breaks and nothing in CI catches it: the command-parity check only compares descriptions, not paths. PR #93 hit exactly this, pointing /spec and /plan at docs/features/[name]/ while /build still required SPEC.md and tasks/plan.md.
validate-artifact-paths.js enforces one canonical set of artifact paths across every file in the pipeline (the spec/plan/build commands, the spec-driven-development and planning-and-task-breakdown skills, and the getting-started and adoption guides). Any spec/plan/todo artifact path outside the allowlist fails CI, so changing the convention has to touch the allowlist and every guarded file in the same change.
Scope is deliberately narrow: only spec/plan/todo artifacts, only the pipeline files; it is not a general markdown path linter. Wired into the validate-commands CI job alongside its test.
actions/checkout@v4 and actions/setup-node@v4 target the deprecated Node 20
runner runtime, so every run logs forced-Node-24 deprecation annotations.
Harmonize checkout on the v6 major already used by the other three jobs,
bump setup-node to v6, and move node-version from Node 20 (EOL 2026-04-30)
to Node 24 LTS.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gives contributors a short intake path when a skill assumes the wrong
ecosystem or outdated steps, so gaps like #404 reach maintainers via
the repo instead of external channels. Closes the intake-only scope
from #412.
Co-authored-by: Cursor <cursoragent@cursor.com>
There was no way to measure whether skills trigger correctly, stay
distinct, or change agent behavior. This adds evals, aligned with what
the community has converged on, with a deterministic CI tier on top:
- evals/cases/<skill>.json for all 24 skills. The evals[] block uses
Anthropic skill-creator's evals.json schema verbatim (id, prompt,
expected_output, expectations[]) so its runner, benchmarks, and eval
viewer work against our files unmodified. A trigger block (this
repo's extension) adds positive/negative routing prompts per skill.
- scripts/run-evals.js, zero-dependency runner:
Tier 2 (CI): trigger evals via stemmed TF-IDF ranking over skill
descriptions (positive prompts must rank top-k, negative prompts
must not rank first), catalog collision detection between skill
descriptions, schema and coverage checks.
Tier 3 (opt-in): --behavioral <skill> executes each eval through
headless claude -p and grades the transcript against expectations[]
(superpowers-style); --dry-run previews without spending tokens.
- CI: run the deterministic tier in the validate-skills job.
- Docs: evals/README.md defines the framework and prior art;
CONTRIBUTING requires an eval file for new skills (warning-level in
the runner until in-flight skill PRs clear); CLAUDE.md pointers.
Current baseline: 120 checks pass, 85% trigger rank-1 rate across 72
positive prompts, zero catalog collisions.
The validate-skills job pinned actions/checkout@v4 while the other two
jobs in the same workflow (test-fresh-install, test-plugin-structure)
use @v6. Align all three on @v6 for consistency.
- scripts/validate-skills.js: zero-dependency Node.js validator that
checks every skill for valid frontmatter, description length (≤1024),
required sections (Overview, When to Use, Common Rationalizations,
Red Flags, Verification), and dead cross-skill references
- Skills with type:meta or exempt:sections in frontmatter skip section
checks; applied to using-agent-skills (meta) and idea-refine (legacy
structure predating the anatomy spec)
- CI: validate-skills job runs before plugin-manifest validation and
blocks merge on any error; uses Node 20, no npm install required
Runs on every push and PR:
1. Validates marketplace and plugin manifests
2. Tests marketplace add + plugin install end-to-end
No ANTHROPIC_API_KEY required — these are CLI/filesystem
operations, not LLM calls.
mavericksea-ai