Merge #644: make the documented verification loop match what CI runs

docs/developer-onboarding.md

@@ -36,8 +36,8 @@ cd agent-skills
 
 There's no build step and no `package.json`; validators are plain Node scripts. You need:
 
-- **Node 20+** (what CI runs) for the `scripts/` validators
-- **bash** (+ `jq` recommended) for the hook regression test
+- **Node 24** (the version CI pins) for the `scripts/` validators
+- **bash** (+ `jq` recommended) for the hook regression tests
 - **`gh` CLI** for the duplicate-PR check before proposing a skill
 - **Claude Code** only if you want to run Tier 3 behavioral evals locally
 
@@ -49,26 +49,44 @@ claude --plugin-dir /path/to/agent-skills
 
 ## 3. The verification loop
 
-The repo eats its own cooking: verification is non-negotiable for skills, and it's non-negotiable for contributions to the repo too. Everything CI runs, you can run locally in seconds:
+The repo eats its own cooking: verification is non-negotiable for skills, and it's non-negotiable for contributions to the repo too. Everything CI runs without the Claude Code CLI, you can run locally in seconds. This is that set, grouped by what each check covers (the workflow spreads them across several jobs):
 
 ```bash
-# Tier 1, structural: frontmatter, naming, required sections
+# Tier 1, structural: frontmatter, naming, required sections, layout and line budget
 node scripts/validate-skills.js
 
+# Links from skills to references/ resolve, including #anchor fragments
+node scripts/validate-reference-links.js
+
+# Every plugin manifest carries the same version
+node scripts/validate-versions.js
+
 # Command parity and description sync across the three command directories
 node scripts/validate-commands.js
 
-# Tier 2, trigger & routing: positive prompts rank top-k, negatives don't collide
-node scripts/run-evals.js
+# Spec/plan/todo artifact paths agree across every command surface and the docs
+node scripts/validate-artifact-paths.js
+
+# Unit tests for the validators and the eval runner
+# (on PowerShell, list the files instead of using the globs)
+node --test scripts/*-test.js scripts/lib/*-test.js
+
+# Tier 2, trigger & routing: positive prompts rank top-k, negatives don't collide.
+# CI enforces the rank-1 floor, so run it the way CI does.
+node scripts/run-evals.js --min-rank1 95
 
 # Tier 3, behavioral (on demand, spends tokens; --dry-run prints the plan)
 node scripts/run-evals.js --behavioral <skill-name> --dry-run
 
-# Hook regression test, required if you touch hooks/session-start.sh
-# or skills/using-agent-skills/SKILL.md
+# Hook regression tests, required if you touch anything under hooks/ or
+# skills/using-agent-skills/SKILL.md (the session-start script embeds that file)
 bash hooks/session-start-test.sh
+bash hooks/sdd-cache-test.sh
+bash hooks/simplify-ignore-test.sh
 ```
 
+CI runs this set on Ubuntu, then runs the validators and unit tests again on macOS and Windows, so path handling and line endings get exercised too. Two further jobs need the Claude Code CLI and are not in the block: `claude plugin validate .` checks the marketplace and plugin manifests, and a marketplace install (`claude plugin marketplace add ./`, then `claude plugin install agent-skills@addy-agent-skills --scope user`) proves the plugin installs from a checkout. Run those if you touch `plugin.json`, the `.claude-plugin/` manifests, or anything the install path reads. When the workflow gains a step, this block should gain the same line; if they ever disagree, [the workflow file](../.github/workflows/test-plugin-install.yml) is the source of truth.
+
 The three eval tiers are worth understanding even if you never touch the harness, because a red Tier 2 usually means *fix your skill's description*, not the eval: Tier 2 is a lexical approximation of routing (stemmed TF-IDF over descriptions), and its two target failure modes are a description missing the vocabulary users actually say, and an over-broad description that outranks the right skill. The full design, schema, and trust-level rules are in [evals/README.md](../evals/README.md).
 
 Run the relevant subset before every PR. A PR that arrives green through Tier 1 + Tier 2 + command parity is reviewable; one that doesn't will bounce on mechanics before anyone reads the content.
@@ -93,14 +111,15 @@ One point worth internalizing rather than looking up: when writing trigger promp
 
 - Docs and skills are **English only**; translations aren't accepted because they drift ([CONTRIBUTING.md](../CONTRIBUTING.md#translations) has the rationale).
 - Changes to `scripts/run-evals.js` or the eval schema should stay compatible with skill-creator's `evals.json` schema (adopted verbatim for the behavioral tier; that compatibility is a feature, not an accident).
-- Anything touching the session-start hook or the meta-skill it embeds requires the hook regression test (§3).
+- Anything touching `hooks/` or `skills/using-agent-skills/SKILL.md` (which the session-start script embeds) requires the three hook regression tests (§3). Anything touching `scripts/` requires its unit tests.
 
 ## 5. Pre-PR checklist
 
-- [ ] Tier 1 green: `node scripts/validate-skills.js`
-- [ ] Tier 2 green: `node scripts/run-evals.js`
-- [ ] Command parity green if you touched any command directory: `node scripts/validate-commands.js`
-- [ ] Hook test green if you touched `hooks/` or `using-agent-skills`
+- [ ] Tier 1 green: `node scripts/validate-skills.js`, plus `node scripts/validate-reference-links.js` if you touched a skill that links to `references/`
+- [ ] Tier 2 green the way CI runs it: `node scripts/run-evals.js --min-rank1 95`
+- [ ] Command parity and artifact paths green if you touched any command directory: `node scripts/validate-commands.js` and `node scripts/validate-artifact-paths.js`
+- [ ] Unit tests green if you touched `scripts/`: `node --test scripts/*-test.js scripts/lib/*-test.js`
+- [ ] Hook tests green if you touched `hooks/` or `using-agent-skills`: `bash hooks/session-start-test.sh`, `bash hooks/sdd-cache-test.sh`, `bash hooks/simplify-ignore-test.sh`
 - [ ] New skill → eval case file present with the minimum trigger/behavioral counts
 - [ ] New skill → gap justified in the PR description; catalog and open PRs checked
 - [ ] No duplicated content; cross-references used instead

evals/README.md

@@ -15,8 +15,8 @@ What neither provides is a **deterministic, CI-safe** check for a multi-skill *c
 
 | Tier | What it checks | Runs | Cost |
 |---|---|---|---|
-| 1. Structural | Frontmatter, naming, required sections, command parity | CI (`validate-skills.js`, `validate-commands.js`) | Free |
-| 2. Trigger & routing | Positive prompts rank their skill top-k; negative prompts don't; no two descriptions near-collide | CI (`run-evals.js`) | Free |
+| 1. Structural | Frontmatter, naming, required sections, layout and line budget, `references/` links, command parity, artifact paths, manifest versions | CI (`validate-skills.js`, `validate-reference-links.js`, `validate-commands.js`, `validate-artifact-paths.js`, `validate-versions.js`) | Free |
+| 2. Trigger & routing | Positive prompts rank their skill top-k; negative prompts don't; no two descriptions near-collide | CI (`run-evals.js --min-rank1 95`) | Free |
 | 3. Behavioral | An agent following the skill satisfies its `expectations[]` | On demand (`run-evals.js --behavioral`) | Tokens |
 
 Tier 2 is a **lexical approximation** of routing (stemmed TF-IDF over descriptions). It cannot judge semantics — that's Tier 3's job — but it catches the two failure modes that dominate real trigger bugs: a description missing the vocabulary users say (false negative), and an over-broad description that outranks the right skill (false positive). A Tier-2 failure usually means *fix the description*, not the eval.