From b7c1cb18e463c8f9ce115aadc3593d5462f760b2 Mon Sep 17 00:00:00 2001 From: ayobamiseun Date: Sat, 3 Oct 2026 21:01:41 +0100 Subject: [PATCH] docs: make the documented verification loop match what CI runs docs/developer-onboarding.md promises that everything CI runs can be run locally, then lists three validators, the eval runner and one hook test. CI runs six validators, seven unit-test files and three hook suites on three operating systems, so a contributor following the doc can fail several steps they never ran. The Tier 1 row in evals/README.md names two validators where there are five, and Tier 2 omits the rank-1 floor. List every workflow command in the onboarding block in workflow order, note the PowerShell glob caveat, state the cross-platform leg, name the workflow file as the source of truth, and align Path 3 and the pre-PR checklist. Update both tier rows in evals/README.md. CLAUDE.md's stale line is left for later because #548 edits that file. Every command in the new block was run from this branch and passes, and every command in the workflow appears in the block. --- docs/developer-onboarding.md | 42 +++++++++++++++++++++++++----------- evals/README.md | 4 ++-- 2 files changed, 32 insertions(+), 14 deletions(-) diff --git a/docs/developer-onboarding.md b/docs/developer-onboarding.md index daaf2281..a7999fa0 100644 --- a/docs/developer-onboarding.md +++ b/docs/developer-onboarding.md @@ -37,7 +37,7 @@ cd agent-skills There's no build step and no `package.json`; validators are plain Node scripts. You need: - **Node 20+** (what CI runs) for the `scripts/` validators -- **bash** (+ `jq` recommended) for the hook regression test +- **bash** (+ `jq` recommended) for the hook regression tests - **`gh` CLI** for the duplicate-PR check before proposing a skill - **Claude Code** only if you want to run Tier 3 behavioral evals locally @@ -49,26 +49,43 @@ claude --plugin-dir /path/to/agent-skills ## 3. The verification loop -The repo eats its own cooking: verification is non-negotiable for skills, and it's non-negotiable for contributions to the repo too. Everything CI runs, you can run locally in seconds: +The repo eats its own cooking: verification is non-negotiable for skills, and it's non-negotiable for contributions to the repo too. Everything CI runs, you can run locally in seconds. This is the full set, in the order the workflow runs it: ```bash -# Tier 1, structural: frontmatter, naming, required sections +# Tier 1, structural: frontmatter, naming, required sections, layout and line budget node scripts/validate-skills.js +# Links from skills to references/ resolve, including #anchor fragments +node scripts/validate-reference-links.js + +# Every plugin manifest carries the same version +node scripts/validate-versions.js + # Command parity and description sync across the three command directories node scripts/validate-commands.js -# Tier 2, trigger & routing: positive prompts rank top-k, negatives don't collide -node scripts/run-evals.js +# Spec/plan/todo artifact paths agree across every command surface and the docs +node scripts/validate-artifact-paths.js + +# Unit tests for the validators and the eval runner +# (on PowerShell, list the files instead of using the globs) +node --test scripts/*-test.js scripts/lib/*-test.js + +# Tier 2, trigger & routing: positive prompts rank top-k, negatives don't collide. +# CI enforces the rank-1 floor, so run it the way CI does. +node scripts/run-evals.js --min-rank1 95 # Tier 3, behavioral (on demand, spends tokens; --dry-run prints the plan) node scripts/run-evals.js --behavioral --dry-run -# Hook regression test, required if you touch hooks/session-start.sh -# or skills/using-agent-skills/SKILL.md +# Hook regression tests, required if you touch anything under hooks/ bash hooks/session-start-test.sh +bash hooks/sdd-cache-test.sh +bash hooks/simplify-ignore-test.sh ``` +CI runs this set on Ubuntu, then runs the validators and unit tests again on macOS and Windows, so path handling and line endings get exercised too. When the workflow gains a step, this block should gain the same line; if they ever disagree, [the workflow file](../.github/workflows/test-plugin-install.yml) is the source of truth. + The three eval tiers are worth understanding even if you never touch the harness, because a red Tier 2 usually means *fix your skill's description*, not the eval: Tier 2 is a lexical approximation of routing (stemmed TF-IDF over descriptions), and its two target failure modes are a description missing the vocabulary users actually say, and an over-broad description that outranks the right skill. The full design, schema, and trust-level rules are in [evals/README.md](../evals/README.md). Run the relevant subset before every PR. A PR that arrives green through Tier 1 + Tier 2 + command parity is reviewable; one that doesn't will bounce on mechanics before anyone reads the content. @@ -93,14 +110,15 @@ One point worth internalizing rather than looking up: when writing trigger promp - Docs and skills are **English only**; translations aren't accepted because they drift ([CONTRIBUTING.md](../CONTRIBUTING.md#translations) has the rationale). - Changes to `scripts/run-evals.js` or the eval schema should stay compatible with skill-creator's `evals.json` schema (adopted verbatim for the behavioral tier; that compatibility is a feature, not an accident). -- Anything touching the session-start hook or the meta-skill it embeds requires the hook regression test (§3). +- Anything touching `hooks/` requires the three hook regression tests (§3). Anything touching `scripts/` requires its unit tests, and a new validator ships with a test file and a CI step. ## 5. Pre-PR checklist -- [ ] Tier 1 green: `node scripts/validate-skills.js` -- [ ] Tier 2 green: `node scripts/run-evals.js` -- [ ] Command parity green if you touched any command directory: `node scripts/validate-commands.js` -- [ ] Hook test green if you touched `hooks/` or `using-agent-skills` +- [ ] Tier 1 green: `node scripts/validate-skills.js`, plus `node scripts/validate-reference-links.js` if you touched a skill that links to `references/` +- [ ] Tier 2 green the way CI runs it: `node scripts/run-evals.js --min-rank1 95` +- [ ] Command parity and artifact paths green if you touched any command directory: `node scripts/validate-commands.js` and `node scripts/validate-artifact-paths.js` +- [ ] Unit tests green if you touched `scripts/`: `node --test scripts/*-test.js scripts/lib/*-test.js` +- [ ] Hook tests green if you touched `hooks/`: all three `bash hooks/*-test.sh` - [ ] New skill → eval case file present with the minimum trigger/behavioral counts - [ ] New skill → gap justified in the PR description; catalog and open PRs checked - [ ] No duplicated content; cross-references used instead diff --git a/evals/README.md b/evals/README.md index 4f501cdb..d218b231 100644 --- a/evals/README.md +++ b/evals/README.md @@ -15,8 +15,8 @@ What neither provides is a **deterministic, CI-safe** check for a multi-skill *c | Tier | What it checks | Runs | Cost | |---|---|---|---| -| 1. Structural | Frontmatter, naming, required sections, command parity | CI (`validate-skills.js`, `validate-commands.js`) | Free | -| 2. Trigger & routing | Positive prompts rank their skill top-k; negative prompts don't; no two descriptions near-collide | CI (`run-evals.js`) | Free | +| 1. Structural | Frontmatter, naming, required sections, layout and line budget, `references/` links, command parity, artifact paths, manifest versions | CI (`validate-skills.js`, `validate-reference-links.js`, `validate-commands.js`, `validate-artifact-paths.js`, `validate-versions.js`) | Free | +| 2. Trigger & routing | Positive prompts rank their skill top-k; negative prompts don't; no two descriptions near-collide | CI (`run-evals.js --min-rank1 95`) | Free | | 3. Behavioral | An agent following the skill satisfies its `expectations[]` | On demand (`run-evals.js --behavioral`) | Tokens | Tier 2 is a **lexical approximation** of routing (stemmed TF-IDF over descriptions). It cannot judge semantics — that's Tier 3's job — but it catches the two failure modes that dominate real trigger bugs: a description missing the vocabulary users say (false negative), and an over-broad description that outranks the right skill (false positive). A Tier-2 failure usually means *fix the description*, not the eval.