mirror of
https://github.com/addyosmani/agent-skills.git
synced 2026-10-26 15:05:04 +00:00
docs: make the documented verification loop match what CI runs
docs/developer-onboarding.md promises that everything CI runs can be run locally, then lists three validators, the eval runner and one hook test. CI runs six validators, seven unit-test files and three hook suites on three operating systems, so a contributor following the doc can fail several steps they never ran. The Tier 1 row in evals/README.md names two validators where there are five, and Tier 2 omits the rank-1 floor. List every workflow command in the onboarding block in workflow order, note the PowerShell glob caveat, state the cross-platform leg, name the workflow file as the source of truth, and align Path 3 and the pre-PR checklist. Update both tier rows in evals/README.md. CLAUDE.md's stale line is left for later because #548 edits that file. Every command in the new block was run from this branch and passes, and every command in the workflow appears in the block.
This commit is contained in:
2 files changed
+32
-14
No files matched your search
@@ -37,7 +37,7 @@ cd agent-skills
|
||||
There's no build step and no `package.json`; validators are plain Node scripts. You need:
|
||||
|
||||
- **Node 20+** (what CI runs) for the `scripts/` validators
|
||||
- **bash** (+ `jq` recommended) for the hook regression test
|
||||
- **bash** (+ `jq` recommended) for the hook regression tests
|
||||
- **`gh` CLI** for the duplicate-PR check before proposing a skill
|
||||
- **Claude Code** only if you want to run Tier 3 behavioral evals locally
|
||||
|
||||
@@ -49,26 +49,43 @@ claude --plugin-dir /path/to/agent-skills
|
||||
|
||||
## 3. The verification loop
|
||||
|
||||
The repo eats its own cooking: verification is non-negotiable for skills, and it's non-negotiable for contributions to the repo too. Everything CI runs, you can run locally in seconds:
|
||||
The repo eats its own cooking: verification is non-negotiable for skills, and it's non-negotiable for contributions to the repo too. Everything CI runs, you can run locally in seconds. This is the full set, in the order the workflow runs it:
|
||||
|
||||
```bash
|
||||
# Tier 1, structural: frontmatter, naming, required sections
|
||||
# Tier 1, structural: frontmatter, naming, required sections, layout and line budget
|
||||
node scripts/validate-skills.js
|
||||
|
||||
# Links from skills to references/ resolve, including #anchor fragments
|
||||
node scripts/validate-reference-links.js
|
||||
|
||||
# Every plugin manifest carries the same version
|
||||
node scripts/validate-versions.js
|
||||
|
||||
# Command parity and description sync across the three command directories
|
||||
node scripts/validate-commands.js
|
||||
|
||||
# Tier 2, trigger & routing: positive prompts rank top-k, negatives don't collide
|
||||
node scripts/run-evals.js
|
||||
# Spec/plan/todo artifact paths agree across every command surface and the docs
|
||||
node scripts/validate-artifact-paths.js
|
||||
|
||||
# Unit tests for the validators and the eval runner
|
||||
# (on PowerShell, list the files instead of using the globs)
|
||||
node --test scripts/*-test.js scripts/lib/*-test.js
|
||||
|
||||
# Tier 2, trigger & routing: positive prompts rank top-k, negatives don't collide.
|
||||
# CI enforces the rank-1 floor, so run it the way CI does.
|
||||
node scripts/run-evals.js --min-rank1 95
|
||||
|
||||
# Tier 3, behavioral (on demand, spends tokens; --dry-run prints the plan)
|
||||
node scripts/run-evals.js --behavioral <skill-name> --dry-run
|
||||
|
||||
# Hook regression test, required if you touch hooks/session-start.sh
|
||||
# or skills/using-agent-skills/SKILL.md
|
||||
# Hook regression tests, required if you touch anything under hooks/
|
||||
bash hooks/session-start-test.sh
|
||||
bash hooks/sdd-cache-test.sh
|
||||
bash hooks/simplify-ignore-test.sh
|
||||
```
|
||||
|
||||
CI runs this set on Ubuntu, then runs the validators and unit tests again on macOS and Windows, so path handling and line endings get exercised too. When the workflow gains a step, this block should gain the same line; if they ever disagree, [the workflow file](../.github/workflows/test-plugin-install.yml) is the source of truth.
|
||||
|
||||
The three eval tiers are worth understanding even if you never touch the harness, because a red Tier 2 usually means *fix your skill's description*, not the eval: Tier 2 is a lexical approximation of routing (stemmed TF-IDF over descriptions), and its two target failure modes are a description missing the vocabulary users actually say, and an over-broad description that outranks the right skill. The full design, schema, and trust-level rules are in [evals/README.md](../evals/README.md).
|
||||
|
||||
Run the relevant subset before every PR. A PR that arrives green through Tier 1 + Tier 2 + command parity is reviewable; one that doesn't will bounce on mechanics before anyone reads the content.
|
||||
@@ -93,14 +110,15 @@ One point worth internalizing rather than looking up: when writing trigger promp
|
||||
|
||||
- Docs and skills are **English only**; translations aren't accepted because they drift ([CONTRIBUTING.md](../CONTRIBUTING.md#translations) has the rationale).
|
||||
- Changes to `scripts/run-evals.js` or the eval schema should stay compatible with skill-creator's `evals.json` schema (adopted verbatim for the behavioral tier; that compatibility is a feature, not an accident).
|
||||
- Anything touching the session-start hook or the meta-skill it embeds requires the hook regression test (§3).
|
||||
- Anything touching `hooks/` requires the three hook regression tests (§3). Anything touching `scripts/` requires its unit tests, and a new validator ships with a test file and a CI step.
|
||||
|
||||
## 5. Pre-PR checklist
|
||||
|
||||
- [ ] Tier 1 green: `node scripts/validate-skills.js`
|
||||
- [ ] Tier 2 green: `node scripts/run-evals.js`
|
||||
- [ ] Command parity green if you touched any command directory: `node scripts/validate-commands.js`
|
||||
- [ ] Hook test green if you touched `hooks/` or `using-agent-skills`
|
||||
- [ ] Tier 1 green: `node scripts/validate-skills.js`, plus `node scripts/validate-reference-links.js` if you touched a skill that links to `references/`
|
||||
- [ ] Tier 2 green the way CI runs it: `node scripts/run-evals.js --min-rank1 95`
|
||||
- [ ] Command parity and artifact paths green if you touched any command directory: `node scripts/validate-commands.js` and `node scripts/validate-artifact-paths.js`
|
||||
- [ ] Unit tests green if you touched `scripts/`: `node --test scripts/*-test.js scripts/lib/*-test.js`
|
||||
- [ ] Hook tests green if you touched `hooks/`: all three `bash hooks/*-test.sh`
|
||||
- [ ] New skill → eval case file present with the minimum trigger/behavioral counts
|
||||
- [ ] New skill → gap justified in the PR description; catalog and open PRs checked
|
||||
- [ ] No duplicated content; cross-references used instead
|
||||
|
||||
+2
-2
@@ -15,8 +15,8 @@ What neither provides is a **deterministic, CI-safe** check for a multi-skill *c
|
||||
|
||||
| Tier | What it checks | Runs | Cost |
|
||||
|---|---|---|---|
|
||||
| 1. Structural | Frontmatter, naming, required sections, command parity | CI (`validate-skills.js`, `validate-commands.js`) | Free |
|
||||
| 2. Trigger & routing | Positive prompts rank their skill top-k; negative prompts don't; no two descriptions near-collide | CI (`run-evals.js`) | Free |
|
||||
| 1. Structural | Frontmatter, naming, required sections, layout and line budget, `references/` links, command parity, artifact paths, manifest versions | CI (`validate-skills.js`, `validate-reference-links.js`, `validate-commands.js`, `validate-artifact-paths.js`, `validate-versions.js`) | Free |
|
||||
| 2. Trigger & routing | Positive prompts rank their skill top-k; negative prompts don't; no two descriptions near-collide | CI (`run-evals.js --min-rank1 95`) | Free |
|
||||
| 3. Behavioral | An agent following the skill satisfies its `expectations[]` | On demand (`run-evals.js --behavioral`) | Tokens |
|
||||
|
||||
Tier 2 is a **lexical approximation** of routing (stemmed TF-IDF over descriptions). It cannot judge semantics — that's Tier 3's job — but it catches the two failure modes that dominate real trigger bugs: a description missing the vocabulary users say (false negative), and an over-broad description that outranks the right skill (false positive). A Tier-2 failure usually means *fix the description*, not the eval.
|
||||
|
||||
Reference in new issue
Block a user