One clause on the description: the skill applies when asked to
simplify, clean up or reduce the nesting of a function that already
works, even when the code is pasted inline. Two Tier 2 positives for
the clause, a fires and stays-quiet pair under evals/plugin/, and a
paragraph in evals/README.md with the measurements.
claude-opus-5-5 pinned, Claude Code 2.1.296, three runs per prompt per
description: the three simplification phrasings went from 0 of 9 to
8 of 9, five adjacent must-not-fire prompts stayed at 0 of 15 both
ways. On the graded pair the judge passed 3 of 3 with the plugin and
0 of 3 without.
Six negative-trigger prompts across six case files declared no `owner`,
so per evals/README.md and scripts/run-evals.js they could only pass
vacuously (skip the pairwise routing assertion) instead of asserting
that the correct skill actually outranks the tested one.
Added owner for the prompts where a real, unambiguous owner exists and
verified against scripts/run-evals.js that it outranks the tested skill
with a non-zero score:
- code-simplification: "Add a feature flag system to the app" -> incremental-implementation
- deprecation-and-migration: "Add dark mode to the settings page" -> frontend-ui-engineering
- doubt-driven-development: "Write the changelog entry for the release" -> git-workflow-and-versioning
- frontend-ui-engineering: "Tag and publish the release" -> git-workflow-and-versioning
- planning-and-task-breakdown: "Encode the output so it is safe against XSS" -> security-and-hardening
- using-agent-skills: "Make the modal accessible for keyboard users" -> frontend-ui-engineering
Left four other owner-less negatives untouched (doubt-driven-development's
"Format this file with prettier", git-workflow-and-versioning's "Add a
loading spinner to the submit button", observability-and-instrumentation's
"Draft the PRD for the referral program", and shipping-and-launch's "Why
is this function so slow on large inputs?") because no skill scores above
zero against them under the current TF-IDF scorer, so declaring an owner
there would either fail CI or force a misleading pick.
Verified: node scripts/run-evals.js still passes (127 checks, 0 errors,
86% rank-1) and node scripts/validate-commands.js passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per review from @federicobartoli and @nucliweb on #342:
Tier 3 (behavioral):
- Grade the execution trace, not the final output: executor runs with
--output-format stream-json --verbose so the grader judges tool calls
and file edits rather than the model's self-reporting.
- Run each eval in a throwaway workspace; files[] fixtures materialize
from evals/fixtures/ so evals can operate on real code.
- Node-level timeouts on executor and grader calls; grader output parsed
and shape-validated before writing (raw saved on failure); the trace is
fenced as untrusted data in the grader prompt.
- All 24 behavioral evals flagged trust_level: "provisional" until they
gain fixtures; the runner surfaces this and exits nonzero on failed
expectations.
Tier 2 (deterministic):
- Negative triggers accept an "owner" skill that must outrank this one,
turning them into pairwise routing tests that cannot pass vacuously;
37 of 48 negatives now declare owners (the rest are tracked in #351).
- Warn when a case file is below the documented minimums (3 positive /
2 negative / 1 behavioral); promotion to error tracked in #352.
- Stemmer: cluster trailing y/i ("simplify"/"simplifies").
Baseline holds: 120 checks, 0 errors, 85% trigger rank-1 rate.
There was no way to measure whether skills trigger correctly, stay
distinct, or change agent behavior. This adds evals, aligned with what
the community has converged on, with a deterministic CI tier on top:
- evals/cases/<skill>.json for all 24 skills. The evals[] block uses
Anthropic skill-creator's evals.json schema verbatim (id, prompt,
expected_output, expectations[]) so its runner, benchmarks, and eval
viewer work against our files unmodified. A trigger block (this
repo's extension) adds positive/negative routing prompts per skill.
- scripts/run-evals.js, zero-dependency runner:
Tier 2 (CI): trigger evals via stemmed TF-IDF ranking over skill
descriptions (positive prompts must rank top-k, negative prompts
must not rank first), catalog collision detection between skill
descriptions, schema and coverage checks.
Tier 3 (opt-in): --behavioral <skill> executes each eval through
headless claude -p and grades the transcript against expectations[]
(superpowers-style); --dry-run previews without spending tokens.
- CI: run the deterministic tier in the validate-skills job.
- Docs: evals/README.md defines the framework and prior art;
CONTRIBUTING requires an eval file for new skills (warning-level in
the runner until in-flight skill PRs clear); CLAUDE.md pointers.
Current baseline: 120 checks pass, 85% trigger rank-1 rate across 72
positive prompts, zero catalog collisions.
ayobamiseun