mirror of
https://github.com/addyosmani/agent-skills.git
synced 2026-11-14 00:22:35 +00:00
Addy Osmani
45e1449138
There was no way to measure whether skills trigger correctly, stay distinct, or change agent behavior. This adds evals, aligned with what the community has converged on, with a deterministic CI tier on top: - evals/cases/<skill>.json for all 24 skills. The evals[] block uses Anthropic skill-creator's evals.json schema verbatim (id, prompt, expected_output, expectations[]) so its runner, benchmarks, and eval viewer work against our files unmodified. A trigger block (this repo's extension) adds positive/negative routing prompts per skill. - scripts/run-evals.js, zero-dependency runner: Tier 2 (CI): trigger evals via stemmed TF-IDF ranking over skill descriptions (positive prompts must rank top-k, negative prompts must not rank first), catalog collision detection between skill descriptions, schema and coverage checks. Tier 3 (opt-in): --behavioral <skill> executes each eval through headless claude -p and grades the transcript against expectations[] (superpowers-style); --dry-run previews without spending tokens. - CI: run the deterministic tier in the validate-skills job. - Docs: evals/README.md defines the framework and prior art; CONTRIBUTING requires an eval file for new skills (warning-level in the runner until in-flight skill PRs clear); CLAUDE.md pointers. Current baseline: 120 checks pass, 85% trigger rank-1 rate across 72 positive prompts, zero catalog collisions.
40 lines
1.1 KiB
JSON
40 lines
1.1 KiB
JSON
{
|
|
"skill_name": "source-driven-development",
|
|
"trigger": {
|
|
"positive": [
|
|
{
|
|
"prompt": "Verify against the official Next.js docs before implementing this",
|
|
"top_k": 3
|
|
},
|
|
{
|
|
"prompt": "I want source-cited code for the new Stripe integration",
|
|
"top_k": 3
|
|
},
|
|
{
|
|
"prompt": "Ground every framework decision in the official documentation",
|
|
"top_k": 3
|
|
}
|
|
],
|
|
"negative": [
|
|
{
|
|
"prompt": "Fix the flaky test in CI"
|
|
},
|
|
{
|
|
"prompt": "Break the spec into ordered tasks"
|
|
}
|
|
]
|
|
},
|
|
"evals": [
|
|
{
|
|
"id": 1,
|
|
"prompt": "Implement session handling with the framework's recommended approach, citing sources.",
|
|
"expected_output": "An implementation grounded in official documentation with citations, flagging anything unverified",
|
|
"expectations": [
|
|
"Claims about framework behavior cite official documentation",
|
|
"Unverified assumptions are flagged rather than presented as fact",
|
|
"Deprecated patterns are avoided in favor of documented current ones"
|
|
]
|
|
}
|
|
]
|
|
}
|