mirror of
https://github.com/addyosmani/agent-skills.git
synced 2026-11-12 15:46:30 +00:00
Addy Osmani
45e1449138
There was no way to measure whether skills trigger correctly, stay distinct, or change agent behavior. This adds evals, aligned with what the community has converged on, with a deterministic CI tier on top: - evals/cases/<skill>.json for all 24 skills. The evals[] block uses Anthropic skill-creator's evals.json schema verbatim (id, prompt, expected_output, expectations[]) so its runner, benchmarks, and eval viewer work against our files unmodified. A trigger block (this repo's extension) adds positive/negative routing prompts per skill. - scripts/run-evals.js, zero-dependency runner: Tier 2 (CI): trigger evals via stemmed TF-IDF ranking over skill descriptions (positive prompts must rank top-k, negative prompts must not rank first), catalog collision detection between skill descriptions, schema and coverage checks. Tier 3 (opt-in): --behavioral <skill> executes each eval through headless claude -p and grades the transcript against expectations[] (superpowers-style); --dry-run previews without spending tokens. - CI: run the deterministic tier in the validate-skills job. - Docs: evals/README.md defines the framework and prior art; CONTRIBUTING requires an eval file for new skills (warning-level in the runner until in-flight skill PRs clear); CLAUDE.md pointers. Current baseline: 120 checks pass, 85% trigger rank-1 rate across 72 positive prompts, zero catalog collisions.
41 lines
1.2 KiB
JSON
41 lines
1.2 KiB
JSON
{
|
|
"skill_name": "code-simplification",
|
|
"trigger": {
|
|
"positive": [
|
|
{
|
|
"prompt": "This function works but it is way too clever, simplify it without changing behavior",
|
|
"top_k": 3
|
|
},
|
|
{
|
|
"prompt": "Reduce the complexity of this module so juniors can maintain it",
|
|
"top_k": 3
|
|
},
|
|
{
|
|
"prompt": "Clean up this working code, it has grown hard to follow",
|
|
"top_k": 3
|
|
}
|
|
],
|
|
"negative": [
|
|
{
|
|
"prompt": "Add a feature flag system to the app"
|
|
},
|
|
{
|
|
"prompt": "Diagnose why the build broke overnight"
|
|
}
|
|
]
|
|
},
|
|
"evals": [
|
|
{
|
|
"id": 1,
|
|
"prompt": "Simplify the provided 80-line function that parses config files, preserving exact behavior.",
|
|
"expected_output": "A simpler implementation with identical behavior and a summary of what was removed and why",
|
|
"expectations": [
|
|
"Behavior is preserved (tests unchanged and passing, or equivalence argued concretely)",
|
|
"Complexity is reduced rather than relocated",
|
|
"The response explains what was removed and why it was safe",
|
|
"No new features are added during the simplification"
|
|
]
|
|
}
|
|
]
|
|
}
|