diff --git a/docs/gateguard/design-notes.md b/docs/gateguard/design-notes.md index 62f132709..8baf6bc59 100644 --- a/docs/gateguard/design-notes.md +++ b/docs/gateguard/design-notes.md @@ -180,6 +180,20 @@ least four characters and is not generic (`index`, `utils`, `config`, ...). Matching is a word-boundary `indexOf` scan; a `RegExp` is never built from transcript text. +## Test stems + +A test is named after the module it exercises, and a search for that module +(`rg tokenizer src tests`) is the search that finds its tests. So for a target +of the test class, one test affix is stripped before the stem rules apply: +a `.test` or `.spec` suffix, a `test_` prefix on `.py`, and a `_test` suffix +on `.py` and `.go`, the same affixes that make a file a test. The result +still has to be at least four characters, not generic (`index.test.js` gives +`index`, which never credits), a whole word in the search, and in the +search's scope; exclusions that mention the stripped stem cover the target. +Only one affix is stripped, and other classes keep the full stem (a +`tokenizer.test.md` under `skills/` is an instruction file). The full stem +(`tokenizer.test`) still matches as a word, since `.` is a word boundary. + ## Search filters A search that excluded the target never saw it: diff --git a/docs/gateguard/evaluation.md b/docs/gateguard/evaluation.md index 0658baff4..1470164b3 100644 --- a/docs/gateguard/evaluation.md +++ b/docs/gateguard/evaluation.md @@ -35,8 +35,10 @@ Metrics per hook: - **Irrelevant questions asked**: for denials of code targets, asked question ids that are not in `relevantQuestions`. Questions are read from the numbered list of a full denial, or from the phrases of a condensed one-line - denial (issued after the first three denials of a session); the condensed - rows count the ones asked in condensed denials only. + denial (issued after the first three denials of a session): the working + tree's per-question phrases (`condensedQuestionPhrase`) and the fixed hint + older hooks use. The condensed rows count the ones asked in condensed + denials only. - **Warranted questions not asked**: `relevantQuestions` ids a code denial did not ask. Asking fewer questions must not drop the ones that matter. - **Estimated denial tokens**: characters of all denial reasons divided by 4. @@ -55,39 +57,39 @@ relatively at that ref, into a temp tree that mirrors the repo layout. ## Results Working tree = branch `gateguard-full`; `upstream/main` = `bd9402f7`; -`pr` = `58a4a0d0` (the pull request head before this round). Node 22, Linux. +`58a4a0d0` = the pull request head before this round. Node 22, Linux. ```text Corpus: 16 scenarios, 144 steps. ``` -| Metric | working tree | upstream/main | pr | +| Metric | working tree | upstream/main | 58a4a0d0 | | --- | ---: | ---: | ---: | | Steps | 144 | 144 | 144 | -| Denials | 99 | 127 | 105 | -| Redundant denials | 0 | 33 | 13 | +| Denials | 97 | 127 | 105 | +| Redundant denials | 0 | 35 | 15 | | Must-deny bypasses | 0 | 2 | 2 | -| Expectation mismatches | 0 | 42 | 20 | -| Irrelevant questions asked | 36 | 173 | 138 | -| Irrelevant questions in condensed denials | 36 | 100 | 79 | -| Warranted questions not asked | 8 | 54 | 41 | -| Estimated denial tokens | 19379 | 22583 | 20142 | -| Allows with a credit note | 8 | 0 | 8 | +| Expectation mismatches | 0 | 44 | 22 | +| Irrelevant questions asked | 2 | 173 | 138 | +| Irrelevant questions in condensed denials | 2 | 100 | 79 | +| Warranted questions not asked | 1 | 54 | 41 | +| Estimated denial tokens | 18991 | 22583 | 20142 | +| Allows with a credit note | 9 | 0 | 8 | | Allows with a sibling note | 11 | 0 | 11 | -| Allows with a trivial-edit note | 3 | 0 | 0 | -| Hook latency p50 (ms) | 1.88 | 0.97 | 1.60 | -| Hook latency p95 (ms) | 7.57 | 2.75 | 6.57 | +| Allows with a trivial-edit note | 4 | 0 | 0 | +| Hook latency p50 (ms) | 1.77 | 0.92 | 1.46 | +| Hook latency p95 (ms) | 7.93 | 3.08 | 6.23 | -| Scenario | Steps | Denials: working tree | Denials: upstream/main | Denials: pr | +| Scenario | Steps | Denials: working tree | Denials: upstream/main | Denials: 58a4a0d0 | | --- | ---: | ---: | ---: | ---: | | docs-heavy-session | 14 | 6 | 12 | 6 | | scaffold-module | 10 | 3 | 10 | 3 | -| bugfix-after-scoped-search | 6 | 2 | 3 | 2 | +| bugfix-after-scoped-search | 6 | 1 | 3 | 2 | | cold-writes | 5 | 3 | 4 | 3 | | exported-api-edits | 6 | 6 | 6 | 6 | | internal-only-edits | 5 | 4 | 4 | 4 | | data-handling-edits | 5 | 5 | 5 | 5 | -| comment-only-edits | 10 | 7 | 9 | 9 | +| comment-only-edits | 10 | 6 | 9 | 9 | | sensitive-targets | 8 | 8 | 8 | 8 | | subagent-edits | 6 | 3 | 1 | 1 | | first-shell-commands | 20 | 11 | 17 | 17 | @@ -101,37 +103,43 @@ Corpus: 16 scenarios, 144 steps. Against `upstream/main`: -- 22% fewer denials (127 to 99) and 14% fewer denial tokens, with every - redundant denial in the corpus gone (33 to 0). +- 24% fewer denials (127 to 97) and 16% fewer denial tokens, with every + redundant denial in the corpus gone (35 to 0). - Both must-deny bypasses closed: `upstream/main` lets a subagent edit `src/auth/oauth.js` and `config/secrets.yaml` without a question. -- Irrelevant questions drop from 173 to 36 and warranted-but-unasked from 54 - to 8, mostly because edits without a public-surface line now ask for local - call sites instead of importers, and the data-schema question is asked only - when the change touches data. +- Irrelevant questions drop from 173 to 2 and warranted-but-unasked from 54 + to 1: edits without a public-surface line ask for local call sites instead + of importers, the data-schema question is asked only when the change + touches data, and condensed denials name the same questions as full ones. - Denials rise only where they should: the subagent's first touch of a sensitive file, the first mutating shell command after a read-only one (`upstream/main` spends its once-per-session routine gate on `ls`), and the first code-changing edit after a comment-only one (the comment edit no longer spends the file's first touch). -Against `pr` (this round's increment): 6 fewer denials net (8 fewer from -trivial comment and whitespace edits and read-only first shell commands, 2 more -from subagent edits of sensitive files), redundant denials 13 to 0, both -subagent bypasses closed, and irrelevant questions 138 to 36. +Against `58a4a0d0` (this round's increment): 8 fewer denials net (3 fewer +in the comment-only scenario: four trivial passes, one of them a shell +script, less the later code edit they no longer pre-check; 6 fewer from +read-only first shell commands; 1 fewer from the test edit credited by +`rg tokenizer src tests`; 2 more from subagent edits of sensitive files), +redundant denials 15 to 0, both subagent bypasses closed, and irrelevant +questions 138 to 2. + +Two corpus labels changed in this round because the working tree now handles +those steps, so both baselines gained a redundant denial: the comment-only +edit of `scripts/build.sh` (`unknown-extension-comment`, now profiled as a +shell script) and the test edit after `rg tokenizer src tests` +(`edit-tokenizer-test`, now credited by the stripped stem `tokenizer`). What the working tree still gets wrong, by the corpus's own labels: -- All 36 remaining irrelevant questions, and 6 of the 8 unasked warranted - ones, come from condensed denials. The condensed hint always mentions "data - schemas if any", and the code-Write hint asks for "affected API" but not - whether an existing file already serves the purpose; full denials in the - corpus ask no irrelevant question. -- A one-line edit of a column-0 Python constant reads as not touching the - public surface (the first line of a snippet may start mid-line), so it asks - for local call sites instead of importers (`python-hash-in-string`). -- A comment-only edit of a file without a change profile (`.sh`) keeps the - full questions (`unknown-extension-comment`). +- Sensitive targets are never profiled, so a sensitive code Write always asks + the data-schema question (`symlinked-auth-first`, `symlinked-auth-sibling`: + the 2 irrelevant questions). +- The C edit that turns a declaration into a comment continuation + (`c-comment-continuation`) has no data words, so the condensed hint no + longer mentions data schemas; the corpus labels that question as warranted + (the 1 unasked one). - Latency roughly doubles against `upstream/main` (p50 about 2 ms, p95 about 8 ms) with the added transcript scanning and path resolution; both stay far below the 200 ms budget for blocking hooks. @@ -141,13 +149,14 @@ What the working tree still gets wrong, by the corpus's own labels: ```bash git fetch upstream main node scripts/dev/gateguard-eval.js --markdown -node scripts/dev/gateguard-eval.js --markdown --baseline upstream/main --baseline pr +node scripts/dev/gateguard-eval.js --markdown --baseline upstream/main --baseline 58a4a0d0 node scripts/dev/gateguard-eval.js --json > gateguard-eval.json node tests/hooks/gateguard-scenarios.test.js ``` `--baseline ` takes any ref and can be repeated (default -`upstream/main`; `pr` is `58a4a0d0`). `--corpus ` points at another +`upstream/main`); pass the pull request head (`58a4a0d0` above) to compare +against it. `--corpus ` points at another scenario directory. The script exits non-zero when the working tree has a mismatch, a must-deny bypass, an explicit `allow` decision or a thrown error. @@ -184,8 +193,8 @@ created, such as Windows without the privilege. - The labels are the corpus author's judgement, made with the design in mind. `redundant` and `relevantQuestions` in particular are opinions: a different author could call the README or CHANGELOG edits after a docs-wide search - redundant, or the test edit after `rg tokenizer src tests` (the stem of - `tokenizer.test.js` is `tokenizer.test`, so that search does not credit it). + redundant, or leave out the data-schema question for + `c-comment-continuation`. Zero redundant denials means the working tree handles the cases this author judged redundant, not that it never repeats a question. - Question relevance is read from denial text. Condensed denials are mapped by diff --git a/docs/ja-JP/skills/gateguard/SKILL.md b/docs/ja-JP/skills/gateguard/SKILL.md index 1ad788f7b..3bc67f3dd 100644 --- a/docs/ja-JP/skills/gateguard/SKILL.md +++ b/docs/ja-JP/skills/gateguard/SKILL.md @@ -126,7 +126,7 @@ Triggers on: `rm -rf`, `git reset --hard`, `git push --force`, `drop table`, etc - **ターン:** 最新の実際のユーザーメッセージ、またはコンパクション（compact summary レコードや `compact_boundary`）から始まります。コンパクション前の検索は数えません。読むのはトランスクリプト末尾（256 KiB / 2000 行）だけで、ターン開始がその範囲より前にある場合は範囲全体を現在のターンとみなします。ターン開始がまったくないトランスクリプトではクレジットしません。 - **結果:** エラーでない `tool_result` が必要です（`is_error` が `true`/`"true"`、または内容が `` で始まるものはエラー）。同じ tool_use id が複数回現れる検索は数えません。 - **同一バッチの除外:** ゲート対象の呼び出しと同じアシスタントメッセージ内の検索は数えません。`tool_use_id` がない、またはトランスクリプトに見つからない場合は、ターン内の最新のアシスタントメッセージを除外します。message id のない検索は数えません。 - - **語幹一致:** ファイル名の語幹（4 文字以上、`index`・`utils`・`config` などの汎用名は除外）が、英数字以外の文字または端で区切られた単語として検索文字列に現れる必要があります（`payment.py` は `payment` や `payment_service` の検索で一致し、`payments` では一致しません）。 + - **語幹一致:** ファイル名の語幹（4 文字以上、`index`・`utils`・`config` などの汎用名は除外）が、英数字以外の文字または端で区切られた単語として検索文字列に現れる必要があります（`payment.py` は `payment` や `payment_service` の検索で一致し、`payments` では一致しません）。テスト対象では先にテストの接辞を 1 つ外します（`.test`、`.spec`、Python と Go の `test_`/`_test`）。`rg tokenizer src tests` は `tests/tokenizer.test.js` をクレジットしますが、`index.test.js` は汎用名 `index` になるため一致しません。 - **除外は数えない:** Grep の `glob` はトップレベルのカンマで分割し、`!` で始まるものは除外です。シェルでは `rg -g`/`--glob`/`--iglob` の `!` 値、`grep --exclude`/`--exclude-dir`、`fd -E`、`ls -I`、`tree -I`、`git ls-files -x`、git の `:!x` パススペック、`-not`/`!` 付きまたは `-prune` が続く `find -name`/`-path`、PowerShell の `-Exclude` が除外です。除外は語幹の出どころになりません。除外が対象（ファイル名またはパス上のディレクトリ、または語幹）に当たる検索はその対象を一切クレジットしません（`rg -g '!widget.py' TODO .` は `widget.py` をクレジットせず、`rg -g '!*.md' widget .` はクレジットします）。`--exclude-from`/`--ignore-file` を含むセグメントはクレジットしません。`*.py`・`--include`・`find -name` などのファイル名グロブで絞った検索は、対象のファイル名に一致する場合だけ語幹一致を数えます。 - **範囲:** `Glob`/`Grep`/`LS` は検索したディレクトリ（`path`、なければツールの `cwd`、Glob はパターンのリテラル部分を加える）の内側にある対象だけをクレジットします。対象自体を読むだけのシェルセグメント（例: `ls src/a.py`）や、パス引数がなく再帰でもない・パイプで stdin を読むセグメントは検索とみなしません。`$( )`、バッククォート、`<( )`、`>( )`、ヒアドキュメントを含むコマンドは対象外です。シェルセグメントは、パス引数（なければ `cwd`）の内側にある対象だけをクレジットします（`rg payment docs` は `src/payment.py` をクレジットしません）。引数より前の入力リダイレクト（`rg payment < file`）は stdin とみなし、別の呼び出しで `cd` した後の相対パス引数は数えません。 - **ディレクトリ一致:** まだ存在しないファイルを作成する Write だけが、対象のディレクトリそのものを指す検索でも一致します（Glob の `path` とリテラル接頭辞、Grep/LS の `path`、シェルのパス引数）。シェル引数では grep 系のパターン引数や値を取るフラグの値を除き、今実在するディレクトリだけを数えます。ターン内のいずれかの Bash/PowerShell 呼び出しに `cd`/`pushd`/`Set-Location` などがあれば、そのターンのシェルのディレクトリ一致は無効です。Edit、MultiEdit、既存ファイルを上書きする Write は語幹一致のみです。 diff --git a/docs/zh-CN/skills/gateguard/SKILL.md b/docs/zh-CN/skills/gateguard/SKILL.md index 0d5e68d10..9d6c62d1d 100644 --- a/docs/zh-CN/skills/gateguard/SKILL.md +++ b/docs/zh-CN/skills/gateguard/SKILL.md @@ -92,7 +92,7 @@ LLM 的自我评估不起作用。问"你是否违反了任何策略？"答案 * **目标类别：** 首次编辑/写入按项目相对路径（去掉开头的 `.claude/worktrees//`）依次判定为 instruction（`CLAUDE.md`、`AGENTS.md`、`SKILL.md`、`.mdc`、`.cursorrules`、`.github/copilot-instructions.md` 等）、test、prose（`.md` 等）、config（`.json`、`.yaml`、`.env` / `.env.*` 等，`.env.example.ts` 仍属 code）或 code，并提出相应的问题。 * **按变更内容提问：** 对 code 类目标还会读取变更内容：未触及公开声明的编辑改问"本文件或其模块中依赖所改行为的调用点"，不涉及数据的变更省略数据格式问题；无法可靠判断（不支持的扩展名、超过 64 KiB 等）以及敏感目标时仍问原来的四个问题。 -* **事先搜索抵免：** 若当前人类回合内已有成功完成（非错误结果）的 `Glob`、`Grep`、`LS` 或 `rg`/`grep`/`find` 等搜索命中该文件，则以附注代替拒绝。`Read` 从不计入；与被门控调用同批次的搜索、重复的 tool_use id、压缩（compaction）之前的搜索均不计入。文件名词干须按单词边界匹配，且目标须位于所搜索目录之内（shell 片段按其路径参数或 `cwd` 判定范围；参数前的输入重定向 `<` 视为读取 stdin）。仅"创建尚不存在的文件"的写入可按目录匹配，且 shell 参数须是真实存在的目录；回合内出现 `cd` 等切换目录命令时不再按 shell 目录匹配。相对路径按工具的 `cwd` 解析。排除规则从不计入：`Grep` 的 `glob` 中以 `!` 开头的项，以及 shell 中 `rg -g '!x'`、`grep --exclude`/`--exclude-dir`、`fd -E`、`ls -I`、`tree -I`、`git ls-files -x`、git `:!x` 路径规格、带 `-not`/`!` 或后接 `-prune` 的 `find -name`/`-path`、PowerShell `-Exclude` 都不作为词干来源；若排除项覆盖目标（文件名、路径上的目录或词干），该搜索完全不能为其抵免（`rg -g '!widget.py' TODO .` 不抵免 `widget.py`，`rg -g '!*.md' widget .` 仍抵免）。含 `--exclude-from`/`--ignore-file` 的片段不抵免；用 `*.py`、`--include`、`find -name` 等文件名 glob 限定的搜索，只有在其匹配目标文件名时词干才计入。 +* **事先搜索抵免：** 若当前人类回合内已有成功完成（非错误结果）的 `Glob`、`Grep`、`LS` 或 `rg`/`grep`/`find` 等搜索命中该文件，则以附注代替拒绝。`Read` 从不计入；与被门控调用同批次的搜索、重复的 tool_use id、压缩（compaction）之前的搜索均不计入。文件名词干须按单词边界匹配（测试目标先去掉一个测试词缀：`.test`、`.spec`，以及 Python/Go 的 `test_`/`_test`，因此 `rg tokenizer src tests` 可抵免 `tests/tokenizer.test.js`，而 `index.test.js` 仍是通用词干 `index`），且目标须位于所搜索目录之内（shell 片段按其路径参数或 `cwd` 判定范围；参数前的输入重定向 `<` 视为读取 stdin）。仅"创建尚不存在的文件"的写入可按目录匹配，且 shell 参数须是真实存在的目录；回合内出现 `cd` 等切换目录命令时不再按 shell 目录匹配。相对路径按工具的 `cwd` 解析。排除规则从不计入：`Grep` 的 `glob` 中以 `!` 开头的项，以及 shell 中 `rg -g '!x'`、`grep --exclude`/`--exclude-dir`、`fd -E`、`ls -I`、`tree -I`、`git ls-files -x`、git `:!x` 路径规格、带 `-not`/`!` 或后接 `-prune` 的 `find -name`/`-path`、PowerShell `-Exclude` 都不作为词干来源；若排除项覆盖目标（文件名、路径上的目录或词干），该搜索完全不能为其抵免（`rg -g '!widget.py' TODO .` 不抵免 `widget.py`，`rg -g '!*.md' widget .` 仍抵免）。含 `--exclude-from`/`--ignore-file` 的片段不抵免；用 `*.py`、`--include`、`find -name` 等文件名 glob 限定的搜索，只有在其匹配目标文件名时词干才计入。 * **仅注释/空白的编辑：** 对受支持语言中非敏感的 code/test/prose 文件，只改动注释和空白的编辑（MultiEdit 须该文件的所有条目都如此）以附注放行并计入 `trivial_allows`，但不标记为已检查，下一次改动代码的编辑仍按首次触碰门控（写入、Python 缩进变化、多行字符串等从不算作此类编辑）。 * **拒绝中指出最接近的未计入搜索：** 非敏感目标被拒绝且本回合有提及该文件却未能抵免的调用时，拒绝消息追加一行"Closest search this turn did not count (...)"，说明最接近的一个及原因（同批次、被排除、范围外、读取 stdin、`Read` 等非搜索、文件名过于通用）；细节经净化并截断至 60 个字符。 * **同回合同级文件合并：** 同一目录、同一类别（仅 code、test、prose）的新文件在同一回合内（完全没有 transcript 路径时为 120 秒内）只拒绝第一个，其余以附注放行。config、instruction，以及任何以 `.` 开头的路径段（点文件和所有点目录）或 8.3 短文件名下的文件从不合并；目录按解析符号链接后的真实位置判定和记录；有 transcript 路径但文件缺失、无法读取或得不到回合 ID 时不合并；编辑从不合并。 diff --git a/manifests/context-packs/skill-triggers@1.json b/manifests/context-packs/skill-triggers@1.json index 97a04ee88..56fb553b4 100644 --- a/manifests/context-packs/skill-triggers@1.json +++ b/manifests/context-packs/skill-triggers@1.json @@ -1 +1 @@ -{"coverage":{"skills":293,"withTriggers":32},"generatedAt":"2026-09-24T23:51:22.784Z","id":"skill-triggers@1","model":{"effort":null,"id":"hand-seeded","source":"manual-curation-pending-regeneration"},"registryDigest":"f307355aabdc54aa83309b70c606373aeaec44d95cb83ad82dcc03396947bb45","schemaVersion":1,"triggers":{"skill:api-connector-builder":["add api integration","new provider connector","match existing integration pattern"],"skill:api-design":["rest endpoint design","pagination api","status codes","api versioning","rate limiting api","resource naming","filtering api","api error responses","offset pagination","limit query parameter","pagination defaults"],"skill:backend-patterns":["express api","node backend architecture","nextjs api routes","server side patterns","data access layer","static file server","url path handling","file server"],"skill:browser-qa":["deployed feature test","visual regression screenshots","core web vitals check","axe accessibility audit","ship do not ship","staging verification"],"skill:canary-watch":["post deploy monitoring","smoke test url","production url check","console errors production","sse stream check","after deploy verification"],"skill:code-tour":["onboarding walkthrough","explain subsystem","architecture tour","pr walkthrough","rca tour"],"skill:coding-standards":["code review standards","naming conventions","readability review","immutability conventions","fix naming typo","export naming","consistent exports"],"skill:content-hash-cache-pattern":["cache file processing","content addressed cache","sha256 hash cache"],"skill:database-migrations":["zero downtime migration","schema change production","add column large table","backfill data","expand contract","concurrent index","migration rollback","prisma migration","django migration"],"skill:deployment-patterns":["ci cd setup","dockerize app","health checks","rollback strategy","production readiness","deploy pipeline","containerize application"],"skill:design-system":["design tokens","visual consistency audit","css custom properties","ui audit","design system bootstrap"],"skill:django-patterns":["django orm","drf api","django rest framework","django caching","django signals","django middleware"],"skill:django-security":["django authentication","csrf protection","sql injection prevention","xss prevention","django deployment security","role based access control","authorization middleware","permissions checks"],"skill:docker-patterns":["dockerfile review","docker compose setup","container security","multi service orchestration"],"skill:error-handling":["error types","retry logic","circuit breaker","user facing errors","exception handling patterns","typed errors","error boundaries","go error handling","custom error class","error codes","config validation"],"skill:evm-token-decimals":["token decimals","wei conversion","erc20 balance off","bridge token precision"],"skill:frontend-a11y":["aria attributes","screen reader support","focus management","semantic html","form labeling","keyboard navigation react","a11y lint errors"],"skill:git-workflow":["merge vs rebase","commit conventions","resolve merge conflict","branching strategy","clean up commits","pull request cleanup","git history tidy"],"skill:hexagonal-architecture":["ports and adapters","dependency injection boundaries","decouple domain from io"],"skill:kubernetes-patterns":["kubernetes manifests","kubectl debugging","pod probes","k8s rbac","autoscaling config","configmap secrets"],"skill:orch-fix-defect":["fix a bug","broken behavior","regression fix","reproduce bug","defect repair"],"skill:postgres-patterns":["slow postgres query","query optimization","index design","rls policies","supabase schema","postgres indexing","database performance","schema design postgres","postgres driver","node postgres","query planner"],"skill:python-patterns":["pythonic code","pep 8","type hints python","python code review","idiomatic python"],"skill:python-testing":["pytest fixtures","mocking python","parametrized tests","coverage python","tdd python"],"skill:redis-patterns":["cache aside pattern","distributed lock","redis rate limiting","cache invalidation"],"skill:regex-vs-llm-structured-text":["parse invoice","extract receipt data","text extraction pipeline","parse form fields","cheap document parser","extract table data","parse log lines","parse access logs","common log format","log line parsing"],"skill:rust-patterns":["rust ownership","borrow checker","rust error handling","traits rust","rust concurrency","idiomatic rust"],"skill:search-first":["find existing library","npm package research","before writing custom code","evaluate existing tools","add dependency research"],"skill:security-review":["security audit","authentication review","sanitize user input","secrets handling","payment security checklist","prevent injection attacks","secure api endpoints","authn authz review","vulnerability checklist","input validation security","parameterized queries","sql injection"],"skill:security-scan":["audit claude config","claudemd security","mcp server audit","agentshield scan","hook configuration audit","settings json security"],"skill:tdd-workflow":["write test first","failing test","red green refactor","test driven development","regression test first","write a regression test"],"skill:verification-loop":["pre pr checks","verification report","quality gates","build lint test coverage","before creating a pr"]},"triggersDigest":"25b97a9e06fc336c7cf95ab854ed1a41033a54bcd6e1fb1cf69dc906332462aa"} +{"coverage":{"skills":293,"withTriggers":32},"generatedAt":"2026-09-24T23:51:22.784Z","id":"skill-triggers@1","model":{"effort":null,"id":"hand-seeded","source":"manual-curation-pending-regeneration"},"registryDigest":"2c34de198272f96fd120fdb1131f87a63bd74c0a5c88275538a9404a0e534575","schemaVersion":1,"triggers":{"skill:api-connector-builder":["add api integration","new provider connector","match existing integration pattern"],"skill:api-design":["rest endpoint design","pagination api","status codes","api versioning","rate limiting api","resource naming","filtering api","api error responses","offset pagination","limit query parameter","pagination defaults"],"skill:backend-patterns":["express api","node backend architecture","nextjs api routes","server side patterns","data access layer","static file server","url path handling","file server"],"skill:browser-qa":["deployed feature test","visual regression screenshots","core web vitals check","axe accessibility audit","ship do not ship","staging verification"],"skill:canary-watch":["post deploy monitoring","smoke test url","production url check","console errors production","sse stream check","after deploy verification"],"skill:code-tour":["onboarding walkthrough","explain subsystem","architecture tour","pr walkthrough","rca tour"],"skill:coding-standards":["code review standards","naming conventions","readability review","immutability conventions","fix naming typo","export naming","consistent exports"],"skill:content-hash-cache-pattern":["cache file processing","content addressed cache","sha256 hash cache"],"skill:database-migrations":["zero downtime migration","schema change production","add column large table","backfill data","expand contract","concurrent index","migration rollback","prisma migration","django migration"],"skill:deployment-patterns":["ci cd setup","dockerize app","health checks","rollback strategy","production readiness","deploy pipeline","containerize application"],"skill:design-system":["design tokens","visual consistency audit","css custom properties","ui audit","design system bootstrap"],"skill:django-patterns":["django orm","drf api","django rest framework","django caching","django signals","django middleware"],"skill:django-security":["django authentication","csrf protection","sql injection prevention","xss prevention","django deployment security","role based access control","authorization middleware","permissions checks"],"skill:docker-patterns":["dockerfile review","docker compose setup","container security","multi service orchestration"],"skill:error-handling":["error types","retry logic","circuit breaker","user facing errors","exception handling patterns","typed errors","error boundaries","go error handling","custom error class","error codes","config validation"],"skill:evm-token-decimals":["token decimals","wei conversion","erc20 balance off","bridge token precision"],"skill:frontend-a11y":["aria attributes","screen reader support","focus management","semantic html","form labeling","keyboard navigation react","a11y lint errors"],"skill:git-workflow":["merge vs rebase","commit conventions","resolve merge conflict","branching strategy","clean up commits","pull request cleanup","git history tidy"],"skill:hexagonal-architecture":["ports and adapters","dependency injection boundaries","decouple domain from io"],"skill:kubernetes-patterns":["kubernetes manifests","kubectl debugging","pod probes","k8s rbac","autoscaling config","configmap secrets"],"skill:orch-fix-defect":["fix a bug","broken behavior","regression fix","reproduce bug","defect repair"],"skill:postgres-patterns":["slow postgres query","query optimization","index design","rls policies","supabase schema","postgres indexing","database performance","schema design postgres","postgres driver","node postgres","query planner"],"skill:python-patterns":["pythonic code","pep 8","type hints python","python code review","idiomatic python"],"skill:python-testing":["pytest fixtures","mocking python","parametrized tests","coverage python","tdd python"],"skill:redis-patterns":["cache aside pattern","distributed lock","redis rate limiting","cache invalidation"],"skill:regex-vs-llm-structured-text":["parse invoice","extract receipt data","text extraction pipeline","parse form fields","cheap document parser","extract table data","parse log lines","parse access logs","common log format","log line parsing"],"skill:rust-patterns":["rust ownership","borrow checker","rust error handling","traits rust","rust concurrency","idiomatic rust"],"skill:search-first":["find existing library","npm package research","before writing custom code","evaluate existing tools","add dependency research"],"skill:security-review":["security audit","authentication review","sanitize user input","secrets handling","payment security checklist","prevent injection attacks","secure api endpoints","authn authz review","vulnerability checklist","input validation security","parameterized queries","sql injection"],"skill:security-scan":["audit claude config","claudemd security","mcp server audit","agentshield scan","hook configuration audit","settings json security"],"skill:tdd-workflow":["write test first","failing test","red green refactor","test driven development","regression test first","write a regression test"],"skill:verification-loop":["pre pr checks","verification report","quality gates","build lint test coverage","before creating a pr"]},"triggersDigest":"25b97a9e06fc336c7cf95ab854ed1a41033a54bcd6e1fb1cf69dc906332462aa"} diff --git a/scripts/dev/gateguard-eval.js b/scripts/dev/gateguard-eval.js index 9adba385d..ee805e35f 100644 --- a/scripts/dev/gateguard-eval.js +++ b/scripts/dev/gateguard-eval.js @@ -12,7 +12,6 @@ const REPO_ROOT = path.join(__dirname, '..', '..'); const HOOK_RELATIVE_PATH = 'scripts/hooks/gateguard-fact-force.js'; const DEFAULT_CORPUS_DIR = path.join(REPO_ROOT, 'tests', 'fixtures', 'gateguard-scenarios'); const DEFAULT_BASELINE = 'upstream/main'; -const BASELINE_ALIASES = Object.freeze({ pr: '58a4a0d0' }); const EXPECTATIONS = new Set(['deny', 'allow']); const TOKEN_CHARS = 4; const CONDENSED_PREFIX = /^\[Fact-Forcing Gate\] \(denial #\d+ this session\) First (edit|creation) of /; @@ -120,10 +119,6 @@ function requireTarget(fromFile, spec) { return path.posix.extname(joined) ? joined : `${joined}.js`; } -function resolveBaselineRef(ref) { - return Object.hasOwn(BASELINE_ALIASES, ref) ? BASELINE_ALIASES[ref] : ref; -} - /** Write the hook at a git ref, and every file it requires relatively, into a temp tree; returns the hook path. */ function materializeHook(ref, outDir) { const seen = new Set(); @@ -474,7 +469,7 @@ const USAGE = [ 'Usage: node scripts/dev/gateguard-eval.js [--markdown | --json] [--baseline ]... [--corpus ]', '', ' --baseline git ref whose hook is compared with the working tree (repeatable;', - ` default ${DEFAULT_BASELINE}; "pr" means ${BASELINE_ALIASES.pr})`, + ` default ${DEFAULT_BASELINE})`, ' --corpus scenario directory (default tests/fixtures/gateguard-scenarios)', ' --markdown print markdown tables (default)', ' --json print the full report as JSON' @@ -503,9 +498,8 @@ async function evaluate({ baselines = [DEFAULT_BASELINE], corpus = DEFAULT_CORPU const hooks = [{ label: 'working tree', ref: null, file: path.join(REPO_ROOT, ...HOOK_RELATIVE_PATH.split('/')) }]; const materialized = fs.mkdtempSync(path.join(os.tmpdir(), 'gateguard-baseline-')); try { - baselines.forEach((label, index) => { - const ref = resolveBaselineRef(label); - hooks.push({ label, ref, file: materializeHook(ref, path.join(materialized, String(index))) }); + baselines.forEach((ref, index) => { + hooks.push({ label: ref, ref, file: materializeHook(ref, path.join(materialized, String(index))) }); }); const evaluated = []; for (const hook of hooks) { diff --git a/scripts/lib/gateguard-search-evidence.js b/scripts/lib/gateguard-search-evidence.js index 6b1d2be62..e5dd0c8fc 100644 --- a/scripts/lib/gateguard-search-evidence.js +++ b/scripts/lib/gateguard-search-evidence.js @@ -2,7 +2,7 @@ const fs = require('fs'); const path = require('path'); -const { WINDOWS_PATH_PATTERN } = require('./gateguard-target-class'); +const { WINDOWS_PATH_PATTERN, classifyTargetFor } = require('./gateguard-target-class'); const { excludedBatchId } = require('./gateguard-turn-scan'); // --- Search commands and flags --- @@ -84,10 +84,20 @@ const MAX_SEARCH_COMMAND_CHARS = 8192; const AMBIGUOUS_SHELL_PATTERN = /`|\$\(|[<>]\(|<= MIN_STEM_LENGTH && !GENERIC_STEMS.has(stem) ? stem : null; } @@ -798,7 +808,7 @@ function createSearchEvidence({ quoteAwareSegments, commandBasename, SHELL_SEGME if (typeof targetPath !== 'string' || !targetPath) return null; const ctx = dirContext(targetPath, data); if (!ctx) return null; - const stem = stemMatcher(eligibleStem(targetPath)); + const stem = stemMatcher(eligibleStem(targetPath, data)); // see docs/gateguard/design-notes.md#same-batch-searches const excluded = excludedBatchId(scan, data); const shellDirsTrusted = !turnChangesDirectory(scan.shellCommands); @@ -846,10 +856,9 @@ function createSearchEvidence({ quoteAwareSegments, commandBasename, SHELL_SEGME if (!scan || !Array.isArray(scan.searches) || typeof targetPath !== 'string' || !targetPath) return null; const ctx = dirContext(targetPath, data); if (!ctx) return null; - const eligible = eligibleStem(targetPath); + const eligible = eligibleStem(targetPath, data); const stem = stemMatcher(eligible); - const base = (String(targetPath).split(/[\\/]/).pop() || '').toLowerCase(); - const bare = base.includes('.') ? base.slice(0, base.lastIndexOf('.')) : base; + const bare = bareStem(targetPath, data); const rawStem = eligible === null && bare.length >= MIN_GENERIC_MISS_STEM ? stemMatcher(bare) : null; const excluded = excludedBatchId(scan, data); const shellDirsTrusted = !turnChangesDirectory(scan.shellCommands); diff --git a/skills/gateguard/SKILL.md b/skills/gateguard/SKILL.md index 3936c59f6..562a3c260 100644 --- a/skills/gateguard/SKILL.md +++ b/skills/gateguard/SKILL.md @@ -321,7 +321,11 @@ An Edit, Write, or MultiEdit entry is allowed, with an a search for `payment` or `payment_service`, but not by one for `payments`. Exclusions are never a stem source. Generic stems (`index`, `main`, `init`, `__init__`, `utils`, `util`, `types`, `readme`, `test`, `tests`, `config`, `mod`, - `lib`, `setup`, `app`, `spec`, `helpers`, `common`) never match. + `lib`, `setup`, `app`, `spec`, `helpers`, `common`) never match. For a test + target, one test affix is stripped first (`.test`, `.spec`, and for Python + and Go `test_` or `_test`), so `rg tokenizer src tests` credits + `tests/tokenizer.test.js`; `index.test.js` still reduces to the generic + `index`. - **Exclusions never credit:** a Grep `glob` is split on top-level commas; entries starting with `!` are exclusions. In shell segments, `rg -g`/ `--glob`/`--iglob` values starting with `!`, `grep --exclude`/ diff --git a/tests/fixtures/gateguard-scenarios/03-bugfix-after-scoped-search.json b/tests/fixtures/gateguard-scenarios/03-bugfix-after-scoped-search.json index ab31e6b05..df66dea39 100644 --- a/tests/fixtures/gateguard-scenarios/03-bugfix-after-scoped-search.json +++ b/tests/fixtures/gateguard-scenarios/03-bugfix-after-scoped-search.json @@ -235,6 +235,7 @@ }, { "id": "edit-tokenizer-test", + "note": "rg tokenizer src tests already searched the test directory for the module the test is named after.", "transcript": [ { "type": "user", @@ -331,9 +332,9 @@ }, "tool_use_id": "toolu_bugfixafterscopedsearch_006" }, - "expect": "deny", + "expect": "allow", "mustDeny": false, - "redundant": false + "redundant": true }, { "id": "npm-test", diff --git a/tests/lib/gateguard-search-evidence.test.js b/tests/lib/gateguard-search-evidence.test.js index 37738253f..bf20ed82c 100644 --- a/tests/lib/gateguard-search-evidence.test.js +++ b/tests/lib/gateguard-search-evidence.test.js @@ -120,6 +120,43 @@ test('findClosestMiss reports a reason code for the nearest non-qualifying searc assert.strictEqual(findClosestMiss(scanOf([s('Grep', { pattern: 'widget_factory', path: '/elsewhere' })]), '', false, data), null); }); +test('a test target is credited by its module stem once the test affix is stripped', () => { + const { findCreditingSearch } = loadHook(); + const data = { cwd: root }; + const scanOf = searches => ({ turnId: null, searches, batchIds: new Map(), newestMessageId: null, shellCommands: [], reads: [] }); + const s = (name, input) => ({ name, input, callsAgo: 1, messageId: 'msg_1' }); + const credits = (search, file) => Boolean(findCreditingSearch(scanOf([search]), `${root}/${file}`, false, data)); + assert.ok(credits(s('Bash', { command: 'rg -n "tokenizer" src tests' }), 'tests/parser/tokenizer.test.js'), '.test'); + assert.ok(credits(s('Grep', { pattern: 'tokenizer', path: `${root}/tests` }), 'tests/parser/tokenizer.spec.ts'), '.spec'); + assert.ok(credits(s('Grep', { pattern: 'widget_factory' }), 'tests/test_widget_factory.py'), 'test_ prefix'); + assert.ok(credits(s('Grep', { pattern: 'widget_factory' }), 'pkg/widget_factory_test.py'), '_test.py suffix'); + assert.ok(credits(s('Grep', { pattern: 'codec' }), 'internal/codec/codec_test.go'), '_test.go suffix'); + assert.ok(credits(s('Grep', { pattern: 'tokenizer.test' }), 'tests/parser/tokenizer.test.js'), 'the full stem still names it'); +}); + +test('stripped test stems keep the generic, length, word-boundary and scope rules', () => { + const { findCreditingSearch, findClosestMiss } = loadHook(); + const data = { cwd: root }; + const scanOf = searches => ({ turnId: null, searches, batchIds: new Map(), newestMessageId: null, shellCommands: [], reads: [] }); + const s = (name, input) => ({ name, input, callsAgo: 1, messageId: 'msg_1' }); + const credits = (search, file) => Boolean(findCreditingSearch(scanOf([search]), `${root}/${file}`, false, data)); + assert.strictEqual(credits(s('Grep', { pattern: 'index' }), 'tests/index.test.js'), false, 'index is generic'); + assert.strictEqual(credits(s('Grep', { pattern: 'app' }), 'src/app.spec.ts'), false, 'app is generic'); + assert.strictEqual(credits(s('Grep', { pattern: 'util' }), 'tests/test_util.py'), false, 'util is generic'); + assert.strictEqual(credits(s('Grep', { pattern: 'main' }), 'cmd/main_test.go'), false, 'main is generic'); + assert.strictEqual(credits(s('Grep', { pattern: 'abc' }), 'tests/abc.test.js'), false, 'too short'); + assert.strictEqual(credits(s('Grep', { pattern: 'tokenizers' }), 'tests/tokenizer.test.js'), false, 'word boundary'); + assert.strictEqual(credits(s('Grep', { pattern: 'tokenizer', path: `${root}/src` }), 'tests/tokenizer.test.js'), false, 'scope'); + assert.strictEqual(credits(s('Grep', { pattern: 'tokenizer', glob: '!*.test.js' }), 'tests/tokenizer.test.js'), false, 'excluded'); + assert.strictEqual(credits(s('Grep', { pattern: 'x', glob: '!tokenizer*' }), 'tests/tokenizer.test.js'), false, 'exclusion names the stem'); + assert.strictEqual(credits(s('Grep', { pattern: 'widget' }), 'tests/widget.test.test.js'), false, 'one affix only'); + assert.strictEqual(credits(s('Grep', { pattern: 'helpers' }), 'src/test_helpers.js'), false, 'test_ prefix only for Python'); + assert.strictEqual(credits(s('Grep', { pattern: 'widget' }), 'src/widget_test.js'), false, '_test suffix only for Python and Go'); + assert.strictEqual(credits(s('Grep', { pattern: 'tokenizer' }), 'skills/tokenizer.test.md'), false, 'instruction targets keep the full stem'); + const miss = findClosestMiss(scanOf([s('Grep', { pattern: 'index' })]), `${root}/tests/index.test.js`, false, data); + assert.strictEqual(miss && miss.reason, 'generic-stem'); +}); + fs.rmSync(transcriptDir, { recursive: true, force: true }); console.log(`\nResults: Passed: ${passed}, Failed: ${failed}`);