From f7f47679b68a593c8c56f6b3354999379b143974 Mon Sep 17 00:00:00 2001 From: montjeffrey Date: Sun, 11 Oct 2026 00:45:01 -0400 Subject: [PATCH] fix(gateguard): make the external-contract question opt-in - questionIdsFor and condensedHintFor take an optional { externalContract: true }; without it a code change that handles data asks data-schema only, and an unknown profile gets the original four questions and condensed hint. - The hook reads GATEGUARD_EXTERNAL_CONTRACT_QUESTION with the same spellings as GATEGUARD_METRICS and passes it to the full denial, the condensed denial and the metrics record, so all three agree. - The effectiveness harness's arm patch forwards the option, so an arm never drops it silently. Docs: change-profile.md gains an External-contract question section and qualifies "byte-identical to the fixed four questions"; design-notes.md, SKILL.md (switch table, data rule, word-for-word set) and CHANGELOG say the fifth question is opt-in; question-effectiveness.md and the harness README note that the runner clears GATEGUARD_* variables, so a gate arm from a current tree measures the gate without the question. skill-triggers@1.json: registryDigest refreshed for the SKILL.md body change; the description and triggers are unchanged. --- CHANGELOG.md | 1 + docker/gateguard-effectiveness/README.md | 7 ++-- docker/gateguard-effectiveness/arm-patch.js | 8 ++--- docs/gateguard/change-profile.md | 18 +++++++++- docs/gateguard/design-notes.md | 4 ++- docs/gateguard/question-effectiveness.md | 8 +++++ manifests/context-packs/skill-triggers@1.json | 2 +- scripts/hooks/gateguard-fact-force.js | 11 +++++-- scripts/lib/gateguard-target-class.js | 33 ++++++++++++++----- skills/gateguard/SKILL.md | 20 +++++++---- 10 files changed, 84 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9b279b0e1..4f0531fa4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ ### Changed - GateGuard first-touch denials ask questions that fit the target: by class (instruction, test, prose, config, code) on its project-relative path, and for code by what the change touches. An edit that changes no public declaration asks for the in-module call sites instead of importers and public API, and the data-schema question is dropped when the change handles no data. Changes the hook cannot read with confidence keep the original four questions word for word. +- GateGuard's external-contract question ("name what outside this repository decides the format, units, timezone or protocol semantics") is opt-in through `GATEGUARD_EXTERNAL_CONTRACT_QUESTION`. Hidden-intent trials did not show it helps, so by default a code change that handles data asks only the data-schema question, and an unreadable change gets the original four questions and condensed hint. - GateGuard credits investigation that already happened: a completed, non-error `Glob`, `Grep`, `LS`, or search-style Bash/PowerShell command in the current human turn that names the target and whose scope contains it passes the first touch with a note. `Read` never counts, nor do searches in the same batch, before a compaction, with error results, or whose filters exclude the target or limit the search to another directory. When a denial follows a search that nearly counted, it names that search and why it did not count. - GateGuard passes comment and whitespace-only edits of code, test and prose files with a note, without marking the file checked, so the next change that alters code is still gated. - GateGuard asks once for a batch of new files: after a new file is denied, later new files of the same class in the same real directory within the same human turn pass with a note. diff --git a/docker/gateguard-effectiveness/README.md b/docker/gateguard-effectiveness/README.md index e4aed748a..646446118 100644 --- a/docker/gateguard-effectiveness/README.md +++ b/docker/gateguard-effectiveness/README.md @@ -21,8 +21,11 @@ where the deciding fact exists only in a simulated user's hidden intent, so an agent that does not ask can only guess. Scenario files omit declared gate question targets, but this does not establish blind or independent authorship: the authors knew the taxonomy, and the external-contract question was added -after inspecting this corpus. Treat those results as exploratory. The report -classifies behavior into descriptive hole categories. +after inspecting this corpus. Treat those results as exploratory. That question +is now opt-in (`GATEGUARD_EXTERNAL_CONTRACT_QUESTION`), and the runner clears +every `GATEGUARD_*` variable from the agent's environment, so a `gate` arm +built from a current candidate tree runs without it. The report classifies +behavior into descriptive hole categories. Each run writes `meta.json`, raw `results.jsonl`, transcripts, `summary.json`, `holes.md`, and `evidence.json`. The evidence manifest fingerprints the frozen diff --git a/docker/gateguard-effectiveness/arm-patch.js b/docker/gateguard-effectiveness/arm-patch.js index 8f2798523..dc59b2848 100644 --- a/docker/gateguard-effectiveness/arm-patch.js +++ b/docker/gateguard-effectiveness/arm-patch.js @@ -43,16 +43,16 @@ function applyArm(arm, targetClass) { const originalText = targetClass.questionText; const originalPhrase = targetClass.condensedQuestionPhrase; const originalHint = targetClass.condensedHintFor; - const questionIdsFor = (cls, isWrite, profile) => armQuestionIds(arm, originalIds(cls, isWrite, profile)); + const questionIdsFor = (cls, isWrite, profile, options) => armQuestionIds(arm, originalIds(cls, isWrite, profile, options)); const questionText = id => (Object.hasOwn(PLACEBO_TEXT, id) ? PLACEBO_TEXT[id] : originalText(id)); const phraseFor = id => { if (Object.hasOwn(PLACEBO_PHRASES, id)) return PLACEBO_PHRASES[id]; if (id === KEPT_ID) return VERBATIM_PHRASE; return originalPhrase(id) || originalText(id).replace(/ \(search the tree.*\)$/, '').replace(/^\w/, c => c.toLowerCase()); }; - const condensedHintFor = (cls, isWrite, profile) => { - const ids = questionIdsFor(cls, isWrite, profile); - if (ids.join('\n') === originalIds(cls, isWrite, profile).join('\n')) return originalHint(cls, isWrite, profile); + const condensedHintFor = (cls, isWrite, profile, options) => { + const ids = questionIdsFor(cls, isWrite, profile, options); + if (ids.join('\n') === originalIds(cls, isWrite, profile, options).join('\n')) return originalHint(cls, isWrite, profile, options); return `briefly state ${joinPhrases(ids.map(phraseFor).filter(Boolean))}, then retry.`; }; Object.assign(targetClass, { questionIdsFor, questionText, condensedHintFor }); diff --git a/docs/gateguard/change-profile.md b/docs/gateguard/change-profile.md index 1bb8d2c1c..653fabfdb 100644 --- a/docs/gateguard/change-profile.md +++ b/docs/gateguard/change-profile.md @@ -116,6 +116,21 @@ or a PowerShell file or web cmdlet (`Out-File`, `Get-Content`, `*-Clixml`; `Import-Csv` matches `csv`). Redirection is found without reading quotes, so a `>` inside a string also counts. +## External-contract question + +`GATEGUARD_EXTERNAL_CONTRACT_QUESTION` (`1`, `true`, `on`, `enabled`, +`enable` or `yes`) adds `external-contract` right after every `data-schema` +in the table above: "Name what outside this repository decides the format, +units, timezone or protocol semantics here (the consumer, the producer, or a +stated convention), or state that the choice is unconstrained". The condensed +hint and the decision metrics name it too. It is off by default because it +has no evidence behind it: it was designed after inspecting the +hidden-intent corpus, and in the rerun that added it the gated arm did not +improve (see +[question-effectiveness.md](question-effectiveness.md#adding-an-external-contract-question-did-not-help)). +`questionIdsFor(class, isWrite, profile, { externalContract: true })` returns +the ids with it. + ## Trivial edits An Edit is trivial when, for every entry, old and new have the same code once @@ -293,7 +308,8 @@ questions were asked without storing text. | config | any | `config-reader`, `config-effect`, `no-plaintext-secrets`, `quote-instruction` | `[data-schema]` is asked only when `touchesData`. With an unknown profile the -code text is byte-identical to the fixed four questions. Sensitive and +code text is byte-identical to the fixed four questions, unless the opt-in +[external-contract question](#external-contract-question) is on. Sensitive and hard-linked targets and NotebookEdit calls are never profiled, so they always get the full code questions. Instruction and config questions carry no change-dependent item, and the test and prose items diff --git a/docs/gateguard/design-notes.md b/docs/gateguard/design-notes.md index 5809dc9aa..c16e036f9 100644 --- a/docs/gateguard/design-notes.md +++ b/docs/gateguard/design-notes.md @@ -207,7 +207,9 @@ their absolute path. A Windows-style path has no real location on a POSIX host The first matching class wins: instruction, test, prose, config, code (the fallback). Cursor `.mdc` rule files are instructions wherever they live. A basename starting with `.env` is config unless it has a code extension -(`.env.example.ts` is source). Code targets ask the original four questions, +(`.env.example.ts` is source). Code targets ask the original four questions +(five with the opt-in +[external-contract question](change-profile.md#external-contract-question)), narrowed by the change profile (see [Questions from the change profile](change-profile.md#questions-from-the-change-profile)); other classes get class-specific questions and a class-specific condensed hint. diff --git a/docs/gateguard/question-effectiveness.md b/docs/gateguard/question-effectiveness.md index f7ab9259a..3135ebca5 100644 --- a/docs/gateguard/question-effectiveness.md +++ b/docs/gateguard/question-effectiveness.md @@ -529,6 +529,14 @@ and the corpus suggests that kind of answer matters. What is not claimed is that adding it improved anything measurable here. A fair test needs scenarios written by independent authors who have not seen that question or these results. +Because nothing here shows it helps, the question is now opt-in: it is asked +only with `GATEGUARD_EXTERNAL_CONTRACT_QUESTION` on (see +[change-profile.md](change-profile.md#external-contract-question)). The +harness clears every `GATEGUARD_*` variable from the agent's environment, so a +`gate` arm built from a candidate tree that includes this change measures the +gate without it, the "before the question" condition above. The runs reported +here used candidate trees from before the change, where it was always asked. + **Standing conclusion.** Across 336 billed trials the gate's benefit is not established, its direction has been positive in the unflagged runs, and its cost is a consistent 35 to 45% in turns and spend. "Not confirmed" is not "no diff --git a/manifests/context-packs/skill-triggers@1.json b/manifests/context-packs/skill-triggers@1.json index ed3be0d40..928c94131 100644 --- a/manifests/context-packs/skill-triggers@1.json +++ b/manifests/context-packs/skill-triggers@1.json @@ -10,7 +10,7 @@ "id": "hand-seeded", "source": "manual-curation-pending-regeneration" }, - "registryDigest": "16549b53055bf5c106e51ca7fb2fe68f8a705b5f6bad40d388349320caedfa63", + "registryDigest": "0136fd919b9b7ffe92e70b3ce28c6d21bd78b658387c0030d9a40eb63e3eddb9", "schemaVersion": 1, "triggers": { "skill:agent-runtime-gateway-smoke-test": [ diff --git a/scripts/hooks/gateguard-fact-force.js b/scripts/hooks/gateguard-fact-force.js index 977ec90d1..ad1fe74de 100644 --- a/scripts/hooks/gateguard-fact-force.js +++ b/scripts/hooks/gateguard-fact-force.js @@ -1945,9 +1945,14 @@ function batchSiblingWarning(safePath) { ); } +// see docs/gateguard/change-profile.md#external-contract-question +function questionOptions() { + return { externalContract: ECC_ENABLE_VALUES.has(normalizeEnvValue(process.env.GATEGUARD_EXTERNAL_CONTRACT_QUESTION)) }; +} + function firstTouchGateMsg(filePath, isWrite, cls, profile) { const safe = sanitizePath(filePath); - const questions = questionIdsFor(cls, isWrite, profile).map(questionText); + const questions = questionIdsFor(cls, isWrite, profile, questionOptions()).map(questionText); return [ '[Fact-Forcing Gate]', '', @@ -1968,7 +1973,7 @@ function firstTouchGateMsg(filePath, isWrite, cls, profile) { */ function condensedGateMsg(action, filePath, ordinal, cls = 'code', targetNote = '', profile = null, missNote = '') { const safe = sanitizePath(filePath); - const hint = condensedHintFor(cls, action === 'creation', profile); + const hint = condensedHintFor(cls, action === 'creation', profile, questionOptions()); return ( `[Fact-Forcing Gate] (denial #${ordinal} this session) First ${action} of ${safe}: ` + `${hint} ` + @@ -2009,7 +2014,7 @@ function firstTouchDenial(filePath, { isWrite, denials, cls, sensitive, linked = cls, sensitive, profile, - questions: questionIdsFor(cls, isWrite, profile) + questions: questionIdsFor(cls, isWrite, profile, questionOptions()) }); if (denials > getFullDenialBudget()) { const action = isWrite ? 'creation' : 'edit'; diff --git a/scripts/lib/gateguard-target-class.js b/scripts/lib/gateguard-target-class.js index 0f2e3a4db..d140bed1c 100644 --- a/scripts/lib/gateguard-target-class.js +++ b/scripts/lib/gateguard-target-class.js @@ -186,7 +186,7 @@ const CLASS_QUESTIONS = Object.fromEntries( ); // see docs/gateguard/change-profile.md#questions-from-the-change-profile -function codeQuestionIds(isWrite, profile) { +function codeQuestionIds(isWrite, profile, externalContract) { const known = Boolean(profile) && profile.known === true; const surface = !known || profile.touchesPublicSurface !== false; const data = !known || profile.touchesData !== false; @@ -194,8 +194,14 @@ function codeQuestionIds(isWrite, profile) { // A change that handles data is usually constrained by something outside the // tree - a consumer's dialect, a reporting timezone, a protocol's semantics - // and the repository cannot settle it. data-schema asks what the change - // touches; external-contract asks who decides what it has to be. - return data ? [...opening, 'data-schema', 'external-contract'] : opening; + // touches; external-contract asks who decides what it has to be. It is + // opt-in: see docs/gateguard/change-profile.md#external-contract-question. + if (!data) return opening; + return externalContract ? [...opening, 'data-schema', 'external-contract'] : [...opening, 'data-schema']; +} + +function asksExternalContract(options) { + return Boolean(options) && options.externalContract === true; } /** True when a class's questions depend on the change profile. */ @@ -203,9 +209,14 @@ function questionsUseProfile(cls) { return !Object.hasOwn(CLASS_QUESTION_IDS, cls); } -/** Stable ids of the first-touch questions for a class, action and change profile. */ -function questionIdsFor(cls, isWrite, profile) { - const ids = Object.hasOwn(CLASS_QUESTION_IDS, cls) ? CLASS_QUESTION_IDS[cls](Boolean(isWrite)) : codeQuestionIds(Boolean(isWrite), profile); +/** + * Stable ids of the first-touch questions for a class, action and change profile. + * `options.externalContract === true` adds the opt-in external-contract question. + */ +function questionIdsFor(cls, isWrite, profile, options) { + const ids = Object.hasOwn(CLASS_QUESTION_IDS, cls) + ? CLASS_QUESTION_IDS[cls](Boolean(isWrite)) + : codeQuestionIds(Boolean(isWrite), profile, asksExternalContract(options)); return [...ids, 'quote-instruction']; } @@ -215,6 +226,8 @@ function questionText(id) { } const CODE_CONDENSED_HINT = + "briefly state importers/callers, affected API, data schemas if any, and the user's verbatim instruction, then retry."; +const CODE_CONDENSED_HINT_EXTERNAL_CONTRACT = "briefly state importers/callers, affected API, data schemas if any, what outside the repository fixes the format or semantics, and the user's verbatim instruction, then retry."; const CONDENSED_PHRASES = Object.freeze({ @@ -252,10 +265,12 @@ const CLASS_CONDENSED_HINTS = { }; /** One-line condensed hint matching `questionIdsFor`. */ -function condensedHintFor(cls, isWrite, profile) { +function condensedHintFor(cls, isWrite, profile, options) { if (Object.hasOwn(CLASS_CONDENSED_HINTS, cls)) return CLASS_CONDENSED_HINTS[cls](Boolean(isWrite)); - if (!isWrite && !(profile && profile.known === true)) return CODE_CONDENSED_HINT; - const phrases = questionIdsFor(cls, isWrite, profile).map(condensedQuestionPhrase).filter(Boolean); + if (!isWrite && !(profile && profile.known === true)) { + return asksExternalContract(options) ? CODE_CONDENSED_HINT_EXTERNAL_CONTRACT : CODE_CONDENSED_HINT; + } + const phrases = questionIdsFor(cls, isWrite, profile, options).map(condensedQuestionPhrase).filter(Boolean); return `briefly state ${joinPhrases(phrases)}, then retry.`; } diff --git a/skills/gateguard/SKILL.md b/skills/gateguard/SKILL.md index e6ebcc1f7..7cda6449b 100644 --- a/skills/gateguard/SKILL.md +++ b/skills/gateguard/SKILL.md @@ -117,19 +117,23 @@ For **code** targets the gate reads the change it is about to allow (Edit and clock reads on either side; in shell scripts also redirection to a file, `curl`, `wget`, `jq` and PowerShell file and web cmdlets. Without them the data - question is dropped. + question is dropped. With `GATEGUARD_EXTERNAL_CONTRACT_QUESTION` on, a + change that keeps the data question is also asked what outside the + repository decides its format or semantics. - **Trivial:** see [Comment and whitespace-only edits](#comment-and-whitespace-only-edits). Supported extensions: `.js .mjs .cjs .jsx .ts .tsx .mts .cts .py .pyi .go .rs .java .kt .kts .cs .c .h .cc .cpp .cxx .hpp .hh .hxx .sh .bash .zsh .ps1 .psm1 .bat .cmd`. Any other extension, a missing or non-string field, more than 64 MultiEdit entries, a side over -64 KiB, or 256 KiB in total means the full questions, word for word. Sensitive +64 KiB, or 256 KiB in total means the full questions, word for word: the +original four (importers, public API, data schemas, the user's instruction), +or five with `GATEGUARD_EXTERNAL_CONTRACT_QUESTION` on. Sensitive and hard-linked targets and NotebookEdit calls always get the full questions. Other classes keep their class questions. Each question has a stable id (`importers`, `public-api`, `local-callers`, `callers`, `no-duplicate`, `data-schema`, -`quote-instruction`, and one per class question), listed in -`docs/gateguard/change-profile.md`. +`external-contract` when opted in, `quote-instruction`, and one per class +question), listed in `docs/gateguard/change-profile.md`. ### Questions by target class @@ -583,10 +587,12 @@ load-bearing destructive-Bash checks keep running: | `GATEGUARD_BASH_EXTRA_DESTRUCTIVE` | unset | Extra destructive-command patterns, as regex source, added to the built-in set. A malformed regex is treated as unset (built-ins still apply) and logged once to stderr. | | `GATEGUARD_STATE_DIR` | `~/.gateguard` | Where per-session gate state is kept. If state cannot be persisted the gate allows the operation rather than looping, and names this variable in the warning. | | `GATEGUARD_METRICS` | unset (off) | Records one line per Edit/Write/MultiEdit/NotebookEdit/Bash/PowerShell decision in `/metrics.jsonl` (see [Decision metrics](#decision-metrics)). Never changes a decision. | +| `GATEGUARD_EXTERNAL_CONTRACT_QUESTION` | unset (off) | Adds a code question after the data-schema question: name what outside the repository decides the format, units, timezone or protocol semantics. Experimental: in hidden-intent trials it did not change outcomes (`docs/gateguard/question-effectiveness.md`). Never changes a decision. | -`GATEGUARD_BASH_ROUTINE_DISABLED` and `GATEGUARD_METRICS` accept `1`, `true`, -`on`, `enabled`, `enable`, or `yes` (case- and whitespace-insensitive); any -other value leaves the routine gate on and metrics off. +`GATEGUARD_BASH_ROUTINE_DISABLED`, `GATEGUARD_METRICS` and +`GATEGUARD_EXTERNAL_CONTRACT_QUESTION` accept `1`, `true`, `on`, `enabled`, +`enable`, or `yes` (case- and whitespace-insensitive); any other value leaves +the routine gate on, metrics off, and the question unasked. #### Decision metrics