8a7d6129 added the external-contract question for every code change that
handles data and updated the pinned tests to match, including the legacy
test "code condensed Edit hint for an unknown profile is unchanged". Its own
evaluation then found the gated arm did not improve (25 to 23 of 48), so the
question has no evidence behind it as a default.
- Restore the test expectations 8a7d6129 changed: questionIdsFor, the
byte-identical code text for src/a.py, the condensed hints, and the hook
question lists default to data-schema without external-contract.
- Add questionIdsFor/condensedHintFor cases with { externalContract: true }.
- Add hook cases: GATEGUARD_EXTERNAL_CONTRACT_QUESTION=1/true/ON asks it
beside data-schema and in the unknown-profile set, never without data;
unset, 0, false and off keep the original four.
Against main: target-class 5 failures, fact-force 10 failures.
Across 240 evaluation trials every question the gate raised was about code
structure - importers, callers, public API, data schemas - while every deciding
fact in the hidden-intent corpus was an external contract: a consumer's CSV
dialect, a reporting timezone, retry safety against a non-idempotent endpoint,
collation rules, a quota policy. Obtaining the deciding fact predicted the
outcome almost perfectly in both arms, and the gate did not reliably cause it.
Add an external-contract question for code targets whose change profile touches
data, beside data-schema rather than replacing it: data-schema asks what the
change touches, external-contract asks who decides what it has to be. It reuses
the existing touchesData signal, so it stays as selective as the question it
sits with and adds no new detection, and it therefore does not reach a change
with no data involvement - pure retry semantics, for instance - which is a known
limit to be measured rather than guessed at.
Code edits that touch data now ask four questions instead of three, so the
condensed hint for an unknown profile changes too. That string was pinned by a
test as legacy-stable; the pin is updated deliberately, because a hint that
names fewer questions than the gate asks is worse than a hint that changed.
Suites: target-class 31, fact-force 1019, scenario-gate 25, eval 15, and the two
effectiveness suites 29 and 26, all passing. The 23 relevantQuestions
annotations that listed data-schema now list external-contract as well.
- Ask class-specific first-touch questions (instruction, test, prose,
config, code); code questions are unchanged.
- Credit a completed Glob/Grep/LS or shell search from the current human
turn that covered the target; Read and exclusion globs never count.
- Collapse same-class new-file creations in one directory within a turn.
- Add opt-in GATEGUARD_FACT_FORCE_MAX_DENIALS session cap (from #2755).
- Always gate sensitive targets, judged on lexical and real paths.
- Canonicalize checked-path keys.
Closes#2608
Co-authored-by: SulimanAbdulrazzaq <suliman9000a@gmail.com>
montjeffrey