A non-numeric, negative, or exponent Points value was persisted and
silently skewed the derived sprint total (bogus -> 0, -3, 1e3 -> 1000)
with no drift reported. parseStory now enforces the documented values,
so apply rejects the batch and status reports the invalid story.
Resolve the Greptile findings on 74262955:
- Refuse symlinked .stories/, epics/, sprints/, .txn/, .ids/ and
symlinked ledger files, so no read or write leaves .stories/.
- Recovery accepts only regular backup-<n> files in .txn/, validates the
whole journal before restoring, and never writes through a planted
restore.tmp.
- Remove a stale lock only under .lock.break after a re-read, so a live
lock is never deleted; a leftover .lock.break times out (exit 3)
instead of spinning.
- apply enforces one-step-forward status transitions, new stories start
as todo, and Sprint must be unassigned or an existing sprint file.
- Refuse a batch that names the same file twice (e.g. a\b vs a/b).
Symlink tests skip on Windows when symlink creation returns EPERM.
The catalog gate validated only the first README project-tree block and
had no skills/commands counterpart to the agents pattern, so:
- README.md project tree (skills) and (commands) were never checked.
- A stale count in a second tree block was invisible because the agents
pattern used String.match with a non-global regex, which reads only the
first occurrence.
Both are now fail-closed. parseReadmeExpectations walks every tree block
for all three categories via matchAll, reports the offending line number
in the source label, and throws when an entry is missing entirely rather
than skipping it. syncEnglishReadme uses replaceAllOrThrow so --write
repairs every block instead of only the first; replaceOrThrow could not
be reused because it replaces only the first match and its regex.test()
probe is unsafe once the pattern carries the g flag.
README.md:847 documented 67 agents against an actual 68 and was never
checked, so it was corrected.
Fixtures in tests/ci/catalog.test.js and tests/ci/validators.test.js now
emit the skills and commands tree lines the gate requires, and both gain
regression coverage that fails against the previous parser (5 new tests).
main resolves SQL clients behind env -S and its launcher grammars, but a
destructive statement still passed under four forms:
- a shell -c inside a short-option cluster (bash -lc, sh -ec), since only
an exact -c was looked up;
- su, whose -c / --command / -lc value runs through the target user's
shell, before or after --;
- taskset, whose mask or CPU list precedes the command, like timeout's
duration;
- a command nested past the shell -c recursion limit, which was allowed
instead of denied.
The first three are checked like sh -c; past the limit the check now
fails closed.
main now inspects SQL clients behind env -S, time, nice, timeout and the
other supported launchers (2b9164c0, from #2829), so this merge takes
main's tree and the cases main still misses are re-applied on its
launcher grammar in the next commit.
96 trials, none invalid. The headline reads gate 23/48 against off 13/48, a
21-point paired difference at exact McNemar p = 0.021, and it is an artifact.
The scenarios, prompts, graders and model alias were identical to the previous
run and the off arm has no gate, yet off fell from 18/48 to 13/48 while the
gated arm went down, 25/48 to 23/48. An intervention meant to raise the gated
arm did not raise it, so the p-value reflects a baseline that fell rather than a
gate that improved.
The two scenarios the question cannot reach, having no data for the profile to
detect, serve as a control: they moved by -3 and +3, more than any of the four
it did reach (-1, -1, +1, -1). Noise on eight trials is about plus or minus
three, which is larger than any effect available to the change.
The question is kept, since it asks something the gate never asked and the
corpus says that kind of answer decides the outcome, but no improvement is
claimed. Testing it fairly needs scenarios authored blind to it, which this
corpus no longer is.
96 trials, none invalid, analysed exactly as pre-registered. passed: gate 25/48
against off 18/48, 48 pairs, discordant 15 to 8, exact McNemar p = 0.210. At the
registered alpha of 0.05 the hypothesis is not confirmed, and the bootstrap
interval agrees this time at -15% with a range of -33% to +4%, so there is no
disagreement between the tests to adjudicate. Secondary measures point the same
way and reach nothing: asked 29/48 against 22/48 at p = 0.230, deciding fact
27/48 against 21/48 at p = 0.327.
The effect halved on replication, from +29 points at p = 0.065 to +15 at
p = 0.210, which is the ordinary fate of a near-threshold finding. The runs are
not pooled: the quota leak was closed between them, and combining them after
seeing both would be the post-hoc choice the pre-registration exists to stop.
Two things survive every run. Obtaining the deciding fact is necessary and
nearly sufficient, with no trial in either arm passing without it across 96
trials. And the gate costs 37 to 45% more turns and spend, which is the most
stable number the evaluation has produced.
soft-limit-overage asked for { allowed, overage }. Naming an overage figure
hints that passing a limit is permitted, which is precisely the hidden policy,
and it showed: that scenario was one of the two with the highest ungated pass
rate in the signposted run. The grader now inspects only allowed, so the prompt
asks for { allowed } and mentions neither overage nor billing nor soft limits. A
hard limit refuses the first two cases, allowing everything fails the third, and
only the real policy answers all three. --check-graders still separates start,
trap and reference on all six.
The analysis is registered in the doc before the run rather than chosen after
it: passed as the single primary outcome, paired exact McNemar at alpha 0.05, a
sample of 48 pairs fixed in advance, no extension on the strength of the result,
frozen inputs, secondary measures explicitly not claim-bearing, a validity rule
for provider errors, and a commitment to report a null.
With the ambiguity signposted, the arms were indistinguishable on every measure.
With the prompts rewritten as ordinary tickets and nothing else changed, they
separate: asked the user 16/24 against 10/24, obtained the deciding fact the
same, passed 15/24 against 8/24, exact McNemar p = 0.065 to 0.070, and a paired
pass-rate difference of 29 points in the gate's favour whose bootstrap interval
excludes zero.
The regime contrast carries it. The gated arm passed 15/24 in both runs while
the ungated arm fell from 16/24 to 8/24, which is what removing the only cue
that a convention was missing would do if the gate supplies that cue itself.
Asking stayed necessary throughout: no trial in either arm passed without the
deciding fact.
Reported as suggestive rather than established. All three p-values sit just
above 5%, one more gate-favouring discordant pair would cross it, and extending
this run after seeing that would be optional stopping. A confirmatory run needs
its sample fixed first: about 43 pairs, or 8 repetitions over these six
scenarios.
The previous prompts signposted what they withheld - "the dialect our consumer
expects", "the calendar our reports are read against", "retrying only the
failures that are safe to repeat" - which invited a question in every arm and
pushed ungated pass rates to 3/4 and 4/4. That bounded the result to a regime
where the agent was already told something was missing.
Each prompt now reads as an ordinary ticket. Shape a grader checks stays, since
removing it would score field-name guessing instead of the hidden decision, but
it is stated mechanically: checkQuota still documents { allowed, overage } and
defines overage as how far usage exceeds the limit, which does not say whether
exceeding it is allowed, so the soft-limit policy stays hidden.
Graders, workspaces and intents are unchanged, so this isolates one variable:
whether the gate changes behaviour when nothing cues the agent that a convention
it cannot see decides the task.
With one decisive fact per scenario the corpus isolates the mechanism:
obtaining the deciding fact predicts the outcome 14/14 in both arms, and not
obtaining it predicts failure in 9 of 10 and 8 of 10. Against that, the gate
changes nothing measurable. Asked the user 14/24 against 15/24, obtained the
deciding fact 14/24 against 14/24, passed 15/24 against 16/24, every exact
McNemar p = 1.000, and the paired pass-rate difference 5 points in off's favour
with a 95% interval of -25% to +35%. At 20 pairs a 40-point effect is ruled out;
a 15-to-20-point one is not.
The gate does fire - denials in 22 of 24 gated trials - so the failure is in the
conversion. timezone-daily-rollup fired four times out of four and produced no
question at all. Every question the gate raised is about code structure, while
every deciding fact in this corpus is an external contract.
Record the two scenario-design faults the earlier runs exposed, and the bound
the correction imposes: moving shape into the prompt also signposts the
ambiguity, which raises asking in both arms, so the claim is limited to the
signposted regime.
Three of six scenarios sat at floor in the 48-trial run, and the run data says
the cause was the scenarios, not the agents.
csv-export-delimiter was asked about in 5 of 8 trials yet disclosedDecisive
never fired: the simulated user disclosed delimiter five times and lineending
twice, and quoting never. disclosedDecisive requires every decisive fact, and a
user who answers only what is asked and never volunteers cannot deliver three of
them to one question. username-collation failed the same way, with only case
ever disclosed.
soft-limit-overage failed differently and worse: the facts reached the agent in
4 of 8 trials and every trial still scored 0, because the grader demanded an
{ allowed, overage } shape that nothing disclosed. It was scoring whether the
agent guessed field names, not whether it learned the policy.
So each scenario now carries exactly one decisive fact, which is one hidden
decision a single well-aimed question can obtain, and anything a grader checks
about shape is stated in the prompt. The three working scenarios are changed the
same way, so disclosedDecisive means the same thing across the corpus: their
second and third decisive facts move into the prompt or become non-decisive
colour.
silence-hole and coverage-hole are claims about the gate, and an ungated arm
never fires it, so classify labelled every ungated miss a coverage-hole by
construction. In the 48-trial run that padded the hole counts with five rows
that say nothing about the gate: coverage-hole read 7 when only 2 were the
gate's. Label an ungated miss unaided-miss, which is the baseline rate and is
excluded from the holes list.
One scenario was the whole blind corpus, which cannot show whether the gate
generalises. These five were written from ambiguities a developer meets, not
from the gate's question list, and each keeps its deciding fact out of the
repository while leaving a plausible wrong answer in plain sight:
- csv-export-delimiter: a comma-parsing importer sits next to an export whose
consumer needs semicolons, CRLF and every field quoted.
- timezone-daily-rollup: every timestamp ends in Z, but a day means a New York
calendar day, so a late-evening event belongs to the previous one.
- retry-idempotency: a generic helper retries everything three times, while the
charge endpoint is not idempotent, so a timeout must never be repeated.
- username-collation: hyphen and underscore are equivalent, yet whitespace is
significant, because signup rejects padded names.
- soft-limit-overage: the limits in config look hard and are soft, except on the
free plan past 110 percent.
Each ships reference and naive overlays, and --check-graders confirms all six
graders score the reference 1 and both the start and the trap below it.
The hidden-intent report printed bare counts. run.js reported Wilson intervals,
a bootstrap paired difference and an exact McNemar test, and that machinery was
lost in the new runner, so four trials rendered as a table that reads like a
result. On this run it is not one: gate 1/2 against off 0/2 gives intervals of
9%-91% and 0%-66%, a paired difference of -50% with a 95% interval spanning
-100% to 0%, and McNemar p = 1.000.
Reuse lib.js's pairedComparison and ai-eval-lib's wilson rather than adding a
second statistics path, and state the shortfall in words whenever a comparison
has fewer than twenty pairs: about 20 trials per arm are needed for 80% power on
a 40-point pass-rate difference at a two-sided 5% level, about 36 for 30 points,
and about 160 for the 15-point threshold the pilot pre-registered.
Mean turns and cost sit in the same table, because the only efficiency figure
this harness has produced points the other way: the gate roughly doubles turns
and cost per trial.
materializeTree passed the archive as an absolute path. GNU tar, which Git for
Windows bundles and puts first on PATH in Git Bash, reads an -f argument
containing a colon as a host:path remote spec and fails with "Cannot connect to
C: resolve failed". bsdtar, the tar in system32 that PowerShell resolves, has no
remote-spec parsing, so the fault only appeared under a bash-based shell - and
there it failed a committed test, not just a dry run.
Name the archive relatively and run tar with dest as its cwd, which extracts
identically under both tars and makes -C unnecessary. git archive keeps the
absolute -o path, which git handles on Windows.
Two tests, because the environment-dependent kind of fault needs both: one
injects a spawn and asserts no tar argument is absolute or drive-lettered, which
holds the invariant on any platform; the other prepends Git's bundled GNU tar to
PATH and extracts into a drive-letter destination, so the original failure is
reproducible from PowerShell too, where it was previously invisible.
The suite was the only file in tests/docker using node:test. The repository's
dominant convention, in 229 of 327 test files including the sibling
gateguard-effectiveness suite, is a local test() helper that counts passes and
sets process.exitCode, which is what tests/run-all.js reads.
Adopt that helper verbatim, along with the sibling's tempDir wrapper so the
scenario fixtures resolve symlinked temp paths. Assertions are unchanged.
Four defects found by running the repository's own checks and one more trial
against it.
ESLint rejected an unused `armSettings` import in intent-eval.js; the caller
supplies `armSettingsByName` already.
A timeout was lost whenever a later turn came back clean: `a || b ? c : false`
evaluates the condition first, so `timedOut` was recomputed from the new turn
instead of being kept.
Isolating each trial's config directory, to stop persisted sessions piling up
under the projects folder, puts the credentials out of reach. Every turn then
returns "Not logged in", and the harness graded that as a coverage-hole - an
auth failure recorded as a finding. Keep the real config directory and delete
only the session folders this run's work root created, and adopt lib.js's own
guards: a trial that is unauthenticated, or whose gated arm edited files with no
metrics line and no denial, stops the run instead of being recorded.
Record denials per turn as well as in total. The hook marks a file checked once
per session and --resume keeps one session id for the conversation, so a zero
after a non-zero is the latch rather than the gate staying quiet, and
silence-hole has to be read with that column in view.
Verified: a real trial authenticates, produces a graded silence-hole, and leaves
the projects directory at the same count it started with.
The pre-registration said results would be reported whichever way they came
out. They came out empty: 72 sessions in which all four arms passed 18 of 18
with mean score 1.000, every arm-against-gate difference 0% (0% to 0%) and
McNemar p = 1. With the outcome fixed for every trial there is no variance to
explain, so this is the absence of a measurement rather than weak evidence that
the gate does nothing. The run does establish the gate's cost in that regime:
about 45% more turns and 28% more tokens for no outcome change.
Record the three reasons it measured nothing - deciding facts hidden in the
repository, the asking channel switched off, and a pass criterion written in
the gate's own vocabulary - and the hidden-intent design that addresses them,
with its hole classes, its first four-trial run and its own limits.
montjeffrey