The existing harness cannot measure what it set out to measure. Every trap
hides its deciding fact in the repository, which agents reliably discover, and
trials run one-shot under "do not ask clarifying questions", so a fact-forcing
gate is evaluated with no user to answer it. Its scenarios also declare the
question the gate ought to raise, chosen by reading the gate's own taxonomy, so
the suite passes by construction and can never find a missing question.
Add a second runner that measures effectiveness instead of coverage:
- scenarios-intent/ keeps the deciding fact in intent.json, as a choice between
plausible conventions the repository cannot settle, so an agent that does not
ask can only guess and the ungated arm cannot sit at the ceiling.
- user-sim.js plays that user with a separate cheap model: it answers the
question in front of it, never volunteers, and reports which fact ids it
disclosed, so asking something and asking the deciding thing are told apart.
An unparseable judge is a recorded failure, not a silent stonewall.
- session.js runs multi-turn sessions joined by --session-id and --resume, and
tells every arm the user is reachable, so the comparison is symmetric and the
gate's contribution is whether it makes the agent ask the deciding question.
- coverage.js classifies each trial from behaviour alone into working, lucky, or
a follow-through, targeting, silence or coverage hole, and reports the
questions the gate actually raised rather than asserting declared ones.
- run-intent.js refuses minus-target, which needs targets these scenarios
deliberately lack, and checks every grader separates the start, the trap and
the reference before a session is billed.
A scenario counts as carrying no information only when nothing varies across
arms. Trap strength is reported separately: the ungated arm failing is what a
working trap looks like, so it must not make a scenario unusable.
The previous fix stopped a throwing hook from aborting the cold replay by
recording it as an allow, matching the warm path. But coldSummary only counted
disagreements, so a failure on a step the worker also allowed was no
disagreement and left no trace: the report showed no failure, a run with no
measured step reported a cold latency of 0 ms because percentile of an empty
list is 0, and the exit status passed. Trading a crash for a clean-looking
success hid exactly the failure the crash had exposed.
coldSummary now counts errors and samples, and reports no latency when nothing
was measured rather than a 0 that reads as a fast hook. The report prints "no
valid sample" in place of a latency and adds a line for steps that failed in a
fresh process, beside the existing disagreement line. The exit check moves into
gateFails, which fails on cold errors as it already did on warm errors and cold
disagreements, and is exported with coldSummary so both can be tested.
The render test builds its report from summarize([]), the real summary shape,
so it keeps working as metrics are added.
measureColdLatency ran each step in a fresh process through execFileSync,
inside try/finally with no catch. A hook that threw made the child exit
non-zero, execFileSync threw, and the error left measureColdLatency and then
evaluate, so neither the report nor the requested SARIF was written for a run
that had hit a single bad step.
The probe now catches a throwing require or run itself and records it the way
workerMain does: the raw payload stands in for the result, so the step reads as
an allow, and the error is kept. Matching the warm path matters because
coldSummary counts every cold decision that differs from the warm one as a
disagreement; any other choice would have reported disagreements that do not
exist. A child that dies without writing parseable output is caught outside the
probe and recorded the same way with no latency, and coldSummary leaves
unmeasured steps out of the percentiles while still counting them.
The new tests run a hook that always throws through both passes. They fail
against the previous code, 2 of 17, and pass against this one.
Brings in the three commits main gained since the branch's merge base: the 2.2.3
release, retiring the Everything Claude Code name for ECC, and the ja-JP sponsor
update. 227 files arrive from main; two conflicted.
CHANGELOG.md: the branch had added an Unreleased section and main a released
2.2.3 section, both at the top. Both are kept verbatim with Unreleased above
2.2.3, since none of the branch's entries shipped in that release.
manifests/context-packs/skill-triggers@1.json: both sides had recomputed
registryDigest from the same earlier value to different results, and neither is
the digest of the merged tree. It is recomputed with loadContextRegistry over the
merged tree and only that field changes; triggers and triggersDigest are
untouched on both sides. validate-context-profiles passes.
All twelve CI validators pass. The full suite reports 44 failures in seven files;
43 of them fail by name and count identically on an unmodified upstream/main, and
the remaining one, plugin-hook-bootstrap, fails only inside run-all and passes in
isolation on both trees. The merge introduces no failing test that main does not
already have.
Across 240 evaluation trials every question the gate raised was about code
structure - importers, callers, public API, data schemas - while every deciding
fact in the hidden-intent corpus was an external contract: a consumer's CSV
dialect, a reporting timezone, retry safety against a non-idempotent endpoint,
collation rules, a quota policy. Obtaining the deciding fact predicted the
outcome almost perfectly in both arms, and the gate did not reliably cause it.
Add an external-contract question for code targets whose change profile touches
data, beside data-schema rather than replacing it: data-schema asks what the
change touches, external-contract asks who decides what it has to be. It reuses
the existing touchesData signal, so it stays as selective as the question it
sits with and adds no new detection, and it therefore does not reach a change
with no data involvement - pure retry semantics, for instance - which is a known
limit to be measured rather than guessed at.
Code edits that touch data now ask four questions instead of three, so the
condensed hint for an unknown profile changes too. That string was pinned by a
test as legacy-stable; the pin is updated deliberately, because a hint that
names fewer questions than the gate asks is worse than a hint that changed.
Suites: target-class 31, fact-force 1019, scenario-gate 25, eval 15, and the two
effectiveness suites 29 and 26, all passing. The 23 relevantQuestions
annotations that listed data-schema now list external-contract as well.
json.loads raises RecursionError on deeply nested input, which escaped the
complete-JSON check and failed the request. Treat it like any other input
that is not a complete JSON document.
Signed-off-by: FenjuFu <fufenjupku@gmail.com>
A signature read that failed after one byte looked like a one-byte text
file and was skipped. The pipeline now runs under pipefail, so the file is
listed as unverified instead.
When cut, sort or uniq is missing, Step 7 says the tally is unavailable.
The count does not depend on them.
Step 1 skipped binaries while the role promised to verify the whole project,
so a credential in a spreadsheet, a PDF or an embedded database got a clean
PASS and never showed up in the report.
Step 7 now lists every file that doesn't read as text, plus any directory it
can't enter, and the pipeline stops on that list until a person confirms. On
this repo that's a few dozen files, all images and video, which is why they're
warnings and not failures. It doesn't try to extract or scan them: that needs
nested-archive recursion and size bounds, and I'd rather list a file than
half-scan it.
Detection is a first-byte signature check plus a whole-file NUL/control-byte
test, with file(1) only widening the list. If a detector fails, the file is
listed as unverified.
Filenames are untrusted, so Step 7 writes the paths to a manifest beside the
project directory, for a person to open, and keeps them out of its stdout
and its report section. Beside and not inside, so a re-run's text scans
can't read it back. The script prints the count itself as its last line. A
scan problem, a failed write, a walk that dies early or a run that never
reaches that line gives an unknown count, and the gate treats unknown as a
stop, not as zero.
Step 1 no longer skips *.min.js.
Tests in tests/skills/sanitizer-unreadable.test.js. Editing SKILL.md changes
the skill registry digest, so skill-triggers@1.json carries the new value.
The merge took main's 68/293 in docs/tr/AGENTS.md, which catalog:sync
does not cover; tests/docs/configure-ecc-install-paths.test.js expects
69 agents and 294 skills.
Co-Authored-By: Kishor Morol <kishormorol@gmail.com>
Keep both the new Polski locale and Bengali in every language switcher,
locale list and alias map; cross-link docs/pl and docs/bn. Resync catalog
counts (69 agents, 294 skills) and the skill-trigger registry digest.
Co-Authored-By: Kishor Morol <kishormorol@gmail.com>
The Plan Canvas library wrote ~/.claude/plan-canvas while the SessionStart
and Stop hooks read CLAUDE_CONFIG_DIR/plan-canvas. Under a second profile
SessionStart missed open reviews and Stop let a turn end with feedback
still queued.
Expand a leading ~ in CLAUDE_CONFIG_DIR the way scripts/lib/agent-data-home.js
does, so a profile written as ~/.claude-work resolves to one directory rather
than two.
Add hook-level coverage that spawns each hook with a distinct
CLAUDE_CONFIG_DIR and no ECC_* override, plus a producer-to-consumer round
trip. Eight of the nine cases fail against the previous revision.
The gateguard SKILL.md note changes the skill registry digest, so
ci/context-profiles.test.js reported the skill triggers manifest as
stale. Update the digest the way the earlier registry binding
refreshes did; the triggers themselves are unchanged.
getDefaultClaudeAgentDataHome() is one of the two roots
resolveAllowedProjectConfigHome() accepts, so following CLAUDE_CONFIG_DIR
moves a security boundary and not just a default. The case asserts the
containment holds at the new location and nowhere else: a path inside the
relocated root is accepted, while the old ~/.claude, a traversal out of the
new root, and an arbitrary home path are all still refused.
It bites. Re-adding ~/.claude as a third allowed root, which is the obvious
way to get this wrong, turns the case red.
Claude Code points every profile-local path at CLAUDE_CONFIG_DIR when it is
set, but ECC hooks kept reading and writing ~/.claude. A session launched
under a second profile left its bash log, cost tracker, metrics, session data
and skill runs in the default profile, where the other one reads them back.
Two groups, as reported:
- getDefaultClaudeAgentDataHome() in scripts/lib/agent-data-home.js now
follows the variable, which covers every hook reaching ~/.claude through
resolveAgentDataHome(). ECC_AGENT_DATA_HOME still wins, so the documented
workaround keeps working.
- plan-canvas-pending, plan-canvas-sessions, mcp-health-check and
post-bash-command-log hard-coded os.homedir()/.claude and now resolve the
same way, matching scripts/hooks/ecc-statusline.js:68.
A blank value is ignored rather than resolved, since path.resolve(' ')
returns the process cwd and would scatter profile data through whatever
directory the session started in.
Fixes#3266
With prefilled reasoning enabled, a structured answer such as
{"closing":"</think>"} was cut at the tag inside its string value.
Reasoning followed by a closing tag is never a valid JSON document, so a
reply that parses as one complete object or array is now returned
unchanged. Also annotate the tests and helpers this PR adds.
Signed-off-by: FenjuFu <fufenjupku@gmail.com>
Only an answered /api/show lookup is cached now, and the cache is replaced rather than mutated. A failed lookup sends no tools for that request and is retried on the next one, so a transient outage no longer disables tools for the provider's lifetime. Also annotate the test helpers and tests this PR adds.
Signed-off-by: FenjuFu <fufenjupku@gmail.com>
The merged registry now has 293 skills, so the hand-seeded manifest's
registryDigest no longer matched and validate-context-profiles failed
npm test. Update registryDigest and coverage.skills; the curated
triggers and triggersDigest are unchanged.
Co-Authored-By: Kishor Morol <kishormorol@gmail.com>
The OpenCode install target resolves its root from OPENCODE_CONFIG_DIR,
then XDG_CONFIG_HOME/opencode, before falling back to HOME. Inheriting
either variable still let a developer's real OpenCode install-state reach
the spawned status CLI and fail the same four tests. Point both into the
isolated home.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The status CLI reconciles install-state discovered under $HOME. The
state-store tests spawned it with the developer's real environment, so on
any machine with ECC installed the real ~/.claude install was counted as an
extra warning and four status/work-items assertions failed (3 attention
items instead of 2). CI stays green only because runners have no install.
Point HOME and USERPROFILE at an empty temp dir for spawned CLIs, cleaned up
on exit. Callers can still override via options.env.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Route every .stories/ write through skills/story-lifecycle/scripts/ledger.js:
- exclusive per-ledger lock; a second writer waits, then exits 3 without writing
- story IDs reserved with exclusive-create markers in the git common dir, so
concurrent agents and worktrees of one clone never share an ID
- journaled batches: stage, journal, rename; derived epic/sprint tables are
regenerated inside the batch; an interrupted batch is rolled back by the
next write command or `recover`; `status` reports it only when no live
writer holds the lock
Bound the ledger to an opt-in planning projection that is never canonical
ECC Task/Evidence/Completion/Integration state, and state that .stories/
follows the checkout.
tests/docs/story-lifecycle-ledger.test.js exercises create, concurrent
create, concurrent cross-worktree allocation, lock contention, overwrite
refusal, interrupted-batch recovery, sprint planning, status drift and
reconcile, and path confinement. Also refresh the skill-triggers manifest
digest for 294 skills.
- Problem: CI's yarn legs fail on ea0a6ff6 with YN0028 (lockfile would be
modified) - the sync merge commit carries a locally regenerated
yarn.lock (2a02200e) even though both merge parents ship the identical
lockfile (7b566ee0), so the merged tree's package.json no longer
matches the committed lockfile.
- Fix: restore yarn.lock to the parents'/main's blob (7b566ee0), which
is the exact combination CI validates green on main and validated
green on ddef0301 (46/46).
- Verification: yarn install --mode=skip-build (CI's exact step,
COREPACK_ENABLE_STRICT=0, skip-build) exits 0 and leaves yarn.lock
unmodified; tree content otherwise identical to ea0a6ff6, whose full
suite passed 704/0 locally.
Tools are now forwarded only when the selected model declares tool support:
catalogued models answer from ModelInfo.supports_tools (llama3.2 and mistral
are tool models on ollama.com, codellama is not), and any other model is
looked up once through /api/show, whose capabilities list is what the
installed model declares. A failed lookup sends no tools.
Native Ollama tool calls often have no id, so each one gets a unique id, and
tool results are sent with tool_name resolved from the call they answer, so
several results stay attributable.
Signed-off-by: FenjuFu <92919259+FenjuFu@users.noreply.github.com>
Strip a balanced <think>...</think> block that opens the reply by default and
leave every other reply untouched, so answers that mention the tags (a lone
or reversed closing tag, JSON string values) are never truncated.
The close-only form produced by chat templates that prefill <think> is now
handled only when the OpenAI or Ollama provider is configured for it, via
prefilled_reasoning=True or OPENAI_PREFILLED_REASONING /
OLLAMA_PREFILLED_REASONING. Hosted providers keep the default.
Signed-off-by: FenjuFu <92919259+FenjuFu@users.noreply.github.com>
Drop home .agents cleanup already adopted in maintainer PR #3165. Retain finding 3 from #3116, which the author identifies as still pending in this PR: keep explicitly target-capable modules while skipping unsupported harness dependencies. Existing manifest regression and 80% scoped coverage checks pass.
Merge current main and retain local review fixes for recovery only. Upstream now contains maintainer implementation 8cfbc267 for issue 3116; this commit is not proposed as PR delivery. Targeted migration, lifecycle, adapter and executor checks pass; full repository CI has not run for this revision.
montjeffrey