From 94ca4b656aede76eda0ebac3af9aed42a27c21a5 Mon Sep 17 00:00:00 2001 From: morluto <76467478+morluto@users.noreply.github.com> Date: Tue, 10 Nov 2026 05:44:04 -0500 Subject: [PATCH] docs(skills): add focused tool audit and agent trial guidance --- .agents/skills/rea-tool-design/SKILL.md | 7 +++ docs/development/client-verification.md | 62 +++++++++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/.agents/skills/rea-tool-design/SKILL.md b/.agents/skills/rea-tool-design/SKILL.md index 536600a42f..d896081a6b 100644 --- a/.agents/skills/rea-tool-design/SKILL.md +++ b/.agents/skills/rea-tool-design/SKILL.md @@ -26,6 +26,13 @@ claims. Check relevant rendered MCP prompts, guided workflows and packaged skill advice against the live contracts and exercise their suggested handoffs. Report working alternatives and evidence limits as well as findings. +For wording audits and agent trials, use +[focused tool-description trials](../../../docs/development/client-verification.md#focused-tool-description-trials). +Start with representative workflow boundaries and repeat failures or ambiguous +outcomes; keep actual arguments and supported final claims separate. Reuse the +repository's evaluation tooling and report actual runtime/model versions and +synthetic versus production coverage rather than copying a harness into a skill. + For a design request, return the proposed boundary, input/result shape, Evidence semantics, nearest alternative, and verification approach. Implement when requested, preserving the user's scope and existing authorization. diff --git a/docs/development/client-verification.md b/docs/development/client-verification.md index ebb37bcd37..24db5ce57e 100644 --- a/docs/development/client-verification.md +++ b/docs/development/client-verification.md @@ -643,3 +643,65 @@ npm run evidence:check The records preserve unsupported and unverified coverage as explicit unknowns. Run the matching real-tool prerequisites described in this guide. + +### Focused tool-description trials + +For a wording or discoverability question, begin with one representative case +per affected workflow boundary and its nearest valid alternative. Repeat a +concrete failure or ambiguous result after repair. Add repetitions when the +requested confidence or comparison needs them; a large matrix is not the default. +Use the user's selected model and budget, and verify the actual client and Node +executables used by the trial rather than assuming the development shell's +versions apply. + +Reuse the existing optional lane and source-owned scenarios where they fit: + +```bash +REA_AGENT_EVAL_SCENARIOS=javascript-module-view npm run verify:agent +``` + +Select the account-supported model with `REA_AGENT_EVAL_MODEL`. Read the selected +prompt and rubric before interpreting the gate: closed factual scenarios request +complete producer Evidence, while `javascript-module-view` exercises retained +views with heuristic scoring. Neither gate alone proves correct selection or +fully supported prose for a different task. Extend the narrow scenario or grader +in the existing evaluation tooling when necessary; reusable runners and fixtures +belong there, not in skills or copied temporary harnesses. + +Capture the actual `initialize` and `tools/list` surface. Compact presentation may +remove parameter descriptions or nested selector shapes, so a concise exact call +example can remain necessary. Verify the producer-to-consumer handoff before +removing its visible syntax. For a wording comparison, hold prompts, schemas, +tool order, fixtures, permissions and host settings fixed; change one surface at +a time. Use fresh agent context without the audit's answers. Record later source, +fixture or runtime changes separately rather than claiming a controlled delta. + +Score these outcomes separately: + +- Tool selection, accepting other valid routes. +- Canonical argument validity and the exact requested paths, IDs, selectors, + fields, units and delivery. Structurally valid arguments can select the wrong + input; fixtures should refuse that selection instead of returning the expected + answer anyway. +- Recovery and cleanup. A valid unresolved lookup is not a malformed request; + normal status reads and cleanup-only retries are not automatically mistakes. + Do not infer that work stopped from a host-only timeout. +- Final claims against returned facts and coverage. Correct text does not erase + malformed-call loops, and valid calls do not prove every claim. A failed name + lookup is not a negative reference search; an exhausted partial page does not + establish absence in unexamined source. Generic responses without state facts + cannot establish that a session is closed. + +Retain prompts, condition/catalog hashes, actual calls/results, failures, final +answers, model/client/runtime versions and cleanup outcomes. Report whole-task +input, cached input, output and elapsed time when available, and label unmeasured +CPU or process-family memory. Fewer description bytes do not establish fewer +task tokens or faster execution. Remove copied authentication after owned +processes settle and review retained artifacts before sharing them. + +Controlled errors can test agent recovery without target effects. Label them as +simulated and verify applicable workflows separately through the production +adapter and real provider boundaries. State tool-subset versus full/progressive +catalog scope and remaining provider/host coverage. Small one-shot trials are +smoke checks, not statistical success rates or evidence that newer models need +less guidance in general.