docs(skills): add focused tool audit and agent trial guidance

This commit is contained in:
morluto committed 2026-11-10 05:44:04 -05:00
1 parent d2f9b8b7de
commit 94ca4b656a
2 files changed
+69

No files matched your search

+7
View File
@@ -26,6 +26,13 @@ claims. Check relevant rendered MCP prompts, guided workflows and packaged skill
advice against the live contracts and exercise their suggested handoffs. Report
working alternatives and evidence limits as well as findings.
For wording audits and agent trials, use
[focused tool-description trials](../../../docs/development/client-verification.md#focused-tool-description-trials).
Start with representative workflow boundaries and repeat failures or ambiguous
outcomes; keep actual arguments and supported final claims separate. Reuse the
repository's evaluation tooling and report actual runtime/model versions and
synthetic versus production coverage rather than copying a harness into a skill.
For a design request, return the proposed boundary, input/result shape, Evidence
semantics, nearest alternative, and verification approach. Implement when
requested, preserving the user's scope and existing authorization.
+62
View File
@@ -643,3 +643,65 @@ npm run evidence:check
The records preserve unsupported and unverified coverage as explicit unknowns.
Run the matching real-tool prerequisites described in this guide.
### Focused tool-description trials
For a wording or discoverability question, begin with one representative case
per affected workflow boundary and its nearest valid alternative. Repeat a
concrete failure or ambiguous result after repair. Add repetitions when the
requested confidence or comparison needs them; a large matrix is not the default.
Use the user's selected model and budget, and verify the actual client and Node
executables used by the trial rather than assuming the development shell's
versions apply.
Reuse the existing optional lane and source-owned scenarios where they fit:
```bash
REA_AGENT_EVAL_SCENARIOS=javascript-module-view npm run verify:agent
```
Select the account-supported model with `REA_AGENT_EVAL_MODEL`. Read the selected
prompt and rubric before interpreting the gate: closed factual scenarios request
complete producer Evidence, while `javascript-module-view` exercises retained
views with heuristic scoring. Neither gate alone proves correct selection or
fully supported prose for a different task. Extend the narrow scenario or grader
in the existing evaluation tooling when necessary; reusable runners and fixtures
belong there, not in skills or copied temporary harnesses.
Capture the actual `initialize` and `tools/list` surface. Compact presentation may
remove parameter descriptions or nested selector shapes, so a concise exact call
example can remain necessary. Verify the producer-to-consumer handoff before
removing its visible syntax. For a wording comparison, hold prompts, schemas,
tool order, fixtures, permissions and host settings fixed; change one surface at
a time. Use fresh agent context without the audit's answers. Record later source,
fixture or runtime changes separately rather than claiming a controlled delta.
Score these outcomes separately:
- Tool selection, accepting other valid routes.
- Canonical argument validity and the exact requested paths, IDs, selectors,
fields, units and delivery. Structurally valid arguments can select the wrong
input; fixtures should refuse that selection instead of returning the expected
answer anyway.
- Recovery and cleanup. A valid unresolved lookup is not a malformed request;
normal status reads and cleanup-only retries are not automatically mistakes.
Do not infer that work stopped from a host-only timeout.
- Final claims against returned facts and coverage. Correct text does not erase
malformed-call loops, and valid calls do not prove every claim. A failed name
lookup is not a negative reference search; an exhausted partial page does not
establish absence in unexamined source. Generic responses without state facts
cannot establish that a session is closed.
Retain prompts, condition/catalog hashes, actual calls/results, failures, final
answers, model/client/runtime versions and cleanup outcomes. Report whole-task
input, cached input, output and elapsed time when available, and label unmeasured
CPU or process-family memory. Fewer description bytes do not establish fewer
task tokens or faster execution. Remove copied authentication after owned
processes settle and review retained artifacts before sharing them.
Controlled errors can test agent recovery without target effects. Label them as
simulated and verify applicable workflows separately through the production
adapter and real provider boundaries. State tool-subset versus full/progressive
catalog scope and remaining provider/host coverage. Small one-shot trials are
smoke checks, not statistical success rates or evidence that newer models need
less guidance in general.