mirror of
https://github.com/morluto/rea.git
synced 2026-11-10 16:57:19 +00:00
docs(skills): add focused tool audit and agent trial guidance
This commit is contained in:
1 parent
d2f9b8b7de
commit
94ca4b656a
2 files changed
+69
No files matched your search
@@ -26,6 +26,13 @@ claims. Check relevant rendered MCP prompts, guided workflows and packaged skill
|
||||
advice against the live contracts and exercise their suggested handoffs. Report
|
||||
working alternatives and evidence limits as well as findings.
|
||||
|
||||
For wording audits and agent trials, use
|
||||
[focused tool-description trials](../../../docs/development/client-verification.md#focused-tool-description-trials).
|
||||
Start with representative workflow boundaries and repeat failures or ambiguous
|
||||
outcomes; keep actual arguments and supported final claims separate. Reuse the
|
||||
repository's evaluation tooling and report actual runtime/model versions and
|
||||
synthetic versus production coverage rather than copying a harness into a skill.
|
||||
|
||||
For a design request, return the proposed boundary, input/result shape, Evidence
|
||||
semantics, nearest alternative, and verification approach. Implement when
|
||||
requested, preserving the user's scope and existing authorization.
|
||||
|
||||
@@ -643,3 +643,65 @@ npm run evidence:check
|
||||
|
||||
The records preserve unsupported and unverified coverage as explicit unknowns.
|
||||
Run the matching real-tool prerequisites described in this guide.
|
||||
|
||||
### Focused tool-description trials
|
||||
|
||||
For a wording or discoverability question, begin with one representative case
|
||||
per affected workflow boundary and its nearest valid alternative. Repeat a
|
||||
concrete failure or ambiguous result after repair. Add repetitions when the
|
||||
requested confidence or comparison needs them; a large matrix is not the default.
|
||||
Use the user's selected model and budget, and verify the actual client and Node
|
||||
executables used by the trial rather than assuming the development shell's
|
||||
versions apply.
|
||||
|
||||
Reuse the existing optional lane and source-owned scenarios where they fit:
|
||||
|
||||
```bash
|
||||
REA_AGENT_EVAL_SCENARIOS=javascript-module-view npm run verify:agent
|
||||
```
|
||||
|
||||
Select the account-supported model with `REA_AGENT_EVAL_MODEL`. Read the selected
|
||||
prompt and rubric before interpreting the gate: closed factual scenarios request
|
||||
complete producer Evidence, while `javascript-module-view` exercises retained
|
||||
views with heuristic scoring. Neither gate alone proves correct selection or
|
||||
fully supported prose for a different task. Extend the narrow scenario or grader
|
||||
in the existing evaluation tooling when necessary; reusable runners and fixtures
|
||||
belong there, not in skills or copied temporary harnesses.
|
||||
|
||||
Capture the actual `initialize` and `tools/list` surface. Compact presentation may
|
||||
remove parameter descriptions or nested selector shapes, so a concise exact call
|
||||
example can remain necessary. Verify the producer-to-consumer handoff before
|
||||
removing its visible syntax. For a wording comparison, hold prompts, schemas,
|
||||
tool order, fixtures, permissions and host settings fixed; change one surface at
|
||||
a time. Use fresh agent context without the audit's answers. Record later source,
|
||||
fixture or runtime changes separately rather than claiming a controlled delta.
|
||||
|
||||
Score these outcomes separately:
|
||||
|
||||
- Tool selection, accepting other valid routes.
|
||||
- Canonical argument validity and the exact requested paths, IDs, selectors,
|
||||
fields, units and delivery. Structurally valid arguments can select the wrong
|
||||
input; fixtures should refuse that selection instead of returning the expected
|
||||
answer anyway.
|
||||
- Recovery and cleanup. A valid unresolved lookup is not a malformed request;
|
||||
normal status reads and cleanup-only retries are not automatically mistakes.
|
||||
Do not infer that work stopped from a host-only timeout.
|
||||
- Final claims against returned facts and coverage. Correct text does not erase
|
||||
malformed-call loops, and valid calls do not prove every claim. A failed name
|
||||
lookup is not a negative reference search; an exhausted partial page does not
|
||||
establish absence in unexamined source. Generic responses without state facts
|
||||
cannot establish that a session is closed.
|
||||
|
||||
Retain prompts, condition/catalog hashes, actual calls/results, failures, final
|
||||
answers, model/client/runtime versions and cleanup outcomes. Report whole-task
|
||||
input, cached input, output and elapsed time when available, and label unmeasured
|
||||
CPU or process-family memory. Fewer description bytes do not establish fewer
|
||||
task tokens or faster execution. Remove copied authentication after owned
|
||||
processes settle and review retained artifacts before sharing them.
|
||||
|
||||
Controlled errors can test agent recovery without target effects. Label them as
|
||||
simulated and verify applicable workflows separately through the production
|
||||
adapter and real provider boundaries. State tool-subset versus full/progressive
|
||||
catalog scope and remaining provider/host coverage. Small one-shot trials are
|
||||
smoke checks, not statistical success rates or evidence that newer models need
|
||||
less guidance in general.
|
||||
Reference in new issue
Block a user