A common first test for a new tool is to call it with valid arguments and check the response. That is necessary. It is not sufficient for an agent-facing workflow.
An agent is a probabilistic user of a deterministic interface. It can choose the wrong tool, form an almost-correct argument, act on a partial result, fail to notice a warning, or stop after a permission denial even though a safe recovery path exists. A tool can have a perfect unit-test suite and still be a poor instrument for an agent.
A tool is not ready when it returns the right answer once. It is ready when an agent can make progress after the common ways it goes wrong.
That broadens the evaluation target. In addition to “Does the endpoint work?”, ask: “Can an agent get from a realistic user goal to a correct, explainable state—and recover when the first attempt is imperfect?”
Why ordinary API tests leave a gap
Traditional API tests focus on contract correctness:
- Is the input accepted or rejected correctly?
- Is the response shaped correctly?
- Does the endpoint perform the intended effect?
- Are permissions enforced?
Those remain important. The MCP tools specification defines schemas, output structures, validation, and error behavior, all of which deserve deterministic tests.
But agents add an interaction layer above the API contract:
- Can the model identify when this is the right tool?
- Can it form an acceptable request from user language?
- Does the result make the next action clear?
- Can it correct a bad argument without a human translating an error code?
- Does it know when to ask for approval instead of retrying?
These are behavioral questions. Task-based evaluation can answer them in ways that a larger collection of isolated endpoint fixtures cannot. Anthropic’s context-engineering guidance also argues for clear, non-overlapping tool boundaries, making tool selection and context cost part of the system under evaluation.
A proposed five-path evaluation suite
Virillio Code Editorial proposes five paths as a compact baseline for agent-facing tools whose failures matter. This is a normative house framework, not an MCP requirement or a complete evaluation standard. Teams should add domain-specific safety, security, latency, and quality scenarios according to the tool’s risk. Each path begins with a user-level task rather than a prefilled function call.
- Path: Success — What you are testing: The agent can select and use the tool for its intended workflow — A healthy result: Correct outcome with a clear explanation and no needless calls
- Path: Ambiguity — What you are testing: The request lacks a fact or could map to several actions — A healthy result: The agent asks a targeted question or uses a safe read-only discovery step
- Path: Correction — What you are testing: The first tool call has a plausible but invalid argument — A healthy result: The error tells the agent what to change and it successfully self-corrects
- Path: Partial outcome — What you are testing: Work may have completed only in part, or a dependency may have timed out — A healthy result: The agent identifies what is known, avoids unsafe replay, and continues from evidence
- Path: Policy boundary — What you are testing: The action needs permission, confirmation, or exceeds allowed scope — A healthy result: The agent stops safely, explains the constraint, and requests the right next step
The baseline is deliberately small. It does not simulate every possible model behavior; it samples failure modes that can turn a functional integration into an unreliable agent loop.
1. Success: measure the whole task, not only the call
The happy path should use a realistic prompt and a realistic starting state. Measure whether the final user-facing outcome is correct, but also record how the agent got there:
- Which tool did it select?
- How many calls did it make?
- How long did the task take?
- How much context did the tool add?
- Could a simpler tool or better description have removed a detour?
Anthropic's tool-design guidance recommends looking beyond top-level accuracy at metrics including tool-call runtime, task runtime, call count, token use, and tool errors. A task can technically complete through an expensive or brittle sequence whose weaknesses appear only across repeated runs.
2. Ambiguity: reward clarification, not bravado
Underspecified requests can create avoidable failures. “Update the deployment,” “clean up these records,” or “send that to the team” may each have several valid interpretations.
A strong tool ecosystem gives the agent a safe way to learn before it acts. That may be a read-only lookup, an explicit choice prompt, or a compact result that names the missing field. The evaluation should not reward the model for guessing correctly by chance.
Score the path as successful when the agent identifies the ambiguity and either asks a useful question or chooses a nondestructive discovery action. Treat an unjustified mutation as a failure even if the final state happens to look plausible.
3. Correction: turn errors into steering
An error message is part of the tool interface. If an agent sends a malformed identifier, an invalid date, or a contradictory filter, the response should say what is wrong and what a valid retry looks like.
The test is simple: give the agent an instruction that predictably produces one plausible mistake. Then inspect whether it can recover without hidden evaluator hints.
Weak result:
“Invalid request.”
Useful result:
“The project identifier is not recognized. Use the project slug from the list result, not its display name.”
The difference is not friendliness. It is whether the tool result creates a next action. In an agent loop, opaque errors multiply calls, consume context, and make safe recovery less likely.
4. Partial outcome: make uncertainty legible
The hardest tools are those with side effects or unreliable dependencies. A timeout after a write request does not always mean “nothing happened.” A bulk operation may complete nine of ten items. A test command may finish with both useful output and a nonzero status.
An agent must be able to distinguish:
- confirmed completion,
- confirmed non-completion,
- partial completion,
- unknown outcome that requires verification before retry.
Include one of these states in the held-out evaluation. Check that the tool returns durable evidence of what it knows, and check that the agent seeks verification instead of blindly replaying a potentially non-idempotent action.
This is where result design and evaluation meet. A model cannot recover from a state the tool refuses to describe.
5. Policy boundary: test the safe stop
Permission denials and confirmation requirements are designed boundaries between an agent's capability and a user's authority, so they belong in evaluation for tools that can cross those boundaries.
The MCP tools specification calls out access control, confirmation, and audit concerns. An evaluation should therefore verify not just that the server blocks an unsafe action, but that the agent responds appropriately:
- It does not keep attempting the blocked action.
- It explains what authority or confirmation is needed.
- It preserves the user's intent without silently broadening scope.
- It continues with a safe alternative only when one is genuinely authorized.
A safe stop is a successful result. In many real workflows, it is more valuable than a superficially completed task with unclear authority.
A useful evaluation record
To improve tools over time, keep a record that separates the task outcome from the model transcript. For each run, capture:
- Field: User task and initial state — Why it matters: Makes the test reproducible
- Field: Expected end state and allowable paths — Why it matters: Avoids overfitting to one exact call sequence
- Field: Tool definitions and versions — Why it matters: Lets you attribute performance changes
- Field: Calls, normalized results, and timing — Why it matters: Reveals unnecessary work and rough edges
- Field: Recovery classification — Why it matters: Shows whether the agent corrected, clarified, verified, or stopped safely
- Field: Human review notes — Why it matters: Captures issues the numeric score misses
Keep a held-out set. If developers repeatedly tune a tool against the same examples, the agent may learn the evaluator's quirks rather than become better at the underlying workflow. Anthropic makes the same point in its tool guidance: held-out tests are how you distinguish genuine improvement from overfitting.
Complementary metrics for different failures
One success-rate number cannot explain how a tool fails. A small dashboard can combine measures such as:
- Task completion: Did the user-level goal reach the intended state?
- Correct tool choice: Did the agent choose an appropriate tool rather than a near-duplicate?
- Recovery rate: After a controlled failure, did it reach a safe next state?
- Unsafe replay rate: Did it repeat a side-effecting call when the result was uncertain?
- Clarification quality: Did it ask for the one fact that unblocked progress?
- Tool-call and token cost: Did it use an efficient amount of work to succeed?
- Explanation quality: Can a reviewer understand what happened and why?
These measures expose different fixes. A low completion rate may need a new capability. A high correction failure rate may need better validation messages. A high call count may signal overlapping tools or confusing descriptions. The remedy should follow the observed failure, not a generic prompt tweak.
How to adopt the suite without building a lab first
One practical adoption path is to start with a few tools in valuable workflows. Write one scenario for each relevant path, run them with the same model and a fixed environment, and review the traces by hand before automating every metric.
When a scenario fails, classify the cause before changing the tool:
- The capability is missing.
- The tool boundary is wrong or overlaps another tool.
- The description or input schema is unclear.
- The result hides the next action.
- The policy boundary is underspecified.
- The agent needs a better task-level instruction or example.
This order matters. Teams often try to prompt around a tool whose contract is the actual problem.
What we are learning building Virillio Code
At Virillio Code, the relevant work remains in development, but the evaluation principle is durable: agent reliability lives at the boundary between a user's request, a tool's deterministic behavior, and the model's next decision. A passing endpoint test verifies only one of those three.
Designing for recovery makes tools more honest for humans as well. Clear errors, explicit partial outcomes, visible approvals, and audit-ready evidence are good interface design whether the caller is a model or a person.
Sources and further reading
- Writing effective tools for AI agents — Agent tool design should use comprehensive evaluations, held-out tests, and metrics beyond response correctness.
- MCP Tools specification — Tool schemas, structured outputs, validation, errors, confirmation, access controls, and audit concerns need explicit treatment.
- Effective context engineering for AI agents — Clear, low-overlap tools and bounded context help agent systems behave reliably.
Editorial disclosure
This article was substantially researched, drafted, and revised with AI through the Virillio Code editorial workflow. Virillio Code publishes the final text under its editorial byline, and the supporting primary sources are linked above.