An agent can pass a static task and still be unsafe to trust after the task changes.
That is not a contradiction. Static evaluation usually asks whether a system can reach an expected result from a fixed starting state. Real long-running work asks an additional question:
What does the agent do when an input it relied on becomes false, unavailable, superseded, or uncertain?
The desirable response is neither “continue confidently” nor “throw everything away.” It is revalidation: identify what changed, find the dependent work, preserve what remains valid, and choose an honest next disposition.
Teams should test that behavior directly.
Completion is only one score
Final completion matters. It is not enough.
The Gaia2 preprint makes the evaluation gap clear by modeling environments that evolve independently of the agent. In those settings, an agent has to deal with temporal constraints, noisy events, ambiguity, and collaboration rather than simply follow a fixed path.
The When Tools Fail preprint introduces ToolMaze, a benchmark for how agents respond to explicit or implicit tool changes and to transient or permanent failures. The CostBench preprint similarly introduces dynamic blocking events, including tool failures and cost changes, in a cost-aware travel-planning setting.
These papers study particular environments and methods. Their findings should not be converted into a universal score for every agent. Taken together, they motivate our broader evaluation lesson: dynamic adaptation should be measured separately from happy-path completion.
The State Perturbation Suite
The State Perturbation Suite is a proposed evaluation method for testing revalidation at the task boundary. Begin with a normal scenario. Let the agent form a bounded plan or produce a partial artifact. Then change one declared dependency in a controlled evaluation environment.
The test does not ask the agent to reveal private scratch work or hidden reasoning. It asks for observable task behavior:
- Did the system detect a relevant change?
- Did it identify the work actually affected?
- Did it take an appropriate disposition: continue, recheck, re-plan, hold, abstain, or seek review?
- Did it preserve valid work while preventing an unsupported claim or effect?
- Did the final artifact or status accurately disclose the change and remaining uncertainty?
The test is about evidence and state transitions, not a transcript of how the model thought.
Change one dependency at a time
A useful suite varies the kind of dependency, not merely the wording of the prompt.
- Perturbation: Source freshness — Controlled change: A source used for a material claim changes or expires — Expected task behavior: Mark the dependent claim for review; refresh or qualify it before presenting it as current
- Perturbation: Tool availability — Controlled change: A selected tool becomes unavailable — Expected task behavior: Hold or re-plan the affected step; do not invent success or substitute a consequential path without review
- Perturbation: Tool semantics — Controlled change: A result shape or operational meaning changes — Expected task behavior: Recheck the planned operation and any acceptance condition that depended on the old behavior
- Perturbation: Authority or approval — Controlled change: A scope, consent, or approval condition changes — Expected task behavior: Stop treating the old plan as permitted; request a bounded new decision where needed
- Perturbation: Shared target state — Controlled change: Another worker or user changes the target artifact — Expected task behavior: Read the current target, reconcile the conflict, and preserve only compatible work
- Perturbation: Time or deadline — Controlled change: The task is delayed past a freshness window or deadline — Expected task behavior: Update the task disposition, revalidate time-sensitive claims, or return a bounded partial result
- Perturbation: Dependency conflict — Controlled change: Two inputs now support incompatible conclusions — Expected task behavior: Preserve the conflict, identify the missing evidence or owner, and avoid false consensus
Each case should be safe, synthetic, or otherwise controlled. The point is not to create production failures. It is to find out whether a system notices conditions it says matter.
Score the response to change
A final pass/fail hides the most useful diagnostic information. The suite instead scores four dimensions.
- Dimension: Detection — Question: Did the agent notice a relevant perturbation? — Good signal: It identifies the changed dependency without treating unrelated noise as a blocker
- Dimension: Scope — Question: Did it identify which work became stale? — Good signal: It rechecks the dependent region rather than all work or none of it
- Dimension: Disposition — Question: Did it choose an honest next state? — Good signal: It continues, re-plans, holds, abstains, or escalates in proportion to the uncertainty and consequence
- Dimension: Evidence — Question: Did it leave a concise basis for the decision? — Good signal: A reviewer can see the changed condition, affected artifact, and validation result without reading private context
This creates a useful distinction between two failures:
- false continuation: the agent moves forward as if nothing changed; and
- false reset: the agent discards valid, independent work because it cannot localize the change.
Both cost users time. The first risks incorrect outcomes. The second makes long-running systems feel fragile and wasteful.
Revalidation should be selective
The word “revalidation” can sound like a demand to rerun everything. That is rarely practical.
A task often contains independent work: an already checked formatting change, a source that remains current, a completed analysis that did not depend on the altered constraint, or a safely bounded draft. A good system should retain that work while marking only the affected region as uncertain.
This is why the test needs explicit dependencies. If a team cannot say which claim, artifact, plan branch, or effect relied on a source or tool, it will not be able to tell whether the agent revalidated selectively.
The State Perturbation Suite makes that gap visible. If every changed source produces a full restart, the task graph may be too opaque. If no changed source affects the result, the agent may be carrying untracked assumptions.
Test dispositions beyond retries
Retrying a failed tool is only one possible response to change. A long-running agent may need to:
- refresh evidence;
- ask for an updated approval;
- choose a safe alternative;
- re-plan a bounded subtask;
- report an unresolved dependency;
- return accepted partial work; or
- abstain because a high-consequence decision no longer has a valid basis.
The When Tools Fail preprint is especially useful as a reminder that recovery is not equivalent to repeating the same action. ToolMaze's perturbation taxonomy includes conditions where the failure is not obvious. Our design recommendation is to define what an agent may claim when an outcome is uncertain, rather than automate more aggressive retries.
A practical test design
- Choose a workflow with an explicit objective, one or two dependencies, and an observable acceptance condition.
- Record the baseline plan, evidence references, tool assumptions, and allowed dispositions at a high level.
- Run the baseline scenario in a safe evaluation environment.
- Introduce one perturbation after the relevant work begins.
- Define the expected observable response before inspecting the run.
- Evaluate detection, affected scope, disposition, evidence, and acceptance outcome.
- Repeat with a different perturbation type and with an unrelated change to measure false alarms.
An example expectation might be: “If a cited source is replaced with materially conflicting information before a public claim is finalized, the agent must mark that claim for revalidation and must not state it as current until the conflict is resolved.” That is testable without exposing sensitive task content.
Metrics that encourage the right behavior
Useful measures include:
- perturbation detection rate: the share of material changes the system identifies;
- selective-revalidation accuracy: whether the system rechecks the work that actually depended on the changed condition;
- false-continuation rate: the share of runs that advance with a material stale assumption;
- false-reset rate: the share of runs that discard valid independent work;
- disposition accuracy: whether the chosen next state matches the task's authority, evidence, and consequence boundaries;
- recovery verification rate: the share of changed scenarios whose final outcome is independently checked; and
- explanation coverage: whether a reviewer can understand the change and response without replaying the entire run.
These are engineering measures, not model rankings. They help a team decide which failure mode to improve next.
Limitations
A perturbation suite cannot prove that an agent will handle every future surprise. It can also create artificial behavior if the test changes are predictable, unrealistic, or too easy to detect.
Keep the scenarios representative, vary the timing of changes, and include both material and immaterial perturbations. Most importantly, do not confuse benchmark success with permission to bypass human review for irreversible or sensitive actions.
The Virillio Code editorial perspective
The general lesson is to make revalidation observable and testable at the task boundary. The State Perturbation Suite is a proposed educational framework, not a description of a shipped or planned Virillio Code capability.
Sources and further reading
- Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments — A recent preprint introduces agent-evaluation environments that evolve independently of agent actions, including temporal constraints, noise, ambiguity, and collaboration.
- When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents — A recent preprint tests explicit and implicit, transient and permanent tool perturbations and evaluates recovery and replanning.
- CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents — A clearly labeled preprint studies cost-aware tool planning in a travel-planning environment with different tool costs and dynamic blocking events such as failures and cost changes.
Editorial disclosure
This article was substantially researched, drafted, and revised with AI through the Virillio Code editorial workflow. Virillio Code publishes the final text under its editorial byline, and the supporting primary sources are linked above.