The Verification Loop for Reliable Computer-Use Agents

Computer use has an unusually dangerous illusion of reliability: a screenshot can look right even when the task is wrong.

The agent may click the intended button but the UI can ignore it. It may fill a visible field while a hidden validation error blocks submission. It may land on a confirmation screen without completing an external action. It may read text embedded in a page that looks like an instruction but has no authority to redirect the task.

That is why a screenshot is an observation, not state.

A computer-using agent should never treat an action as complete merely because it issued a click. It needs fresh evidence that the intended state transition occurred.

Both OpenAI and Anthropic describe computer use as an agent loop: the model requests UI actions, the application executes them, and the results—including screenshots or errors—return to the model for the next decision. The important design insight is not the click itself. It is the closed loop around the click.

Why visual automation is different

A programmatic API normally expresses a named operation and returns a structured response. A UI gives the agent a visual surface designed for a human. Meaning is distributed across text, layout, focus state, transient notifications, and sometimes hidden application logic.

This creates four reliability gaps:

• Gap: Ambiguous target — What the agent observes: Several similar buttons or fields — What might actually be true: The action may land on the wrong control
• Gap: Invisible state — What the agent observes: A field appears filled — What might actually be true: Validation or focus may have failed
• Gap: Delayed effect — What the agent observes: A loading indicator or changed page — What might actually be true: The underlying action may still fail or be pending
• Gap: Untrusted content — What the agent observes: Text, email, PDF, or webpage instructions — What might actually be true: The content may be irrelevant, malicious, or unauthorized

The answer is not to ask the agent to be more confident. It is to make every important interaction follow a small, verifiable state machine.

A proposed OIAV loop: Observe, Interpret, Act, Verify

This article proposes four steps for every computer-mediated change, with a separate authorization check before any consequential action. OIAV is a design shorthand used here, not an industry or platform standard.

• Step: Observe — Question: What is actually on the current screen? — Useful artifact: Fresh screenshot, window metadata, accessible tree, or a trusted tool result
• Step: Interpret — Question: What does the screen mean relative to the user's goal? — Useful artifact: Chosen target, uncertainty, and intended state transition
• Step: Act — Question: What is the smallest safe action that advances the goal? — Useful artifact: One ordered UI action with a bounded scope
• Step: Verify — Question: What evidence proves the intended transition happened? — Useful artifact: Fresh screenshot, confirmation receipt, changed record, or deterministic read-back

The loop is intentionally boring. Reliability comes from avoiding the gap between “the agent attempted an action” and “the system reached the intended state.”

Observe: start from a fresh, scoped view

An old screenshot is a stale cache. Before any significant action, the agent needs an observation that corresponds to the current display and coordinate space.

This is a practical issue, not only a philosophical one. Computer-use systems may transform screenshots to fit model input limits, while UI actions use a coordinate space that must stay aligned with the image. The visual frame can also change as a result of asynchronous loads, modal dialogs, focus shifts, or another user interaction.

Capture enough information to interpret the frame:

• a fresh screenshot or trusted UI-state query,
• the application or window identity,
• screen dimensions and scaling context,
• the task goal and constraints that are still in force,
• whether the UI contains anything that requires confirmation.

Do not solve ambiguity by clicking. If the agent cannot distinguish two likely targets, it should zoom, inspect a trusted semantic surface, or ask for clarification.

Interpret: separate what the page says from what the user authorized

The model sees third-party content through a screen, browser, document, or message. That content can be useful evidence. It is not a new source of authority.

OpenAI's computer-use guidance is explicit on this point: page text, screenshots, emails, and other third-party material should be treated as untrusted input, and only the user's instructions grant permission. This is a sound rule for every computer-use stack.

Before acting, the agent should classify the proposed transition:

• Class: Observe — Example: Open a detail page or search a list — Default behavior: Proceed within the allowed scope
• Class: Reversible local change — Example: Edit a draft or change a local preference — Default behavior: Proceed if the task clearly authorizes it; verify
• Class: Consequential external change — Example: Send, submit, purchase, publish, delete, or grant access — Default behavior: Apply the governing platform policy; many workflows require fresh action-time confirmation even when earlier authority was broad
• Class: Ambiguous request — Example: “Update the account” with multiple possible targets — Default behavior: Ask a focused question or use a read-only discovery step

The classification is more useful than a generic “risky / not risky” flag because it tells the agent what to do next.

Act: prefer a short, ordered action group

Computer use is sequential even when a model can request several actions at once. In a batch such as click, type, and screenshot, the second action depends on the first action successfully setting focus; the third is meaningful only if the first two completed.

Anthropic's implementation guidance reflects this: run computer actions in order, stop after the first failure, return a result for every action, and let the agent replan from the observed outcome.

That suggests a simple policy:

• Make action groups short.
• Keep actions ordered.
• Stop a group when a prerequisite fails.
• Do not speculate past a failed or ambiguous step.
• Use a fresh observation after the group when the next decision depends on state.

This costs a few more turns than issuing a long script. It saves far more turns than recovering from a string of actions that were correct only under an assumption that stopped being true after the first click.

Verify: prove the state transition, not the gesture

Verification must match the kind of action.

• Action: Type into a field — Weak evidence: Cursor moved — Stronger evidence: Field value is visible and accepted by validation
• Action: Click Save — Weak evidence: Button press animation — Stronger evidence: Saved-state indicator or a trusted read-back of the updated record
• Action: Submit a form — Weak evidence: Navigation occurred — Stronger evidence: Confirmation identifier, receipt, or status change
• Action: Delete an item — Weak evidence: The item disappeared from one view — Stronger evidence: A scoped list refresh or immutable audit evidence
• Action: Download a file — Weak evidence: Browser showed a notification — Stronger evidence: File metadata, checksum, or successful use in the intended workflow

The strongest evidence is usually a read-back from an authoritative system. A screenshot can be enough for a low-consequence local task, but it should not be the sole proof of an irreversible external effect.

This is where many computer agents become brittle: they stop after a gesture instead of asking what the user would need to see to trust the outcome.

Authorization is a gate, not a note

Verification answers “did it happen?” Authorization answers “may it happen?” They are separate checks.

For consequential actions, put the authorization gate before execution:

1. State the specific proposed change in user language.
2. Identify the target and scope.
3. Explain the nontrivial consequence or irreversibility.
4. Request confirmation unless the user already granted bounded permission.
5. Execute only after that boundary is satisfied.
6. Verify and report the result truthfully.

OpenAI's guidance recommends a human in the loop for purchases, authenticated flows, destructive actions, or actions that are hard to reverse. The exact confirmation rule should come from the governing product policy and the user's scoped authority; the agent should not invent a weaker rule for convenience.

Context discipline matters for visual loops

Screenshots are expensive context. Keeping every past image in the prompt causes the current target, newest error, and user intent to compete with stale visual clutter.

The answer is not to forget everything. Keep a compact execution record:

• current goal and confirmed constraints,
• latest verified state,
• last action and result,
• pending authorization request if any,
• stable references to larger artifacts if they are needed later.

Prune stale images once their facts have been extracted. Retain the evidence that matters, not every frame of the animation.

This is the same broader lesson as context engineering: the agent needs the right state for the next decision, not an unbounded record of every possible state.

Design the recovery path before the demo

A reliable computer-use loop has defined behavior for these ordinary events:

• The screenshot is too low-resolution to identify a target.
• A click fails or lands on the wrong control.
• A required field is missing or invalid.
• A site redirects to an unexpected page.
• A confirmation or login boundary appears.
• The page contains instructions that conflict with the user's request.
• The UI indicates a partial, pending, or unknown outcome.

For each case, decide whether the agent should observe again, use another permitted tool, ask the user, stop safely, or verify before retrying. Do not let a generic retry policy decide an action's semantic safety.

A practical preflight checklist

Before enabling a new computer-use workflow, ask:

1. What observation confirms the exact current screen and target?
2. Which transitions are read-only, reversible, consequential, or ambiguous?
3. What user authority is required for the consequential transitions?
4. Which actions must be executed sequentially?
5. What evidence will prove the requested outcome?
6. What should the agent do if evidence is missing or contradictory?
7. Which third-party inputs must be treated as untrusted?

If those answers are unclear, the agent is being asked to navigate a workflow that a human has not fully specified either.

What we are learning building Virillio Code

The computer-use implications for Virillio Code are general rather than product-specific: a trustworthy agent should keep human intent, observed UI state, action authority, and proof of completion separate. The moment an agent crosses from reading a screen to changing a user's world, it needs a stronger loop than “the click looked right.”

Sources and further reading

• OpenAI Computer use guide — Computer-use systems act through screenshots and UI actions, and need isolated environments, user review for high-impact actions, and untrusted-content handling.
• Claude computer use tool — UI actions should run in order, report results and failures, and use screenshots to support replanning and verification.
• Effective context engineering for AI agents — Long-running agents need deliberate, bounded management of the context available for the next decision.

Editorial disclosure

This article was substantially researched, drafted, and revised with AI through the Virillio Code editorial workflow. Virillio Code publishes the final text under its editorial byline, and the supporting primary sources are linked above.