A Coding Agent’s Result Needs a Claim-Coverage Map

An agent finishes a coding task and reports:

“The bug is fixed. The tests pass. Nothing else changed. This is ready to merge.”

That sentence is not one claim. It is at least four:

1. the reported behavior now occurs;
2. relevant checks passed;
3. unrelated behavior or scope was preserved; and
4. the remaining risk is low enough for the next decision.

Each claim needs different evidence. A single passing check may support one of them and leave the others open. A clean diff may support a narrow scope statement but not a compatibility claim. An unchanged working tree may show that a particular agent session did not leave a file edit, but it does not prove that no other effect occurred.

An agent’s result needs a Claim-Coverage Map.

A Result Supports Only the Claims Its Evidence Covers

The Coding-agents can replicate scientific machine learning papers preprint describes a workflow that turns selected computational claims into explicit targets. It records generated evidence, provenance, comparisons, and validation checks rather than letting a final agent message stand in for support. The study concerns twelve runs across four scientific machine-learning papers, not everyday repository maintenance. An architectural inference that should be validated in other settings is that a generated artifact and an evidence-backed claim remain different objects.

The AutoMat paper, arXiv v2 and marked COLM 2026, evaluates coding agents as they reconstruct computational workflows from papers. Its error analysis identifies incomplete procedures, methodological deviations, and execution fragility as failure modes. This is a domain-specific benchmark, not proof that all software tasks behave the same way. It still illustrates why a successful-looking output cannot automatically establish every conclusion an agent draws from it.

The design response should not be to save a transcript of every tool call. It should be to bind each material assertion to the narrow evidence that bears on it.

The Claim-Coverage Map

The Claim-Coverage Map is a proposed application-level record for the assertions that matter to a reviewer, task owner, or downstream agent. It is not a proof system, a replacement for independent review, or an instruction to retain private test output.

• Map element: Claim — Question it answers: What exactly is the agent asserting in plain language? — Why it matters: Splits a polished summary into statements that can be evaluated separately.
• Map element: Claim type — Question it answers: Is this a behavior, compatibility, scope, absence, performance, readiness, or no-action claim? — Why it matters: Different claims need different kinds of support.
• Map element: Observable criterion — Question it answers: What observation would count for or against the claim? — Why it matters: Prevents a generic “checks passed” statement from doing too much work.
• Map element: Evidence reference — Question it answers: Which permitted artifact, check, or review result bears on it? — Why it matters: Makes the support inspectable without copying raw output.
• Map element: Evidence quality — Question it answers: Was the result targeted, partial, environment-limited, flaky, independently reviewed, or unavailable? — Why it matters: Preserves the difference between evidence and certainty.
• Map element: Scope and freshness — Question it answers: Which inputs, versions, time window, and conditions does the evidence cover? — Why it matters: Avoids extending a bounded result into a universal statement.
• Map element: Disposition — Question it answers: Is the claim supported, indirect, unsupported, contradicted, or awaiting independent review? — Why it matters: Gives uncertainty an honest home in the final result.
• Map element: Next decision — Question it answers: What may safely happen next, and what additional check or approval is required? — Why it matters: Connects evidence to action without treating the map as authority by itself.

This is intentionally smaller than a full code-review handoff. A handoff helps a reviewer understand the entire change. A Claim-Coverage Map asks whether each sentence in the agent’s conclusion can carry its own weight.

Keep Evidence Bound to Its Claim Type

• Claim type: Behavior — A common overreach: “One successful scenario proves the feature is fixed everywhere.” — A more honest evidence posture: State the tested behavior, its input boundary, and untested variants.
• Claim type: Compatibility — A common overreach: “The targeted regression check passed, so no existing behavior changed.” — A more honest evidence posture: Name the compatibility surface and the checks that cover it; disclose what remains untested.
• Claim type: Scope — A common overreach: “Nothing else changed.” — A more honest evidence posture: Describe the reviewed change boundary and any known incidental files or effects.
• Claim type: Performance — A common overreach: “It is faster.” — A more honest evidence posture: Identify the measurement, workload, environment, and comparison boundary; avoid an unqualified global claim.
• Claim type: Readiness — A common overreach: “Ready to merge.” — A more honest evidence posture: State which review, policy, verification, and approval conditions have or have not been met.
• Claim type: No action — A common overreach: “No change is needed.” — A more honest evidence posture: Record the symptom boundary, evidence of current behavior, and any limitation in the reproduction.

The FixedBench preprint studies 200 human-verified reports for which no code change is required and reports that the tested agents still proposed undesirable non-test, non-documentation changes in 35–65% of cases. That benchmark-specific result supports treating a no-change conclusion as a claim with a bounded basis; it is not a universal estimate of coding-agent behavior.

The map makes an important habit visible: evidence for one claim should not silently migrate to a stronger neighbor.

A targeted test can be excellent evidence for the reported regression. It is not automatically evidence that the whole repository is unchanged, that an external deployment is authorized, or that a user’s broader intent was satisfied. Those may be reasonable next questions, but they are different questions.

A small illustrative map

Suppose an agent receives a report that filtered results sometimes lose their selection constraint after continuation. A truthful final summary might need several entries:

• Claim: The reported continuation scenario preserves the initial selection constraint. — Evidence boundary: A targeted, permitted regression scenario covering the stated input shape. — Disposition: Supported for the recorded scenario.
• Claim: Existing continuation behavior remains compatible for adjacent cases. — Evidence boundary: A limited set of related checks; broader integration coverage is not available. — Disposition: Partially supported; broader compatibility remains unverified.
• Claim: The change is confined to the intended behavior boundary. — Evidence boundary: Reviewed change map plus the current working-state observation. — Disposition: Supported within the review boundary; not a claim about every runtime effect.
• Claim: The change is ready for merge. — Evidence boundary: Required review and release conditions are not yet complete. — Disposition: Awaiting independent review; do not describe as merged or released.

The map adds documentation work. A compact form can keep that overhead bounded while making the agent’s conclusions legible enough for a reviewer to decide which ones are adequate for the risk.

Keep assertion coverage separate from completion

A Completed Agent Task Needs an Acceptance Contract proposes a task-level Completion Certificate. It asks whether the objective’s acceptance conditions are met. The Evidence Package Behind a Strong Coding-Agent Handoff proposes a review-oriented package covering intent, change map, verification, residual risk, and review route.

The Claim-Coverage Map is more granular:

• Record: Claim-Coverage Map — Unit of review: A material assertion — Main question: What evidence supports this sentence, under what conditions?
• Record: Code-review handoff — Unit of review: A proposed change — Main question: How should a reviewer understand and evaluate this diff?
• Record: Completion Certificate — Unit of review: A task outcome — Main question: Do the stated acceptance conditions justify a completion claim?

The three can reinforce each other. The map can feed the verification and residual-risk portions of a handoff, and it can supply evidence links for a completion decision. But it should not be collapsed into either one. A task can be complete while some nonessential claims remain deliberately absent; a review handoff can be excellent while one performance claim still lacks adequate evidence.

Make unsupported claims useful instead of hidden

An unsupported claim is not always a failure. It may be a prompt to request a required environment, wait for an independent check, narrow the summary, or remove an unnecessary assertion.

Useful dispositions include:

• Disposition: Supported — Appropriate agent language: “The bounded check supports this stated behavior under these conditions.”
• Disposition: Indirect — Appropriate agent language: “The available evidence is suggestive but does not directly test the claim.”
• Disposition: Unverified — Appropriate agent language: “The agent produced a candidate result, but this claim needs a named check or review.”
• Disposition: Contradicted — Appropriate agent language: “The current observation conflicts with the claim; do not carry it into the summary.”
• Disposition: Not applicable — Appropriate agent language: “The task did not make this claim, so no evidence is implied.”

This vocabulary prevents the agent from replacing a missing result with confidence language. It also helps a reviewer focus only on the claims that materially affect the next decision.

Retain References While Minimizing Evidence Payloads

Evidence can contain sensitive task data, private paths, source snippets, credentials, personal information, or internal traces. A Claim-Coverage Map does not need to duplicate it. It can retain a minimum reference, a permitted result summary, the evidence boundary, and the disposition.

This is both a privacy discipline and an engineering discipline. More collected output is not automatically more reviewable. A small map with clear scope is often more useful than an enormous transcript that makes it impossible to find which result supports which statement.

A practical adoption path

1. Take the final summary from one reviewed coding-agent task.
2. Underline every sentence that asserts behavior, scope, compatibility, readiness, performance, or no action.
3. Create one map row for each material claim.
4. Link each row to the smallest permitted evidence reference and state the evidence boundary.
5. Change unsupported claims into limitations, next checks, or questions rather than leaving them as confident prose.
6. Ask an independent reviewer to challenge the map before promoting high-impact claims.

The first implementation can be a table in a pull request or task review. Over time, teams can decide which claim types deserve structured support in their own workflow.

A general lesson for coding agents

An agent’s summary is a bundle of claims, not a single result. The Claim-Coverage Map is a proposed educational framework for making the evidence, scope, and uncertainty of those claims reviewable. It does not describe a released Virillio Code capability.

Sources and further reading

• Coding-agents can replicate scientific machine learning papers — A preprint reports 12 runs across four scientific machine-learning papers using explicit claim targets, generated evidence and provenance, claim-to-output links, and validation checks before completion.
• Can Coding Agents Reproduce Findings in Computational Materials Science? — The AutoMat paper, arXiv v2 and marked COLM 2026, evaluates coding agents reconstructing scientific workflows and reports incomplete procedures, methodological deviations, and execution fragility among its failure modes.
• Coding Agents Don’t Know When to Act — The FixedBench preprint evaluates 200 human-verified reports where no code change is required, reinforcing that a no-change conclusion is itself a claim with a bounded basis.

Editorial disclosure

This article was substantially researched, drafted, and revised with AI through the Virillio Code editorial workflow. Virillio Code publishes the final text under its editorial byline, and the supporting primary sources are linked above.