A coding agent does not solve a unit test. It enters an environment.
It inherits a repository snapshot, a dependency graph, a toolchain, a shell, a test runner, a task description, a set of permissions, and a definition of success. Change any of those and the same model behavior may produce a different patch, a different failure, or a different score.
That is why a patch-only evaluation is easy to run but difficult to trust.
A coding-agent evaluation is not a prompt paired with an expected diff. It is a reproducible environment with an explicit task contract and a verifier.
SWE-bench made this point concrete by evaluating agents on real software issues in real repositories. The task is not “write a function that matches this output.” It is “given this codebase and issue, make the change that resolves it.” Its evaluation harness uses containerized environments because the environment is part of what turns an agent action into an evaluable result.
The practical implication for agent builders is simple: if someone cannot reproduce the conditions under which an agent acted, they cannot tell whether they measured the agent, the machine, or a fortunate accident.
Why code-agent evaluations drift
Traditional code-generation benchmarks can often isolate the input and expected output. A coding agent cannot. It sees and acts on a living software system, which introduces at least five sources of drift.
- Source of drift: Repository state — What changes: Commit, generated artifacts, fixtures, or lockfiles — Why the score becomes ambiguous: The agent may be solving a different problem
- Source of drift: Environment setup — What changes: Runtime version, dependencies, operating system, or service stubs — Why the score becomes ambiguous: A valid patch can fail for unrelated setup reasons
- Source of drift: Task description — What changes: Issue wording, linked context, acceptance criteria — Why the score becomes ambiguous: Different prompts may imply different scopes
- Source of drift: Action surface — What changes: Shell commands, network access, test tools, write permissions — Why the score becomes ambiguous: Agents may have unequal ability to investigate or verify
- Source of drift: Verifier — What changes: Tests, timeouts, grading rules, and hidden checks — Why the score becomes ambiguous: Passing can mean different things from run to run
Treating these as background details makes results look cleaner, but it makes them less meaningful.
One practical approach is the five-invariant framework proposed below by Virillio Code Editorial. It is a design aid, not an industry standard; teams can adapt it to their own evaluation risks before comparing runs, models, prompts, tools, or orchestration strategies.
A proposed five-invariant evaluation framework
1. Snapshot: define the codebase the agent actually saw
The snapshot is more than a repository name. It includes the exact revision, relevant generated files, submodules if they exist, test fixtures, and any worktree state intentionally placed in scope.
An agent that is tested against a moving default branch is not being tested against a stable task. A dependency upgrade, a newly added test, or a refactor can silently change what “correct” means.
Record:
- repository identity and immutable revision,
- included and excluded files,
- initial working-tree state,
- issue or task version,
- known failing tests before the agent starts.
This also makes failure analysis honest. If an agent cannot reproduce a baseline failure because the repository changed, the problem may be the benchmark snapshot rather than the agent.
2. Setup: pin the execution environment, not just the package manifest
A package manifest is an instruction to build an environment; it is not the environment itself. Language runtime versions, system libraries, path behavior, operating-system differences, test sharding, service availability, and cache state can all change a coding workflow.
SWE-bench's use of containerized evaluation is instructive here. The container is not a magic guarantee of validity, but it establishes a repeatable boundary around the runtime where a patch will be exercised.
For a useful internal evaluation, capture:
- base image or operating-system identity,
- language and package-manager versions,
- dependency-install procedure and cache policy,
- available commands, compilers, and test runners,
- service mocks, fixtures, and network policy,
- resource limits and timeout budget.
The goal is not to make an environment impossibly sterile. It is to make the variation explicit enough that another run can recreate it.
3. Task contract: specify the work without smuggling in the answer
The task contract is the agreement between the evaluator and the agent. It should say what the agent is expected to accomplish, what evidence is available, and what constraints are non-negotiable.
An issue title alone is rarely enough. It may omit a user-visible acceptance criterion, a backwards-compatibility requirement, or a known limitation that a human engineer would learn from surrounding context. Conversely, attaching every historical discussion and accepted patch can turn the test into answer retrieval.
A good contract separates three things:
- Element: Goal — Purpose: The change the user or maintainer wants — Example: Fix pagination so a continuation preserves filters
- Element: Constraints — Purpose: What must not change — Example: Preserve the public wire format and existing error behavior
- Element: Evidence budget — Purpose: What the agent may inspect — Example: Repository, tests, documentation, and issue discussion; no future patch or hidden fix
This is not bureaucratic overhead. It prevents two common benchmark failures: under-specifying the user problem, and leaking the solution through the evaluation context.
4. Action policy: make the agent's operating authority comparable
An agent's result depends on what it can do. One agent may run tests, inspect logs, search the repository, install dependencies, create a local fixture, and edit files. Another may only write a diff in a text box. Comparing their scores without recording those differences compares two different products.
The action policy should name both available tools and denied actions:
- Read scope: which files, repositories, issue systems, and logs may be read?
- Write scope: which paths may be changed?
- Execution scope: which commands, services, and test targets may run?
- Network scope: whether package install, documentation lookup, or external search is available.
- Human boundary: which decisions require review or approval?
- Resource boundary: time, memory, cost, and retry limits.
Policy is especially important for agents that can make external changes. An evaluation should not reward an agent for completing a task by using authority that would be unavailable—or unacceptable—in the intended workflow.
5. Verifier: grade the requested behavior, not the patch shape
The verifier is the part most likely to turn a convincing-looking demo into a misleading benchmark.
Tests are essential, but test pass/fail is not always a complete measure of correctness. A patch may pass a narrow test while violating a public contract. It may cause a hidden regression. It may be functionally correct but exceed the allowed scope. Or the test suite may fail for unrelated infrastructure reasons.
Build the verifier as a layered proof:
- Proof layer: Baseline — Question: Does the unmodified snapshot reproduce the expected starting state?
- Proof layer: Targeted behavior — Question: Does the requested bug fix or feature work?
- Proof layer: Regression — Question: Do relevant existing tests still pass?
- Proof layer: Scope — Question: Did the agent stay within the permitted files and authority?
- Proof layer: Explanation — Question: Can a reviewer connect the agent's change to the observed outcome?
The evaluator should record raw failures, not only a scalar score. A timeout, flaky service, dependency conflict, and incorrect patch are different findings. Treating them as one red mark hides the next engineering decision.
The evaluation record should be inspectable
OpenAI's evaluation guidance frames an eval around test inputs and explicit testing criteria, then calls for analysis and iteration. That translates naturally to coding agents, but the input must include the environment as well as the task.
A useful evaluation record has three layers:
- Task record: snapshot, task contract, policy, and expected evidence.
- Execution record: commands, tool calls, changed files, time, retries, and resource use.
- Verification record: baseline state, test outputs, grading criteria, verdict, and caveats.
Do not confuse this with capturing unlimited model transcripts. The record should preserve the artifacts needed to reproduce and audit the result. It does not need to expose every internal intermediate or retain sensitive data indefinitely.
What to measure besides “resolved”
Resolution rate matters, but it is a lagging metric. Add measures that explain why a run did or did not succeed:
- Baseline validity: How often does the evaluation environment start in the expected state?
- Task understanding: Did the agent locate the relevant code and constraints?
- Verification discipline: Did it run the relevant checks before declaring success?
- Scope compliance: Did it modify only what the task and policy allowed?
- Recovery behavior: Did it distinguish a task failure from an environmental failure?
- Cost profile: How many turns, tool calls, and execution minutes did the task consume?
- Reproducibility: Can another run recreate the recorded conditions closely enough to explain differences? A pinned environment is reproducible; a nondeterministic model execution is not guaranteed to repeat exactly.
These metrics help avoid the wrong intervention. Low task understanding may call for better retrieval or repository maps. Poor verification discipline may call for a stronger completion policy. Baseline failures may require fixing the harness rather than changing the model.
Start small: build one trustworthy slice
Teams do not need a large public benchmark to adopt this approach. Start with five to ten recurring engineering tasks that matter to your own product:
- Freeze a repository snapshot.
- Containerize or otherwise record the setup.
- Write a concise task contract and allowed-action policy.
- Confirm the baseline behavior before the agent runs.
- Grade the result with targeted tests and an artifact review.
- Store failures by category, not just as pass or fail.
Only then compare prompts, models, toolsets, or agent architectures. Otherwise, a chart may be precise without being informative.
What we are learning building Virillio Code
Virillio Code's agent-runtime work is still in development, but the evaluation lesson transfers directly: durable systems need durable evidence. For coding agents, the evidence is not merely the final patch. It includes the initial state, the authority the agent had, what it observed, and how the outcome was verified.
The more autonomous the agent becomes, the more this matters. A believable result is one a human can replay, inspect, and explain.
Sources and further reading
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Repository-level coding-agent tasks require a real codebase, issue context, and execution environment rather than isolated code generation.
- SWE-bench repository and evaluation harness — Containerized harnesses and explicit task execution are central to reproducible coding-agent evaluation.
- Working with evals — Reliable evaluation uses task-specific inputs, explicit testing criteria, result analysis, and iteration.
Editorial disclosure
This article was substantially researched, drafted, and revised with AI through the Virillio Code editorial workflow. Virillio Code publishes the final text under its editorial byline, and the supporting primary sources are linked above.