Model Routing Belongs in the Agent Policy Layer

Routing an agent task to a different model is not a mere optimization.

It is a decision about how the system will reason, use tools, preserve a structured contract, handle uncertainty, and respond when work goes wrong. It can change the time a user waits and the resources a platform consumes. It can also change the quality floor the user experiences.

That means model routing belongs in the policy layer.

The common framing is simple: send easy work to a cheaper model and hard work to a stronger one. The problem is that “easy” is not a stable property of a prompt. A task can look simple while requiring an exact output format, a narrow authority boundary, a nuanced source comparison, a reliable tool call, or a graceful response to ambiguity.

A route is acceptable only when the selected configuration can satisfy the task's required behavioral floor under the current conditions.

OpenAI's model guidance recommends comparing configurations on representative tasks and measuring task success, completeness, required evidence, tokens, latency, and cost. The order matters. Resource use is one dimension of a successful task, not the definition of success. The current model catalog also makes model choice an explicit operational decision by documenting families with different capability and performance profiles; it does not establish a universally best route.

The RouteLLM paper explores the technical side of this tradeoff by studying dynamic selection between models with different strengths and costs. That research makes routing look attractive—and it can be. But a production agent still needs a human-understandable answer to a more basic question: what is this route allowed to compromise, and what must it never compromise?

Routing changes more than answer quality

A model or configuration can differ across many properties that matter to an agent:

• Property: Task reasoning — Why it matters at runtime: A route may change whether the agent can reliably decompose or validate a complex task
• Property: Tool interaction — Why it matters at runtime: It may affect tool choice, structured arguments, error interpretation, and recovery behavior
• Property: Evidence handling — Why it matters at runtime: It may change whether the final response preserves required sources, caveats, and uncertainty
• Property: Output contract — Why it matters at runtime: It may affect schema compliance, content completeness, language, or format expectations
• Property: Latency and availability — Why it matters at runtime: It may alter how long the user waits, whether a task can run in the background, or when it should be deferred
• Property: Cost and capacity — Why it matters at runtime: It may affect per-task effort and the fairness of shared platform resources
• Property: Safety and authority fidelity — Why it matters at runtime: It may affect how consistently the agent respects approval, scope, and stopping boundaries

No single routing score captures all of these. A predicted probability of a good answer is not a permission to violate an output, evidence, or authority requirement.

The first rule of routing is therefore negative: never route below a task's non-negotiable floor.

The Model Route Card

The Model Route Card is a proposed record for a runtime routing decision. It is intentionally compact. It does not prescribe a particular router, vendor, or algorithm. It gives an operator, reviewer, and future evaluator enough context to understand why a task was placed on a particular eligible path.

• Field: Task class and stakes — Question it answers: What kind of work is this, and what would a poor outcome affect? — What it should make explicit: Interactive support, coding change, research synthesis, structured extraction, low-risk classification, or another defined product class
• Field: Capability floor — Question it answers: What must the selected route be able to do? — What it should make explicit: Required reasoning, tool use, evidence, language, structured-output, or recovery behaviors validated for the task class
• Field: Quality constraints — Question it answers: What cannot be traded away? — What it should make explicit: Acceptance criteria, required citations, uncertainty handling, confirmation boundaries, and terminal-state requirements
• Field: Service objective — Question it answers: What resource objective applies after the floor is met? — What it should make explicit: Latency target, token or effort budget, queue class, capacity availability, and user-selected preference where applicable
• Field: Integration compatibility — Question it answers: Can this configuration operate with the required tools and state? — What it should make explicit: Tool availability, input types, output contract, context requirements, and known limitations
• Field: Fallback and escalation — Question it answers: What happens if the preferred route is unavailable, uncertain, or fails its gate? — What it should make explicit: Escalate, defer, request clarification, use a bounded alternative, or return an honest failure state
• Field: Review and expiry — Question it answers: When must this decision be reevaluated? — What it should make explicit: Model or configuration change, new task class, behavior regression, policy revision, or evidence staleness

The card is a policy record, not a prompt template. A platform may store it as configuration, an evaluation artifact, or a runtime decision trace. The key is that the choice becomes inspectable instead of being hidden behind a label such as “fast mode” or “smart mode.”

Start with a capability floor

The most important field is the capability floor.

A capability floor is not a claim that one model is objectively better than another. It is a requirement for a particular task class. For example:

• A low-stakes classification task may require a validated output label and a clear abstention path.
• A research synthesis may require source preservation, conflict disclosure, and evidence-quality checks.
• A coding task may require a specific verification boundary and accurate reporting of changes versus proposals.
• A task with consequential effects may require the same authority and confirmation behavior regardless of the selected model.

If a candidate route cannot meet the floor under evaluation, it is not an eligible route. That eliminates the most dangerous version of optimization: saving resources by silently removing a behavior the product has promised.

This makes routing compatible with behavioral-compatibility testing. An agent team can reuse the same kind of invariant suite that governs a model migration, but apply it to each task class and routing candidate. Routing then becomes a constrained choice among eligible options rather than an unbounded prediction contest.

Complexity is a weak routing signal

Many routers begin with apparent complexity: prompt length, certain keywords, or a model's guess that a task is hard. Those signals can be useful, but they are not enough.

A short request may carry a strict output contract. A simple-looking question may require a high-authority source. A routine tool call may sit behind an important confirmation boundary. A long task may be low-risk if it is purely exploratory and its outputs remain reviewable.

The route should consider the task's constraints, not just its shape.

That is why the Route Card distinguishes task class, capability floor, and service objective. The system can favor lower resource use only after it knows the task remains in a class whose required behaviors are supported.

Fallbacks must preserve truth

Fallback behavior is where a routing policy becomes visible to users.

If a preferred route is unavailable, a platform has several legitimate options:

• wait for the eligible route when the task's quality floor is non-negotiable;
• ask the user to choose between more delay and a narrower, clearly described outcome;
• route to a validated alternative that satisfies the same floor;
• split a task so a lower-cost route handles only a bounded preparatory stage; or
• return an honest blocked or deferred state.

What it should not do is silently substitute a route that cannot keep the same promise.

An agent can be helpful while remaining honest about its limits. It can produce a draft rather than a verified answer, a plan rather than a completed change, or a request for clarification rather than an invented resolution. The fallback field makes those distinctions part of the route design rather than an afterthought.

Use evaluation evidence, not anecdote

Routing requires evidence for the specific workloads it will serve. A single benchmark result is not enough, because agent quality depends on the interaction among prompt, tools, context, state, task boundary, and user expectation.

Representative evaluation cases should check:

• task completion against the defined success condition;
• required source, caveat, and output-contract preservation;
• tool selection and structured-result handling;
• authority, confirmation, and scope behavior;
• recovery from an unavailable or ambiguous dependency;
• latency and resource use; and
• how the route behaves at its declared fallback boundary.

The point is not to endlessly benchmark every request. It is to avoid using a routing heuristic as a substitute for a validated product contract.

RouteLLM shows, in its evaluated settings, that dynamic routing can make meaningful cost-quality tradeoffs. A production system should treat that result as a reason to evaluate routing carefully, not as a license to assume a universal threshold will transfer to every agent workflow.

Measure the policy, not just the router

Useful routing metrics are tied to the card:

• Floor-pass rate: What share of selected routes meet the required behavior and output criteria for their task class?
• Fallback honesty: When a preferred route is unavailable, how often does the system explain its narrowed, deferred, or escalated state accurately?
• Route rationale coverage: What share of consequential route decisions can be reconstructed from a task class, floor, objective, and review state?
• Regression containment: When a model or configuration changes, how quickly are affected routes re-evaluated or paused?
• Resource efficiency after quality: How much latency, effort, or cost is reduced without increasing floor violations?
• Fairness under load: Does routing improve shared capacity use without moving lower-quality outcomes onto a particular user or task class?

These measures preserve the intended tradeoff. Cost reduction counts only when the agent continues to meet the behavior users were promised.

Common anti-patterns

Routing to the cheapest option by default

Cost-sensitive work may be an appropriate class, but “cheapest” is not a capability floor.

Hiding quality changes behind a mode label

If a user sees a materially different evidence, latency, tool, or completion posture, the product should make that route's limits understandable.

Using only prompt length as a proxy for difficulty

Task constraints and risk matter more than the number of tokens in the request.

Treating a fallback as an invisible implementation detail

A fallback can change the product promise. It needs a declared behavior and an honest terminal state.

Never expiring route evidence

Model behavior, tools, prices, constraints, and task distributions change. A route card needs a review trigger, not permanent confidence.

A practical adoption path

1. Identify one task class that already has clear success criteria and a measurable resource tradeoff.
2. Define its capability floor and non-negotiable quality constraints.
3. Evaluate a small set of eligible configurations on representative cases.
4. Create a Route Card that records objective, compatibility, fallback, and expiry.
5. Make only bounded resource tradeoffs within the eligible set.
6. Observe failure and fallback behavior before widening the policy.
7. Re-evaluate when models, tools, task distribution, or user expectations change.

The durable rule is simple: select models by policy-constrained suitability, not by an opaque prediction of which request deserves fewer resources.

What we are learning building Virillio Code

Virillio Code is still in development. Its high-level work on bounded execution and scoped tools suggests a transferable lesson: decisions that alter an agent's effective capability should be explicit enough to review. Runtime routing is one of those decisions.

Sources and further reading

• OpenAI Model Guidance — Teams should compare configurations on representative tasks across success, completeness, evidence, tokens, latency, and cost.
• OpenAI Models — Models have distinct current capability, performance, and cost characteristics, making selection an operational decision.
• RouteLLM: Learning to Route LLMs with Preference Data — An original paper studies dynamic routing between stronger and weaker models to balance response quality and cost.

Editorial disclosure

This article was substantially researched, drafted, and revised with AI through the Virillio Code editorial workflow. Virillio Code publishes the final text under its editorial byline, and the supporting primary sources are linked above.