Model Routing Is Simple. Until It Isn’t.

Definition: Model routing is the runtime decision about which model handles a request or execution step.
Example: An agent may begin with a cheaper model, encounter a harder tool-use path, and need to trade cost against latency and reliability as the task unfolds.
Key takeaway: Routing is not just classification; it is optimization across the whole serving system.
Business impact: A router that optimizes the wrong metric can increase the cost or slow down the experience it was meant to improve.

Model routing sounds like the obvious next step after model choice.

Send easy requests to a cheaper model. Reserve the expensive model for difficult work. Use a coding specialist for code and a multimodal model for images. Put a classifier in front, lower the bill, and move on.

That mental model is useful for a diagram. It is incomplete for production.

In “Model Routing Is Simple. Until It Isn’t.”, IBM Research describes what happened when its team built routing into agentic systems. The problem stopped looking like “which model is best for this task?” and started looking like a systems-optimization problem involving cost, quality, latency, caching, infrastructure, compliance, and reliability.

The distinction matters because the model selected at the beginning is not the system that ultimately produces the result. An agent may call a model repeatedly, reuse a large context, invoke tools, retry a failed action, wait on a busy endpoint, and cross a governance boundary. The right model on a pricing sheet can become the wrong choice after those interactions are included.

The cheapest token price can lose

IBM Research’s clearest example is a cost comparison that initially looks backwards.

Across 417 tasks in the AppWorld Test Challenge, using the same CodeAct agent, Claude Sonnet 4.6 cost $79 in total, or $0.19 per task. GPT-4.1 cost $155, or $0.37 per task—nearly twice as much. Yet GPT-4.1 had lower input and output token prices in the comparison, and Sonnet used roughly three times as many reasoning steps.

If the router looks only at the published price per token, GPT-4.1 should win. It did not in this workload.

The explanation was caching. Agent runs repeatedly carry forward large pieces of context. When those repeated inputs are cache hits, the effective cost can fall sharply. IBM Research reports that Sonnet’s lower cache-read price benefited disproportionately from this behavior, enough to outweigh both its higher base price and its longer execution trajectories.

That creates a more useful cost equation:

The lesson is not that list prices are useless. It is that they are not the objective. A router that optimizes a proxy can confidently send work down the more expensive path.

“Easy” and “hard” are unstable labels

Difficulty-based routing is the next intuitive strategy. Estimate how hard a request looks, then send the hard ones to a stronger model.

The problem is that difficulty is often hidden until the agent starts working.

“Summarize this contract” can be a short generation—or it can trigger retrieval, compliance checks, tool calls, document disambiguation, and several rounds of refinement. A technical request may look difficult while being routine for a specialist. A short request may be difficult because the missing context is discovered only after execution begins.

There is also no single dimension of difficulty. In production, a router may need to balance:

• Quality: Will the answer or action meet the task’s acceptance criteria?
• Cost: How much will the complete trajectory consume, not just the first call?
• Latency: How quickly does the user receive a successful result?
• Specialization: Is a particular model better suited to the modality or tool?
• Reliability: How often does the route finish without retries or escalation?
• Governance: Is the model approved for this data, region, and workload?

These signals can disagree. A smaller model may be cheap but require more retries. A larger model may take longer per token but finish in fewer steps. A model that is best on quality may be unavailable under the data-residency policy for a particular request.

The router is not choosing a model in isolation. It is choosing an operating point under constraints.

Latency is a serving problem too

Model size is an easy explanation for latency: larger models should be slower, smaller models should be faster. End-to-end latency is less tidy.

The router itself adds work. The endpoint may be running on different hardware. A cache may be warm or cold. The provider may be under load. Network and queueing behavior can dominate the difference between two model runtimes.

Routing granularity changes the equation again. Routing once per task adds little overhead and keeps the control plane simple. Routing at every agent step offers more opportunities to adapt, but every decision creates another latency event and another operational state to observe.

That does not make per-step routing wrong. It makes it a design choice. The system needs to know whether the potential quality or cost gain justifies the extra decisions.

IBM’s shift: from classification to optimization

IBM Research says its team changed the central question. Instead of asking which model is best for a task, its router optimizes across cost, quality, and latency simultaneously while remaining lightweight enough not to become a bottleneck.

The reported AppWorld results are presented as a cost-accuracy frontier rather than a single winner. Different router configurations provide different operating points:

• A latency-optimized configuration reached 84% accuracy for $93 and 83 seconds.
• Compared with running Opus alone, that configuration represented a 21% cost reduction and a 9% latency reduction, with a 4% accuracy drop.
• Another configuration pushed cost lower, trading against the other objectives.
• The optimization itself was reported at roughly 6 milliseconds and 2 kilobytes of memory per task.

The practical meaning is more important than any one configuration. A router does not need to discover the universal best model. It needs to expose useful tradeoffs so an operator can choose whether the current workload prioritizes cost, response time, or accuracy.

A conventional difficulty-based router can land near the same accuracy range while spending more because it explores a narrower part of that tradeoff space. The optimization framing makes the tradeoff explicit instead of hiding it behind a binary “easy versus hard” decision.

What a production routing policy should measure

The source does not publish a complete implementation recipe, and it says more technical details will follow. The current findings still imply a concrete evaluation discipline.

Measure completed work, not isolated calls

A model call is not the unit that matters when an agent can use tools, retry, or continue for many steps. Track cost and latency per successful task, then inspect the trajectory that produced it.

Include cache behavior

Record cache-hit and cache-miss rates by workload. The same model can have very different effective economics depending on how much context is reused and how the serving system bills that reuse.

Treat latency as end-to-end

Measure from the user’s request to a successful result. Include routing overhead, queueing, endpoint load, tool calls, retries, and any additional model decisions.

Evaluate governance before the route is needed

Enterprise constraints are not a final checklist after quality testing. Approved model lists, privacy rules, regional processing, and data-residency requirements can remove a route before the model is even considered. The router needs graceful alternatives when its ideal choice is disallowed.

Make the objective visible

A team should be able to state what its router is optimizing for on a given workload. “Cheapest model” is not a sufficient policy. “Minimum cost subject to 85% task success and a two-second p95 latency target” is closer to an operational objective, even if the actual thresholds differ.

The failure mode is optimization theater

A routing layer can look sophisticated while optimizing the wrong thing.

A classifier trained on prompt difficulty may be accurate at predicting a label but blind to cache economics. A benchmark-based selector may rank models correctly while ignoring provider load. A price table may identify the cheapest first call while missing the retries required to finish. A per-step policy may improve adaptation while making latency unpredictable.

These systems fail quietly because the router’s decision can look reasonable in isolation. The damage appears later as a higher total bill, a slower user experience, more failed tool calls, or a compliance exception.

This is why routing needs the same discipline as any other production control plane: observability, evaluation, fallback behavior, and a clear objective. The model is one component. The trajectory and serving environment are the rest.

A useful architecture for the problem

The safest mental model is not “router in front of an LLM.” It is a feedback loop around a workload:

1. Observe: capture task type, context shape, policy constraints, cache state, and current infrastructure conditions.
2. Select: choose an eligible model and routing granularity under the active objective.
3. Execute: run the model and tools while recording intermediate events.
4. Evaluate: check quality, success, latency, cost, retries, and governance outcomes.
5. Adapt: update the policy or route the next step when the execution reveals new information.

This does not mean every team needs an elaborate online optimizer. It means the system should be evaluated against the complete loop rather than credited for selecting a plausible model once.

For readers building agents, the distinction connects directly to how agent systems can fail silently even when an evaluation says “success”. A routing policy can produce a successful-looking answer while wasting budget, violating a policy boundary, or relying on a fragile trajectory. Routing metrics need to include the conditions around success, not just the final text.

The bigger lesson

Model routing is simple when the task is a single request, model quality is static, pricing is the whole cost, and the environment is predictable.

Production agents violate all four assumptions.

They reuse context. They call tools. They retry. They face variable infrastructure. They operate under data and compliance constraints. They may need a model decision at the start, in the middle, or not at all. Under those conditions, the “best model” is not a stable property of the prompt. It is a property of the whole system at a particular operating point.

IBM Research’s result is therefore less about one winning router than about a change in engineering vocabulary. Stop treating routing as a thin classifier that sits before inference. Treat it as optimization over a live workload, with real cost, quality, latency, and governance constraints.

The simple version is still useful for a whiteboard. The production version is where the work begins.

FAQ

Is model routing only about sending simple prompts to small models?

No. Difficulty is one possible signal, but production routing also has to consider cost, quality, latency, specialization, reliability, caching, infrastructure conditions, and governance. A task’s true difficulty may not be visible until an agent begins executing it.

Why did Claude Sonnet cost less than GPT-4.1 in IBM Research’s comparison?

In the reported 417-task AppWorld comparison, Sonnet’s lower cache-read pricing changed the effective cost of repeated context enough to outweigh its higher listed token prices and longer reasoning trajectories. The result demonstrates why a pricing-sheet comparison is incomplete.

Does the result prove one model is better than another?

No. The comparison is workload- and configuration-specific. It shows that model rankings can change when the full agent trajectory, caching, and serving conditions are included. The useful output is a set of tradeoff points, not a universal champion.

Should routing happen at every agent step?

Not automatically. Per-step routing can adapt to new information, but each decision adds overhead and complexity. Teams should compare task-level and step-level routing against their latency, cost, observability, and reliability requirements.

What should a team optimize first?

Start with cost and quality per completed task, then add end-to-end latency and governance constraints. Include cache behavior, retries, tool calls, and infrastructure state. A router is useful only when its objective matches the system’s real operating conditions.

The point is not to make routing sound mysterious. It is to measure the thing the system actually does: a sequence of model, cache, tool, infrastructure, and policy decisions that produces—or fails to produce—the result the user asked for.