Semantic routing: how agents pick tools without an LLM

The Towards AI article makes a straightforward architectural argument: routing is classification, not reasoning. If an agent has dozens of tools or SaaS integrations, sending every tool-selection decision through a large language model can add latency, token overhead, and another place for an API name to be hallucinated.

Definition: Semantic routing classifies a request by meaning before dispatch.

Example: A fast classifier can send a scheduling intent directly toward calendar and messaging tools.

Key takeaway: Keep expensive reasoning for ambiguous or genuinely complex work.

Business impact: A separate routing layer can make latency and cost easier to measure, but only if its errors and fallbacks are included in the evaluation.

The problem is not communication. It is unnecessary mediation.

Many agent stacks treat the main LLM as an interpreter for every step: inspect the available tools, infer which one fits, construct a call, and recover if the choice was wrong. That approach is flexible, but the context may contain a large tool catalog even when the user’s intent is routine.

The source article argues that this produces three operational pressures: larger prompts, slower decisions, and more opportunities for a model to select the wrong endpoint. Its proposed alternative is to move predictable classification outside the main reasoning loop. The agent still communicates with tools and other agents, but it does not need a full LLM pass to decide every destination.

The distinction is important. Semantic routing is not a claim that agents have become independent of language models. It is a claim about where a system should spend model inference: a small, fast classifier for dispatch; a larger model for interpretation, planning, and synthesis.

What does the routing flow look like?

A semantic router sits between an incoming request and the execution layer. It turns a natural-language or structured request into an intent, complexity estimate, category, or other routing signal, then chooses from an approved set of destinations.

A practical flow looks like this:

1. Receive the user or agent request.
2. Classify its intent and relevant constraints.
3. Match the result against tool, model, or agent capabilities.
4. Dispatch directly when confidence and policy allow it.
5. Escalate to an LLM or human-controlled fallback when the request is ambiguous.
6. Record the decision, outcome, latency, and cost for evaluation.

That last step is not optional in a serious deployment. A router can appear cheap while creating rework through misroutes. Teams need to measure the entire path, not only the classifier’s inference time.

What is “direct” agent communication here?

The headline’s “directly” is best understood as direct dispatch, not a new guarantee that agents exchange arbitrary messages without language models. A classifier can route a structured intent to a known tool or peer-agent capability without asking an LLM to narrate the handoff.

This resembles a network control plane more than a conversation. The request carries meaning or classification signals; a routing layer matches those signals to an available capability; and the selected agent or service receives a bounded task. A language model can still enter later when the task needs decomposition, negotiation, or open-ended reasoning.

That separation also clarifies how semantic routing relates to adjacent protocols:

The vLLM Semantic Router project presents this as signal-driven decision routing across a mixture of models, with signals and policies that can account for cost, latency, privacy, and safety. The IETF Internet-Draft for SIRP describes a protocol direction in which content-level classification can produce interoperable routing metadata before an inference backend is called. These are related efforts, not proof that one universal standard has already won.

Why is the cost argument persuasive—and incomplete?

A classifier can be much cheaper and faster than a large reasoning call, especially when the routing decision is repetitive. The Towards AI article cites approximately 100 milliseconds for a vector-classification path and discusses large reductions in token overhead. Those figures are claims from the source article’s examples, not an independent benchmark or a guarantee for every workload.

The complete economics are harder. A router adds infrastructure, training or threshold-tuning work, observability, and fallback paths. It can also create a second cost when a misroute causes a retry or sends a complex request to a model that cannot finish it. Cache behavior, data transfer, security checks, and the price of the chosen backend all belong in the measurement.

The right comparison is therefore not “classifier versus zero cost.” It is fixed-model or LLM-mediated routing versus a routed system with classifier inference, policy checks, execution, fallbacks, and rework included.

Where do standards fit?

Semantic routing is appearing at several layers at once. Application frameworks can classify tool intent. Gateways can route inference requests across model backends. Agent networks can advertise capabilities and resolve a task to a peer. The common idea is to make meaning and policy part of the dispatch decision instead of treating every destination as a static URL.

The SIRP draft describes content-level signals such as category, sensitivity, and complexity, with optional extensions for cost, urgency, domain specialization, and privacy. Those dimensions point to the real production challenge: routing is not only about finding the semantically closest destination. It is also about deciding what the system is allowed to send, how quickly it must respond, and how much uncertainty is acceptable.

Interoperability will matter if these layers are to compose. A tool protocol can expose capabilities, an agent protocol can publish an Agent Card, and a routing protocol can carry the decision. But shared vocabulary, identity, authorization, and failure semantics are still required before a collection of independent agents behaves like a dependable network.

What should teams evaluate first?

Start with a route inventory rather than a protocol rollout. List the recurring intents, available tools or peer agents, model paths, confidence thresholds, and fallback behavior. Then compare routed and LLM-mediated traffic using the same task-quality review.

Track at least:

• classifier latency and total end-to-end latency;
• route confidence and escalation rate;
• wrong-tool or wrong-agent rate;
• token use and backend cost;
• privacy and authorization violations;
• retries, rework, and human corrections;
• task success segmented by intent and complexity.

A hybrid router is usually the practical starting point: deterministic or vector-based dispatch for high-confidence intents, with an LLM fallback for uncertainty. That design preserves the cost and latency benefits of direct routing without pretending that every request can be reduced to a fixed label.

The larger shift: reasoning becomes a scarce resource

The architectural idea behind semantic routing is simple but consequential. An LLM should not be the default control plane for every operation merely because it can understand language. Classification, capability matching, authorization, and dispatch can be handled by smaller or deterministic components when the domain supports it.

That does not make agents less intelligent. It gives the reasoning model a narrower job—and gives the surrounding system clearer places to measure latency, cost, safety, and failure. As agent deployments grow from demos into networks of tools and specialized peers, that separation may become as important as the protocols that carry the messages themselves.

FAQ

What is semantic routing?

Semantic routing classifies a request by its meaning and constraints, then sends it to an appropriate tool, model, or agent. The main LLM is reserved for reasoning that cannot be handled by a fast classifier or policy layer.

Does it remove LLMs from agent systems?

No. It removes unnecessary LLM-mediated routing from predictable paths. LLMs remain useful for ambiguity, planning, decomposition, negotiation, and synthesis.

Is it the same as MCP or A2A?

No. MCP connects agents to tools and data, while A2A supports agent discovery and communication. Semantic routing is the decision layer that selects a destination based on intent and policy.

Is it a guaranteed way to reduce costs?

No. It can reduce routing overhead, but the result depends on classifier accuracy, workload mix, backend prices, cache behavior, fallbacks, and rework. Measure the complete system before claiming savings.