Learn · Beginner
Model Routing and Cascades: Spending Frontier Money Only Where It Buys Something
Model routing is the practice of sending each request to the cheapest model capable of handling it, rather than sending everything to your best one. A cascade is the closely related pattern of trying a cheap model first and escalating only when its answer fails a check. Both work for the same unglamorous reason: in almost every real workload, the large majority of requests are easy, and paying frontier prices for them buys nothing at all.
The skew that makes it work
Look at the traffic hitting a deployed AI product and you will find a long tail of genuinely hard requests sitting on top of a mountain of trivial ones. Reformat this address. Extract the invoice number. Summarise this paragraph in one line. Decide whether the user is asking about billing or shipping. Pick which of four tools to call next.
A small model answers all of those exactly as well as a frontier model does, at somewhere between a tenth and a hundredth of the price. Sending them to your most capable endpoint is like taking a taxi to the corner shop: it works perfectly and the outcome is identical to walking. The savings are not clever - they are just the difference between two prices for the same result, multiplied by however much of your traffic is trivial.
The catch is the word "capable." You have to know in advance which requests are easy, and that is the entire technical problem.
Routing: decide first
A router looks at the incoming request and predicts, before running anything, which model will handle it acceptably. The prediction can be a small trained classifier, a nearest-neighbour lookup against labelled examples, or a heuristic over features like length and task type.
The research foundation here is RouteLLM from LMSYS, which trains routers on human preference data between a strong and a weak model, and Hybrid LLM from Microsoft, which routes on predicted quality drop and lets you tune the cost-quality dial explicitly. The consistent finding across this work is that a large fraction of queries can be served by the small model with no measurable quality loss - the exact fraction depending entirely on your traffic mix.
The most reliable way to build a router is to fit it to your own traffic rather than to a public benchmark. Score every model you have available against held-out tasks drawn from your own logs, then fit a policy from those measurements. Your workload's definition of "hard" is not a general one, and a router trained on somebody else's distribution will be confidently wrong about yours.
Cascades: try first, escalate if needed
A cascade inverts the order. Run the cheap model, check the answer, and escalate to the expensive model only if the check fails. FrugalGPT by Chen, Zaharia and Zou is the canonical treatment, chaining models in ascending order of cost with a scorer deciding whether to accept each answer or move up the chain.
Cascades trade latency for accuracy of the decision. A router might misjudge a hard request from its surface features alone; a cascade actually sees the cheap model's attempt before deciding. The cost is that escalated requests pay twice - once for the failed cheap attempt, once for the expensive retry - plus the added round trip.
The quality of the check is what makes or breaks a cascade. Options range from cheap to expensive: the model's own confidence, which is often poorly calibrated; a verifier that checks the answer against a rule or a schema; a small LLM-as-a-judge scoring the response; or, best of all, a real ground-truth test such as running the code the model wrote. Verification is generally easier than generation, which is the property that makes the whole pattern work.
Cascades also relate closely to speculative decoding, which applies the draft-then-verify idea at the level of individual tokens inside a single model rather than at the level of whole requests across models.
Where it actually bites
Agents are where routing pays for itself fastest, because agents burn tokens on a scale that chat never did. A single agent task cycles through planning, tool calls, reading results, revising and verifying - dozens of model calls where a chat interaction had one. Most of those intermediate calls are mechanical. Routing them to a cheap model while reserving the frontier model for the genuinely hard planning steps can change an agent product's unit economics outright.
Routing also composes with everything else in the cost toolkit. Prompt caching cuts the cost of repeated context, quantization cuts the cost of running your own weights, and distillation can produce the cheap model that the router points at in the first place. A router does not replace those; it decides which of them gets used.
The honest failure modes
Routing is not free money, and three things go wrong regularly.
The first is misrouting cost. Sending a hard request to a weak model costs you the failed attempt plus the retry plus the latency plus, if nothing catches it, a wrong answer shipped to a user. A router that is wrong ten percent of the time on hard traffic can easily be more expensive than no router.
The second is incomplete accounting. Token-price comparisons routinely omit retries, latency budgets, the compute spent training and periodically refitting the routing policy, and the engineering time to maintain it. Any savings claim that counts only input and output tokens is an upper bound, not a result.
The third is drift. Your traffic changes, models get deprecated and replaced, and a policy fitted six months ago is scoring today's requests against a model lineup that no longer exists. Routing policies need to be refitted, and treating one as a set-and-forget component is how a quiet quality regression starts.
The rule of thumb worth remembering: route when you can measure, cascade when you can verify, and be suspicious of any savings number that does not include what the mistakes cost.
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
RouteLLM: Learning to Route LLMs with Preference Data
Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Key questions
What is the difference between a router and a cascade?
Why does routing save so much money?
What is the main risk of routing?
Cite this
APA
Ground Truth. (2026, July 26). Model Routing and Cascades: Spending Frontier Money Only Where It Buys Something. Ground Truth. https://groundtruth.day/learn/model-routing-and-cascades.html
BibTeX
@misc{groundtruth:model-routing-and-cascades,
title = {Model Routing and Cascades: Spending Frontier Money Only Where It Buys Something},
author = {{Ground Truth}},
year = {2026},
month = {jul},
url = {https://groundtruth.day/learn/model-routing-and-cascades.html}
}