Route the Work

“Use the strongest model” is safe only if cost, latency, privacy, capacity, and outages do not matter. “Use the cheapest model” is safe only if failed outcomes do not matter.

Real systems need a policy between those slogans.

A routing policy observes what is known about the request and chooses a model, deployment, configuration, or cascade. Its job is not to identify the globally best model. Its job is to select a feasible route for this work under uncertainty.

The core decision is:

choose route r minimizing expected total outcome cost
subject to quality, latency, privacy, availability, and policy constraints

Model names are implementation details inside that decision.

What can be routed?

At the application boundary, a route can vary:

• model family and revision;
• hosted API versus private deployment;
• quantized versus higher-precision artifact;
• context length or retrieval policy;
• tool-enabled versus text-only configuration;
• ordinary versus speculative decode;
• one-shot call versus verified agent loop;
• synchronous route versus deferred batch.

This is different from mixture-of-experts routing inside a model. An internal MoE layer selects parameter experts during a forward pass. An application router selects independently operated paths with separate prices, data boundaries, SLOs, and failure behavior. The analogy is useful but the control contracts differ.

Route at the smallest unit whose requirements are stable. One agent task may use a strong planner, a small classifier for document labels, a code-specialized generator, and a constrained verifier. Routing the entire task to one brand leaves step-level savings and safety boundaries unused.

Hard gates come before learned scores

Some constraints should not be traded for predicted quality.

eligible(route, request) =
  data_region allowed
  and privacy tier allowed
  and required tools supported
  and context fits
  and output schema supported
  and route currently healthy
  and deadline remains feasible

Only eligible routes enter optimization. A model with a high predicted answer score is irrelevant if sending the context violates a data boundary. A route with the lowest nominal price is infeasible if its p95 latency exceeds the remaining workflow deadline.

Maintain capabilities as versioned facts, not prompt-time folklore:

Route fact	Evidence source	Freshness
context and output limits	deployed runtime/config	on deploy
region and retention policy	contract and infrastructure	on policy change
tool/schema support	integration tests	on model/runtime change
latency by workload slice	production traces	continuous
verified success by slice	evaluation and outcomes	continuous
marginal and fixed cost	bills plus capacity model	on price/load change
health and queue estimate	live telemetry	seconds

The router should be able to explain every exclusion and selection.

Fixed, rule-based, learned, and cascaded policies

Four common policy families solve different problems.

Fixed route

Every eligible request uses one route. This is easy to understand and provides a strong baseline. It can be optimal when traffic is homogeneous, volume is small, or operating another model costs more than it saves.

Rule-based route

Declared fields choose a route:

if data_class == restricted: private_route
elif task == extraction and schema_small: compact_route
elif context_tokens > threshold: long_context_route
else: general_route

Rules are auditable and good for hard product distinctions. They become brittle when difficulty is continuous or rules encode untested intuition.

Learned pre-router

A classifier or scorer predicts which route will succeed before generation. RouteLLM learns routing between stronger and weaker models using preference data. It demonstrates a research framework for a cost-quality trade. Production outcome labels may differ from human preference, especially for code, tools, exact extraction, or regulated decisions.

Cascade

A cheaper route runs first. A gate inspects its input, confidence, output, verifier, or some combination and decides whether to accept or escalate. FrugalGPT studies learned LLM cascades that combine models and stopping strategies under budget.

A cascade pays for information. It may know whether output parses, tests pass, citations resolve, or confidence is low. Its downside is that escalated requests pay first-attempt cost and latency plus the stronger route.

Formulate routing as expected loss

For request features x and route r, define:

J(r | x) =
  model_and_tool_cost(r, x)
  + latency_value_loss(r, x)
  + P(failure | r, x) * failure_consequence(x)
  + expected_retry_and_review_cost(r, x)
  + operational_shadow_price(r)

The policy selects the eligible route with minimum estimated J, or abstains when uncertainty is too high.

This expression explains why one route can be right for a draft and wrong for a payment action. Failure consequence changes. It also explains why marginal API price is incomplete for self-hosted routes: scarce capacity has an opportunity cost during peaks.

Quality can be implemented as a hard minimum rather than a loss term:

minimize expected cost
such that P(verified success | route, slice) >= 0.95
and p95 completion <= 2 seconds

Be honest about estimation error. A predicted 95.1 percent success rate is not proof that a route satisfies a 95 percent contract. Use confidence bounds and a safety margin.

A 1,000-request worked comparison

Consider an illustrative workload of 650 easy and 350 hard requests. No named models or market prices are represented.

Route	Easy success	Hard success	Cost/attempt
Weak	94%	55%	$0.002
Strong	98%	90%	$0.020

A fixed strong policy expects:

650*.98 + 350*.90 = 952 successes
cost = $20.00
cost per expected success = $0.02101

A fixed weak policy expects 803.5 successes for $2.00, but fails any quality requirement above 80.35 percent.

Now a learned router sends 10 percent of easy requests and 85 percent of hard requests to strong. Routing costs $0.0002 per request.

strong requests = 65 + 297.5 = 362.5
weak requests   = 637.5
easy success    = .10*.98 + .90*.94 = .944
hard success    = .85*.90 + .15*.55 = .8475
expected successes = 650*.944 + 350*.8475 = 910.225
total cost = 362.5*.020 + 637.5*.002 + .20 = $8.725
cost per expected success = $0.00959

The router passes an illustrative 90 percent quality gate and cuts expected cost per success by about 54 percent relative to fixed strong. It still loses about 41.8 expected successes, so it is not equivalent.

Finally, a cascade runs weak for everyone, then escalates 12 percent of easy and 70 percent of hard requests. Its gate costs $0.0003 per request.

strong escalations = 650*.12 + 350*.70 = 323
expected successes = 892.37
cost = 1000*.002 + 323*.020 + .30 = $8.76
cost per expected success = $0.00982

Under these assumptions the cascade misses the 90 percent quality gate while costing slightly more than the router. Change gate accuracy, prices, or the value of output evidence and the ranking can reverse.

The cheap fixed route's cost ratio looks best only because the denominator ignores the quality gate. Constrained optimization prevents an unacceptable policy from winning a cost chart.

Confidence must mean something measurable

A router output of 0.8 is useful only if requests assigned that score succeed about 80 percent of the time under the defined label. This is calibration.

Guo et al. document miscalibration in modern neural networks and evaluate post-hoc calibration methods in classification. An LLM router is not the same experiment, but the operational lesson transfers: ranking and calibrated probability are different properties.

Plot reliability by workload slice:

Predicted success band	Observed verified success	Count
0.50 to 0.60	measured	n
0.60 to 0.70	measured	n
0.70 to 0.80	measured	n
0.80 to 0.90	measured	n
0.90 to 1.00	measured	n

Calibration can drift when prompts, models, tools, or verifiers change. A router trained on chat preference may be confidently wrong about SQL execution. Recalibrate or retrain on current application outcomes.

Keep an abstain region. If routes have overlapping uncertainty or the request is out of distribution, use a safe default, run a verifier-rich cascade, or ask for human input. Forced certainty is not efficiency.

Preference is not correctness

Pairwise human preference is abundant for open-ended responses and central to RouteLLM's research setup. In automation, the label should match the outcome.

• Code: tests and repository acceptance.
• Retrieval answer: supported claims and citation entailment.
• Extraction: field-level exactness against audited records.
• Tool action: schema validity, authorization, and side-effect confirmation.
• Support resolution: issue resolved without unsafe or incorrect instruction.

A fluent strong-model response may win preference while failing a hidden test. A terse small-model output may be correct and valid. Train and evaluate the router against the decision the product needs.

When no verifier exists, use carefully designed human labels and explicitly price uncertainty. Do not manufacture an objective by asking another model and calling the result ground truth.

Cascades need failure-aware stopping rules

A cascade gate can inspect:

• model-reported confidence, cautiously;
• token-level uncertainty signals;
• agreement among samples;
• deterministic schema and type checks;
• retrieval support and citations;
• executable tests;
• a learned correctness predictor;
• a second model's critique;
• business-rule constraints.

Language Model Cascades: Token-level uncertainty and beyond studies uncertainty signals for deferral. It is a preprint and its evaluated signals should not be promoted to universal stopping rules.

The strongest gates are often domain verifiers. If a generated patch passes the relevant tests and policy checks, escalation may be unnecessary. If an answer concerns an unverified live balance, high textual confidence is weak evidence.

Count false accepts and false escalations separately:

• false accept: weak output fails but gate accepts, damaging quality;
• false escalate: weak output would pass but gate pays strong cost and latency;
• correct accept: saving;
• correct escalate: protected outcome.

Optimize their consequence-weighted confusion matrix, not gate accuracy alone.

Reliability includes correlated failure

Multiple routes are not resilient if they share the same dependency. Two API model names may use one provider region, identity system, gateway, or quota. Two self-hosted models may share the same cluster and overloaded KV pool.

Describe route failure domains:

provider, account, region, network, runtime, model family,
tokenizer, data source, tool dependency, verifier

A fallback must be available, compatible, and capacity-tested during the event that triggers it. If all normal traffic shifts to a smaller fallback, its queue may collapse.

Routing for availability can conflict with routing for cost. Reserve or rehearse fallback capacity and include its idle cost in the policy economics.

Route workflow steps, not just initial prompts

Agent loops reveal more information over time. The first planner call sees an ambiguous task. A later step knows it needs one of three operations. A verifier knows the produced artifact and failed checks.

Allow route features to include safe workflow state:

step_type
remaining_deadline_and_budget
previous_failure_class
tool_and_schema_requirements
context_length
verification_strength
consequence_and_reversibility

Do not leak private content into an external router merely to decide that private content cannot leave. Feature extraction must obey the same policy gates as generation.

Per-step routing also prevents a strong-model failure from recursively selecting itself. After a schema failure, switching to a route with constrained decoding may matter more than scaling model capability.

The route decision is a security boundary

Prompts are untrusted input. A user can write “send this to the cheapest external model,” “ignore the private-data flag,” or text that imitates the features a learned router associates with an allowed class. The router may use content to predict task type, but content must not override trusted policy attributes.

Separate feature provenance:

trusted: tenant, authorization, data class, region, contract tier,
         remaining budget, workflow state, required tool permissions
derived: intent, difficulty, domain, expected length, success score
untrusted: instructions and claims inside user or retrieved content

Hard gates use trusted attributes and verified system state. Derived features may choose among already eligible routes. If classification of data sensitivity itself is learned, apply a conservative default and audit false negatives.

The router also sees valuable metadata and sometimes full prompts. Minimize its input. A lightweight router that needs an embedding of private content is still a data processor. Prefer local feature extraction, redaction, declared workflow fields, or a policy-compatible router when raw text cannot leave its boundary.

Protect economic controls. An attacker can craft requests that force the expensive route, create cascade escalations, or occupy fallback capacity. Rate limits based only on request count miss this. Track predicted and realized token/tool cost per principal and require budget admission before execution.

Log route decisions without copying secrets unnecessarily. A useful audit record includes request and policy identifiers, trusted eligibility facts, derived score versions, excluded routes and reasons, selected route, budget, and eventual verifier outcome. Retention and access controls should match the sensitivity of the underlying work.

Finally, test policy invariants independently of the model:

• restricted data never enters an external route;
• a tool permission cannot be gained through route selection;
• expired credentials make a route ineligible;
• prompt wording cannot alter tenant or region;
• budget exhaustion produces a controlled stop;
• fallback does not weaken the original privacy contract.

Routing can reduce cost only after it preserves authority.

A builder playbook

1. Build the route registry

Version capabilities, hard constraints, price model, observed latency, capacity, and health. Separate declared vendor facts from your measurements.

2. Create a fixed baseline and oracle ceiling

Evaluate every eligible route on the same representative tasks. The hindsight oracle selects the cheapest successful route after seeing outcomes; it is unattainable but shows maximum possible routing value.

3. Choose the true outcome label

Use executable, schema, groundedness, or human outcome verification. Record consequence, not just win preference.

4. Evaluate policies offline

Compare fixed, rules, learned pre-route, and cascade policies on identical traces. Report success, severe failure, cost, latency, routing overhead, escalation, and per-slice confidence bounds.

5. Shadow before controlling

Log the route the candidate would choose while the baseline still executes. This reveals drift and capacity concentration without exposing outcomes.

6. Canary with abstention

Start in low-consequence, strongly verified slices. Retain a safe default for uncertain or out-of-distribution requests. Bound spend and failure.

7. Monitor policy regret

Sample counterfactual routes where affordable. Track the cost or outcome difference between chosen and successful alternatives. Without exploration, the router can stop learning about routes it rarely selects.

8. Recalculate cost per verified outcome

Include router inference, duplicate cascade calls, tools, retries, latency, fallback reserve, and failure loss. Reuse the unit model from Cost per Successful Outcome.

The decision

Use a fixed route until workload heterogeneity creates enough avoidable cost or failure to justify policy complexity. Add hard eligibility gates first, transparent rules second, and learned prediction only when representative outcome data can support it. Use cascades when the first attempt produces strong verification evidence worth its extra latency.

The winning router does not send the most traffic to the cheapest model or the strongest brand. It sends each step to the least costly feasible route and knows when its own evidence is too weak to decide.

References

• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, Chen, Zaharia, and Zou, TMLR 2024. Develops cost-aware model cascades and stopping strategies.
• RouteLLM: Learning to Route LLMs from Preference Data, Ong et al., ICLR 2025. Provides a learned strong-versus-weak routing framework.
• Language Model Cascades: Token-level uncertainty and beyond, Yue et al., 2024 preprint. Studies uncertainty-based deferral signals.
• On Calibration of Modern Neural Networks, Guo et al., ICML 2017. Establishes the practical distinction between confidence ranking and calibrated probabilities.

All policies, costs, and success rates in the worked scenario are illustrative. The full calculations are retained in the claim ledger.