The router exists to answer one question: which model handles this request? Over the last eleven months we have shipped three different versions of that rule, and all three were wrong. The fourth is not wrong, but it is not perfect either — this is a post about why.
v1 — pick_cheapest_first
The first rule was simple: pick the cheapest model that can in principle handle the task. "In principle" meant a static capability matrix read from a Postgres table (eng_model_capabilities). We liked the logic: cheapness first, quality later, fallback chain catches the rest. Two weeks in, the numbers told a different story. 28% of users were suffering because of fallback — the cheap model botched the request, retried on the next tier, and the response that finally went back had a median latency of 2.4 seconds against a 600ms SLO. The cleanup was costing more than the bargain saved. The retry rate alone was 18% on our two heaviest tenants.
The lesson: cost optimisation is a bad default because fallback activation is more expensive than picking the right model from the start. Not in token price — in the request's final latency, in SRE on-call time, and in user patience.
v2 — pick_fastest_first
The second version was the inverse. Take the fastest model from whichever provider is online; do not worry about price, the user's patience is more expensive than tokens. It worked — median latency dropped to 340ms. The bill doubled inside three weeks. Finance was not happy. The CFO's note ("team, what happened in the first week of April?") landed in Slack #engineering and travelled up to the planning committee. "Fastest" almost always meant gpt-4o, even when the request was an 80-token classification a nano-model would handle for one-third the price. The "too expensive because you have not measured against what" reasoning solidified inside us during this phase.
v3 — first_to_meet_slo
The third version is built around a per-tenant SLO row (eng_tenant_slo) storing three fields: latency_p95_ms, cost_ceiling_per_request_eur, quality_floor. The router walks the candidate model list (capability filter first) and picks the FIRST one that satisfies all three constraints across the last 30-minute telemetry window. This worked — financially, in latency, and in support-ticket counts. For eight months.
It was wrong twice. The first was a rare combination: one tenant's SLO was so tight (latency_p95 < 200ms AND cost < 0.001 EUR) that no candidate satisfied it. The router fell out with a NoEligibleModel exception, surfaced to callers as a 500. The fix: if no model satisfies every constraint, pick the one that violates the fewest and emit an slo.degraded counter. The second failure was subtler. The telemetry window is 30 minutes, but freshly activated tenants start with an empty window. v3 fell back to the last known global median as a proxy, which made a slow model look unfairly fast. The fix: during warm-up, use a pessimistic estimate — the worst published latency for the candidate.
The metric that finally settled the argument
Not latency, not cost, not routing accuracy. One number: resolved-request-cost — the amount billed to the tenant divided by the count of requests handled successfully in a single round (no retry, no fallback). That is the number, and that number is now the router rule's sole objective. v3 improved it by 32% over v2 and 47% over v1. v4 (in development) watches this number's per-tenant drift daily and auto-recalibrates the candidate list when drift exceeds threshold — no human in the loop for the routine recalibration, only for the policy changes.
The lesson is not that routers are hard. The lesson is that choosing the objective function is much harder than writing the algorithm.