Multi-provider LLM routing — why and when

Three weeks. That is how much time passed between the sentence "we need a router" and the first per-request, four-dimensional, production provider selection. Today we route across five LLM providers in production: OpenAI, Anthropic, Azure OpenAI, Google Gemini, Mistral. This is a post about why the effort is worth it, and why staying on a single provider stops being a defensible choice once you take the cost/latency/SLO triad seriously.

Why one provider is not enough

Everyone starts with one provider. OpenAI, because it is the default. Six months in, something happens: a provider-level outage (two hours on February 14, GPT-4o returning 503s), a price change (Anthropic Claude 3.5 Sonnet dropped 40% in June), or a capability gap (Gemini's 2M context window when you need to read a long document). That is when you realise the single provider is also a single vendor lock, a single failure mode, and a single pricing contract.

The 4 dimensions the router decides on

The RouterPolicyService runs every request through a four-dimensional filter:

1. Capability — can the model do what is needed? Function calling, JSON mode, vision input, 100k+ context, Hungarian fluency. Stored as a matrix in eng_model_capabilities, refreshed per provider.
2. Cost — per-token price × expected token count. Lives in eng_pricing_table, refreshes every 6 hours (see the cost-attribution post). Per-tenant cost ceiling from eng_tenant_slo.
3. Latency — the provider's last 30-minute p95 latency for the given model. If the provider has not served in 30 minutes, pessimistic estimate (worst published).
4. Fallback chain — when the primary choice fails (rate limit, 5xx, timeout), which is the backup. Chain depth capped at 3.

The core routing code

async route(req: LlmRequest): Promise<RoutingDecision> {
  const candidates = await this.capabilityFilter(req);
  const priced = await this.costAnnotate(candidates, req);
  const eligible = priced.filter(c => c.cost <= req.tenantSlo.costCeiling);
  const ranked = eligible.sort((a, b) =>
    a.p95LatencyMs - b.p95LatencyMs
  );
  return {
    primary: ranked[0],
    fallbacks: ranked.slice(1, 4),
    reason: this.explainChoice(ranked[0], req),
  };
}

explainChoice returns a human-readable string we also pin to the span attribute. "picked anthropic/claude-3-5-sonnet because openai/gpt-4o p95=2400ms exceeds tenant SLO 1500ms." That field is the most valuable debuggability surface in the entire router.

The three weeks

Week 1: load the capability matrix, build the provider adapter interface. Five providers, each documents function calling differently. Week 2: pricing table and refresh worker. Two providers do not publish a price API; manual lookup. Week 3: telemetry (latency tracking), tenant SLO editor, and the fallback chain. The last three days are regression tests: 200 prompts × five providers, verifying that routing decisions are reproducible and deterministic under the same SLO.

Today's numbers

In production we route ~340,000 requests/day. Cost-ceiling violation rate: 0.4%. Fallback-chain activation rate: 2.1%. Median routing decision time: 8ms. Median end-to-end request latency: 720ms. In one year no tenant has filed a feature request to roll back to a single provider — six have asked us to add another. We are landing the sixth now (Cohere).

The lesson: multi-provider routing is not cost optimisation; it is optionality. Optionality against vendor lock, against outages, against pricing shifts. And the three weeks turn out to be one of the best ROI investments the platform shipped in 2026.