The engine's fallback chain has four layers, and for a while we thought it was bullet-proof. The four layers in order: (1) another provider on the same model family, (2) another family on the standard provider, (3) a cheaper model on the same family, (4) a web-search-augmented last response. The first three only fire when the previous layer explicitly errors out or violates SLO within 8 seconds. The fourth is a degraded mode that returns a "we could not answer" plus a web search snapshot.
It ran for two months without trouble. Then the first week of April it caused an 18-minute incident we did not immediately understand.
The non-obvious pitfall
The fallback chain can deadlock if every layer is in degraded mode at the same time. This is not hypothetical: on April 4, OpenAI Realtime API was in a regional outage (eu-west-1), Anthropic Claude was rate-limiting because of a big enterprise tenant under heavy load, and ElevenLabs was experiencing TTS slowness (5x normal latency). A fallback request went through all three layers, each rejected ("timeout", "rate-limited", "degraded"), and the fourth layer's web-search-augmented response was nonsense because the user's question was about an internal product the web knows nothing about.
Result: 18 minutes during which every request that flowed through the engine and landed on a European data centre got the same response: "Sorry, I cannot answer your question at this time." 4,217 requests, 412 tenants.
The circuit-breaker pattern
The fix has two parts. First: every fallback layer has its own circuit breaker. If a layer is above 25% error rate over the last 60 seconds, the circuit is open and the router SKIPS that layer entirely — straight to the next. The circuit goes half-open between 60-90 seconds (one trial request fires; if it succeeds the circuit closes) and fully closed after 90 seconds of successful trials.
Second: every fallback chain has a global hard timeout of 12 seconds. If total chain time exceeds 12 seconds, the request gets a degraded fast response (a templated "the engine is currently overloaded, please retry" message), and a chain.exceeded_max_time counter ticks. It is not pretty, but the request dies in a controlled way, not by accident.
The 12 seconds is not arbitrary. We measured the median per-request user-patience window at 28 seconds; the 95th percentile is 14 seconds. The 12-second cap ensures 95% of users would rather see an "overloaded" message than abandon the application.
What we still cannot do
Predict chain dynamics. Right now we are reactive — we wait for the circuit breaker to fire. A proactive system (provider-health prediction) would help a lot, but we have not yet measured whether provider-health signals reliably predict chain deadlock. That is next quarter's research topic. The signals we already collect — provider-side latency histograms, regional error rates, rate-limit-window utilisation — could feed a small predictor, but we have not yet found a model whose precision is high enough to act on (false-positive circuit-open events are themselves a source of degraded latency).
The post-incident dashboard
The April incident produced one durable artifact: a Grafana board that shows the live state of every fallback layer for every provider, per region. A green tile means circuit closed, yellow means half-open with trial firing, red means open. A second row shows the rolling 60-second error rate per layer. The on-call engineer's first action on any voice-or-chat alert is now to open this board. Before the incident, the same information existed across four separate dashboards in three different tools; the consolidation was as important as the circuit-breaker code itself.
Four months of operating with this pattern: zero chain-deadlock incidents. Three near-misses where two layers degraded simultaneously and the third absorbed the load. One false-positive circuit-open (a provider's transient 5xx spike that turned out to be a single bad pod, recovered in 40 seconds) that cost us roughly 200 EUR in unnecessary fallback to a more expensive model. We consider 200 EUR of false-positive cost a reasonable price for the prevented incident class.