One million requests. That is not a big number on the industry scale — Google turns over multiples of that in a single day on one search property. For us it took three months, and over those three months enough data accumulated for us to replace our initial hypotheses with statistically defensible claims. This is a post about what we measured.
Median latency by domain
- Chat: 412ms
- Voice: 280ms
- Illustrate (image generation): 6.8 seconds
Voice looking faster than chat may surprise people. It should not — voice is a streaming pipeline where the first audio chunk leaves the server before the full response is ready (the user hears the response as it is being generated). Chat is measured on full response time; token-by-token streaming is enabled, but the median is taken at the full-response moment. If we measure chat first-token-latency, it lands at 180ms, comparable to voice.
Illustrate is long, and not easy to shorten. The DALL-E 3 + post-processing pipeline spends almost all of its time at the upstream provider. The only way that number moves is with a different provider (FLUX, SDXL-turbo), or if the user accepts a lower resolution.
Cost per request distribution
Cost-per-request is a long-tailed distribution. Median 0.0028 EUR. 95th percentile 0.034 EUR. 99th percentile 0.21 EUR. The top 1% accounts for 38% of monthly cost. These top-tail requests are almost always long multi-turn voice sessions, image generation series, or large context-window RAG calls from customers who query a 100K-token document.
The 95th-percentile users
4.3% of our tenant base accounts for 38% of total cost. They are not the bad ones — they are the most engaged. But our pricing model is linear and this skew shows up on the margin. Two options: (a) tier-based pricing (heavy-use plan), (b) per-request optimisation for those tenants. For now we have picked (b): these tenants get a dedicated router tuning where the fallback strategy is more aggressive (cheaper models preferred when SLO allows). Result: 17% cost reduction for those tenants with no change in user experience (their retention rate did not move).
The wrong-model rate
Wrong-model means: the router picked a model that, in retrospect (per eval-suite judgement), was NOT the best choice for this request. The current rate is 3.1%. We measure this by running a 1% shadow-request sample on the other candidate models for every response and comparing answer quality via an LLM-as-judge. If another model is consistently better on a given request class, it is a signal the router rule needs updating.
3.1% is not bad — the industry benchmark (from our own prior measurements) sits at 8-12%. But every tenth wrong-model is a potential customer complaint, so we walk through the router rules quarterly using this data and refresh the candidate lists.
What we measured and stays quiet
Chat session length distribution. Median is 4 turns, 95th percentile is 18 turns. This says: most users get what they want quickly and leave. The 18+ turn sessions are either support deep-dives (good) or the engine has gone in circles (bad). Distinguishing the two is the next engineering task.
What surprised us
Three things we did not predict before the data arrived. First: time-of-day distribution is flatter than the consumer-internet baseline. We assumed a strong Western European business-hours peak (9-18 UTC+1) but saw only a 2.3x peak-to-trough ratio; multi-region tenants and overnight batch workloads smooth the curve more than expected. Second: tool-call frequency tracks model strength inversely. The strongest model in the catalog calls tools 18% less often than the cheap fast model on the same prompts — it solves more in-context. Third: voice sessions exit cleanly via end-of-turn 84% of the time, but the remaining 16% are aborts mid-response, almost all from mobile users on poor connections. Voice retry-with-buffered-context is now a queued feature.
One final note: every number above is a snapshot. The engine, the providers, and the customers all change weekly. The Grafana board with these metrics is bookmarked by every engineer, and the weekly engineering meeting opens with a five-minute review of which numbers moved and why. None of this works without the discipline to look at the same numbers every week — the measurements are only useful if you compare them to themselves over time.