Voice AI is no longer a novelty — but the gap between a working demo and a production-grade system is where most teams lose months and budget.
Speech recognition accuracy has crossed the threshold where it is genuinely useful in noisy, real-world conditions. Text-to-speech has become indistinguishable from human speech in many contexts. And large language models can now reason, summarise and respond in milliseconds. Yet organisations still struggle to wire these capabilities into a coherent, maintainable stack. This article lays out how to think about the architecture before writing a single line of code.
The Three-Layer Mental Model
Every voice AI system — whether it is a call-centre bot, a meeting transcription tool or an in-car assistant — maps onto three distinct layers:
- STT (Speech-to-Text): Converts raw audio into a text transcript. Latency and word error rate (WER) are the primary quality signals here.
- LLM (reasoning & response generation): Takes the transcript, applies context (system prompt, conversation history, knowledge base), and produces a text response.
- TTS (Text-to-Speech): Converts the LLM output back to audio and streams it to the user.
These layers appear simple in sequence. The engineering challenge is at the seams: how fast can you pass data between them, and how do you handle failure at each boundary?
Latency Budget — the Silent Killer
Human conversational tolerance for silence is roughly 1.5–2 seconds end-to-end. A typical breakdown:
- STT transcription: 200–400 ms (streaming) or 600–1200 ms (batch)
- LLM first token: 300–800 ms depending on model size and provider
- TTS audio start: 150–400 ms (streaming synthesis)
If you design each layer independently and concatenate their worst-case timings, you will consistently exceed user tolerance. Streaming is not optional — it is the architectural primitive that keeps you inside the budget.
Reference Stack: What Production Teams Are Actually Running
Practical insight: The most resilient teams treat STT, LLM and TTS as swappable components behind a thin adapter layer — not as permanent dependencies. This pays dividends the first time a provider raises prices or degrades quality.
A common production pattern looks like this:
- STT: Whisper (self-hosted for cost control) or a managed streaming API for real-time use cases. Deepgram and AssemblyAI are popular choices for low-latency pipelines.
- LLM: GPT-4o, Claude 3.5 or an open-weight model (Llama 3, Mistral) served via vLLM for throughput. Prompt caching and context window management are critical at scale.
- TTS: ElevenLabs or Cartesia for high-naturalness voices; OpenAI TTS for speed and simplicity. Self-hosted XTTS or Coqui for cost-sensitive deployments.
- Orchestration layer: A lightweight Python service (FastAPI or similar) that manages websocket connections, buffers audio chunks, and coordinates the pipeline state machine.
- Observability: Trace every utterance with a correlation ID. Log WER samples, LLM token counts and TTS render time. You cannot tune what you cannot measure.
Where Teams Get Into Trouble
- Neglecting turn-detection: Knowing when the user has finished speaking is harder than transcription itself. Voice Activity Detection (VAD) models like Silero are underrated infrastructure.
- Ignoring interruption handling: Users cut off AI responses. Your pipeline must be able to cancel in-flight TTS and LLM calls cleanly.
- Monolithic deployments: Bundling STT, LLM and TTS into a single service makes independent scaling impossible. Separate them from day one.
Choosing Between Managed APIs and Self-Hosted
The decision comes down to three variables: volume, latency requirements and data sensitivity. Managed APIs win on speed-to-market and maintenance burden. Self-hosted wins on unit economics above roughly 100,000 minutes per month and on compliance in regulated industries.
Many mature teams run a hybrid: managed APIs in development and low-traffic environments, self-hosted inference for high-volume production workloads.
Key takeaways
- Design the full pipeline around a strict latency budget from day one — streaming at every layer is non-negotiable.
- Abstract each AI layer behind an adapter interface so providers remain swappable.
- Invest early in VAD and interruption handling; they determine perceived conversational quality more than raw WER or voice naturalness.
- Separate STT, LLM and TTS into independently deployable services to enable targeted scaling and cost optimisation.
As voice AI moves from experimental to core infrastructure, the organisations that will have an advantage are those that treat speech pipelines with the same engineering rigour as any other distributed system — so the real question is: does your team have a clear latency budget and failure-mode playbook for every seam in your voice stack?