How to Build a Voice Agent with LiveKit: Architecture and Pitfalls

LiveKit's agents framework makes the happy path genuinely easy: connect an agent to a room, wire up STT, an LLM, and TTS, and you have a working voice bot in an afternoon. The gap between that demo and something that survives real callers is where most of the actual engineering work lives.

The core architecture

A LiveKit voice agent is a participant in a room, same as a human caller would be. The room is the WebRTC session; the agent subscribes to the caller's audio track, and publishes its own synthesized audio track back. Inside the agent process, the pipeline is typically:

1. VAD (voice activity detection) — decides when the caller is speaking, so you're not running STT on silence.
2. STT (speech-to-text) — streaming transcription of the caller's audio.
3. LLM — takes the transcript (plus conversation state) and produces a response.
4. TTS (text-to-speech) — synthesizes the response back to audio, streamed to the room.

For phone calls specifically, LiveKit SIP sits in front of this: it bridges a PSTN call into a LiveKit room, so the same agent architecture handles both browser-based and phone-based callers with the same pipeline.

There's a second architecture worth knowing about: speech-to-speech models that skip the STT→LLM→TTS chain entirely and go audio-in, audio-out. LiveKit supports both patterns. The cascaded pipeline gives you more control and easier debugging (you can inspect the transcript); speech-to-speech can reduce latency by removing serialization steps, at the cost of losing that inspectable middle layer.

Pitfall #1: turn-detection tuning is not optional

The default VAD/endpointing settings will feel wrong for your actual use case in one direction or the other — either cutting callers off mid-sentence because it detected a pause as "done speaking," or leaving long dead air before the agent responds because it's waiting too long to be sure the caller finished. This isn't a one-time config; it needs tuning against real call recordings from your actual caller base, because pause patterns vary by language, by whether someone's reading from a script, and by call context (a caller reciting an account number pauses differently than one asking a question).

Pitfall #2: interruption handling is a UX requirement, not a nice-to-have

If a caller talks over the agent's response — which real callers do constantly, especially anyone used to human phone conversation — and your agent doesn't support barge-in, it will keep talking over them. This reads as broken, not just unpolished. Implementing this properly means the agent needs to detect the interruption via VAD on the caller's track while its own TTS is playing, stop synthesis immediately, and correctly truncate its own conversational turn so the LLM's next response accounts for being cut off — not resume as if it finished its sentence.

Pitfall #3: session state across reconnects

Real network conditions mean WebRTC sessions drop and reconnect, especially on mobile. If your agent's conversation state lives only in the room-session process memory, a reconnect silently resets context — the caller experiences this as the agent suddenly "forgetting" what they just said. State needs to be externalized (or at minimum, checkpointed) so a reconnect resumes the same conversation rather than starting fresh.

Pitfall #4: scaling agent workers

Each active call needs its own agent worker instance. Under load, you need a dispatch/scaling strategy that spins up workers ahead of demand (cold-start latency on a new worker is directly added to the caller's wait time), and you need monitoring on worker pool saturation specifically — not just overall server CPU, which can look fine while your worker pool is actually exhausted and new calls are queuing.

Pitfall #5: the latency budget is a budget, not a target for one component

Callers notice round-trip delay above roughly 500ms as a "the bot is slow" experience. That budget gets split across VAD/endpointing wait, STT streaming latency, LLM time-to-first-token, and TTS synthesis start — and it's consumed by network transport twice, once each direction. Optimizing your LLM call in isolation doesn't help if your endpointing timeout alone is eating 400ms waiting to be sure the caller stopped talking.

Where teams actually get stuck

Almost never on the initial pipeline wiring — the LiveKit docs and quickstarts cover that well. It's in the tuning that only real call volume reveals: endpointing thresholds, interruption edge cases, and the monitoring to catch worker pool exhaustion before callers do. If you're comparing LiveKit against a more managed option before committing to this level of pipeline ownership, see our LiveKit vs. Twilio comparison for the control-vs-operational-burden tradeoff.

If you're mid-build and stuck on one of these — especially the endpointing/interruption tuning, which is genuinely fiddly to get right — get in touch.