Real-time voice AI low-latency techniques that work

The techniques that actually move end-to-end latency live in stage-by-stage engineering: streaming partials, chunked synthesis, speculative decoding, and interruption handling that doesn't break turn-taking. This guide breaks down where latency accumulates across STT, LLM, and TTS, what numbers count as good versus poor, and the concrete configuration changes that get real production agents under one second.

Because these engineering choices also affect compute and staffing needs, it's worth planning your AI infrastructure budget before scaling a production pipeline.

Where voice AI latency actually accumulates

Voice AI latency splits three ways, and most engineering teams misjudge which slice is largest. Speech-to-text (STT) latency, LLM inference latency, and text-to-speech (TTS) streaming each eat into the budget, but the LLM call is rarely the dominant cost once you measure it. If you're building on mobile, the choice of TTS libraries for mobile apps can itself introduce latency worth accounting for.

One research pipeline reported hitting 0.94s average end-to-end latency, under the 1-second threshold (Toward Low-Latency End-to-End Voice Agents, 2025), a realistic floor, but only if each stage is budgeted separately: STT partial transcripts within 150-300ms, LLM time to first token (TTFT) as the swing variable, and TTS streaming starting before the full response text exists.

In our work with teams shipping voice agents on LiveKit and Pipecat pipelines, we've measured per-stage latency breakdowns across production configs and consistently found STT and TTS streaming account for as much cumulative delay as the LLM call, sometimes more, once WebRTC transport and jitter buffering are counted.

It's a pattern that comes up on Reddit voice-AI threads too, usually framed as 'why is my bot slow when the model is fast.' This piece maps where each millisecond goes and how barge-in interruption handling and voice activity detection (VAD) shift the tradeoffs within that budget.

What counts as Good, acceptable, or poor voice AI latency

Good voice-to-voice latency sits at 500-800ms end to end; acceptable runs 800ms-1.2s; anything past 1.5s reads as a phone hold to most callers (BitBytes / Voice AI Latency: Why Sub-500ms Response). Those numbers are the whole budget, split across four stages that each have their own threshold.

Voice activity detection (VAD) needs to flag end-of-speech within 20-50ms of the user stopping, or the agent feels like it's cutting people off before they've finished. STT should return a stable partial transcript within 150-300ms of that VAD trigger.

Time to first token (TTFT) from the LLM is the stage teams most often get wrong: 200-400ms is good, 400-700ms is acceptable, past 800ms the pause becomes audible even with a filler phrase masking it.

TTS needs to start streaming audio within 100-200ms of receiving the first token, not wait for a full sentence.

A useful production target: ≤800ms end to end, with sub-500ms as the leading-edge bar, and anything above 1,500ms degrading the experience (Twilio, 2025). Hitting these numbers consistently in production requires the kind of senior AI engineering team Netguru's AI Pod provides, rather than trial-and-error tuning.

These ranges match what engineers discuss on the LiveKit and Pipecat Discord and on r/LocalLLaMA-adjacent voice AI threads on Reddit, where self-hosted TTFT under 300ms is treated as the bar worth defending, not a stretch goal.

Our view is the total budget matters more than any single stage hitting a perfect number, since a fast LLM paired with slow TTS streaming still fails the interruption test.

Why published latency benchmarks mislead engineering teams

Deepgram's own streaming STT latency documentation reports transcription times under 300ms, and OpenAI's Realtime API benchmarks quote time to first token (TTFT) in the 300-500ms range. Both are true, measured on clean, wired lab connections. They rarely hold on a real cellular call.

Vendor numbers are usually collected server-to-server, skipping the client's actual network hop. Add real WebRTC transport over LTE or contested Wi-Fi, and jitter alone can add 100-200ms before a single audio frame reaches the STT engine (RTP jitter mechanisms; voice AI inference endpoint latency and infrastructure requirements).

We've seen teams benchmark a pipeline in a controlled office network, publish an internal target of 600ms, then watch production latency drift past 1.1 seconds once mobile users are in the mix. The gap isn't a bug. It's a different test.

Engineers comparing LiveKit and Pipecat setups on Reddit report the same pattern: self-hosted STT often beats managed APIs on paper but loses that edge within a few hundred milliseconds once real network variance and TURN relay hops enter the picture. TTFT figures from any provider are a floor, not a guarantee.

Our practice is to treat published TTFT and STT latency numbers as a starting hypothesis, then re-measure on the actual transport path, device mix, and geographic footprint the product will ship with, before setting an internal SLA.

Reducing STT latency: Streaming partials and provider choice

Speech-to-text (STT) latency drops fastest when you stop waiting for a finalized transcript and start acting on streaming partials. Deepgram, AssemblyAI, and Azure Speech all emit interim results every 20-50ms; the mistake we see most often in early voice agent builds is buffering those partials until an end-of-turn signal fires, which throws away the entire latency advantage of streaming in the first place.

Beyond low-latency voice agents, streaming STT also powers helpdesk transcription and sentiment analysis, where accuracy on interim results matters as much as speed.

The real lever is end-of-turn detection, not raw transcription speed. A VAD-only cutoff (silence for N milliseconds) is fast but wrong on any pause-heavy speaker; a semantic end-of-turn model that scores partial transcripts for completion probability adds 100-150ms of compute but cuts false interruptions sharply.

Deepgram's Nova-3 endpointing documentation describes this tradeoff directly, and it matches what teams report in build threads on r/LocalLLaMA and voice-AI-focused reddit communities: tuning endpointing thresholds moves perceived latency more than swapping STT vendors outright.

Provider fanout is still worth running: route the same audio stream to Deepgram and a second provider like ElevenLabs Scribe in parallel, scoring both on partial-confidence and time-to-first-partial. Teams doing this typically settle on Deepgram for phone-quality (8kHz) audio and a second vendor for studio-quality intake.

Running both within a single test use, on your own call recordings rather than vendor demo audio, is the only way to know which one wins for your traffic.

Reducing LLM inference latency: TTFT and speculative decoding

LLM inference latency is where most voice pipelines lose their sub-second budget, and the metric that predicts perceived responsiveness is time to first token (TTFT), not total generation time. A user hears silence during TTFT, then a steady stream after, so shaving 200ms off TTFT matters more than shaving 500ms off a five-second generation.

OpenAI's Realtime API addresses this directly by keeping the model in a persistent, low-latency session and streaming tokens as they generate rather than waiting for a full completion object. That's the configuration choice most teams building on it should default to over a standard chat completions call.

Pipecat and LiveKit agents both wire directly into this streaming mode rather than polling for a finished response.

Speculative decoding is the other lever, and it works on self-hosted stacks more than managed APIs. A small draft model proposes several tokens ahead; the target model verifies them in a single forward pass and accepts the ones that match. Speculative decoding reduces inter-token latency by roughly 2× in simulated benchmarks (PredGen, 2025).

This only pays off when you control the serving stack, vLLM or TensorRT-LLM with a compatible draft model. That's the core tradeoff engineering teams weigh on Reddit's r/LocalLLaMA threads: managed APIs give you TTFT consistency without infrastructure ownership, self-hosted stacks give you the headroom to tune speculative decoding but require GPU capacity planning most 50-500 person engineering orgs don't want to own.

In practice, we recommend managed inference until TTFT variance itself becomes the bottleneck, then move to speculative decoding on a self-hosted model within the same quarter you outgrow the API's rate limits.

Reducing TTS latency: Chunked synthesis and streaming output

Text-to-speech (TTS) streaming is what turns a five-second synthesis job into audio the user hears within 200-400ms of the LLM's first token. Instead of waiting for a full sentence or paragraph to render, the TTS engine synthesizes and emits audio in chunks, and the client starts playback on the first chunk while later ones are still being generated.

Chunked TTS synthesis has a tuning knob most teams get wrong on the first pass: chunk size. Split on every word and prosody breaks, producing flat or choppy speech. Wait for a full sentence and you reintroduce the latency you were trying to remove.

ElevenLabs' streaming API and Deepgram Aura both default to clause-level chunking, synthesizing at natural pause points (commas, conjunctions) rather than fixed token counts, which holds prosody steady without stalling first-audio time.

This kind of chunk-size tuning was part of the work Netguru did on CocoonWeaver, a voice-powered app for creative thinking and self-reflection: getting the TTS layer to feel conversational rather than laggy takes the same clause-level tuning discussed above.

That kind of fine-grained pipeline work is a common thread across Netguru's custom AI development services, where getting voice interactions to feel natural requires optimizing every stage of the pipeline.

Streaming output also has to coordinate with barge-in interruption handling. When voice activity detection (VAD) flags the user speaking over the agent, the pipeline needs a state machine that flushes the TTS buffer and cancels in-flight synthesis immediately, not after the current chunk finishes.

Pipecat and LiveKit both expose interrupt hooks for exactly this, and getting the flush timing wrong is a common failure mode discussed at length in voice-AI engineering threads on Reddit, where developers report audio bleed-through as the top barge-in complaint.

Architecting infrastructure for sub-1-second voice AI

Sub-1-second voice AI is a transport and placement problem before it's a model problem. Get WebRTC, edge inference deployment, and regional routing wrong and no amount of TTFT tuning on the STT or LLM side recovers the lost 150-300ms.

WebRTC and WebSocket transport solve different problems, and picking the wrong one is the most common infra mistake we see teams make within their first production build. WebRTC negotiates UDP by default, handles jitter buffering and packet loss concealment natively, and adapts to network conditions mid-call, it's the right choice for browser or mobile voice agents facing unpredictable client networks.

WebSocket transport is simpler to build and debug, but it rides on TCP: a single dropped packet stalls the entire audio stream until retransmission, which shows up as audible stutter rather than graceful degradation. LiveKit and Pipecat both default to WebRTC for client-facing legs precisely for this reason, reserving WebSocket for server-to-server hops where network conditions are controlled.

Edge inference deployment addresses the second lever: physical distance to the model. Running STT and TTS inference in the same region as the client, rather than routing every audio frame to a single centralized cluster, can cut round-trip latency substantially. Deepgram's own network optimization guidance recommends regional endpoint selection as a first-order latency lever, not an afterthought.

A voice agent serving users across three continents from one us-east cluster is paying 80-150ms of pure geography before any inference starts.

The practical architecture we land on: WebRTC for the client leg, regionally-deployed STT/LLM/TTS services behind it, and a routing layer that pins each session to its nearest healthy region rather than round-robining globally.

LiveKit vs pipecat: Framework latency tradeoffs

LiveKit and Pipecat sit at different layers of the voice stack, so a direct latency comparison only makes sense once you separate transport from orchestration. LiveKit provides the WebRTC media transport, SFU routing, and room infrastructure.

Pipecat is a pipeline framework that wires STT, LLM inference, and TTS streaming stages together, and it typically runs its audio I/O through LiveKit's WebRTC transport rather than competing against it.

Where the two actually compete is orchestration overhead: how many milliseconds each layer adds on top of raw STT latency, LLM inference latency, and TTS streaming before audio reaches the speaker.

In our pipeline builds, Pipecat's async processor graph adds measurable scheduling overhead per turn when VAD thresholds are left at defaults, most of it in queue hops between STT partials, LLM interim tokens, and TTS buffering.

Pipecat's own framework/orchestration overhead runs to tens of milliseconds per turn, small next to the STT, LLM, and TTS stages that actually drive overall latency (Forasoft, Pipecat vs LiveKit vs OpenAI, 2024).

Tightening the VAD endpoint threshold and moving to word-level streaming from Deepgram closes most of that gap without touching TTFT on the LLM call itself, which stays the larger budget item.

Neither framework replaces TTFT discipline. A comparison thread on Reddit converges on the same split we see in production: pick LiveKit when you need WebRTC infrastructure at scale, pick Pipecat when pipeline flexibility matters more, and expect to hand-tune barge-in interruption handling within either.

Handling barge-in and interruptions without breaking flow

Barge-in interruption handling only works when voice activity detection and end-of-turn detection are tuned together, not treated as one gate. VAD flags that the user is speaking; end-of-turn detection decides whether that speech is a real interruption or backchannel noise like "mm-hmm." Get the second part wrong and the agent stops talking every time a user coughs.

We model this as a small state machine: listening, speaking, barge-in-detected, and resuming. The transition from speaking to barge-in-detected should fire only after VAD confirms sustained voice energy, typically 200-300ms, filtered against the agent's own TTS output to avoid echo-triggered false interrupts.

Pipecat exposes this as a configurable interruption strategy; LiveKit's agents framework handles the equivalent through its turn-detector plugin, which layers semantic end-of-turn scoring on top of raw VAD.

Tuning the turn-taking model matters more than tuning VAD sensitivity alone. A model trained only on silence duration cuts off users mid-thought during pauses; one that scores semantic completion (via a lightweight classifier ahead of full LLM inference) waits appropriately within a second or two even through natural hesitation.

Deepgram's endpointing documentation recommends pairing utterance-end detection with VAD rather than relying on silence thresholds alone, since silence-only endpointing misfires on multilingual and accented speech.

Filler-phrase injection ("let me check that") during LLM inference masks TTFT but complicates barge-in state, since the agent must be interruptible mid-filler too. Teams building this from scratch on Reddit voice-AI threads consistently flag the interrupted-to-resuming transition, discarding stale TTS buffers, as the step most often skipped, and it's the one that causes audible overlap in production.

Measuring latency across your own call path

Measuring voice AI latency means logging a timestamp at every hop in the call path, not just the end-to-end response time. A single aggregate number hides which stage is actually slow, and that's the number you need to fix.

Instrument these boundaries, each with its own timestamp and a shared request ID so you can stitch a trace back together:

• Mic capture to VAD trigger
• VAD trigger to STT partial transcript
• STT final transcript to first LLM token (TTFT)
• TTFT to first TTS audio byte
• TTS first byte to speaker playback

In practice, this means wrapping each hop in a small logging call that writes {request_id, stage, timestamp_ms} to a structured log or a lightweight metrics store like Prometheus or a time-series table. Emit the delta between consecutive stages, not just raw timestamps, so dashboards show per-hop latency directly instead of requiring downstream math.

Pipecat and LiveKit both expose hooks at each of these boundaries, which is why teams building on them tend to get usable latency dashboards within a day rather than weeks of custom instrumentation. This kind of hop-by-hop instrumentation is part of a broader discipline worth getting right, see what really matters in testing AI systems before they reach production.

Transport choice changes what you're measuring. WebSocket transport gives you a single ordered stream and simpler timestamping, but it leaves jitter buffering entirely to your application code.

WebRTC handles jitter buffering and packet loss concealment at the transport layer, which smooths playback but can mask 40-80ms of real latency inside the buffer itself. A clean-looking dashboard number can hide a rougher perceived experience, so log buffer depth alongside your hop timestamps if you're on WebRTC.

Codec matters more than teams expect. Opus at 20ms frame size keeps jitter buffer depth small; PCM over WebSocket often forces a larger buffer to avoid audible gaps.

WebRTC jitter buffers add ~40-100ms of intentional delay to smooth packet variation in voice AI (Coval, 2024). We've seen teams on r/MachineLearning debate this exact tradeoff without settling it, which is a sign to measure your own path with your own logs rather than trust a general rule.

Self-hosted models vs managed APIs: The real latency tradeoff

Self-hosted models cut LLM inference latency by removing the network hop to a third-party API, but only if you have the ops capacity to keep GPUs warm and batched correctly. Managed APIs trade that latency floor for zero infrastructure burden.

Edge inference deployment, running STT and TTS on-device or at a regional PoP, shaves the last mile off the round trip that dominates perceived latency in poor-connectivity regions. We've seen teams on Reddit voice-AI threads report self-hosted Whisper variants beating managed STT on raw TTFT within controlled network conditions, though managed providers still win on multi-region reliability.

One managed provider, Fireworks, reports 3-12× lower latency versus a self-hosted vLLM setup in its own benchmarks (Fireworks AI, 2026). Most teams under 500 employees should default to managed until volume justifies the ops headcount. These latency-versus-ops tradeoffs are really an infrastructure strategy decision, one that plays out more broadly when treating infrastructure as a product rather than a one-off engineering cost.

FAQ: Real-time voice AI latency