Why Your Voice AI Sounds Slow (It's Not the Model - It's the Pipes)
Why Your Voice AI Sounds Slow (It's Not the Model - It's the Pipes)
Table of Contents
"We spent months building this agent. In our internal tests, it felt natural. Then we went live - and users were pausing, repeating themselves, hanging up."— Suman Gandham, Co-founder & CEO, Vobiz
If you have ever shipped a voice AI product, you know exactly what this feels like. The sandbox sounds great. The latency numbers in the dashboard look fine. But the moment your agent hits a real PSTN call - coming in from a mobile phone, routing through a carrier network, bouncing across SIP trunks - something breaks. Not the model. Not your prompt. The conversation just feels... wrong. Like talking to someone on a satellite call from 2003.
The instinct is to blame the AI. Upgrade the model. Switch LLM providers. Spend another sprint on prompt engineering. None of it helps, because the model was never the problem to begin with.
The problem is the pipes.
The Voice AI Pipeline: Where Every Millisecond Lives
To understand why this happens, you need a clear picture of what actually happens between the moment a user finishes speaking and the moment your agent responds.
The Voice AI Pipeline (Mouth-to-Ear)
Add those up and you get a theoretical minimum of around 600–900ms for the AI stack alone, if everything is optimized. Real-world analysis of 4M+ production voice agent calls - published by Hamming AI - shows that the industry median mouth-to-ear response time is 1.4–1.7 seconds. For 10% of calls, it exceeds 3–5 seconds.
Human conversational norms tolerate a turn gap of roughly 200–300ms before something feels off. Research published in PNAS found that extremely short response times - under 250ms - are what make conversations feel natural and connected. Once that threshold is crossed consistently, users stop trusting the agent.
The Real Culprit: Legacy Telephony Infrastructure
The public switched telephone network (PSTN) and the SIP trunking infrastructure built on top of it were designed in an era when the only thing traveling over them was human voice. Specifically, they were optimized for voice that can tolerate 150–400ms of one-way delay without the listener noticing - because human brains compensate naturally.
Jitter Buffers
Receiving endpoints use a jitter buffer: a small delay introduced intentionally to reorder packets. Typical delay: 100–500ms.
Codec Transcoding
Each conversion between formats adds 20–80ms per hop and degrades fidelity. Standard G.711 (8kHz) vs AI native (24kHz+).
Carrier Handoffs
Calls bouncing between mobile carriers, PSTN gateways, and SIP providers adds cumulative latency at each step.
How Multi-Hop Architecture Compounds the Problem
The phrase "multi-hop" describes what happens when audio data crosses multiple independent infrastructure nodes before it reaches your AI.
Legacy SIP Deployment (Multi-Hop)
"Legacy SIP routes were never built with response time in mind. They were built for resilience - for making sure the call completed, not that it completed fast. Every hop that made sense for a human conversation is a structural tax on an AI one."— Vikash Srivastava, Co-founder & CTO, Vobiz
This is compounded by interruption handling breaking entirely at high latency. The experience is deeply unnatural and erodes user trust rapidly.
What the Numbers Actually Look Like in Production
The data from real production deployments makes the problem concrete:
- Industry median latency: 1.4–1.7 seconds mouth-to-ear, based on Hamming AI's analysis of 4M+ calls.
- WebRTC vs PSTN transport gap: WebRTC calls add 100–200ms of transport latency; PSTN/SIP adds 400–600ms.
- Hop reduction impact: A direct WebRTC path uses 2 hops, approaching 0 in P2P configurations.
- Market Growth: The Voice AI market is projected to grow at a 34.8% CAGR to USD 47.5B by 2034.
The Infrastructure Upgrade: AI-Native Single-Hop Routing
An AI-native telephony stack replaces multi-hop carrier routing with a single-hop media path - audio goes from the caller directly to an edge media server colocated with the AI inference stack.
AI-Native Infrastructure (Single-Hop)
- ✅ Ultra-low latency routing
- ✅ Noise isolation at infra layer
- ✅ AI-tuned jitter buffers
- ✅ Instant DID provisioning
Benchmark: Legacy Routes vs. AI-Native Infrastructure
| Metric | Legacy PSTN/SIP | Vobiz (AI-Native) |
|---|---|---|
| Avg Telephony Latency | 400–600ms | <80ms |
| Jitter Buffer Overhead | 100–500ms | <20ms |
| Codec Transcoding Steps | 2–4 per call | 1 (or none) |
| Network Hops | 4–5 | 1 (edge-terminated) |
| Audio Quality | 8kHz G.711 (degraded) | Up to 24kHz |
| Word Error Rate | 15–20% (noisy) | Significantly reduced |
| DID Provisioning | Days–weeks | Seconds (API) |
| Countries (Inbound) | Varies by provider | 190+ |
| Countries (Outbound) | Varies by provider | 190+ |
| Integration Time | Weeks | ~10 minutes |
| Uptime | Carrier SLA | 99.99% completion |
| Spam Flag Rate | Baseline | 30% reduction |
What Noise Does to the Problem
Latency is not the only axis where legacy infrastructure fails AI agents. Background noise creates a second structural problem.
Modern ASR systems like Deepgram Nova-3 achieve impressively low WER of 6.84% on clean audio. But in noisy real-world conditions, standard pipelines climb to 15–20% WER.
VAPI AI's analysis confirms that signal-to-noise ratio drops directly impact intent detection. Fixing the pipe fixes everything downstream.
"When noise cancellation runs at the media server - before audio reaches the STT model - the entire response chain improves."
What This Means for Builders
- 01Optimize transport layer first. The bigger gain is in the pipes (400-600ms gain) than in model tuning.
- 02Measure mouth-to-ear. Platform internal latency is a vanity metric. Real users experience the network transit.
- 03Do not chase model upgrades if your infrastructure is broken. A better LLM cannot compensate for 600ms of jitter buffer overhead.
- 04Noise is a telephony problem, not a model problem. You cannot prompt-engineer your way out of 15–20% WER caused by background noise on a PSTN call. Handle it at the infrastructure layer.
- 05Compliance belongs at the transport layer too. PII exposure and consent logging need to happen before audio touches third-party systems.
The model was never the bottleneck. The pipes were always the problem. Now there is a platform built to fix them.
Frequently Asked Questions
Why does my voice AI agent feel slow even though my model latency is under 500ms?
Because model latency is only one part of the total response time. The full mouth-to-ear latency includes network transmission, jitter buffering, codec transcoding, and carrier handoffs in the telephony layer - which typically adds 400–600ms on legacy PSTN/SIP routes.
What is the difference between PSTN/SIP latency and WebRTC latency for voice AI?
WebRTC establishes a direct UDP path between the client and the server, adding approximately 100–200ms of transport latency. PSTN and SIP routes pass through multiple carrier handoffs, jitter buffers, and codec conversions, adding 400–600ms.
What is 'mouth-to-ear latency' and why does it matter more than platform latency?
Mouth-to-ear latency measures the total time from when the user stops speaking to when they hear the AI agent's response - the delay as experienced by the caller. Platform latency only measures the AI stack's internal processing time and excludes network transit.
What word error rate should I target for production voice AI?
For enterprise production voice agents, target a WER below 5% in clean audio conditions. In noisy real-world calls, standard ASR pipelines can reach 15–20% WER without infrastructure-level noise cancellation.
How does AI-native telephony infrastructure differ from CPaaS?
CPaaS platforms were built primarily for human-to-human communication. AI-native telephony is designed from the ground up for AI agents: ultra-low-latency media routing, noise cancellation at the transport layer, and single-hop architecture.
Can voice AI agents be made compliant with HIPAA, GDPR, and TRAI on a single platform?
Yes, but compliance must be handled at the infrastructure layer to be effective. Vobiz provides ISO, GDPR, HIPAA, DPDP, and TRAI compliance out of the box as part of the infrastructure, not as an optional add-on.
How long does it take to integrate an AI-native telephony platform?
On AI-native platforms designed for developer velocity, integration takes minutes. Vobiz averages a 7-minute integration time with 10 minutes to first successful live call.