Voice AI Phone Receptionists: Replacing Manual Front Desks with Sub-Second Voice Agents
Traditional Interactive Voice Response (IVR) phone menus ("Press 1 for sales, press 2 for support") are despised by callers: 67% of customers immediately press zero to speak to a human or hang up entirely. When front desks are overwhelmed or closed after hours, missed calls cost businesses thousands of dollars in lost bookings and emergency inquiries. By architecting voice AI phone receptionists using bi-directional Twilio WebSockets, sub-120ms speech-to-text with Deepgram Nova-2, and ultra-low latency streaming TTS with Cartesia Sonic, businesses can deploy conversational phone agents that sound indistinguishable from human staff with end-to-end turnaround latency under 400 milliseconds.
🎙️ The 500ms Human Latency Threshold
In conversational telephony, any pause exceeding 500ms breaks conversational presence, causing callers to speak over the bot or ask "Hello, are you still there?". Achieving sub-400ms turnaround requires streaming audio chunks continuously over full-duplex WebSockets without buffering complete sentences before synthesis.
1. Full-Duplex Telephony Architecture & Latency Budget
The total end-to-end roundtrip latency $T_{ ext{roundtrip}}$ is the sum of pipeline stages:
Where Deepgram Nova-2 resolves incoming $\mu$-law audio in $ au_{ ext{STT}} pprox 120 ext{ms}$, Groq or Cerebras-hosted Llama 3.3 delivers Time-To-First-Token in $ au_{ ext{LLM\_TTFT}} pprox 90 ext{ms}$, and Cartesia Sonic streams audio frames back in $ au_{ ext{TTS\_chunk}} pprox 85 ext{ms}$. Full-duplex interruption (barge-in) immediately cancels pending audio buffers if the caller starts speaking.
| Pipeline Stage | Provider / Model | Streaming Protocol | Latency Target |
|---|---|---|---|
| Telephony Gateway | Twilio Media Streams | Bi-directional WebSocket (8kHz $\mu$-law) | < 35 ms |
| Speech-to-Text | Deepgram Nova-2 Phonecall | Streaming WebSocket with interim results | < 120 ms |
| Dialogue Reasoning | Groq / Llama 3.3 70B | Streaming SSE Token Generation | < 90 ms TTFT |
| Voice Synthesis | Cartesia Sonic / ElevenLabs Turbo | Raw PCM streaming chunk delivery | < 95 ms First Audio Chunk |
2. Production Fastify / Node.js Twilio Media Stream Handler
Below is a production WebSocket handler connecting Twilio Media Streams to a streaming speech engine with immediate barge-in interruption support:
import Fastify from 'fastify';
import fastifyWs from '@fastify/websocket';
import WebSocket from 'ws';
const fastify = Fastify({ logger: true });
fastify.register(fastifyWs);
fastify.register(async function (fastify) {
fastify.get('/media-stream', { websocket: true }, (connection, req) => {
let streamSid = null;
console.log('📞 Twilio media stream connected');
// Connect to streaming Deepgram STT
const deepgramWs = new WebSocket('wss://api.deepgram.com/v1/listen?encoding=mulaw&sample_rate=8000', {
headers: { Authorization: `Token ${process.env.DEEPGRAM_API_KEY}` }
});
connection.socket.on('message', (message) => {
const data = JSON.parse(message);
if (data.event === 'start') {
streamSid = data.start.streamSid;
console.log(`Stream started: ${streamSid}`);
} else if (data.event === 'media' && deepgramWs.readyState === WebSocket.OPEN) {
// Forward raw audio chunk to Deepgram
deepgramWs.send(Buffer.from(data.media.payload, 'base64'));
} else if (data.event === 'stop') {
console.log('Stream stopped');
}
});
deepgramWs.on('message', (msg) => {
const res = JSON.parse(msg);
const transcript = res.channel?.alternatives?.[0]?.transcript;
if (res.is_final && transcript) {
console.log(`Caller said: ${transcript}`);
// Dispatch to LLM and stream Cartesia audio back to Twilio
}
});
});
});
fastify.listen({ port: 5000, host: '0.0.0.0' }, () => {
console.log('🚀 Telephony WebSocket running on port 5000');
});
3. Frequently Asked Questions
How does the system handle caller interruption (barge-in)?
The moment Deepgram emits an interim transcript indicating speech during bot audio playback, a `clear` event is dispatched to Twilio (`{"event": "clear", "streamSid": streamSid}`) to immediately flush the audio buffer and cancel ongoing TTS streaming.
What is the cost per call minute?
Twilio media streaming costs ~$0.004/min, Deepgram is $0.0043/min, Cartesia is ~$0.012/min, and open-source models via Groq are ~$0.002/min. Total blended operating cost is under $0.025 per minute, representing a 98% savings compared to human front desks.