Voice AI Agent

The technical stack

A voice AI agent stitches together four components in real time:

1. ASR (Automatic Speech Recognition): converts the caller's speech to text. Streaming ASR is effectively mandatory: a system that waits for the caller to finish before it starts transcribing has already spent the whole conversational gap before it begins thinking.
2. LLM (Large Language Model): decides what to say back. Production systems choose between models on a latency-versus-quality tradeoff, since the fastest model is rarely the most capable one.
3. TTS (Text-to-Speech): converts the response into spoken audio.
4. Orchestration: manages the call lifecycle, interruption handling, function calls (book appointment, transfer to human), and CRM integrations.

How voice AI agents differ from chatbots

Chatbots are text-first and turn-based, you type, it answers. Voice AI agents are speech-first and continuous, both sides speak, often overlapping ("barge-in"). The engineering is harder: the model has to detect when the caller has finished speaking, decide whether to interrupt, and recover gracefully when the conversation goes off-rails.

The hardest technical problem in voice AI is end-of-turn detection: knowing when the caller has finished a sentence vs paused mid-thought. Most production systems use a combination of voice activity detection (VAD), semantic completion estimation, and a silence threshold. The threshold is a tuning decision with a real tradeoff rather than a settled number: set it short and the agent interrupts people who were still thinking, set it long and every reply feels slow.

Latency budget

End-to-end latency, from caller stops speaking to agent starts speaking, is the single most-measured metric in voice AI. It is also the metric most often quoted without a source, so it is worth separating what is actually established from what vendors assert.

What is established. Human conversation runs on very short gaps. A ten-language study published in PNAS found the most common gap between one speaker finishing and the next starting is between zero and 200 milliseconds, with every language studied clustering within about a quarter second of that. That is the bar a voice agent is unconsciously measured against, and it is why a delay that looks small on paper feels like hesitation on a call.

Separately, the ITU's telephony recommendation on one-way transmission delay treats under 150ms as essentially transparent to callers and delays beyond 400ms as unacceptable for general network planning. That figure covers network transmission rather than an agent's thinking time, so it is a floor the plumbing must clear, not a target for response latency.

What is not established. The component-by-component latency budgets published across this industry do not agree with each other, are not independently benchmarked, and come almost entirely from vendors describing their own products. Any specific millisecond breakdown you read, including one attached to a total, should be treated as that vendor's claim about that vendor's stack rather than a measured industry standard.

The practical version, without a fabricated number attached: a voice agent should begin responding fast enough that the pause does not read as hesitation, and the only honest way to know whether a given system clears that bar is to call it yourself.

Common deployments

• Inbound receptionist: answer + qualify + book or dispatch. Largest commercial deployment in 2026.
• Outbound sales / appointment-setting: high-volume cold outbound (regulatory friction in most US states; check TCPA).
• In-call assist: coach a human agent in real time with prompts during the call.
• IVR replacement: replace touch-tone menus with natural language ("press 1 for…" becomes "what can I help you with?").

Limitations in 2026

• Multi-party conferences: most agents handle 1:1 calls only.
• Code-switching: switching languages mid-conversation works but accents reduce ASR accuracy.
• Emotional escalation: agents detect frustration but don't always defuse it. Routing to a human is still the standard pattern.
• Complex authentication: voice biometric KYC is improving but not yet trusted for high-stakes verification.

How the market is organised

Named vendors were removed from this section on 2026-07-30. Naming competitors in outward copy breaks a standing rule, and a glossary entry is outward copy. The taxonomy is the useful part and it survives without the names.

• Vertical receptionists: built for one industry, with that industry's call shapes, integrations and escalation rules already in place. HiGrovi is one of these, built for home and pet services.
• Developer platforms: toolkits for building your own voice agent. Powerful, and they assume you have an engineer.
• Enterprise customer-experience suites: large-scale contact-centre systems, priced and scoped for enterprises rather than a contractor with four trucks.
• Voice infrastructure: the speech and language layers that the other three categories are built on top of, rarely bought directly by a business.