CtrlK
An AI voice agent is a conversational artificial intelligence system that communicates with users through spoken language in real time. Unlike text-based chatbots or simple interactive voice response (IVR) menus, AI voice agents can hold fluid, natural-sounding telephone or in-app conversations, interpret intent, handle interruptions, and execute tasks such as booking appointments or transferring calls. The technology sits at the intersection of speech recognition, large language models, and text-to-speech synthesis, and it has become one of the fastest-growing segments of the broader conversational AI market.
The global voice AI agents market was valued at approximately USD 2.4 billion in 2024 and is projected to reach USD 47.5 billion by 2034, growing at a compound annual growth rate (CAGR) of 34.8%. Enterprises across banking, healthcare, retail, and telecommunications have driven this growth by deploying voice agents for customer service automation, outbound sales, appointment scheduling, and phone triage. Gartner predicts conversational AI will reduce contact center agent labor costs by $80 billion in 2026, and a Forrester Consulting study found that companies using voice AI report three-year ROI between 331% and 391%.
AI voice agents rely on one of two primary architectural paradigms: the cascaded (pipeline) approach and the end-to-end (speech-to-speech) approach.
The cascaded architecture is the most widely deployed design in production systems as of 2025. It chains together three distinct components in sequence:
Because each component operates independently, the cascaded pipeline is highly modular. Developers can swap in different STT, LLM, or TTS providers depending on cost, latency, or language requirements. The text intermediary between stages also provides a convenient point for applying content filters, compliance checks, and logging before the caller hears anything. These properties make the cascaded approach attractive for enterprise deployments where control, debuggability, and regulatory compliance are priorities.
The primary drawback is latency. Each handoff between components adds delay. A well-optimized cascaded pipeline typically achieves 500 to 800 milliseconds of end-to-end latency, though poorly tuned setups can exceed two seconds. Information is also lost at each boundary: the STT stage discards prosodic cues like tone, emphasis, and emotion, meaning the LLM works only with flat text.
End-to-end models process the entire exchange in a single latent space, accepting audio input and producing audio output without intermediate text representations. This design eliminates the multiple conversion steps and handoffs between models, achieving latency as low as 200 to 300 milliseconds. The model can also preserve and respond to non-verbal cues such as tone of voice, hesitation, and emotion, because audio features are never discarded.
OpenAI's GPT-4o and the gpt-realtime model family are leading examples of this approach. Google's Gemini Live similarly operates as an end-to-end voice model.
Despite the latency and expressiveness advantages, end-to-end models present significant challenges for enterprise adoption. Four main issues limit their use: the lack of a text intermediary makes content filtering harder; debugging is more difficult because there is no transcript to inspect mid-pipeline; there are no straightforward fallback mechanisms if one component fails; and evaluation tooling for speech-to-speech quality remains immature. Costs also tend to run higher, sometimes roughly ten times that of a chained pipeline, because the model re-processes the entire conversation context on each turn.
As a result, while speech-to-speech models saw major investment from AI labs throughout 2025, cascaded architectures remain the dominant solution for complex agentic tasks in production environments.
A third approach, sometimes called the "half-cascade" architecture, has gained traction as a middle ground. In this design, audio input is processed by a native audio encoder that feeds directly into a text-based language model for reasoning and response generation. The text output is then synthesized into speech by a TTS component. This preserves the speed advantage of native audio input (because the STT step is replaced by a faster audio encoder) while maintaining the debuggability and tool-calling reliability of text-based reasoning.
Both OpenAI and Google use variants of this half-cascade architecture in their consumer voice products. Google's Gemini 2.5 Flash Native Audio model, for instance, processes raw audio natively through a single low-latency model while still producing text-based reasoning internally. OpenAI's gpt-realtime model similarly combines native audio understanding with structured text-based tool calling.
Some platforms take a different hybrid approach: using a speech-to-speech model for initial response generation to minimize latency, then falling back to a cascaded pipeline for tool calls or knowledge retrieval that require structured text reasoning. These designs attempt to capture the low latency of end-to-end models while retaining the controllability of cascaded pipelines.
As of early 2026, less than 15% of enterprise deployments use pure speech-to-speech architectures, with the majority relying on cascaded or hybrid systems for production workloads.
| Attribute | Cascaded (STT + LLM + TTS) | End-to-end (S2S) | Hybrid (half-cascade) |
|---|---|---|---|
| Typical latency | 500-2,000+ ms | 200-300 ms | 300-800 ms |
| Debuggability | High (full text transcripts at each stage) | Low (no intermediate transcripts) | Medium (text available at reasoning stage) |
| Emotional understanding | Low (prosody lost in STT conversion) | High (native audio preserves tone, emotion) | Medium to high (audio encoder retains some cues) |
| Tool calling reliability | High | Less reliable | High |
| Compliance and auditing | Strong (full transcripts for review) | Weak (no text intermediary) | Moderate to strong |
| Modularity | High (swap STT, LLM, TTS independently) | Low (single model) | Medium |
| Cost per minute | ~$0.15/min | Higher (up to 10x cascaded) | Varies |
| Enterprise adoption (2026) | Dominant | <15% | Growing |
Several major AI companies have launched consumer-facing voice conversation products that showcase the state of the art.
OpenAI demonstrated Advanced Voice Mode as part of the GPT-4o announcement in May 2024, but the feature did not ship immediately. It began rolling out to a limited group of ChatGPT Plus subscribers in late July 2024 and expanded to all Plus and Teams subscribers on September 24, 2024. Advanced Voice Mode uses the GPT-4o model's native audio capabilities, allowing it to process speech input and generate speech output in a single model. Users can interrupt the model mid-sentence, and the system can sense and interpret emotions from tone of voice and adjust its responses accordingly. As of September 2024, ChatGPT offered nine voice options: Breeze, Juniper, Cove, Ember, Arbor, Maple, Sol, Spruce, and Vale.
For developers building their own voice agents, OpenAI introduced the Realtime API in public beta on October 1, 2024, at OpenAI Dev Day. The API allows third-party applications to stream audio to and from OpenAI's speech-to-speech models over WebSocket or WebRTC connections. On October 30, 2024, OpenAI added five new voices with greater range and expressiveness.
In August 2025, the Realtime API reached general availability with the launch of the gpt-realtime model. This model showed significant improvements in instruction following accuracy (30.5% on the MultiChallenge audio benchmark, up from 20.6% for the previous model) and function calling accuracy (66.5% on ComplexFuncBench, up from 49.7%). The generally available API added support for remote MCP servers, image inputs, and phone calling through Session Initiation Protocol (SIP), enabling developers and enterprises to build production-ready voice agents.
Google unveiled Gemini Live at its Pixel launch event in August 2024. The feature allows users to have free-form voice conversations with Gemini, including the ability to interrupt the AI and change topics mid-conversation. Gemini Live initially launched for English-speaking Android users who subscribed to Gemini Advanced. By early October 2024, Google expanded availability to all Android users at no cost. The feature subsequently rolled out to iOS and Google Workspace accounts.
Anthropic launched voice mode for Claude in late May 2025, initially available to paid subscribers on iOS and Android. Voice access was extended to all users on June 3, 2025. The feature offers five distinct voice options (Buttery, Airy, Mellow, Glassy, and Rounded) and allows users to switch between text and voice during a conversation. In March 2026, Anthropic introduced a voice mode for Claude Code, its command-line coding assistant, using a push-to-talk interface activated via the /voice command.
A growing ecosystem of startups and platforms enables businesses to build, deploy, and manage AI voice agents without developing the underlying infrastructure from scratch. These platforms typically provide orchestration layers that combine STT, LLM, and TTS components along with telephony integration, analytics, and compliance tooling.
| Platform | Founded | Headquarters | Key Features | Notable Funding | Typical Latency |
|---|---|---|---|---|---|
| Bland AI | 2023 | San Francisco, CA | No-code Conversational Pathways workflow builder; HIPAA and GDPR compliant self-hosted infrastructure; voice cloning; multi-language support; post-call analytics | $65M total (including $40M Series B, January 2025, led by Emergence Capital) | Not publicly disclosed |
| Vapi | 2023 | San Francisco, CA | Developer-first API; real-time voice orchestration over WebRTC; supports bring-your-own STT/LLM/TTS; 100+ languages; Squad multi-agent routing; GoHighLevel and Make.com integrations | ~$25M total ($20M Series A, December 2024, led by Bessemer Venture Partners) | 550 to 800 ms |
| Retell AI | 2023 | San Francisco, CA | No-code agent builder; proprietary turn-taking model; function calling for appointments and CRM updates; 31+ languages with automatic language detection; HIPAA, SOC 2 Type II, GDPR compliant; Retell Assure automated QA (launched late 2025) | ~$5M seed (Y Combinator, 2024) | ~600 ms |
| ElevenLabs Conversational AI | 2022 | New York, NY | Sub-100 ms voice latency; 32+ languages; RAG integration; SDKs for JavaScript, Python, Swift; Conversational AI 2.0 (May 2025) with multimodal text and voice input | $180M Series C (January 2025, a16z and ICONIQ Growth); $500M Series D (February 2026, $11B valuation) | Sub-100 ms |
| Voiceflow | 2019 | San Francisco, CA | Visual drag-and-drop flow builder; Agent Step for autonomous AI decisions (Winter 2025); 300+ native integrations; SOC 2 and ISO compliant; custom TTS voices via ElevenLabs | ~$39M total | Sub-500 ms |
| Play.ai (PlayAI) | 2022 | San Francisco, CA | PlayDialog model with emotional prompting; Play 3.0 mini for low-latency multilingual TTS (30+ languages); web, phone, and app deployment; 24/7 voice agents | $21M seed (November 2024, led by Kindred Ventures, with Y Combinator) | Not publicly disclosed |
AI voice agents have found traction across a range of industries and business functions. The banking, financial services, and insurance (BFSI) sector leads adoption with a 32.9% market share as of 2024, followed by healthcare, retail, and telecommunications.
The most common deployment scenario involves replacing or augmenting traditional call center operations. Voice agents can handle frequently asked questions, account inquiries, password resets, billing disputes, and order status checks without human intervention. Retell AI reports that companies deploying its technology automate up to 80% of inbound calls. By operating around the clock and handling unlimited concurrent calls, voice agents eliminate hold times and reduce labor costs.
Healthcare providers, dental offices, salons, and service businesses use voice agents to manage appointment scheduling over the phone. The agent accesses the business's calendar system through function calling or API integration, checks availability, and confirms bookings in real time during the call. This use case is well suited to voice AI because the conversation follows a relatively predictable structure while still requiring natural language understanding to handle variations in how callers express their needs.
Voice agents can initiate outbound calls to prospects, deliver a scripted pitch, answer questions, and qualify leads based on predefined criteria before routing interested prospects to human sales representatives. Bland AI's Conversational Pathways feature allows sales teams to design branching call flows that adapt based on the prospect's responses. The scalability of AI-driven outbound calling allows businesses to reach thousands of prospects simultaneously, though this use case faces particular regulatory scrutiny (see Ethical and Legal Considerations below).
In healthcare settings, voice agents perform initial patient triage by asking about symptoms, urgency, and medical history before routing the call to the appropriate department or scheduling a telehealth consultation. In corporate environments, voice agents serve as intelligent receptionists that understand caller intent and route calls to the correct department or individual, replacing rigid IVR menu trees with natural conversation.
Financial institutions and utilities deploy voice agents to make collections calls, negotiate payment plans, and process payments over the phone. The agent can access account information in real time, verify the caller's identity, and complete transactions, all while maintaining compliance with regulations such as the Fair Debt Collection Practices Act.
Voice agents conduct post-interaction surveys, customer satisfaction calls, and market research interviews. Because the agent can ask follow-up questions and probe for detail, voice surveys often yield richer qualitative data than automated text-based surveys or pre-recorded robocalls.
Building voice agents that feel natural and reliable in production requires solving several difficult engineering problems.
Conversational fluency demands that the agent respond within a window that feels natural to the caller. Research on human conversation patterns suggests that pauses longer than approximately 500 milliseconds begin to feel unnatural, and pauses beyond one second are perceived as the system being "stuck" or broken. Achieving sub-500-millisecond end-to-end latency in a cascaded pipeline requires aggressive optimization at every stage: streaming STT that emits partial transcripts, speculative LLM inference that begins generating before the user finishes speaking, and streaming TTS that starts synthesizing audio from the first output tokens.
End-to-end speech-to-speech models can achieve 200 to 300 milliseconds of latency by eliminating inter-component handoffs, but they come with the tradeoffs described in the Architecture section. ElevenLabs claims sub-100-millisecond voice latency for its Conversational AI platform, though this figure likely measures only the TTS component rather than full end-to-end latency.
Human conversations involve constant, subtle negotiation over who speaks next. Speakers use pauses, intonation changes, and filler words to signal that they are yielding the floor or holding it. Replicating this behavior in a voice agent is one of the field's hardest unsolved problems.
The specific challenge of "barge-in" detection (recognizing when a caller interrupts the agent mid-utterance) illustrates the difficulty. Most voice agents rely on Voice Activity Detection (VAD) to notice when the caller is speaking during the agent's turn. But VAD alone cannot distinguish between a genuine interruption ("Actually, never mind, I want something else"), a backchannel acknowledgment ("mm-hmm," "yeah"), ambient noise (a cough, typing, or background chatter), and an echo of the agent's own output.
Treating every detected sound as a full interruption makes the agent jittery: it constantly stops mid-sentence and restarts, creating a frustrating experience. Ignoring all sounds during agent speech makes the agent seem oblivious when the caller genuinely wants to interject. Advanced systems use a combination of VAD, acoustic echo cancellation, semantic analysis of partial transcripts, and trained classifiers to categorize detected speech as interruption, backchannel, or noise. Retell AI, for example, has developed a proprietary turn-taking model specifically designed to determine when to stop speaking and when to continue.
Human callers convey frustration, confusion, urgency, and satisfaction through their tone, pace, and inflection. A voice agent that responds to an angry customer in a cheerful tone risks escalating the situation. Detecting caller emotion from audio signals and adapting the agent's response (both in content and delivery) remains an active area of research.
On the generation side, most TTS engines produce speech with a limited emotional range. Play.ai's PlayDialog model introduces "emotional prompting" to control the tone, pacing, and inflection of generated speech. GPT-4o's Advanced Voice Mode can express a range of emotions and adjust its delivery based on the caller's detected emotional state, representing one of the most advanced capabilities in production as of 2025.
Global enterprises require voice agents that operate across multiple languages and understand a wide range of accents. While the leading platforms claim support for 30 to 100+ languages, performance varies significantly across languages and dialects. Code-switching (when a caller mixes two languages in a single sentence) poses particular challenges for both STT and LLM components. The gpt-realtime model introduced in 2025 specifically improved its ability to switch languages within a single sentence and process alphanumeric sequences across languages.
Voice conversations are less forgiving of errors than text interactions. If a text chatbot misunderstands a query, the user can simply retype their message. In a voice conversation, misunderstandings compound: the agent may respond to something the caller did not say, the caller may not realize the misunderstanding occurred, and the conversation can drift far off track before either party recognizes the problem. Building robust error detection and graceful recovery mechanisms (such as confirming understanding before taking irreversible actions) is critical for production voice agents.
ElevenLabs, founded in 2022 and headquartered in New York, has established itself as a leading platform for voice AI. Originally known for its high-fidelity text-to-speech and voice cloning technology, the company expanded into the conversational AI space in November 2024 with the launch of its Conversational AI developer platform.
The platform allows developers to build interactive voice agents powered by leading LLMs (including Claude, GPT-4, and Gemini) or custom models. It includes built-in retrieval-augmented generation (RAG) so agents can ground their answers in business-specific data. SDKs are available for JavaScript, Python, Swift, and additional languages.
In May 2025, ElevenLabs released Conversational AI 2.0, introducing multimodal input support (simultaneous text and voice), improved enterprise readiness, and enhanced agent capabilities. The platform claims sub-100-millisecond latency and supports 32 or more languages.
ElevenLabs has raised significant capital to fuel its growth. In January 2025, the company announced a $180 million Series C round co-led by a16z and ICONIQ Growth, valuing the company at $3.3 billion. In February 2026, ElevenLabs raised $500 million in a Series D round at an $11 billion valuation as the company began exploring a potential initial public offering.
The rapid advancement of AI voice technology has raised significant ethical and legal concerns that regulators, industry groups, and civil society organizations are actively working to address.
Modern TTS systems can clone a person's voice from as little as a few seconds of reference audio. While this capability has legitimate applications (such as creating personalized voice agents or preserving the voices of people with degenerative diseases), it also enables misuse. Unauthorized voice cloning has been used to create fraudulent audio of public figures, conduct phone scams impersonating family members, and generate non-consensual content.
The core ethical issue is consent. Courts in the United States and the European Union began classifying voice data as biometric property in early 2024, allowing individuals to claim ownership of their vocal signatures. In the case of Lehrman v. Lovo, Inc. (2024), professional voice actors alleged that Lovo Inc. used their recorded voices without proper authorization to train AI models. The U.S. District Court for the Southern District of New York partially granted and partially denied a motion to dismiss, establishing that unauthorized use of voice data for AI training may constitute a viable legal claim.
AI-generated voice deepfakes pose risks to public trust, political discourse, and individual reputation. Deepfake audio can fabricate statements that an individual never made, potentially leading to defamation, market manipulation, or political interference. The EU AI Act classifies voice cloning as high-risk AI, demanding transparency and strict safeguards. Several U.S. states have passed deepfake and voice cloning laws requiring consent and clear disclosure.
On February 8, 2024, the U.S. Federal Communications Commission (FCC) unanimously adopted a Declaratory Ruling confirming that calls made with AI-generated voices qualify as "artificial or prerecorded voice" calls under the Telephone Consumer Protection Act (TCPA). This ruling requires that callers obtain prior express consent from the called party before making AI-generated voice calls, provide identification and disclosure information about the party responsible for initiating the call, and offer opt-out mechanisms. The ruling gave State Attorneys General new enforcement tools against AI-powered robocall scams.
For businesses deploying AI voice agents for outbound calling, this ruling means that consent management and disclosure practices are legally mandatory, not optional best practices.
A growing consensus among regulators and industry participants holds that AI voice agents should disclose their non-human identity at the beginning of every interaction. California's B.O.T. Act (2019) already requires bots to disclose their artificial identity when communicating with consumers for sales or political purposes. The EU AI Act imposes similar disclosure requirements for AI systems that interact with humans. Several voice AI platforms have built disclosure features into their products, automatically playing a statement such as "This call is powered by AI" at the start of each conversation.
Voice agents may exhibit biased behavior based on the caller's accent, dialect, speech patterns, or language. STT models have been shown to perform less accurately on speakers with non-standard accents, which can lead to higher error rates and poorer service for certain demographic groups. Ensuring equitable performance across diverse speaker populations is an ongoing challenge that requires careful dataset curation, testing across demographic groups, and monitoring in production.
The conversational AI market is projected to grow from $17.97 billion in 2026 to $82.46 billion by 2034. Gartner forecasts that 40% of enterprise applications will integrate task-specific AI agents by the end of 2026, up from less than 5% in 2025. An estimated 80% of all businesses plan to integrate AI-driven voice technology into customer service operations by 2026.
In the near term, cascaded architectures continue to dominate enterprise deployments because of their reliability, debuggability, and compliance features. Speech-to-speech models are gaining ground in consumer-facing applications where natural conversational feel is prioritized over auditability. Hybrid (half-cascade) architectures are emerging as the likely medium-term standard for enterprise use, combining the latency benefits of native audio processing with the control and transparency of text-based reasoning.
The banking, financial services, and insurance (BFSI) sector leads voice AI adoption with a 32.9% market share as of 2024. Healthcare, retail, and telecommunications follow. Competition among platform vendors has driven down per-minute costs, with entry-level pricing dropping below $0.07 per minute for some providers.
Several trends are shaping the trajectory of AI voice agent technology.
Agentic capabilities. Voice agents are evolving from conversational interfaces into autonomous AI agents that can take complex, multi-step actions on behalf of the caller. This includes navigating multiple backend systems, making decisions based on business logic, and completing transactions end to end without human handoff.
Multimodal interaction. The boundary between voice and other modalities is blurring. ElevenLabs' Conversational AI 2.0 supports simultaneous voice and text input. Google's Gemini Live can process visual input alongside voice. Future voice agents will likely combine speech, vision, screen sharing, and text in unified interactions.
On-device processing. Anthropic has been preparing offline voice packs that allow voice processing without an internet connection for short prompts, designed for educational institutions and sensitive enterprise environments. On-device STT and TTS models are becoming feasible on modern smartphones and edge devices, which could reduce latency to near zero and address data privacy concerns.
Improved evaluation and quality assurance. Retell AI's launch of Retell Assure in late 2025, the first automated QA solution for voice AI, signals growing industry recognition that voice agents need systematic monitoring and evaluation. As deployment scales, automated tools for detecting hallucinations, measuring conversation quality, and identifying failure modes will become essential.
Cost reduction. The cost of operating voice agents is declining as competition intensifies among STT, LLM, and TTS providers. Models like Play.ai's Play 3.0 mini and smaller open-source alternatives are making production voice agents accessible to smaller businesses.