For three years the agent stack was a text problem. The interesting commercial agents in 2026 — the ones eating airline call centres, drive-through kiosks, and tier-1 IT helpdesks — are voice agents, and they require a fundamentally different architecture: bidirectional audio streams, sub-second turn-taking, interruption handling, and a model that can listen, think, and talk at the same time. This is the loop that’s redefining where conversational AI lives.
Core Concept
From Cascade to Speech-to-Speech — Why Voice Is a Different Stack
Until 2024, every voice product was a sandwich. Audio in → ASR (automatic speech recognition, e.g. Whisper) → transcribed text into an LLM → LLM text out → TTS (text-to-speech, e.g. ElevenLabs) → audio out. This is the “cascade” or “pipeline” architecture. It works, the components are interchangeable, and every customer-service voice bot you’ve called this decade is probably built this way. It also has three structural failures: latency stacks up (each box adds 100–500 ms), paralinguistics get thrown away (the LLM never sees that the caller is angry, hesitant, or sarcastic — the transcript is flat text), and the model can’t talk while it listens (turn-taking is enforced by silence detection rather than by understanding).
Analogy first. The cascade is like writing a letter, mailing it, getting a reply, mailing back. A real conversation is two people in a room who can interrupt, finish each other’s sentences, hum “mm-hmm” while the other talks. To get the second behaviour out of a model you have to build it differently: the model itself reads and writes audio tokens directly, both streams run in parallel, and there is no “turn” concept — just two voices that happen to be present in the same context window simultaneously. That is full-duplex speech-to-speech, and it’s the architecture Moshi, GPT-realtime, and Gemini Live are converging on.
Mechanism. There are three architectures alive in the field right now, and an engineer should be able to draw all three on a whiteboard.
The three voice architectures
ASR → LLM → TTS — the pipeline
How do most production voice bots work today?
Three separate models in series, each streaming chunks. Modern stacks (Pipecat, LiveKit Agents) heavily optimise the seams: streaming Whisper variants emit partials, the LLM begins generating from the first ASR partial, the TTS streams phonemes as the LLM emits its first token. End-to-end latency lands around 800–1,500 ms in production. Wins on debuggability and model swap-ability; loses on naturalness and emotional bandwidth.
Audio in → LLM → Audio out — one model, half-duplex
How does GPT-4o ‘voice’ or LLaMA-Omni work?
A single multimodal model takes audio tokens (from a neural codec like Mimi, SNAC, or HuBERT-derived units) and emits audio tokens. There is no transcript intermediate, so paralinguistics survive: the model hears that you sighed and can choose to sigh back. But the model still produces one turn at a time — it listens, thinks, then speaks. LLaMA-Omni reports 226 ms response latency (Fang et al., ICLR 2025).
Two parallel streams, no turns — full-duplex
Why is Moshi different from everything before it?
The model maintains two audio streams in parallel: one is its own output, the other is the user’s input. Both are tokenised at the same frame rate (12.5 Hz for Moshi) and the model predicts the next token in both streams every step. Speaker turns disappear. The model can decide to stay silent for a beat, hum acknowledgement, or interrupt — behaviours emerging from the data, not a turn-taking heuristic. Moshi reports 160 ms theoretical / 200 ms practical latency (Défossez et al., 2024).
Anatomy of a single conversational round-trip
The number every voice engineer obsesses over is response latency: from user-stops-talking to first-audio-byte-received. Below is what the budget looks like across the three architectures, mapped onto a 1-second wall clock. The cascade has to spend latency three times; full-S2S spends it once and overlaps with input.
# Latency budgets (rough order-of-magnitude, production)
cascade:
user stops —> VAD(120) ASR-final(180) LLM-TTFT(280) TTS-first(150) net(80)
~830ms total · 4 hops, no overlap inside hops
half-s2s:
user stops —> VAD(80) audio-LM-first-token(180) net(40)
~300ms total · one hop, paralinguistics preserved
full-s2s:
user still talking —> model can already pre-attend & co-emit net(40)
~160-200ms perceived · turn boundary becomes optional
Two more pieces matter to claim engineer-level fluency. First, VAD & barge-in. Voice Activity Detection is the millisecond-resolution classifier that decides when a human started or stopped speaking. Modern stacks use Silero VAD or a server-side model on a 10 ms hop. When the human starts talking while the model is talking, that’s barge-in, and the right behaviour is to instantly mute the TTS, snapshot what the model already said, and resume listening. Get this wrong and your agent talks over the customer or, worse, ignores them. Second, WebRTC vs WebSocket. WebRTC is built for voice (jitter buffer, opus codec, packet loss recovery, NAT traversal); WebSocket is a TCP stream. For anything user-facing in the browser or on a phone, WebRTC is the right transport — it is what LiveKit, Daily, and the OpenAI Realtime API use under the hood. WebSocket is fine for server-to-server.
A useful test of an engineer’s grasp: ask them what happens when the user says “wait, no” 200 ms after the agent starts speaking. If they can sketch VAD → interruption event → cancel pending TTS → truncate model output → emit barge-in turn back to the LLM — they understand voice agents. If they say “the LLM handles it”, they don’t.
Papers to Know
Three Papers That Defined the Voice-Agent Stack
arXiv preprint
Seminal
2024 (Kyutai)
Moshi: a speech-text foundation model for real-time dialogue
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, Neil Zeghidour (Kyutai) · arXiv 2410.00037
Defined the modern blueprint for full-duplex speech-to-speech: a 7B Transformer that simultaneously predicts tokens for its own audio stream and the user’s audio stream, plus an inner monologue text stream for reasoning. Introduced Mimi, a streaming neural audio codec at 12.5 Hz with semantic + acoustic tokens, replacing earlier two-step codecs like SoundStream/Encodec for dialogue. Reports 160 ms theoretical and 200 ms practical end-to-end latency.
Why it matters: First open-source proof that a single neural network can hold a real-time, full-duplex spoken conversation — the engineering dream that made voice agents the hottest 2026 frontier. Every later S2S model (LLaMA-Omni 2, Step-Audio 2, Sesame CSM) cites Moshi’s parallel-stream design.
arXiv 2410.00037
Published
Recent
ICLR 2025
LLaMA-Omni: Seamless Speech Interaction with Large Language Models
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, Yang Feng (ICTNLP, Chinese Academy of Sciences) · arXiv 2409.06666
Built on Llama-3.1-8B-Instruct: a speech encoder + a speech adaptor feeds audio embeddings directly to the LLM, and a streaming speech decoder emits audio tokens. No intermediate transcript step. The team curated InstructS2S-200K, a 200,000-sample speech-instruction dataset. Reports response latency as low as 226 ms; accepted at ICLR 2025.
Why it matters: Showed you can bolt audio I/O onto a strong open-weights LLM with modest compute (a few hundred GPU-hours), bypassing the need to retrain a foundation model from scratch. This is the recipe most enterprises will follow for domain-specific voice agents — take Llama or Qwen, glue on speech adapters, fine-tune on your call recordings.
arXiv 2409.06666
arXiv preprint
Recent
2024 (Tsinghua · gpt-omni)
Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
Zhifei Xie, Changqiao Wu (gpt-omni / Tsinghua) · arXiv 2408.16725
Proposed a text-audio parallel decoding scheme: at each step the model emits both an audio token and a text token, so the language model’s “reasoning” trace is preserved while audio streams out. Used a batch-parallel inference trick to keep generation real-time. Released VoiceAssistant-400K, a 400,000-sample fine-tuning corpus. The smallest fully end-to-end open-source S2S model when it shipped.
Why it matters: Established the “think in text, speak in audio” pattern that lets a small model punch above its weight in voice. The implementation lives at gpt-omni/mini-omni and is the most-forked starting point for hobbyist S2S projects in 2026.
arXiv 2408.16725
Community Pulse
What Practitioners Are Saying
Andrej Karpathy · paraphrased position
Karpathy has argued that voice is the natural input modality for the LLM era and that the next interesting interfaces will be speech-first — treating the keyboard as a fallback rather than the default.
Recurring theme across his late-2025 talks (notably the “Software 3.0” framing on Latent Space) and his coverage of ambient computing. No single quotable URL on the realtime API itself, so phrased as a position rather than a verbatim quote.
Swyx (Latent Space) · paraphrased position
Swyx has argued that “Agent Engineering” — not chat — is the unit of work for AI in 2025-26, and that the production-grade voice agent is the canonical end-form: real-time, multimodal, tool-using, and observable end-to-end.
Theme of his AI Engineer Summit 2025 keynote (“Agent Engineering”) on latent.space; voice as the dominant deployment surface is a recurring thread in his Realtime API coverage.
Harrison Chase (LangChain) · paraphrased position
Chase has argued that the more agentic an application becomes, the more the LLM dictates control flow — and that voice agents collapse the LangGraph state machine onto a model that decides, in real time, when to listen, plan, call a tool, or talk back.
Synthesis of his widely-cited definition of “agentic” and his 2025 blog series on voice as a first-class LangGraph runtime. Paraphrased — no single canonical quote URL.
Simon Willison · paraphrased position
Willison has flagged the OpenAI Realtime API and the broader voice-agent rollouts as one of the more under-appreciated unlocks of 2024-25 — not because the model is smarter, but because audio bandwidth changes which jobs an agent can plausibly do.
Recurring coverage on simonwillison.net and his joint year-in-review with Swyx on Latent Space; phrased as a position because it spans many posts rather than a single quote.
Platform Deep-Dive
How the Frontier Labs Implement Voice in 2026
The four providers Susan benchmarks against on every sales call — Claude (Anthropic), OpenClaw, Codex/GPT-5.4 (OpenAI), Gemini 3.1 (Google) — have each picked a slightly different lane. Understanding the trade-offs is the difference between repeating marketing copy and being credible in a CTO meeting.
If a customer asks “which one should we build on?”, the honest answer in 2026 is: build the orchestration layer (LiveKit or Pipecat) around a swappable provider, then pick OpenAI for raw quality, Gemini for vision-rich flows, and Claude/OpenClaw when reasoning depth + tool-use safety dominates over raw latency.
CGO Lens
What Voice Agents Mean for botlearn.ai
$Voice unlocks a new revenue surface for tutoring
Text tutoring caps at the user’s typing speed and patience. A voice tutor that speaks back at conversational latency turns a chat product into something that competes with a human tutor — the same shift Khan Academy’s Khanmigo has been chasing. botlearn.ai’s upsell story becomes “text tier → voice tier” with a clear price ladder, because each voice minute carries 30× the API cost of a text exchange and customers will pay for it.
Pricing arithmetic to remember in customer calls: $64 / M output audio tokens at gpt-realtime ≈ $0.06 per minute of agent speech. A 20-minute lesson costs ~$1.20 in raw model cost; package at $5/lesson and you have ~75% margin pre-amortisation.
!The S2S vs cascade decision is your platform bet
If botlearn.ai picks gpt-realtime exclusively, you inherit OpenAI’s pricing and policy curves. If you pick Pipecat or LiveKit with swappable backends, you defer the choice and gain leverage in vendor negotiations — but you take on the orchestration burden. The right move for a CGO is to push engineering toward the orchestration layer in 2026 H1, then lock in a primary provider once the price sheets stabilise.
✓Customer success cares about three numbers
(1) p95 turn latency — if it crosses 700 ms, NPS collapses. (2) Barge-in success rate — how often the agent stops cleanly when interrupted; should be >95%. (3) Voice-to-text fallback rate — what fraction of users abandon voice for text, your reverse churn signal. Build these three into the dashboard you show enterprise buyers; they’ll all ask within the first ten minutes.
·Competitive positioning
The legacy IVR vendors (Five9, NICE, Genesys) are racing to bolt LLMs onto cascades they already own. The pure-AI entrants (Vapi, Bland, Retell) are racing to wrap the Realtime API. botlearn.ai’s wedge is neither — it is education-domain voice, where the right agent has to track curriculum state, learner mastery, and pedagogical strategy across multi-session voice histories. That’s a memory + voice + RAG problem (Days 2, 4, 20), not a generic call-centre problem. Sell against the IVRs on quality and against the wrappers on durability.
✓Sales objection handler
“Why not just use GPT-realtime directly?” — Because the value sits in the curriculum graph, the spaced-repetition scheduler, the voice persona that adapts to learner age, and the parental safety layer. The model is a commodity input; the product is everything wrapped around it. Same answer Notion gives when prospects ask why not just use a raw LLM.
Curriculum Tracker
Where We Are — Day 20 of 17 + 3 Bonus
D01Full Agent Stack
D02Memory Architecture
D03Planning & Tool Use
D04RAG Deep Dive
D05Agent Frameworks
D06Benchmarks & Eval
D07Multi-Agent Systems
D08Computer Use Agents
D09Code Agents
D10Long-Horizon Tasks
D11Agent Safety
D12Agent Economics
D13Research Frontiers
D14OpenClaw Deep-Dive
D15A2A Protocols
D16Agentic Commerce
D17Synthesis
D18Reasoning & Test-Time Compute
D19Agent Observability
D20Voice & Realtime Agents · today
Three weeks in. The curriculum has now covered the full agent surface a CGO needs to hold a peer-level conversation with a frontier-lab engineer: cognition (planning, reasoning, memory), I/O (tools, computer-use, voice), operations (eval, observability, economics), and ecosystem (frameworks, A2A, OpenClaw, commerce). Day 21 is open: candidate topics include verifier models & process reward, agent fine-tuning & distillation, embodied/robotic agents, or a final capstone — pick based on which conversation feels like it would land best with an inbound investor or partner this week.