Moshi Guide: Real-Time Full-Duplex Speech-Text AI

Problem: The Cascaded Voice Pipeline Latency and Turn-Based Bottleneck

Conversational voice interaction is widely recognized as the most intuitive, natural interface between humans and intelligent systems. When operating high-throughput enterprise platforms—whether resolving real-time user inquiries from data pipelines established via real-time streaming data ingestion or deploying customer-facing conversational assistants informed by automated data extraction pipelines—voice provides unmatched immediacy.

Yet, despite massive advances in Large Language Models (LLMs), conversational voice agents in commercial deployment have historically suffered from an artificial, frustrating, and mechanical user experience. Users speak, wait in silence for multiple seconds, listen to a rigid monologue, and find themselves unable to interrupt, laugh, or offer simple conversational acknowledgments without derailing the system.

This fundamental breakdown stems from the ubiquitous architectural paradigm that has dominated voice engineering for over a decade: the cascaded turn-based pipeline.

The Compounding Latency Crisis

In human interpersonal communication, conversational turn-taking is remarkably fast. Psycholinguistic research across diverse languages demonstrates that the average gap between conversational turns in natural dialogue is approximately 230 milliseconds (Stivers et al., 2009). When a gap exceeds 500 milliseconds, human listeners subconsciously perceive awkwardness, hesitation, or cognitive disconnect.

Cascaded voice pipelines violate this physiological expectation by design. Because each module is decoupled, latency compounds additively across three independent stages:

1. Automatic Speech Recognition (ASR) Delay: The system cannot begin transcribing audio accurately until Voice Activity Detection (VAD) confirms that the speaker has completed an utterance. This requires a silence threshold (typically 300ms to 700ms) to avoid cutting the speaker off mid-sentence. Once silence is detected, acoustic feature extraction and autoregressive token decoding add another 200ms to 400ms.
2. Text LLM Processing Latency: The transcribed text must be wrapped in system instructions, context history, and tool schemas before being ingested by the LLM. Time-to-First-Token (TTFT) on a 70B parameter model typically consumes 200ms to 500ms, followed by token generation time.
3. Text-to-Speech (TTS) Buffering Delay: Modern neural TTS engines (such as diffusion vocoders or autoregressive audio transformers) cannot synthesize natural prosody from single isolated words. They require complete phrases or punctuation-delimited clauses (4 to 8 words) before commencing audio synthesis, adding an unavoidable buffering delay of 300ms to 600ms before streaming audio begins.

When these independent latencies are combined, even an optimized enterprise cascade exhibits an end-to-end response time between 1.5 and 3.0 seconds. This delay shatters conversational flow, turning dialogue into a transactional walkie-talkie exchange.

The Turn-Based Half-Duplex Trap

Beyond latency, the cascaded approach enforces a rigid half-duplex turn-taking constraint:

• Discrete Speaker Ownership: The architecture assumes that a conversation is an alternating sequence of discrete, non-overlapping turns: Speaker A speaks while Speaker B listens, then Speaker B speaks while Speaker A listens.
• Barge-In Pathology: If the user speaks while the synthetic TTS voice is playing, the system must trigger a blunt “barge-in” mechanism that violently mutes the audio channel and flushes the LLM generation buffer. It cannot distinguish between a genuine interruption (“Stop, that’s incorrect!”) and a conversational affirmation (“Right, I agree”).
• Absence of Backchanneling: In natural human conversation, between 10% and 20% of spoken time consists of overlapping speech (Çetin and Shriberg, 2006). Attentive listeners continuously emit non-interruptive backchannels—such as “uh-huh,” “yeah,” “mm-hmm,” and slight head gestures—to communicate engagement without seizing the conversational floor. Cascaded systems are structurally incapable of backchanneling; any sound from the listener either halts the agent or is discarded as background noise.

The Paralinguistic Information Bottleneck

The most insidious limitation of the cascaded architecture is the textual bottleneck.

Speech carries vast amounts of communicative data that cannot be captured in a textual transcript:

• Emotional State & Tone: Sarcasm, urgency, frustration, confidence, and warmth are conveyed through pitch contours, vocal timbre, and speech rate. When an ASR system transcribes “Sure, that sounds great,” the LLM receives identical tokens whether the phrase was uttered with genuine enthusiasm or biting cynicism.
• Non-Verbal Vocalizations: Laughter, sighs, throat clearing, hesitations (“um,” “uh”), and breath patterns modulate the meaning of words. ASR pipelines either sanitize these vocalizations or misinterpret them as phonetic errors.
• Acoustic Environment: Reverberation, background voices, and ambient soundscapes contextualize the speaker’s environment, but are obliterated when converted to ASCII strings.

To resolve these structural bottlenecks, researchers at Kyutai—an open-science AI research lab in Paris—developed Moshi: a speech-text foundation model for real-time dialogue (arXiv:2410.00037). Moshi discards the cascaded pipeline entirely, unifying speech comprehension and speech synthesis into a single, end-to-end, multi-stream foundation model capable of full-duplex communication with sub-200ms latency.

---

Solution: End-to-End Multi-Stream Full-Duplex Speech Architecture

Moshi reconceptualizes conversational AI from first principles. Rather than converting audio to text, passing text to an LLM, and converting text back to audio, Moshi casts spoken dialogue as direct speech-to-speech autoregressive sequence modeling.

Core Tenets of the Moshi Paradigm

Moshi’s breakthrough rests upon four foundational architectural innovations:

1. Multi-Stream Token Representation: Instead of merging user and system utterances into a single serialized string of words, Moshi models two parallel audio streams continuously. At every 80ms time frame, the model ingests 8 discrete tokens representing user audio and generates 8 discrete tokens representing its own speech. Because both streams exist simultaneously, there are no speaker turns. The model naturally handles silence, single-speaker speech, overlapping interruptions, and conversational backchanneling.

2. The Mimi Neural Audio Codec: To enable autoregressive modeling over speech without exploding sequence length, Kyutai developed Mimi, an ultra-low-latency neural audio codec that operates at a remarkably low frame rate of 12.5 Hz (one frame every 80 milliseconds). Mimi employs a split Residual Vector Quantization (RVQ) scheme that distills semantic knowledge from WavLM into its first codebook while capturing fine acoustic detail across 7 subsequent codebooks, achieving pristine 24 kHz audio synthesis at a bandwidth of only 1.1 kbps.

3. Dual-Transformer Hierarchy: Rather than forcing a massive 7B parameter model to predict 16+ audio codebooks sequentially at every time step, Moshi splits compute between:

• A Temporal Transformer (the Helium 7B backbone) that attends across multi-minute conversational history along the time axis ($T$).
• A lightweight Depth Transformer (6 layers, 1024 hidden dimensions) that autoregressively predicts the hierarchical tokens (text, semantic audio, and fine acoustic codebooks) within each individual time step ($t$).

4. Inner Monologue Co-Speech Reasoning: Unconditioned speech language models often generate phonetic babble and hallucinate facts because audio tokens lack the abstract syntactic density of text. Moshi solves this by introducing Inner Monologue: before predicting the audio tokens for a given 80ms frame, the model predicts the corresponding time-aligned text token. The text token acts as an autoregressive prefix that guides semantic and acoustic generation, allowing Moshi to leverage the deep factual knowledge of its 7B text LLM backbone while outputting native speech.

---

Architecture: Helium Backbone, Mimi Codec, and Dual-Transformer Hierarchy

To understand how Moshi achieves streaming real-time performance, we must examine its three structural components: the Helium language model backbone, the Mimi neural audio codec, and the Dual-Transformer generation engine.

The Helium 7B Text LLM Backbone

The intellectual engine powering Moshi is Helium, a 7-billion-parameter text language model trained from scratch by Kyutai on 2.1 trillion tokens of curated English text.

Helium adheres to modern transformer best practices:

• Normalization: Root Mean Square Layer Normalization (RMSNorm) applied at the input of attention blocks, feed-forward networks, and output projection layers.
• Activation Functions: Gated Linear Units with Sigmoid Linear Units (SwiGLU) for enhanced expressiveness in MLP projections.
• Positional Encodings: Rotary Position Embeddings (RoPE) enabling smooth length extrapolation across extended sequence windows.
• Attention Mechanism: FlashAttention integration for memory-efficient computation.
• Tokenizer: Unigram SentencePiece vocabulary containing 32,000 tokens, with numbers split into single digits and byte-fallback encoding to prevent out-of-vocabulary serialization failures.

During conversational inference, Helium’s context window spans 3,000 temporal steps. Because each step represents an 80ms audio slice, the model maintains active temporal awareness across 4 minutes of continuous, unbroken dialogue.

The Mimi Neural Audio Codec: Semantic-Acoustic Split RVQ

Audio language modeling has historically faced an intractable dilemma:

• Semantic Tokens (e.g., w2v-BERT, HuBERT, WavLM): Carry rich linguistic information, making them easy for an LLM to predict, but discard speaker identity, pitch, emotion, and acoustic quality. They cannot reconstruct high-fidelity audio.
• Acoustic Codecs (e.g., EnCodec, SoundStream): Reconstruct pristine 24 kHz audio using Residual Vector Quantization (RVQ), but their codebooks are linguistically disorganized, causing LLMs to generate unintelligible garble.

Kyutai resolved this dilemma through Mimi, an innovative streaming codec featuring semantic distillation into a split RVQ:

By distilling WavLM features into the very first quantization layer ($Q_0$), Mimi ensures that token 0 represents pure semantic content. The subsequent 7 quantization layers ($Q_1$ through $Q_7$) model acoustic residuals. This allows Moshi to predict language structure and acoustic timbre in a unified, hierarchical framework.

Dual-Transformer Hierarchy: Conquering the Combinatorial Explosion

In a full-duplex setup modeling user audio, system audio, and text, each 80ms timestep contains:

• 1 User Text Token (optional)
• 8 User Audio Tokens ($U_0 \dots U_7$)
• 1 System Text Token (Inner Monologue)
• 8 System Audio Tokens ($M_0 \dots M_7$)

Predicting all 18 tokens sequentially inside a 7B parameter autoregressive model would increase sequence length by a factor of 18, crushing inference throughput.

To solve this, Moshi adopts a Dual-Transformer hierarchy:

1. The Temporal Transformer (Helium 7B): Operates exclusively along the time axis ($t = 1 \dots T$). At each step $t$, it consumes the combined representation of the preceding frame’s tokens and outputs a rich contextual hidden state $H_t \in \mathbb{R}^{4096}$.
2. The Depth Transformer (Small 6-Layer Transformer): Operates within the current timestep ($t$). Conditioned on $H_t$, it autoregressively predicts the text token, then semantic token $M_0$, then acoustic tokens $M_1 \dots M_7$.

This division of labor keeps the massive 7B model running at a relaxed 12.5 Hz clock speed while the tiny 6-layer model rapidly handles intra-frame token dependencies in parallel.

---

Implementation: Inner Monologue Alignment and Streaming Multi-Stream Inference

To understand how Moshi synchronizes text reasoning with raw acoustic generation, we must explore the mechanics of Inner Monologue and examine a functional PyTorch reference implementation.

Inner Monologue and Token Delay Dynamics

Human speakers do not formulate speech phoneme-by-phoneme in acoustic isolation; the brain plans semantic word concepts milliseconds before motor neurons engage the vocal cords.

Inner Monologue replicates this cognitive architecture:

• At timestep $t$, the Depth Transformer first emits a text token from Helium’s 32,000-token vocabulary.
• If a word spans multiple 80ms frames, subsequent frames emit a special padding token (<pad>), maintaining strict time alignment with speech audio.
• The emitted text token is projected into the Depth Transformer’s state, directly conditioning the emission of semantic token $M_0$ and acoustic tokens $M_1 \dots M_7$.

Furthermore, Kyutai introduced an Acoustic Delay ($\tau_{\text{audio}}$) and a Text Delay ($\tau_{\text{text}}$) during training. Setting an acoustic delay of 1 frame ($\tau_{\text{audio}} = 1$) ensures that the model observes user audio tokens at frame $t$ before finalizing its own acoustic response at frame $t + 1$. This subtle delay provides the transformer with causal foresight, drastically reducing phonemic stuttering and audio artifacts.

PyTorch Architecture Implementation

The following complete PyTorch module demonstrates the multi-stream conditioning loop, the Depth Transformer execution, and the Inner Monologue token generation pipeline:

---

Comparison: Cascaded Pipelines vs. End-to-End Voice AI Benchmarks

To quantify the performance gains realized by Moshi’s end-to-end architecture, we contrast it against traditional cascaded systems, proprietary speech models (such as GPT-4o Voice and Gemini Live), and earlier open-source research prototypes (Mini-Omni, SpeechGPT, dGSLM).

Empirical Benchmark Findings

In empirical evaluations conducted across spoken reasoning, speech intelligibility, and acoustic quality benchmarks, Moshi demonstrates remarkable capabilities:

1. Spoken Question Answering (Spoken SQuAD & CoQA):

• Evaluated on Spoken SQuAD, Moshi achieves an Exact Match (EM) score of 71.4% and an F1 score of 80.2%.
• Prior speech-to-speech models that lacked an Inner Monologue mechanism (such as Spectron and SpeechGPT) scored under 58% F1 due to phonetic divergence during multi-sentence generation.
• By prefixing audio generation with time-aligned text tokens, Moshi matches the factual question-answering accuracy of pure text LLMs while maintaining live speech delivery.

2. Speech Intelligibility & Word Error Rate (WER):

• When transcribing Moshi’s generated audio through an independent Whisper-Large-v3 ASR model, Moshi achieves a Word Error Rate of 4.2% on conversational English.
• Unconditioned audio language models (AudioLM, TWIST) exhibit WERs exceeding 18.5% due to phonetic drift over extended sequences. Moshi’s semantic WavLM distillation in Mimi Codebook 0 ensures crisp consonantal boundaries and natural vowel articulation.

3. Audio Reconstruction & Naturalness (UTMOS & PESQ):

• Mimi achieves a Perceptual Evaluation of Speech Quality (PESQ) score of 3.42 and a UTMOS naturalness score of 4.15 at just 1.1 kbps.
• In comparison, EnCodec requires 6.0 kbps to achieve a 3.38 PESQ score. Mimi’s adversarial training loss and causal transformer bottleneck deliver superior subjective audio fidelity at less than one-fifth the transmission bitrate.

---

Production: Deployment Strategies, Quantization, and Real-Time Serving

Deploying an end-to-end full-duplex conversational model in enterprise production introduces unique infrastructure considerations. Because Moshi continuously streams bidirectional audio at 12.5 Hz, standard HTTP request-response patterns are obsolete. Production deployments must leverage low-latency protocols, GPU kernel optimizations, and intelligent memory management.

Quantization and VRAM Budgeting

The standard FP16 checkpoint of Moshi Helium 7B requires approximately 15.5 GB of VRAM for weights alone, plus 2 to 4 GB for KV cache allocation across active sessions.

To maximize concurrency and enable deployment on cost-effective enterprise hardware (such as single Nvidia L4 or RTX 4090 GPUs), Moshi supports modern post-training quantization:

• INT4 Weight-Only Quantization (AWQ / Marlin): Compresses model weight footprint to 4.2 GB. Executes matrix-vector multiplications via optimized Marlin FP16xINT4 kernels, maintaining full 16-bit activation fidelity. Reduces VRAM consumption by 72% with less than 0.8% degradation in Spoken SQuAD accuracy.
• FP8 Quantization (Ada Lovelace & Hopper Architectures): Supported natively on Nvidia H100, L4, and RTX 4090. Fits the full model and a 16-session concurrent KV cache into 8.5 GB of VRAM. Eliminates dequantization overhead, accelerating temporal forward passes by up to 1.8x.

Managing Streaming Jitter and Buffer Underruns

In a full-duplex system operating at 80ms per frame, any network jitter or scheduling delay will cause audible clicks or dropouts in user audio:

• Jitter Buffer Configuration: The WebRTC gateway must maintain an adaptive 40ms to 80ms jitter buffer. Because Mimi’s causal lookahead requires one future frame for optimal vocoding, aligning the network buffer with the codec lookahead introduces zero net perceptual penalty.
• Silence Masking: When network packets are lost, the gateway injects Mimi padding tokens corresponding to neutral ambient silence rather than zeroing the waveform, preventing harsh acoustic pops.

Grounding and Knowledge Retrieval

While Moshi processes audio end-to-end, enterprise dialogue systems frequently require access to live external databases.

Developers can integrate live knowledge through a hybrid retrieval loop:

1. As Moshi’s Inner Monologue emits text tokens in real time, a background daemon parses completed noun phrases and entities.
2. The daemon queries indexed vector stores or enterprise data pipelines built on custom API integration endpoints.
3. Relevant factual snippets are injected into Helium’s temporal context buffer as hidden prompting signals, dynamically updating Moshi’s factual grounding without interrupting ongoing vocal delivery. For organizations scaling multimodal agent infrastructure, consulting production infrastructure tiers and reviewing developer integration documentation ensures optimal hardware sizing and deployment reliability.

By merging the expressive power of neural audio codecs with the intellectual depth of 7B language backbones, Moshi establishes the blueprint for the next generation of conversational AI: real-time, emotionally aware, and truly full-duplex.