moshikyutaispeech-aimimifull-duplexvoice-agentsaudio-modelstutorial

Moshi Guide: Full-Duplex Real-Time Speech AI Architecture

September 27, 2026

Moshi Architecture: Full-Duplex Native Speech AI vs Cascaded Half-Duplex Pipelines

The Problem: The Latency and Half-Duplex Traps of Cascaded Voice Pipelines

In modern enterprise automation, customer engagement platforms, and multimodal agent environments like AiFetchly, voice interfaces are rapidly transitioning from scripted IVR menus to autonomous conversational assistants. However, despite rapid advances in Large Language Models (LLMs), interacting with voice assistants in production has historically felt unnatural, robotic, and sluggish.

For nearly a decade, conversational voice systems have relied on a cascaded, half-duplex architecture: chaining three or four disconnected machine learning models across a sequential pipeline.

The Traditional Cascaded Voice Pipeline:
[User Speech] ──► [VAD: Wait 500ms Silence] ──► [Whisper STT (300-500ms)]
                                                        │
[User Hears Audio] ◄── [TTS Vocoder (300-500ms)] ◄── [Text LLM (600-1000ms)]
Total Round-Trip Latency: 1,700ms - 2,500ms! (Strictly Turn-Based)

While individual components in this cascade have become increasingly capable, the composite architecture suffers from three fatal systemic bottlenecks:

1. The Cascaded Latency Tax (The 2,000ms Awkward Pause)

Human conversation operates under tight cognitive timing constraints. Psycholinguistic research demonstrates that the typical gap between turns in natural human dialogue is 200 milliseconds. When conversational latency exceeds 600 milliseconds, users perceive the interaction as disconnected; at 1,500 milliseconds, users assume the system crashed or disconnect the call.

In a cascaded architecture, latency compounds across every sequential stage:

Voice Activity Detection (VAD) Silence Penalty (400–700ms): To determine that a user has finished speaking, the VAD must observe a mandatory silence threshold. If set too short (e.g., 200ms), the system interrupts users who pause to think. If set conservatively (e.g., 600ms), this delay is permanently baked into every conversational turn before processing even begins.
Speech-to-Text (STT) Ingestion (250–500ms): ASR models like Whisper or Conformer must ingest the audio buffer, compute mel-spectrograms, and decode the text transcription.
Large Language Model TTFT & Generation (400–800ms): The text model evaluates the prompt, computes attention over context history, and streams tokens.
Text-to-Speech (TTS) Synthesis (300–600ms): Neural TTS engines must accumulate an initial clause or sentence fragment, generate acoustic tokens or mel-spectrograms, and pass them through a vocoder.

The composite end-to-end latency reliably lands between 1,500ms and 2,500ms, forcing an artificial, halting conversational rhythm.

2. The Half-Duplex Turn-Taking Straightjacket (Audio Collisions)

Traditional pipelines are fundamentally half-duplex: the system can either listen or speak, but never both at the same time:

Audio Collisions: If a user speaks while the model is playing synthesized audio, the VAD either suffers from acoustic echo leakage or violently cuts off the agent’s playback mid-word.
Inability to Handle Barge-Ins: When humans talk, we constantly execute subtle conversational negotiations: interjecting clarifying questions, adjusting our volume, or acknowledging understanding. Cascaded pipelines cannot process user speech concurrently with agent speech; any interruption requires completely discarding the active generation graph.
Absence of Backchanneling: Natural dialogue relies heavily on non-intrusive vocal feedback (“uh-huh”, “right”, “yeah”, “mhm”) that signals active listening without claiming conversational turns. Half-duplex architectures treat every incoming syllable as a demand for a full LLM inference cycle.

3. The Prosody and Emotion Stripping Bottleneck

Language models operating on raw text strings are completely blind to non-verbal vocal acoustics:

Information Loss at the Transcription Boundary: When speech is collapsed into plain UTF-8 text strings, vital communicative signals are permanently erased: pitch, cadence, vocal stress, sarcasm, hesitation, frustration, whisper, and breathing.
Flat Synthetic Monotony: Because the LLM only outputs punctuation and words, the downstream TTS engine must invent synthetic prosody from scratch. The agent sounds artificially cheerful or uniformly robotic, failing to mirror the emotional state of the speaker.

The Architecture: Kyutai Moshi’s Native Full-Duplex Paradigm

To eliminate the cascaded latency wall and restore the natural dynamics of human speech, Kyutai (a French non-profit AI research lab founded by Xavier Niel, Rodolphe Saadé, and Eric Schmidt) developed Moshi (Défossez, Copet, et al., late 2024 / 2025).

Moshi is the world’s first fully open-source, full-duplex speech-text foundation model. Instead of cascading separate models, Moshi integrates audio encoding, linguistic reasoning, and acoustic synthesis into a unified end-to-end neural network.

The Moshi Full-Duplex Architecture:
User Mic (24kHz) ──► [Mimi Audio Encoder] ──► 12.5 Hz Tokens (8 RVQ streams) ──┐
                                                                               ▼
Agent Speaker ◄── [Mimi Audio Decoder] ◄── 12.5 Hz Tokens (8 RVQ streams) ◄── [Helium 7B
                                                ▲                              Multi-Stream
Agent Text Out ─────────────────────────────────┴── Inner Monologue Stream ── Transformer]
End-to-End Latency: ~160ms - 200ms (Simultaneous Full-Duplex Listening & Speaking!)

Moshi is built upon three foundational architectural breakthroughs:

1. Mimi: The 12.5 Hz Neural Audio Codec

The core challenge of processing audio with transformer foundation models is token rate. Standard CD-quality audio (24 kHz or 48 kHz) consists of tens of thousands of samples per second. Previous neural audio codecs (such as Meta’s EnCodec or SoundStream) produced token frame rates of 50 Hz to 75 Hz. Feeding 75 audio tokens per second into a multi-billion-parameter transformer exhausts GPU memory bandwidth and explodes Key-Value (KV) cache memory.

Kyutai solved this with Mimi, an ultra-low-framerate neural audio tokenizer:

12.5 Hz Temporal Resolution: Mimi compresses 24 kHz mono audio into discrete token frames every 80 milliseconds (12.5 frames per second)—a 4x to 6x reduction compared to EnCodec.
1,920x Downsampling Factor: The encoder utilizes a SeaNet-inspired convolutional backbone with strided 1D convolutions and Transformer downsamplers to compress raw waveforms directly.
Residual Vector Quantization (RVQ): Within each 80ms frame, Mimi uses 8 hierarchical codebooks (each with a codebook size of 2,048 entries), yielding a bandwidth of only 1.1 kbps.
Semantic-Acoustic Distillation: Uniquely, the first quantizer ($Q_1$) in Mimi’s RVQ hierarchy is trained with a distillation loss against self-supervised WavLM representations. This ensures that the primary codebook captures rich linguistic semantics, while subsequent quantizers ($Q_2$ through $Q_8$) encode acoustic details, speaker timbre, and spatial acoustics.

2. Helium 7B: The Multi-Stream Transformer Backbone

The cognitive engine of Moshi is Helium, a 7-billion-parameter decoder-only transformer pre-trained on 2.1 trillion tokens of multilingual text.

Rather than processing a single 1D sequence of text tokens, Helium is engineered to model multiple synchronized streams in parallel. At each 80ms time step $t$, the transformer simultaneously ingests:

User Audio Stream: 8 discrete Mimi tokens representing user speech incoming from the microphone.
Agent Audio Stream: 8 discrete Mimi tokens representing the model’s own outgoing vocalization.
Agent Text Stream (Inner Monologue): Standard text subword tokens representing the linguistic transcription of what the model is saying.

By maintaining unified causal self-attention across these parallel tracks, Helium continuously conditions its speech generation on incoming user audio while it is actively talking.

3. Inner Monologue: Anchoring Acoustic Generation in Textual Reasoning

Early experiments in training end-to-end speech models directly on audio tokens revealed a major weakness: semantic drift. Because audio waveforms contain immense high-frequency entropy (background hiss, vocal timbre, room reverberation), models trained solely on audio tokens quickly descend into “acoustic babbling”—producing grammatically meaningless syllables with convincing prosody.

Moshi solves this through Inner Monologue:

At each time step, Helium predicts its own text transcript as an internal reasoning stream slightly ahead of its acoustic codebook tokens.
Predicting the text token acts as a semantic anchor, allowing Moshi to leverage the deep world knowledge and reasoning capabilities acquired during Helium’s 2.1T text pre-training.
The acoustic generation heads ($Q_1 \dots Q_8$) are conditioned directly on the hidden states of this text prediction, ensuring that every spoken syllable aligns with a coherent textual thought.

Core Mechanics: Multi-Stream Interleaving and Residual Vector Quantization

To understand how Moshi achieves full-duplex communication without audio collisions, we must examine the token alignment mechanics and delay schedules that synchronize user audio, model text, and model speech.

Moshi Multi-Stream Interleaving and Mimi Codec Mechanics

1. Multi-Stream Token Alignment and Time Delays

In standard autoregressive language modeling, predicting token $y_t$ relies on historical tokens $y_{<t}$. In conversational speech, however, physical acoustics require lookahead: humans formulate an idea linguistically before vocal cords vibrate.

Moshi establishes a time-delayed multi-stream matrix:

Time Step (80ms frames):   t=0        t=1        t=2        t=3        t=4
User Audio (In):           U_0        U_1        U_2        U_3        U_4
Inner Monologue (Text):    [PAD]      [PAD]      "Hello"    ","        "how"
Agent Audio (Out):         [PAD]      [PAD]      [PAD]      A_0        A_1

Inner Monologue Text Delay: Text tokens are generated with a deliberate temporal offset, appearing one to two frames ahead of the corresponding acoustic tokens. This gives the transformer time to formulate semantic intent before committing to acoustic waveform synthesis.
Acoustic Token Staggering (Delayed RVQ): Emitting 8 codebook tokens ($Q_1 \dots Q_8$) per frame simultaneously would require sampling an 8-way joint distribution. Moshi uses a delayed autoregressive factorization: $Q_1$ is predicted first, $Q_2$ is predicted conditioning on $Q_1$, and so forth, or offsets codebooks across small sub-frame steps.

2. Asymmetric Dual-Stream Audio Modeling

Moshi maintains two distinct operational tracks in memory:

Stream 1: User Acoustic Channel
Input: Audio from client microphone -> Streamed into Mimi Encoder -> RVQ Tokens (U_t)
Attention: Model's query vectors attend to historical user tokens via causal attention masks.

Stream 2: Agent Acoustic Channel
Output: Helium hidden states -> Multi-head linear classifiers -> RVQ Tokens (A_t)
Feedback: Agent tokens are fed back autoregressively into the model at step t+1.

Because both channels exist inside the same attention context, Moshi continuously computes cross-correlations between what the user is saying and what it is synthesizing. If the user begins speaking while the model is generating audio, the attention heads immediately register incoming non-zero acoustic energy.

3. Natural Barge-In, Interruption Handling, and Backchanneling

In traditional voice bots, interruption handling requires an external software circuit: stopping playback, clearing audio buffers, and flushing state.

In Moshi, interruption is an emergent behavior of the generative policy:

Continuous Listening During Generation: Because user audio tokens stream into Helium every 80ms regardless of whether the agent is speaking, the model detects speech onset immediately.
Autonomous Yielding: Helium is fine-tuned on naturalistic conversational datasets (such as Fisher and Kyutai’s synthetic multi-speaker dialogues). The model learns that when the user speaks with rising intonation or authoritative cadence, the highest-probability next token for the agent stream is the silence/pause token.
Natural Backchanneling: When the user is delivering a long narrative, Moshi does not sit in dead silence. Instead, the model spontaneously emits short, low-energy affirmative vocables (“mhm”, “yeah”, “I see”) into the agent channel without derailing the user’s ongoing speech.

Implementation: Streaming Full-Duplex Speech with Moshi in Python

Deploying Moshi locally or on an inference server allows developers to build low-latency voice assistants without proprietary third-party APIs. Below is a complete, runnable PyTorch implementation demonstrating how to load Kyutai’s Mimi audio codec and Moshi language model, process live audio chunks, and stream synthesized responses.

Prerequisites

Ensure you have PyTorch, Hugging Face Transformers, and the Kyutai client packages installed:

pip install torch torchaudio transformers moshi numpy

Complete Full-Duplex Streaming Pipeline

import torch
import numpy as np
from moshi.models import loaders

class MoshiVoiceAgent:
    def __init__(self, device: str = "cuda"):
        self.device = torch.device(device if torch.cuda.is_available() else "cpu")
        print(f"Initializing Moshi engine on {self.device}...")

        # Load Mimi neural audio codec (12.5 Hz frame rate, 24 kHz audio)
        self.mimi = loaders.get_mimi(loaders.MIMI_NAME, device=self.device)
        self.mimi.eval()

        # Load Moshi 7B foundation model (Helium backbone)
        self.moshi = loaders.get_moshi(loaders.MOSHI_NAME, device=self.device)
        self.moshi.eval()

        # Audio parameters
        self.sample_rate = 24000
        self.frame_size = int(self.sample_rate / 12.5)  # 1920 samples = 80ms

        # Initialize streaming state buffers
        self.reset_session()
        print("Moshi Voice Engine ready for streaming!")

    def reset_session(self):
        # Initializes stateful KV-caches and multi-stream context
        self.mimi_state = self.mimi.init_state(batch_size=1)
        self.moshi_state = self.moshi.init_state(batch_size=1)
        self.is_interrupted = False

    @torch.inference_mode()
    def process_audio_chunk(self, audio_chunk_pcm: np.ndarray) -> tuple[np.ndarray, str]:
        # Processes an 80ms chunk of incoming 24kHz mono PCM audio
        assert len(audio_chunk_pcm) == self.frame_size, (
            f"Expected {self.frame_size} samples (80ms), got {len(audio_chunk_pcm)}"
        )

        # 1. Format input waveform: shape (1, 1, 1920)
        chunk_tensor = torch.from_numpy(audio_chunk_pcm).float().to(self.device)
        chunk_tensor = chunk_tensor.unsqueeze(0).unsqueeze(0)

        # 2. Encode user audio into discrete tokens via Mimi (8 RVQ codebooks)
        # user_tokens shape: (1, 8, 1) -> 8 codebook tokens for this 80ms frame
        user_tokens = self.mimi.encode_step(chunk_tensor, self.mimi_state)

        # 3. Step Helium multi-stream transformer
        # Ingests user audio tokens and emits text and agent audio tokens
        step_output = self.moshi.step(
            user_tokens=user_tokens,
            state=self.moshi_state
        )

        # 4. Extract Inner Monologue text token
        text_token_id = step_output.text_token
        decoded_text = self.moshi.tokenizer.decode([text_token_id])

        # 5. Decode agent acoustic tokens into raw audio waveform via Mimi
        agent_tokens = step_output.agent_tokens # shape: (1, 8, 1)
        synthesized_waveform = self.mimi.decode_step(agent_tokens, self.mimi_state)

        # 6. Convert synthesized tensor to numpy PCM float32
        out_pcm = synthesized_waveform.squeeze().cpu().numpy()

        return out_pcm, decoded_text

if __name__ == "__main__":
    agent = MoshiVoiceAgent(device="cuda")

    # Simulate 3 seconds of streaming audio (37 frames of 80ms each)
    total_frames = 37
    print(f"Simulating live audio stream ({total_frames} frames)...")

    full_inner_monologue = []

    for frame_idx in range(total_frames):
        # Generate simulated microphone input (e.g., audio tone or silence)
        simulated_mic = np.random.uniform(-0.05, 0.05, size=agent.frame_size).astype(np.float32)

        # Execute 80ms streaming step
        agent_audio, text_fragment = agent.process_audio_chunk(simulated_mic)

        if text_fragment.strip():
            full_inner_monologue.append(text_fragment)

        if frame_idx % 10 == 0:
            print(f"Frame {frame_idx:02d}/37: Emitted 80ms audio, Latency ~160ms")

    print("
Synthesized Inner Monologue transcript:")
    print("".join(full_inner_monologue))

Architectural Comparison: Moshi vs. Cascaded Pipelines vs. GPT-4o Audio

To evaluate where Moshi fits within the speech artificial intelligence landscape, consider how it compares against traditional modular cascades and proprietary cloud solutions across core engineering metrics:

Architectural Dimension	Traditional Cascaded Pipeline	OpenAI GPT-4o (Native Audio)	Kyutai Moshi (Open Foundation Model)
System Paradigm	Cascaded (VAD $\to$ STT $\to$ LLM $\to$ TTS)	Proprietary End-to-End Multimodal	Open-Weights End-to-End Multimodal
Duplex Capability	Strict Half-Duplex (Turn-based)	Native Full-Duplex	Native Full-Duplex
Typical End-to-End Latency	1,500ms – 2,800ms	250ms – 400ms	160ms – 220ms
Audio Tokenizer	N/A (Operates on text / spectrograms)	Proprietary Neural Audio Codec	Mimi (12.5 Hz, 8-layer RVQ, 1.1 kbps)
Language Backbone	Any standard text LLM (e.g., Llama 3)	Proprietary Frontier Multimodal	Helium 7B (Trained on 2.1T tokens)
Reasoning Anchoring	Pure text generation	Unified multimodal attention	Inner Monologue (Synchronized Text Stream)
Interruption Handling	Brittle software abort / VAD mute	Seamless conversational barge-in	Native autoregressive yielding & barge-in
Vocal Expressivity	Flat synthetic TTS prosody	Rich, expressive prosody	Rich prosody, whispers, laughter, sighs
Hosting & Privacy	Multi-vendor cloud / API endpoints	Closed proprietary cloud API	100% Local / Self-Hosted on single GPU
Hardware Footprint	Distributed microservices	Unknown cloud cluster	Single NVIDIA GPU (24GB VRAM: RTX 4090)

Production Engineering: Deploying Moshi in Real-Time Agent Stacks

Transitioning full-duplex speech AI from a research artifact to a production-grade enterprise system requires addressing several high-concurrency infrastructure challenges:

1. WebRTC Ingestion and Jitter Buffering

Unlike REST APIs or standard text SSE streaming, real-time voice demands continuous bidirectional UDP streaming:

WebRTC Data & Media Tracks: Voice streams should be ingested via WebRTC (using protocols like Opus audio compression at 24 kHz mono).
Chunk Reassembly: Client audio packets must be buffered into strict 80ms windows (1,920 PCM samples). A small jitter buffer (20–40ms) is required to absorb network latency variance without introducing perceptible lag.

2. GPU Memory Footprint and Quantization

Moshi’s 7B Helium backbone and Mimi codec require approximately 16 GB of VRAM in 16-bit precision (bfloat16):

Single Consumer GPU Deployment: Moshi runs comfortably on a single NVIDIA GeForce RTX 4090 (24GB), an NVIDIA L4 (24GB), or an A100 (40GB/80GB).
Quantization Optimizations: Deploying Moshi with INT8 or INT4 weight quantization via BitsAndBytes or AWQ shrinks VRAM consumption to under 8 GB, opening the door to local edge deployments on Apple Silicon (M2/M3/M4 via MLX) and high-end workstations.

3. Integration with Autonomous Extraction Pipelines

In platforms like AiFetchly, voice assistants are not mere conversational novelties—they serve as high-speed data ingestion and query interfaces.

By streaming live spoken instructions through Moshi, enterprise users can direct autonomous scraping agents, clarify extracted data schemas, and receive spoken executive summaries in real time. Because Moshi simultaneously outputs an Inner Monologue text transcript, the system automatically logs a clean, deterministic textual audit trail of every voice interaction for compliance, database storage, and downstream analytics.

Summary Checklist: Evaluating Full-Duplex Voice Architectures

Before deploying a conversational voice stack in 2026, verify your technical architecture against these criteria:

Framerate Efficiency: Audio tokenizer operates at or below 25 Hz (Mimi achieves 12.5 Hz) to prevent KV cache memory saturation.
Latency Budget: Total round-trip audio latency remains under 300ms to preserve natural conversational cadence.
Bidirectional Duplexing: System continuously ingests user audio while actively generating speech tokens.
Semantic Anchoring: Model employs an Inner Monologue or text prediction stream to prevent acoustic drift and hallucinations.
Data Privacy & Governance: Model weights are auditable and deployable on private local hardware without transmitting sensitive audio to third-party endpoints.