mooncakekimikv-cachellm-servingtutorial

Mooncake Architecture: Kimi Disaggregated KV Cache Serving

September 22, 2026

Mooncake Architecture: Kimi Disaggregated KV Cache Serving Engine

The Problem: The 2-Million Token Wall and the KV Cache Memory Cliff

In enterprise generative AI deployments—from large-scale legal document parsing and autonomous web research to continuous code extraction in AiFetchly—expanding context windows from 8k tokens to 200,000, 500,000, and 2,000,000 tokens is essential.

However, scaling foundation models to million-token contexts exposes an acute infrastructure bottleneck in traditional collocated serving architectures (where both prompt prefill and token decoding execute on the same GPU workers):

1. The Astronomical KV Cache Memory Footprint

During autoregressive generation, a transformer model stores historical Key ($\mathbf{K}$) and Value ($\mathbf{V}$) activations to avoid recalculating past context.

For a 70B parameter model serving a 1,000,000-token document in 16-bit precision, the KV cache consumes over 500 gigabytes of memory per active query. A single user request exceeds the physical VRAM capacity of six dedicated NVIDIA H100 80GB GPUs solely to store attention history, starving the cluster and capping concurrent batch capacity to single digits.

2. Prefill Interference and Severe Decode Jitter

Ingesting a 500,000-token prompt requires massive compute-bound matrix multiplications that can monopolize GPU Tensor Cores for 5 to 15 seconds.

In a collocated setup, while this massive prefill executes, all concurrent interactive decoding streams sharing the GPU freeze. Time-Per-Output-Token (TPOT) spikes by 20x to 50x, causing streaming responses to stutter and violating enterprise Service Level Objectives (SLOs).

3. The Devastating Recomputation Tax

When GPU memory pressure spikes, traditional serving schedulers evict older request caches from VRAM.

If an agent or user asks a follow-up question on an existing 1-million-token document, the server must re-ingest and re-compute the entire 1M-token prompt from scratch. This “recomputation tax” wastes millions of GPU FLOPs, creates massive server queues, and multiplies cloud hosting expenses.

The Architecture: Mooncake and Disaggregated Ephemeral KV Storage

To conquer the million-token barrier, Moonshot AI (the creators of the Kimi commercial platform) designed and open-sourced Mooncake (Qin et al., 2024 / 2025).

Mooncake abandons the paradigm of treating GPU memory as an isolated, collocated silo. Instead, it introduces Prefill-Decode Disaggregation coupled with a Distributed Ephemeral KV Cache Store.

Mooncake System Topology: Disaggregated Prefill and Tiered KV Store

The Three Structural Pillars of Mooncake

Mooncake organizes inference infrastructure into three specialized, horizontally scalable subsystems:

1. Physical Prefill-Decode Disaggregation

Rather than forcing a single GPU to alternate between compute-heavy prefill and memory-heavy decoding:

Prefill GPU Cluster: Specialized clusters equipped with high Tensor Parallelism (TP) that saturate GPU Tensor Cores on dense prompt ingestion without being interrupted by decoding cycles.
Decode GPU Cluster: Specialized clusters equipped with high pipeline and continuous batching concurrency that stream tokens sequentially with rock-solid, predictable latency.

2. The Distributed Ephemeral KV Store

In Mooncake, the KV cache is decoupled from any individual GPU or server. It operates as a cluster-wide distributed ephemeral storage layer.

As soon as a prefill node finishes computing Key-Value representations for a prompt chunk, it immediately offloads the tensors into the distributed cache via high-speed Remote Direct Memory Access (RDMA) over InfiniBand or RoCE (RDMA over Converged Ethernet), freeing the prefill GPU for the next incoming query.

3. Three-Tier Hierarchical Storage Architecture

To store petabytes of active KV blocks without purchasing tens of thousands of GPUs, Mooncake establishes a hierarchical memory pool:

Tier 1: GPU High Bandwidth Memory (HBM): Ultra-fast (>3 TB/s) memory reserved strictly for the active tokens currently undergoing attention matrix multiplication.
Tier 2: Host CPU DRAM: Terabytes of server memory (pooled across the cluster at ~400 GB/s via PCIe Gen 5) storing active, warm conversational context prefixes.
Tier 3: Local NVMe SSDs: Massive, low-cost local solid-state storage storing cold historical contexts, multi-turn documents, and dormant agent sessions.

Key Mechanics: High-Speed RDMA Transfer and Zero Recomputation

Mooncake unlocks million-token context through several low-level systems optimizations:

1. Zero-Copy Asynchronous RDMA Streaming

Transferring hundreds of megabytes of KV cache across servers traditionally causes CPU serialization bottlenecks.

Mooncake bypasses the host CPU networking stack completely using kernel-bypass RDMA. Prefill workers stream KV blocks directly into the target decode worker’s memory or cluster DRAM buffers at speeds exceeding 100 GB/s, adding less than 15ms of end-to-end network transit time.

2. Multi-Turn Prefix Reuse Without Recomputation

When a user submits a follow-up query in a multi-turn 1M-token session:

The Mooncake scheduler consults its global prefix index.
It detects that 99.9% of the historical context tokens are already cached in Tier 2 (Host DRAM) or Tier 3 (NVMe SSD).
The prefill node only computes the small user prompt delta (e.g., 50 new tokens).
The decode worker fetches the historical 1M-token KV blocks from host DRAM via RDMA in milliseconds, completely bypassing the multi-second prefill phase.

3. Predictive Chunk-Level Eviction

Mooncake uses a predictive cache management policy. By monitoring request arrival patterns and sequence life cycles, the engine automatically demotes inactive KV blocks from GPU HBM $\to$ Host DRAM $\to$ NVMe SSD, and proactively promotes blocks back to HBM before a scheduled decoding turn begins.

Long-Context Serving Architecture Comparison

Architectural Dimension	Collocated Serving (vLLM V0)	Chunked Prefill (Sarathi)	Disaggregated P/D (DistServe)	Mooncake (Kimi Architecture)
Prefill & Decode Coupling	Strictly Collocated	Collocated (Interleaved)	Physically Disaggregated	Physically Disaggregated
KV Cache Storage	GPU HBM Only	GPU HBM Only	Local Node GPU/DRAM	Cluster-Wide Multi-Tier (HBM-DRAM-SSD)
Max Practical Context	32k – 64k tokens	64k – 128k tokens	128k tokens	2,000,000+ tokens
Inter-Token Latency Jitter	Severe (Prefill freezes)	Moderate	Minimal	Near-Zero (Zero prefill collision)
KV Cache Recomputation	Frequent on memory pressure	Frequent on memory pressure	Moderate	Zero (Persistent across cluster)
Inter-Node Network	Standard TCP / Ray	Standard TCP	Basic RDMA transfer	Optimized Zero-Copy RDMA Fabric
Serving Cost / Million Tokens	Baseline (1.0x)	0.75x	0.55x	0.25x (75% cost reduction)

Reference Implementation: Hierarchical KV Cache Eviction and Routing

Below is an educational Python implementation illustrating how Mooncake coordinates hierarchical storage tiers (HBM, Host DRAM, and NVMe SSD) and routes requests based on prefix cache hits:

import time
from typing import Dict, Optional, Tuple

class TieredKVCacheStore:
    """
    Minimal simulation of Mooncake's hierarchical KV cache store.
    Manages Tier 1 (GPU HBM), Tier 2 (Host DRAM), and Tier 3 (NVMe SSD).
    """
    def __init__(self, hbm_capacity: int = 2, dram_capacity: int = 5):
        self.hbm_capacity = hbm_capacity
        self.dram_capacity = dram_capacity

        self.tier1_hbm: Dict[str, dict] = {}   # Active GPU HBM
        self.tier2_dram: Dict[str, dict] = {}  # Pooled Host DRAM
        self.tier3_ssd: Dict[str, dict] = {}   # High-speed NVMe SSD

    def store_kv_chunk(self, chunk_id: str, token_count: int, kv_tensor: str):
        """
        Stores newly computed KV block from prefill node.
        """
        block_data = {"tokens": token_count, "data": kv_tensor, "timestamp": time.time()}

        if len(self.tier1_hbm) < self.hbm_capacity:
            self.tier1_hbm[chunk_id] = block_data
            print(f"[STORE] Block {chunk_id} ({token_count} tokens) placed in Tier 1 (GPU HBM).")
        else:
            # Demote oldest HBM block to Host DRAM (Tier 2)
            oldest_hbm = min(self.tier1_hbm.keys(), key=lambda k: self.tier1_hbm[k]["timestamp"])
            evicted_block = self.tier1_hbm.pop(oldest_hbm)
            self._demote_to_dram(oldest_hbm, evicted_block)

            self.tier1_hbm[chunk_id] = block_data
            print(f"[STORE] Block {chunk_id} placed in Tier 1 (GPU HBM) after eviction.")

    def _demote_to_dram(self, chunk_id: str, block: dict):
        if len(self.tier2_dram) < self.dram_capacity:
            self.tier2_dram[chunk_id] = block
            print(f"[DEMOTE] Block {chunk_id} transferred to Tier 2 (Host DRAM via RDMA).")
        else:
            # Demote oldest DRAM block to NVMe SSD (Tier 3)
            oldest_dram = min(self.tier2_dram.keys(), key=lambda k: self.tier2_dram[k]["timestamp"])
            evicted_dram = self.tier2_dram.pop(oldest_dram)
            self.tier3_ssd[oldest_dram] = evicted_dram
            print(f"[DEMOTE] Block {oldest_dram} flushed to Tier 3 (NVMe SSD cold store).")

            self.tier2_dram[chunk_id] = block

    def retrieve_kv_chunk(self, chunk_id: str) -> Tuple[Optional[dict], str]:
        """
        Resolves KV block across hierarchical tiers without recomputation.
        """
        if chunk_id in self.tier1_hbm:
            return self.tier1_hbm[chunk_id], "Tier 1 (GPU HBM - Instant)"
        elif chunk_id in self.tier2_dram:
            block = self.tier2_dram[chunk_id]
            print(f"[FETCH] Block {chunk_id} streamed from Host DRAM via RDMA (~100 GB/s).")
            return block, "Tier 2 (Host DRAM via RDMA)"
        elif chunk_id in self.tier3_ssd:
            block = self.tier3_ssd[chunk_id]
            print(f"[FETCH] Block {chunk_id} loaded from NVMe SSD cold storage.")
            return block, "Tier 3 (NVMe SSD)"
        return None, "Cache Miss (Recomputation Required)"

if __name__ == "__main__":
    store = TieredKVCacheStore(hbm_capacity=2, dram_capacity=2)

    print("--- Simulating Mooncake Hierarchical Ingestion ---")
    # Simulate storing long-context chunks (500k tokens each)
    store.store_kv_chunk("doc_chunk_1", 250000, "KV_TENSOR_1")
    store.store_kv_chunk("doc_chunk_2", 250000, "KV_TENSOR_2")
    store.store_kv_chunk("doc_chunk_3", 250000, "KV_TENSOR_3") # Triggers HBM demotion
    store.store_kv_chunk("doc_chunk_4", 250000, "KV_TENSOR_4")
    store.store_kv_chunk("doc_chunk_5", 250000, "KV_TENSOR_5") # Triggers DRAM demotion to SSD

    # Retrieval for follow-up query without prefill recomputation
    data, location = store.retrieve_kv_chunk("doc_chunk_1")
    print(f"Result for doc_chunk_1: Found in {location} -> Zero recomputation needed!")

Production Scaling: Serving Million-Token Workflows in aiFetchly

In automated data extraction architectures like AiFetchly, foundation models process hundreds of long-context enterprise documents, financial earnings transcripts, and multi-page web extractions simultaneously:

1. Eliminating the Multi-Turn Recomputation Tax

When autonomous agents perform deep web research or recursive link extraction, they continually reference previous extraction rounds. In legacy clusters, multi-turn contexts exceed GPU limits, forcing the system to re-prefill historical text on every iteration.

With Mooncake’s tiered architecture, historical contexts are preserved across pooled host DRAM and SSDs. When an agent dispatches step 15 of an extraction loop, historical KV blocks load in milliseconds via RDMA, slashing latency and cutting cloud token costs by up to 75%.

2. High-Density Concurrency for Enterprise Workloads

By offloading dormant KV cache blocks to host DRAM and disaggregating compute-intensive prefill clusters from decoding pools, enterprise infrastructure serves 5x more concurrent users on identical GPU hardware.

By decoupling attention memory from individual GPU chips and orchestrating tiered storage over high-speed networks, Mooncake establishes the gold standard for high-throughput, multi-million-token LLM serving in 2026.