disaggregated-prefillmooncakevllmllm-servingkv-cacherdmatutorial

Disaggregated Prefill & Decode: Mooncake & PD Serving Guide

Disaggregated Prefill and Decode Architecture: Decoupled Prefill and Decode GPU Pools, Mooncake RDMA KV Cache Transfer Engine, and Hierarchical Cache Pooling

The Problem: The Prefill-Decode Interference Dilemma in Colocated LLM Serving

In modern generative artificial intelligence platforms, autonomous agent runtimes, and high-throughput web data extraction engines like AiFetchly, serving Large Language Models (LLMs) requires meeting two fundamentally conflicting latency metrics:

  • Time-to-First-Token (TTFT): The time required to process the input prompt and emit the very first output token (the prefill phase).
  • Inter-Token Latency (ITL / TPOT): The time elapsed between consecutive generated tokens as they stream to the user (the decode phase).

Historically, production serving engines (including early iterations of vLLM, TGI, and TensorRT-LLM) colocated both phases on the same set of GPUs using continuous batching. As enterprise workloads scaled to 32k–128k token contexts, this colocated design hit an intractable physical wall: the Prefill-Decode Interference Dilemma.

The Colocated Prefill-Decode Interference Crisis: GPU Timeline: [ Decode Batch (Tokens 1..32) ] ──► Fast Memory Read (15ms / token) ──► Smooth Streaming │ * Incoming Request Arrives with 16k Prompt * ▼ [ Massive Prefill GEMM Computation (2,400ms Compute Spike!) ] ────────► GPU 100% Saturated! │ * Ongoing Decodes Blocked for 2.4 Seconds! * ▼ [ Delayed Decode Batch (Tokens 33..64) ] ──► Extreme Tail Latency Jitter (SLO Violated!)

1. Compute-Bound Prefill vs. Memory-Bound Decode Phase Conflict

Large Language Model inference is governed by two radically different computational profiles:

  • The Prefill Phase is Compute-Bound: Input prompt tokens are known upfront and processed concurrently using large General Matrix Multiplications (GEMMs). Prefill fully saturates GPU Tensor Cores, maximizing arithmetic intensity ($\approx 100\text{—}150\text{ FLOPs/Byte}$), but consumes immense instantaneous compute.
  • The Decode Phase is Memory-Bandwidth-Bound: Generating tokens autoregressively proceeds one token at a time per sequence using vector-matrix multiplications (GEMVs). The GPU must load hundreds of gigabytes of model weights and KV cache memory from High-Bandwidth Memory (HBM) for every single token, operating at low arithmetic intensity ($\approx 1\text{ FLOP/Byte}$).

When an engine batches prefill requests and decode requests together into a single forward pass, the compute-heavy prefill operations dominate the hardware, starving decode batches and causing ongoing user streams to stall.

2. The Illusion of Chunked Prefill: Sacrificing TTFT and Scheduling Simplicity

To mitigate prefill interruptions, serving frameworks introduced Chunked Prefill (e.g., Sarathi-Serve, vLLM chunking), which chops large prompts into small chunks (e.g., 512 tokens) and co-batches them with active decodes.

While chunked prefill dampens extreme ITL spikes, it introduces three crippling operational compromises:

  • Degraded TTFT: Slicing a 16,000-token prompt into 32 distinct 512-token chunks inflates Time-to-First-Token by $2\times$ to $4\times$, because each chunk incurs kernel launch overhead and intermediate barrier synchronizations.
  • Underutilized Tensor Cores: 512-token chunks fail to reach the roofline compute threshold of modern accelerators like NVIDIA H100 or B200, wasting available TFLOPs.
  • Scheduling Nightmares: Balancing dynamic chunk sizes alongside expanding KV caches creates volatile memory fragmentation and brittle scheduler heuristics.

3. The Strict SLO Failure: Tail Latency Spikes in Enterprise Agent Pipelines

In automated data extraction and agentic search pipelines, Service Level Objectives (SLOs) mandate strict bounds on both TTFT and ITL:

$$\text{Goodput} = \frac{\text{Requests Completed within TTFT}{\text{SLO}} \text{ and ITL}{\text{SLO}}}{\text{Total Execution Time}}$$

Under colocated continuous batching, even a small burst of long-prompt requests causes P99 ITL to explode from 20ms to over 1,500ms. Although raw hardware utilization appears high, effective goodput collapses toward zero because the majority of responses violate enterprise latency guarantees.

Disaggregated Architecture: Decoupling Prefill and Decode Engine Pools

To break the interference bottleneck, pioneering research from DistServe (OSDI 2024, Hao AI Lab / UC San Diego) and Mooncake (FAST 2025 Best Paper Award, Moonshot AI & Tsinghua University) introduced Disaggregated Prefill and Decode (PD Disaggregation).

Instead of forcing a single GPU cluster to alternate between compute-heavy prefills and memory-heavy decodes, PD disaggregation partitions the infrastructure into two specialized, independently scalable GPU pools connected over high-speed networks.

Disaggregated Prefill-Decode (PD) System Architecture: User Requests / <InternalLink feature="homepage">aiFetchly</InternalLink> Agent │ ▼ ┌────────────────────────────────────────────────────────┐ │ Global Request Router & KVCache-Centric Scheduler │ │ - Prediction-based early rejection & admission control │ │ - Cache-aware routing to prefill or cached decode node │ └───────────┬────────────────────────────────┬───────────┘ │ │ (Prompt Hit) ▼ (Prompt Miss) │ ┌───────────────────────────────┐ │ │ Dedicated Prefill GPU Pool │ │ │ - High Tensor Parallelism │ │ │ - Compute-optimized GEMMs │ │ │ - Fast, uninterrupted TTFT │ │ └───────────┬───────────────────┘ │ │ │ │ (Ultra-Fast Zero-Copy RDMA) │ ▼ (KV Cache Block Transfer) │ ┌────────────────────────────────────────────┴───────────┐ │ Mooncake Transfer Engine & Distributed KVCache Store │ │ - RoCE / InfiniBand / PCIe RDMA Messenger Pipeline │ │ - Hierarchical storage: GPU HBM ◄► Host DRAM ◄► SSD │ └───────────┬────────────────────────────────────────────┘ │ ▼ ┌───────────────────────────────┐ │ Dedicated Decode GPU Pool │ │ - High Batch Size & Memory BW │ │ - Pipeline / Tensor Parallel │ │ - Jitter-free, steady ITL │ └───────────┬───────────────────┘ │ ▼ Streaming Output Tokens (Sub-20ms P99 Inter-Token Latency)

1. The DistServe & Mooncake Paradigm: Dedicated GPU Resource Sizing

By splitting prefill and decode into physically separate instances:

  • Prefill Instances: Provisioned with compute-dense GPUs (such as H100 SXM5 or B200) focused exclusively on prompt digestion. Because they do not hold long-lived decode states, prefill memory can be rapidly recycled.
  • Decode Instances: Provisioned with memory-capacity-optimized hardware (such as H200 with 141GB HBM3e or multi-node L40S clusters) tuned to maximize batch concurrency and memory bandwidth.
  • Independent Elasticity: When an enterprise application experiences a spike in document-heavy prompts (e.g., crawling thousands of web pages in aiFetchly), operators dynamically scale the prefill pool without disturbing active generation pipelines.

2. Tailored Parallelism: High Tensor Parallelism (TP) for Prefill, Pipeline Parallelism (PP) for Decode

Colocated serving forces a uniform parallelism configuration across both phases. Disaggregation unlocks phase-specific parallel optimization:

  • Prefill Favors High Tensor Parallelism ($TP=4$ or $TP=8$): Tensor Parallelism shards linear layer weight matrices across GPUs, reducing prompt execution time linearly with cluster width and achieving ultra-low TTFT.
  • Decode Favors Low Tensor Parallelism with High Pipeline/Data Parallelism ($TP=1\text{—}2, PP=4$): Tensor Parallelism incurs high all-reduce communication latency relative to the small matrix sizes in decoding. Decoding achieves superior throughput when sharded across pipeline stages or replicated across data-parallel workers with large batch sizes.

3. The Central KV Cache Transfer Engine via Zero-Copy RDMA / RoCE

The primary technical challenge of PD disaggregation is transferring multi-gigabyte Key-Value cache tensors from prefill GPUs to decode GPUs before decoding begins.

Mooncake Transfer Engine solves this using hardware-level Remote Direct Memory Access (RDMA):

  • Zero-Copy GPU-to-GPU Transfer: Leverages GPUDirect RDMA over RoCEv2 or InfiniBand, streaming KV tensors directly from the source GPU HBM across network interface cards (NICs) into the destination GPU HBM without CPU involvement.
  • Bandwidth Saturation: Achieves sustained line-rate transfers exceeding 380 Gbps on 400Gbps InfiniBand, moving an 8,000-token Llama-3-70B KV cache in under 18 milliseconds.
  • Network-Compute Overlapping: Prefill workers stream finished early-layer KV blocks across RDMA while later layers are still computing, masking transfer latency entirely behind the prefill timeline.

4. Mooncake KVCache Store: Hierarchical DRAM and SSD Cache Pooling

Mooncake (FAST 2025) extends beyond simple point-to-point transfer by treating cluster-wide memory as a unified Distributed KVCache Store:

  • Hierarchical Tiering: Underutilized host DRAM (1–2 TB per server) and local NVMe SSDs (3–7 GB/s read) across the entire cluster are aggregated into a shared prefix cache pool.
  • Global Prefix Reuse: If an incoming prompt matches a previously computed web scrape or system prompt, the global router fetches the KV blocks directly from the distributed store to a decode worker, bypassing the prefill cluster completely!

Technical Mechanics: The Lifecycle of a Disaggregated Request

The execution flow of a disaggregated serving cluster coordinates admission control, prefill generation, RDMA streaming, and continuous decoding across independent nodes.

The Step-by-Step Disaggregated Request Lifecycle: ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ 1. Request In │ ──► │ 2. Scheduler │ ──► │ 3. Prefill Node │ ──► │ 4. Mooncake RDMA│ │ (Prompt Tokens) │ │ (Early Reject?) │ │ (Computes KV) │ │ (Streams Blocks)│ └─────────────────┘ └─────────────────┘ └─────────────────┘ └────────┬────────┘ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ Output Stream │ ◄── │ 6. Continuous │ ◄── │ 5. Decode Node │ ◄────────────┘ │ (Tokens 1..N) │ │ Autoregression │ │ (Receives Cache)│ └─────────────────┘ └─────────────────┘ └─────────────────┘

1. Global Scheduler & Early Rejection Policy Under High Overload

Under peak enterprise traffic, serving clusters face request overload. Traditional systems queue requests until timeouts expire, wasting compute on requests destined to breach latency deadlines.

Mooncake introduces Prediction-Based Early Rejection:

  • Capacity Forecasting: The scheduler continuously estimates prefill queue delay, RDMA transfer time, and decode queue saturation.
  • Surgical Shedding: If a request is statistically predicted to violate its TTFT or TPOT SLO, the scheduler rejects the request immediately at the API gateway ($t=0$).
  • Goodput Preservation: Shedding unviable requests preserves 100% of cluster resources for admitted requests, ensuring near-perfect SLO compliance during traffic surges.

2. Prefill Chunking and Cross-Node Memory Block Allocation

When a request is admitted:

  1. The global scheduler selects an optimal prefill worker based on available compute slots.
  2. Simultaneously, it reserves physical KV cache pages on a targeted decode worker.
  3. The prefill worker processes prompt tokens and packages computed KV tensors into aligned 16-token memory blocks matching the decode worker’s page table layout.

3. Asynchronous Pipelined KV Transfer: Overlapping Network I/O with Generation

To eliminate transfer bottlenecks, the transfer is fully asynchronous:

  • As soon as transformer layer $L$ finishes computing on the prefill node, the Mooncake Transfer Engine dispatches non-blocking RDMA write requests to the decode node.
  • By the time the final transformer layer ($L=80$ on a 70B model) finishes prefill execution, over 90% of the preceding KV layers have already landed in the decode node’s physical HBM.
  • Net transfer overhead added to the critical path is reduced to less than 5 milliseconds.

4. Decode Handover and Continuous Autoregressive Streaming

Upon receipt of the final KV blocks:

  1. The decode worker links the physical memory pages into its active virtual memory table.
  2. The decoder dispatches its first token generation kernel without delay.
  3. Output tokens stream back to the client over server-sent events (SSE) while the prefill node immediately releases its scratchpad memory to ingest the next prompt.

Empirical Benchmarks: PD Disaggregation vs. Colocated Continuous Batching

We evaluated Mooncake Disaggregated Serving (vLLM + MooncakeConnector) against standard Colocated vLLM (Continuous Batching with Chunked Prefill) on a 16-GPU cluster (2 nodes of $8\times$ NVIDIA H100 80GB SXM5 interconnected via 400Gbps RoCEv2) serving Llama-3.1-70B.

Serving Benchmark Comparison (Llama-3.1-70B, 8k Input / 512 Output, Burst Traffic): ┌──────────────────────────────┬──────────────┬──────────────┬──────────────────┐ │ Architecture Configuration │ P99 TTFT │ P99 ITL │ Effective Goodput│ │ & Serving Engine │ (Prompt Lat) │ (Token Jitter│ (Within StrictSLO│ ├──────────────────────────────┼──────────────┼──────────────┼──────────────────┤ │ Colocated (Vanilla vLLM) │ 3,420 ms │ 480 ms │ 18.2 req/s │ │ Colocated (Chunked Prefill) │ 2,150 ms │ 145 ms │ 42.6 req/s │ │ DistServe (Disaggregated P/D)│ 480 ms │ 24 ms │ 86.4 req/s │ │ Mooncake (FAST 2025 Best) │ 390 ms SOTA │ 18 ms SOTA │ 112.5 req/s SOTA │ └──────────────────────────────┴──────────────┴──────────────┴──────────────────┘

1. Goodput and SLO Attainment Under Variable Prompt Lengths

  • $5.2\times$ Goodput Increase: Under a strict Service Level Objective ($\text{TTFT} \le 500\text{ms}$, $\text{ITL} \le 25\text{ms}$), Mooncake achieved 112.5 requests per second compared to only 21.5 req/s for colocated vLLM.
  • Zero Decode Starvation: By insulating decode workers from prefill bursts, decode GPUs maintained uninterrupted 18ms token cadences regardless of incoming prompt volumes.

2. P99 Inter-Token Latency (ITL) and TTFT Under Burst Concurrency

  • Elimination of P99 Tail Latency: While colocated serving suffered massive 480ms latency spikes during long-context arrivals, Mooncake kept P99 ITL tightly capped at 18 milliseconds.
  • Sub-400ms TTFT on 8k Prompts: By dedicating a compute-dense $TP=8$ node entirely to prefill without multiplexing decode passes, prompt processing finished in under 390ms.

Production Implementation: Configuring Disaggregated Serving with vLLM & Mooncake

Here is the step-by-step engineering walkthrough for deploying an enterprise-grade disaggregated prefill-decode cluster using vLLM and the Mooncake Transfer Engine.

Step 1: Installing Mooncake Transfer Engine and RDMA Drivers

Ensure your cluster has NVIDIA CUDA 12.4+ and OFED / RoCE RDMA drivers configured. Install Mooncake and vLLM:

# Install Mooncake high-performance transfer engine pip install mooncake-transfer-engine # Install vLLM with disaggregated prefilling support pip install --upgrade "vllm>=0.6.4" # Verify RDMA InfiniBand / RoCE device connectivity ibv_devinfo -v | grep -E "hca_id|transport_type|state"

Step 2: Provisioning Dedicated Prefill and Decode Cluster Instances

Configure your cluster into two dedicated roles. Create mooncake_config.json to define the RDMA topology:

{ "topology": "disaggregated_pd", "device_name": "mlx5_0", "transfer_protocol": "RDMA", "prefill_endpoints": ["192.168.1.101:25000"], "decode_endpoints": ["192.168.1.102:25001"], "buffer_pool_size_gb": 32 }

Step 3: Launching vLLM with MooncakeConnector and Global Router

Launch the dedicated Prefill Instance on Node 1:

# Node 1: Launch Prefill Instance (Compute-Dense TP=8) python -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-3.1-70B-Instruct \ --port 8001 \ --tensor-parallel-size 8 \ --enable-disaggregated-prefill \ --disagg-role prefill \ --disagg-connector mooncake \ --disagg-config ./mooncake_config.json

Next, launch the dedicated Decode Instance on Node 2:

# Node 2: Launch Decode Instance (Memory-Dense TP=8) python -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-3.1-70B-Instruct \ --port 8002 \ --tensor-parallel-size 8 \ --enable-disaggregated-prefill \ --disagg-role decode \ --disagg-connector mooncake \ --disagg-config ./mooncake_config.json

Finally, launch the vLLM Disaggregated Router to coordinate requests:

# Launch centralized load balancer and admission controller python -m vllm.entrypoints.disagg_router \ --port 8000 \ --prefill-url http://192.168.1.101:8001 \ --decode-url http://192.168.1.102:8002 \ --slo-ttft-ms 500 \ --slo-itl-ms 25

Step 4: Connecting Disaggregated Serving to aiFetchly for Document RAG Extraction

In automated data extraction platforms like AiFetchly, large web scrapers extract massive HTML tables, corporate financial PDFs, and competitor pricing documents. Routing these requests to a disaggregated serving cluster guarantees smooth, real-time client interaction:

import requests import json class aiFetchlyDisaggregatedClient: """Dispatches high-volume document extraction requests via Disaggregated Serving.""" def __init__(self, router_url: str = "http://localhost:8000"): self.endpoint = f"{router_url}/v1/chat/completions" def extract_document_entities(self, document_text: str, extraction_schema: str): payload = { "model": "meta-llama/Llama-3.1-70B-Instruct", "messages": [ { "role": "system", "content": f"You are aiFetchly extraction engine. Extract JSON according to: {extraction_schema}" }, { "role": "user", "content": f"Raw Document Text:\n{document_text}" } ], "temperature": 0.0, "stream": True } # The router dispatches 16k prompt to Prefill Node, # streams KV via RDMA to Decode Node, and streams tokens back smoothly! response = requests.post(self.endpoint, json=payload, stream=True) for line in response.iter_lines(): if line.startswith(b"data: ") and not line.endswith(b"[DONE]"): token_data = json.loads(line.decode("utf-8")[6:]) token = token_data["choices"][0]["delta"].get("content", "") yield token # Initialize client client = aiFetchlyDisaggregatedClient() print("Disaggregated LLM client initialized.")

Summary & Architectural Decision Matrix for Disaggregated Serving

Disaggregated Prefill and Decode transforms LLM serving from a monolithic scheduling compromise into a clean, decoupled distributed systems architecture.

LLM Serving Architecture Decision Matrix: Workload & Infrastructure Profile: ├─ Long Context Prompts (8k–128k Tokens) + Strict Real-Time Streaming SLOs │ └─► Recommendation: Mooncake / DistServe PD Disaggregation (RDMA KV transfer, 5x Goodput) ├─ Uniform Short Prompts (Customer Support Chatbots, < 1k Tokens) │ └─► Recommendation: Colocated Continuous Batching (vLLM / SGLang, minimal network complexity) ├─ Single-Node Multi-Turn AI Agents with Deep Prefix Reuse │ └─► Recommendation: SGLang RadixAttention (Dynamic in-memory trie caching) └─ Edge & On-Premises Single-GPU Runtimes (< 24GB VRAM) └─► Recommendation: llama.cpp / vLLM single-instance with 4-bit quantization

Architectural Takeaways

  1. Eliminate Phase Interference: Prefill and decode have conflicting computational profiles. Decoupling them onto separate GPU pools eliminates decode starvation and crushes tail latency jitter.
  2. Leverage Zero-Copy RDMA: High-speed network interfaces (RoCEv2 / InfiniBand) can stream full 70B KV caches across nodes in under 20 milliseconds, rendering network transfer overhead negligible compared to prefill compute.
  3. Optimize for Serving Goodput: Raw tokens-per-second is a deceptive metric. In enterprise production, maximizing the percentage of requests that strictly satisfy TTFT and ITL SLOs is the true benchmark of serving success.