llm-inferencedisaggregated-servinggpu-clustersvllmtutorial

Disaggregated Prefill-Decode: Scaling LLM Serving in 2026

September 20, 2026

Disaggregated Prefill-Decode Architecture: Decoupling Compute-Bound Prefill Nodes from Memory-Bandwidth-Bound Decode Workers via RDMA KV Streaming

The Problem: The Inherent Tension Between Prefill and Decode in Collocated LLM Serving

When serving Large Language Models in production, text generation is divided into two fundamentally conflicting computational phases:

The Prefill Phase (Prompt Ingestion): The model ingests all prompt tokens in parallel to generate the very first output token and build the initial Key-Value (KV) cache. This phase is compute-bound (high arithmetic intensity): it saturates GPU Tensor Cores, consumes massive FLOPs, and benefits from high Tensor Parallelism (TP).
The Decode Phase (Autoregressive Generation): The model generates subsequent tokens one-by-one. Each new token requires reading the model weights and the entire accumulated KV cache from High-Bandwidth Memory (HBM) into SRAM. This phase is memory-bandwidth-bound (low arithmetic intensity): Tensor Cores sit largely idle while the memory bus operates at 100% capacity.

The Breakdown of Collocated Serving Under Long-Context Workloads

In traditional LLM serving engines (such as standard vLLM, TGI, or TensorRT-LLM deployments), both prefill and decode execute on the exact same GPU workers—a paradigm known as collocated serving.

While continuous iteration-level batching manages short-context conversations effectively, collocated serving degrades catastrophically when applied to 32k, 64k, or 128k prompt lengths in RAG, legal document parsing, and agentic web scraping:

Head-of-Line Blocking and Severe ITL Spikes: When a user submits a 64k-token document for extraction, the GPU must spend several seconds calculating the prefill matrix. Because standard continuous batchers co-schedule prefill with active decoding streams, ongoing user streams suffer massive Inter-Token Latency (ITL) spikes (jitter). Streaming responses pause mid-sentence, destroying real-time user experience.
Resource Underutilization and GPU Stranding: A GPU optimized for compute-heavy prefill (such as an 8-way Tensor Parallel H100 cluster) is drastically oversized and underutilized when executing single-token autoregressive decoding. Conversely, a cluster provisioned for large batch decoding leaves its Tensor Cores 80% idle while waiting on memory bandwidth.
Interrupted Chunked Prefills: Workarounds like chunked prefill split prompt ingestion into smaller token chunks (e.g., 512 tokens per step) to interleave with decoding. However, chunking stretches Time-to-First-Token (TTFT) by 3x to 5x, fails to eliminate ITL variance completely, and introduces complex KV cache management overhead.

The Architecture: Physically Decoupling Prefill and Decode Pools

To resolve this structural mismatch, modern infrastructure engineering has transitioned to Disaggregated Prefill-Decode (P/D Disaggregation)—championed by systems like Splitwise, DistServe, Mooncake, and native vLLM disaggregation modules.

Rather than forcing one GPU to alternate between compute-bound matrix multiplications and memory-bound memory scans, disaggregated serving physically partitions the infrastructure into two specialized, independently scalable GPU tiers.

Disaggregated Serving Topology: Prompt Routers, Dedicated High-FLOP Prefill Workers, Asynchronous RDMA KV-Cache Transfer, and High-HBM Decode Workers

The Three Core Pillars of P/D Disaggregation

Dedicated Prefill Pool (Compute Optimized): High-FLOP GPUs (such as NVIDIA H100, B200, or TPU v5p) configured with high Tensor Parallelism (e.g., $TP=4$ or $TP=8$) process incoming prompts at maximum compute saturation. Once the prompt is ingested and the initial KV cache is computed, the request immediately vacates the prefill worker.
Dedicated Decode Pool (Memory & KV Capacity Optimized): High-HBM GPUs (such as H200, L40S, or nodes with quantized KV cache pools) configured with Pipeline or Data Parallelism ($TP=1$ or $TP=2$) handle continuous autoregressive generation. Because no heavy prefill operations ever enter these GPUs, decoding streams achieve a rock-solid, non-fluctuating ITL of under 20 milliseconds.
High-Speed RDMA KV Cache Transfer: The critical bridge between pools is an asynchronous tensor transfer engine. When a prefill worker finishes generating the KV cache for a prompt, it streams the key-value tensors directly across InfiniBand or RoCE (RDMA over Converged Ethernet) into the target decode worker’s paged memory pool via Remote Direct Memory Access (RDMA), bypassing CPU host memory.

Collocated vs. Chunked Prefill vs. Disaggregated Serving

Architectural Dimension	Collocated Serving (Standard vLLM)	Chunked Prefill (Interleaved)	Disaggregated Prefill-Decode (Splitwise / Mooncake)
Compute Profile	Compute and Memory contending on 1 GPU	Compute squeezed into fixed slice chunks	Dedicated GPU tiers for compute vs. memory
Time-to-First-Token (TTFT)	Fast under low load; terrible under concurrency	Slow (artificially throttled by chunking)	Consistently ultra-low (< 200ms on 32k tokens)
Inter-Token Latency (ITL)	High jitter (spikes up to 2,000ms during prefill)	Moderate jitter (frequent sub-spikes)	Zero prefill interference (< 25ms steady stream)
GPU Hardware Flexibility	Homogeneous cluster mandatory	Homogeneous cluster mandatory	Heterogeneous hardware arbitrage (mix H100 + L40S)
Cluster Throughput	Baseline (1.0x)	1.3x – 1.5x	2.2x – 3.5x tokens/second per dollar

Implementation: Simulating Disaggregated KV Cache Streaming and Load Balancing

Let us implement a complete Python simulation demonstrating how a disaggregated serving router dispatches prompts to prefill workers, transfers the computed KV cache tensors asynchronously over simulated high-speed network buffers, and executes decoding without latency spikes.

Step 1: Define the Disaggregated Request and KV Cache Transfer Envelope

import asyncio
import time
import dataclasses
from typing import List, Dict, Optional
import torch

@dataclasses.dataclass
class ServingRequest:
    request_id: str
    prompt_tokens: List[int]
    max_decode_tokens: int
    arrival_time: float
    prefill_start_time: Optional[float] = None
    prefill_end_time: Optional[float] = None
    kv_transfer_start_time: Optional[float] = None
    kv_transfer_end_time: Optional[float] = None
    decode_start_time: Optional[float] = None
    generated_tokens: List[int] = dataclasses.field(default_factory=list)
    inter_token_latencies: List[float] = dataclasses.field(default_factory=list)

@dataclasses.dataclass
class KVCachePayload:
    request_id: str
    kv_tensors: torch.Tensor # Shape: [layers, 2, batch, heads, seq_len, head_dim]
    num_tokens: int
    memory_size_mb: float

Step 2: Implement the Dedicated Prefill Worker

The prefill worker runs high-batch matrix multiplications to compute the prompt representation and output the KV cache:

class PrefillWorker:
    def __init__(self, worker_id: str, compute_tflops: float = 989.0):
        self.worker_id = worker_id
        self.compute_tflops = compute_tflops
        self.lock = asyncio.Lock()

    async def execute_prefill(self, request: ServingRequest) -> KVCachePayload:
        async with self.lock:
            request.prefill_start_time = time.perf_counter()
            seq_len = len(request.prompt_tokens)

            # Prefill FLOPs approximation: 2 * params * seq_len (e.g. 70B model)
            simulated_prefill_delay = (seq_len * 0.00004) # e.g., ~250ms for 6.4k tokens
            await asyncio.sleep(simulated_prefill_delay)

            # Simulated KV cache tensor generation (Float16: 2 bytes * 2 (K+V) * layers * heads * dim * seq_len)
            num_layers, num_heads, head_dim = 32, 32, 128
            kv_size_bytes = 2 * 2 * num_layers * num_heads * head_dim * seq_len
            kv_size_mb = kv_size_bytes / (1024 * 1024)

            # In production, this resides in GPU VRAM ready for RDMA transfer
            dummy_kv = torch.empty((num_layers, 2, 1, num_heads, seq_len, head_dim), dtype=torch.float16)

            request.prefill_end_time = time.perf_counter()
            return KVCachePayload(
                request_id=request.request_id,
                kv_tensors=dummy_kv,
                num_tokens=seq_len,
                memory_size_mb=kv_size_mb
            )

Step 3: Implement the RDMA Transfer Bridge and Dedicated Decode Worker

class RDMANetworkFabric:
    def __init__(self, bandwidth_gbps: float = 400.0): # 400 Gbps InfiniBand / RoCE
        self.bandwidth_mb_per_sec = (bandwidth_gbps * 1024 / 8)

    async def transfer_kv_cache(self, payload: KVCachePayload, target_decode_worker: 'DecodeWorker'):
        # RDMA latency = (Payload Size / Bandwidth) + link overhead (~5 microseconds)
        transfer_duration = (payload.memory_size_mb / self.bandwidth_mb_per_sec) + 0.0005
        await asyncio.sleep(transfer_duration)
        target_decode_worker.register_kv_cache(payload)

class DecodeWorker:
    def __init__(self, worker_id: str):
        self.worker_id = worker_id
        self.active_kv_caches: Dict[str, KVCachePayload] = {}
        self.running_streams: Dict[str, ServingRequest] = {}

    def register_kv_cache(self, payload: KVCachePayload):
        self.active_kv_caches[payload.request_id] = payload

    async def run_decode_loop(self):
        # Memory-bandwidth bound iteration step: constant steady clock
        while True:
            if not self.running_streams:
                await asyncio.sleep(0.005)
                continue

            step_start = time.perf_counter()
            # Simulated decode step across all batched active streams
            await asyncio.sleep(0.015) # Constant 15ms step latency (sub-20ms ITL)

            finished_ids = []
            for req_id, req in list(self.running_streams.items()):
                now = time.perf_counter()
                if req.generated_tokens:
                    req.inter_token_latencies.append(now - step_start)
                else:
                    req.decode_start_time = now

                req.generated_tokens.append(999) # Append next token
                if len(req.generated_tokens) >= req.max_decode_tokens:
                    finished_ids.append(req_id)

            for req_id in finished_ids:
                del self.running_streams[req_id]
                if req_id in self.active_kv_caches:
                    del self.active_kv_caches[req_id]

Step 4: Run the End-to-End Disaggregated Orchestrator

async def run_disaggregated_serving_benchmark():
    fabric = RDMANetworkFabric(bandwidth_gbps=400.0)
    prefill_node = PrefillWorker("prefill-h100-01")
    decode_node = DecodeWorker("decode-h200-01")

    # Start the continuous decode worker in the background
    decode_task = asyncio.create_task(decode_node.run_decode_loop())

    request = ServingRequest(
        request_id="req-rag-9912",
        prompt_tokens=[101] * 16384, # 16k prompt
        max_decode_tokens=20,
        arrival_time=time.perf_counter()
    )

    print("[Orchestrator] Step 1: Routing 16k prompt to Prefill Node...")
    kv_payload = await prefill_node.execute_prefill(request)
    ttft_prefill = (request.prefill_end_time - request.prefill_start_time) * 1000
    print(f"  -> Prefill completed in {ttft_prefill:.2f} ms (KV Cache size: {kv_payload.memory_size_mb:.2f} MB)")

    print("[Orchestrator] Step 2: Streaming KV Cache over 400Gbps RDMA to Decode Node...")
    transfer_start = time.perf_counter()
    await fabric.transfer_kv_cache(kv_payload, decode_node)
    transfer_duration = (time.perf_counter() - transfer_start) * 1000
    print(f"  -> RDMA KV Transfer completed in {transfer_duration:.2f} ms")

    print("[Orchestrator] Step 3: Commencing token decoding...")
    decode_node.running_streams[request.request_id] = request

    # Wait for decode completion
    while request.request_id in decode_node.running_streams:
        await asyncio.sleep(0.01)

    avg_itl = (sum(request.inter_token_latencies) / len(request.inter_token_latencies)) * 1000
    print(f"[Results] Tokens generated: {len(request.generated_tokens)}")
    print(f"  -> Average Inter-Token Latency (ITL): {avg_itl:.2f} ms")
    print(f"  -> ITL Jitter: Zero prefill interference observed.")

    decode_task.cancel()

# asyncio.run(run_disaggregated_serving_benchmark())

Production Serving and Deployment Trade-offs in 2026

When architecting a disaggregated LLM serving cluster at scale, infrastructure teams must navigate several operational considerations:

Network Bandwidth Feasibility Condition: Disaggregation delivers net latency gains only when the KV Cache Transfer Time is significantly less than the Prefill Processing Time: $$\frac{\text{KV Size (GB)}}{\text{Network Bandwidth (GB/s)}} \ll \text{Prefill Execution Duration}$$ On a 400 Gbps or 800 Gbps RDMA interconnect, transferring a 16k context KV cache requires just 10ms to 20ms—orders of magnitude faster than the hundreds of milliseconds required for prefill computation. On standard 10Gbps or 25Gbps commodity Ethernet, however, network transfer becomes the primary bottleneck, making collocated serving preferable.
Heterogeneous Hardware Arbitrage: In collocated setups, all nodes must run identical, top-tier GPUs (e.g. 8x H100). Disaggregated clusters allow cost optimization: deploying expensive H100/B200 nodes for the prefill tier while deploying cost-effective, memory-dense GPUs (such as L40S, PCIe cards, or H200s) for the decode tier, slashing overall cluster compute expenditure by 30% to 45%.
Dynamic Pool Autoscaling: Prompt traffic in production is highly volatile. Daytime hours experience large bursts of enterprise RAG queries (prefill heavy), while code completion and interactive chat exhibit long generation streams (decode heavy). Disaggregated architectures permit Kubernetes or Slurm autoscalers to scale prefill and decode worker pools completely independently based on queue depths.

Conclusion

Disaggregated Prefill-Decode represents the defining evolution of high-concurrency LLM serving in 2026. By decoupling compute-bound prompt ingestion from memory-bandwidth-bound token generation and connecting them via high-speed RDMA tensor streaming, AI platform engineers eliminate head-of-line blocking, slash TTFT, and deliver ultra-smooth, jitter-free generation streams at scale.