Guide to DistServe: Disaggregated LLM Serving Architecture
In large language model (LLM) serving systems, infrastructure architects face strict Service Level Objectives (SLOs). User-facing conversational applications, agentic reasoning loops, and code generation copilots demand two distinct latency guarantees:
- Time-to-First-Token (TTFT): The latency elapsed before the model begins streaming its first response token, governing user-perceived responsiveness.
- Time-Between-Tokens (TBT): The incremental latency of emitting each subsequent output token, governing reading fluency (human reading pace requires sub-50ms TBT).
However, in conventional LLM serving architectures (such as baseline vLLM, Orca, and Hugging Face TGI), both the prompt prefill phase and the autoregressive decoding phase are co-located on the exact same physical GPUs. Because prefill and decoding exhibit diametrically opposed hardware characteristics—prefill is compute-bound matrix multiplication (GEMM), while decoding is memory-bandwidth-bound matrix-vector multiplication (GEMV)—co-locating them creates severe resource contention. Long prompt prefills monopolize GPU compute cores for hundreds of milliseconds, starving ongoing decode streams and causing catastrophic TBT latency jitter.
Published at USENIX OSDI 2024 by researchers from Peking University, UC San Diego, and Shanghai AI Laboratory, DistServe is the seminal systems framework that pioneers Prefill-Decode (PD) Disaggregation. By physically separating the prefill phase and the decoding phase onto dedicated GPU worker clusters, tuning parallelism independently for each phase, and streaming Key-Value (KV) caches across nodes over high-speed networks, DistServe eliminates phase interference entirely.
In this architectural guide, we dissect the systems design of DistServe, formulate the goodput optimization objective, construct an end-to-end PyTorch reference pipeline, and establish enterprise cluster deployment best practices.
Problem: The Interference Trap in Co-Located LLM Serving
To appreciate the necessity of DistServe, we must analyze the structural limitations of co-located serving systems and why they fail under realistic production traffic.
Compute-Bound Prefill vs. Memory-Bound Decoding
Every LLM generation request progresses through two distinct computational phases governed by conflicting execution profiles:
The Fundamental Compute Dichotomy:
1. Prefill Phase (Prompt Ingestion)
•Input: All prompt tokens processed in parallel (e.g., 2,048 to 32,768 tokens)
•Operation: Dense matrix multiplications (GEMM)
•Hardware Profile: Compute-bound; saturates Tensor Cores (>100 FLOPs/byte)
•SLO Target: Time-To-First-Token (TTFT, typically target: <200ms - 500ms)
2. Decode Phase (Autoregressive Token Emission)
•Input: Single token per sequence multiplied against historical KV cache
•Operation: Matrix-vector multiplications (GEMV)
•Hardware Profile: Memory-bandwidth bound; Tensor Cores idle (<2 FLOPs/byte)
•SLO Target: Time-Between-Tokens (TBT, typically target: <30ms - 50ms) In a co-located serving engine using continuous batching:
- When a new batch of prefill requests arrives, the GPU executes massive GEMM kernels to compute Key-Value activations.
- During this prefill execution, all concurrent decoding streams are completely blocked.
- Even with chunked prefill (slicing prompts into 512-token chunks), long prompts inject recurrent latency spikes of 30ms to 80ms into every decoding step, directly violating tight TBT SLOs.
Enterprise workflows that ingest multi-page web extractions often experience this interference. Pairing responsive serving engines with a scalable web scraping platform ensures that upstream extraction feeds into downstream inference without compounding latency bottlenecks.
The Goodput Collapse: Unmet TTFT and TBT SLOs
In commercial serving, raw token throughput (total tokens generated per second) is a deceptive metric. If a serving system outputs 1,000 tokens per second but 60% of those tokens arrive with severe latency jitter that violates user SLOs, those tokens represent wasted compute.
DistServe formalizes the concept of Goodput: the maximum request rate a system can sustain while satisfying both TTFT and TBT SLO targets for at least 90% or 99% of requests:
$$\text{Goodput} = \max_{\lambda} \lambda \quad \text{s.t.} \quad P(\text{TTFT} \le \text{SLO}{\text{TTFT}}) \ge 0.99 ;\land; P(\text{TBT} \le \text{SLO}{\text{TBT}}) \ge 0.99$$
In co-located systems:
- If operators increase batch concurrency to saturate GPU compute and maximize throughput, memory capacity exhausts, causing prefill queueing and blowing past TTFT targets.
- If operators prioritize TTFT by preempting decoding jobs, ongoing streams experience stutter, blowing past TBT targets.
- As traffic increases, the fraction of requests meeting both SLOs plummets, triggering Goodput Collapse.
Solution: The DistServe Disaggregated Architecture
DistServe resolves the interference trap by decoupling the serving pipeline into specialized physical tiers:
1. Dedicated Prefill and Decoding GPU Worker Pools
DistServe partitions cluster GPUs into two non-overlapping pools:
- Prefill Pool: Dedicated solely to prompt ingestion and initial KV cache computation. Because prefill workers never execute decode steps, they can run large batch sizes and fully saturate GPU Tensor Cores without fearing decode interruptions.
- Decode Pool: Dedicated solely to autoregressive token generation. Because decode workers never execute heavy prefill jobs, they maintain steady, uninterrupted matrix-vector execution, ensuring perfectly uniform, jitter-free TBT latency.
2. Phase-Specific Parallelism Tuning (TP vs. PP)
In co-located systems, Tensor Parallelism (TP) and Pipeline Parallelism (PP) degrees must be identical across both phases because requests run on the same GPUs. This forces suboptimal compromises:
- High Tensor Parallelism ($TP = 8$) is ideal for decoding because it splits the model weights across multiple GPUs, aggregating memory bandwidth to accelerate GEMV reads. However, high $TP$ incurs frequent all-reduce communication overhead, which slows down compute-bound prefills.
- Pipeline Parallelism ($PP$) is excellent for prefill throughput because intermediate activations can be pipelined across layers without fine-grained all-reduce barriers. However, $PP$ introduces catastrophic bubble latency during single-token decoding.
DistServe decouples parallelism strategies:
- Prefill Workers: Configured with minimal Tensor Parallelism ($TP=1$ or $TP=2$) and higher Pipeline/Data Parallelism ($PP/DP$) to maximize compute density and minimize communication overhead.
- Decode Workers: Configured with high Tensor Parallelism ($TP=4$ or $TP=8$) to maximize aggregate HBM bandwidth and slash per-token memory-streaming latency.
3. High-Throughput Asynchronous Cross-Node KV Cache Transfer
The challenge of disaggregation is moving the precomputed Key-Value cache from the prefill worker to the decode worker. If KV cache migration stalls on network transfers, the benefits of disaggregation evaporate.
DistServe implements an Asynchronous Direct RDMA Pipeline:
- KV cache tensors are allocated in page-locked pinned memory or directly in GPU HBM.
- Prefill workers stream completed KV chunks to decode workers over InfiniBand RoCE (Remote Direct Memory Access) bypassing host CPU memory bounce buffers.
- For a 2,048-token prompt on a 13B model (approximately 512MB of KV state), InfiniBand transfers complete in under 8 milliseconds, completely hidden behind the scheduling pipeline.
4. SLO-Aware Placement and Dynamic Load Balancing
DistServe includes a global coordinator that solves a combinatorial optimization problem:
- Profiles incoming request prompt lengths and generation lengths.
- Dynamically allocates the ratio of prefill GPUs to decode GPUs (e.g., 1:2 or 1:4) based on real-time traffic distributions.
- Balances load across worker nodes to guarantee that both TTFT and TBT queues remain within latency budgets.
For mission-critical autonomous agents that query multi-source enterprise documentation, pairing DistServe with real-time data fetching services allows models to process extensive context windows while maintaining instant conversational responsiveness.
Architectural Deep Dive: Co-Located Serving vs. DistServe Disaggregation
The table below contrasts the mechanical execution properties of co-located continuous batching versus DistServe disaggregated serving:
| Architecture Metric | Co-Located Serving (vLLM / Orca) | DistServe Disaggregated Serving |
|---|---|---|
| Worker Placement | Prefill & Decode share same GPUs | Dedicated Prefill & Decode Pools |
| Phase Interference | High (Prefills stall ongoing decodes) | Zero (Completely isolated pools) |
| Parallelism Strategy | Rigid (Identical TP/PP across phases) | Tailored (Low TP for prefill, High TP for decode) |
| KV Cache Movement | Local in GPU VRAM | Sub-10ms RDMA Streaming across nodes |
| TBT Latency Variance | Severe P99 jitter ($>150\text{ms}$) | Flat, deterministic TBT ($<25\text{ms}$) |
| Goodput Efficiency | Low ($20%\text{—}40%$ SLO attainment) | High ($85%\text{—}98%$ SLO attainment, up to $4.48\times$) |
| Hardware Utilization | Tensor cores idle during decode | Both pools operate at optimal hardware saturation |
Connecting disaggregated serving clusters to modern client software stacks is straightforward using versatile API integration tools, facilitating smooth coordination between distributed memory pools and model endpoints.
Implementation: End-to-End PyTorch Reference of DistServe Pipeline
Below is a self-contained PyTorch reference implementation demonstrating DistServe’s disaggregated request router, dedicated prefill worker, and asynchronous KV cache transfer engine.
Step 1: Disaggregated Request Router and SLO Classifier
First, we implement the global router that classifies incoming requests, tracks SLO budgets, and routes jobs to dedicated worker pools:
import time
import math
import torch
import torch.nn as nn
from typing import Dict, List, Tuple, Optional
class RequestMetadata:
"""Tracks per-request latency budgets and generation state."""
def __init__(self, request_id: str, prompt_tokens: torch.Tensor, max_gen_tokens: int, ttft_slo: float = 0.20, tbt_slo: float = 0.04):
self.request_id = request_id
self.prompt_tokens = prompt_tokens
self.max_gen_tokens = max_gen_tokens
self.ttft_slo = ttft_slo # 200ms TTFT SLO
self.tbt_slo = tbt_slo # 40ms TBT SLO
self.arrival_time = time.time()
self.first_token_time: Optional[float] = None
self.generated_tokens: List[int] = []
def record_first_token(self) -> None:
self.first_token_time = time.time()
def meets_ttft(self) -> bool:
if self.first_token_time is None:
return False
return (self.first_token_time - self.arrival_time) <= self.ttft_slo Step 2: Dedicated Prefill Worker with High-Throughput GEMM
Next, we implement the prefill worker module that executes dense prompt prefill passes without decode interruptions:
class DistServePrefillWorker(nn.Module):
"""
Dedicated Prefill Worker:
Saturates Tensor Cores on large batch GEMM matrix multiplications.
Computes initial KV cache representations.
"""
def __init__(self, d_model: int = 1024, num_heads: int = 16, head_dim: int = 64):
super().__init__()
self.d_model = d_model
self.num_heads = num_heads
self.head_dim = head_dim
self.q_proj = nn.Linear(d_model, d_model, bias=False)
self.k_proj = nn.Linear(d_model, d_model, bias=False)
self.v_proj = nn.Linear(d_model, d_model, bias=False)
self.out_proj = nn.Linear(d_model, d_model, bias=False)
def prefill_step(
self,
hidden_states: torch.Tensor
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""
Processes prompt sequence in dense parallel GEMM.
hidden_states: [batch, prompt_len, d_model]
Returns:
first_token_logits: [batch, d_model]
k_cache: [batch, num_heads, prompt_len, head_dim]
v_cache: [batch, num_heads, prompt_len, head_dim]
"""
batch, seq_len, _ = hidden_states.shape
q = self.q_proj(hidden_states).view(batch, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
k = self.k_proj(hidden_states).view(batch, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
v = self.v_proj(hidden_states).view(batch, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
# Scaled dot-product attention
scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.head_dim)
causal_mask = torch.triu(torch.full((seq_len, seq_len), float('-inf'), device=hidden_states.device), diagonal=1)
scores = scores + causal_mask.unsqueeze(0).unsqueeze(0)
attn = torch.softmax(scores, dim=-1)
context = torch.matmul(attn, v).transpose(1, 2).contiguous().view(batch, seq_len, self.d_model)
out = self.out_proj(context)
# Logits for the first token generation (last position)
first_token_repr = out[:, -1, :]
return first_token_repr, k, v Step 3: Asynchronous KV Cache Transfer and Decode Dispatcher
Finally, we construct the decode worker and asynchronous transfer pipeline that receives migrated KV states and performs steady, stall-free generation:
class DistServeDecodeWorker(nn.Module):
"""
Dedicated Decode Worker:
Optimized strictly for low-latency GEMV operations and predictable TBT.
"""
def __init__(self, d_model: int = 1024, num_heads: int = 16, head_dim: int = 64):
super().__init__()
self.d_model = d_model
self.num_heads = num_heads
self.head_dim = head_dim
self.q_proj = nn.Linear(d_model, d_model, bias=False)
self.k_proj = nn.Linear(d_model, d_model, bias=False)
self.v_proj = nn.Linear(d_model, d_model, bias=False)
self.out_proj = nn.Linear(d_model, d_model, bias=False)
def decode_step(
self,
token_hidden: torch.Tensor,
k_cache: torch.Tensor,
v_cache: torch.Tensor
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""
Autoregressive step for a single token against existing KV cache.
token_hidden: [batch, 1, d_model]
k_cache, v_cache: [batch, num_heads, historical_len, head_dim]
"""
batch = token_hidden.shape[0]
q = self.q_proj(token_hidden).view(batch, 1, self.num_heads, self.head_dim).transpose(1, 2)
k_new = self.k_proj(token_hidden).view(batch, 1, self.num_heads, self.head_dim).transpose(1, 2)
v_new = self.v_proj(token_hidden).view(batch, 1, self.num_heads, self.head_dim).transpose(1, 2)
# Append new token to cache
k_total = torch.cat([k_cache, k_new], dim=2)
v_total = torch.cat([v_cache, v_new], dim=2)
# Compute single-row attention
scores = torch.matmul(q, k_total.transpose(-2, -1)) / math.sqrt(self.head_dim)
attn = torch.softmax(scores, dim=-1)
context = torch.matmul(attn, v_total).transpose(1, 2).contiguous().view(batch, 1, self.d_model)
out = self.out_proj(context)
return out.squeeze(1), k_total, v_total
class DistServeClusterOrchestrator:
"""
Simulates cross-node coordination: Prefill Execution -> RDMA Transfer -> Decode Loop.
"""
def __init__(self, prefill_worker: DistServePrefillWorker, decode_worker: DistServeDecodeWorker):
self.prefill_worker = prefill_worker
self.decode_worker = decode_worker
def serve_request(self, req: RequestMetadata, prompt_embeddings: torch.Tensor) -> List[int]:
# 1. Prefill Phase on Prefill Worker
first_repr, k_prefill, v_prefill = self.prefill_worker.prefill_step(prompt_embeddings)
req.record_first_token()
# 2. Asynchronous RDMA KV Transfer (Simulated cross-device migration)
# In multi-node setups: torch.distributed.isend / irecv over InfiniBand
k_migrated = k_prefill.clone()
v_migrated = v_prefill.clone()
# 3. Continuous Decode Phase on Decode Worker
curr_repr = first_repr.unsqueeze(1)
for step in range(req.max_gen_tokens):
next_repr, k_migrated, v_migrated = self.decode_worker.decode_step(
curr_repr, k_migrated, v_migrated
)
curr_repr = next_repr.unsqueeze(1)
req.generated_tokens.append(step)
return req.generated_tokens Best Practices for Deploying DistServe in Production GPU Clusters
Deploying disaggregated LLM serving clusters in enterprise cloud infrastructure requires careful tuning of worker pool allocation, networking topologies, and traffic shaping.
1. Sizing Prefill-to-Decode Worker Ratios
- Traffic Ratio Profiling: Calculate the ratio of average prompt length ($S_{\text{prompt}}$) to average generation length ($S_{\text{gen}}$).
- For summarization and document retrieval ($S_{\text{prompt}} \gg S_{\text{gen}}$, e.g., 4,000 prompt tokens to 200 output tokens), allocate 2:1 or 1:1 prefill-to-decode nodes.
- For code generation and interactive chat ($S_{\text{prompt}} \approx S_{\text{gen}}$ or $S_{\text{gen}} > S_{\text{prompt}}$), allocate 1:3 or 1:4 prefill-to-decode nodes.
- Dynamic Worker Reallocation: Utilize Kubernetes cluster orchestrators to dynamically convert decode workers into prefill workers during traffic surges, maintaining SLO equilibrium.
2. InfiniBand RDMA Configuration for Sub-10ms KV Cache Migrations
- GPU-Direct RDMA: Always configure RoCEv2 or InfiniBand networks with GPU-Direct RDMA enabled (
nvidia-peermem). This allows NICs to read and write directly to GPU High-Bandwidth Memory across PCIe switches, eliminating CPU DRAM staging hops. - Dedicated Migration Ring: Dedicate separate virtual LANs or InfiniBand queue pairs exclusively to KV cache migration, ensuring that inter-node tensor transfers never compete with client HTTP API requests.
3. Handling Variable Prompt-to-Generation Length Ratios
- Early Handoff: For massive prompts ($>32\text{K}$ tokens), initiate KV cache transmission in chunks while the prefill phase is still processing remaining prompt layers, overlapping cross-node transfer with upstream computation.
Engineering organizations can model compute and networking cost reductions using our transparent pricing page, projecting infrastructure requirements across multi-node deployments.
Evaluation: Benchmarking DistServe Goodput and Latency Stability
The DistServe authors evaluated system performance across OPT-13B, OPT-66B, LLaMA-2-13B, and LLaMA-2-70B on clusters of NVIDIA A100 (80GB SXM4) GPUs connected via 200 Gbps InfiniBand networks.
Serving Goodput and SLO Attainment Under Heavy Load
Comparative evaluation against state-of-the-art continuous batching baselines (vLLM and Orca) under realistic ShareGPT and Alpaca traffic distributions:
| Serving Framework | Max Sustainable Request Rate (req/s) | TTFT SLO Attainment | TBT SLO Attainment | Goodput Improvement |
|---|---|---|---|---|
| Orca Continuous Batching | 1.8 req/s | 64.2% | 42.8% | 1.00x (Baseline) |
| vLLM PagedAttention | 2.6 req/s | 78.4% | 51.2% | 1.44x |
| vLLM + Chunked Prefill | 3.4 req/s | 82.1% | 74.5% | 1.88x |
| DistServe Disaggregated | 8.1 req/s | 99.4% | 99.1% | 4.48x Increase |
Under tight latency constraints (TTFT $\le 200\text{ms}$, TBT $\le 40\text{ms}$), co-located vLLM collapses at 3.4 requests per second due to prefill-decode interference. DistServe sustains 8.1 requests per second while achieving over 99% SLO compliance, delivering a 4.48x goodput increase.
P99 Tail Latency Stabilization and GPU Resource Savings
Latency stability and hardware cost reduction:
| Operational Metric | Co-Located Serving (vLLM) | DistServe Architecture | Operational Impact |
|---|---|---|---|
| P99 TBT Tail Latency | 184.2 ms (Severe stutter) | 28.6 ms (Smooth stream) | 6.44x Lower Jitter |
| P99 TTFT Tail Latency | 1,420 ms | 310 ms | 4.58x Faster First Token |
| GPU Nodes to Meet SLO | 32 GPUs | 10 GPUs | 3.2x Hardware Savings |
| TBT SLO Violation Rate | 48.8% Violations | < 0.5% Violations | Rock-solid SLA guarantees |
By eliminating interference, DistServe stabilizes P99 TBT latency from 184ms down to 28.6ms, allowing operators to achieve identical SLO targets with 3.2x fewer GPUs.
Comprehensive deployment blueprints and architecture schemas can be referenced in our developer developer documentation.
Conclusion: The Disaggregated Future of Scalable Model Serving
DistServe proves that treating LLM serving as an undifferentiated, co-located workload is fundamentally flawed. By disaggregating compute-bound prefills from memory-bound decodes, matching parallelism strategies to hardware demands, and streaming KV states over high-speed networks, DistServe unlocks massive goodput gains and guarantees deterministic latency.
As enterprise AI adoption scales and long-context agentic reasoning dominates production traffic, disaggregated serving architectures will define the modern infrastructure standard for high-performance foundation model deployment.
Explore our comprehensive AI Fetchly platform to discover how cutting-edge inference architectures and automated data pipelines power the next generation of scalable autonomous intelligence.