Chunked Prefill: Taming Latency Spikes in LLM Serving
The Problem: Prefill-Decode Interference and the Latency Jitter Wall
In production Large Language Model (LLM) serving, requests transition through two fundamentally distinct phases governed by opposing computational characteristics on modern GPUs:
- The Prefill Phase (Prompt Ingestion): The model processes all input tokens simultaneously to initialize the Key-Value (KV) cache. Because operations occur across a full sequence matrix, prefill is compute-bound (GEMM). It boasts high arithmetic intensity (frequently $>100 \text{ FLOPs per byte}$), saturating GPU Tensor Cores. Processing a long 4,096-token or 8,192-token prompt can monopolize an entire GPU for 500ms to 1,500ms.
- The Decode Phase (Autoregressive Generation): The model generates one token at a time sequentially. Because only a single token vector is multiplied against the model weights and past KV cache, decoding is memory-bandwidth bound (GEMV). It exhibits low arithmetic intensity ($<2 \text{ FLOPs per byte}$), leaving GPU compute cores over 80% idle while waiting for weights to stream from High Bandwidth Memory (HBM). Each decoding step typically executes in 20ms to 35ms.
In early continuous batching systems (such as Orca and baseline vLLM), schedulers treated prompt prefills as atomic, monolithic operations. When a new long-prompt request arrived at the server, the scheduler immediately allocated GPU resources to run its full prefill.
The Consequence: Catastrophic TPOT Spikes
This monolithic scheduling model creates severe Prefill-Decode Interference:
- Decode Starvation: While the GPU executes a massive 1,000ms prefill kernel, all 64 ongoing interactive decode streams are completely frozen.
- Perceived Latency Stutter: In human-facing chat interfaces, agentic tool loops, and real-time voice agents, users perceive any delay greater than 50ms as a stutter. A sudden 800ms freeze destroys real-time responsiveness.
- Service Level Objective (SLO) Violations: While average Time Per Output Token (TPOT) may appear acceptable, the tail latency (P99 TPOT) explodes by 10x to 25x, violating enterprise latency SLAs.
Historically, engineering teams attempted to separate prefills and decodes onto dedicated physical GPU clusters (Disaggregated Prefill-Decode, such as Splitwise). While effective for massive hyper-scale data centers, disaggregation introduces substantial network latency, requires expensive multi-gigabyte RDMA KV cache transfers across InfiniBand switches, and remains impractical for single-node deployments and local workstations.
The Solution: Chunked Prefill and Piggybacking
Pioneered in research by Agrawal et al. (Sarathi-Serve, OSDI 2024) and standardized in production engines like vLLM v1 and SGLang, Chunked Prefill re-engineers iteration-level batch scheduling through two foundational mechanisms:
- Prefill Slicing: Long input prompts are divided into fixed-size token chunks (typically $C = 512$ or $C = 1,024$ tokens). Rather than executing an 8,192-token prompt in one monolithic 800ms burst, the scheduler executes 16 discrete 512-token chunks over 16 consecutive scheduling iterations.
- Piggybacking (Decode Co-Scheduling): In every scheduling step, ongoing decode requests are never paused. Instead, the scheduler constructs a heterogeneous batch combining all active single-token decodes alongside a single prefill chunk, governed by a global token budget (e.g., $T_{\text{budget}} = 544$ tokens).
The Hardware Synergy: Why Piggybacking is Essentially “Free”
The mathematical genius of piggybacking lies in hardware resource complementarity:
$$\text{Total Work} = \underbrace{\text{Decodes (Memory-Bandwidth Bound)}}{\text{Saturates HBM Read Bus}} + \underbrace{\text{Prefill Chunk (Compute Bound)}}{\text{Saturates Idle Tensor Cores}}$$
During an isolated decode iteration, the GPU streams 140 GB of model weights from HBM to compute only a few dozen matrix-vector dot products—leaving Tensor Cores idling in an unutilized state.
By co-scheduling a 512-token prefill chunk into that exact same forward pass, the model re-uses the model weights already streamed into GPU SRAM to compute high-density GEMM operations. The prefill computation is effectively hidden beneath the memory-bandwidth latency wall of the decode tokens!
As a result, an iteration containing 32 decodes plus a 512-token prefill chunk executes in ~35ms—scarcely longer than an isolated decode step—while making steady, deterministic progress on the prompt.
Key Architectural Mechanics
Implementing chunked prefill inside modern attention kernels requires solving several algorithmic challenges regarding attention masking and state management:
1. Mixed Attention Masking (Causal Self-Attention + Historical Cross-Attention)
When processing chunk $k$ of a long prompt (spanning token indices $[k \cdot C, (k+1) \cdot C - 1]$):
- Intra-Chunk Attention: Tokens within chunk $k$ attend to one another using standard lower-triangular causal masking.
- Inter-Chunk Attention: Tokens within chunk $k$ attend to all historical KV cache activations generated by chunks $0, 1, \dots, k-1$ using an unmasked, full cross-attention matrix.
Modern attention kernels (such as FlashAttention-2, FlashAttention-3, and FlashInfer) provide native APIs for varlen (variable-length) mixed prefill-decode batches, computing block-sparse attention across heterogeneous sequence lengths without padding overhead.
2. Token Budgeting and Slot Scheduling
In each scheduling cycle, the engine calculates the allowable prefill chunk size based on active decode demand:
$$C_{\text{allowed}} = \min\left(C_{\max}, \quad T_{\text{budget}} - N_{\text{active decodes}}\right)$$
If 32 requests are currently in the decoding phase and $T_{\text{budget}} = 544$, the scheduler allocates exactly $544 - 32 = 512$ tokens for the leading prefill chunk. If decode concurrency swells to 128 requests, the prefill chunk size automatically scales down to $544 - 128 = 416$ tokens.
This mathematical ceiling guarantees that the total execution time of any single GPU iteration remains strictly bounded, preventing latency spikes regardless of incoming traffic bursts.
3. Progressive KV Cache Allocation
Under PagedAttention, chunked prefill integrates seamlessly with virtual memory block tables:
- As chunk $k$ is processed, only the physical blocks required to store its 512 tokens are allocated from the free frame pool.
- Once the final chunk is processed, the request’s first output token is generated, and the request seamlessly transitions into the active decode pool.
Serving Architecture Comparison Matrix
| Architectural Dimension | Monolithic Batching | Continuous Batching (No Chunking) | Disaggregated Serving (Splitwise) | Chunked Prefill & Piggybacking |
|---|---|---|---|---|
| Prefill Scheduling | Static batch at start | Monolithic single-pass | Dedicated prefill cluster | Sliced into fixed-size chunks |
| Decode Interleaving | None (All wait) | Paused during prefills | Physically decoupled | Concurrent (Piggybacked) |
| Hardware Efficiency | Low (<25% GPU util) | Medium (Periodic idle) | High (Specialized nodes) | Optimal (Simultaneous HBM & Tensor Core saturation) |
| P99 TPOT Variance | Extreme (>2,000ms) | Severe (400ms – 1,200ms) | Low (<30ms) | Extremely Low (<40ms) |
| Cluster Complexity | Minimal | Minimal | Very High (RDMA, InfiniBand, KV network transfer) | Low (Runs on single GPU or standard cluster) |
| Time to First Token (TTFT) | Slow | Fast for single prompts | Fast | Slight increase for long prompts; stable overall |
Reference Implementation: Chunked Prefill Scheduler Simulator
Below is a complete Python reference implementation demonstrating how a production-grade inference engine organizes heterogeneous requests, calculates token budgets, slices long prompts, and executes piggybacked iterations.
from dataclasses import dataclass
from typing import List, Optional
import time
@dataclass
class Request:
req_id: str
prompt_tokens: int
max_decode_tokens: int
prompt_tokens_processed: int = 0
tokens_generated: int = 0
arrival_time: float = 0.0
@property
def is_prefill_complete(self) -> bool:
return self.prompt_tokens_processed >= self.prompt_tokens
@property
def is_finished(self) -> bool:
return self.is_prefill_complete and (self.tokens_generated >= self.max_decode_tokens)
@property
def remaining_prefill_tokens(self) -> int:
return max(0, self.prompt_tokens - self.prompt_tokens_processed)
class ChunkedPrefillScheduler:
"""
Simulates a token-budgeted Chunked Prefill and Piggybacking engine.
"""
def __init__(self, max_batched_tokens: int = 544, default_chunk_size: int = 512):
self.max_batched_tokens = max_batched_tokens
self.default_chunk_size = default_chunk_size
self.waiting_queue: List[Request] = []
self.running_decodes: List[Request] = []
def add_request(self, request: Request):
self.waiting_queue.append(request)
def schedule_iteration(self) -> dict:
"""
Builds a co-scheduled heterogeneous iteration batch:
1. Guarantees priority to all active ongoing decode tokens.
2. Fills remaining token budget with a sliced prefill chunk.
"""
batch_decodes: List[Request] = []
prefill_task: Optional[dict] = None
total_tokens = 0
# Step 1: Always service active decodes first to maintain streaming TPOT
for req in self.running_decodes:
if not req.is_finished:
batch_decodes.append(req)
total_tokens += 1
# Step 2: Calculate remaining token budget for prefill chunking
budget_remaining = self.max_batched_tokens - total_tokens
# Step 3: Inspect waiting queue to select or continue a prefill chunk
if self.waiting_queue and budget_remaining > 0:
target_req = self.waiting_queue[0]
chunk_tokens = min(target_req.remaining_prefill_tokens, budget_remaining, self.default_chunk_size)
target_req.prompt_tokens_processed += chunk_tokens
total_tokens += chunk_tokens
prefill_task = {
"req_id": target_req.req_id,
"chunk_size": chunk_tokens,
"remaining": target_req.remaining_prefill_tokens,
"completed": target_req.is_prefill_complete
}
# If this chunk completed the prompt, transition to running decodes
if target_req.is_prefill_complete:
self.waiting_queue.pop(0)
self.running_decodes.append(target_req)
return {
"decode_count": len(batch_decodes),
"prefill_task": prefill_task,
"total_tokens": total_tokens
}
def step(self):
"""Simulates execution of one forward iteration."""
iteration_info = self.schedule_iteration()
# Simulate generating 1 token for each active decode
for req in self.running_decodes:
if req.is_prefill_complete and not req.is_finished:
req.tokens_generated += 1
# Clean up finished requests
self.running_decodes = [r for r in self.running_decodes if not r.is_finished]
return iteration_info
# Demonstration: An 8k prompt arriving while 8 active decodes are streaming
if __name__ == "__main__":
scheduler = ChunkedPrefillScheduler(max_batched_tokens=544, default_chunk_size=512)
# Pre-populate with 8 active decode streams
for i in range(8):
req = Request(req_id=f"active_decode_{i}", prompt_tokens=100, max_decode_tokens=20)
req.prompt_tokens_processed = 100 # Prefill already finished
scheduler.running_decodes.append(req)
# A massive 2,048-token prompt arrives
long_prompt = Request(req_id="massive_prompt_1", prompt_tokens=2048, max_decode_tokens=10)
scheduler.add_request(long_prompt)
print("--- Executing Scheduling Iterations with Chunked Prefill ---")
for iter_idx in range(1, 8):
info = scheduler.step()
p_task = info["prefill_task"]
if p_task:
task_str = f"Chunk: {p_task['chunk_size']} tokens (Remaining in prompt: {p_task['remaining']})"
else:
task_str = "None (Prefill finished)"
print(f"Iter {iter_idx:02d} | Decodes: {info['decode_count']:02d} | Total Tokens: {info['total_tokens']:03d} | Prefill {task_str}") Production Deployment and Ecosystem Integration
Chunked prefill has become an essential configuration primitive in enterprise model serving:
1. vLLM and SGLang Configuration
In modern vLLM, chunked prefill is enabled out of the box in v1 or activated via CLI flags:
# Production launch script enabling chunked prefill in vLLM
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--enable-chunked-prefill \
--max-num-batched-tokens 512 \
--max-num-seqs 256 \
--tensor-parallel-size 4 --enable-chunked-prefill: Activates prefill slicing and heterogeneous piggybacking.--max-num-batched-tokens: Governs $T_{\text{budget}}$. Setting this to 512–1,024 achieves an optimal balance, ensuring iterations complete within 30ms–45ms while providing ample compute density for prompt ingestion.
2. Impact on Interactive Chat and Agentic Workflows
In agentic swarms (such as automated web researchers or coding assistants), agents frequently submit prompts containing 16k–32k tokens of scraped documentation or database schemas. Without chunked prefill, a single agent prompt freezes all concurrent user queries. With chunked prefill, the agent’s prompt ingests smoothly across background iterations without disrupting interactive users.
3. Local Desktop AI Acceleration in aiFetchly
For engineering teams and analysts deploying local inference models via AiFetchly, chunked prefill provides a vital stabilizing mechanism. When connecting aiFetchly desktop clients to local Ollama or vLLM instances running on high-end workstation hardware (such as NVIDIA RTX 4090 or Apple Silicon Mac Studio), chunked prefill prevents long document extraction tasks from stalling concurrent UI interactions, delivering consistent token streaming and fluid AI assistance.