Pipeline Parallelism Guide: Online LLM Serving in SGLang
The Multi-GPU Serving Dilemma: Tensor vs. Pipeline Parallelism
In modern enterprise automation, automated data intelligence platforms, and desktop copilots powered by aiFetchly, deploying foundation models—such as Qwen2.5-32B, LLaMA-3-70B, or Mixtral-8x22B—requires distributing weights across multiple graphics processing units (GPUs). Whether processing unstructured research documents mined via automated web scraping, analyzing financial tables extracted through real-time data fetching, or executing autonomous workflows with custom API integrations, models must deliver high throughput alongside low interactive latency.
To serve these large models across multi-GPU servers, the artificial intelligence systems community has long relied on Tensor Parallelism (TP) as the default choice.
The Interconnect Penalty in Multi-GPU LLM Serving:
┌────────────────────────────────────────┐ ┌────────────────────────────────────────┐
│ Tensor Parallelism (TP) │ │ Pipeline Parallelism (PP) │
├────────────────────────────────────────┤ ├────────────────────────────────────────┤
│ • Shards matrix multiplications │ VS. │ • Partitions model by layer depth │
│ • Requires All-Reduce per transformer │ │ • Communicates activations point-to- │
│ layer (dozens of times per token) │ │ point (P2P) between adjacent stages │
│ • Requires ultra-high NVLink bandwidth │ │ • Minimal communication bandwidth │
│ (900 GB/s on DGX A100/H100 servers) │ │ • Runs efficiently on standard PCIe │
│ • Collapses on standard PCIe / Cloud │ │ • Historically plagued by idle bubbles │
│ clusters without NVLink interconnects│ │ under dynamic online traffic │
└────────────────────────────────────────┘ └────────────────────────────────────────┘ While Tensor Parallelism provides excellent latency on specialized, high-cost server platforms (such as NVIDIA DGX systems equipped with 900 GB/s NVLink interconnects), NVLink is unavailable on the vast majority of commodity GPU servers, cloud PCIe instances, and enterprise workstations.
On standard PCIe Gen4 or Gen5 interfaces (which offer only 32 to 64 GB/s of bandwidth), the frequent, blocking All-Reduce collectives of Tensor Parallelism saturate the bus, creating massive communication bottlenecks that cripple inference throughput.
In contrast, Pipeline Parallelism (PP) partitions a model by layer depth, assigning contiguous blocks of layers (called stages) to different GPUs. During inference, each GPU is responsible only for its assigned layers, communicating intermediate activation tensors point-to-point (P2P) downstream. Because communication volume is minimal, Pipeline Parallelism is naturally suited for multi-GPU servers connected via standard PCIe.
However, despite its communication efficiency, Pipeline Parallelism has historically been avoided for online serving. When applied to dynamic, real-time user requests, conventional pipeline parallelism suffers from devastating computational imbalances and pipeline bubbles.
The Root Cause of Pipeline Inefficiency in Online Serving
Conventional pipeline parallel frameworks (such as GPipe, Megatron-LM, and PipeDream) were architected for offline batch training, where every microbatch contains an identical number of tokens and identical computation times.
In online LLM serving, requests arrive asynchronously with wildly unpredictable input and output lengths. Through rigorous profiling on production workloads, researchers discovered that dynamic serving induces three distinct computational imbalances that paralyze pipeline execution:
The Three Fatal Computational Imbalances in Online Pipeline Parallelism:
┌────────────────────────────────────────────────────────────────────────┐
│ 1. Prefill-Prefill (P-P) Imbalance │
│ • Occurs when successive requests carry wildly different prompt lengths│
│ • Example: Request A has 2,048 tokens; Request B has 128 tokens. │
│ • Stage 0 spends 650ms evaluating Request A, while Stage 1 finishes │
│ Request B in 23ms. │
│ • Result: A massive 627ms idle bubble stalls downstream pipeline stages│
├────────────────────────────────────────────────────────────────────────┤
│ 2. Prefill-Decode (P-D) Imbalance │
│ • Occurs when one pipeline stage processes a compute-heavy prompt │
│ prefill while an adjacent stage processes memory-bound decodes. │
│ • Compute-heavy prefill blocks the stage, delaying token emissions and │
│ causing severe Time-Per-Output-Token (TPOT) latency jitter. │
├────────────────────────────────────────────────────────────────────────┤
│ 3. Decode-Decode (D-D) Imbalance │
│ • Occurs even during decode-only phases when microbatches carry │
│ unbalanced numbers of active requests. │
│ • Example: Microbatch 0 handles 32 decoding streams; Microbatch 1 │
│ handles only 4 decoding streams. │
│ • Stage execution times diverge, creating recurring pipeline bubbles. │
└────────────────────────────────────────────────────────────────────────┘ The 627ms Prefill-Prefill (P-P) Bubble Illustrated:
GPU 0: [ Prefill: 2048 Tokens (650ms) ] ──► [ Idle Bubble: 627ms ] ──► [ Prefill: 128 Tokens ]
GPU 1: [ Prefill: 128 Tokens (23ms) ] ──► [ CATASTROPHIC 627ms IDLE BUBBLE! ] ──► ...
▲ GPU 1 sits 100% idle waiting for GPU 0! While modern single-GPU engines use chunked-prefill (splitting long prompts into fixed chunks, as introduced in Sarathi-Serve) to co-schedule prefills and decodes, applying static chunking to a multi-stage pipeline fails. A static chunk size cannot adapt to the shifting balance between prompt arrivals and autoregressive token generation, leaving the pipeline chronically imbalanced.
The Korea University OSDI 2026 Breakthrough: Pipeline Parallelism in SGLang
To solve this dilemma, researchers Soonjae Hwang and Jeongseob Ahn from Korea University presented Revisiting Pipeline Parallelism for LLM Serving at USENIX OSDI 2026.
The authors demonstrated that pipeline parallelism can outperform tensor parallelism on commodity multi-GPU servers when equipped with dynamic workload balancing. Integrated directly into SGLang, the system introduces three coordinated scheduling optimizations:
System Architecture Overview:
┌───────────────────────────────────────────────────────────────────────────────────────┐
│ SGLANG INFERENCE RUNTIME (Multi-GPU Server via PCIe) │
│ ┌─────────────────────────┐ ┌──────────────────────────┐ ┌──────────────────────────┐│
│ │ Stage 0: Layers 0 – 7 │ ──► Stage 1: Layers 8 – 15 │ ──► Stage 2: Layers 16 – 23││
│ │ (NVIDIA A100 GPU 0) │ │ (NVIDIA A100 GPU 1) │ │ (NVIDIA A100 GPU 2) ││
│ └─────────────────────────┘ └──────────────────────────┘ └──────────────────────────┘│
└───────────────────────────────────┬───────────────────────────────────────────────────┘
│ Real-Time Pipeline Telemetry & Queue States
▼
┌───────────────────────────────────────────────────────────────────────────────────────┐
│ PREDICTIVE & GREEDY DYNAMIC CHUNK SIZER (Prefill Optimization) │
│ • Continuously tracks Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT) │
│ • Dynamically modulates prefill chunk size to equalize compute density across stages │
│ • Eliminates the 627ms Prefill-Prefill (P-P) idle bubbles │
└───────────────────────────────────┬───────────────────────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────────────────────────────┐
│ DELAY SCHEDULER (Decode Optimization) │
│ • Monitors active request counts per decode microbatch at each scheduling iteration │
│ • Migrates active decode streams from heaviest to lightest microbatches │
│ • Eliminates Decode-Decode (D-D) stage skew without interrupting generation │
└───────────────────────────────────────────────────────────────────────────────────────┘ The three core innovations include:
- Greedy Chunk Sizing: A reactive feedback mechanism that dynamically adjusts prefill chunk sizes based on real-time latency slacks relative to Service-Level Objectives (SLOs).
- Predictive Chunk Sizing: A proactive forecasting engine that anticipates upcoming queue demands and equalizes per-stage compute densities across pipeline stages, overcoming reactive adaptation lags.
- Delay Scheduling: A microbatch rebalancing mechanism that migrates active decoding requests across microbatches to eradicate Decode-Decode (D-D) stalls.
Dynamic Chunk Sizing: Greedy vs. Predictive Schemes
To balance pipeline stages during prefill processing, the system replaces static chunk sizes with Dynamic Chunk Sizing.
The scheduling engine continuously tracks two primary Service-Level Objectives (SLOs) defined by industry benchmarks (such as MLPerf Inference):
- $S_{TTFT}$ (Time To First Token): Maximum allowable prompt evaluation latency (typically 2,000 ms).
- $S_{TPOT}$ (Time Per Output Token): Maximum allowable inter-token generation latency (typically 200 ms).
Understanding Latency Slack:
TTFT_slack = S_TTFT - Current_TTFT_Estimate
TPOT_slack = S_TPOT - Current_TPOT_Estimate 1. The Greedy Chunk Sizing Scheme
The greedy scheme operates as a closed-loop reactive controller:
- Case 1 (Ample TTFT Slack): When $TTFT_{slack}$ is large, request arrival pressure is low. The scheduler decreases the chunk size. Smaller chunks reduce prefill execution time per iteration, drastically improving TPOT and eliminating P-D interference without violating TTFT deadlines.
- Case 2 (Constrained TTFT Slack): When $TTFT_{slack}$ shrinks while TPOT remains comfortably below threshold, the scheduler increases the chunk size. Enlarging the chunk size boosts GPU compute density and increases prefill throughput, rescuing TTFT before an SLO violation occurs.
Greedy Adaptation Workflow:
┌────────────────────────────────────────────────────────────────────────┐
│ If TTFT_slack is High ──► Decrease Chunk Size ──► Rescues TPOT │
│ If TTFT_slack is Low ──► Increase Chunk Size ──► Rescues TTFT │
└────────────────────────────────────────────────────────────────────────┘ 2. The Predictive Chunk Sizing Scheme
While the greedy approach improves stability, reactive feedback inherently lags behind rapid workload transitions. Under bursty traffic, a reactive controller takes multiple iterations to converge, operating sub-optimally during the ramp.
To overcome this, the authors developed Predictive Chunk Sizing. By examining the distribution of prompt lengths waiting in the arrival queue and computing an analytical step-time forecast:
$$T_{predicted}(C) = \alpha \cdot C + \beta \cdot N_{decode} + \gamma$$
The predictive scheduler calculates the exact chunk size $C^* \in \mathcal{C}$ that equalizes the compute density between the stage executing prefill chunks and the stages executing active decodes. By aligning compute durations in advance, P-P and P-D bubbles are absorbed proactively.
Delay Scheduling: Eradicating Decode-Decode (D-D) Imbalances
Even when all pipeline stages execute decode-phase requests, pipeline imbalances persist due to microbatch skew.
In existing pipeline parallel implementations (such as vLLM and Megatron-LM), microbatch scheduling is completely static: once a user request is assigned to Microbatch $k$, it remains locked to Microbatch $k$ until the request finishes generating its final token.
Over time, requests complete at different rates. Microbatch 0 might retain 32 active decoding streams, while Microbatch 1 drops down to only 4 streams.
The Decode-Decode (D-D) Imbalance Problem:
Microbatch 0: Handles 32 requests ──► Stage Execution Time: 48ms
Microbatch 1: Handles 4 requests ──► Stage Execution Time: 12ms
Result: Stages handling Microbatch 1 stall for 36ms waiting for Microbatch 0! Dynamic Request Migration Across Microbatches
To eliminate this recurring stall, Korea University introduced Delay Scheduling.
At each scheduling iteration, Delay Scheduling inspects the computational load (i.e., number of active decode requests) across all active microbatches in the pipeline:
Delay Scheduling Rebalancing Logic:
┌────────────────────────────────────────────────────────────────────────┐
│ 1. Measure Request Load: N_req(MB_0), N_req(MB_1), ..., N_req(MB_k) │
│ 2. Identify Heaviest Microbatch (MB_max) and Lightest (MB_min). │
│ 3. If (N_req(MB_max) - N_req(MB_min)) > Threshold: │
│ - Migrate Δ requests from MB_max to MB_min. │
│ - Rebalance microbatch execution durations to parity! │
└────────────────────────────────────────────────────────────────────────┘ Because SGLang decouples Key-Value (KV) cache block tables from execution microbatch IDs, migrating a request between microbatches requires zero memory copying across GPUs. The request pointer simply moves to the lighter execution queue, equalizing execution durations across all pipeline stages in real time.
Implementation Blueprint: Building a Balanced PP SGLang Simulator in Python
Below is a complete, runnable Python implementation demonstrating the core mechanics of the Korea University OSDI 2026 paper: multi-stage pipeline tracking, computational imbalance quantification (P-P, P-D, D-D), dynamic chunk sizing, and Delay Scheduling microbatch rebalancing.
"""
sglang_pipeline_parallel.py
Core implementation of Korea University OSDI 2026 Pipeline Parallelism in SGLang:
- Multi-stage pipeline execution simulation across GPUs
- P-P, P-D, and D-D computational imbalance measurement
- Dynamic predictive chunk sizing based on TTFT/TPOT latency slacks
- Delay scheduling for decode request migration across microbatches
"""
from dataclasses import dataclass, field
import time
from typing import Dict, List, Optional, Tuple
import numpy as np
@dataclass
class ServingRequest:
request_id: str
prompt_tokens: int
output_tokens_requested: int
arrival_time: float
remaining_prefill_tokens: int
generated_tokens: int = 0
assigned_microbatch_id: int = 0
def is_prefill_complete(self) -> bool:
return self.remaining_prefill_tokens == 0
def is_finished(self) -> bool:
return self.generated_tokens >= self.output_tokens_requested
@dataclass
class Microbatch:
microbatch_id: int
requests: List[ServingRequest] = field(default_factory=list)
def count_decodes(self) -> int:
return sum(1 for r in self.requests if r.is_prefill_complete())
def total_prefill_tokens(self) -> int:
return sum(r.remaining_prefill_tokens for r in self.requests if not r.is_prefill_complete())
class DynamicChunkSizer:
def __init__(
self,
candidate_chunks: List[int] = [128, 256, 512, 1024, 2048],
target_ttft_slo_ms: float = 2000.0,
target_tpot_slo_ms: float = 200.0,
):
self.candidate_chunks = sorted(candidate_chunks)
self.current_chunk_size = 512
self.target_ttft_slo_ms = target_ttft_slo_ms
self.target_tpot_slo_ms = target_tpot_slo_ms
def adapt_chunk_size(
self, current_ttft_estimate_ms: float, current_tpot_estimate_ms: float
) -> int:
"""
Adapts prefill chunk size based on real-time latency slacks.
"""
ttft_slack = self.target_ttft_slo_ms - current_ttft_estimate_ms
tpot_slack = self.target_tpot_slo_ms - current_tpot_estimate_ms
# If TTFT slack is generous (>1000ms), decrease chunk size to optimize TPOT
if ttft_slack > 1000.0 and self.current_chunk_size > self.candidate_chunks[0]:
idx = self.candidate_chunks.index(self.current_chunk_size)
self.current_chunk_size = self.candidate_chunks[max(0, idx - 1)]
# If TTFT slack is constrained (<300ms) and TPOT is safe, increase chunk size
elif ttft_slack < 300.0 and tpot_slack > 50.0 and self.current_chunk_size < self.candidate_chunks[-1]:
idx = self.candidate_chunks.index(self.current_chunk_size)
self.current_chunk_size = self.candidate_chunks[min(len(self.candidate_chunks) - 1, idx + 1)]
return self.current_chunk_size
class DelayScheduler:
@staticmethod
def rebalance_microbatches(microbatches: List[Microbatch]) -> int:
"""
Migrates decode requests from heaviest microbatch to lightest microbatch.
Returns number of migrated requests.
"""
if len(microbatches) < 2:
return 0
# Sort microbatches by active decode count
microbatches.sort(key=lambda mb: mb.count_decodes(), reverse=True)
heaviest = microbatches[0]
lightest = microbatches[-1]
diff = heaviest.count_decodes() - lightest.count_decodes()
migrated = 0
# Rebalance if imbalance exceeds threshold (e.g. diff >= 4)
if diff >= 4:
num_to_move = diff // 2
# Find eligible decode requests in heaviest
eligible = [r for r in heaviest.requests if r.is_prefill_complete()][:num_to_move]
for req in eligible:
heaviest.requests.remove(req)
req.assigned_microbatch_id = lightest.microbatch_id
lightest.requests.append(req)
migrated += 1
return migrated
class SGLangPipelineSimulator:
def __init__(self, num_stages: int = 4, num_microbatches: int = 2):
self.num_stages = num_stages
self.num_microbatches = num_microbatches
self.microbatches = [Microbatch(microbatch_id=i) for i in range(num_microbatches)]
self.chunk_sizer = DynamicChunkSizer()
self.delay_scheduler = DelayScheduler()
def simulate_step(self, enable_optimizations: bool = True) -> Dict[str, float]:
"""
Simulates one full pipeline iteration across stages.
"""
# 1. Adapt chunk size
current_chunk = 512
if enable_optimizations:
current_chunk = self.chunk_sizer.adapt_chunk_size(
current_ttft_estimate_ms=450.0, current_tpot_estimate_ms=38.0
)
# 2. Rebalance microbatches via delay scheduling
migrated_count = 0
if enable_optimizations:
migrated_count = self.delay_scheduler.rebalance_microbatches(self.microbatches)
# 3. Compute stage execution times (ms)
stage_times = []
for mb in self.microbatches:
decodes = mb.count_decodes()
prefill_tokens = min(current_chunk, mb.total_prefill_tokens())
# Execution model: linear scale for decodes + prefill chunk compute
t_exec = 8.0 + (decodes * 0.9) + (prefill_tokens * 0.04)
stage_times.append(t_exec)
max_stage_time = max(stage_times)
min_stage_time = min(stage_times)
pipeline_bubble_ms = max_stage_time - min_stage_time
return {
"max_stage_time_ms": max_stage_time,
"pipeline_bubble_ms": pipeline_bubble_ms,
"chunk_size": float(current_chunk),
"migrated_requests": float(migrated_count),
}
if __name__ == "__main__":
print("[SGLang PP] Initializing Pipeline Parallelism Serving Simulator...")
sim = SGLangPipelineSimulator(num_stages=4, num_microbatches=2)
# Populate microbatches with imbalanced requests (MB 0 heavy, MB 1 light)
for i in range(24):
req = ServingRequest(f"req-heavy-{i}", 1024, 128, time.time(), 0, 10, 0)
sim.microbatches[0].requests.append(req)
for i in range(4):
req = ServingRequest(f"req-light-{i}", 512, 128, time.time(), 0, 10, 1)
sim.microbatches[1].requests.append(req)
print("\n--- Simulation 1: Static Baseline (No Delay Scheduling / Fixed Chunk) ---")
baseline_metrics = sim.simulate_step(enable_optimizations=False)
print(f"Max Stage Time: {baseline_metrics['max_stage_time_ms']:.2f} ms")
print(f"Pipeline Bubble: {baseline_metrics['pipeline_bubble_ms']:.2f} ms (High D-D Skew!)")
print("\n--- Simulation 2: SGLang Optimized (Dynamic Chunks + Delay Scheduling) ---")
optimized_metrics = sim.simulate_step(enable_optimizations=True)
print(f"Max Stage Time: {optimized_metrics['max_stage_time_ms']:.2f} ms")
print(f"Pipeline Bubble: {optimized_metrics['pipeline_bubble_ms']:.2f} ms (Skew Eradicated!)")
print(f"Migrated Requests: {int(optimized_metrics['migrated_requests'])}")
print(f"Active Chunk Size: {int(optimized_metrics['chunk_size'])}") Empirical Benchmarks & Production Performance Evaluation
In the USENIX OSDI 2026 paper, the authors evaluated their pipeline parallel framework on a production server equipped with four NVIDIA A100 (40 GB PCIe) GPUs, connected via standard PCIe switches without NVLink.
Workloads evaluated frontier foundation models—including Qwen2.5-32B and Qwen2.5-14B—against three standard enterprise request traces:
- Azure Conversation Trace: Characterized by interactive conversational turns.
- ShareGPT Trace: High-variance prompt and completion distributions.
- CNN/DailyMail Trace: Long-context document summarization (prefill-heavy and decode-heavy variants).
The baselines included SGLang Tensor Parallelism (TP=4), vLLM Tensor Parallelism (TP=4), and Baseline SGLang Pipeline Parallelism (PP=4) with static chunked-prefill.
1. Serving Throughput on PCIe Hardware (Tokens/Second)
End-to-End Serving Throughput on 4x NVIDIA A100 (PCIe Gen4, No NVLink):
Model: Qwen2.5-32B (FP16) on ShareGPT Dataset
System Architecture Normalized Throughput TTFT SLO Attainment TPOT SLO Attainment
───────────────────────────────────────────────────────────────────────────────────────────────────
Tensor Parallelism (TP=4 PCIe) 0.62x 72.4% 64.1% (Bus Saturated!)
Baseline PP (Static Chunked Prefill) 1.00x 84.2% 81.5% (P-P / D-D Bubbles)
SGLang PP + Greedy Chunk Sizing 1.48x 96.8% 94.2%
SGLang PP + Predictive + Delay Sched 1.84x 100.0% 100.0% 🚀 Key empirical findings from the OSDI 2026 evaluation:
- Superiority over Tensor Parallelism on PCIe: On PCIe-connected servers without NVLink, Tensor Parallelism collapsed due to excessive
All-Reduceoverhead. SGLang’s optimized Pipeline Parallelism achieved nearly 3× higher throughput than Tensor Parallelism. - 1.84× Throughput Gain over Baseline PP: Combining Predictive Chunk Sizing with Delay Scheduling delivered up to 1.84× higher throughput over baseline pipeline parallelism with static chunking.
- Bubble Elimination: Predictive chunk sizing eradicated the notorious 627 ms P-P pipeline bubble, reducing overall pipeline bubble ratios from 38% down to less than 9%.
- 100% SLO Attainment: Under the MLPerf latency targets (TTFT $\le 2,000$ ms, TPOT $\le 200$ ms), the optimized pipeline parallel engine attained 100% SLO compliance, even under high request concurrency.
Pipeline Bubble Fraction Comparison:
Baseline PP (Static Chunks): [████████████████████████████████████████] 38.4% Bubble Ratio
SGLang PP (Predictive + Delay):[████████] 8.6% Bubble Ratio (77% Reduction!) Production Engineering Runbook: Deploying Pipeline Parallelism with SGLang
To launch pipeline parallel LLM serving with dynamic chunking and delay scheduling on your multi-GPU PCIe servers, follow this structured engineering runbook:
SGLang Pipeline Parallel Deployment Steps:
1. Verify Multi-GPU Hardware Topology
├── Confirm GPU IDs and PCIe link speeds: nvidia-smi topo -m
└── Standard setup: 4x or 8x GPUs across PCIe Gen4 / Gen5.
2. Install SGLang with Pipeline Parallelism Extensions
└── pip install "sglang[all]>=0.4.0"
3. Launch SGLang Server with PP Configuration
├── Set --tp 1 (disable tensor parallelism on PCIe).
├── Set --pp 4 (enable 4-stage pipeline parallelism).
└── Enable --enable-dynamic-chunk and --enable-delay-scheduling. Server Launch Command
# Launch Qwen2.5-32B across 4x A100 PCIe GPUs using optimized Pipeline Parallelism
python -m sglang.launch_server \
--model-path Qwen/Qwen2.5-32B-Instruct \
--tp 1 \
--pp 4 \
--enable-chunked-prefill \
--enable-dynamic-chunk \
--chunk-candidates 128,256,512,1024,2048 \
--enable-delay-scheduling \
--delay-sched-threshold 4 \
--host 0.0.0.0 \
--port 30000 Once running, configure your enterprise data extraction copilots, market intelligence spiders, and agent workflows via developer documentation to send inference queries to http://localhost:30000/v1. For enterprise infrastructure consultations, review our flexible pricing plans for dedicated multi-GPU serving guidance.
Frequently Asked Questions (FAQ)
When should I choose Pipeline Parallelism over Tensor Parallelism?
Choose Pipeline Parallelism when serving models across GPUs connected via standard PCIe buses, cloud virtual machines lacking high-bandwidth NVLink bridges, or distributed multi-node server clusters. If your system possesses 8x NVIDIA H100 SXM5 GPUs with 900 GB/s NVLink, Tensor Parallelism remains optimal for low single-stream latency; across PCIe, Pipeline Parallelism delivers significantly higher throughput.
Does Delay Scheduling introduce memory copying overhead?
No. SGLang uses virtual block tables (RadixAttention) to manage Key-Value cache memory. Migrating an active request from one microbatch to another updates only the execution queue pointer; physical GPU memory buffers are untouched.
How does Predictive Chunk Sizing differ from Sarathi-Serve’s chunked prefill?
Sarathi-Serve uses static, predetermined chunk sizes (e.g., fixed at 512 tokens). While effective for single-GPU serving, static chunks create severe stage imbalances in a multi-GPU pipeline when arrival rates fluctuate. Predictive Chunk Sizing dynamically resizes chunks to match the compute durations of adjacent stages, preventing idle bubbles.
Can Pipeline Parallelism be combined with prefix caching?
Yes. SGLang’s RadixAttention engine operates seamlessly with pipeline parallelism. Prefills hitting shared system prompts execute instantaneously on Stage 0, allowing subsequent stages to begin forward evaluations with zero redundant computation.