Distributed SystemsLLM ServingPrefill OptimizationCUDA GraphsMLSys 2026

PLA-Serve Guide: Prefill-Length-Aware LLM Serving in 2026

October 14, 2026

The Prefill Heterogeneity Bottleneck in Modern LLM Serving

The evolution of modern distributed large language model (LLM) serving has increasingly centered on Disaggregated Prefill and Decode (PD Disaggregation). In foundational systems architectures like DistServe, Splitwise, and Mooncake, the serving pipeline divides physical GPU worker pools into two specialized tiers:

Prefill Instances: Optimized for compute-bound General Matrix Multiplications (GEMMs) over input prompts to generate initial Key-Value (KV) cache activations.
Decode Instances: Optimized for memory-bandwidth-bound autoregressive token generation across expanding KV cache buffers.

By physically decoupling prompt processing from token generation, PD disaggregation effectively insulates decoding latency—measured by Time-Between-Tokens (TBT)—from the computational spikes of incoming prompt prefills.

However, resolving the interference between prefill and decode revealed an even deeper, previously unaddressed bottleneck: Extreme Length Heterogeneity Within the Prefill Tier Itself.

+---------------------------------------------------------------------------------------------------------+
|                                  THE PREFILL HETEROGENEITY DILEMMA                                      |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|   INCOMING WORKLOAD DISTRIBUTION (Bimodal & Long-Tailed Prompt Distribution):                           |
|   - Short-Prefill Requests: 64 to 512 tokens (e.g., Conversational greetings, single-turn QA, queries)  |
|   - Long-Prefill Requests: 4,096 to 64,000+ tokens (e.g., Multi-turn history, dense RAG, code analysis) |
|                                                                                                         |
|   NAIVE HOMOGENEOUS BATCHING (SGLang / vLLM / Standard PD Disaggregation):                              |
|   +-------------------------------------------------------------------------------------------------+   |
|   | GPU Execution Batch: [ Short (128 tok) ][ Long (16,384 tok) ][ Short (256 tok) ]                 |   |
|   +-------------------------------------------------------------------------------------------------+   |
|                                                    |                                                    |
|                                                    v                                                    |
|   THREE SYSTEM-LEVEL PATHOLOGIES:                                                                       |
|   1. Head-of-Line Blocking: Short prompts stall behind massive O(L^2) attention GEMMs for seconds.     |
|   2. CUDA Graph Breakdown: Heterogeneous shapes force eager CPU kernel launches; GPU sits idle.        |
|   3. Router Blindness: Load balancers route by token count, misjudging quadratic compute complexity.    |
+---------------------------------------------------------------------------------------------------------+

The Three Pathologies of Naive Prefill Serving

In production environments, user traffic rarely conforms to uniform prompt lengths. Across typical conversational traces, requests follow an extreme bimodal distribution: millions of short prompts (under 512 tokens) arriving alongside heavy enterprise requests (multi-turn chat histories, legal document reviews, and code repositories exceeding 16,000 to 64,000 tokens).

When serving systems treat all prefill requests as homogeneous tasks within a common scheduling queue, three fatal performance breakdowns occur:

Head-of-Line Latency Blocking: Because transformer self-attention scales quadratically with sequence length ($\mathcal{O}(L^2)$ in compute and $\mathcal{O}(L)$ in memory bandwidth), processing a 16,384-token prompt takes orders of magnitude longer than a 256-token query. When a short request is batched with or queued behind a long request, the short request suffers massive queue wait times, blowing past interactive Time-to-First-Token (TTFT) Service Level Objectives (SLOs).
CUDA Graph Fragmentation and Kernel Launch Overhead: CUDA Graph execution is the cornerstone of sub-millisecond GPU inference, eliminating CPU-side kernel dispatch latency and driver synchronization. However, CUDA Graphs require strictly static tensor dimensions and pre-allocated memory buffers. When short and long prompts are batched together arbitrarily, dynamic sequence padding and jagged tensor dimensions shatter shape uniformity. Frameworks are forced to fall back to eager PyTorch kernel launches, leaving GPU streaming multiprocessors (SMs) starving during kernel launch bubbles.
Router Metric Blindness: Existing distributed routers (such as the standard SGLang router or vLLM round-robin proxies) balance load by counting raw request numbers or aggregate token volumes. However, a single 32,000-token prompt generates $32,000^2 \approx 1.02 \times 10^9$ attention operations, whereas 64 distinct 500-token prompts require only $64 \times 500^2 \approx 1.6 \times 10^7$ operations—a 64x difference in computational intensity despite representing identical token counts! Naive routing inadvertently assigns massive computational spikes to already congested prefill instances.

To resolve these challenges, a research team from the Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) and Carnegie Mellon University (CMU) introduced PLA-Serve at MLSys 2026. PLA-Serve (Length-Aware Prefill Serving) introduces an architectural redesign for the prefill tier: a dual-queue architecture that segregates long and short prefills, combines dynamic batch waiting windows with CUDA Graph clustering, and deploys quadratic-aware cluster routing.

Deployed on top of state-of-the-art serving backends (including SGLang and vLLM), PLA-Serve reduces prefill latency by over 30%, slashes multi-instance SLO violations by 28%, and boosts serving throughput by 35% on modern foundation models like Qwen2.5-32B.

Architectural Foundations of PLA-Serve

PLA-Serve reorganizes the prefill serving tier by transforming the monolithic scheduling loop into a specialized, length-aware pipeline. Instead of pushing incoming prompts into a single FIFO (First-In, First-Out) or greedy batching engine, PLA-Serve categorizes requests based on their attention complexity profile and routes them through specialized execution paths.

+---------------------------------------------------------------------------------------------------------+
|                                       PLA-SERVE SYSTEM ARCHITECTURE                                     |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|   +-------------------------------------------------------------------------------------------------+   |
|   | INCOMING CLIENT REQUESTS (API Gateways, Web Crawlers, Enterprise Chatbots, Agent Tool Chains)   |   |
|   +-------------------------------------------------------------------------------------------------+   |
|                                                    |                                                    |
|                                                    v                                                    |
|   +-------------------------------------------------------------------------------------------------+   |
|   | 1. LENGTH-AWARE ADAPTIVE ROUTER                                                                 |   |
|   |    - Computes Quadratic Attention Complexity: C(req) = a * L + b * L^2                          |   |
|   |    - Tracks Worker-Specific Virtual Compute Backlog & HBM Utilization                           |   |
|   +-------------------------------------------------------------------------------------------------+   |
|                         /                                                   \                           |
|      Short Prefill Path (L < L_thresh)                    Long Prefill Path (L >= L_thresh)             |
|                       /                                                       \                         |
|                      v                                                         v                        |
|   +------------------------------------+                    +------------------------------------+      |
|   | 2. SHORT-PREFILL QUEUE             |                    | 3. LONG-PREFILL QUEUE              |      |
|   |    +-----------------------------+ |                    |    +-----------------------------+ |      |
|   |    | Dynamic Waiting Window (W_t)| |                    |    | Chunked Pipelined Scheduler | |      |
|   |    +-----------------------------+ |                    |    +-----------------------------+ |      |
|   |                   |                |                    |                   |                |      |
|   |                   v                |                    |                   v                |      |
|   |    +-----------------------------+ |                    |    +-----------------------------+ |      |
|   |    | CUDA Graph Bucket Clusterer | |                    |    | FlashAttention-3 VarLen     | |      |
|   |    | (Bucket: 128, 256, 512 tok) | |                    |    | Non-Graph Compute Engine    | |      |
|   |    +-----------------------------+ |                    |    +-----------------------------+ |      |
|   +------------------------------------+                    +------------------------------------+      |
|                      \                                                         /                        |
|                       \                                                       /                         |
|                        v                                                     v                          |
|   +-------------------------------------------------------------------------------------------------+   |
|   | 4. SPATIAL-TEMPORAL WORKER ARBITRATION                                                          |   |
|   |    - Single Instance Mode: Temporal Time-Slicing (Short batches interleave long chunks)         |   |
|   |    - Multi-Instance Mode: Spatial Disaggregation (Dedicated Short-GPU vs Long-GPU pools)        |   |
|   +-------------------------------------------------------------------------------------------------+   |
|                                                    |                                                    |
|                                                    v                                                    |
|   +-------------------------------------------------------------------------------------------------+   |
|   | DISAGGREGATED DECODE TIER (vLLM / SGLang Decode Workers with PagedAttention)                    |   |
|   +-------------------------------------------------------------------------------------------------+   |
+---------------------------------------------------------------------------------------------------------+

The Three Pillar Innovations

PLA-Serve comprises three coordinated systems:

Dual-Queue Length Disaggregation: Incoming prefill requests are dynamically partitioned at an empirically calibrated threshold $L_{\text{thresh}}$ (typically between 1,024 and 2,048 tokens). Requests with $L \< L_{\text{thresh}}$ are routed to the Short-Prefill Queue, while requests with $L \ge L_{\text{thresh}}$ enter the Long-Prefill Queue. This dual-queue separation eliminates head-of-line blocking, ensuring that interactive conversational turns are never delayed behind multi-second document ingestion jobs.
Dynamic Waiting Window & CUDA Graph-Based Clustering: For short-prefill requests, PLA-Serve introduces a calibrated micro-waiting window ($W_t \approx 2\text{ to }5\text{ ms}$). During this tiny window, the scheduler groups short requests into discrete length buckets (e.g., 128, 256, 512, 1,024 tokens). Each bucket maps directly to a pre-captured CUDA Graph. By executing clustered short prefills entirely via static CUDA Graph replay, PLA-Serve eliminates CPU driver overhead, maximizes GPU occupancy, and achieves ultra-low batch execution latency.
Adaptive Spatial-Temporal Dispatching: Depending on available hardware infrastructure, PLA-Serve executes either:
- Temporal Disaggregation (Single-Instance Setup): On standalone prefill workers, the engine executes short-prefill CUDA Graph batches during the scheduling gaps of chunked long prefills.
- Spatial Disaggregation (Cluster Setup): Across multi-GPU clusters, the router dedicates specialized GPU nodes to short-prefill graph clusters and other nodes to long-prefill throughput pipelines, dynamically adjusting node assignments as workload distributions shift.

The Mathematics of Prefill Complexity and Routing

Traditional distributed routers fail because they assume a linear relationship between input token count and computational load. To make prefill scheduling predictable, PLA-Serve formalizes the cost of prompt processing.

Decomposing Linear GEMM vs. Quadratic Attention

In an autoregressive transformer layer, the prefill phase involves two distinct mathematical operations:

Projection GEMMs (Linear Complexity): Projecting input tokens into Query ($Q$), Key ($K$), Value ($V$), and Output ($O$) matrices, along with Feed-Forward Network (FFN / MLP) up-projection and down-projection. The computational work is strictly linear in sequence length: $$\text{FLOPs}{\text{proj}} = L \cdot \left( 4 \cdot d{\text{model}}^2 + 2 \cdot d_{\text{model}} \cdot d_{\text{ffn}} \right)$$
Self-Attention Score Computation (Quadratic Complexity): Evaluating $Q K^T$ and multiplying by $V$ across all attention heads. Even with optimized FlashAttention kernels, the number of arithmetic operations scales with the square of sequence length: $$\text{FLOPs}{\text{attn}} = 2 \cdot L^2 \cdot d{\text{model}}$$

Combining these terms across all $N_{\text{layers}}$ transformer layers yields the total prefill computational cost: $$\text{Cost}{\text{prefill}}(L) = N{\text{layers}} \cdot \left( \alpha \cdot L + \beta \cdot L^2 \right)$$ where $\alpha$ and $\beta$ are architectural constants determined by hidden dimension $d_{\text{model}}$ and FFN intermediate dimension $d_{\text{ffn}}$.

Prefill Latency Scaling: Linear vs. Quadratic Divergence
Tokens (L)      Linear Term (alpha * L)     Quadratic Term (beta * L^2)    Relative Compute
-----------------------------------------------------------------------------------------
256             256 units                   65,536 units                   1.0x (Baseline)
512             512 units                   262,144 units                  3.8x
2,048           2,048 units                 4,194,304 units                58.2x
8,192           8,192 units                 67,108,864 units               896.0x
32,768          32,768 units                1,073,741,824 units            14,112.0x

The Length-Aware Routing Algorithm

PLA-Serve’s centralized load balancer replaces token-counting proxies with a Virtual Compute Backlog Estimator.

When a request $r_i$ arrives with prompt length $L_i$, the router calculates its estimated execution duration: $$E(r_i) = \gamma_1 \cdot L_i + \gamma_2 \cdot L_i^2$$ where $\gamma_1$ and $\gamma_2$ are empirically calibrated coefficients profiled on the target GPU architecture (e.g., NVIDIA H100 or A100).

For each prefill worker instance $k \in {1, \dots, M}$, the router maintains a virtual backlog score $B_k$: $$B_k = \sum_{r_j \in \text{Queue}_k} E(r_j)$$

To route incoming request $r_i$:

If $L_i \< L_{\text{thresh}}$, the router identifies the subset of workers configured for short-prefill CUDA Graph execution and assigns $r_i$ to the worker with the lowest $B_k$.
If $L_i \ge L_{\text{thresh}}$, the request is routed to a dedicated long-prefill instance or scheduled using chunked execution, updating $B_k$ by the full quadratic cost.

By accounting for the $\mathcal{O}(L^2)$ divergence, PLA-Serve prevents long prompts from overwhelming workers, reducing multi-instance load skew from 45% down to under 6%.

Smart Batching and CUDA Graph-Based Clustering

The crowning technical achievement of PLA-Serve is making CUDA Graphs viable for prompt prefill.

Why CUDA Graphs Historically Failed for Prefill

In autoregressive token decoding (generating 1 token at a time), CUDA Graphs are universal: every request in a batch generates exactly 1 token, meaning tensor dimensions are static ([batch_size, 1, hidden_dim]). Serving engines capture a handful of static graphs (e.g., batch sizes 1, 2, 4, 8, 16, 32, 64) and replay them with zero CPU overhead.

In prefill, however, prompt lengths vary wildly: one user sends 47 tokens, another 182 tokens, another 419 tokens. Naive padding to a fixed maximum length wastes immense memory bandwidth and tensor core cycles computing attention over useless padding tokens. As a result, systems like vLLM and SGLang disable CUDA Graphs during the prefill phase entirely, accepting substantial CPU driver dispatch overhead.

The Dynamic Waiting Window and Bucketing Strategy

PLA-Serve overcomes this trade-off through Micro-Windowed Bucketed Clustering:

+---------------------------------------------------------------------------------------------------------+
|                                    CUDA GRAPH CLUSTERING PIPELINE                                       |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|   1. INCOMING REQUEST ARRIVAL STREAM:                                                                   |
|      Req A (78 tok) ---> Req B (210 tok) ---> Req C (95 tok) ---> Req D (480 tok) ---> Req E (110 tok)    |
|                                                                                                         |
|   2. DYNAMIC MICRO-WAITING WINDOW (e.g., W_t = 3.5 ms):                                                 |
|      Accumulates requests in a micro-buffer until window expires or batch target is reached             |
|                                                                                                         |
|   3. LENGTH BUCKET CLUSTERING:                                                                          |
|      +---------------------------------+     +---------------------------------+                        |
|      | Bucket 128 (Tokens <= 128)      |     | Bucket 256 (129 < Tokens <= 256)|                        |
|      | - Req A (pad 78 -> 128)         |     | - Req B (pad 210 -> 256)        |                        |
|      | - Req C (pad 95 -> 128)         |     +---------------------------------+                        |
|      | - Req E (pad 110 -> 128)        |     +---------------------------------+                        |
|      | Total batch: 3 requests         |     | Bucket 512 (257 < Tokens <= 512)|                        |
|      +---------------------------------+     | - Req D (pad 480 -> 512)        |                        |
|                       |                      +---------------------------------+                        |
|                       v                                                                                 |
|   4. STATIC CUDA GRAPH REPLAY:                                                                          |
|      Dispatches directly to pre-captured CUDA Graph: `graph_prefill_b3_seq128`                          |
|      - CPU Launch Time: 4 microseconds (vs. 250 microseconds eager launch)                              |
|      - GPU SM Occupancy: 96% (Zero inter-kernel dispatch gaps)                                          |
+---------------------------------------------------------------------------------------------------------+

Micro-Waiting Window ($W_t$): When the short-prefill queue is non-empty, PLA-Serve waits for a small, configurable window $W_t$ (calibrated between 1.5 ms and 4.0 ms). Because interactive request arrival rates are high in multi-tenant environments, accumulating tokens over 3 ms yields multiple co-batchable short requests.
Power-of-Two Bucketing: Requests are partitioned into discrete sequence length buckets: $B \in {64, 128, 256, 512, 1024}$. Prompts are minimally padded to their nearest bucket ceiling. The minimal padding overhead (\< 8% FLOP waste) is vastly outweighed by the massive throughput acceleration gained from static kernel scheduling.
Pre-Captured CUDA Graph Execution: PLA-Serve pre-compiles and warms up a grid of static CUDA Graphs indexed by (batch_size, bucket_length). When a cluster of requests is assembled, PLA-Serve copies input tokens into pre-allocated memory buffers and invokes cudaGraphLaunch. Kernel launch latency drops from over 200 microseconds down to under 5 microseconds, allowing short prefills to complete in record time.

Step-by-Step Implementation: Building a Prefill-Length-Aware Serving Engine

To understand how PLA-Serve operates in practice, let us implement a functional prototype demonstrating:

Quadratic-aware request cost calculation.
Dual-queue length disaggregation.
Micro-waiting window clustering with bucketed execution.

Step 1: Request Complexity & Dual-Queue Architecture

# pla_serve_queues.py
import time
import math
from dataclasses import dataclass, field
from typing import List, Optional, Tuple
from collections import deque

@dataclass
class PrefillRequest:
    request_id: str
    prompt_tokens: List[int]
    prompt_len: int = field(init=False)
    arrival_time: float = field(default_factory=time.time)
    slo_deadline_ms: float = 200.0  # 200ms TTFT SLO

    def __post_init__(self):
        self.prompt_len = len(self.prompt_tokens)

    @property
    def estimated_compute_cost(self) -> float:
        """
        Calculates quadratic attention complexity:
        Cost = alpha * L + beta * L^2
        """
        alpha = 1.0     # Linear projection weight
        beta = 0.0025   # Quadratic self-attention weight
        return (alpha * self.prompt_len) + (beta * (self.prompt_len ** 2))

class DualQueuePrefillManager:
    """
    Segregates incoming prefill requests into Short and Long queues
    based on calibrated length thresholds.
    """
    def __init__(self, length_threshold: int = 1024):
        self.length_threshold = length_threshold
        self.short_queue = deque()
        self.long_queue = deque()

    def enqueue(self, request: PrefillRequest):
        if request.prompt_len < self.length_threshold:
            self.short_queue.append(request)
        else:
            self.long_queue.append(request)

    def has_pending_requests(self) -> bool:
        return len(self.short_queue) > 0 or len(self.long_queue) > 0

Step 2: Bucketed CUDA Graph Clusterer for Short Prefills

# pla_serve_clusterer.py
import torch
import time
from typing import List, Dict, Tuple
from pla_serve_queues import PrefillRequest

class BucketedGraphClusterer:
    """
    Clusters short-prefill requests into discrete length buckets
    and executes them using pre-captured CUDA Graphs.
    """
    BUCKETS = [64, 128, 256, 512, 1024]
    BATCH_SIZES = [1, 2, 4, 8]

    def __init__(self, waiting_window_ms: float = 3.0):
        self.waiting_window_ms = waiting_window_ms
        self.graphs: Dict[Tuple[int, int], torch.cuda.CUDAGraph] = {}
        self.static_inputs: Dict[Tuple[int, int], torch.Tensor] = {}
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

        # Warmup mock static CUDA graphs if CUDA is available
        self._initialize_graphs()

    def _initialize_graphs(self):
        if not torch.cuda.is_available():
            print("[PLA-Serve] CUDA not detected. Operating in simulated CUDA Graph replay mode.")
            return

        hidden_dim = 2048
        for b_size in self.BATCH_SIZES:
            for b_len in self.BUCKETS:
                static_in = torch.zeros(b_size, b_len, hidden_dim, device=self.device)
                self.static_inputs[(b_size, b_len)] = static_in

                # Capture static forward GEMM graph
                g = torch.cuda.CUDAGraph()
                s = torch.cuda.Stream()
                s.wait_stream(torch.cuda.current_stream())
                with torch.cuda.stream(s):
                    for _ in range(3):  # Warmup
                        _ = torch.matmul(static_in, static_in.transpose(-1, -2))
                    torch.cuda.synchronize()

                    with torch.cuda.graph(g):
                        _ = torch.matmul(static_in, static_in.transpose(-1, -2))
                torch.cuda.current_stream().wait_stream(s)
                self.graphs[(b_size, b_len)] = g

    def cluster_requests(self, candidate_requests: List[PrefillRequest]) -> Dict[int, List[PrefillRequest]]:
        """
        Groups requests into nearest bucket ceilings to minimize padding waste.
        """
        clusters: Dict[int, List[PrefillRequest]] = {b: [] for b in self.BUCKETS}
        for req in candidate_requests:
            assigned_bucket = None
            for b in self.BUCKETS:
                if req.prompt_len <= b:
                    assigned_bucket = b
                    break
            if assigned_bucket:
                clusters[assigned_bucket].append(req)
        return {b: reqs for b, reqs in clusters.items() if reqs}

    def execute_bucket(self, bucket_len: int, batch: List[PrefillRequest]) -> float:
        """
        Executes a bucketed batch using static graph replay or optimized kernel.
        Returns execution duration in milliseconds.
        """
        batch_size = len(batch)
        start_time = time.perf_counter()

        # In production, select closest power-of-two pre-captured CUDA Graph
        graph_key = (batch_size, bucket_len)
        if torch.cuda.is_available() and graph_key in self.graphs:
            self.graphs[graph_key].replay()
            torch.cuda.synchronize()
        else:
            # Simulated high-performance GEMM
            time.sleep(0.0008 * (bucket_len / 128.0) * batch_size)

        return (time.perf_counter() - start_time) * 1000.0

Step 3: Adaptive Spatial-Temporal Prefill Scheduler

# pla_serve_scheduler.py
import time
from typing import List, Dict, Any
from pla_serve_queues import DualQueuePrefillManager, PrefillRequest
from pla_serve_clusterer import BucketedGraphClusterer

class PLAServeScheduler:
    """
    Coordinates dual-queue prefill execution, enforcing zero interference
    between short conversational turns and long RAG/context requests.
    """
    def __init__(self, queue_manager: DualQueuePrefillManager, clusterer: BucketedGraphClusterer):
        self.qm = queue_manager
        self.clusterer = clusterer
        self.completed_requests: List[Dict[str, Any]] = []

    def run_scheduling_cycle(self) -> Dict[str, Any]:
        """
        Executes a single length-aware prefill scheduling iteration.
        Prioritizes short-prefill CUDA Graph batches while interleaving
        long-prefill sequence chunks.
        """
        cycle_start = time.perf_counter()
        executed_short = 0
        executed_long = 0

        # 1. Process Short-Prefill Queue with Micro-Waiting Window
        if len(self.qm.short_queue) > 0:
            # Collect short requests within micro-waiting window
            batch_candidates = []
            while len(self.qm.short_queue) > 0 and len(batch_candidates) < 16:
                batch_candidates.append(self.qm.short_queue.popleft())

            # Cluster by sequence length
            clusters = self.clusterer.cluster_requests(batch_candidates)
            for bucket_len, req_list in clusters.items():
                dur_ms = self.clusterer.execute_bucket(bucket_len, req_list)
                executed_short += len(req_list)
                now = time.time()
                for r in req_list:
                    ttft = (now - r.arrival_time) * 1000.0
                    self.completed_requests.append({
                        "request_id": r.request_id,
                        "type": "SHORT",
                        "prompt_len": r.prompt_len,
                        "ttft_ms": ttft,
                        "slo_met": ttft <= r.slo_deadline_ms
                    })

        # 2. Process Long-Prefill Queue if Short Queue is clear
        if len(self.qm.long_queue) > 0 and len(self.qm.short_queue) == 0:
            long_req = self.qm.long_queue.popleft()

            # Execute long prefill using non-graph high-throughput FlashAttention
            # Simulating execution duration proportional to quadratic complexity
            compute_time_s = 0.005 * (long_req.prompt_len / 1024.0) ** 1.8
            time.sleep(compute_time_s)
            executed_long += 1

            now = time.time()
            ttft = (now - long_req.arrival_time) * 1000.0
            self.completed_requests.append({
                "request_id": long_req.request_id,
                "type": "LONG",
                "prompt_len": long_req.prompt_len,
                "ttft_ms": ttft,
                "slo_met": ttft <= (long_req.slo_deadline_ms * 4)  # Relaxed SLO for large contexts
            })

        cycle_dur_ms = (time.perf_counter() - cycle_start) * 1000.0
        return {
            "cycle_duration_ms": cycle_dur_ms,
            "executed_short": executed_short,
            "executed_long": executed_long,
            "pending_short": len(self.qm.short_queue),
            "pending_long": len(self.qm.long_queue)
        }

Empirical Benchmarks: Evaluating PLA-Serve

The authors of PLA-Serve evaluated the system across extensive testbed environments, benchmarking against vanilla SGLang, vLLM under prefill-decode disaggregation, and the SGLang load-balancing router.

The experiments evaluated popular open-weight foundation models—including Qwen2.5-32B and LLaMA-3-70B—driven by real-world multi-turn conversational traces (ShareGPT, LMSYS-Chat-1M, and synthetic long-document benchmarks).

Evaluation Metric	Vanilla vLLM (PD Disagg)	Vanilla SGLang (PD Disagg)	SGLang Load-Balancing Router	PLA-Serve (MLSys 2026)
Prefill Latency (Short Prompts)	148 ms	118 ms	104 ms	72 ms (> 30% Reduction)
SLO Violation Rate (TTFT)	38.4%	29.2%	23.5%	11.2% (28% Relative Drop)
CUDA Graph Acceleration	Disabled (Eager)	Disabled (Eager)	Disabled (Eager)	Enabled (Bucketed Replay)
Kernel Dispatch Overhead	~220 us per launch	~180 us per launch	~180 us per launch	\< 5 us (CUDA Graph Replay)
Multi-GPU Load Skew	46.2% Variance	38.5% Variance	24.1% Variance	\< 5.8% (Uniform Balance)
Aggregate Cluster Throughput	Baseline (1.0x)	1.12x	1.18x	1.35x (35% Speedup)

Key Empirical Findings

Over 30% Reduction in Prefill Latency: By clustering short prompts into discrete length buckets and executing them via pre-captured CUDA Graphs, PLA-Serve eliminates CPU driver dispatch delays, reducing average short-prompt TTFT from 118 ms to 72 ms.
28% Reduction in SLO Violations: In multi-instance data-parallel deployments, segregating long and short prefills prevents multi-thousand-token prompts from hijacking GPU streaming multiprocessors, preserving strict interactive response guarantees.
Quadratic-Aware Load Balancing: Against the state-of-the-art SGLang load-balancing router, PLA-Serve cuts SLO violations by an additional 12% in multi-GPU configurations by estimating true $\mathcal{O}(L^2)$ attention compute rather than raw token volume.
35% Increase in System Throughput: When serving Qwen2.5-32B under high concurrency and mixed-length traffic distributions, PLA-Serve increases total sustained requests-per-second by 35% without requiring additional GPU hardware.

Integrating PLA-Serve with AIFetchly Infrastructure

In distributed enterprise AI ecosystems, inference serving does not operate in isolation. Real-world foundation model architectures depend on continuous web intelligence, document extraction, and API orchestration:

High-Throughput Web Mining Pipelines: When enterprise pipelines ingest massive multi-page web corpora via automated web scraping, document extraction produces extreme sequence length variance—short navigation headlines arriving alongside dense 40-page financial audits. Deploying PLA-Serve ensures that real-time conversational agents querying the database are never stalled by background document ingestion.
Reliable API Integration & Data Prefetching: In high-concurrency portals powered by API integration and real-time data fetching, API gateways can tag incoming requests with preliminary length hints, allowing PLA-Serve’s router to pre-assign requests to optimal CUDA Graph buckets before network deserialization completes.
Infrastructure Cost Reduction: Achieving 35% higher throughput on existing GPU hardware directly reduces cloud hosting costs. Organizations avoid over-provisioning expensive NVIDIA H100 or A100 clusters simply to handle prompt-length spikes. Review our transparent pricing tiers and explore our comprehensive developer documentation to discover how the AIFetchly platform unites web data harvesting, vector search, and high-performance LLM serving into a cost-effective enterprise stack.

Future Horizons: The Next Generation of Prefill Systems

PLA-Serve establishes a foundation for the future of prompt processing:

Zero-Padding Dynamic CUDA Graphs: Emerging GPU runtime extensions will allow dynamic shape compilation within CUDA Graphs, eliminating the minimal padding overhead of discrete bucketing.
Hardware-Assisted Sequence Partitioning: Future accelerator silicon with native tensor-level context switching will enable hardware-level preemption of long prefill GEMMs, allowing short prompts to interrupt long sequences at microsecond granularity.
Cross-Model Length-Aware Scheduling: As multimodal foundation models incorporate high-resolution video and audio tokens, length heterogeneity will expand from thousands of tokens to millions. Extending PLA-Serve’s dual-queue principles across multimodal encoders will become essential for real-time AI agents.

Key Technical Takeaways

Length Heterogeneity is the True Prefill Bottleneck: Monolithic prefill queues suffer from head-of-line blocking, CUDA Graph disablement, and router metric blindness when processing mixed short and long prompts.
Dual-Queue Separation: Segregating short conversational turns from long RAG/context requests eliminates prompt interference and guarantees sub-100ms Time-to-First-Token.
CUDA Graph Clustering for Prefill: Combining a micro-waiting window ($W_t \approx 3\text{ ms}$) with discrete length bucketing unlocks static CUDA Graph replay for prompts, slashing kernel dispatch overhead from 200 microseconds to under 5 microseconds.
Quadratic Complexity Routing: Factoring in the $\mathcal{O}(L^2)$ computational scaling of self-attention prevents cluster load skew, boosting aggregate system throughput by 35% and cutting SLO violations by 28%.