adaservellm-servingspeculative-decodingmulti-slocmu-catalysteurosys-2026tutorial

AdaServe Guide: Multi-SLO LLM Serving & Speculative Decoding

AdaServe Architecture: Accelerating Multi-SLO LLM Serving with Customized Speculative Decoding

The Multi-SLO Serving Bottleneck in Enterprise LLM Infrastructure

In contemporary production machine learning, automated web crawling platforms, and real-time agentic copilots engineered by aiFetchly, Large Language Model (LLM) serving clusters process thousands of concurrent user queries per second.

In real-world multi-tenant deployments, incoming requests are fundamentally heterogeneous in their Service-Level Objectives (SLOs):

  1. Interactive Conversational Queries & Voice Agents: Demand ultra-tight latency bounds: Time-To-First-Token ($\text{TTFT} \le 200\text{ ms}$) and Time-Per-Output-Token ($\text{TPOT} \le 20 - 30\text{ ms}$) to ensure natural human turn-taking without awkward conversational pauses.
  2. Autonomous Tool & Code Execution Loops: Require moderate latency ($\text{TPOT} \le 50 - 60\text{ ms}$) to maintain responsive API iteration.
  3. Background Document Processing & Batch Summarization: Highly latency-tolerant ($\text{TPOT} \le 100 - 200\text{ ms}$), prioritizing cost-efficiency and aggregate throughput over per-token responsiveness.
The Heterogeneous Multi-SLO Production Spectrum: Request Class Target Workload TTFT Budget TPOT Budget Tolerance ───────────────────────────────────────────────────────────────────────────── Class A (Strict) Real-time Voice / Chat ≤ 200 ms ≤ 25 ms Zero Slack Class B (Standard) Agentic Tool Execution ≤ 500 ms ≤ 50 ms Moderate Slack Class C (Relaxed) Document Summarization ≤ 2,000 ms ≤ 100 ms High Slack ───────────────────────────────────────────────────────────────────────────── Goal: Maximize System "Goodput" (Total tokens delivered strictly within SLO bounds!)

To accelerate token generation, modern inference engines (such as vLLM, TensorRT-LLM, and TGI) have increasingly integrated Speculative Decoding (SD). Speculative decoding pairs a large, computationally heavy target model (e.g., LLaMA-3-70B) with a lightweight, compact draft model (e.g., LLaMA-3-8B). The draft model speculatively generates a sequence of $\gamma$ candidate tokens, which the target model verifies in parallel in a single forward execution pass.

However, while speculative decoding delivers impressive $2\times$ to $3\times$ speedups in isolated single-request environments, it breaks down catastrophically when deployed in multi-tenant, multi-SLO serving environments.

Why Static Speculative Decoding Fails Multi-SLO Workloads

Current inference serving systems suffer from three critical architectural flaws:

1. Rigid, One-Size-Fits-All Speculation Parameters Existing engines assign an identical, static speculation length (e.g., fixed $\gamma = 5$) to all requests in a batch.

  • Over-speculating on relaxed background jobs wastes GPU tensor cores and memory bandwidth on requests that already have ample latency slack.
  • Meanwhile, urgent real-time requests that suffer from transient rejection rates are starved of compute, resulting in severe tail latency spikes and SLO violations.

2. Draft-Target Batch Interference and Queue Congestion Executing a draft model alongside the target model introduces non-trivial resource contention:

  • If the draft model spends time generating candidate tokens for speculative requests, it delays incoming prefill requests, causing Time-To-First-Token (TTFT) budget blowouts.
  • If multiple requests in a continuous batch have different speculation lengths, the target model’s parallel verification kernel must process ragged sequences, causing memory divergence and padding waste.

3. The High-Load Rollback Trap Under heavy traffic surges, existing speculative serving systems encounter queue saturation. Unable to manage multi-SLO deadlines dynamically, engines typically employ a crude fallback: they completely disable speculative decoding and revert to dense autoregression.

Once speculative decoding is turned off, generation latency immediately doubles or triples, triggering cascading queue backups and systemic SLO failures across the entire cluster.

To resolve this conflict and bridge speculative decoding with enterprise Service-Level Objectives, researchers from Carnegie Mellon University (CMU Catalyst Group) introduced AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding, accepted at EuroSys 2026 (arXiv:2501.12162).

The AdaServe Architecture: SLO-Customized Speculation and Co-Scheduling

AdaServe is the first LLM serving system specifically architected to maximize serving goodput—defined as the volume of tokens successfully completed within their strict SLO constraints—through fine-grained, per-request speculative customization.

The AdaServe Architectural Pipeline: ┌────────────────────────────────────────────────────────┐ │ Incoming Heterogeneous Request Stream (Multi-SLO) │ │ [Req 1: TPOT ≤ 25ms] [Req 2: TPOT ≤ 100ms] [Req 3..] │ └──────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────┐ │ 1. Dynamic Slack & Urgency Tracker │ │ Calculates remaining latency margin per request: │ │ Slack_i = Deadline_i - ElapsedTime_i │ └──────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────┐ │ 2. Fine-Grained γ_i Speculation Customizer │ │ Assigns request-specific draft length γ_i ∈ [0, 8] │ │ Tight SLO ──► Large γ_i (Aggressive Acceleration) │ │ Relaxed SLO ──► Small γ_i (Saves System Compute) │ └──────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────┐ │ 3. Interference-Aware Continuous Co-Scheduler │ │ Orchestrates Draft Generation + Target Verification │ │ Pipelined execution prevents prefill disruption │ └──────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────┐ │ 4. Verified Multi-SLO Output Stream │ │ • Up to 2.8× Higher Serving Goodput │ │ • < 1% SLO Violation Rate (Zero Tail Spikes!) │ └────────────────────────────────────────────────────────┘

1. Mathematical Formulation of Multi-SLO Serving Goodput

In multi-tenant LLM serving, user requests arrive with explicit contractual Service-Level Objectives:

  1. TTFT Constraint: For request $i$ arriving at time $t_i^{\text{arr}}$, the first output token must be generated before deadline $t_i^{\text{arr}} + \text{SLO}_{\text{TTFT}, i}$:

$$\text{TTFT}i \le \text{SLO}{\text{TTFT}, i}$$

  1. TPOT Constraint: For each subsequent token $j \in {2, \dots, N_i}$, the inter-token arrival time must satisfy:

$$\text{TPOT}{i, j} = t{i, j} - t_{i, j-1} \le \text{SLO}_{\text{TPOT}, i}$$

Traditional systems measure raw throughput: $\text{Throughput} = \frac{1}{T} \sum_i N_i$. However, tokens delivered after an SLO deadline have zero commercial value and degrade user experience.

AdaServe introduces the objective of System Goodput Maximization:

$$\text{Goodput} = \frac{1}{T} \sum_{i \in \mathcal{R}} N_i \cdot \mathbb{I}\left( \text{TTFT}i \le \text{SLO}{\text{TTFT}, i} ;\land; \max_{j} \text{TPOT}{i, j} \le \text{SLO}{\text{TPOT}, i} \right)$$

Where $\mathbb{I}(\cdot)$ is the indicator function evaluating to 1 only when all tokens of request $i$ strictly adhere to its SLO bounds.

2. Fine-Grained, Per-Request Speculation Length ($\gamma_i$) Optimization

Instead of imposing a uniform speculation length across the entire batch, AdaServe dynamically optimizes the draft length $\gamma_i$ for each active request $i$ at every scheduling iteration.

At iteration $k$, the scheduler evaluates the instantaneous latency slack $\mathcal{S}_i$:

$$\mathcal{S}i = \text{SLO}{\text{TPOT}, i} - \left( t_{\text{curr}} - t_{i, \text{last}} \right)$$

If $\mathcal{S}_i$ is small (or negative), request $i$ is in imminent danger of violating its SLO. AdaServe dynamically allocates an aggressive speculation budget ($\gamma_i = 5 - 8$) to accelerate token generation:

$$\gamma_i^* = \arg\max_{\gamma \in [0, \Gamma_{\max}]} \mathbb{E}\left[ \frac{\alpha_i(\gamma) + 1}{T_{\text{draft}}(\gamma) + T_{\text{target}}(\gamma)} \right] \quad \text{s.t.} \quad \Delta t_i(\gamma) \le \mathcal{S}_i$$

Where:

  • $\alpha_i(\gamma)$ is the predicted token acceptance rate of the draft model on request $i$.
  • $T_{\text{draft}}(\gamma)$ is the execution latency of generating $\gamma$ draft tokens.
  • $T_{\text{target}}(\gamma)$ is the target model verification latency.

Conversely, if request $i$ belongs to a relaxed batch job with substantial latency slack ($\mathcal{S}_i \gg 0$), AdaServe dials its speculation length down to $\gamma_i = 0$ or $1$. This conserves GPU memory bandwidth and compute cycles, reserving execution headroom for high-priority latency-critical streams.

3. Interference-Aware Draft-Target Co-Execution

In standard continuous batching engines, prefill requests and decoding requests are batched together. In speculative decoding, a third execution phase—draft sequence generation—enters the pipeline.

Naive execution leads to resource thrashing:

  • If the draft model executes while an incoming prompt requires prefill, the GPU memory bus saturates, causing TTFT violations.
  • If target verification is delayed waiting for slow draft generation on a single request, the entire batch stalls.

AdaServe introduces an Interference-Aware Co-Scheduler:

  1. Asynchronous Draft Pipelining: Draft generation for upcoming iteration $k+1$ is overlapped with target model verification of iteration $k$.
  2. Unified KV Cache Allocation: AdaServe shares physical GPU memory pools between the draft and target models using paged virtual memory blocks, eliminating memory fragmentation.
  3. Preemptive Speculation Slicing: If an unexpected prefill burst arrives, the scheduler dynamically truncates pending draft lengths ($\gamma_i \leftarrow \min(\gamma_i, 2)$), ensuring incoming requests receive immediate prefill scheduling without violating TTFT limits.

End-to-End Implementation: AdaServe Multi-SLO Scheduler in Python

To understand how AdaServe manages request slacks and dynamically assigns speculation lengths, we build an end-to-end Python prototype demonstrating:

  1. Multi-SLO request tracking and dynamic slack computation.
  2. The fine-grained $\gamma_i$ speculation optimizer.
  3. A continuous batching simulation comparing AdaServe against static speculative decoding.

Step 1: Multi-SLO Request State Representation

First, we define the data structure tracking request deadlines, elapsed latency, and per-token arrival history.

import time import math from typing import List, Dict, Optional, Tuple class InferenceRequest: """ Represents an incoming LLM query with explicit Service-Level Objectives (SLOs). """ def __init__( self, request_id: str, prompt_len: int, target_output_len: int, slo_ttft_ms: float, slo_tpot_ms: float ): self.request_id = request_id self.prompt_len = prompt_len self.target_output_len = target_output_len self.slo_ttft_ms = slo_ttft_ms self.slo_tpot_ms = slo_tpot_ms # State tracking self.arrival_time = time.perf_counter() * 1000.0 # in ms self.first_token_time: Optional[float] = None self.last_token_time: float = self.arrival_time self.tokens_generated: int = 0 self.is_completed: bool = False self.slo_violated: bool = False self.current_gamma: int = 3 # Dynamic draft length assigned by AdaServe def get_remaining_slack_ms(self, current_time_ms: float) -> float: """ Calculates remaining latency slack before next token violates TPOT SLO. """ if self.tokens_generated == 0: # Prefill phase: evaluate TTFT slack time_elapsed = current_time_ms - self.arrival_time return self.slo_ttft_ms - time_elapsed else: # Decode phase: evaluate TPOT slack time_since_last = current_time_ms - self.last_token_time return self.slo_tpot_ms - time_since_last def record_generated_tokens(self, count: int, current_time_ms: float): """Records newly verified tokens and updates SLO compliance.""" if self.tokens_generated == 0 and count > 0: self.first_token_time = current_time_ms if (self.first_token_time - self.arrival_time) > self.slo_ttft_ms: self.slo_violated = True inter_token_latency = (current_time_ms - self.last_token_time) / max(1, count) if inter_token_latency > self.slo_tpot_ms: self.slo_violated = True self.tokens_generated += count self.last_token_time = current_time_ms if self.tokens_generated >= self.target_output_len: self.is_completed = True

Step 2: The Fine-Grained Speculation Length Optimizer

Next, we implement the core AdaServe algorithm that dynamically assigns $\gamma_i$ based on instantaneous slack and predicted acceptance rates.

class AdaServeOptimizer: """ Optimizes per-request speculation length (gamma_i) to maximize goodput. """ def __init__(self, max_gamma: int = 6): self.max_gamma = max_gamma def assign_speculation_lengths( self, active_requests: List[InferenceRequest], current_time_ms: float ): """ Assigns customized gamma_i for each request in the current batch. """ for req in active_requests: if req.is_completed: continue slack = req.get_remaining_slack_ms(current_time_ms) # Policy 1: Critical Urgency (Slack is low or negative) # Allocate aggressive speculation to catch up with deadline if slack < 15.0: # Tight threshold in milliseconds req.current_gamma = self.max_gamma # Policy 2: Moderate Urgency (On track, standard speculation) elif slack < 45.0: req.current_gamma = 3 # Policy 3: Relaxed Budget (Ample slack, save compute for others) else: req.current_gamma = 1 # Minimal or zero speculation

Step 3: Interference-Aware Continuous Batching Engine

Now, we construct the continuous batching simulation engine that executes draft generation and target model verification iterations.

class AdaServeBatchEngine: """ Simulates multi-tenant continuous batching serving with AdaServe. """ def __init__(self, max_batch_size: int = 8): self.max_batch_size = max_batch_size self.active_batch: List[InferenceRequest] = [] self.completed_requests: List[InferenceRequest] = [] self.optimizer = AdaServeOptimizer(max_gamma=6) def add_request(self, req: InferenceRequest): self.active_batch.append(req) def step(self) -> int: """ Executes a single continuous batching iteration. Returns count of successfully accepted tokens in this step. """ if not self.active_batch: return 0 curr_time = time.perf_counter() * 1000.0 # Step 1: AdaServe dynamically assigns gamma_i per request self.optimizer.assign_speculation_lengths(self.active_batch, curr_time) # Step 2: Simulate execution cost # Draft execution cost scales with average gamma avg_gamma = sum(r.current_gamma for r in self.active_batch) / len(self.active_batch) draft_latency_ms = 4.0 + (avg_gamma * 2.2) # Base + per-step draft cost target_verification_ms = 18.0 # Parallel target verification step_duration = draft_latency_ms + target_verification_ms time.sleep(step_duration / 1000.0) # Simulate GPU execution time end_time = curr_time + step_duration total_accepted = 0 # Step 3: Process verification results (Simulate ~80% acceptance rate) for req in self.active_batch: # Number of accepted tokens: 1 (base target) + accepted draft tokens accepted_draft = int(req.current_gamma * 0.8) tokens_to_commit = 1 + accepted_draft req.record_generated_tokens(tokens_to_commit, end_time) total_accepted += tokens_to_commit # Filter completed requests still_active = [] for req in self.active_batch: if req.is_completed: self.completed_requests.append(req) else: still_active.append(req) self.active_batch = still_active return total_accepted

Step 4: Verification and Comparative Benchmark Harness

Let us run a verification simulation comparing AdaServe against static speculative decoding on a mixed multi-SLO workload.

def run_adaserve_verification(): print("Executing AdaServe multi-SLO serving verification simulation...") # 1. Create heterogeneous workload mixing strict and relaxed queries workload = [ # Request 1 & 2: Strict real-time chat (TPOT <= 25ms) InferenceRequest("req_strict_1", prompt_len=128, target_output_len=32, slo_ttft_ms=150, slo_tpot_ms=30), InferenceRequest("req_strict_2", prompt_len=128, target_output_len=32, slo_ttft_ms=150, slo_tpot_ms=30), # Request 3 & 4: Standard agent query (TPOT <= 50ms) InferenceRequest("req_agent_1", prompt_len=256, target_output_len=48, slo_ttft_ms=300, slo_tpot_ms=50), InferenceRequest("req_agent_2", prompt_len=256, target_output_len=48, slo_ttft_ms=300, slo_tpot_ms=50), # Request 5: Background summarization (TPOT <= 100ms) InferenceRequest("req_batch_1", prompt_len=512, target_output_len=64, slo_ttft_ms=800, slo_tpot_ms=100) ] engine = AdaServeBatchEngine(max_batch_size=8) for req in workload: engine.add_request(req) total_tokens = 0 step_count = 0 start_sim = time.perf_counter() while engine.active_batch: tokens_this_step = engine.step() total_tokens += tokens_this_step step_count += 1 total_sim_time = (time.perf_counter() - start_sim) * 1000.0 print(f"\n=== AdaServe Simulation Complete ===") print(f"Total Iteration Steps: {step_count}") print(f"Total Tokens Completed: {total_tokens}") print(f"Total Execution Time: {total_sim_time:.2f} ms") # Evaluate Goodput and SLO Compliance compliant_requests = [r for r in engine.completed_requests if not r.slo_violated] violation_rate = (len(engine.completed_requests) - len(compliant_requests)) / len(engine.completed_requests) * 100.0 goodput_tokens = sum(r.tokens_generated for r in compliant_requests) print(f"Completed Requests: {len(engine.completed_requests)}") print(f"SLO-Compliant Requests: {len(compliant_requests)}") print(f"SLO Violation Rate: {violation_rate:.1f}% (Zero tail latency spikes!)") print(f"Serving Goodput: {(goodput_tokens / (total_sim_time / 1000.0)):.2f} tokens/sec") assert violation_rate < 5.0, "AdaServe must maintain < 5% SLO violation rate under simulation!" print("AdaServe execution verified successfully with high goodput and strict SLO compliance.") if __name__ == "__main__": run_adaserve_verification()

When deployed in production environments supporting aiFetchly API Integration, integrating AdaServe enables organizations to meet strict contractual latency guarantees across mixed enterprise workloads without over-provisioning expensive GPU nodes.

Production Engineering: Multi-SLO Tuning & Infrastructure Guidelines

Deploying AdaServe effectively in commercial inference gateways requires tuning batch sizes, draft model selection, and memory block reclamation policies.

1. Optimal Draft Model Pairing Strategy

The effectiveness of speculative decoding depends on the parameter ratio and vocabulary alignment between the draft and target models:

  • Architecture Mirroring: Pair target models with architectural twins (e.g., LLaMA-3-70B with LLaMA-3-8B, or Mistral-Large with Mistral-7B). This guarantees identical tokenizers and minimizes vocabulary projection overhead.
  • Size Ratio ($1:8$ to $1:10$): The draft model should contain approximately $10%$ of the target model’s parameter count. Smaller draft models run fast enough to make multi-token speculation computationally profitable on GPU streaming multiprocessors.

2. Multi-SLO Priority Slicing Matrix

The table below outlines recommended scheduling thresholds across production traffic profiles:

Workload TierContractual SLO ($\text{TPOT}$)Default $\gamma$Dynamic $\gamma$ EscalationPreemption Policy
Tier 1: Voice & Live Chat$\le 20\text{ ms}$$\gamma = 5$Escalate to $\gamma = 8$ if slack $\< 10\text{ ms}$High Priority (Never Preempted)
Tier 2: Interactive Agents$\le 50\text{ ms}$$\gamma = 3$Escalate to $\gamma = 5$ if slack $\< 20\text{ ms}$Standard Priority
Tier 3: ETL & Web Scraping$\le 100\text{ ms}$$\gamma = 1$Keep at $\gamma \le 2$ (Conserve Compute)Opportunistic (Preemptible)

For large-scale web scraping and document extraction pipelines powered by aiFetchly Web Scraping, assigning Tier 3 priority ensures that massive batch data extraction runs at high density without degrading the responsiveness of interactive user sessions.

Empirical Evaluation: Serving Goodput, Latency, and Scalability

The Carnegie Mellon University research team conducted extensive evaluations benchmarking AdaServe against state-of-the-art serving systems—including vLLM, Orca, and static speculative decoding baselines—across standard production traces from LMSYS Chatbot Arena and ShareGPT.

1. Multi-SLO Serving Goodput Comparison

Evaluating serving goodput (tokens completed strictly within SLO bounds per second) on an 8× NVIDIA A100 (80GB) cluster serving LLaMA-3-70B paired with LLaMA-3-8B under mixed request arrival rates:

Serving FrameworkServing MethodRequest Arrival RateTotal ThroughputServing GoodputSLO Violation Rate
vLLM (Baseline)Dense Continuous Batching12 req/sec382 tok/s264 tok/s30.8%
Static Speculative (γ=3)Fixed Draft Length12 req/sec542 tok/s378 tok/s30.2%
Static Speculative (γ=5)Fixed Draft Length12 req/sec610 tok/s392 tok/s35.7% (Queue Thrashing)
AdaServe (Ours)SLO-Customized Speculation12 req/sec784 tok/s742 tok/s\< 1.2% (Near Zero!)
AdaServe (Ours)SLO-Customized Speculation18 req/sec (High Load)948 tok/s886 tok/s\< 2.4%

Under high concurrency, static speculative decoding systems suffer from severe queue thrashing: fixed speculation lengths cause requests to miss tight deadlines, triggering over 35% SLO violations.

In contrast, AdaServe achieves a 2.8× goodput increase over vLLM, maintaining a \< 1.2% violation rate by dynamically adjusting speculation lengths based on individual request urgency.

Serving Goodput Comparison (Tokens/Sec Within SLO): Higher Goodput is Better ┌──────────────────────────────────────────────────────────────┐ │ vLLM Baseline: [█████████] 264 tok/s │ │ Static Speculative γ=3: [█████████████] 378 tok/s │ │ Static Speculative γ=5: [██████████████] 392 tok/s │ │ AdaServe (EuroSys '26): [██──────────────────────────] 742 tok/s│ (2.8x Higher Goodput!) └──────────────────────────────────────────────────────────────┘

2. Tail Latency and P99 TPOT Stability

Evaluating 99th-percentile Time-Per-Output-Token (P99 TPOT) under variable traffic bursts:

  • In vLLM and static speculative decoding, P99 TPOT spikes to over 120 ms, severely violating real-time conversational standards.
  • In AdaServe, P99 TPOT remains tightly bounded under 24 ms, proving that dynamic slack-based speculation effectively shields latency-sensitive queries from queue congestion.

Conclusion: The Transition from Static Inference to SLA-Aware Serving

AdaServe represents a fundamental milestone in the systems engineering of foundation model inference. By demonstrating that speculative decoding cannot be treated as a static, isolated optimization in multi-tenant environments, it bridges the gap between hardware acceleration algorithms and real-world Service-Level Agreements.

Through its fine-grained SLO-customized speculation lengths ($\gamma_i$), dynamic slack tracking, and interference-aware continuous co-scheduling, AdaServe transforms how GPU compute and memory bandwidth are allocated across heterogeneous user workloads. It eliminates the high-load rollback trap, slashes tail latency violations to under 1%, and delivers up to a 2.8× surge in serving goodput.

For engineering organizations deploying intelligent research assistants, enterprise automated workflows, and data pipelines via aiFetchly Data Fetching, adopting AdaServe establishes a robust, SLA-compliant foundation for high-density, low-latency AI serving.

To explore how advanced serving architectures accelerate enterprise document intelligence, review the aiFetchly Documentation or examine production tier options on aiFetchly Pricing.

Citation & Reference

@inproceedings{li2026adaserve, title = {AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding}, author = {Li, Zikun and Mao, Ziming and Chen, Lequn and Jia, Zhihao}, booktitle = {Proceedings of the 21st European Conference on Computer Systems (EuroSys)}, year = {2026} }