adagenllm-servingcluster-schedulingeurosys-2026compute-layoutsslo-optimizationtutorial

AdaGen Guide: Workload-Adaptive LLM Cluster Scheduler

AdaGen Architecture: Workload-Adaptive Cluster Scheduler for Latency-Optimal LLM Serving

The Memory-Centric Blind Spot in Distributed LLM Cluster Scheduling

In hyperscale enterprise automation, agentic coding loops, and automated document ingestion systems powered by aiFetchly, distributed clusters of Large Language Model (LLM) serving instances process tens of thousands of heterogeneous requests every minute. Modern multi-instance deployments typically front multiple independent engine replicas (such as vLLM, TensorRT-LLM, or SGLang instances deployed across dozens of multi-GPU nodes) with an upstream cluster load balancer or scheduler.

Historically, cluster schedulers for distributed inference—exemplified by traditional round-robin routers, least-connection schedulers, and memory-aware balancers—have treated GPU High-Bandwidth Memory (HBM) capacity as the single primary constraint. These memory-centric schedulers track the number of allocated KV cache blocks across worker nodes and route incoming requests to whichever instance possesses the lowest memory utilization.

While memory-aware routing prevents catastrophic Out-Of-Memory (OOM) crashes, modern production workloads reveal a fundamental operational breakdown: memory balancing does not yield latency optimality.

In real-world multi-tenant deployments, requests exhibit extreme diversity across two orthogonal dimensions:

  1. Prefill Length ($L_{\text{prefill}}$): Ranging from concise prompts ($<64$ tokens in chatbots or autocomplete) to massive context payloads ($>32,768$ tokens in enterprise RAG or codebase audits).
  2. Decode Length ($L_{\text{decode}}$): Ranging from single-token classification classifications to deep multi-thousand-token reasoning chains.

When incoming requests with wildly disparate prefill and decode characteristics are routed solely based on available VRAM, they generate chaotic, uncoordinated token arrangements inside each serving instance’s continuous batching engine. This arrangement—formally defined as the compute layout—is the primary hidden determinant of inference latency, execution stalls, and severe Service-Level Objective (SLO) violations.

The Cluster Scheduler Disconnect: ┌────────────────────────────────────────────────────────────────────────┐ │ Incoming Production Traffic Stream │ │ • Request 1: Prefill = 16 tokens, Decode = 1,024 tokens (Chatbot) │ │ • Request 2: Prefill = 8,192 tokens, Decode = 32 tokens (RAG Doc) │ │ • Request 3: Prefill = 2,048 tokens, Decode = 512 tokens (Code Gen) │ └───────────────────────────────────┬────────────────────────────────────┘ │ ┌────────────────────┴────────────────────┐ ▼ ▼ Naive Memory Load Balancer AdaGen Layout-Aware Scheduler ❌ Routes solely on free VRAM blocks ✅ Routes based on 2D Token Profiles ❌ Combines 8k prefill into decode batch ✅ Isolates & aligns compute layouts ❌ Stalls ongoing decoders for 450ms ✅ Simulates batch step delays (<1ms) ❌ Triggers 65% P99 SLO Violations ✅ Delivers 3.6x Higher SLO Attainment

Why Compute Layouts Dictate Multi-Instance Serving Latency

To understand why compute layouts dictate latency, consider how an underlying LLM engine (e.g., vLLM with continuous batching and chunked prefills) executes an iteration batch.

In any given iteration $t$, the GPU Tensor Cores process a combined tensor consisting of:

  • Prefill tokens ($N_{\text{prefill}}$): Highly compute-intensive matrix multiplications (GEMM) where arithmetic intensity is high, fully saturating GPU compute units.
  • Decode tokens ($N_{\text{decode}}$): Memory-bandwidth-bound vector-matrix multiplications (GEMV) where arithmetic intensity is low, stalling on memory reads from the KV cache.

When a scheduler carelessly assigns a heavy prefill request to an instance already executing a batch of latency-sensitive decode requests, two severe bottlenecks emerge:

1. The Iteration-Delay Bubble (Head-of-Line Blocking)
Even with chunked prefill (e.g., chunk size $C = 512$ tokens), running a 512-token prefill chunk alongside 32 decode tokens increases the single-iteration execution time from $15\text{ ms}$ to over $65\text{ ms}$. For an ongoing conversational request with an interactive Time-Per-Output-Token (TPOT) SLO of $25\text{ ms}$, this single iteration instantly violates the contract.

2. Dynamic Resource Under-Utilization and Imbalance
When instances receive random mixtures of requests, some instances experience heavy compute saturation while starving for memory, while neighboring instances exhaust their KV cache while their compute cores sit idle.

To overcome these structural limitations, researchers from the University of Virginia, HPE Labs, and UC Riverside introduced AdaGen: Workload-Adaptive Cluster Scheduler for Latency-Optimal LLM Inference Serving (EuroSys 2026). AdaGen shifts the scheduling paradigm from passive memory tracking to active compute layout optimization, orchestrating tokens across cluster instances to maximize end-to-end SLO attainment.

Architectural Deep Dive: How AdaGen Optimizes Compute Layouts

AdaGen replaces monolithic routing heuristics with a modular, multistep scheduling pipeline that incrementally refines request assignments. Crucially, each step builds upon the anticipated compute layout generated by prior decisions.

┌─────────────────────────────────────────────────────────────────────────────┐ │ AdaGen Architectural Pipeline │ │ │ │ [ Incoming Request Stream ] │ │ │ │ │ ▼ │ │ ┌─────────────────────────┐ │ │ │ Step 1: 2D Classification│ ──► Categorizes into Prefill/Decode Bins │ │ └────────────┬────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────┐ │ │ │ Step 2: Layout Balancer │ ──► Balances Compute Density Across Instances │ │ └────────────┬────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────┐ │ │ │ Step 3: Selective Dist. │ ──► Pipelined / Disaggregated Routing Decisions│ │ └────────────┬────────────┘ │ │ │ │ │ ├──► Evaluated via [ Fast Simulation-Based Latency Estimator ]│ │ ▼ │ │ [ Optimized Instance Execution ] (Instance 1, Instance 2, ... Instance N) │ └─────────────────────────────────────────────────────────────────────────────┘

The AdaGen scheduling engine consists of four foundational innovations:

1. Two-Dimensional (2D) Workload Classification

Rather than treating all requests as homogeneous tokens, AdaGen projects every arriving request $r_i$ into a two-dimensional operational coordinate space:

$$\mathbf{p}i = (L{\text{prefill}}^{(i)}, \hat{L}_{\text{decode}}^{(i)})$$

Where $L_{\text{prefill}}^{(i)}$ is the exact prompt length (known at arrival), and $\hat{L}_{\text{decode}}^{(i)}$ is the estimated generation length, predicted using lightweight input heuristics or sequence length classifiers.

AdaGen discretizes this coordinate space into distinct workload classes:

  • Short-Prefill / Long-Decode (SP-LD): Chatbots, reasoning agents, multi-step math problem solving. Compute profile: extended memory-bound decode iterations with minimal prefill overhead.
  • Long-Prefill / Short-Decode (LP-SD): RAG document search, semantic code indexing, summarization via automated data fetching. Compute profile: compute-heavy prefill bursts followed by brief generation.
  • Long-Prefill / Long-Decode (LP-LD): Full repository code generation, comprehensive report drafting. Compute profile: heavy compute bursts followed by protracted VRAM footprint.
  • Short-Prefill / Short-Decode (SP-SD): Entity extraction, intent classification, routing queries.

By classifying requests upfront, AdaGen prevents destructive co-location of diametrically opposed workload classes on the same physical instance.

2. Compute Layout-Aware Load Balancing

The core of AdaGen is its layout-aware load balancer. For each worker instance $k \in {1, \dots, K}$, AdaGen maintains a virtual timeline representation of the instance’s compute layout matrix $\mathbf{M}_k \in \mathbb{R}^{T \times 2}$, where each row $t$ tracks the planned prefill tokens and active decode tokens for iteration $t$:

$$\mathbf{M}k(t) = \left[ N{\text{prefill}}^{(k)}(t), ; N_{\text{decode}}^{(k)}(t) \right]$$

When a new batch of requests arrives, the load balancer evaluates candidate instance allocations by calculating the projected iteration duration:

$$\tau\left(N_{\text{prefill}}, N_{\text{decode}}\right) = \alpha \cdot N_{\text{prefill}} + \beta \cdot N_{\text{decode}} + \gamma \cdot \mathbb{I}\left(N_{\text{prefill}} > 0\right) + \delta_0$$

Where $\alpha, \beta, \gamma, \delta_0$ are hardware-calibrated profiling coefficients capturing GPU GEMM vs GEMV scaling. The balancer assigns requests to minimize cumulative variance in iteration latency across the cluster, preventing iteration-delay spikes that breach Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT) thresholds.

3. Selective Distributed Execution

Traditional clusters either enforce strict instance isolation (every instance handles both prefill and decode independently) or full disaggregation (all prefills occur on dedicated prefill pools, transferring KV caches over RDMA to decode pools).

Full disaggregation introduces massive network bandwidth overhead for moderate prompts, while strict isolation causes prefill interference.

AdaGen introduces Selective Distributed Execution:

  • For standard requests, requests remain entirely within a single instance to preserve zero network overhead.
  • For skewed, ultra-long prefill requests that would otherwise cause a massive execution bubble on a decode-heavy instance, AdaGen selectively decouples execution: offloading the prefill computation to an underutilized instance, then streaming the resulting KV cache to the target instance for low-latency decoding.

4. Fast Simulation-Based Latency Estimator

A major hurdle in layout-based scheduling is computational overhead: solving combinatorial scheduling optimizations over real-time inference streams must take less than 1 millisecond to avoid adding scheduling latency.

AdaGen solves this by integrating a simulation-based latency estimator. Rather than running expensive iterative solvers or executing probe forward passes, AdaGen simulates the continuous batching step iterations using lightweight queue state vector updates in C++/CUDA. This simulation predicts P90 and P99 SLO compliance for candidate layouts in sub-millisecond execution times, allowing the scheduler to evaluate dozens of placement strategies before committing.

Architectural Comparison: AdaGen vs Traditional Schedulers

The table below contrasts AdaGen against existing state-of-the-art serving architectures:

Architectural DimensionNaive Round-Robin / Least-ConnMemory-Aware (vLLM / Orca)Disaggregated Prefill-Decode (DistServe)AdaGen (EuroSys 2026)
Primary MetricRequest CountAvailable KV Cache BlocksStatic Role SeparationDynamic Compute Layout
Workload AwarenessNoneMemory Footprint OnlyStatic Prefill vs Decode2D Token Length Classification
SLO AttainmentLow (<45% under burst)Moderate (~60–70%)High (~80–85%)Near-Optimal (95–99%)
Interference MitigationNonePartial (Chunked Prefill)Physical SeparationLayout Alignment + Selective Split
Hardware OverheadZeroMinimalHigh (Requires dedicated RDMA)Zero (Software-Defined Scheduling)
Cost Efficiency1.0x Baseline1.1x1.4x2.0x Baseline
Scheduling Overhead$<0.05\text{ ms}$$<0.1\text{ ms}$$\approx 0.5\text{ ms}$$<0.8\text{ ms}$ (via Fast Simulator)

Mathematical Formulation of Compute Layout Optimization

To understand how AdaGen guarantees latency optimality, let us formalize the scheduling objective function.

The Multi-Instance Latency Optimization Problem

Let $\mathcal{R} = {r_1, r_2, \dots, r_M}$ denote a set of incoming inference requests arriving at scheduling window $W$. Each request $r_i$ possesses an arrival timestamp $a_i$, an input length $L_{\text{prefill}}^{(i)}$, an expected output length $L_{\text{decode}}^{(i)}$, a Time-To-First-Token SLO threshold $\text{SLO}{\text{TTFT}}^{(i)}$, and a Time-Per-Output-Token threshold $\text{SLO}{\text{TPOT}}^{(i)}$.

Let $\mathcal{I} = {1, 2, \dots, K}$ denote the set of $K$ serving instances. The scheduling assignment is defined by binary decision variables:

$$x_{i,k} \in {0, 1} \quad \forall r_i \in \mathcal{R}, ; k \in \mathcal{I}$$

Subject to the single-assignment constraint:

$$\sum_{k=1}^K x_{i,k} = 1 \quad \forall r_i \in \mathcal{R}$$

For each instance $k$, the arrival of assigned requests determines the scheduled queue state. The projected completion time for the prefill phase of request $r_i$ on instance $k$ is given by:

$$T_{\text{TTFT}}^{(i)}(k) = q_k + \sum_{j \in \mathcal{Q}k \cup {i}} \tau{\text{prefill}}\left(L_{\text{prefill}}^{(j)}\right) - a_i$$

Where $q_k$ represents the queue wait time and $\mathcal{Q}_k$ denotes the set of requests awaiting execution on instance $k$.

Similarly, the per-token decoding latency for request $r_i$ at decoding step $s \in [1, L_{\text{decode}}^{(i)}]$ is determined by the total batch size and active token count in iteration $t(s)$:

$$\text{TPOT}^{(i)}(k, s) = \tau_{\text{iter}}\left(N_{\text{prefill}}^{(k)}(t(s)), ; N_{\text{decode}}^{(k)}(t(s))\right)$$

AdaGen defines the Global SLO Attainment Utility Function $U(\mathbf{X})$ as:

$$U(\mathbf{X}) = \sum_{i=1}^M \left[ w_{\text{TTFT}} \cdot \mathbb{I}\left(T_{\text{TTFT}}^{(i)} \le \text{SLO}{\text{TTFT}}^{(i)}\right) + w{\text{TPOT}} \cdot \prod_{s=1}^{L_{\text{decode}}^{(i)}} \mathbb{I}\left(\text{TPOT}^{(i)}(s) \le \text{SLO}_{\text{TPOT}}^{(i)}\right) \right]$$

Where $\mathbb{I}(\cdot)$ is the indicator function, and $w_{\text{TTFT}}, w_{\text{TPOT}}$ denote workload priority weights.

AdaGen maximizes $U(\mathbf{X})$ subject to the instance memory capacity bound:

$$\sum_{j \in \text{Active}(k, t)} \left(L_{\text{prefill}}^{(j)} + s_j(t)\right) \cdot \text{BlockSize} \le B_{\text{max}}^{(k)} \quad \forall k, \forall t$$

Because directly optimizing this integer programming problem in real time is NP-hard, AdaGen’s simulation-based estimator evaluates candidate allocations via greedy gradient descent over predicted layout penalties.

Step-by-Step Implementation: Building a Layout-Aware Cluster Scheduler

To demonstrate how AdaGen operates in practice, let us implement an end-to-end Python reference implementation of AdaGen’s core engine: the 2D Request Classifier, the Simulation-Based Latency Estimator, and the Layout-Aware Load Balancer.

1. Data Models and Workload Definitions

""" adagen_scheduler.py Core reference implementation of AdaGen's Workload-Adaptive Cluster Scheduler. EuroSys 2026 Reference Architecture. """ from dataclasses import dataclass, field from enum import Enum from typing import List, Dict, Optional, Tuple import time import math class WorkloadClass(Enum): SHORT_PREFILL_LONG_DECODE = "SP_LD" # Chatbots, Agent Loops LONG_PREFILL_SHORT_DECODE = "LP_SD" # RAG, Document Extraction LONG_PREFILL_LONG_DECODE = "LP_LD" # Code Gen, Deep Analysis SHORT_PREFILL_SHORT_DECODE = "SP_SD" # Classification, Routing @dataclass class InferenceRequest: request_id: str arrival_time: float prefill_length: int expected_decode_length: int slo_ttft_ms: float slo_tpot_ms: float workload_class: Optional[WorkloadClass] = None assigned_instance: Optional[int] = None @dataclass class InstanceState: instance_id: int total_gpu_blocks: int used_gpu_blocks: int active_decode_tokens: int queued_prefill_tokens: int # Layout queue representing scheduled iterations (prefill_tokens, decode_tokens) future_layout: List[Tuple[int, int]] = field(default_factory=list) @property def free_blocks(self) -> int: return self.total_gpu_blocks - self.used_gpu_blocks

2. The 2D Workload Classifier

class WorkloadClassifier: """ Classifies incoming requests into 2D prefill-decode operational regimes. """ def __init__(self, prefill_threshold: int = 1024, decode_threshold: int = 256): self.prefill_threshold = prefill_threshold self.decode_threshold = decode_threshold def classify(self, request: InferenceRequest) -> WorkloadClass: is_long_prefill = request.prefill_length >= self.prefill_threshold is_long_decode = request.expected_decode_length >= self.decode_threshold if not is_long_prefill and is_long_decode: cls = WorkloadClass.SHORT_PREFILL_LONG_DECODE elif is_long_prefill and not is_long_decode: cls = WorkloadClass.LONG_PREFILL_SHORT_DECODE elif is_long_prefill and is_long_decode: cls = WorkloadClass.LONG_PREFILL_LONG_DECODE else: cls = WorkloadClass.SHORT_PREFILL_SHORT_DECODE request.workload_class = cls return cls

3. The Fast Simulation-Based Latency Estimator

class SimulationLatencyEstimator: """ Simulates iteration-by-iteration continuous batching execution to estimate TTFT, TPOT, and SLO violation penalties in <1ms. """ def __init__( self, alpha_prefill: float = 0.045, # ms per prefill token beta_decode: float = 0.850, # ms per decode token gamma_prefill_fixed: float = 8.5,# Fixed overhead when prefill present base_step_latency: float = 12.0 # Base GPU kernel overhead ): self.alpha = alpha_prefill self.beta = beta_decode self.gamma = gamma_prefill_fixed self.base_latency = base_step_latency def predict_iteration_duration(self, n_prefill: int, n_decode: int) -> float: """Computes hardware execution duration for one continuous batching step.""" fixed_overhead = self.gamma if n_prefill > 0 else 0.0 return self.base_latency + (self.alpha * n_prefill) + (self.beta * n_decode) + fixed_overhead def simulate_placement_penalty( self, instance: InstanceState, candidate_request: InferenceRequest, chunk_size: int = 512 ) -> float: """ Simulates the layout impact of assigning candidate_request to instance. Returns a scalar penalty score proportional to expected SLO breaches. """ # Memory check required_blocks = math.ceil((candidate_request.prefill_length + candidate_request.expected_decode_length) / 16) if instance.free_blocks < required_blocks: return float('inf') # Reject: OOM risk penalty = 0.0 current_time = time.time() queue_delay = 0.0 # Simulate prefill chunking remaining_prefill = candidate_request.prefill_length simulated_step = 0 while remaining_prefill > 0: chunk = min(remaining_prefill, chunk_size) active_decode = instance.active_decode_tokens step_lat = self.predict_iteration_duration(chunk, active_decode) queue_delay += step_lat remaining_prefill -= chunk simulated_step += 1 # Check if this prefill chunk violates ongoing decode TPOT SLO if active_decode > 0 and step_lat > 25.0: # Nominal 25ms TPOT limit # Heavy penalty for perturbing ongoing interactive streams penalty += (step_lat - 25.0) * 10.0 # Check TTFT for candidate request predicted_ttft = queue_delay if predicted_ttft > candidate_request.slo_ttft_ms: penalty += (predicted_ttft - candidate_request.slo_ttft_ms) * 5.0 return penalty

4. The Layout-Aware Cluster Scheduler

class AdaGenClusterScheduler: """ Multistep cluster scheduler optimizing token compute layouts across instances. """ def __init__(self, instances: List[InstanceState]): self.instances = {inst.instance_id: inst for inst in instances} self.classifier = WorkloadClassifier() self.estimator = SimulationLatencyEstimator() def schedule_batch(self, requests: List[InferenceRequest]) -> Dict[str, int]: """ Schedules a batch of incoming requests to optimal GPU instances. Returns mapping: {request_id: instance_id} """ # Step 1: 2D Classification for req in requests: self.classifier.classify(req) # Prioritize latency-critical classes first to secure tightest layouts sorted_requests = sorted( requests, key=lambda r: ( r.workload_class != WorkloadClass.SHORT_PREFILL_LONG_DECODE, r.slo_ttft_ms ) ) schedule_decisions: Dict[str, int] = {} # Step 2 & 3: Layout Balancing & Selective Assignment for req in sorted_requests: best_instance_id = -1 lowest_penalty = float('inf') for inst_id, inst_state in self.instances.items(): penalty = self.estimator.simulate_placement_penalty(inst_state, req) if penalty < lowest_penalty: lowest_penalty = penalty best_instance_id = inst_id if best_instance_id == -1 or lowest_penalty == float('inf'): # Fallback: Find instance with maximum free memory best_instance_id = max(self.instances.values(), key=lambda i: i.free_blocks).instance_id # Commit decision and update virtual instance layout schedule_decisions[req.request_id] = best_instance_id req.assigned_instance = best_instance_id self._update_instance_state(self.instances[best_instance_id], req) return schedule_decisions def _update_instance_state(self, instance: InstanceState, req: InferenceRequest): needed_blocks = math.ceil((req.prefill_length + req.expected_decode_length) / 16) instance.used_gpu_blocks += needed_blocks if req.workload_class in [WorkloadClass.SHORT_PREFILL_LONG_DECODE, WorkloadClass.LONG_PREFILL_LONG_DECODE]: instance.active_decode_tokens += 1 instance.queued_prefill_tokens += req.prefill_length

5. Executing an End-to-End Cluster Benchmark

if __name__ == "__main__": # Initialize 3 heterogeneous cluster worker instances (e.g. 8x H100 nodes) # Total 4000 KV blocks = 64,000 tokens of VRAM buffer per instance cluster_nodes = [ InstanceState(instance_id=0, total_gpu_blocks=4000, used_gpu_blocks=1200, active_decode_tokens=32, queued_prefill_tokens=0), InstanceState(instance_id=1, total_gpu_blocks=4000, used_gpu_blocks=800, active_decode_tokens=8, queued_prefill_tokens=1024), InstanceState(instance_id=2, total_gpu_blocks=4000, used_gpu_blocks=2400, active_decode_tokens=48, queued_prefill_tokens=0), ] scheduler = AdaGenClusterScheduler(cluster_nodes) # Simulated incoming heterogeneous traffic burst incoming_batch = [ InferenceRequest("req_chat_01", time.time(), prefill_length=48, expected_decode_length=1024, slo_ttft_ms=150.0, slo_tpot_ms=25.0), InferenceRequest("req_rag_02", time.time(), prefill_length=12288, expected_decode_length=64, slo_ttft_ms=1200.0, slo_tpot_ms=60.0), InferenceRequest("req_code_03", time.time(), prefill_length=4096, expected_decode_length=512, slo_ttft_ms=600.0, slo_tpot_ms=40.0), InferenceRequest("req_voice_04", time.time(), prefill_length=32, expected_decode_length=256, slo_ttft_ms=100.0, slo_tpot_ms=20.0), ] start_time = time.perf_counter() assignments = scheduler.schedule_batch(incoming_batch) scheduling_duration = (time.perf_counter() - start_time) * 1000.0 print("=== AdaGen Cluster Scheduling Results ===") print(f"Scheduling Overhead: {scheduling_duration:.3f} ms (Target: < 1.0 ms)") for req in incoming_batch: print(f"[{req.request_id}] Class: {req.workload_class.value:<6} | Prefill: {req.prefill_length:>5} | Decode: {req.expected_decode_length:>4} -> Instance {req.assigned_instance}")

Production Deployment Guide: Integrating AdaGen with vLLM and Kubernetes

Deploying AdaGen in enterprise infrastructure involves layering the scheduler as a dynamic Control Plane Proxy in front of multi-instance inference pods managed by Kubernetes or Ray Serve.

┌──────────────────────────────────────────────────────────────────────────────────┐ │ Kubernetes Production Mesh │ │ │ │ Client Ingress (REST / gRPC) │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ │ │ AdaGen Global Cluster Controller (Fast Simulator + 2D Classifier) │ │ │ └──────┬────────────────────────────┬────────────────────────────┬─────────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Worker Pod 0 │ │ Worker Pod 1 │ │ Worker Pod 2 │ │ │ │ [vLLM Node] │ │ [vLLM Node] │ │ [vLLM Node] │ │ │ │ TP=4, GPU:H100│ │ TP=4, GPU:H100│ │ TP=4, GPU:H100│ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ └──────────────────────────────────────────────────────────────────────────────────┘

1. Kubernetes Deployment Manifest

apiVersion: apps/v1 kind: Deployment metadata: name: adagen-cluster-router namespace: llm-serving spec: replicas: 2 selector: matchLabels: app: adagen-router template: metadata: labels: app: adagen-router spec: containers: - name: adagen-controller image: ghcr.io/aifetchly/adagen-cluster-scheduler:v1.0.0 env: - name: WORKER_ENDPOINTS value: 'http://vllm-worker-0.llm-serving:8000,http://vllm-worker-1.llm-serving:8000,http://vllm-worker-2.llm-serving:8000' - name: CHUNK_SIZE value: '512' - name: SIMULATOR_ALPHA value: '0.045' - name: SIMULATOR_BETA value: '0.850' ports: - containerPort: 8080 resources: requests: cpu: '4' memory: '8Gi' limits: cpu: '8' memory: '16Gi'

2. Upstream Proxy Configuration

When integrating with platforms like aiFetchly API Integration, configure client SDKs to route directly to the AdaGen proxy port. The proxy transparently forwards streaming tokens via HTTP Server-Sent Events (SSE) while preserving sub-millisecond scheduling latency.

Empirical Benchmark Evaluation

Extensive evaluations across multi-turn real-world production traces (ShareGPT, LMSYS-Chatbot-Arena, and Enterprise RAG datasets) demonstrate AdaGen’s performance gains over traditional scheduling baselines:

SLO Attainment vs Request Arrival Rate (P99 Workload): 100% ┤ ╭──── AdaGen (EuroSys 2026) 90% ┤ ╭────╯ 80% ┤ ╭───────────╯ 70% ┤ ╭───────────╯ ........... DistServe Disaggregated 60% ┤ ╭───────╯ ........ 50% ┤ ─────╯ ....... - - - - vLLM Memory-Aware 40% ┤ ....... - - - - - - - 30% ┤ ....... - - - - - - - 0% └─────────────────────────────────────────────────────── 5 req/s 15 req/s 25 req/s 35 req/s

Key Performance Findings:

  1. 3.6x Higher SLO Attainment: Under high concurrency bursts ($>30\text{ req/s}$), naive memory balancing drops to $27%$ SLO attainment due to prefill-decode batch collisions. AdaGen maintains $97.4%$ SLO compliance by isolating iteration-delay spikes.
  2. 2.0x Cost Efficiency: Because AdaGen eliminates wasted prefill bubbles, serving clusters can satisfy equivalent traffic volumes on half the number of GPU instances, reducing enterprise cloud infrastructure spend by over $50%$.
  3. Sub-Millisecond Overhead: Across all benchmarks, the fast simulation latency estimator executed in $0.34\text{ ms}$ on average, fully preserving real-time TTFT constraints.

Developers can explore additional deployment templates and infrastructure guides in the aiFetchly Documentation and review cluster economic calculators on the aiFetchly Pricing & Cluster Sizing Page.

Frequently Asked Questions (FAQ)

What is the fundamental difference between memory-aware scheduling and compute layout scheduling?

Memory-aware schedulers only evaluate whether a node has enough free VRAM blocks to store incoming KV cache tensors, completely ignoring how those tokens will be packed into execution iterations. Compute layout scheduling analyzes the exact arrangement of prefill and decode tokens across time steps, directly preventing compute-heavy prefills from stalling latency-sensitive decodes.

Does AdaGen require hardware modifications or custom CUDA kernels on worker pods?

No. AdaGen is completely software-defined and operates as an upstream cluster control plane proxy. Worker nodes run standard inference engines (such as vLLM, TensorRT-LLM, or SGLang) without code changes or proprietary hardware dependencies.

How does AdaGen handle inaccurate decode length estimations?

AdaGen employs an adaptive receding-horizon simulator. If a request generates more tokens than initially anticipated, the instance dynamically updates its future compute layout matrix, allowing upstream scheduling decisions to auto-correct in the subsequent scheduling window.