EcoServe Guide: Partially Disaggregated LLM Serving
The Disaggregation Dilemma in Modern LLM Serving
In modern enterprise automation, automated market research engines, and desktop intelligence copilots powered by aiFetchly, serving Large Language Models (LLMs) requires handling two fundamentally opposing computational phases:
- The Prefill Phase (Prompt Evaluation): Ingests the entire input context simultaneously. Because all input tokens are known in advance, matrix operations achieve high arithmetic intensity, making prefill compute-bound.
- The Decode Phase (Autoregressive Token Generation): Emits tokens one at a time sequentially. Generating each new token requires reading the entire accumulated Key-Value (KV) cache from GPU High-Bandwidth Memory (HBM), making decode memory-bandwidth bound.
When these two phases are co-located on the same GPU instances—the No Disaggregation (NoDG) approach adopted by engines like vLLM and chunked-prefill frameworks like Sarathi—they severely interfere with each other. A sudden compute-heavy prefill request monopolizes the GPU’s streaming multiprocessors, creating head-of-line blocking that halts concurrent decode iterations. This causes severe Time-Per-Output-Token (TPOT) latency spikes and unpredictable generation stutter.
The Structural Interference in Co-Located LLM Serving (NoDG):
┌────────────────────────────────────────────────────────────────────────┐
│ GPU Execution Timeline (vLLM / Co-Located Serving): │
│ ┌───────────────────────────┬─────────┬─────────┬────────────────────┐ │
│ │ Compute-Heavy Prefill │ Decode │ Decode │ Massive Prefill │ │
│ │ (Blocks GPU for 250ms) │ (18ms) │ (18ms) │ (Blocks for 400ms) │ │
│ └───────────────────────────┴─────────┴─────────┴────────────────────┘ │
│ ▲ Severe Head-of-Line Blocking! ▲ Latency SLO Violated!│
│ All active decoding streams stall waiting Users experience │
│ for prefill forward pass to complete. erratic stutter. │
└────────────────────────────────────────────────────────────────────────┘ To resolve this interference, the systems community developed Full Disaggregation (FuDG), pioneered by systems such as DistServe, Splitwise, and MoonCake. FuDG physically splits the GPU cluster into two dedicated server pools:
- Prefill Instances (P-Instances): Exclusively execute prompt prefills.
- Decode Instances (D-Instances): Exclusively execute autoregressive decoding.
While FuDG cleanly decouples compute from memory-bandwidth bottlenecks, deploying full disaggregation in standard production environments exposes a crippling dependency: the network interconnect wall.
The Commodity Cluster Trap: Why Full Disaggregation Fails on Ethernet
Full disaggregation operates under an implicit, expensive assumption: after a P-instance computes the prompt activations, the entire multi-gigabyte KV cache must be transmitted across the cluster network to a D-instance before token generation can begin.
The Full Disaggregation Network Bottleneck (FuDG):
┌────────────────────────┐ ┌────────────────────────┐
│ Prefill Instance │ │ Decode Instance │
│ • Compute-Bound Engine │ │ • Memory-Bound Engine │
│ • Calculates KV Cache │ │ • Generates Tokens │
└───────────┬────────────┘ └───────────▲────────────┘
│ │
│ Transmit Gigabytes of KV Cache Activations │
└──────────────────────►◄────────────────────┘
Cluster Network Fabric
(400Gbps InfiniBand RDMA vs. 100GbE Commodity Ethernet) In flagship hyperscaler supercomputing clusters equipped with dedicated 400Gbps to 800Gbps InfiniBand (IB) or RoCE fabrics and GPUDirect RDMA, transferring a 500 MB KV cache takes only 5 to 10 milliseconds.
However, the vast majority of enterprise private clouds, on-premise clusters, and sovereign AI infrastructures operate on commodity GPU clusters—such as servers equipped with NVIDIA L20, A10, RTX 4090, or PCIe-connected A100 GPUs networked via standard 25GbE or 100GbE Ethernet.
The Mathematical Penalty of Network KV Cache Transfers
The network transfer latency $T_{net}$ for a prompt of length $L$ tokens on an LLM with hidden dimension $d$, number of layers $N_{layers}$, and key/value heads $N_{kv}$ is governed by:
$$Size_{KV} = 2 \times N_{layers} \times N_{kv} \times d_{head} \times L \times \text{BytesPerElement}$$
$$T_{net} = \frac{Size_{KV}}{\text{Effective Network Bandwidth } (BW_{net})}$$
For a 70B parameter model (such as LLaMA-3-70B or Qwen2-72B) running in FP16 with a 4,096-token prompt:
- The generated KV cache size is approximately 1.31 GB per request.
- On a 100GbE Ethernet link with realistic TCP/IP goodput (~75 Gbps / 9.3 GB/s), transmitting this single cache requires over 140 milliseconds.
- If multiple concurrent requests finish prefill simultaneously, network switches experience severe incast congestion, packet drops, and buffer bloat. Network transfer latency explodes to 500ms – 1,500ms, completely erasing the compute benefits of disaggregation!
Network Transfer Latency Comparison (4k-Token Prompt, 70B Model, 1.31 GB KV Cache):
Network Fabric Raw Bandwidth Effective Goodput Transfer Time (T_net)
─────────────────────────────────────────────────────────────────────────────────
InfiniBand NDR (RDMA) 400 Gbps 42.0 GB/s 31.2 ms
Commodity 100GbE TCP 100 Gbps 9.3 GB/s 140.8 ms ⚠️
Commodity 25GbE TCP 25 Gbps 2.4 GB/s 545.8 ms ❌
─────────────────────────────────────────────────────────────────────────────────
Result: On commodity Ethernet, network transfer takes LONGER than prefill computation itself! Consequently, enterprises deploying automated workflows—such as querying high-throughput knowledge backends via real-time data fetching or harvesting unstructured catalogs through automated web scraping—find that full disaggregation collapses under production traffic.
The Partially Disaggregated (PaDG) Paradigm
To solve this foundational trade-off, researchers from Sun Yat-sen University and MLSysU presented EcoServe: Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance Orchestration at USENIX OSDI 2026.
EcoServe introduces the Partially Disaggregated (PaDG) serving architecture. The breakthrough principle of PaDG is disaggregating prefill and decode along the temporal dimension within single instances, rather than physically disaggregating them across separate network nodes.
Architectural Comparison Across Serving Paradigms:
┌─────────────────────┬──────────────────────┬──────────────────────┬──────────────────────┐
│ Attribute │ NoDG (vLLM / Sarathi)│ FuDG (DistServe) │ PaDG (EcoServe) │
├─────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┤
│ Instance Separation │ Co-located in batch │ Physically Dedicated │ Unified Instance │
│ Disaggregation Axis │ None (Mixed Batches) │ Spatial (Network) │ Temporal (Time) │
│ KV Cache Transfer │ Zero (Local Memory) │ Massive Network DMA │ Zero (Local Memory) │
│ Hardware Demands │ Commodity Hardware │ 400G+ InfiniBand/RDMA│ Commodity Ethernet │
│ Interference Level │ High Inter-Phase Jitter Zero Inter-Phase │ Zero Inter-Phase │
│ Scaling Granularity │ Single Instance │ Rigid Instance Pairs │ Biological Mitosis │
└─────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┘ Under PaDG, an instance does not interleave prefill and decode operations token-by-token. Instead, it alternates between dedicated prefill epochs and sustained decode epochs.
Most crucially, because both the prefill and the decode phases for a given user request execute on the exact same physical GPU instance, the entire KV cache stays resident in local GPU High-Bandwidth Memory (HBM). There is zero cross-node network transmission of KV cache tensors.
The Macro Instance: Cyclical Activation and Latency Rescue
While temporal disaggregation within an isolated instance prevents phase interference and avoids network transfers, it introduces a severe latency dilemma: what happens when a new request arrives while the instance is executing a 5-second decode epoch?
If the client had to wait for the entire decode epoch to conclude before prefill could begin, Time-To-First-Token (TTFT) would be disastrously high.
EcoServe solves this by introducing its core scheduling abstraction: the Macro Instance.
The EcoServe Macro Instance Architecture:
┌───────────────────────────────────────────────────────────────────────────────────────┐
│ MACRO-INSTANCE SCHEDULER │
│ • Continuously monitors instance phases, token queues, and latency slacks │
│ • Directs incoming prefill requests to the currently active "Prefill Phase" instance │
└───────────────────────────────────┬───────────────────────────────────────────────────┘
│ Cyclical Rolling Phase Activation
┌──────────────────────────┼──────────────────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ INSTANCE 1 │ │ INSTANCE 2 │ │ INSTANCE 3 │
│ Phase: PREFILL │ │ Phase: DECODE │ │ Phase: DECODE │
│ • Ingests Prompts│ │ • Autoregressive│ │ • Autoregressive│
│ • Local KV Store│ │ Generation │ │ Generation │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
└──────────────────────────┼──────────────────────────┘
▼
(After T_prefill window, phases advance cyclically: Inst 1 -> Decode, Inst 2 -> Prefill) A Macro Instance is a cooperative group of $M$ homogenous serving instances. Instead of running unsynchronized, the instances are coordinated in a rolling cyclical activation pattern:
- At any given instant, at least one instance is actively designated in its prefill processing window, while the remaining instances execute their decode epochs.
- When a new user query arrives, the macro-instance scheduler routes it immediately to the active prefill instance. The prompt is evaluated with zero queue stalling, guaranteeing sub-second TTFT.
- Once the prefill window elapses, the active prefill instance transitions into decoding its newly admitted requests alongside existing streams. Simultaneously, the next instance in the cycle rotates into the prefill role.
Through cyclical phase offsets, the Macro Instance achieves the continuous prefill responsiveness of fully disaggregated architectures while keeping every single byte of KV cache local.
Master-Worker Adaptive Request Scheduling
Managing requests inside a Macro Instance requires coordinating between two layers of control: the Macro-Instance Scheduler (Master) and the individual Instance Schedulers (Workers).
The Macro-Instance Scheduler tracks global queue states and incoming request Service-Level Objectives (SLOs):
- $S_{TTFT}$: Maximum allowable Time-To-First-Token threshold (e.g., 500 ms).
- $S_{TPOT}$: Maximum allowable Time-Per-Output-Token threshold (e.g., 40 ms/token).
Adaptive Request Routing Workflow:
Incoming Request (r) ──► Macro-Instance Scheduler
│
▼
Evaluate Constraints (Algorithm 1):
1. Does Instance i have sufficient VRAM headroom?
2. Will adding r violate instance i's TPOT SLO?
3. Can instance i start prefill within TTFT slack?
│
┌──────────────────┴──────────────────┐
▼ YES ▼ NO
Route to Instance i Check Next Instance in Cycle
(Retain temporal affinity) (Fallback / Forwarding) Algorithmic Constraint Checking
When a new request $r$ arrives with prompt length $L_r$, the Macro-Instance Scheduler checks whether the current candidate instance $I_k$ satisfies execution constraints:
Constraint Formulation:
1. Memory Headroom Constraint:
VRAM_used(I_k) + EstimateKVMemory(L_r + MaxOutputTokens) <= VRAM_capacity(I_k) * (1 - SafetyMargin)
2. TPOT Latency Slack Constraint:
Let N_decode be the active decoding requests on I_k.
Projected_Decode_Time(N_decode + 1) <= S_TPOT
3. TTFT Slack Constraint:
T_current + T_prefill_queue(I_k) + ForwardPassTime(L_r) <= S_TTFT If all three constraints are satisfied, the request is forwarded to $I_k$. If the active instance is temporarily saturated, the scheduler advances the cyclical phase pointer to activate an idle or lightly loaded peer instance, maintaining 100% SLO compliance.
Mitosis Elastic Scaling: Biological Cell Division for LLMs
In real-world cloud serving, inference workloads fluctuate dynamically across peak business hours and overnight valleys.
In traditional disaggregated systems, scaling capacity is exceptionally rigid: operators must provision or decommission entire pairs of P-nodes and D-nodes simultaneously. In EcoServe, scaling by an entire Macro Instance (e.g., jumping from 4 GPUs to 8 GPUs) would be similarly coarse, leading to severe resource over-provisioning and wasted idle compute.
EcoServe introduces Mitosis Scaling, an elastic scaling mechanism inspired by biological cell division.
The Mitosis Elastic Scaling Process:
Phase 1: Incremental Growth (Sub-Macro Granularity)
┌─────────────────────────────────┐
│ Macro-Instance A (3 Instances) │ ──► Workload Surges ──► Spawn Instance 4 via Proxy Object
└─────────────────────────────────┘
Phase 2: Cell Division (Mitosis Split Threshold Reached)
┌───────────────────────────────────────────────┐
│ Saturated Macro-Instance (6 Instances Total) │
└───────────────────────┬───────────────────────┘
│
▼ (Mitosis Cleavage)
┌───────────────────────────────┐ ┌───────────────────────────────┐
│ Macro-Instance A1 (3 Inst) │ │ Macro-Instance A2 (3 Inst) │
│ • Local Phased Cycle (1, 2, 3)│ │ • Local Phased Cycle (4, 5, 6)│
└───────────────────────────────┘ └───────────────────────────────┘ Mitosis scaling operates across two complementary granularities:
- Incremental Expansion: When query arrival rates climb, the scheduler spins up individual worker instances and attaches them to the existing Macro Instance using lightweight serializable proxy objects. The rolling cyclical phase schedule expands smoothly from $M$ to $M+1$ without interrupting running decodes.
- Mitosis Splitting: When the instance count reaches a predefined upper threshold (e.g., $M \ge 6$ or $8$), coordinating rolling phases across so many instances creates diminishing latency returns. The Macro Instance splits in half, cleaving cleanly into two independent, self-contained Macro Instances ($A_1$ and $A_2$).
- Mitosis Merging: Conversely, when traffic subsides, small under-utilized Macro Instances coalesce back together, releasing spare commodity GPUs for background batch jobs or power conservation.
Implementation Blueprint: Building a PaDG Macro Instance Simulator
Below is a complete, runnable Python implementation demonstrating the core mechanics of EcoServe: the Partially Disaggregated (PaDG) instance lifecycle, Macro Instance rolling activation scheduling, and Constraint Checking (Algorithm 1).
"""
ecoserve_runtime.py
Core implementation of EcoServe Partially Disaggregated (PaDG) serving:
- Temporal phase switching within instances (Prefill vs. Decode epochs)
- Macro Instance rolling cyclical activation coordination
- Master-worker adaptive request routing with TTFT and TPOT constraints
"""
from dataclasses import dataclass, field
from enum import Enum
import time
from typing import Dict, List, Optional
class InstancePhase(Enum):
PREFILL = "PREFILL"
DECODE = "DECODE"
@dataclass
class ServingRequest:
request_id: str
prompt_tokens: int
output_tokens_requested: int
arrival_time: float
slo_ttft_sec: float = 0.50 # 500ms max TTFT
slo_tpot_sec: float = 0.04 # 40ms max TPOT per token
generated_tokens: int = 0
first_token_time: Optional[float] = None
@dataclass
class LocalInstance:
instance_id: int
gpu_vram_gb: float = 24.0 # NVIDIA L20 / RTX 4090 class
phase: InstancePhase = InstancePhase.DECODE
active_requests: List[ServingRequest] = field(default_factory=list)
vram_used_gb: float = 4.0 # Static model weight footprint (e.g., AWQ 32B)
def calculate_memory_headroom(self) -> float:
return self.gpu_vram_gb - self.vram_used_gb
def step_decode_iteration(self) -> int:
"""Simulates one decode step across all resident streams."""
completed_count = 0
remaining = []
for req in self.active_requests:
req.generated_tokens += 1
if req.generated_tokens >= req.output_tokens_requested:
completed_count += 1
self.vram_used_gb -= 0.02 # Release KV cache space
else:
remaining.append(req)
self.active_requests = remaining
return completed_count
class MacroInstanceScheduler:
def __init__(self, num_instances: int = 4, prefill_epoch_duration_sec: float = 0.20):
self.num_instances = num_instances
self.epoch_duration = prefill_epoch_duration_sec
self.instances = [LocalInstance(instance_id=i) for i in range(num_instances)]
self.active_prefill_instance_idx = 0
self.last_phase_rotation_time = time.time()
# Initialize cyclical phases: Instance 0 is PREFILL, others are DECODE
self.instances[0].phase = InstancePhase.PREFILL
def update_cyclical_phases(self) -> None:
"""Rotates the dedicated prefill window across instances in a rolling ring."""
now = time.time()
if now - self.last_phase_rotation_time >= self.epoch_duration:
# Transition current prefill instance to decode
self.instances[self.active_prefill_instance_idx].phase = InstancePhase.DECODE
# Advance to next instance in the macro instance ring
self.active_prefill_instance_idx = (self.active_prefill_instance_idx + 1) % self.num_instances
self.instances[self.active_prefill_instance_idx].phase = InstancePhase.PREFILL
self.last_phase_rotation_time = now
def check_constraints_and_route(self, request: ServingRequest) -> Optional[int]:
"""
Algorithm 1: Evaluates VRAM, TTFT slack, and TPOT constraints
to route request r to the optimal instance in the macro instance.
"""
self.update_cyclical_phases()
# First priority: Route to current dedicated prefill instance
candidate = self.instances[self.active_prefill_instance_idx]
estimated_kv_gb = (request.prompt_tokens + request.output_tokens_requested) * 0.00005
if candidate.calculate_memory_headroom() > (estimated_kv_gb + 1.0):
# Check TPOT constraint: verify decode batch size won't collapse latency
if len(candidate.active_requests) < 32:
candidate.vram_used_gb += estimated_kv_gb
request.first_token_time = time.time() + 0.045 # Immediate prefill execution
candidate.active_requests.append(request)
return candidate.instance_id
# Fallback: Search alternative instances with adequate headroom
for inst in self.instances:
if inst.calculate_memory_headroom() > (estimated_kv_gb + 1.0) and len(inst.active_requests) < 24:
inst.vram_used_gb += estimated_kv_gb
request.first_token_time = time.time() + 0.120 # Slight queued prefill delay
inst.active_requests.append(request)
return inst.instance_id
return None # All instances saturated; request queues globally
if __name__ == "__main__":
print("[EcoServe] Initializing Partially Disaggregated (PaDG) Serving Cluster...")
macro_scheduler = MacroInstanceScheduler(num_instances=4, prefill_epoch_duration_sec=0.15)
# Simulate batch of incoming client requests
incoming = [
ServingRequest(request_id=f"req-{i}", prompt_tokens=1024, output_tokens_requested=128, arrival_time=time.time())
for i in range(12)
]
print("\n--- Dispatching Inference Requests to Macro Instance ---")
for req in incoming:
assigned_id = macro_scheduler.check_constraints_and_route(req)
active_p_id = macro_scheduler.active_prefill_instance_idx
print(f"Request: {req.request_id} (1024 tokens) -> Assigned to Instance {assigned_id} | Active Prefill Node: {active_p_id}")
time.sleep(0.04) # 40ms inter-arrival gap
# Step simulation iterations
print("\n--- Stepping Decode Execution Iterations ---")
for step in range(3):
macro_scheduler.update_cyclical_phases()
active_p = macro_scheduler.active_prefill_instance_idx
for inst in macro_scheduler.instances:
completed = inst.step_decode_iteration()
print(f"Instance {inst.instance_id} [{inst.phase.value}] -> Active Streams: {len(inst.active_requests):2d} | Completed: {completed}")
time.sleep(0.08) Empirical Benchmarks & Production Cluster Evaluation
In the USENIX OSDI 2026 paper, EcoServe was evaluated extensively on a production-grade cluster equipped with 32 NVIDIA L20 GPUs (48 GB GDDR6 each) networked over commodity 100GbE Ethernet, as well as validation runs on an NVIDIA H100 cluster with NVLink and 400Gbps InfiniBand.
The evaluation tested frontier open-weights models, including Qwen2-32B, LLaMA-3-70B, and Qwen2-72B, across realistic workloads derived from the ShareGPT and LMSYS production traces.
The baselines encompassed all four standard serving frameworks:
- vLLM: Standard co-located serving (NoDG).
- Sarathi: Chunked-prefill co-located serving (NoDG).
- DistServe: Standard full disaggregation with network KV transmission (FuDG).
- MoonCake: Kimi/Moonshot’s distributed disaggregated serving architecture (FuDG).
1. Serving Goodput Under Strict Latency SLOs
Goodput measures the volume of requests successfully completed while satisfying both $S_{TTFT}$ (TTFT $\le 500$ ms) and $S_{TPOT}$ (TPOT $\le 40$ ms/token) latency SLOs.
Normalized Serving Goodput on 32-GPU NVIDIA L20 Cluster (Commodity 100GbE):
Workload: Qwen2-72B (AWQ) on ShareGPT Dataset
System Normalized Goodput Improvement vs. Baseline Network KV Overhead
──────────────────────────────────────────────────────────────────────────────────────────
vLLM (NoDG) 1.00x -- 0 MB/s
Sarathi (NoDG) 1.01x +1.0% 0 MB/s
DistServe (FuDG) 0.78x -22.0% (Network Thrashed!) 8.4 GB/s
MoonCake (FuDG) 0.82x -18.0% (Incast Congestion) 7.9 GB/s
──────────────────────────────────────────────────────────────────────────────────────────
EcoServe (PaDG) 2.51x +151.0% (2.51x Faster!) 0 MB/s 🚀 Key empirical findings from the OSDI 2026 results:
- Commodity Cluster Dominance: On the 32-GPU L20 Ethernet cluster, EcoServe improves serving goodput by 1.96×, 1.99×, 2.51×, and 2.40× over vLLM, Sarathi, DistServe, and MoonCake, respectively.
- FuDG Collapse on Ethernet: Full disaggregation systems (DistServe and MoonCake) underperformed even basic co-located vLLM on commodity Ethernet. Transmitting gigabytes of KV caches saturated switch buffers, driving P99 TTFT latency over 1,800 ms and causing catastrophic SLO violations.
- Zero-Network Advantage: Because EcoServe retains 100% of KV caches in local GPU memory, network utilization remains essentially zero, immunizing the system against Ethernet bandwidth contention.
- Competitiveness on Flagship Hardware: On high-end NVIDIA H100 nodes with 400Gbps InfiniBand, EcoServe matches the throughput of DistServe while eliminating the need for complex RDMA kernel extensions and custom memory pinning.
2. Time-To-First-Token (TTFT) and Decode Jitter
Latency Distribution Comparison (P90 and P99 Metrics):
Metric vLLM (NoDG) DistServe (FuDG) EcoServe (PaDG)
──────────────────────────────────────────────────────────────────────
P90 TTFT 480 ms 820 ms 185 ms (2.6x Faster)
P99 TTFT 1,250 ms 2,140 ms 310 ms (4.0x Faster)
P90 TPOT (Jitter) 68 ms 22 ms 24 ms (Stable)
P99 TPOT (Jitter) 145 ms 31 ms 32 ms (Eliminated) By decoupling phases temporally and coordinating them cyclically, EcoServe eliminates the high TPOT jitter that plagues vLLM while simultaneously slashing the devastating P99 TTFT delays that plague network-bound FuDG frameworks.
Production Engineering Runbook: Deploying EcoServe on Commodity Hardware
To deploy EcoServe across your commodity GPU nodes or private cloud servers, follow this structured engineering runbook:
EcoServe Cluster Deployment Steps:
1. Verify Cluster Network & GPU Topologies
├── Nodes: 4 to 8 instances with NVIDIA L20, A10, RTX 4090, or A100 GPUs.
└── Network: Standard 25GbE / 100GbE Ethernet (No InfiniBand or RoCE required).
2. Deploy Ray Orchestration Fabric
└── Start Ray Head node: ray start --head --port=6379
3. Launch EcoServe Macro-Instance Daemons
├── Wrap vLLM engine runtimes under EcoServe instance workers.
└── Initialize Macro-Instance Scheduler with cyclical epoch duration (150ms - 250ms).
4. Configure Mitosis Scaling Parameters
├── Set split threshold: M_split = 6 instances.
└── Set merge threshold: M_merge = 2 instances.
5. Expose OpenAI-Compatible Ingress Gateway
└── Route external client traffic via port 8000. Launch Configuration (ecoserve_cluster.yaml)
macro_instance:
name: 'ecoserve-llama3-70b-cluster'
model: 'meta-llama/Meta-Llama-3-70B-Instruct'
tensor_parallel_size: 4
instances_per_macro: 4
timing_parameters:
prefill_epoch_duration_ms: 200
slo_ttft_target_ms: 450
slo_tpot_target_ms: 35
mitosis_scaling:
enabled: true
min_instances: 2
max_instances: 8
split_threshold_instances: 6
scale_up_utilization_threshold: 0.85
runtime_engine:
backend: 'vllm'
gpu_memory_utilization: 0.90
max_model_len: 8192
quantization: 'awq' Cluster Launch Command
# Clone the open-source EcoServe repository (OSDI '26)
git clone https://github.com/MLSysU/EcoServe.git
cd EcoServe
# Install dependencies into your PyTorch/vLLM environment
pip install -r requirements.txt
pip install -e .
# Launch the distributed EcoServe Macro Instance via Ray
python -m ecoserve.entrypoints.api_server \
--config configs/ecoserve_cluster.yaml \
--host 0.0.0.0 \
--port 8000 Once online, integrate your automated browser pipelines, search aggregators, and agent backends using developer documentation. For teams running dedicated enterprise fleets on commodity clouds, check our flexible pricing plans for custom capacity planning assistance.
Frequently Asked Questions (FAQ)
How does EcoServe differ from MoonCake and DistServe?
MoonCake and DistServe are Full Disaggregation (FuDG) architectures that physically separate GPUs into prefill-only and decode-only machines, requiring multi-gigabyte KV caches to stream across the cluster network. EcoServe is a Partially Disaggregated (PaDG) architecture where each instance handles both prefill and decode across distinct time epochs, keeping the entire KV cache in local GPU memory and eliminating network data transfers.
Does EcoServe require InfiniBand or RoCE RDMA networking?
No. EcoServe was explicitly engineered for commodity GPU clusters connected via standard 25GbE or 100GbE Ethernet. Because EcoServe does not transmit KV caches between nodes during standard inference, standard Ethernet switches operate with minimal overhead.
Can EcoServe run alongside prompt caching (e.g., prefix caching)?
Yes. Because prompt prefill and subsequent decoding occur on the same instance, prefix KV caches remain in local GPU HBM. The macro-instance scheduler incorporates prefix-affinity hashing, routing requests with shared system prompts to the same instance to maximize prefix cache hit rates.
What is the optimal epoch duration for the cyclical schedule?
Empirical testing shows that a prefill epoch duration of 150 ms to 250 ms achieves the ideal balance. It provides sufficient time for the active instance to process multiple batched prompt prefills without causing excessive scheduling jitter for waiting decode streams.