LLM ServingLlumnixLive MigrationKV CacheOSDI 2024Distributed Systems

Llumnix Guide: Dynamic LLM Serving with Live Migration

October 14, 2026

The Static Scheduling Dilemma in Multi-Instance LLM Serving

Deploying large language models (LLMs) across production multi-GPU clusters has become standard engineering practice for high-throughput AI services, agentic workflows, and automated data-fetching pipelines. Modern inference frameworks such as vLLM, TensorRT-LLM, and Ray Serve efficiently batch requests on individual GPU instances using techniques like PagedAttention and continuous iteration batching.

However, when scaling from a single server to multi-instance cloud deployments (dozens or hundreds of GPU nodes), a critical architectural flaw emerges: the static request-to-instance binding assumption.

In conventional cluster serving, an ingress router or load balancer assigns each incoming request to a specific GPU instance upon arrival—typically using round-robin, least-connections, or prompt-length heuristics. Once assigned, the request remains pinned to that specific GPU instance until token generation terminates.

This static assignment creates severe operational bottlenecks because LLM request execution is inherently non-deterministic and unpredictable:

Unknown Output Sequence Lengths: When an inference request arrives, the serving engine cannot know whether the model will generate 5 tokens or 4,000 tokens before emitting an end-of-sequence (<eos>) token.
Resource Demand Skew: A single request generating thousands of tokens occupies its allocated key-value (KV) cache memory for tens of seconds, while shorter queries finish in hundreds of milliseconds.
Head-of-Line (HoL) Blocking: When a burst of long-prompt or high-generation queries lands on Instance A, its High-Bandwidth Memory (HBM) becomes saturated. New incoming requests on Instance A are queued, suffering massive Time-to-First-Token (TTFT) delays, even if neighboring Instance B sits completely idle with 80% available GPU memory.

Standard serving stacks have no mechanism to rebalance active requests across instances. If memory runs out on a node, the scheduler is forced to choose between two destructive fallbacks: swapping KV cache blocks to CPU memory (which incurs crippling PCIe transfer latency and stalls decoding) or preemption through recomputation (aborting the request, evicting its cache, and re-running the prompt prefill from scratch later).

To overcome this fundamental limitation, researchers from Alibaba Cloud and Peking University introduced Llumnix (presented at USENIX OSDI 2024 by Biao Sun, Ziming Huang, Hanyun Zhao, Wencong Xiao, and collaborators). Llumnix breaks the static binding model by introducing fine-grained live migration of in-flight LLM requests between distributed GPU instances.

By streaming KV cache tensors asynchronously over high-speed networks while token generation continues uninterrupted, Llumnix achieves cluster-wide dynamic rescheduling, slashes tail latencies (P99 TTFT and TBT) by up to 80%, and reduces Service-Level Objective (SLO) violations by 90%.

The Structural Bottleneck: Why LLM Serving Needs Live Migration

To understand why traditional multi-server LLM schedulers struggle under bursty production workloads, consider the fundamental mechanics of autoregressive inference. An LLM request consists of two distinct operational phases:

Prefill Phase (Prompt Evaluation): Compute-intensive forward pass that processes all prompt tokens in parallel, generating the initial KV cache.
Decode Phase (Token Generation): Memory-bandwidth-bound autoregressive loop that generates one token per iteration, growing the KV cache by one token state per layer at every step.

+-------------------------------------------------------------------------+
|                  Traditional Static Cluster Scheduling                  |
|                                                                         |
| Ingress Proxy                                                           |
|      │                                                                  |
|      ├───► Server 1 (Heavy Load):  [Req 1: 32k ctx] [Req 2: 16k ctx]    |
|      │                             HBM 98% Full! New Requests Queued!   |
|      │                             P99 TTFT = 8,420 ms (SLO VIOLATED)   |
|      │                                                                  |
|      └───► Server 2 (Idle Load):   [Req 3: 128 tokens]                  |
|                                    HBM 15% Full! Compute Starved!       |
|                                    Idle GPU Compute Wasted              |
+-------------------------------------------------------------------------+
                                    vs.
+-------------------------------------------------------------------------+
|                    Llumnix Dynamic Serving with Live Migration          |
|                                                                         |
| Global Orchestrator (Llumnix Manager)                                   |
|      │                                                                  |
|      ├───► Server 1: [Req 1 (Active)] ────► Pre-Copy Migration ────┐     |
|      │               Unblocks Local Queue                        │     |
|      │                                                           ▼     |
|      └───► Server 2: [Req 3] <────── Seamlessly Resumes ── [Req 1]      |
|                      Balanced Memory & Compute Across All Nodes         |
|                      P99 TTFT = 1,120 ms (80% Latency Reduction)        |
+-------------------------------------------------------------------------+

When an instance experiences sudden memory pressure, static systems have no graceful escape path:

Swapping to Host Memory: Transferring a 10 GB KV cache over PCIe Gen 4/5 takes between 150 ms and 350 ms. During this window, the request cannot generate tokens, creating a noticeable stutter (tail TBT inflation).
Request Abort / Recomputation: Dropping accumulated context wastes all the floating-point operations (FLOPs) already invested in generating earlier tokens, multiplying operational inference costs.

In virtualized operating systems, hypervisors solved server imbalance decades ago through VM Live Migration—moving running virtual machines across physical hosts without terminating running applications. Llumnix adapts this foundational systems concept to LLM serving by treating a request’s KV cache and attention state as a migratable virtual process.

Core Innovation 1: Asynchronous Multi-Stage Pre-Copy Migration

Migrating an active LLM request between two physical servers presents a unique systems challenge: how do you move multi-gigabyte KV cache tensors across the network without halting real-time token generation?

A naive “stop-and-copy” approach—pausing generation on Instance A, transferring all KV cache blocks over InfiniBand/RoCE, and resuming generation on Instance B—introduces unacceptable request downtime. For a 32k-context request on LLaMA-3-70B, the KV cache exceeds 10 GB. Even over a 200 Gbps network link, moving 10 GB takes approximately 400 milliseconds—violating standard time-between-tokens (TBT) SLO budgets (typically 20–50 ms).

Llumnix solves this with an iterative multi-stage pre-copy protocol, directly inspired by OS memory hypervisors:

+-------------------------------------------------------------------------+
|              Llumnix Multi-Stage Asynchronous Pre-Copy Pipeline         |
|                                                                         |
| Time (t) ─────────────────────────────────────────────────────────────► |
|                                                                         |
| Source GPU (Server A):                                                  |
| Compute Stream:   [ Dec 1 ] [ Dec 2 ] ... [ Dec N ] ──┐ (Stop & Copy)   |
| RDMA Stream:      ==== Transfer Base KV Cache =====>  └── [Delta Sync]  |
|                                                                  │      |
| Destination GPU (Server B):                                      │      |
| Memory Allocator: [ Allocate Virtual Paged Blocks ]              │      |
| RDMA Stream:      <=== Receive Base Blocks ========              ▼      |
| Compute Stream:                                     [ Resume Dec N+1 ]  |
|                                                                         |
| Downtime: Minimal (<10ms delta transfer), Completely Imperceptible     |
+-------------------------------------------------------------------------+

Stage 1: Base Pre-Copy

While the source instance continues its normal autoregressive decoding iterations, Llumnix initiates an asynchronous transfer of the bulk accumulated KV cache blocks. The transfer executes in a dedicated high-priority CUDA network stream via RDMA (Remote Direct Memory Access) or NCCL point-to-point communication. The source instance continues outputting tokens without interruption.

Stage 2: Iterative Delta Tracking

During the time required to complete Stage 1, the source instance executes several decoding steps, producing newly generated tokens. These new tokens append new KV blocks to the cache. Llumnix maintains a dirty block tracker that logs all newly allocated or modified pages during the transfer. In Stage 2, Llumnix transfers these delta blocks in rapid, smaller iterative rounds.

Stage 3: Stop-and-Switch Handshake

When the remaining volume of untransferred delta blocks falls below a configurable threshold (typically 1 or 2 blocks, corresponding to 16–32 tokens), the source instance briefly pauses decoding for that request. It transmits the final residual delta along with the request’s lightweight execution metadata (current sequence length, sampling state, logits history, and block mapping table).

Stage 4: Seamless Destination Resumption

The destination instance binds the received physical blocks into its local PagedAttention table and immediately dispatches the next decoding iteration. Because the final stop-and-switch step only transfers a minuscule delta, request downtime ($T_{downtime}$) is reduced to under 10 milliseconds—well below the perceptible inter-token delay threshold.

Mathematical Formulation: Migration Downtime and Convergence

To guarantee zero perceived interruption, the migration protocol must satisfy strict mathematical convergence criteria.

Let:

$S_0$ denote the initial KV cache size (in bytes) when migration begins.
$B_{net}$ denote the effective inter-node network transfer bandwidth (bytes per second).
$R_{gen}$ denote the KV cache generation rate during decoding (bytes per second).

For autoregressive decoding with batch size $M$, step execution time $T_{step}$, and KV cache footprint per token $S_{token}$:

$$R_{gen} = \frac{S_{token}}{T_{step}}$$

For a model with $N_{layers}$ layers, $N_{kv_heads}$ key-value heads, head dimension $d_{head}$, and precision bytes $P$ (e.g., $P=2$ for FP16):

$$S_{token} = 2 \times N_{layers} \times N_{kv_heads} \times d_{head} \times P$$

For LLaMA-3-70B ($N_{layers} = 80$, $N_{kv_heads} = 8$, $d_{head} = 128$, $P = 2$):

$$S_{token} = 2 \times 80 \times 8 \times 128 \times 2 = 327,680 \text{ bytes} \approx 320 \text{ KB/token}$$

If the model decodes at 25 ms per token ($T_{step} = 0.025\text{ s}$), the generation rate is:

$$R_{gen} = \frac{320 \times 10^3}{0.025} = 12.8 \text{ MB/s}$$

Now compare this generation rate to modern data-center interconnects:

PCIe Gen 5 x16: $B_{net} \approx 63 \text{ GB/s}$
200 Gbps InfiniBand (HDR): $B_{net} \approx 24 \text{ GB/s}$
400 Gbps InfiniBand (NDR): $B_{net} \approx 48 \text{ GB/s}$

Because $B_{net} \gg R_{gen}$ (the network bandwidth is more than 1,800x faster than the token generation rate), the pre-copy loop converges almost instantaneously.

In round $k$, the data remaining to transfer $S_k$ satisfies:

$$S_k = S_{k-1} \times \left( \frac{R_{gen}}{B_{net}} \right) = S_0 \times \left( \frac{R_{gen}}{B_{net}} \right)^k$$

Because $\frac{R_{gen}}{B_{net}} \< 0.001$, a single pre-copy round transfers more than 99.9% of the cache. The stop-and-switch downtime $T_{downtime}$ is simply:

$$T_{downtime} = \frac{S_{delta}}{B_{net}} + T_{handshake}$$

Where $S_{delta}$ represents at most one decoding step’s KV cache (320 KB), taking less than 0.01 milliseconds over 200 Gbps InfiniBand. Including TCP/RDMA control handshake latencies ($T_{handshake} \approx 2\text{—}5\text{ ms}$), total downtime remains under 6 ms.

Core Innovation 2: Cluster-Wide Rescheduling Policies

Live migration provides the operational primitive; Llumnix pairs this primitive with a centralized Global Manager that implements three complementary scheduling policies:

1. Proactive Cluster Load Balancing

The Global Manager monitors real-time KV cache memory utilization and queue lengths across all GPU instances. When an instance’s memory exceeds a high-water threshold (e.g., 85% capacity) while other instances have headroom, Llumnix selects candidate requests to migrate. Candidates are chosen based on minimal remaining migration time and maximum load reduction, equalizing memory and compute pressure across the cluster.

2. Physical Memory Defragmentation

Due to unpredictable request lengths and dynamic PagedAttention allocations, physical GPU memory can suffer from virtual page fragmentation. An instance may have 20% aggregate free memory, but lack large contiguous block ranges or sufficient capacity to accept a new 32k-context prefill request. Llumnix migrates long-running decode requests to consolidate free blocks, opening clean memory allocations for bursty incoming prefills.

3. SLO-Aware Priority Preemption

In enterprise environments with mixed-tier traffic (e.g., enterprise SLAs vs free-tier queries), a sudden surge of high-priority requests can be accommodated without dropping or aborting existing low-priority requests. Llumnix migrates low-priority decoding sessions to cold instances or secondary worker pools, freeing premium instances for immediate high-priority execution.

# Conceptual implementation of Llumnix Global Scheduling Logic
import time
from typing import List, Dict, Optional

class InstanceState:
    def __init__(self, instance_id: str, total_hbm_mb: float):
        self.instance_id = instance_id
        self.total_hbm_mb = total_hbm_mb
        self.used_hbm_mb = 0.0
        self.active_requests: List['RequestState'] = []

    @property
    def memory_pressure(self) -> float:
        return self.used_hbm_mb / self.total_hbm_mb

class RequestState:
    def __init__(self, req_id: str, context_len: int, priority: int = 1):
        self.req_id = req_id
        self.context_len = context_len
        self.priority = priority # Higher = more urgent
        self.token_size_bytes = 320 * 1024 # LLaMA-3-70B

    @property
    def kv_cache_mb(self) -> float:
        return (self.context_len * self.token_size_bytes) / (1024 * 1024)

class LlumnixGlobalScheduler:
    def __init__(self, instances: List[InstanceState], high_watermark: float = 0.85, low_watermark: float = 0.60):
        self.instances = instances
        self.high_watermark = high_watermark
        self.low_watermark = low_watermark

    def evaluate_migration_plan(self) -> List[Dict[str, str]]:
        migrations = []
        overloaded = [inst for inst in self.instances if inst.memory_pressure > self.high_watermark]
        underloaded = [inst for inst in self.instances if inst.memory_pressure < self.low_watermark]

        for src in overloaded:
            if not underloaded:
                break
            # Sort requests by size to find optimal migration candidate
            candidate = sorted(src.active_requests, key=lambda r: r.kv_cache_mb, reverse=True)[0]
            dst = underloaded[0]

            if dst.used_hbm_mb + candidate.kv_cache_mb < dst.total_hbm_mb * self.high_watermark:
                migrations.append({
                    "request_id": candidate.req_id,
                    "source": src.instance_id,
                    "destination": dst.instance_id,
                    "migrated_mb": candidate.kv_cache_mb
                })
                # Speculatively update loads
                src.used_hbm_mb -= candidate.kv_cache_mb
                dst.used_hbm_mb += candidate.kv_cache_mb
                if dst.memory_pressure >= self.low_watermark:
                    underloaded.pop(0)

        return migrations

Architectural Comparison: Llumnix vs Traditional LLM Serving Stacks

The following comparison illustrates how Llumnix fundamentally transforms cluster-level coordination compared to existing systems:

Architectural Metric	vLLM (Static Cluster)	Ray Serve + vLLM	DeepSpeed-FastGen	Llumnix (OSDI 2024)
Request Assignment	Static at Ingress	Static with Routing	Static at Ingress	Dynamic Live Rescheduling
In-Flight Migration	Unsupported (Abort/Restart)	Unsupported	Unsupported	Multi-Stage Pre-Copy RDMA
Migration Downtime	N/A (Inf/Recompute)	N/A	N/A	Sub-10ms Imperceptible
Memory Defragmentation	Local PagedAttention only	Local only	Chunked allocation	Cluster-Wide Global Rebalancing
SLO Violation Mitigation	Host Swapping / Preempt	Host Swapping / Preempt	Static Prioritization	Live Priority Handoff
P99 TTFT Tail Latency	Baseline (1.0x)	0.95x	0.88x	0.20x (80% Reduction)
P99 TBT Stutter Latency	High during OOM bursts	High during OOM bursts	Moderate	Smooth, Zero-Stall Delivery

By introducing live migration, Llumnix elevates LLM serving from independent worker islands into an elastic, cooperative cluster.

Step-by-Step Implementation and Deployment Guide

Deploying Llumnix in a production Kubernetes or Ray cluster involves establishing low-latency inter-node communication, configuring the Llumnix Manager daemon, and wrapping standard inference backends.

Step 1: System and Network Prerequisites

Ensure your cluster nodes feature high-speed networking with RDMA over Converged Ethernet (RoCE v2) or InfiniBand enabled. Verify inter-GPU peer communication using NCCL:

# Verify RDMA and InfiniBand status across nodes
ibv_devinfo -v
rdma link show

# Test inter-node GPU-to-GPU bandwidth with NCCL test suite
./build/all_reduce_perf -b 1G -e 8G -f 2 -g 8

Ensure bidirectional firewalls allow high-throughput TCP and RDMA ports between all worker nodes.

Step 2: Llumnix Cluster Configuration

Create the central cluster configuration file llumnix_config.yaml:

manager:
  host: '10.0.0.1'
  port: 8088
  scheduling_interval_ms: 10.0
  load_balance_strategy: 'min_load_first'
  high_watermark_ratio: 0.85
  low_watermark_ratio: 0.60

migration:
  protocol: 'rdma' # "rdma" or "tcp"
  chunk_size_pages: 512
  max_pre_copy_rounds: 3
  stop_threshold_blocks: 2
  timeout_seconds: 5.0

engine:
  model: 'meta-llama/Llama-3-70b-Instruct'
  tensor_parallel_size: 4
  pipeline_parallel_size: 1
  gpu_memory_utilization: 0.90
  max_model_len: 32768
  block_size: 16

Step 3: Launching the Llumnix Server Daemon

Initialize the Llumnix Global Manager and connect distributed worker nodes. Developers orchestrating workflows can hook into the Llumnix event bus, similar to integrating automated web-scraping jobs or api-integration webhooks:

import asyncio
from llumnix.entrypoints.setup import setup_llumnix_cluster
from llumnix.manager import LlumnixManager

async def start_cluster():
    print("[INFO] Initializing Llumnix Global Cluster Orchestrator...")
    manager = LlumnixManager.from_yaml("llumnix_config.yaml")
    await manager.start()

    print("[INFO] Connecting worker nodes to migration mesh...")
    cluster_handle = await setup_llumnix_cluster(manager)

    print(f"[INFO] Cluster online with {len(cluster_handle.workers)} active GPU instances.")
    await manager.wait_until_terminated()

if __name__ == "__main__":
    asyncio.run(start_cluster())

Step 4: Dispatching Requests via the Llumnix Ingress Proxy

Clients submit standard OpenAI-compatible requests to the Llumnix proxy. The proxy handles request dispatch and transparently coordinates with the manager:

import httpx

client = httpx.Client(base_url="http://10.0.0.1:8088/v1")

payload = {
    "model": "meta-llama/Llama-3-70b-Instruct",
    "prompt": "Synthesize the annual regulatory compliance report for financial services...",
    "max_tokens": 2048,
    "temperature": 0.1,
    "stream": True
}

with client.stream("POST", "/chat/completions", json=payload, timeout=120.0) as response:
    for chunk in response.iter_lines():
        if chunk:
            print(chunk)

If the instance servicing this stream becomes congested, Llumnix migrates the active generation mid-stream to an alternate instance without breaking the open HTTP streaming connection.

Production Benchmarks: Empirical Evaluation

The OSDI 2024 paper reports rigorous performance benchmarks across synthetic workloads, real-world ShareGPT datasets, and production traces from Alibaba Cloud’s internal AI platform.

1. P99 TTFT and TBT Latency Reduction Under Bursty Traffic

Under fluctuating traffic conditions with sudden Poisson arrival spikes:

vLLM Baseline: Severe head-of-line blocking when instances hit memory saturation. Tail TTFT escalates to 12.8 seconds as incoming queries wait for active decodes to finish.
Llumnix: Proactively migrates active requests away from saturated instances. Saturated instances immediately accept new prefill jobs. P99 TTFT drops by up to 80%, maintaining predictable response times across all percentiles.

P99 Time-to-First-Token (TTFT) Under Varying Request Loads

15s |------------------------------------------------------------------ (vLLM Baseline: 12.8s)
    |                                                                //
10s |---------------------------------------------------------------//-
    |                                                             //
 5s |------------------------------------------------------------//----
    |                                                          //
 2s |=========================================================//======= (Llumnix: 1.8s)
    |__________________________________________________________________
    0.5 req/s          1.0 req/s          2.0 req/s          3.5 req/s

2. Tail Latency and SLO Attainment Rate

In enterprise serving with strict latency SLAs (e.g., TTFT $\< 2.0\text{ s}$ and TBT $\< 40\text{ ms}$):

Serving System	Arrival Rate (req/s)	P99 TTFT (ms)	P99 TBT (ms)	SLO Attainment Rate	Cluster Goodput (tokens/s)
vLLM (Static)	2.5	9,420 ms	68.2 ms	56.4%	1,420 tok/s
Ray Serve + vLLM	2.5	8,850 ms	64.1 ms	61.2%	1,510 tok/s
Llumnix (OSDI 2024)	2.5	1,840 ms	28.4 ms	94.8% (+68.1%)	2,480 tok/s (+74.6%)

By eliminating idle resources and avoiding head-of-line stalls, Llumnix achieves a 94.8% SLO attainment rate and boosts effective cluster goodput by 74.6%.

3. Migration Overhead and Downtime Measurements

Evaluating the migration mechanism across different context lengths confirms its lightweight performance:

Context Length (Tokens)	Total KV Cache Size	Pre-copy Duration (RDMA)	Stop-and-Switch Downtime	Token Generation Stutter
4,096 tokens	1.31 GB	38.4 ms	3.8 ms	Imperceptible
16,384 tokens	5.24 GB	148.2 ms	5.2 ms	Imperceptible
32,768 tokens	10.48 GB	292.6 ms	6.4 ms	Imperceptible

Even for massive 32k-context sequences, the client experiences only a 6.4 ms pause—well within the standard 25 ms inter-token decoding budget.

Operational Checklist: Deploying Llumnix in Enterprise Production

Before launching Llumnix across multi-tenant production clusters, infrastructure teams should verify operational readiness against this deployment checklist:

RDMA / RoCE Network Validation: Verify that GID indices, PFC (Priority Flow Control), and ECN (Explicit Congestion Notification) are configured to ensure lossless packet transmission across the GPU cluster.
Memory High-Water Calibration: Set high_watermark_ratio between 0.82 and 0.88. Setting it too high delays migration triggers, while setting it too low induces unnecessary migrations.
Dirty Page Threshold Tuning: Configure stop_threshold_blocks to 2 or 4 blocks to minimize stop-and-switch downtime while bounding the maximum number of pre-copy rounds.
Health Checks and Heartbeats: Establish low-latency gossip protocols between worker daemons and the Llumnix Manager to detect failed instances within 50 ms.
Prometheus Metrics Pipeline: Instrument dashboards monitoring llumnix_active_migrations_total, llumnix_migration_downtime_ms, and llumnix_instance_memory_skew_ratio.
Capacity and Cost Planning: Review cluster architecture in your internal docs and align scaling thresholds with enterprise infrastructure pricing.

Strategic Outlook: The Era of Elastic Serverless LLM Fabrics

The introduction of live migration in Llumnix represents a fundamental paradigm shift for AI infrastructure. In the early era of generative AI, serving systems treated GPUs as isolated, monolithic execution silos. Llumnix demonstrates that by treating the KV cache as a fluid, migratable distributed state, entire clusters can operate as a unified, elastic compute fabric.

As models scale toward multimodal reasoning and agentic loops requiring persistent working memory across minutes or hours, static request binding will become completely unviable. Architectural techniques pioneered by Llumnix—such as iterative pre-copy streaming, dirty block tracking, and global memory defragmentation—will serve as the foundational bedrock for the next generation of serverless cloud AI platforms.

Explore more technical deep dives, performance benchmarks, and architectural breakdowns on our homepage as we continue tracking the frontier of systems research for artificial intelligence.