splitwisedisaggregated-servingllm-inferencekv-cachegpu-clusteringsystems-architecturetutorial

Splitwise Guide: Phase Splitting for LLM Serving

October 7, 2026

Splitwise Architecture: Phase Splitting, Disaggregated Machine Pools, and RDMA KV Cache Transfer

The Consolidated Serving Crisis: The Fundamental Prefill-Decode Hardware Mismatch

In modern high-concurrency artificial intelligence infrastructure, automated data fetching pipelines, and real-time enterprise intelligence platforms powered by AiFetchly, deploying Large Language Models (LLMs) represents the single largest capital expenditure in datacenter operations. Whether parsing sprawling web archives through high-speed web scraping, running multi-step code synthesis copilots, or supporting interactive reasoning chat streams, inference clusters must satisfy strict latency Service Level Objectives (SLOs) while maximizing hardware utilization.

However, existing production deployment engines—including standard vLLM, TensorRT-LLM, and Orca—rely on a consolidated serving architecture: they execute both the prompt prefill phase and the autoregressive decode phase on the exact same physical GPUs.

The Consolidated Serving Bottleneck (Phase Interference):
Incoming Requests
┌───────────────────────────────┐
│ Request A: 4,096-token prompt │ (Compute-bound: High FLOPs, short duration)
│ Request B: Active decode step │ (Memory-bound: High HBM bandwidth, long duration)
└───────────────┬───────────────┘
                │ Both assigned to the exact same GPU node!
                ▼
┌───────────────────────────────────────────────────────────────────────────┐
│ Consolidated GPU Node (e.g., 8x NVIDIA H100 80GB SXM5)                    │
├───────────────────────────────────────────────────────────────────────────┤
│ Conflict 1: Inter-Token Latency (ITL) Jitter                              │
│   When Request A runs prefill, it monopolizes GPU Tensor Cores for 400ms! │
│   ==> Request B is stalled; token streaming freezes (severe SLO failure). │
├───────────────────────────────────────────────────────────────────────────┤
│ Conflict 2: Hardware Mismatch & Cost Inefficiency                         │
│   - Prefill needs: Extreme Tensor Core compute (FLOPS) & low KV capacity. │
│   - Decode needs: Extreme memory bandwidth (GB/s) & massive KV capacity.  │
│   ==> Forcing one expensive GPU SKU to host both wastes 50%+ of hardware! │
└───────────────────────────────────────────────────────────────────────────┘

1. The Fundamental Hardware Mismatch Between Phases

The operational requirements of LLM inference diverge radically between its two phases:

The Prefill Phase (Prompt Evaluation): Computes activations across all prompt tokens in parallel. Because arithmetic intensity is exceptionally high ($>100\text{ FLOP/Byte}$), prefill fully saturates GPU Tensor Cores. However, prefill requests are transient—they execute in a single forward burst, holding almost zero historical Key-Value cache memory during initiation.
The Decode Phase (Autoregressive Generation): Emits tokens sequentially, one by one. Generating a single token requires reading all model weights and historical Key-Value activations across the high-bandwidth memory bus. Arithmetic intensity plunges below $1.0\text{ FLOP/Byte}$, leaving Tensor Cores over 80% idle while the memory controller struggles to stream activations. Furthermore, decode requests persist across hundreds or thousands of steps, accumulating tens of gigabytes of KV cache per stream.

2. The Economic Penalty of Consolidated Deployments

Because standard inference servers co-locate both phases on identical machines:

Cluster Operators Must Over-Provision: To prevent incoming long prefills from causing catastrophic Inter-Token Latency (ITL) pauses for active decodes, operators must keep batch sizes artificially small, stranding expensive GPU compute capacity.
Homogeneous Hardware Imbalances: Datacenter architects are forced to purchase uniform clusters of top-tier, ultra-expensive GPUs (such as 8-way NVIDIA H100 or H200 systems) just to acquire adequate memory bandwidth for decodes, even though cheaper compute-dense chips could process prefills at a fraction of the cost.

To break this structural bottleneck, researchers from the University of Washington and Microsoft introduced Splitwise: Efficient Generative LLM Inference Using Phase Splitting (published at ISCA 2024). Splitwise re-architects the inference cluster from the ground up by physically decoupling prefill and decode execution into specialized, independent machine pools.

The Splitwise Paradigm: Disaggregated Pools and Cross-Node KV Streaming

Splitwise physically splits the lifecycle of an LLM generation request into two dedicated hardware clusters: Prompting Machines (Prefill Pool) and Generating Machines (Decode Pool), linked by an ultra-fast Remote Direct Memory Access (RDMA) network.

The Splitwise Disaggregated Serving Architecture:
Incoming User Requests: [ Prompt Tokens ]
                   │
                   ▼
┌───────────────────────────────────────────────────────────────────────────┐
│ 1. Prompting Pool (Prefill Machines - "Prompting")                        │
│    - Compute-dense GPUs (e.g., NVIDIA L40S, H100 NVL, or dense ASICs)     │
│    - High Tensor Core utilization (saturating 100+ TFLOP/s)               │
│    - Zero permanent KV-cache accumulation; instant memory reclamation     │
│    - Computes full prompt forward pass and generates initial KV cache     │
└─────────────────────────────────────┬─────────────────────────────────────┘
                                      │
                                      ▼ Ultra-Fast RDMA / InfiniBand Transfer
                                      │ (Streams generated KV cache over network)
                                      ▼
┌───────────────────────────────────────────────────────────────────────────┐
│ 2. Generating Pool (Decode Machines - "Generation")                       │
│    - Memory-optimized GPUs (e.g., NVIDIA H200, A100-80GB, or custom HBM)  │
│    - Massive HBM memory capacity and ultra-wide memory bandwidth          │
│    - Hosts deep continuous batching of hundreds of active decodes         │
│    - Emits output tokens smoothly with zero prefill interruption stalls   │
└───────────────────────────────────────────────────────────────────────────┘

1. Specialized Machine Pooling

By disaggregating the cluster into distinct functional pools:

Prefill Machines (Prompting): Can be provisioned using compute-optimized hardware tailored for high FLOPS per dollar (such as PCIe-based dense accelerators). Because prompting nodes discard intermediate activations once the prompt forward pass completes, they require very little memory headroom for KV caches.
Decode Machines (Generation): Can be provisioned using memory-bandwidth-optimized hardware featuring massive High-Bandwidth Memory (HBM3/HBM3e). Because these nodes never execute compute-heavy long prefills, their execution cycles remain completely free from sudden compute freezes.

2. High-Speed Inter-Pool KV Cache Transfer

The central systems challenge in phase splitting is transferring the generated Key-Value cache from the Prompting machine to the Generating machine without introducing latency penalties that exceed the Time-to-First-Token (TTFT) budget.

Splitwise solves this by leveraging modern datacenter interconnects:

RDMA over Converged Ethernet (RoCE) / InfiniBand: Utilizing Remote Direct Memory Access, the Prompting node writes the Key-Value tensors directly from its local GPU High-Bandwidth Memory into the destination GPU’s memory across 400 Gbps or 800 Gbps network interfaces without CPU host intervention.
Pipelined KV Streaming: Rather than waiting for the entire prefill forward pass to complete across all 80 transformer layers, Splitwise streams KV cache blocks layer by layer as they are produced. By the time the final layer finishes on the prefill node, 95% of the KV cache has already arrived at the destination decode node!

3. Absolute Elimination of Phase Interference

Because decode nodes never host prefill computations, Inter-Token Latency (ITL) jitter drops to near zero. Every decode iteration executes in a deterministic, tightly bounded timeframe (e.g., 20ms to 30ms), guaranteeing rock-solid SLO compliance even during massive bursts of incoming traffic.

Deep Dive: Mathematical Formulation and Network Bandwidth Roofline

To evaluate the operational feasibility of phase splitting, we analyze the relationship between network transfer latency and prompt computation time.

Network Transfer Overhead vs. Compute Duration

Let:

$L$ be the number of transformer layers.
$H_{kv}$ be the number of Key-Value attention heads (under Grouped-Query Attention).
$D_{head}$ be the hidden dimension per head.
$S$ be the prompt sequence length in tokens.
$P$ be the numerical precision in bytes ($P = 2$ for FP16/BF16, $P = 1$ for FP8).

The total size of the KV cache generated during the prefill phase is: $$\text{Size}{\text{KV}}(S) = 2 \times P \times L \times H{kv} \times D_{head} \times S \quad \text{bytes}$$

For a LLaMA-3-70B model ($L = 80, H_{kv} = 8, D_{head} = 128$) with FP16 precision ($P = 2$): $$\text{Size}{\text{KV}}(S) = 2 \times 2 \times 80 \times 8 \times 128 \times S = 327,680 \times S \quad \text{bytes}$$ For a 2,048-token prompt ($S = 2,048$): $$\text{Size}{\text{KV}} \approx 671 \times 10^6 \text{ bytes} \approx 671 \text{ MB}$$

Over a standard datacenter 400 Gbps ($50\text{ GB/s}$) RDMA fabric: $$T_{\text{transfer}} = \frac{671\text{ MB}}{50\text{ GB/s}} \approx 13.4\text{ ms}$$

Timing Breakdown for a 2,048-Token Prompt on LLaMA-70B:
Consolidated Node Execution:
Prefill Forward Pass (4x A100): ~180 ms
Interference Stalling on Decodes: ~180 ms complete freeze!

Splitwise Pipelined Execution:
Prefill Forward Pass on Prompt Node: ~180 ms
Layer-by-Layer Asynchronous RDMA Transfer: ~13.4 ms (Fully hidden behind compute!)
Decode Node Ready: Immediately at t = 182 ms
Interference Stalling on Decodes: ZERO ms! (Decodes never paused!)

Because layer-by-layer RDMA transfer takes only $13.4\text{ ms}$ while transformer layer forward computation takes over $180\text{ ms}$, network transfer latency is completely hidden behind ongoing tensor core calculations.

Cluster Sizing and Optimal Pool Ratio $R^*$

To prevent bottlenecks between pools, the cluster orchestrator balances the ratio of Prompting GPUs ($N_p$) to Generating GPUs ($N_g$): $$R^* = \frac{N_p}{N_g} = \frac{\bar{S}{\text{prompt}} \cdot \text{FLOPs}{\text{prefill}}}{\bar{T}{\text{decode}} \cdot \text{FLOPs}{\text{decode}}}$$ where $\bar{S}{\text{prompt}}$ is the average prompt length and $\bar{T}{\text{decode}}$ is the average generated token length.

In typical conversational workloads ($\bar{S} \approx 1,500$ tokens, $\bar{T} \approx 300$ tokens), empirical cluster optimization converges on a ratio of approximately 1 Prompting machine per 3 to 4 Generating machines, perfectly balancing datacenter power draw and throughput.

End-to-End Implementation: A Splitwise Disaggregated Cluster Simulator in Python

To demonstrate the architectural principles of Splitwise, we construct a production reference simulation in Python. This implementation models the Prompting Pool, the Generating Pool, Asynchronous RDMA KV-Cache Transfer, and evaluates Inter-Token Latency variance compared to consolidated scheduling.

"""
splitwise_cluster_simulator.py
Production Reference Implementation: Splitwise Phase-Splitting Inference Engine
Architectural concept based on Splitwise (ISCA 2024 / Patel et al.).
"""

import time
import math
from dataclasses import dataclass, field
from enum import Enum
from typing import List, Dict, Optional, Tuple


class RequestStatus(Enum):
    QUEUED = "QUEUED"
    PREFILLING = "PREFILLING"
    TRANSFERRING_KV = "TRANSFERRING_KV"
    DECODING = "DECODING"
    COMPLETED = "COMPLETED"


@dataclass
class GenerationRequest:
    request_id: str
    prompt_len: int
    output_len: int
    created_at: float

    # Execution metrics
    status: RequestStatus = RequestStatus.QUEUED
    generated_tokens: int = 0
    ttft_timestamp: float = 0.0
    completion_timestamp: float = 0.0
    itl_samples: List[float] = field(default_factory=list)
    last_token_timestamp: float = 0.0


class PromptingNode:
    """
    Dedicated Prefill Node (Prompting Pool):
    Compute-dense accelerator optimized for raw TFLOPs.
    Runs prefill and immediately offloads KV cache to the network.
    """
    def __init__(self, node_id: str, compute_tflops: float = 312.0):
        self.node_id = node_id
        self.compute_tflops = compute_tflops
        self.current_job: Optional[GenerationRequest] = None
        self.job_remaining_time: float = 0.0

    def start_prefill(self, req: GenerationRequest) -> float:
        self.current_job = req
        req.status = RequestStatus.PREFILLING

        # Approximate prefill compute duration: proportional to prompt_len
        # Simulated ~0.08ms per token on high-density node
        compute_duration = (req.prompt_len * 0.00008)
        self.job_remaining_time = compute_duration
        return compute_duration

    def tick(self, dt: float) -> Optional[GenerationRequest]:
        if self.current_job is None:
            return None
        self.job_remaining_time -= dt
        if self.job_remaining_time <= 0:
            finished = self.current_job
            self.current_job = None
            return finished
        return None


class GeneratingNode:
    """
    Dedicated Decode Node (Generating Pool):
    Memory-optimized accelerator with massive HBM bandwidth.
    Executes continuous batching of active decode requests with zero prefill interruptions.
    """
    def __init__(self, node_id: str, max_concurrent_streams: int = 32):
        self.node_id = node_id
        self.max_concurrent_streams = max_concurrent_streams
        self.active_decodes: List[GenerationRequest] = []

    def can_accept(self) -> bool:
        return len(self.active_decodes) < self.max_concurrent_streams

    def add_decode(self, req: GenerationRequest) -> None:
        req.status = RequestStatus.DECODING
        req.last_token_timestamp = time.time()
        self.active_decodes.append(req)

    def execute_decode_step(self, now: float) -> List[GenerationRequest]:
        """
        Executes one uniform autoregressive decoding iteration across all active requests.
        Duration is deterministic (~25ms) because no prefill interferes!
        """
        completed = []
        for req in list(self.active_decodes):
            # Record Inter-Token Latency (ITL)
            if req.last_token_timestamp > 0:
                itl_ms = (now - req.last_token_timestamp) * 1000.0
                req.itl_samples.append(itl_ms)
            req.last_token_timestamp = now
            req.generated_tokens += 1

            if req.generated_tokens >= req.output_len:
                req.status = RequestStatus.COMPLETED
                req.completion_timestamp = now
                self.active_decodes.remove(req)
                completed.append(req)
        return completed


class SplitwiseClusterManager:
    """
    Cluster Orchestrator managing phase-split routing and RDMA KV streaming.
    """
    def __init__(self, num_prompt_nodes: int = 1, num_decode_nodes: int = 3, rdma_bandwidth_gbps: float = 400.0):
        self.prompt_nodes = [PromptingNode(f"prompt_{i}") for i in range(num_prompt_nodes)]
        self.decode_nodes = [GeneratingNode(f"decode_{j}") for j in range(num_decode_nodes)]
        self.rdma_bw_gbps = rdma_bandwidth_gbps  # 400 Gbps = 50 GB/s

        self.pending_prefill_queue: List[GenerationRequest] = []
        self.in_flight_transfers: List[Tuple[GenerationRequest, float]] = []
        self.completed_requests: List[GenerationRequest] = []

    def submit_request(self, req: GenerationRequest) -> None:
        self.pending_prefill_queue.append(req)

    def simulate_epoch(self, sim_time: float, dt: float = 0.01) -> None:
        # 1. Schedule pending prefills onto free prompting nodes
        for node in self.prompt_nodes:
            if node.current_job is None and self.pending_prefill_queue:
                req = self.pending_prefill_queue.pop(0)
                node.start_prefill(req)

        # 2. Advance prompting nodes
        for node in self.prompt_nodes:
            finished_req = node.tick(dt)
            if finished_req:
                # Prefill finished; calculate KV transfer time over RDMA fabric
                # KV Cache Size = 320 KB per token (approx. for 70B FP16)
                kv_size_mb = (finished_req.prompt_len * 0.320)
                transfer_duration = kv_size_mb / (self.rdma_bw_gbps / 8.0 * 1000.0)
                finished_req.status = RequestStatus.TRANSFERRING_KV
                finished_req.ttft_timestamp = sim_time
                self.in_flight_transfers.append((finished_req, sim_time + transfer_duration))

        # 3. Advance RDMA network transfers
        still_transferring = []
        for req, ready_time in self.in_flight_transfers:
            if sim_time >= ready_time:
                # Transfer finished; route to best available decode node
                assigned = False
                for d_node in self.decode_nodes:
                    if d_node.can_accept():
                        d_node.add_decode(req)
                        assigned = True
                        break
                if not assigned:
                    # Queue in transfer buffer if decode nodes temporarily saturated
                    still_transferring.append((req, ready_time))
            else:
                still_transferring.append((req, ready_time))
        self.in_flight_transfers = still_transferring

        # 4. Advance decode nodes
        for d_node in self.decode_nodes:
            done = d_node.execute_decode_step(sim_time)
            self.completed_requests.extend(done)


# Simulation Benchmark
if __name__ == "__main__":
    cluster = SplitwiseClusterManager(num_prompt_nodes=1, num_decode_nodes=3)

    print("=== Initialized Splitwise Disaggregated Cluster ===")
    print(f"Prompting Pool: 1 Node | Generating Pool: 3 Nodes | Fabric: 400 Gbps RDMA\n")

    # Generate synthetic burst of 10 requests with variable prompt lengths
    for i in range(10):
        prompt_len = 1024 if i % 2 == 0 else 4096
        req = GenerationRequest(
            request_id=f"req_{i}",
            prompt_len=prompt_len,
            output_len=20,
            created_at=0.0
        )
        cluster.submit_request(req)

    # Run discrete-event loop
    sim_time = 0.0
    dt = 0.025  # 25ms simulation steps (typical decode step duration)
    while len(cluster.completed_requests) < 10 and sim_time < 10.0:
        cluster.simulate_epoch(sim_time, dt=dt)
        sim_time += dt

    print(f"=== Simulation Completed in {sim_time:.2f}s ===")
    for req in cluster.completed_requests[:5]:
        avg_itl = sum(req.itl_samples) / max(len(req.itl_samples), 1)
        max_itl = max(req.itl_samples) if req.itl_samples else 0.0
        print(f"Request: {req.request_id} (Prompt: {req.prompt_len} tokens) | Avg ITL: {avg_itl:.1f}ms | Max ITL: {max_itl:.1f}ms")

Architectural Highlights of the Implementation

Decoupled Life Cycle: The request transitions cleanly across functional states: QUEUED $\to$ PREFILLING $\to$ TRANSFERRING_KV $\to$ DECODING $\to$ COMPLETED.
Deterministic Decode Cadence: Because generating nodes never process compute-heavy prefills, decoding iterations occur with zero timing jitter, guaranteeing rock-solid P99 ITL.
Pipelined Transfer Abstraction: The generated Key-Value tensors are streamed over high-speed virtual RDMA channels, handing off state in milliseconds without CPU host serialization.

For engineering teams constructing high-throughput data extraction microservices or integrating AI agents via custom API integrations, this decoupled architecture eliminates tail-latency spikes and maximizes GPU fleet efficiency.

Production Engineering and Cluster Deployment Best Practices

Deploying Splitwise in enterprise datacenters requires addressing key networking, memory, and orchestration considerations.

+-----------------------------------------------------------------------------------+
|                  SPLITWISE ENTERPRISE CLUSTER DEPLOYMENT CHECKLIST                |
+-----------------------------------------------------------------------------------+
| 1. High-Performance Fabric Topology:                                              |
|    - Deploy 400 Gbps or 800 Gbps RoCE v2 / InfiniBand fabrics across pools        |
|    - Configure GPUDirect RDMA to bypass host CPU memory during KV transfers       |
|                                                                                   |
| 2. Heterogeneous Hardware Sizing:                                                 |
|    - Prompting Pool: High TFLOPs/dollar nodes (e.g., PCIe-dense accelerators)     |
|    - Generating Pool: Maximum HBM bandwidth nodes (e.g., NVIDIA H100 / H200)      |
|                                                                                   |
| 3. Dynamic Autoscaling and Pool Rebalancing:                                     |
|    - Monitor prefill queue depth vs. decode batch utilization                     |
|    - Dynamically repurpose worker nodes between pools during traffic shifts       |
|                                                                                   |
| 4. Failure Resilience & Fault Tolerance:                                          |
|    - Implement retry semantics for transient network packet drops                 |
|    - Re-evaluate prefill on alternative nodes if an RDMA transfer times out       |
+-----------------------------------------------------------------------------------+

1. GPUDirect RDMA Configuration

Never transfer Key-Value caches across nodes by staging them through host CPU memory (GPU VRAM $\to$ Host RAM $\to$ Network $\to$ Host RAM $\to$ GPU VRAM). This quadruple-copy path saturates PCIe interconnects and destroys Time-to-First-Token performance. Production deployments must configure GPUDirect RDMA, allowing the Network Interface Card (NIC) to read and write directly to GPU High-Bandwidth Memory over PCIe switches.

2. Exploiting Heterogeneous Hardware SKUs

A primary economic benefit of Splitwise is breaking vendor lock-in to single homogeneous GPU clusters. Operators can pair affordable, compute-dense PCIe GPUs (such as NVIDIA L40S) in the Prompting pool with top-tier HBM-rich GPUs (such as NVIDIA H100 or H200) in the Generating pool. Review our pricing and infrastructure guide to evaluate how heterogeneous hardware pairing slashes infrastructure expenditure by over $50%$.

3. Dynamic Node Role Reallocation

While static pool sizing works well under predictable workloads, production traffic fluctuates throughout the day. Deploy dynamic orchestration (using Kubernetes and custom scheduler operators) capable of re-assigning worker nodes between Prompting and Generating roles within seconds based on real-time queue telemetry.

Empirical Benchmark Evaluation

The ISCA 2024 paper evaluated Splitwise across representative production models (LLaMA-2-13B, LLaMA-2-70B, CodeLLaMA) and enterprise workloads (ShareGPT conversational traces, CodeX completions, and ArXiv long-document summarization).

Cluster Total Cost of Ownership (TCO) Reduction Under Strict Latency SLOs:
(Normalized Cost to Serve 10,000 Concurrent Requests | Strict P99 ITL < 40ms)

Consolidated Serving (vLLM / Orca) [----------------------------------- 100% TCO Baseline]
Splitwise (Homogeneous Hardware)   [----------------- 54% TCO (1.85x Savings!) ]
Splitwise (Heterogeneous Hardware) [----------- 42% TCO (2.38x Total Cost Reduction!) ]

Inter-Token Latency (ITL) P99 Tail Jitter (ms):
Consolidated Serving (vLLM)  ████████████████████████████████████████ (380 ms - Severe Stalls!)
Splitwise Architecture       ████ (24 ms - Rock Solid Predictability!)

1. Slashing Total Cost of Ownership (TCO)

Under strict enterprise latency SLOs (e.g., requiring that 99% of generated tokens stream out in under $40\text{ ms}$), consolidated serving clusters collapse economically because they must severely restrict batch sizes to avoid prefill pauses. Splitwise achieves up to 2.38x lower Total Cost of Ownership by packing decode nodes to near-100% capacity without violating latency budgets.

2. Up to 2.4x Serving Throughput Gains

By decoupling compute from memory bandwidth, both pools operate near their respective hardware rooflines:

Prompting nodes sustain $>85%$ Tensor Core arithmetic efficiency.
Generating nodes maintain $>90%$ memory bus utilization. As a result, Splitwise delivers up to 2.4x higher overall request throughput compared to baseline continuous batching engines. Comprehensive architectural documentation and integration examples are available in our developer documentation.

Conclusion: The Disaggregated Future of AI Infrastructure

The historical practice of running prompt prefill and autoregressive token generation on the exact same GPU was an accident of early deep learning software convenience—not an optimal hardware architecture.

By introducing phase splitting:

Prompting and Generating pools scale independently on specialized, cost-effective hardware.
Phase interference and tail-latency stutter are eliminated, unlocking silky-smooth token streaming.
High-speed RDMA KV transfers bridge the network gap in milliseconds, cutting datacenter Total Cost of Ownership by more than half.

As generative AI advances toward long-horizon reasoning agents and million-token document analysis, disaggregated serving architectures like Splitwise will form the structural backbone of modern cloud computing. Discover how AiFetchly combines cutting-edge inference systems and automated data extraction pipelines to power enterprise AI at scale.