Prism Guide: Multi-LLM Serving via GPU Memory Ballooning

Operating modern generative artificial intelligence infrastructure has become an exercise in economic survival. Cloud inference providers—such as Hyperbolic, Novita AI, Alibaba Cloud, and Hugging Face—must maintain continuous online availability for hundreds of heterogeneous foundation models and specialized fine-tuned checkpoints. Simultaneously, token pricing has plunged rapidly, turning what was once a high-margin service into a razor-thin operational business.

In production environments, the overwhelming operational expense does not stem from exotic hardware or optical interconnects, but from chronic GPU under-utilization. To strictly honor customer Service Level Objectives (SLOs)—specifically Time-to-First-Token (TTFT) and Time-Per-Output-Token (TPOT)—cloud providers routinely provision dedicated GPU worker groups for individual models. For high-volume base models like LLaMA-3.3-70B, dedicated instances achieve respectable duty cycles. However, the vast majority of deployed models represent specialized or long-tail endpoints: domain-adapted reasoning agents, code verification tools, multilingual translators, and task-specific adapters.

Empirical analyses of multi-month production traces across major inference providers demonstrate that real-world model duty cycles frequently collapse below 30%. Long-tail models sit idle for more than 70% of the day. Yet, cloud providers cannot simply shut these instances down without inducing catastrophic multi-second cold-start delays that violate enterprise latency guarantees.

Presented at the prestigious 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2026) by a joint research consortium from UCLA, UC Berkeley, Harvard, CMU, Stanford, Rice University, ByteDance, and Alibaba Cloud, Prism introduces a transformative architectural paradigm: GPU Memory Ballooning for Multi-LLM Serving. By taking inspiration from virtual machine hypervisor memory management and introducing the open-source kvcached balloon driver, Prism unifies spatial and temporal GPU sharing into an elastic, demand-aware co-serving substrate. Deployed across clusters exceeding 10,000 GPUs, Prism achieves up to 3.3x higher TTFT SLO attainment, 2x higher TPOT SLO attainment, and over 2x infrastructure cost reduction.

In this architectural guide, we dissect the systemic failure modes of conventional GPU sharing, unpack the virtual memory mechanics of kvcached, step through production deployment on SGLang, evaluate performance against SOTA systems, and explore cluster-scale scheduling strategies for enterprise AI infrastructure.

Problem: The Multi-LLM Memory Wall and Spatial-Temporal Dilemma

To understand why multi-LLM serving requires a clean-sheet memory management architecture, systems engineers must examine the structural realities of multi-tenant model workloads and the fatal limitations of existing GPU sharing techniques.

Production Trace Analysis: Shifting Bursty Groups and Request Volatility

Prior academic research on LLM serving predominantly evaluated single-model systems under synthetic, idealized Poisson or Gamma arrival distributions. To ground systems design in empirical reality, the authors of Prism analyzed multi-tenant production traces from four major cloud environments: Hyperbolic (24 models, 4 months), Novita AI (16 models, 1 month), Chatbot Arena Battle (129 models, 16 months), and Chatbot Arena Chat (84 models, 11 days), spanning models from 1B to 70B parameters.

This workload analysis revealed two defining operational patterns:

1. Vertical Bursty Groups: At any single point in time, only a transient subset of deployed models—a “bursty group”—is actively processing requests. Across the evaluated traces, between 50% and 77% of provisioned models are completely idle. Yet, this active subset is not static; it changes between 54 and 766 times per hour as compound AI workflows, human user sessions, and automated agent loops activate different tools.
2. Horizontal Activation Heterogeneity: Large general-purpose planner models sustain continuous activity, while small, specialized sub-models (such as mathematical verifiers or text formatters) activate sporadically.

The Breakdown of Pure Space Sharing

Faced with this workload profile, infrastructure operators typically evaluate two standard GPU-sharing regimes: Space Sharing (colocating multiple models concurrently on the same physical GPU) or Time Sharing (swapping models in and out of GPU High-Bandwidth Memory). Both approaches fail under production traffic:

Under pure space sharing (such as static memory partitioning via NVIDIA MIG or MuxServe), the serving framework statically carves physical GPU memory between colocated models. While this prevents swapping overhead during steady-state execution, it creates artificial resource scarcity.

In production traces where models are idle for over 70% of the time, static memory partitioning leaves more than 50% of GPU High-Bandwidth Memory (HBM) completely unused. When an active model experiences a sudden traffic burst, its statically allocated Key-Value (KV) cache pool exhausts immediately. Incoming requests must queue or drop, causing massive TTFT spikes—even though vast gigabytes of reserved KV buffer sit idle inside the adjacent inactive model’s partition!

The Breakdown of Pure Time Sharing

Conversely, pure time sharing (such as QLM or ServerlessLLM) unloads inactive models to host CPU DRAM, loading weights only when requests arrive.

While this eliminates memory fragmentation during prolonged idle periods, it suffers from catastrophic model thrashing during volatile, interleaved traffic. When user interactions alternate rapidly between two or more models, the serving cluster spends more wall-clock time executing PCIe weight transfers and engine re-initializations than computing inference tokens. Queues explode, and Time-to-First-Token latencies climb into multi-second delays that violate enterprise SLAs.

High-throughput enterprise workloads—whether orchestrating complex multi-agent reasoning chains or executing distributed web scraping workflows—demand an infrastructure layer that bridges this divide. The industry required an elastic systems mechanism capable of fluidly shifting between spatial and temporal sharing at runtime.

Solution: GPU Memory Ballooning and Elastic Virtual Paging

To resolve the spatial-temporal sharing dilemma, the architects of Prism established a foundational insight: GPU memory is the single unifying bottleneck in multi-LLM serving, and spatial vs. temporal sharing merely govern different allocations of that exact same physical memory.

The Concept of GPU Memory Ballooning

In operating systems and hypervisor virtualization (such as VMware ESXi or Linux KVM), memory ballooning is a classic technique used to dynamically adjust the memory allocation of guest virtual machines without restarting them. A pseudo-device driver (the balloon driver) inside the guest VM allocates memory from the guest OS when instructed by the hypervisor, pinning those physical pages so the hypervisor can safely reassign them to another VM under memory pressure. When the guest VM requires memory again, the balloon deflates, releasing pages back to the guest.

Prism translates this elegant operating system abstraction directly into GPU silicon for multi-LLM co-serving:

1. The LLM Balloon Driver (kvcached): An elastic memory management layer inserted between the LLM serving runtime (e.g., SGLang) and the low-level CUDA driver.
2. Decoupled Virtual and Physical GPU Addressing: Traditional LLM engines statically allocate their entire KV-cache tensor pool at boot time (e.g., allocating 40GB of contiguous physical memory). In contrast, Prism reserves a massive virtual address space during engine initialization, but allocates and maps physical GPU memory pages on-demand at a fine-grained 2 Megabyte granularity.
3. Transparent Inflation and Deflation: When a colocated model sits idle or processes low-token streams, kvcached inflates the balloon, unmapping physical 2MB pages from its virtual KV tensor and returning them to a global GPU physical memory pool. When an active model faces an intense burst, the balloon deflates, dynamically binding physical memory pages into its virtual address space in milliseconds—with zero changes to PagedAttention CUDA kernels.

Four Structural Architectural Invariants

To make GPU memory ballooning practical for production serving, Prism enforces four strict system invariants:

• R1: Fast Reallocation Between Weights and KV-Cache: Released model weight memory can be repurposed immediately as KV-cache blocks for running models within sub-second timescales.
• R2: Universal Architecture Compatibility: Different foundation models possess radically different layer counts ($L$), attention heads ($H$), and head dimensions ($D$). Prism operates beneath tensor layouts at raw 2MB physical page boundaries, making memory fungible across diverse model families (e.g., LLaMA, Qwen, DeepSeek, Mistral).
• R3: Minimal Fragmentation: By standardizing allocations to uniform 2MB physical tiles, Prism completely avoids external memory fragmentation and eliminates complex secondary indexing layers.
• R4: Zero Kernel Disruption: Existing high-performance FlashAttention and PagedAttention kernels expect contiguous virtual pointers. By manipulating the virtual-to-physical page table via CUDA Virtual Memory Management (VMM) APIs, the underlying attention kernels execute without modification.

For applications requiring continuous, high-volume data streams—such as live real-time data fetching pipelines—Prism guarantees that memory expands and contracts dynamically without stalling active inference streams.

Architecture: The kvcached Driver and Two-Tier Control Plane

The Prism framework is structured as a decoupled, two-tier architecture: a high-performance GPU Data Plane governed by the kvcached balloon driver, and an intelligent Memory-Centric Control Plane comprising a Global Placement Scheduler and a Local Request Arbitrator.

The kvcached Driver Microarchitecture

The kvcached balloon driver operates as a shared dynamic library (libkvcached.so) loaded directly into the serving engine process. Built upon low-level CUDA Driver Virtual Memory Management (VMM) APIs, kvcached manages memory across three explicit lifecycles:

1. Virtual Address Space Reservation: During engine boot, kvcached invokes cuMemAddressReserve() to claim a continuous 64GB–128GB virtual address window for each model’s KV tensor. Crucially, this operation reserves address space in the GPU’s memory management unit (MMU) but allocates zero physical bytes of HBM.
2. On-Demand Page Backing: As input tokens arrive and KV blocks are allocated by SGLang’s scheduler, kvcached checks whether the target virtual offsets are mapped. If unmapped, it requests physical memory chunks from the GPU driver using cuMemCreate(), binds them to the virtual addresses via cuMemMap(), and sets read-write permissions with cuMemSetAccess().
3. Instant Memory Deflation: When a request completes, or when the global scheduler commands memory reclamation, kvcached unmaps physical handles via cuMemUnmap() and destroys physical allocations with cuMemRelease(), immediately freeing physical HBM for other models.

Because memory allocation and mapping occur at the 2MB boundary, mapping latencies remain under 10 to 25 microseconds, making dynamic ballooning effectively invisible to running attention computations.

Global Scheduler: KVPR-Driven Model Placement

To place heterogeneous models across a cluster of GPUs without triggering memory contention, Prism introduces the Key-Value Pressure Ratio (KVPR) metric.

For any model $m_i$ characterized by an average request arrival rate $r_i$, an SLO latency target $s_i$, and physical model weight footprint $w_i$, its SLO-weighted compute demand is defined as $d_i = \frac{r_i}{s_i}$. When a set of models $\mathcal{M}g$ is colocated on a GPU group $g$ with total physical HBM capacity $C$, the shared memory available for KV caching is: $$S_g = C - \sum{m_i \in \mathcal{M}_g} w_i$$

The Key-Value Pressure Ratio for GPU group $g$ is formulated as the ratio of aggregate SLO-weighted demand to remaining shared KV-cache capacity: $$\text{KVPR}(g) = \frac{\sum_{m_i \in \mathcal{M}g} d_i}{S_g} = \frac{\sum{m_i \in \mathcal{M}g} \frac{r_i}{s_i}}{C - \sum{m_i \in \mathcal{M}_g} w_i}$$

By balancing models with complementary demand patterns—pairing high-demand, low-weight models with low-demand, high-weight models—the global scheduler prevents any individual GPU from becoming a tail-latency bottleneck.

Local Scheduler: Slack-Aware Request Arbitration

Once models are assigned to a GPU group, incoming requests must be scheduled to prevent short, latency-critical requests from being blocked by heavy, long-context requests.

Prism implements a local request queue based on a generalized variant of the Moore-Hodgson Algorithm for minimizing the number of late jobs:

• Slack Estimation: For each request $r$, Prism computes its remaining latency slack: $$\text{Slack}(r) = a_r + s_r - \text{CurrentTime} - e_r$$ where $a_r$ is the arrival timestamp, $s_r$ is the TTFT SLO target, and $e_r$ is the estimated prefill execution duration ($e_r = \frac{p_r}{c}$, proportional to prompt length $p_r$ and hardware compute throughput $c$).
• Priority-Based Admission: Requests are sorted by earliest deadline. If the scheduler detects that admitting request $r$ causes a downstream deadline violation, it identifies the request with the largest execution time in the active batch ($r_{\max} = \arg\max \frac{p_r’}{c}$), evicting or deferring it to CPU memory to salvage the SLOs of all remaining requests.

Implementation: Integrating kvcached with SGLang and CUDA VMM

Deploying Prism in a production cluster requires integrating kvcached into the core tensor management layer of the serving engine. Below, we examine the low-level C++ CUDA VMM driver implementation and its Python bindings for SGLang.

Step 1: Low-Level CUDA VMM Driver (kvcached_driver.cpp)

The following C++ source demonstrates how kvcached uses the CUDA Driver API to manage physical memory pages beneath an arbitrary PyTorch virtual tensor pointer:

Step 2: Python SGLang Integration Hook (prism_sglang_manager.py)

In SGLang, the KV-cache manager (KVCachePool) interacts with the balloon driver through lightweight Python bindings. The following implementation demonstrates dynamic page provisioning during token batch generation:

Step 3: Fast Model Activation Pipeline

When a model is completely idle, its weights are offloaded to host CPU DRAM. To eliminate cold-start lag when new requests arrive, Prism implements a three-stage accelerated activation pipeline:

• Pinned Host Staging: Model weights in CPU memory are stored in pinned (page-locked) host memory pools, enabling maximum PCIe Gen5 line-rate saturation (up to 64 GB/s per x16 slot).
• Asynchronous Pipelined Transfer: Weight loading is decomposed into layer chunks. As Layer 0 is copied over PCIe, the GPU initializes Layer 0 CUDA Graph executors concurrently.
• Fast Startup Metrics: As evaluated on NVIDIA H100 systems, small models (1B–8B) activate from host memory in under 0.7 seconds, 14B models in 1.3 seconds, and large 70B models in 1.5 seconds—completing well within multi-second conversational TTFT budgets.

Comparison: Prism vs MuxServe vs QLM vs ServerlessLLM

To contextualize Prism’s systems contributions, we compare its architectural attributes against leading multi-LLM serving frameworks: MuxServe (spatial sharing), QLM (temporal queue-based sharing), ServerlessLLM (checkpoint-accelerated serverless sharing), and vLLM / SGLang baseline (dedicated instances).

Key Performance Trade-offs

1. SLO Attainment Under Bursting: In the presence of unpredictable traffic surges, MuxServe’s static partitions cause active models to run out of KV memory, driving TTFT SLO violations to over 40%. QLM swaps weights so aggressively between alternating models that engine restarts consume more time than inference. Prism continuously harvests unused memory from idle colocated models, maintaining 99% TTFT and TPOT attainment.
2. Hardware Efficiency and GPU Counts: When evaluated across 58 real-world models from the Chatbot Arena and Hyperbolic traces, Prism satisfies 99% of all TTFT and TPOT SLOs using only 16 NVIDIA H100 GPUs. In contrast, MuxServe++ requires 32 GPUs to achieve comparable SLO adherence, and static partitioning fails to reach 99% attainment even with 32 GPUs provisioned.
3. Runtime Overhead: Under constant, worst-case traffic streams with zero idle gaps (where static partitioning is theoretically optimal), Prism’s VMM mapping layer adds merely 3ms (4%) TTFT overhead and 4ms (13%) TPOT overhead, demonstrating that dynamic elasticity incurs negligible runtime penalties.

Enterprise platforms deploying complex reasoning agents—such as automated enterprise AI data extraction systems—achieve massive throughput multipliers by avoiding over-provisioning.

Production: Cluster Deployment, KVPR Optimization, and Case Studies

Deploying Prism across enterprise GPU clusters involves configuring the global scheduler, provisioning the Redis request buffer, and verifying telemetry metrics across production nodes.

Production Cluster Deployment Script

The following deployment script initializes Prism’s global coordinator, loads the KVPR placement optimizer, and mounts the SGLang inference engines:

Real-World Production Case Studies

Prism was validated in dual production shadow deployments across two major industrial environments:

• Company A (Enterprise Internal Cloud): Deployed on an internal cluster serving thousands of research and enterprise engineers running mixed workloads (3B to 70B models). With Prism dynamically managing model placement and elastic ballooning, Company A achieved a 3.89x increase in average token throughput per GPU over multi-week measurement periods, with zero SLO violations and identical p99 tail latency.
• Company B (Commercial Multi-Tenant Inference Provider): Optimized revenue per GPU under bursty public API traffic. Prior to Prism, rigid memory partitioning forced Company B to over-provision GPU worker pools. Deploying Prism increased billable revenue per GPU by 2.86x, directly monetizing previously stranded memory capacity.

By transforming GPU High-Bandwidth Memory from a set of rigid, fragmented silos into an elastic, self-balancing pool, Prism establishes the definitive systems blueprint for high-efficiency, multi-tenant AI inference infrastructure.