kv-cachelmcachevllmllm-servingtutorial

Hierarchical KV Cache: Scaling LLM Memory Beyond VRAM

September 21, 2026

Hierarchical KV Cache Offloading Architecture: Decoupling Context Capacity from GPU VRAM

The Problem: The Prompt Recomputation Tax in Long-Context Serving

In modern AI agent loops, enterprise Retrieval-Augmented Generation (RAG), and interactive code assistants, requests are characterized by massive shared prefixes:

System prompts defining multi-agent personas and behavioral boundaries.
OpenAPI schemas and function-calling JSON specifications.
Multi-thousand-token customer documentation, codebases, or legal contracts.
Accumulating multi-turn conversation histories.

While advanced GPU memory allocators (such as PagedAttention) eliminate internal fragmentation within active memory, High Bandwidth Memory (HBM) remains strictly physical and scarce—capped at 80 GB on NVIDIA H100s and 141 GB on H200s.

When a server handles hundreds of concurrent users, GPU memory quickly reaches 100% capacity. To prevent Out-of-Memory (OOM) crashes, the memory manager must evict idle KV caches. When an interactive user pauses for 30 seconds before submitting their next query, their 32,000-token conversation cache is evicted from GPU VRAM to make room for active generation.

The Massive Cost of Cache Eviction

When the user submits their follow-up message, the server experiences a complete cache miss:

Massive Compute Waste: For an open-weights model like LLaMA-3-70B, recomputing the attention activations for a 32k prompt requires over 4,500 TFLOPs of matrix multiplications.
Brutal Time to First Token (TTFT) Spikes: The user stares at a frozen screen for 3.5 to 8.0 seconds while the GPU re-executes the identical prefill computation it already completed minutes earlier.
GPU Thrashing: During traffic peaks, GPUs spend over 70% of their operational cycles recomputing identical prompt prefixes rather than generating new tokens, collapsing total serving throughput.

The Solution: Hierarchical Storage Tiering (LMCache & Mooncake)

Introduced in open-source systems like LMCache (UC Berkeley / University of Chicago) and distributed infrastructure like Mooncake Store, Hierarchical KV Cache Offloading decouples context memory capacity from physical GPU VRAM.

Instead of discarding evicted KV cache blocks or keeping them locked inside high-cost GPU HBM, the system organizes memory into an automated four-tier storage hierarchy, analogous to CPU cache hierarchies (L1, L2, L3, and Virtual Memory).

Hierarchical Storage Tiers and Data Flow: How Pipelined Cache Transfers Bypass Heavy Prompt Prefill Computation

The Fundamental Asymmetry: Moving Bytes vs. Computing FLOPs

The economic and latency justification for hierarchical offloading rests on a stark hardware reality: loading precomputed KV activations over PCIe or high-speed NVMe is an order of magnitude faster than re-executing attention FLOPs on GPU Tensor Cores.

Consider a 32,000-token prompt on an 8-GPU node running an FP16 70B parameter model:

Recomputing Prefill: Requires ~4,500 TFLOPs of computation. On an 8x H100 cluster under real-world serving load, this takes ~3,500 milliseconds.
Transferring Precomputed KV Cache: With 8-bit quantization, the 32k KV cache occupies only ~1.0 GB of memory. Transferring 1.0 GB across a modern PCIe 5.0 x16 bus ($64 \text{ GB/s}$) takes ~15.6 milliseconds!

Streaming precomputed activations across the PCIe bus is over 200x faster than re-running the forward pass. By replacing GPU computation with high-bandwidth memory transfers, Time to First Token (TTFT) drops from seconds to sub-200ms intervals.

The Four-Tier Storage Hierarchy

Modern hierarchical KV engines manage attention activations across four hardware layers:

1. Tier 1: GPU High Bandwidth Memory (HBM)

Speed: 3.35 TB/s (H100) to 4.8 TB/s (H200).
Role: Stores the active working set of tokens currently undergoing generation or immediate prefill.

2. Tier 2: Host CPU Main Memory (DRAM)

Speed: 64 GB/s per GPU over PCIe 5.0 x16 (bidirectional DMA).
Capacity: 512 GB to 2.0 TB per server node (using standard DDR5 memory).
Role: Sub-millisecond warm cache pool. When a request finishes its turn, its paged KV blocks are immediately transferred to host DRAM via asynchronous cudaMemcpyAsync. If the user responds within minutes, the cache streams back to GPU HBM in $<100\text{ms}$.

3. Tier 3: Local NVMe Solid-State Drives

Speed: 7.0 GB/s to 14.0 GB/s using Direct I/O (io_uring).
Capacity: 8 TB to 30 TB of low-cost enterprise storage.
Role: Persistent multi-hour conversation storage. Inactive sessions in CPU DRAM are flushed to local NVMe, allowing servers to retain days of conversation history without consuming expensive RAM.

4. Tier 4: Distributed Remote Object Store (Mooncake / Ceph / S3)

Speed: 100 Gbps to 400 Gbps RDMA over Converged Ethernet (RoCE).
Capacity: Petabytes of centralized cluster storage.
Role: Cross-Node Cache Sharing. In distributed multi-node clusters, a load balancer may route a user’s initial prompt to GPU Worker A, and their subsequent query to GPU Worker B. Tier 4 allows Worker B to pull the precomputed KV cache directly from the shared cluster store over RDMA, eliminating node-affinity constraints.

Serving Architecture Comparison Matrix

Dimension	Pure GPU Caching (vLLM Native)	Recompute on Miss	Disaggregated PD (Splitwise)	Hierarchical Offload (LMCache)
Max Context Retention	Bounded by 80GB HBM	Zero retention	Bounded by cluster HBM	Virtually Infinite (Terabytes NVMe)
Multi-Turn TTFT (Hit)	~30ms	3,000ms – 8,000ms	200ms – 500ms	120ms – 200ms
Hardware Cost	High ($30k+ GPUs required)	Extreme compute waste	High (Dedicated prefill GPUs)	Low (Standard DDR5 & NVMe)
Cross-Node Sharing	None (Node-locked)	N/A	None	Native (Shared Mooncake/Redis)
Prefill FLOP Savings	Limited by HBM evictions	0%	0%	Up to 90% FLOP reduction
Cluster Complexity	Low	Low	Very High (Network KV piping)	Moderate (Storage abstraction)

Reference Implementation: Hierarchical Cache Tier Simulator

Below is an educational Python implementation demonstrating hierarchical cache tiering across GPU VRAM, CPU Host Memory, and Local Disk, featuring automated LRU eviction down the storage ladder and instant cache retrieval.

from dataclasses import dataclass
from typing import Dict, Optional, List
import time

@dataclass
class KVCacheChunk:
    prefix_hash: str
    num_tokens: int
    size_mb: float
    data: bytes

class HierarchicalKVManager:
    """
    Simulates a 3-tier hierarchical KV cache manager:
    Tier 1: GPU HBM (Fastest, Smallest)
    Tier 2: Host CPU DRAM (Fast, Medium)
    Tier 3: Local NVMe Disk (Slower, Massive)
    """
    def __init__(self, gpu_cap_mb: float = 2048, cpu_cap_mb: float = 8192, disk_cap_mb: float = 32768):
        self.caps = {"gpu": gpu_cap_mb, "cpu": cpu_cap_mb, "disk": disk_cap_mb}
        self.usage = {"gpu": 0.0, "cpu": 0.0, "disk": 0.0}

        # Maps prefix_hash -> (tier_name, KVCacheChunk, last_access_time)
        self.index: Dict[str, dict] = {}

    def put(self, chunk: KVCacheChunk):
        """Stores a new chunk, placing it directly into GPU HBM."""
        self._ensure_capacity("gpu", chunk.size_mb)
        self.index[chunk.prefix_hash] = {
            "tier": "gpu",
            "chunk": chunk,
            "last_access": time.time()
        }
        self.usage["gpu"] += chunk.size_mb

    def get(self, prefix_hash: str) -> Optional[KVCacheChunk]:
        """
        Retrieves a chunk. If found in CPU or Disk, promotes it back to GPU HBM!
        """
        if prefix_hash not in self.index:
            return None # Cache miss: must recompute prefill

        entry = self.index[prefix_hash]
        current_tier = entry["tier"]
        chunk = entry["chunk"]
        entry["last_access"] = time.time()

        if current_tier == "gpu":
            print(f"CACHE HIT [GPU HBM]: Instant zero-transfer access for {chunk.num_tokens} tokens!")
            return chunk

        # Promotion to GPU HBM
        print(f"CACHE HIT [{current_tier.upper()}]: Promoting {chunk.size_mb:.1f} MB to GPU HBM (Bypassing Prefill)...")
        self._ensure_capacity("gpu", chunk.size_mb)

        # Demote from current tier
        self.usage[current_tier] -= chunk.size_mb
        entry["tier"] = "gpu"
        self.usage["gpu"] += chunk.size_mb
        return chunk

    def _ensure_capacity(self, tier: str, required_mb: float):
        """Evicts oldest chunks to the lower storage tier when space is exhausted."""
        while self.usage[tier] + required_mb > self.caps[tier]:
            # Find LRU chunk in this tier
            candidates = [k for k, v in self.index.items() if v["tier"] == tier]
            if not candidates:
                break
            lru_key = min(candidates, key=lambda k: self.index[k]["last_access"])
            self._demote_chunk(lru_key, from_tier=tier)

    def _demote_chunk(self, key: str, from_tier: str):
        """Pushes an evicted chunk down the storage ladder."""
        entry = self.index[key]
        chunk = entry["chunk"]

        if from_tier == "gpu":
            next_tier = "cpu"
        elif from_tier == "cpu":
            next_tier = "disk"
        else:
            # Exceeded disk capacity: complete eviction
            self.usage["disk"] -= chunk.size_mb
            del self.index[key]
            print(f"DISK PURGE: Dropped {key}")
            return

        print(f"EVICTION CASCADE: Demoting {chunk.size_mb:.1f} MB from {from_tier.upper()} -> {next_tier.upper()}")
        self._ensure_capacity(next_tier, chunk.size_mb)
        self.usage[from_tier] -= chunk.size_mb
        entry["tier"] = next_tier
        self.usage[next_tier] += chunk.size_mb


# Demonstration: Simulating Cache Eviction and Multi-Turn Retrieval
if __name__ == "__main__":
    # Small test capacities to demonstrate eviction cascades
    manager = HierarchicalKVManager(gpu_cap_mb=500, cpu_cap_mb=1000, disk_cap_mb=2000)

    # User A starts a 16k conversation (occupies 350 MB)
    chunk_a = KVCacheChunk(prefix_hash="user_alpha_32k", num_tokens=32000, size_mb=350.0, data=b"kv_alpha")
    manager.put(chunk_a)
    print(f"Stored User A | GPU Usage: {manager.usage['gpu']:.1f} MB / {manager.caps['gpu']:.1f} MB")

    # User B arrives with a large 300 MB prompt, forcing User A out of GPU HBM into CPU DRAM
    chunk_b = KVCacheChunk(prefix_hash="user_beta_30k", num_tokens=30000, size_mb=300.0, data=b"kv_beta")
    manager.put(chunk_b)
    print(f"Stored User B | GPU Usage: {manager.usage['gpu']:.1f} MB | CPU Usage: {manager.usage['cpu']:.1f} MB")

    # User A sends follow-up query after 30 seconds: Retrieved from CPU DRAM without prefill!
    print("\n--- User A Sends Follow-Up Turn ---")
    retrieved = manager.get("user_alpha_32k")
    assert retrieved is not None

Production Deployment and Ecosystem Integration

Hierarchical offloading is rapidly standardizing across open-source and enterprise inference stacks:

1. Native vLLM & LMCache Configuration

Modern vLLM instances connect to LMCache via modular plugin adapters:

# Launch vLLM with LMCache hierarchical storage connector
lmcache_vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --lmcache-config ./lmcache_config.yaml \
    --max-model-len 65536 \
    --gpu-memory-utilization 0.90

Inside lmcache_config.yaml, engineers configure local DRAM buffer pools (local_cpu: true, max_local_cpu_size: 64GB), local NVMe caching directories, and remote Mooncake store endpoints.

2. Radical Reduction in Cloud GPU Spend

In enterprise environments running customer-facing agents, over 60% of daily tokens consist of repeated prompt prefixes (company documentation, system instructions, and multi-turn history).

By retaining these prefixes in low-cost host DRAM and NVMe drives, engineering teams cut their required GPU cluster footprint by 3x to 5x while delivering faster, more consistent conversational latency.

3. Local Desktop AI Acceleration in aiFetchly

For software engineers and data analysts using AiFetchly, hierarchical caching provides seamless desktop performance. When querying local documentation sets or conducting deep web research across multi-turn sessions, aiFetchly pairs with local Ollama or vLLM instances to retain project context in host RAM. Even after hours of inactivity, follow-up queries begin streaming responses instantaneously—eliminating irritating prompt precomputation delays on local workstation hardware.