ShadowKVKV CacheLLM ServingLong-ContextHeterogeneous Computing

Guide to ShadowKV: Scalable Long-Context LLM Inference

October 7, 2026

The era of long-context large language models (LLMs) has transformed how applications interact with enterprise knowledge. Today, models regularly process prompts spanning 128,000 to one million tokens—ingesting massive code repositories, dense legal discovery filings, and extensive conversational histories.

However, scaling sequence lengths in production serving systems has exposed a punishing hardware barrier: the GPU VRAM Capacity Wall. During autoregressive generation, every active sequence must maintain its complete Key-Value (KV) cache in fast GPU High-Bandwidth Memory (HBM). For an 8B model processing a 1M-token prompt or a 70B model serving a 128K context, the KV cache footprint alone exceeds 30GB to 64GB per single request. Because a single request consumes nearly the entire memory budget of an 80GB GPU, serving concurrency collapses to a batch size of one, causing operational serving costs to explode.

While researchers previously proposed offloading idle KV caches to inexpensive host CPU memory (DRAM), transferring complete KV caches back to the GPU across PCIe links introduces severe I/O stalls, dropping generation throughput below interactive thresholds.

Presented by ByteDance Seed researchers at ICML 2025 as a Spotlight paper, ShadowKV resolves this fundamental systems bottleneck. By keeping a compact, low-rank Key cache on the GPU (in the “light”) while offloading the full Value cache to host CPU memory (in the “shadows”), ShadowKV dynamically evaluates attention scores on-device and streams only the top-k critical Value tokens over PCIe. This heterogeneous approach slashes GPU memory consumption by up to 8x, increases serving throughput by up to 6x, and expands concurrent batch capacity by 3x on 1M-token workloads without degrading model accuracy.

In this technical guide, we dissect the architecture of ShadowKV, formalize the mathematics of low-rank Key projection and sparse Value streaming, construct an end-to-end PyTorch reference implementation, and share production deployment strategies.

Problem: The VRAM Capacity and PCIe Bandwidth Bottlenecks in Long-Context Serving

To appreciate ShadowKV’s design, we must examine the physical resource conflicts that arise when serving long-context LLMs.

The VRAM Capacity Wall and Concurrency Collapse

The Key-Value cache memory requirement grows strictly linearly with context length $S$, number of layers $L$, KV head count $H_{\text{kv}}$, and hidden dimension $d$:

$$\text{Memory}{\text{KV}} = 2 \times L \times H{\text{kv}} \times d \times S \times \text{sizeof(dtype)}$$

For a modern 70B parameter model ($L = 80, H_{\text{kv}} = 8, d = 128$) at FP16 precision:

A 32,768-token prompt requires 10.48 GB of dedicated VRAM.
A 131,072-token prompt requires 41.94 GB of dedicated VRAM.
A 1,000,000-token prompt requires 320.00 GB of dedicated VRAM.

On an 80GB NVIDIA H100 or A100 GPU:

The base model weights already consume 35GB to 40GB (even when quantized to 4-bit/8-bit).
Serving just two concurrent 32K requests completely saturates the remaining VRAM.
Long-context queries force the serving runtime to reduce the active batch size to 1, leaving the GPU’s 3,350 TFLOPS of compute capability over 80% idle while waiting for sequential token generation.

Enterprise automation systems that ingest large volumes of live web documentation frequently collide with this concurrency wall. Pairing efficient serving architectures with an enterprise web scraping platform allows organizations to extract and process real-time web content without suffering from memory starvation.

Why Full KV Cache Offloading Stalls on PCIe Bandwidth

Because host CPU DRAM is inexpensive and abundant (modern server nodes routinely pack 512GB to 2TB of DDR5 memory), a natural alternative is KV Cache Offloading: store the KV cache in host DRAM and transfer it to the GPU via PCIe during decoding.

However, standard full offloading fails because the PCIe bus becomes a crippling bottleneck:

The PCIe Bandwidth Bottleneck:
Host CPU DRAM (1TB Capacity, 300 GB/s)
   │   │PCIe 4.0 x16 Bus (Max 32 GB/s Bandwidth)
   │GPU HBM (80GB Capacity, 3,350 GB/s Bandwidth)

Scenario: Decoding 1 token with 64K historical tokens (16GB KV cache)
•Bandwidth needed: Transfer 16GB over PCIe at 32 GB/s
•Transfer latency: 16 GB / 32 GB/s = 500 ms per token!
•Result: Decoding slows to 2 tokens per second (Unusable for real-time applications!)

Transferring 16GB of KV state over a PCIe 4.0 link takes 500 milliseconds per token, reducing generation speed to a crawl. The fundamental challenge is clear: we must exploit host DRAM capacity without saturating the PCIe interface.

Solution: How ShadowKV Coordinates GPU Light and CPU Shadows

ShadowKV recognizes an essential mathematical asymmetry between Key and Value tensors in self-attention:

Key vectors are used exclusively for computing attention scores: $\text{Score} = Q K^T / \sqrt{d}$.
Value vectors are used exclusively to aggregate the final representation: $O = \text{Softmax}(\text{Score}) V$.
Attention Sparsity: In long sequences, the softmax distribution is exceptionally sparse. A single query token typically attends to only 5% to 10% of historical tokens, while 90% to 95% of Value tokens receive negligible attention mass and have almost zero influence on the final output.

ShadowKV leverages this property through a four-part heterogeneous architecture:

1. Low-Rank Key Cache on GPU (Light Tier)

Rather than storing full-precision Key vectors ($K \in \mathbb{R}^{S \times d}$) on the GPU, ShadowKV projects them into a low-dimensional subspace of rank $r \ll d$ (e.g., projecting from $d = 128$ down to $r = 16$ or $r = 32$):

$$K_{\text{low}} = K \mathbf{W}{K} \quad \text{where } \mathbf{W}{K} \in \mathbb{R}^{d \times r}$$

The projection matrix $\mathbf{W}_K$ is computed offline via singular value decomposition (SVD) on calibration activations or learned with a lightweight linear projection.

Because $r \ll d$, the memory footprint of $K_{\text{low}}$ on the GPU is 4x to 8x smaller than the original Key cache. Storing only $K_{\text{low}}$ on the GPU requires negligible VRAM, allowing the GPU to host long contexts and large batch sizes directly in HBM.

2. Full Value Cache Offloading to Host CPU (Shadow Tier)

The complete Value cache ($V \in \mathbb{R}^{S \times d}$) and the full-precision Key cache are stored in host CPU DRAM (“in the shadows”). Because host memory is expansive (often hundreds of gigabytes), storing full 1M-token Value caches incurs zero VRAM penalty on the GPU.

3. Query-Aware Sparse Top-K Attention Evaluation

When a query token $q \in \mathbb{R}^d$ arrives during decoding:

The query is projected into the same low-rank subspace: $q_{\text{low}} = q \mathbf{W}_{K}$.
The GPU computes low-rank attention scores directly against all tokens in HBM:

$$S_{\text{low}} = \frac{q_{\text{low}} K_{\text{low}}^T}{\sqrt{r}}$$

Because low-rank projections preserve the principal directional components of key vectors, $S_{\text{low}}$ accurately preserves the relative ranking order of attention weights. 3. The GPU identifies the top-$k$ token positions (or top-$k$ memory chunks) that carry the largest attention scores (typically the top 5% to 10% of the sequence).

4. Asynchronous Overlapped PCIe Streaming Pipeline

Instead of transferring the entire 16GB Value cache over PCIe, the GPU issues an asynchronous DMA copy requesting only the selected top-k Value tokens from host DRAM:

Transferring 5% to 10% of the Value cache requires moving only 800MB to 1.6GB instead of 16GB!
At PCIe 4.0 speeds, this transfer completes in 25 to 50 milliseconds.
ShadowKV overlaps PCIe transfers with ongoing MLP compute and layer-wise prefetching using dedicated CUDA streams, hiding the remaining I/O latency behind GPU tensor calculations.

When deploying high-throughput agent swarms that analyze continuous data feeds, pairing ShadowKV with fast data fetching services ensures that external context ingestion and on-device token generation proceed without latency degradation.

Architectural Deep Dive: Dataflow and Hardware Dynamics

The table below contrasts the mechanical execution properties of standard full attention, naive CPU offloading, and ShadowKV:

Architectural Metric	Dense GPU Execution	Naive Host CPU Offloading	ShadowKV Heterogeneous Architecture
Key Cache Placement	GPU HBM (Full Precision)	Host CPU DRAM	GPU HBM (Low-Rank $r=16/32$)
Value Cache Placement	GPU HBM (Full Precision)	Host CPU DRAM	Host CPU DRAM (Full Precision)
VRAM Footprint	$100%$ (Extremely High)	$\approx 0%$	$12.5%\text{—}25%$ ($4\text{—}8\times$ reduction)
PCIe Bus Traffic	0 GB (No transfers)	$100%$ of KV Cache ($16\text{GB}+$)	Only Top-K Values ($5\text{—}10%$, $\approx 1\text{GB}$)
PCIe Transfer Latency	None	500ms+ per token (Unusable)	$<30\text{ms}$ (Fully overlapped with compute)
Max Batch Size (128K)	1 (VRAM exhausted)	Large (Capped by PCIe)	$3\times\text{—}4\times$ larger concurrency
Retrieval Accuracy	100% (Baseline)	100%	$>99.8%$ on LongBench & Needle tests

Integrating these heterogeneous inference pipelines into modern microservices is straightforward using versatile API integration tools, facilitating smooth coordination between distributed memory pools and model endpoints.

Implementation: End-to-End PyTorch Reference of ShadowKV Engine

Below is a self-contained PyTorch reference implementation illustrating ShadowKV’s low-rank Key projection, CPU-GPU memory manager, and sparse streaming attention execution.

Step 1: Low-Rank Projection and Offline Key Profiler

First, we implement the projection module that maps Key vectors into a compact low-rank subspace:

import math
import torch
import torch.nn as nn
from typing import Tuple, List, Dict, Optional

class LowRankKeyProjector(nn.Module):
    """
    Projects high-dimensional Key vectors into a compact low-rank subspace
    to minimize GPU VRAM consumption while preserving attention rankings.
    """
    def __init__(self, head_dim: int = 128, rank: int = 24):
        super().__init__()
        self.head_dim = head_dim
        self.rank = rank

        # Learnable or SVD-initialized projection matrix
        self.proj = nn.Linear(head_dim, rank, bias=False)
        self.scale = 1.0 / math.sqrt(rank)

    def forward(self, keys: torch.Tensor) -> torch.Tensor:
        """
        Projects keys: [batch, num_heads, seq_len, head_dim] -> [batch, num_heads, seq_len, rank]
        """
        return self.proj(keys)

Step 2: Heterogeneous CPU-GPU Cache Allocator

Next, we build the memory manager that allocates low-rank Key tensors in GPU HBM while retaining full-precision Value tensors in pinned host CPU memory:

class ShadowKVCache:
    """
    Heterogeneous KV Cache Manager:
    - Low-rank Keys in GPU HBM ('Light Tier')
    - Full-precision Values in Host CPU DRAM ('Shadow Tier')
    """
    def __init__(
        self,
        max_batch_size: int,
        max_seq_len: int,
        num_heads: int,
        head_dim: int,
        rank: int = 24,
        dtype: torch.dtype = torch.float16,
        gpu_device: torch.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    ):
        self.max_batch_size = max_batch_size
        self.max_seq_len = max_seq_len
        self.num_heads = num_heads
        self.head_dim = head_dim
        self.rank = rank
        self.dtype = dtype
        self.gpu_device = gpu_device

        # GPU HBM Tier: Low-rank Key cache [Batch, Num_Heads, Seq_Len, Rank]
        self.k_low_gpu = torch.zeros(
            (max_batch_size, num_heads, max_seq_len, rank),
            dtype=dtype,
            device=gpu_device
        )

        # Host CPU Tier: Full Value cache in pinned memory [Batch, Num_Heads, Seq_Len, Head_Dim]
        # Pinned memory enables asynchronous zero-copy DMA transfers over PCIe
        self.v_full_cpu = torch.zeros(
            (max_batch_size, num_heads, max_seq_len, head_dim),
            dtype=dtype,
            device="cpu"
        ).pin_memory() if gpu_device.type == "cuda" else torch.zeros(
            (max_batch_size, num_heads, max_seq_len, head_dim),
            dtype=dtype,
            device="cpu"
        )

        # Track active sequence lengths per batch
        self.seq_lengths = torch.zeros(max_batch_size, dtype=torch.long, device=gpu_device)

    def append_kv(
        self,
        batch_idx: int,
        k_low_tokens: torch.Tensor,
        v_full_tokens: torch.Tensor
    ) -> None:
        """
        Appends newly generated or prefilled token states.
        k_low_tokens: [num_heads, num_new_tokens, rank] (on GPU)
        v_full_tokens: [num_heads, num_new_tokens, head_dim] (on CPU or GPU)
        """
        curr_len = self.seq_lengths[batch_idx].item()
        num_new = k_low_tokens.shape[1]
        new_len = curr_len + num_new
        assert new_len <= self.max_seq_len, "Context capacity exceeded"

        # 1. Update Low-Rank Key Cache on GPU
        self.k_low_gpu[batch_idx, :, curr_len:new_len, :] = k_low_tokens

        # 2. Update Full Value Cache on Host CPU DRAM
        self.v_full_cpu[batch_idx, :, curr_len:new_len, :] = v_full_tokens.to("cpu", non_blocking=True)

        self.seq_lengths[batch_idx] = new_len

Step 3: Sparse Attention Evaluator and Streaming Execution Engine

Finally, we construct the sparse attention execution engine that computes low-rank attention scores on the GPU, selects top-k token indices, and streams active Value slices over PCIe:

class ShadowKVAttentionEngine(nn.Module):
    """
    Evaluates low-rank attention on GPU and streams active top-k Value tokens over PCIe.
    """
    def __init__(
        self,
        projector: LowRankKeyProjector,
        head_dim: int = 128,
        rank: int = 24,
        top_k_ratio: float = 0.10
    ):
        super().__init__()
        self.projector = projector
        self.head_dim = head_dim
        self.rank = rank
        self.top_k_ratio = top_k_ratio
        self.scale = 1.0 / math.sqrt(head_dim)

    def forward(
        self,
        query: torch.Tensor,
        cache: ShadowKVCache,
        batch_idx: int,
        sink_tokens: int = 4,
        local_window: int = 32
    ) -> torch.Tensor:
        """
        Decodes a single query token:
        query shape: [num_heads, 1, head_dim] (on GPU)
        Returns: [num_heads, head_dim]
        """
        num_heads, _, head_dim = query.shape
        curr_len = cache.seq_lengths[batch_idx].item()
        device = query.device
        dtype = query.dtype

        if curr_len == 0:
            return torch.zeros((num_heads, head_dim), dtype=dtype, device=device)

        # 1. Project Query to Low-Rank Subspace on GPU
        # [num_heads, 1, head_dim] -> [num_heads, 1, rank]
        q_low = self.projector.proj(query)
        q_low_scaled = q_low * (1.0 / math.sqrt(self.rank))

        # 2. Compute Low-Rank Attention Scores on GPU
        # k_active shape: [num_heads, curr_len, rank]
        k_active = cache.k_low_gpu[batch_idx, :, :curr_len, :]

        # [num_heads, 1, rank] @ [num_heads, rank, curr_len] -> [num_heads, curr_len]
        scores_low = torch.bmm(q_low_scaled, k_active.transpose(1, 2)).squeeze(1)

        # 3. Top-K Index Selection
        budget_k = max(sink_tokens + local_window, int(curr_len * self.top_k_ratio))
        budget_k = min(budget_k, curr_len)

        if budget_k >= curr_len:
            # Short sequence: fetch entire sequence
            selected_indices = torch.arange(curr_len, device=device).unsqueeze(0).repeat(num_heads, 1)
        else:
            # Mask out mandatory sinks and local window from competitive selection
            candidate_mask = torch.ones(curr_len, dtype=torch.bool, device=device)
            candidate_mask[:sink_tokens] = False
            candidate_mask[-local_window:] = False

            masked_scores = scores_low.clone()
            masked_scores[:, ~candidate_mask] = float("-inf")

            comp_k = budget_k - (sink_tokens + local_window)
            _, top_indices = torch.topk(masked_scores, k=comp_k, dim=-1)

            # Concatenate mandatory sinks, local window, and competitive top-k
            mandatory = torch.cat([
                torch.arange(sink_tokens, device=device),
                torch.arange(curr_len - local_window, curr_len, device=device)
            ]).unsqueeze(0).repeat(num_heads, 1)

            selected_indices = torch.cat([mandatory, top_indices], dim=-1)

        # 4. Asynchronous Sparse DMA Transfer of Value Tokens from CPU to GPU
        # CPU slice: [num_heads, curr_len, head_dim]
        # Gather only selected indices across heads
        v_selected_list = []
        for h in range(num_heads):
            idx_h = selected_indices[h].cpu()
            # Fetch slice from pinned host CPU memory to GPU
            v_h_cpu = cache.v_full_cpu[batch_idx, h, idx_h, :]
            v_h_gpu = v_h_cpu.to(device, non_blocking=True)
            v_selected_list.append(v_h_gpu)

        v_sparse_gpu = torch.stack(v_selected_list, dim=0) # [num_heads, budget_k, head_dim]

        # 5. Compute Final Attention Weights on Sparse Tokens
        # Gather corresponding scores: [num_heads, budget_k]
        sparse_scores = torch.gather(scores_low, dim=-1, index=selected_indices)
        attn_weights = torch.softmax(sparse_scores, dim=-1).unsqueeze(1) # [num_heads, 1, budget_k]

        # Aggregate output: [num_heads, 1, budget_k] @ [num_heads, budget_k, head_dim] -> [num_heads, head_dim]
        out = torch.bmm(attn_weights, v_sparse_gpu).squeeze(1)
        return out

Best Practices for Deploying ShadowKV in Production Serving

Deploying ShadowKV in production LLM inference clusters requires fine-tuning rank dimensions, memory pooling, and PCIe pipeline overlap.

1. Calibrating Projection Rank vs. Recall Fidelity

Rank Dimension ($r$): For models with standard head dimensions ($d = 128$), an offline rank of $r = 24$ to $r = 32$ captures over 94% of singular value variance while achieving a 4x to 5.3x memory reduction on Key tensors.
Dynamic Head Budgets: In Grouped-Query Attention (GQA) architectures (such as LLaMA-3 with 32 query heads and 8 KV heads), compute low-rank scores across query heads and take the union of selected Value indices per KV group. This ensures DMA transfers over PCIe are shared across all query heads within the group.

2. Pinned Host Memory and CUDA Streams for Zero-Copy DMA Transfers

Asynchronous Prefetching: Always allocate host CPU buffers using torch.Tensor.pin_memory() or CUDA host-pinned allocators (cudaHostAlloc). Pinned memory bypasses OS page tables and allows the GPU Direct Memory Access (DMA) engine to stream data across PCIe without host CPU interruption.
Double Buffering Across Layers: While layer $l$ executes dense MLP feed-forward operations on the GPU, issue the PCIe DMA transfer for layer $l+1$‘s sparse Value tokens on a background CUDA stream. This completely hides the 25ms PCIe transfer latency behind arithmetic compute.

3. Co-locating ShadowKV with PagedAttention and GQA Models

Paged Allocation: Organize the low-rank Key cache on the GPU using PagedAttention virtual memory blocks (e.g., 16 tokens per page) to eliminate internal fragmentation.
Fallback to Full GPU: For sequences shorter than 8,192 tokens, keep both Keys and Values entirely in GPU VRAM. Trigger ShadowKV offloading dynamically only when sequence lengths exceed 16K tokens, ensuring short requests experience maximum single-device speed.

Engineering teams can evaluate infrastructure cost savings using our interactive pricing page, calculating exact GPU and server memory hardware requirements across expected concurrency loads.

Evaluation: Benchmarking ShadowKV Throughput and Memory Scaling

ShadowKV was benchmarked on NVIDIA A100 (80GB SXM4) and H100 (80GB HBM3) nodes with 512GB DDR5 system memory across LLaMA-3-8B-Instruct (64K context) and LLaMA-2-7B-32K.

Serving Throughput and Concurrency Scaling

Evaluation on continuous batching inference workloads (LLaMA-3-8B, 64K context length, PCIe 4.0 x16):

Concurrency Metric	Full GPU HBM Attention	Full CPU DRAM Offload	ShadowKV Architecture	Improvement vs Baseline
Max Batch Size (64K Context)	2 Requests (OOM limit)	16 Requests	12 Requests	6.0x Larger Batch Size
Serving Throughput (tokens/s)	42 tokens/sec	18 tokens/sec	128 tokens/sec	3.05x Higher Throughput
GPU VRAM per Request (64K)	16.8 GB	0.8 GB	2.6 GB	6.46x Memory Savings
PCIe Bus Traffic per Step	0 MB	16,800 MB	1,680 MB	90% Less PCIe Traffic

By shrinking Key memory on the GPU by over 6x and fetching only 10% of Values over PCIe, ShadowKV enables servers to run 6x larger concurrent batches, increasing overall token throughput by over 3x compared to baseline GPU execution.

Retrieval Accuracy on LongBench and Needle-in-a-Haystack

Evaluated on multi-document question answering (LongBench) and Needle-in-a-Haystack passkey retrieval across context depths from 16K to 128K tokens:

Benchmark / Depth	Dense Full Attention	Static Eviction (H2O, 20%)	ShadowKV ($r=24$, Top-10% Value)
LongBench (Avg F1 Score)	48.6	39.2	48.4 (99.6% Retention)
Needle-in-a-Haystack (32K)	100.0%	61.2%	100.0%
Needle-in-a-Haystack (64K)	100.0%	38.0%	99.8%
Needle-in-a-Haystack (128K)	99.8%	22.4%	99.5%

While static eviction baselines degrade rapidly as sequence lengths expand, ShadowKV retains over 99.5% of full attention retrieval accuracy because its low-rank on-device Keys accurately locate factual needles and fetch their full-precision Values from CPU memory on demand.

Comprehensive deployment blueprints and architecture schemas can be referenced in our developer developer documentation.

Conclusion: The Heterogeneous Era of Long-Context Intelligence

ShadowKV demonstrates that conquering the long-context memory wall does not require buying multi-million-dollar clusters of specialized GPUs. By coordinating high-speed GPU tensor cores with high-capacity host CPU memory through low-rank Key projections and sparse PCIe Value streaming, ShadowKV delivers high-throughput 1M-token inference on standard single-node servers.

As foundation models continue to scale their context horizons, heterogeneous memory management architectures will form the operational backbone of enterprise AI serving infrastructure.

Explore our unified AI Fetchly platform to discover how cutting-edge inference optimizations and real-time data pipelines power the next generation of scalable autonomous agents.