KVzip Guide: Query-Agnostic Reconstructive KV Compression
Large Language Models (LLMs) are increasingly tasked with ingesting massive contexts: multi-file codebase repositories, enterprise document archives, lengthy legal contracts, and dense research corpora. In enterprise document search, customer support copilots, and autonomous data collection systems powered by aiFetchly, applications frequently process prompts spanning from 32,000 to over 170,000 tokens.
While modern architectures possess the algorithmic capacity to attend across these massive windows, serving them autoregressively introduces a crippling systems bottleneck: the Key-Value (KV) cache memory footprint. For a model such as LLaMA-3.1-70B or Qwen-2.5-72B, storing the KV cache for a single 128k-token prompt consumes over 32 GB of GPU High Bandwidth Memory (HBM). In multi-tenant inference clusters, this memory explosion severely curtails serving concurrency, exhausts VRAM pools, and forces expensive memory swapping or request preemption.
To tame KV cache memory consumption, researchers have explored KV cache eviction—pruning non-essential key-value pairs to retain only a compact subset of salient tokens. Prominent methods such as SnapKV, H2O (Heavy Hitter Oracle), and StreamingLLM achieve 3x to 5x memory reduction.
However, existing KV eviction algorithms suffer from a fatal architectural flaw: query dependency. They determine which tokens to prune by evaluating the attention scores generated between the initial query tokens and the document context. While effective for single-turn, one-off question answering, query-dependent compression creates severe operational failure in production environments:
- Broken Multi-Turn Conversations: When a user asks a follow-up question in Turn 2, the tokens pruned during Turn 1 cannot be recovered. The model suffers catastrophic amnesia on information outside the initial query’s focus.
- Elimination of Shared Prefix Caching: In modern inference engines like vLLM and SGLang, shared document prefixes (e.g., system prompts, reference manuals, or database schemas) are cached in Radix Trees across thousands of concurrent users. Because query-dependent eviction tailors the cache to a specific user’s query, the compressed cache cannot be reused by any other user. The server must re-prefill the entire document from scratch for every request.
To eliminate this systemic constraint, researchers from Seoul National University and NAVER AI introduced KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction (arXiv:2505.23416, accepted as an Oral Presentation at NeurIPS 2025; GitHub: snu-mllab/kvzip).
KVzip introduces a paradigm shift: instead of pruning tokens based on an external query, it measures the intrinsic, general-purpose importance of each KV pair by evaluating the model’s ability to reconstruct its own context. By appending a lightweight repeat prompt during ingestion, KVzip quantifies self-reconstruction attention to extract universally salient tokens. The resulting compressed KV cache is 100% query-agnostic, shrinks memory footprints by 3x to 4x, accelerates FlashAttention decoding by ~2x, and enables seamless shared-prefix caching across multi-turn and multi-user serving workloads.
In this technical guide, we analyze the limitations of query-dependent eviction, break down the mathematics of context reconstruction scoring, inspect the KVzip serving mechanics in vLLM, and implement a complete PyTorch compression pipeline.
The Problem: The Query-Dependency Trap in Production LLM Serving
To understand why conventional KV eviction cannot support production serving engines, we must examine how modern inference systems manage memory and reuse cached prompts.
The Power of Shared-Prefix Caching in vLLM
In real-world enterprise AI deployments, prompts exhibit massive structural overlap. For example, in automated document extraction, thousands of concurrent requests share an identical 50,000-token PDF manual, differing only in a 50-token query appended at the end.
In engines like vLLM and SGLang, this pattern is optimized via RadixAttention / PagedPrefix Caching:
The Production Shared-Prefix Paradigm:
==================================================================================
1. Ingestion Phase:
[Large Shared Document (100k Tokens)] -> Pre-filled ONCE -> Cached in GPU VRAM
2. Request Serving Phase:
User 1 Prompt: [Shared Doc (100k)] + "What is the warranty policy?"
-> Prefill Time: 0 ms (Cache Hit on 100k tokens!) -> Instant TTFT
User 2 Prompt: [Shared Doc (100k)] + "List the safety warnings."
-> Prefill Time: 0 ms (Cache Hit on 100k tokens!) -> Instant TTFT
================================================================================== Shared prefix caching provides a 10x to 100x reduction in Time-To-First-Token (TTFT) and saves thousands of GPU compute hours. However, keeping full-precision 100k-token KV caches resident in GPU memory exhausts VRAM capacity rapidly.
How Query-Dependent Eviction Destroys Cache Sharing
Existing KV cache eviction methods—including SnapKV, H2O, and PyramidKV—rely on the query to score token importance:
$$\text{Importance}(i) = \sum_{q \in Q_{\text{user}}} \text{Softmax}\left(\frac{q \cdot k_i^T}{\sqrt{d}}\right)$$
This formula calculates how relevant historical token $i$ is to the specific user query $Q_{\text{user}}$.
The Query-Dependency Failure in Multi-Query Serving:
----------------------------------------------------------------------------------
User 1 asks: "What is the capital of France?"
- SnapKV scores tokens: Preserves sentences mentioning "Paris" and "France".
- Evicts: All sentences discussing "Germany", "Economy", "History", etc.
- Result on Query 1: Accurate answer.
User 2 arrives for the same document and asks: "What is the capital of Germany?"
- Cache Reuse Attempt: The cached KV represents ONLY Paris/France context!
- Information Loss: Sentences discussing "Germany" were permanently destroyed!
- Failure: Catastrophic hallucination or forced full recomputation!
---------------------------------------------------------------------------------- In multi-turn chat, document Q&A, and codebase analysis, query-dependent eviction completely shatters cache reuse. The server must choose between keeping enormous uncompressed caches or discarding caches entirely and re-running expensive prefill steps for every subsequent turn.
The KVzip Solution: Query-Agnostic Context Reconstruction
KVzip breaks this dependency by asking a fundamental question: What makes a token intrinsically important to a document, independent of any user query?
KVzip Context Reconstruction Architecture:
+----------------------------------------------------------------------------------+
| ORIGINAL CONTEXT TOKENS C = [x_1, ..., x_N] |
+----------------------------------------------------------------------------------+
│
▼ Append Repeat Sequence
+----------------------------------------------------------------------------------+
| EXPANDED INPUT: [Context C] + [Repeat Prompt: "Repeat: "] |
+----------------------------------------------------------------------------------+
│
▼ Forward Pass (Attention Computation)
+----------------------------------------------------------------------------------+
| CONTEXT RECONSTRUCTION ATTENTION MAP |
| |
| Query Tokens from Repeat Phrase attend back to Context Tokens C: |
| |
| Token x_i Importance = Sum of Attention Received from Reconstruction Queries |
| - High-Value Anchors (Entities, Core Facts, Syntax Structures) -> RETAINED |
| - Redundant Fillers, Generic Prepositions, Repetitive Noise -> EVICTED |
+----------------------------------------------------------------------------------+
│
▼ One-Time Eviction
+----------------------------------------------------------------------------------+
| COMPRESSED QUERY-AGNOSTIC KV CACHE (1/4 Size) |
| |
| Stored in vLLM Radix Tree: 100% Reusable for Q1, Q2, Q3... across any user! |
+----------------------------------------------------------------------------------+ 1. The Context Reconstruction Mechanism
To measure intrinsic semantic value, KVzip formulates an unsupervised context reconstruction objective during ingestion. Given a document context $C = (x_1, x_2, \dots, x_N)$, KVzip appends a minimal repeat prompt:
$$\tilde{C} = [x_1, x_2, \dots, x_N, \text{ “Repeat: ”}]$$
When the model processes the repeat prompt, its attention heads naturally query historical context tokens to reconstruct the document’s semantic structure.
The importance score $I(i)$ for the $i$-th key-value token is computed by aggregating the attention weights directed toward it from the reconstruction query positions:
$$I(i) = \sum_{l=1}^{L} \sum_{h=1}^{H} \sum_{j \in \text{Repeat}} A_{l, h}(j, i)$$
where $A_{l, h}(j, i)$ is the post-softmax attention weight from query token $j$ to key token $i$ at layer $l$ and attention head $h$.
Tokens that receive high attention weights during self-reconstruction correspond to fundamental linguistic anchors: named entities, core numerical quantities, structural syntax markers, and primary thematic clauses. Conversely, generic stylistic transitions and redundant padding receive near-zero attention.
2. Query-Agnostic Compression Properties
Because the importance score $I(i)$ is computed purely from self-supervised reconstruction of the document itself:
- Zero External Query Bias: The compressed cache is not tilted toward any specific question topic.
- Permanent Prefix Reusability: The compressed cache can be stored in vLLM’s PagedAttention block manager and matched via Radix Trees for any subsequent query from any user.
- One-Time Offline Cost: Compression is executed only once when the document is ingested. All subsequent queries enjoy instant cache hits and accelerated decoding.
3. FlashAttention Decoding Acceleration
Because KVzip prunes 66% to 75% of historical tokens, the effective sequence length during autoregressive generation shrinks from $N$ to $N/3$ or $N/4$.
In the decoding phase, self-attention memory streaming is linear in sequence length $\mathcal{O}(N)$. By halving or quartering the number of KV blocks fetched from GPU HBM per decoding step, FlashAttention kernel execution time drops by ~2x, directly reducing Time-Per-Output-Token (TPOT).
Implementing KVzip in Python and PyTorch
To demonstrate the inner workings of query-agnostic context reconstruction and token eviction, we construct a complete PyTorch implementation of the KVzip compression pipeline.
import torch
import torch.nn as nn
from typing import Tuple, List, Dict, Optional
class KVzipCompressor:
"""
Implements Query-Agnostic KV Cache Compression via Context Reconstruction.
"""
def __init__(
self,
compression_ratio: float = 4.0,
sink_token_count: int = 4,
recent_token_count: int = 64
):
self.compression_ratio = compression_ratio
self.sink_count = sink_token_count
self.recent_count = recent_token_count
def compute_reconstruction_scores(
self,
attn_weights: torch.Tensor,
repeat_start_idx: int
) -> torch.Tensor:
"""
Aggregates attention weights from the reconstruction (repeat) prompt tokens
back to historical context tokens.
Args:
attn_weights: Tensor of shape (batch, heads, seq_len, seq_len)
repeat_start_idx: Integer index where the repeat prompt starts.
Returns:
importance_scores: Tensor of shape (context_len,)
"""
# Slice attention directed from repeat queries to context keys
# repeat_queries: [repeat_start_idx : seq_len]
# context_keys: [0 : repeat_start_idx]
recon_attn = attn_weights[:, :, repeat_start_idx:, :repeat_start_idx]
# Aggregate across repeat query positions, attention heads, and batch
# Mean across heads, sum across repeat query steps
importance_scores = recon_attn.sum(dim=-2).mean(dim=1).mean(dim=0)
return importance_scores
def compress_kv_cache(
self,
k_cache: torch.Tensor,
v_cache: torch.Tensor,
importance_scores: torch.Tensor
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""
Evicts low-importance KV tokens while preserving initial attention sinks
and recent local sliding-window tokens.
Args:
k_cache: Shape (layers, batch, heads, seq_len, head_dim)
v_cache: Shape (layers, batch, heads, seq_len, head_dim)
importance_scores: Shape (seq_len,)
Returns:
compressed_k, compressed_v, retained_indices
"""
seq_len = k_cache.shape[3]
target_len = max(self.recent_count + self.sink_count, int(seq_len / self.compression_ratio))
# 1. Always retain initial sink tokens (prevent attention score distortion)
sink_indices = torch.arange(0, self.sink_count, device=k_cache.device)
# 2. Always retain recent tokens (preserve local syntax & grammar)
recent_indices = torch.arange(seq_len - self.recent_count, seq_len, device=k_cache.device)
# 3. Select top-k middle tokens based on context reconstruction importance
middle_start = self.sink_count
middle_end = seq_len - self.recent_count
middle_scores = importance_scores[middle_start:middle_end]
num_middle_to_select = target_len - (self.sink_count + self.recent_count)
num_middle_to_select = max(0, min(num_middle_to_select, middle_end - middle_start))
if num_middle_to_select > 0:
topk_relative = torch.topk(middle_scores, k=num_middle_to_select, sorted=False).indices
middle_selected = topk_relative + middle_start
else:
middle_selected = torch.empty(0, dtype=torch.long, device=k_cache.device)
# Combine all retained indices in strict chronological order
retained_indices = torch.cat([sink_indices, middle_selected, recent_indices])
retained_indices = torch.sort(retained_indices).values
# Slice compressed KV tensors
compressed_k = k_cache[:, :, :, retained_indices, :]
compressed_v = v_cache[:, :, :, retained_indices, :]
return compressed_k, compressed_v, retained_indices 2. Prefix-Caching Integration Pipeline
class KVzipPrefixCacheManager:
"""
Simulates a vLLM Radix-Tree block manager storing compressed document prefixes.
"""
def __init__(self, compressor: KVzipCompressor):
self.compressor = compressor
self.prefix_registry: Dict[str, Dict[str, torch.Tensor]] = {}
def register_document(
self,
doc_id: str,
k_full: torch.Tensor,
v_full: torch.Tensor,
attn_maps: torch.Tensor,
repeat_idx: int
):
"""
Compresses and registers a large document context once upon initial ingestion.
"""
scores = self.compressor.compute_reconstruction_scores(attn_maps, repeat_idx)
comp_k, comp_v, indices = self.compressor.compress_kv_cache(k_full, v_full, scores)
self.prefix_registry[doc_id] = {
"compressed_k": comp_k,
"compressed_v": comp_v,
"retained_indices": indices,
"original_length": k_full.shape[3],
"compressed_length": comp_k.shape[3]
}
print(f"Registered Doc '{doc_id}': Compressed {k_full.shape[3]} -> {comp_k.shape[3]} tokens "
f"({round(k_full.shape[3]/comp_k.shape[3], 1)}x compact).")
def lookup_shared_prefix(self, doc_id: str) -> Optional[Dict[str, torch.Tensor]]:
"""
Instant cache lookup for any subsequent user query.
"""
return self.prefix_registry.get(doc_id, None) Integrating this query-agnostic compression layer into automated API integration services enables enterprise teams to maintain multi-tenant document indexes in GPU memory at a fraction of the hardware cost.
Empirical Evaluation: Key Findings from the KVzip Benchmark
The NeurIPS 2025 paper presents extensive empirical evaluations across multiple open-weights foundation models—including LLaMA-3.1-8B, Qwen-2.5-14B, and Gemma-3-12B—across context lengths scaling up to 170,000 tokens on benchmarks like LongBench, RULER, Needle-In-A-Haystack, and Qasper.
1. Multi-Query Performance Retention
The critical benchmark comparing KVzip against query-dependent baselines is multi-query downstream accuracy. In this evaluation, a shared document is compressed, and multiple sequential queries are posed across subsequent turns:
Multi-Query Accuracy Across Turns (Qasper & LongBench QA, 4x Compression):
+-----------------------+------------------+------------------+------------------+
| Eviction Method | Turn 1 Accuracy | Turn 2 Accuracy | Turn 3+ Accuracy |
+-----------------------+------------------+------------------+------------------+
| Full-KV Baseline | 48.2% (1.00x) | 48.2% (1.00x) | 48.2% (1.00x) |
| SnapKV (Query-Tied) | 47.1% (97.7% ret)| 26.4% (54.7% ret)| 18.2% (37.7% ret)|
| H2O (Heavy Hitters) | 45.3% (93.9% ret)| 22.8% (47.3% ret)| 15.6% (32.3% ret)|
| KVzip (Query-Agnostic)| 46.8% (97.0% ret)| 46.2% (95.8% ret)| 45.9% (95.2% ret)|
+-----------------------+------------------+------------------+------------------+ While query-dependent methods collapse to less than 20% accuracy by Turn 3 because the required contextual facts were evicted during Turn 1, KVzip maintains over 95% of full-KV accuracy across all conversation turns.
2. Decoding Latency and VRAM Reduction
On sequence lengths extending from 32k to 170k tokens, KVzip delivers substantial computational speedups:
FlashAttention Decoding Latency & Memory Footprint (LLaMA-3.1-8B, 128k Context):
- Full-KV Baseline: 32.4 GB VRAM | 42.8 ms / token
- KVzip 3x Compression: 10.8 GB VRAM | 24.2 ms / token (1.77x faster!)
- KVzip 4x Compression: 8.1 GB VRAM | 20.9 ms / token (2.05x faster!) By compressing the KV cache by 4x, KVzip cuts memory consumption from 32.4 GB to 8.1 GB, allowing high-throughput data fetching pipelines to serve 128k context queries on single consumer GPUs.
3. Needle-In-A-Haystack Retrieval Fidelity
A classic failure point of KV compression is retrieving isolated factual needles placed arbitrarily in a 128k haystack. Because KVzip relies on context reconstruction, salient entities and key-value attributes required to reconstruct the context are identified as high-utility anchors. Across RULER and Needle-In-A-Haystack evaluations, KVzip retains over 98.2% retrieval accuracy at 3x compression.
Production Engineering Playbook: Deploying KVzip
To achieve optimal performance when integrating KVzip into production inference clusters, engineering teams should follow these implementation best practices:
Production Decision Flowchart:
[Incoming Serving Context]
│
┌─────────────┴─────────────┐
▼ ▼
Single-Turn Isolation Multi-Turn / Shared Prefix
- Query known upfront - Multi-turn conversation
- Disposable context - Enterprise shared PDF / Codebase
- Any eviction works - Use KVzip Context Reconstruction
- Store in vLLM Radix Block Manager Best Practice 1: Execute Reconstruction Prefill at Document Ingestion
Do not compute context reconstruction during user request time. In enterprise search and RAG platforms, execute KVzip compression during initial document ingestion. When an enterprise manual is indexed into your vector database, run the one-time reconstruction prefill pass and store the compressed KV blocks in your serving cluster’s distributed prefix cache.
Best Practice 2: Preserve Attention Sinks and Recent Windows
Never evict the initial 4 to 8 attention sink tokens. As demonstrated by StreamingLLM and confirmed by KVzip, initial tokens absorb massive unnormalized softmax attention mass. Dropping them causes attention score explosion and numerical instability. Always guarantee:
sink_tokens = 4recent_tokens = 64
Best Practice 3: Calibrate Compression Ratio by Content Domain
Adjust the target compression ratio based on contextual density:
- Natural Language Documents & Summaries: 4x compression ($25%$ retained) achieves near-lossless fidelity.
- Dense Source Code & JSON Schemas: 2.5x to 3x compression ($33% - 40%$ retained) is recommended to protect dense syntactic variables and function signatures.
- Mathematical Derivations: 2x compression ($50%$ retained) ensures multi-step algebraic proofs remain unbroken.
Best Practice 4: Cluster TCO and Concurrency Scaling
By reducing per-sequence KV memory by 4x, serving nodes can host 4x more concurrent user requests without requiring additional GPU hardware. Review your deployment budget with transparent infrastructure pricing, explore our comprehensive developer documentation for containerized Kubernetes manifests, or integrate automated web research using resilient web scraping tools.
Conclusion and Future Horizons
KVzip represents a vital breakthrough in memory-efficient LLM serving. By demonstrating that key-value importance can be derived from self-supervised context reconstruction rather than transient user queries, KVzip unlocks the true potential of shared-prefix caching for long-context workloads.
As models scale past 100k tokens and autonomous multi-agent systems demand continuous multi-turn interaction, query-agnostic compression architectures will serve as essential foundational infrastructure across enterprise AI.