cache-craftragllm-servingkv-cachechunk-cachingsigmod-2025tutorial

Cache-Craft Guide: Chunk-Level KV Caching for RAG Serving

Cache-Craft Architecture: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation

The RAG Prefill Bottleneck: Why Prefix Caching Fails on Retrieved Knowledge

Retrieval-Augmented Generation (RAG) has established itself as the architectural backbone of enterprise intelligence copilots, automated compliance analyzers, and unstructured data ingestion platforms powered by aiFetchly. By dynamically retrieving relevant document passages from vector databases and prepending them to the user prompt, RAG grounds Large Language Models (LLMs) in verified external facts, preventing hallucination.

However, scaling RAG to high-throughput production environments introduces a crippling computational penalty: the prefill phase latency wall.

In standard production RAG pipelines (such as enterprise document assistants answering queries over corporate wikis), each query incorporates between 5 and 20 retrieved text chunks, expanding the prompt length to 4,000–32,000 tokens. In multi-tenant inference clusters, processing this massive input context—the prefill phase—accounts for up to 77% of total end-to-end execution time. Because transformer attention computational complexity scales quadratically ($O(N^2)$) with prompt length, Time-To-First-Token (TTFT) routinely balloons to dozens of seconds, degrading interactive user experience.

The RAG Prefix Caching Breakdown Dilemma: ┌────────────────────────────────────────────────────────────────────────┐ │ Query 1: Context = [ Chunk A | Chunk B | Chunk C ] ──► Question 1 │ │ • Engine computes & caches Radix prefix tree for [A -> B -> C] │ ├────────────────────────────────────────────────────────────────────────┤ │ Query 2: Context = [ Chunk D | Chunk B | Chunk A ] ──► Question 2 │ │ ❌ Chunk D does not match root Chunk A │ │ ❌ Radix Tree Prefix Cache COLLAPSES: 0% Hit Rate! │ │ ❌ Engine recomputes Chunk B & Chunk A from scratch! │ └────────────────────────────────────────────────────────────────────────┘

The Three Critical Inefficiencies of Existing RAG Caching

To accelerate prompt processing, modern inference engines (such as vLLM’s Chunked PagedAttention and SGLang’s RadixAttention) rely heavily on Prefix Caching. However, in production RAG workloads, prefix caching breaks down completely due to three structural barriers:

1. The Arbitrary Permutation Fragility of Prefix Trees
Prefix caching requires prompt tokens to match the cached sequence from left to right starting strictly from the first token (position 0). In RAG systems, however, vector retrieval ranks chunks dynamically based on query similarity. Even when two user queries retrieve the exact same set of documents, variations in query phrasing or semantic distance alter the chunk ordering (e.g., $[C_1, C_2, C_3]$ vs. $[C_3, C_1, C_2]$). Because the prefix matches fail at the very first chunk, existing engines achieve near-zero cache reuse, recomputing millions of redundant Key-Value (KV) tensors.

2. The Contextualization Drift and Attention Mask Trap
Why can’t systems simply cache and concatenate isolated KV tensors for individual text chunks? The answer lies in bidirectional contextualization and causal attention:

  • In autoregressive transformers, the Key and Value representations of token $x_j$ in Chunk $C_2$ depend strictly on all preceding tokens in the context window.
  • If Chunk $C_2$ was originally cached after Chunk $C_1$, its keys and values incorporate historical cross-attention signals from $C_1$.
  • If $C_2$ is reused behind a different prefix (e.g., $C_4$), naive KV reuse causes severe contextualization drift: the cached representations clash with the new prompt context, producing garbled syntax, hallucinated entities, and degraded output quality.

3. Rotational Positional Embedding (RoPE) Invalidation
Modern transformer foundation models (such as LLaMA-3.1, Qwen-2.5, and Mistral) apply Rotary Position Embedding (RoPE) to keys and queries based on absolute sequence indices. When a chunk appears at token offsets $[2048 - 3072]$ instead of $[0 - 1024]$, its cached rotary angles are mathematically incorrect, requiring complex rotary un-shifting.

To overcome these fundamental barriers, researchers from Adobe Research, IIT Bombay, and IIT Kanpur developed Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation (arXiv:2502.15734 / SIGMOD 2025). Cache-Craft introduces a paradigm shift: managing KV cache at the granularity of individual text chunks while repairing contextualization drift through strategic, selective recomputation.

Architectural Deep Dive: How Cache-Craft Operates

Cache-Craft replaces rigid prefix trees with a chunk-centric Key-Value management layer tightly integrated into the vLLM serving runtime.

┌─────────────────────────────────────────────────────────────────────────────┐ │ Cache-Craft System Pipeline │ │ │ │ Incoming RAG Request: Query Q + Ordered Chunks [C1, C2, C3, ... Cn] │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ CPU Metadata Hash Store & Hit Classifier │ │ │ │ Categorizes Chunks: C_hit (Precomputed KV) vs C_miss (Needs Prefill)│ │ │ └───────────────────┬───────────────────────────────┬─────────────────┘ │ │ │ │ │ │ Chunks in C_hit │ │ Chunks in C_miss │ │ ▼ ▼ │ │ ┌───────────────────────────────┐ ┌─────────────────────────────────┐ │ │ │ Reusability & Impact Scorer │ │ GPU Chunk Prefill Engine │ │ │ │ Computes Adjusted Prefix (β') │ │ Standard Forward Computation │ │ │ │ & Chunk Context Impact (CCI) │ │ Saves New KV into Chunk Cache │ │ │ └───────────────┬───────────────┘ └───────────────┬─────────────────┘ │ │ │ │ │ │ └─────────────────┬─────────────────┘ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ Selective Recomputation Engine (Top 15 - 25% Contextualized Tokens) │ │ │ │ Re-attends high-sensitivity boundary tokens to repair drift │ │ │ └─────────────────────────────────┬───────────────────────────────────┘ │ │ ▼ │ │ [ High-Throughput Token Generation ] (2.0x Lower TTFT, 95% Output Quality)│ └─────────────────────────────────────────────────────────────────────────────┘

The Cache-Craft architecture rests upon three core innovations:

1. Chunk-Level KV Storage and Metadata Hash Management

Rather than hashing entire prompt prefixes, Cache-Craft decomposes the knowledge base into discrete chunks (e.g., $256 - 1,024$ tokens each).

  • CPU-Resident Hash Table: Cache-Craft maintains a fast metadata store in pinned host CPU RAM mapping document chunk hashes to physical memory locations across CPU and GPU pools.
  • Variant Versioning: Because a chunk may be reused behind different prefix contexts, Cache-Craft maintains $N \times M$ candidate variants per chunk, dynamically evicting variants that offer low reuse frequency.
  • Asynchronous PCIe Prefetching: When a RAG query arrives via automated data fetching, Cache-Craft asynchronously streams the required chunk-caches from CPU RAM to GPU High-Bandwidth Memory (HBM) over high-speed PCIe DMA channels while the vector database finishes ranking, masking memory loading latency.

2. The Selective Recomputation Strategy

The central breakthrough of Cache-Craft is recognizing that not all tokens within a cached chunk require recomputation to maintain output fidelity:

  • The researchers discovered that inside any retrieved passage, the majority of tokens exhibit high intra-chunk attention (attending to words within the same chunk) and very low inter-chunk attention (attending to preceding documents).
  • Only a small fraction of tokens—specifically entities, introductory clauses, and transitional terms—are strongly contextualized by preceding chunks.
  • Cache-Craft formulates the Cache Fixing Overhead (CFO) by evaluating:
    1. Adjusted Prefix Overlap ($\beta’$): The degree of semantic alignment between the chunk’s current prefix and its cached historical prefix.
    2. Chunk Context Impact (CCI): The distribution of inter-chunk attention weights across transformer layers.

By strategically recomputing only the top 15% to 25% most sensitive tokens ($\mathbb{T}$), Cache-Craft completely repairs contextual drift, achieving output quality virtually indistinguishable from full recomputation while skipping 75% to 85% of prefill FLOPs.

3. Hardware-Aware Frequency-Reuse Eviction Policy

Standard eviction algorithms like Least Recently Used (LRU) or Least Frequently Used (LFU) fail for chunk-caches because they treat all memory blocks as having equal computational value.

Cache-Craft introduces Frequency-Reuse ($f_r$) Scoring:

$$f_r \mathrel{+}= \frac{1}{\text{CFO}}$$

Where $\text{CFO}$ represents the computational cost required to fix the cache. Chunks that require minimal recomputation and appear frequently across user queries receive high $f_r$ scores and are permanently pinned in GPU VRAM, while low-utility variants are evicted to host memory.

Architectural Comparison: Cache-Craft vs Existing Caching Paradigms

Architectural DimensionFull Recomputation (Vanilla vLLM)RadixTree Prefix Caching (SGLang)Prompt Caching (DeepSeek / Anthropic)Cache-Craft (SIGMOD 2025)
GranularityNone (Computes everything)Strict Prefix SequenceCoarse System PromptsIndividual Document Chunks
Permutation InvarianceN/AFails on reordered chunksFails on reordered chunks100% Permutation Invariant
Handling of DriftExact (100% compute)Exact (Identical prefix only)Exact (Static prefix only)15-25% Selective Recomputation
Computational Savings0% Baseline~10–20% on RAG~15% on RAG51% over Prefix, 75% over Full
TTFT Speedup (70B)1.0x Baseline1.15x1.2x2.3x Latency Reduction
System Throughput1.0x Baseline1.1x1.15x1.6x Higher Serving Throughput
Output Fidelity100% Baseline100%100%95% ROUGE F1 Maintained

Mathematical Formulation of Selective Recomputation

Let us formalize Cache-Craft’s contextualization scoring and token selection criteria.

Quantifying Contextualization Drift

Let $C_i$ denote a text chunk of length $|C_i|$ tokens, and let $C_{1:i-1}$ represent the sequence of preceding chunks in the prompt context.

The true hidden state of chunk $C_i$ at layer $L$ under full contextualization is denoted by $\mathbf{H}{C_i}^L\left(C{1:i}\right) \in \mathbb{R}^{|C_i| \times d_{\text{model}}}$. When chunk $C_i$ is computed in isolation without preceding context, its uncontextualized representation is $\mathbf{H}_{C_i}^L(C_i)$.

The Contextualization Error Matrix $\mathbf{E}i \in \mathbb{R}^{|C_i| \times d{\text{model}}}$ is:

$$\mathbf{E}i = \mathbf{H}{C_i}^L\left(C_{1:i}\right) - \mathbf{H}_{C_i}^L\left(C_i\right)$$

In self-attention, token $t \in C_i$ attends to preceding tokens $k \in C_{1:i-1}$ with inter-chunk attention energy:

$$\alpha_{t, k} = \frac{\exp\left( \frac{\mathbf{q}t \mathbf{k}k^T}{\sqrt{d{\text{head}}}} \right)}{\sum{j=1}^t \exp\left( \frac{\mathbf{q}_t \mathbf{k}j^T}{\sqrt{d{\text{head}}}} \right)}$$

The Chunk Context Impact (CCI) for token $t$ is the cumulative cross-attention mass directed outside the chunk:

$$\text{CCI}(t) = \sum_{k \in C_{1:i-1}} \alpha_{t, k}$$

The Optimal Recomputation Subset Selection

Given a computational budget allowing recomputation of at most $K = \lceil \gamma \cdot |C_i| \rceil$ tokens (where $\gamma \in [0.15, 0.25]$), Cache-Craft selects the optimal token subset $\mathbb{T}^* \subset C_i$ that maximizes captured context impact:

$$\mathbb{T}^* = \arg\max_{\mathbb{T} \subset C_i, ; |\mathbb{T}| \le K} \sum_{t \in \mathbb{T}} \text{CCI}(t)$$

During forward execution, the GPU attention kernel loads the cached keys and values for all tokens in $C_i \setminus \mathbb{T}^$ and performs active forward passes **only for the tokens in $\mathbb{T}^$**, updating their representations and restoring global attention distribution accuracy.

Step-by-Step Implementation: Building a Cache-Craft RAG Serving Engine

Below is a complete, self-contained Python reference implementation demonstrating Cache-Craft’s core architecture: the Chunk Metadata Store, the Context Sensitivity Scorer, and the Selective Recomputation Engine.

1. Data Models and System Definitions

""" cache_craft_engine.py Reference Implementation of Cache-Craft: Managing Chunk-Caches for RAG Serving. SIGMOD 2025 Reference Architecture. """ from dataclasses import dataclass, field from typing import List, Dict, Optional, Tuple, Set import numpy as np import time import math @dataclass class DocumentChunk: chunk_id: str text: str tokens: List[int] @property def num_tokens(self) -> int: return len(self.tokens) @dataclass class ChunkKVCache: chunk_id: str cached_keys: np.ndarray # Shape: (num_tokens, num_heads, d_head) cached_values: np.ndarray # Shape: (num_tokens, num_heads, d_head) frequency_reuse: float = 1.0 creation_time: float = field(default_factory=time.time) @dataclass class RAGInferenceRequest: query_id: str query_text: str retrieved_chunks: List[DocumentChunk] arrival_time: float = field(default_factory=time.time)

2. The Chunk Metadata & Hash Store

class ChunkMetadataStore: """ CPU-resident hash table managing precomputed chunk-caches and frequency-reuse scoring. """ def __init__(self, d_model: int = 4096, num_heads: int = 32): self.d_model = d_model self.num_heads = num_heads self.d_head = d_model // num_heads self.cache_table: Dict[str, ChunkKVCache] = {} def lookup_chunks(self, chunks: List[DocumentChunk]) -> Tuple[List[DocumentChunk], List[DocumentChunk]]: """Separates chunks into C_hit (cached) and C_miss (needs prefill).""" c_hit = [] c_miss = [] for c in chunks: if c.chunk_id in self.cache_table: c_hit.append(c) else: c_miss.append(c) return c_hit, c_miss def store_chunk_cache(self, chunk: DocumentChunk): """Simulates precomputing and storing a new chunk-cache.""" # Simulated FP16 Key/Value matrices k_tensor = np.random.randn(chunk.num_tokens, self.num_heads, self.d_head).astype(np.float32) * 0.05 v_tensor = np.random.randn(chunk.num_tokens, self.num_heads, self.d_head).astype(np.float32) * 0.05 self.cache_table[chunk.chunk_id] = ChunkKVCache(chunk.chunk_id, k_tensor, v_tensor) def update_reuse_score(self, chunk_id: str, cfo: float): """Updates frequency-reuse score f_r += 1 / CFO.""" if chunk_id in self.cache_table: self.cache_table[chunk_id].frequency_reuse += (1.0 / max(0.1, cfo))

3. The Selective Recomputation Engine

class SelectiveRecomputationEngine: """ Identifies the top 15-25% most context-sensitive tokens and repairs KV drift. """ def __init__(self, recompute_ratio: float = 0.20): self.recompute_ratio = recompute_ratio def identify_context_sensitive_tokens(self, chunk: DocumentChunk, preceding_tokens_count: int) -> List[int]: """ Simulates calculating Chunk Context Impact (CCI) for each token in chunk. Tokens near chunk boundaries and key entities exhibit highest sensitivity. """ n = chunk.num_tokens num_to_recompute = max(1, int(n * self.recompute_ratio)) # Synthetic sensitivity curve: boundary tokens have higher inter-chunk attention sensitivity_scores = [] for idx in range(n): # Positional factor: tokens at the start and end of chunk attend more to preceding context pos_weight = math.exp(-idx / 16.0) + (0.3 if idx % 5 == 0 else 0.05) sensitivity_scores.append((pos_weight, idx)) # Sort by sensitivity descending sensitivity_scores.sort(key=lambda x: x[0], reverse=True) tokens_to_recompute = [idx for _, idx in sensitivity_scores[:num_to_recompute]] return tokens_to_recompute def execute_selective_prefill( self, request: RAGInferenceRequest, metadata_store: ChunkMetadataStore ) -> Dict[str, float]: """ Executes Cache-Craft inference plan: 1. Retrieves cached chunks from C_hit. 2. Computes uncached chunks in C_miss. 3. Recomputes top 20% tokens for C_hit to repair contextual drift. """ c_hit, c_miss = metadata_store.lookup_chunks(request.retrieved_chunks) total_tokens = sum(c.num_tokens for c in request.retrieved_chunks) recomputed_tokens_count = 0 prefilled_tokens_count = 0 preceding_count = 0 start_time = time.perf_counter() # Step A: Handle C_miss (Full prefill required) for chunk in c_miss: prefilled_tokens_count += chunk.num_tokens preceding_count += chunk.num_tokens metadata_store.store_chunk_cache(chunk) # Step B: Handle C_hit (Selective 20% recomputation) for chunk in c_hit: tokens_to_fix = self.identify_context_sensitive_tokens(chunk, preceding_count) recomputed_tokens_count += len(tokens_to_fix) preceding_count += chunk.num_tokens # Update frequency reuse score cfo_score = len(tokens_to_fix) / chunk.num_tokens metadata_store.update_reuse_score(chunk.chunk_id, cfo_score) elapsed_ms = (time.perf_counter() - start_time) * 1000.0 # Calculate computation savings standard_full_prefill = total_tokens cache_craft_active_compute = prefilled_tokens_count + recomputed_tokens_count savings_percent = ((standard_full_prefill - cache_craft_active_compute) / max(1, standard_full_prefill)) * 100.0 return { "total_prompt_tokens": total_tokens, "c_hit_chunks": len(c_hit), "c_miss_chunks": len(c_miss), "recomputed_tokens": recomputed_tokens_count, "actual_tokens_computed": cache_craft_active_compute, "computation_savings_percent": savings_percent, "planning_latency_ms": elapsed_ms }

4. Running a Production RAG Serving Benchmark

if __name__ == "__main__": print("=== Cache-Craft RAG Serving Engine Benchmark ===") meta_store = ChunkMetadataStore(d_model=4096, num_heads=32) engine = SelectiveRecomputationEngine(recompute_ratio=0.20) # 20% selective recomputation # Pre-seed popular chunks into knowledge base cache (e.g. system docs) doc_corpus = [ DocumentChunk(f"doc_chunk_{i:02d}", f"Text content of legal chunk {i}", [100 + i] * 512) for i in range(10) ] # Warm 6 of the 10 chunks in cache for chunk in doc_corpus[:6]: meta_store.store_chunk_cache(chunk) # Simulate an incoming RAG query retrieving chunks in arbitrary permutation [doc_04, doc_01, doc_08, doc_03] incoming_rag_query = RAGInferenceRequest( query_id="query_compliance_01", query_text="Summarize audit liabilities under GDPR Section 4", retrieved_chunks=[doc_corpus[4], doc_corpus[1], doc_corpus[8], doc_corpus[3]] ) results = engine.execute_selective_prefill(incoming_rag_query, meta_store) print("\n--- Benchmark Execution Results ---") print(f"Total Input Tokens: {results['total_prompt_tokens']} tokens") print(f"Chunks Reused (C_hit): {results['c_hit_chunks']} chunks") print(f"Chunks Computed (C_miss): {results['c_miss_chunks']} chunks") print(f"Tokens Selectively Repaired: {results['recomputed_tokens']} tokens (20% of C_hit)") print(f"Net Active Tokens Processed: {results['actual_tokens_computed']} tokens") print(f"Computational Work Saved: {results['computation_savings_percent']:.1f}% Reduction!") print(f"System Planning Latency: {results['planning_latency_ms']:.3f} ms")

Production Deployment: Integrating Cache-Craft with vLLM and LangChain

In enterprise microservices architectures, Cache-Craft operates as an extended memory backend inside vLLM worker engines.

┌──────────────────────────────────────────────────────────────────────────────────┐ │ Enterprise RAG Serving Mesh │ │ │ │ RAG Orchestrator (LangChain / LlamaIndex / Semantic Kernel) │ │ Retrieves vector chunks from Pinecone / Milvus / pgvector │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ │ │ vLLM Engine with Cache-Craft Plugin (Port 8000) │ │ │ │ • Chunk Metadata Hash Store in Host CPU Memory │ │ │ │ • Asynchronous PCIe Chunk-Cache Loader Stream │ │ │ │ • Fused FlashAttention Kernel with Selective Token Masking │ │ │ └──────────────────────────────────┬───────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ │ │ Physical GPU Cluster (NVIDIA H100 80GB) │ │ │ │ 2.3x Faster TTFT • 1.6x Higher Serving Concurrency • Zero Drift │ │ │ └──────────────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────────────┘

1. vLLM Engine Configuration

apiVersion: apps/v1 kind: Deployment metadata: name: vllm-cachecraft-worker namespace: rag-serving spec: replicas: 2 template: metadata: labels: app: vllm-worker spec: containers: - name: vllm image: vllm/vllm-openai:latest env: - name: ENABLE_CACHECRAFT_CHUNK_CACHE value: 'true' - name: CACHECRAFT_RECOMPUTE_RATIO value: '0.20' - name: CPU_METADATA_STORE_SIZE_GB value: '32' args: - '--model=meta-llama/Meta-Llama-3.1-70B-Instruct' - '--tensor-parallel-size=4' - '--enable-chunked-prefill=true' resources: limits: nvidia.com/gpu: 4

Organizations automating distributed web data ingestion can pipe scraped and chunked documents directly into Cache-Craft memory pools using aiFetchly API Integration and aiFetchly Web Scraping Workflows.

Empirical Benchmark Evaluation on Real-World Workloads

Cache-Craft was rigorously evaluated across two enterprise production RAG systems at Adobe (Sys-X and Sys-Y) as well as open multi-hop question answering benchmarks (2WikiMQA, MuSiQue, and HotpotQA) using LLaMA-3-8B and LLaMA-3-70B:

Time-To-First-Token (TTFT) Latency on LLaMA-3-70B (Prompt = 8k tokens): Full Recomputation: ████████████████████ 3,850 ms Prefix-Caching (SOTA): ████████████████ 3,120 ms Cache-Craft (SIGMOD '25):████████ 1,670 ms (2.3x Latency Reduction!) Output Quality (ROUGE-F1 Score vs Full Recomputation): Cache-Craft (20% Recompute): ███████████████████ 94.8% Quality Preserved Naive KV Reuse (0% Recompute):██████████ 48.2% (Severe Hallucinations!)

Key Performance Findings:

  1. 2.0x to 2.3x TTFT Latency Reduction: On LLaMA-3-70B, Time-To-First-Token latency is slashed from $3.85\text{ s}$ down to $1.67\text{ s}$ by bypassing 75% of prompt prefill matrix operations.
  2. 51% Computation Reduction over Prefix Caching: Because Cache-Craft reuses chunks regardless of their position or ordering in the prompt, it achieves massive cache hit rates where prefix trees fail.
  3. 1.6x Higher Serving Throughput: Reclaiming GPU prefill compute cycles allows clusters running continuous batching to service 40% to 60% more user queries per minute on identical GPU clusters.
  4. 95% Output Quality Maintained: By selectively recomputing the top 15-25% most context-sensitive tokens, output responses remain indistinguishable from expensive full recomputation.

For detailed hardware configuration tables and enterprise sizing models, visit the aiFetchly Documentation and explore cluster cost savings on the aiFetchly Sizing & Pricing Portal.

Frequently Asked Questions (FAQ)

How is Cache-Craft different from standard prefix caching like SGLang’s RadixAttention?

Prefix caching requires prompt tokens to match the cached sequence from position 0 in exact linear order. If chunks are reordered (e.g., $[C_2, C_1]$ instead of $[C_1, C_2]$), prefix caching yields a 0% hit rate. Cache-Craft caches at the granularity of individual chunks, allowing documents to be reused at arbitrary positions with arbitrary neighbors.

Why does naive chunk-cache reuse degrade model output quality?

Autoregressive transformers utilize causal attention and Rotary Position Embeddings (RoPE). When a chunk is reused without fixing, its keys and values reflect previous contexts and stale token positions, distorting inter-token attention scores and causing severe hallucination loops.

What is the ideal recomputation ratio for production workloads?

Extensive empirical testing reveals that recomputing 15% to 25% of tokens represents the optimal trade-off: it achieves over 95% of full-recomputation ROUGE-F1 accuracy while eliminating 75% of prefill execution latency.