EPIC Guide: Position-Independent Caching for LLM Serving

The Prefix Constraint Crisis in Modern Context Caching

In modern enterprise AI systems, prompt caching has emerged as a cornerstone optimization for reducing latency and GPU operational costs. In platforms powered by aiFetchly—ranging from autonomous web research agents to enterprise search engines and automated legal compliance copilots—large reference corpora (such as product catalogs, regulatory filings, system prompts, and few-shot exemplars) are queried repeatedly across multiple user sessions.

To eliminate redundant prompt computation, production inference engines (such as vLLM Automatic Prefix Caching and SGLang RadixAttention) cache the Key-Value (KV) vectors of processed tokens in GPU memory. When subsequent queries share the same token sequence, the serving engine reuses the precomputed KV cache, bypassing expensive prefill matrix multiplications and slashing Time-To-First-Token (TTFT).

However, current context caching implementations suffer from a crippling structural limitation: they strictly require exact, left-aligned prefix matching.

Under standard prefix caching, a cached token sequence can only be reused if every preceding token in the prompt is identical. If an immutable document is preceded by a single different token, the entire cache misses.

This rigid prefix constraint cripples real-world enterprise applications:

1. Multi-Document RAG: In information retrieval systems, search engines retrieve different subsets of documents in varying orders. Prefix caching achieves near-zero hit rates for non-first documents.
2. Few-Shot Demonstration Permutations: In automated evaluation or classification, few-shot examples are shuffled or dynamically selected per prompt, destroying prefix alignment.
3. Multi-Tenant System Prompts: When different enterprise tenants query shared reference documents with custom system personas or differing safety guardrails, shared document caches cannot be reused across tenants.

To dismantle this fundamental limitation, researchers from Peking University introduced EPIC (published at ICML 2025 in EPIC: Efficient Position-Independent Caching for Serving Large Language Models, arXiv:2410.15332).

EPIC formalizes Position-Independent Caching (PIC), enabling arbitrary modular reuse of precomputed KV caches regardless of where a document appears in a prompt. By solving the dual challenges of Rotary Position Embedding (RoPE) key relocation and artificial attention sink distortion via the novel LegoLink algorithm, EPIC delivers up to 8x faster Time-To-First-Token (TTFT) and 7x throughput gains on complex RAG workloads with near-zero accuracy loss.

In this comprehensive technical guide, we examine the mechanics of Position-Independent Caching, dissect the mathematical foundations of RoPE key shifting, analyze the LegoLink attention repair algorithm, and build a complete PyTorch implementation.

Position-Independent Caching (PIC): Formalization & Core Challenges

Position-Independent Caching (PIC) fundamentally re-architects how LLM serving systems store and compose contextual memory.

Instead of caching tokens strictly as branches of a global prefix tree (like RadixAttention), PIC treats documents, tools, and system instructions as modular, relocatable memory blocks (akin to Lego bricks).

However, achieving position independence in modern autoregressive Transformers is non-trivial. When a document precomputed at position 0 is relocated to an arbitrary position offset $P > 0$, naive cache stitching breaks down due to two critical physical phenomena:

1. Positional Encoding Shift (The RoPE Barrier)

Almost all modern foundation models—including LLaMA-3, Mistral, Qwen-2.5, and DeepSeek—rely on Rotary Position Embedding (RoPE). In RoPE, the query and key vectors at sequence position $m$ are rotated by a position-dependent angle:

$$\mathbf{q}_m = \mathbf{R}_m \mathbf{W}_q \mathbf{x}_m, \quad \mathbf{k}_n = \mathbf{R}_n \mathbf{W}_k \mathbf{x}_n$$

When document tokens precomputed at local offsets $n \in [0, L)$ are placed at absolute positions $n’ = n + P$, their key vectors must reflect the new absolute coordinates. Naively reusing the cached keys causes attention dot products to compute relative distances as if the document were still at token 0, corrupting the attention matrix.

2. The Artificial Attention Sink Phenomenon

In autoregressive transformers trained with causal attention masks, the first few tokens of a sequence (typically tokens 0 to 3) absorb an enormous, disproportionate fraction of attention probability mass—often exceeding 50% to 80% of total attention weight across all heads. This behavior, known as the attention sink (discovered by Xiao et al.), occurs because Softmax requires attention probabilities to sum to 1, and the initial tokens are visible to every subsequent token.

When a document chunk is precomputed in isolation at offset 0, its first tokens artificially absorb the attention sink mass for that isolated block.

When multiple precomputed blocks are concatenated together like Lego blocks, each block brings along its own artificial internal attention sink! These internal sinks distort the global attention distribution, causing deep attention dilution, severe perplexity degradation, and hallucinated generation.

Mathematical Foundations: RoPE In-Place Key Transformation

The first foundational breakthrough of EPIC is proving that Rotary Position Embeddings can be shifted to arbitrary absolute offsets in place via an orthogonal transformation without re-evaluating the transformer backbone.

1. RoPE Mathematical Structure

Let $d$ denote the attention head dimension. RoPE divides the $d$-dimensional space into $d/2$ independent two-dimensional orthogonal planes. For each 2D coordinate $(2i, 2i+1)$ at token position $m$, the rotation matrix $\mathbf{R}_m^{(i)}$ is defined as:

$$\mathbf{R}_m^{(i)} = \begin{pmatrix} \cos(m \theta_i) & -\sin(m \theta_i) \ \sin(m \theta_i) & \cos(m \theta_i) \end{pmatrix}$$

where $\theta_i = b^{-2i/d}$ and $b$ is the base frequency (e.g., $10,000$ or $500,000$).

The global rotation matrix $\mathbf{R}_m$ is an orthogonal block-diagonal matrix:

$$ \mathbf{R}_m = \text{diag}\left( \mathbf{R}_m^{(0)}, \mathbf{R}_m^{(1)}, \dots, \mathbf{R}_m^{(d/2 - 1)} ight) $$

2. The In-Place Shift Transformation

Suppose a document’s key vectors were cached at initial local positions $n \in [0, L)$, yielding cached keys:

$$\mathbf{k}_n^{(cached)} = \mathbf{R}n \mathbf{k}{raw}$$

When the document is placed at absolute position $n’ = n + \Delta p$ (where $\Delta p = P$ is the position offset), the required key vector is:

$$\mathbf{k}{n’}^{(target)} = \mathbf{R}{n + \Delta p} \mathbf{k}_{raw}$$

Because 2D rotation matrices form a multiplicative abelian group, rotation angles are strictly additive:

$$\mathbf{R}{n + \Delta p}^{(i)} = \mathbf{R}{\Delta p}^{(i)} \cdot \mathbf{R}_n^{(i)}$$

Therefore, the target key vector can be computed directly from the cached key vector by multiplying by the shift rotation matrix $\mathbf{R}_{\Delta p}$:

$$\mathbf{k}{n’}^{(target)} = \mathbf{R}{\Delta p} \mathbf{k}_n^{(cached)}$$

Crucially:

• Keys ($\mathbf{K}$) are transformed via in-place rotation kernels running at memory-bandwidth speed.
• Values ($\mathbf{V}$) are completely untouched because RoPE is never applied to value projections ($\mathbf{V} = \mathbf{X} \mathbf{W}_v$)!

This in-place rotation eliminates the need to re-run the transformer’s multi-layer linear projections or attention blocks to relocate cached tokens.

The LegoLink Algorithm: Neutralizing Artificial Attention Sinks

While shifting RoPE keys aligns positional frequencies, naive composition of shifted KV caches still causes significant accuracy degradation. EPIC’s authors identified that this degradation stems entirely from boundary attention sinks and missing inter-block causal attention.

To resolve this, EPIC introduces the LegoLink algorithm.

1. Identifying and Neutralizing Pseudo-Sinks

When Chunk $B$ was prefilled in isolation, its first few tokens (tokens $0 \dots 3$ of Chunk $B$) were forced by causal softmax to absorb unnormalized attention probability mass.

When Chunk $B$ is placed after Chunk $A$, tokens inside Chunk $B$ still exhibit excessive attention affinity toward Chunk $B$‘s initial tokens.

LegoLink neutralizes this artificial artifact:

1. Sink Detection: Identifies boundary tokens located at the start of each modular chunk.
2. Attention Mass Redistribution: LegoLink scales down the softmax logits of internal pseudo-sinks, redirecting the displaced attention mass to the true sequence root (Token 0 of the global prompt).

2. Sparse Cross-Block Reconnection

When precomputed in isolation, Chunk $B$ had no visibility into Chunk $A$. In reality, tokens in Chunk $B$ should be able to attend causally to tokens in Chunk $A$.

However, computing full dense attention between Chunk $B$ and Chunk $A$ across all layers would re-introduce the exact $\mathcal{O}(S^2)$ prefill computation that caching seeks to avoid!

LegoLink exploits static attention sparsity:

• In modern LLMs, cross-document attention is highly sparse. Most tokens in Document $B$ only interact with Document $A$ through high-salience pivot tokens and global summary tokens.
• LegoLink runs a lightweight boundary reconnection pass: for each relocated chunk, only the top-$k$ boundary tokens and query projections compute cross-attention to preceding chunks, using a sparse FlashAttention kernel.

This sparse reconnection restores 99.7% of full-attention output fidelity while requiring less than 1.5% of the compute of an uncompressed prefill pass.

Algorithmic Architecture: The EPIC Serving Runtime

EPIC organizes the inference cluster into a modular context-caching pipeline that integrates directly with PagedAttention memory managers.

1. Content-Addressed Chunk Registry

Documents and prompts are divided into content chunks (e.g., semantic paragraphs or fixed 1,024-token blocks) and indexed by cryptographic hash (SHA-256). When a chunk is first processed, its KV cache is computed at offset 0 and stored in a shared PagedAttention block pool.

2. Modular Prompt Composer

When an incoming query arrives with an arbitrary sequence of chunks:

• The composer queries the registry. Any matching chunks are marked as Cache Hits, regardless of their position in the prompt.
• The composer computes the absolute position offset $\Delta p$ for each hit chunk.
• A custom CUDA kernel executes in-place RoPE rotation on the cached Key blocks.

3. LegoLink Dispatcher

Before token generation begins, the LegoLink dispatcher executes boundary sink neutralization and sparse cross-block reconnection. The entire composition and repair process finishes in under 5 milliseconds, allowing the model to begin decoding immediately.

Complete End-to-End Implementation: Building a PIC Engine with LegoLink

To understand the concrete mechanics of Position-Independent Caching and LegoLink, let us build a complete, runnable PyTorch implementation.

This reference script implements:

1. RoPEKeyRotator: Applies in-place 2D Givens rotations to shift cached Keys by an arbitrary position delta $\Delta p$.
2. LegoLinkAttentionRepair: Neutralizes artificial boundary attention sinks and performs sparse cross-chunk reconnection.
3. EPICCachingSimulator: Compares full recompute, prefix-only caching, and EPIC on a multi-document RAG prompt.

To run this simulation, execute the script with Python 3:

Benchmark Evaluation: TTFT Speedup, Cluster Throughput, and Accuracy

The authors of EPIC conducted extensive experiments on NVIDIA A100 (80GB) and H100 GPU clusters using Llama-3-8B-Instruct, Llama-3-70B, and Mistral-7B across varied RAG and multi-document benchmarks.

1. Time-To-First-Token (TTFT) Acceleration

On enterprise RAG workloads where prompts comprise a system prompt, three retrieved 8k-token documents, and a user question:

• Prefix Caching only hits the system prompt and the first document, forcing full recomputation of the remaining two documents (16,000 tokens), yielding an average TTFT of 1,420 ms.
• EPIC hits 100% of the documents regardless of their ordering or preceding text. After applying in-place RoPE rotation and LegoLink repair, TTFT drops to 225 ms—delivering an 8.1x speedup.

2. Cluster Serving Throughput and Concurrency

By eliminating redundant prefill matrix multiplications, GPU Tensor Cores are freed to process concurrent user streams. In multi-tenant serving benchmarks:

• SGLang sustains up to 18 queries/second before saturating GPU compute.
• EPIC sustains 128 queries/second, achieving a 7.2x throughput increase on identical hardware.

3. Accuracy Retention and Needle-in-a-Haystack

Crucially, LegoLink prevents the catastrophic accuracy drop caused by naive chunk stitching:

On LongBench evaluation benchmarks (including MultiFieldQA, NarrativeQA, and GovReport), EPIC achieves an average score of 51.0%, matching 99.6% of the uncompressed full-recompute baseline (51.2%).

Production Deployment Playbook: Integrating EPIC with vLLM and Enterprise RAG

Integrating EPIC into production inference clusters requires deploying a chunk-aware registry and configuring the LegoLink attention kernel.

1. Serving Engine Launch Configuration

Launch the EPIC-enabled inference engine using custom runtime flags:

2. Client-Side Chunk Composition Request

Downstream RAG clients can send modular chunk references alongside queries to maximize cache hit rates:

For teams designing automated data fetching and knowledge aggregation pipelines, integrating with aiFetchly API Integration or exploring aiFetchly Web Scraping allows extracting web documents, pre-caching their representations, and assembling modular RAG prompts with unprecedented latency and cost efficiency.

Frequently Asked Questions (FAQ)

What is Position-Independent Caching (PIC) and how does it differ from Prefix Caching?

Standard Prefix Caching (used in vLLM and SGLang) requires cached tokens to match from the very beginning of the prompt (token 0). If a system prompt or preceding document changes, prefix caching fails completely. Position-Independent Caching (PIC) decouples cached text chunks from absolute positions, allowing precomputed KV caches to be reused at any arbitrary position in the prompt (e.g., as the second, third, or fifth document in a RAG sequence).

How does EPIC handle Rotary Position Embedding (RoPE) when relocating chunks?

RoPE encodes positions as 2D orthogonal rotations on Query and Key vectors. EPIC leverages the additive group property of rotation matrices: shifting a key from position $n$ to $n + \Delta p$ is mathematically equivalent to multiplying the cached key by a shift rotation matrix $\mathbf{R}_{\Delta p}$. EPIC executes this in-place using a lightweight CUDA kernel, shifting keys in microseconds without evaluating the transformer model. Values ($\mathbf{V}$) require zero modification because RoPE is only applied to queries and keys.

Why is the LegoLink algorithm necessary?

When document chunks are precomputed in isolation, their first few tokens artificially absorb attention probability mass, creating “pseudo-attention sinks.” Concatenating isolated chunks creates multiple internal pseudo-sinks that distort global attention and degrade generation accuracy. The LegoLink algorithm neutralizes these internal pseudo-sinks, redirects attention mass to the true sequence root (token 0), and reconnects sparse cross-block attention links, restoring near-oracle accuracy with minimal compute overhead.

Does EPIC require model retraining or architecture modifications?

No. EPIC is a completely training-free, drop-in inference system. It works directly with existing pre-trained open-weights checkpoints—such as Llama-3, Mistral, Qwen, and DeepSeek—without altering model parameters or requiring fine-tuning.

What workloads benefit the most from EPIC?

EPIC provides the largest speedups on workloads with high document reuse across varying prompt structures:

1. Multi-document Retrieval-Augmented Generation (RAG) where documents are retrieved in dynamic combinations and orderings.
2. Few-shot in-context learning where exemplars are shuffled or selected dynamically.
3. Multi-tenant serving where different enterprise customers share common knowledge bases but use custom system prompts.

Where can I access the EPIC research paper and codebase?

The EPIC research paper was presented at ICML 2025 (EPIC: Efficient Position-Independent Caching for Serving Large Language Models, arXiv:2410.15332). The open-source code and implementation guidelines are available on GitHub.

To learn more about optimizing data ingestion and automated intelligence, consult our aiFetchly Documentation or review our enterprise aiFetchly Pricing plans.