AttentionStoreKV CacheMulti-Turn ServingDisaggregated StorageLLM SystemsInference Optimization

AttentionStore Guide: Multi-Turn KV Cache Architecture

October 14, 2026

The Multi-Turn Conversational Memory Tax

Conversational agents, multi-step code generation copilots, automated customer service representatives, and autonomous data-fetching pipelines represent the fastest-growing categories of enterprise AI deployments. Unlike single-turn question-answering systems, these applications operate across extended, multi-turn dialogue histories.

In a typical customer support session or developer debugging loop, a conversation evolves across 10 to 50 sequential turns. With each new turn, the client submits a new query that appends to the preceding conversational context. By Turn 15, the prompt fed into the Large Language Model (LLM) includes:

System instructions and developer guidelines.
Grounded retrieved knowledge (RAG documents and database schemas).
All previous user inputs and model assistant responses.

To generate a response to the latest query, standard LLM serving systems must process the entire accumulated conversation history. This creates an operational dilemma known as the Multi-Turn Memory Tax:

The Recomputation Penalty: If the serving engine does not retain the key-value (KV) cache from previous turns, it must re-execute the prefill phase over the entire historical context for every single new user interaction. For a 32k-token conversation history, recomputing the KV cache burns tens of trillions of floating-point operations (FLOPs), causing Time-to-First-Token (TTFT) latency to explode from 200 milliseconds to over 6 seconds.
The In-VRAM Pinning Penalty: To avoid recomputation, early serving frameworks attempted to pin the KV cache in GPU High-Bandwidth Memory (HBM) across conversational turns. However, human users take between 10 seconds and several minutes to read a response, think, and compose their next message. Pinning a 10 GB KV cache in scarce $30,000 GPU memory while waiting for human typing severely starves GPU capacity, drops concurrent serving capacity by over 80%, and forces companies to massively overprovision GPU clusters.

Existing caching systems like SGLang’s RadixAttention or vLLM’s automatic prefix caching mitigate this within single GPU nodes using local Least-Recently-Used (LRU) evictions in GPU memory. However, when memory pressure spikes or when multi-node clusters route user turns to different physical servers, local VRAM caches are evicted or missed entirely.

To resolve this fundamental trade-off between compute recomputation and memory waste, systems researchers introduced AttentionStore (published by Bin Gao, Zhuomin Rao, and collaborators).

AttentionStore establishes a hierarchical disaggregated KV cache storage architecture. By decoupling KV cache persistence from active GPU workers into a multi-tiered storage pool (GPU HBM, Host CPU DRAM, and distributed NVMe SSDs connected over RDMA), and pairing this architecture with look-ahead prefetching and layer-wise asynchronous loading, AttentionStore slashes multi-turn TTFT by up to 86% (a 7.2x speedup), cuts GPU memory consumption by 80%, and reduces overall cluster operational costs by over 60%.

The Architectural Bottleneck: Why In-VRAM Prefix Caching Fails

To appreciate how AttentionStore transforms conversational serving, consider what happens when a multi-turn user interacts with a standard LLM cluster running vLLM or TensorRT-LLM:

+-------------------------------------------------------------------------+
|                  Standard Single-Node In-VRAM Prefix Caching            |
|                                                                         |
| Turn 1: User asks query ──► Prefill ──► Output generated ──► Cache in HBM|
|                                                                         |
| [ Human Idle Window: 45 seconds while user reads and types ]            |
|                                                                         |
| Scenario A: High-Concurrency Spike Arrives!                             |
| - GPU HBM reaches 95% saturation.                                       |
| - LRU Eviction triggers: The idle user's 16k KV cache is DROPPED!       |
|                                                                         |
| Turn 2: User sends follow-up ──► CACHE MISS!                            |
| - Must recompute all 16k tokens from scratch!                           |
| - TTFT spikes to 4,800 ms (Latency SLA Violated).                       |
|                                                                         |
| Scenario B: Load Balancer Routes Turn 2 to Server Node 2!               |
| - Server 2 has no knowledge of Server 1's local VRAM cache.             |
| - CACHE MISS! Full recomputation required!                              |
+-------------------------------------------------------------------------+
                                    vs.
+-------------------------------------------------------------------------+
|                  AttentionStore Hierarchical Disaggregated Cache         |
|                                                                         |
| Turn 1: Output generated ──► Asynchronously Saved to Disaggregated Pool |
|                                                                         |
| [ Human Idle Window: 45 seconds ]                                       |
| - GPU HBM immediately FREED for active jobs! Zero idle memory waste.   |
| - KV cache safely staged in Host DRAM / Distributed NVMe via RDMA.      |
|                                                                         |
| Turn 2: Request arrives at Ingress Gateway (Any Server Node)            |
| - Prefix-Hash Tree instantly matches conversation ID.                   |
| - Look-Ahead Prefetching streams KV blocks to GPU HBM layer-by-layer.   |
| - TTFT = 380 ms (86% Latency Reduction across Multi-Turn sessions).     |
+-------------------------------------------------------------------------+

Single-tier in-VRAM caching suffers from three critical deficiencies:

Capacity Mismatch: An 80GB GPU node can hold the active weights of a 70B model plus only 10 to 15 concurrent 32k-token KV caches. In a multi-tenant cloud application serving thousands of concurrent chat sessions, 95% of conversational caches are evicted before the user sends their next message.
Cluster Routing Fragmentation: In elastic multi-server clusters, an ingress load balancer routes requests based on real-time server utilization. When Turn 1 executes on Server A and Turn 2 is dispatched to Server B, Server B experiences a complete cache miss, rendering Server A’s cached state useless.
Synchronization Latency: Synchronously fetching multi-gigabyte KV caches over standard network protocols stalls the GPU engine, introducing pipeline bubbles that negate the latency savings of caching.

Core Innovation 1: Hierarchical Disaggregated KV Storage Architecture

AttentionStore resolves these structural limitations by separating compute from memory persistence. Rather than treating GPU VRAM as the sole storage medium for attention history, AttentionStore establishes an elastic, three-tier storage hierarchy:

Tier 1: GPU Cluster Memory (Active Working Pool)

GPU High-Bandwidth Memory (HBM) is reserved strictly for active computation: model weights, activation buffers, and the KV cache blocks of currently executing forward passes. As soon as a request finishes generating its final token, its KV cache blocks are marked for asynchronous evacuation, freeing HBM for waiting requests.

Tier 2: Host CPU DRAM (Mid-Tier Staging Buffer)

Each GPU server utilizes its high-capacity host system memory (typically 512 GB to 2 TB of DDR5 RAM) as a low-latency secondary cache. Transferring KV cache pages between host DRAM and GPU HBM across the local PCIe bus (or NVLink-C2C on superchips) proceeds at 30 GB/s to 63 GB/s, allowing multi-gigabyte caches to be swapped in hundreds of milliseconds.

Tier 3: Distributed NVMe Storage Pool (Disaggregated Elastic Tier)

For conversations spanning extended idle times or distributed across multi-node clusters, AttentionStore pools local NVMe SSDs across the datacenter into a shared, disaggregated object store connected via Remote Direct Memory Access (RDMA over Converged Ethernet - RoCE v2 or InfiniBand).

Any GPU worker node in the cluster can fetch or store conversation blocks from any storage node with microsecond access times, eliminating cluster routing fragmentation.

Core Innovation 2: Look-Ahead Prefetching & Layer-Wise Asynchronous Loading

Moving multi-gigabyte KV caches across storage tiers introduces potential I/O latency. If a GPU engine waits for an entire 10 GB cache to transfer before starting computation, Time-to-First-Token is penalized.

AttentionStore completely eliminates this I/O overhead using a two-pronged execution strategy:

1. Look-Ahead Prefetching

In production LLM serving, incoming requests do not arrive in isolation; they are enqueued in an ingress scheduling queue. AttentionStore monitors the look-ahead scheduling window.

While the GPU is busy executing the final decoding tokens of currently running Batch $N$, AttentionStore inspects the waiting jobs in Queue $N+1$. It calculates the prefix hash of the waiting requests, queries the global cache index, and begins prefetching matching historical KV blocks into Host DRAM and GPU HBM before the previous batch has even finished executing.

+-------------------------------------------------------------------------+
|              AttentionStore Layer-Wise Asynchronous Streaming           |
|                                                                         |
| Time (t) ─────────────────────────────────────────────────────────────► |
|                                                                         |
| GPU Compute:     [ Layer 1 Prefill ] [ Layer 2 Prefill ] [ Layer 3 ... ]|
|                                                                         |
| DMA / RDMA Pipe: [ Stream Layer 2 ]  [ Stream Layer 3 ]  [ Stream L4...]|
|                                                                         |
| The I/O latency of Layer (L+1) is completely hidden behind Layer L math!|
+-------------------------------------------------------------------------+

2. Layer-Wise Asynchronous Loading

A Transformer model evaluates attention sequentially, layer by layer (Layer 1 $\to$ Layer 2 $\to \dots \to$ Layer $L$). To compute Layer 1’s attention output, the GPU only needs Layer 1’s key and value tensors; it does not need Layer 20’s tensors until tens of milliseconds later.

AttentionStore exploits this temporal pipeline by loading KV cache tensors layer-by-layer on dedicated asynchronous CUDA copy streams:

While the GPU computes Layer 1’s self-attention, the DMA controller transfers Layer 2’s KV cache blocks from host DRAM or NVMe storage into GPU HBM.
While the GPU evaluates Layer 2, Layer 3 streams in.

By the time the execution reach reaches each subsequent layer, its required KV tensors are already resident in VRAM. The network and PCIe transfer latency is 100% overlapped and hidden behind ongoing compute.

Mathematical Modeling: Prefetching Windows and TTFT Bounds

To evaluate the mathematical conditions under which AttentionStore achieves zero-overhead retrieval, consider the timing constraints of the layer-wise prefetching pipeline.

Let:

$L$ denote the total number of layers in the model (e.g., $L = 64$ for LLaMA-3-70B).
$T_{\text{layer_comp}}$ denote the compute duration of a single Transformer layer during prefill for the newly appended prompt tokens ($N_{\text{new}}$).
$S_{\text{layer_kv}}$ denote the size (in bytes) of a single layer’s historical KV cache for the shared conversation context ($N_{\text{history}}$).
$B_{\text{transfer}}$ denote the effective sustained transfer bandwidth between the storage tier and GPU VRAM (e.g., PCIe Gen 5 at 63 GB/s or RoCE RDMA at 25 GB/s).

1. Single-Layer Transfer Time ($T_{\text{layer_trans}}$)

For historical context length $N_{\text{history}}$, attention heads $H_{kv}$, head dimension $d_{head}$, and precision bytes $P$:

$$S_{\text{layer_kv}} = 2 \times H_{kv} \times d_{head} \times N_{\text{history}} \times P$$

$$T_{\text{layer_trans}} = \frac{S_{\text{layer_kv}}}{B_{\text{transfer}}}$$

2. The Zero-Stall Overlap Criterion

To ensure the GPU never stalls waiting for historical KV cache blocks, the computation time of layer $l$ must equal or exceed the transfer time of layer $l+1$:

$$T_{\text{layer_comp}}(N_{\text{new}}) \ge T_{\text{layer_trans}}(N_{\text{history}})$$

If the new user query is short ($N_{\text{new}} = 64$ tokens), layer compute time is small (~0.5 ms). If the historical context is massive ($N_{\text{history}} = 32,768$ tokens), streaming a single layer’s cache ($S_{\text{layer_kv}} \approx 4.19 \text{ MB}$) over PCIe Gen 5 ($B_{\text{transfer}} \approx 63 \text{ GB/s}$) takes:

$$T_{\text{layer_trans}} = \frac{4.19 \times 10^6}{63 \times 10^9} \approx 0.066 \text{ ms (66 microseconds)}$$

Because $T_{\text{layer_trans}} (0.066\text{ ms}) \ll T_{\text{layer_comp}} (0.5\text{ ms})$, the transfer completes nearly 8x faster than the compute time, easily satisfying the overlap condition.

The entire 32k-token historical KV cache is streamed into the GPU across the model’s 64 layers with zero exposed I/O latency.

TTFT Comparison Across Serving Paradigms

For a 32k-token conversation history with a 128-token user follow-up on LLaMA-3-70B:

Serving Paradigm	Storage Mechanism	Historical Context Processing	Exposed Transfer Latency	Total Time-to-First-Token (TTFT)
Naive Recomputation	None (Drop cache)	Full 32k Forward Pass	0 ms	4,850 ms (4.85 seconds)
Synchronous Host Swap	CPU DRAM	Prefill 128 tokens only	310 ms (Synchronous PCIe)	580 ms
In-VRAM LRU (Hit)	GPU HBM (Pinned)	Prefill 128 tokens only	0 ms (Already in VRAM)	210 ms (High VRAM Waste)
AttentionStore	Hierarchical Disaggregated	Prefill 128 tokens only	0 ms (Layer-Wise Overlap)	225 ms (86% TTFT Reduction)

AttentionStore matches the latency performance of pinning caches in scarce GPU VRAM, but achieves it while storing the data on cost-effective disaggregated storage.

Core Innovation 3: Prefix-Hash Tree Indexing and Read-Eviction Policies

To coordinate millions of cached conversation blocks across distributed nodes, AttentionStore implements a high-performance Prefix-Hash Tree Index:

+-------------------------------------------------------------------------+
|                  AttentionStore Prefix-Hash Tree Index                  |
|                                                                         |
|                          [ Root System Prompt ]                         |
|                             Hash: 0x8F3A...                             |
|                                    │                                    |
|                   ┌────────────────┴────────────────┐                   |
|                   ▼                                 ▼                   |
|         [ Turn 1: User Query A ]          [ Turn 1: User Query B ]      |
|             Hash: 0xC1D4...                   Hash: 0x5E9B...           |
|                   │                                 │                   |
|                   ▼                                 ▼                   |
|        [ Turn 2: Assistant Resp ]         [ Turn 2: Assistant Resp ]    |
|             Hash: 0x99A2...                   Hash: 0x44B1...           |
|                                                                         |
| Node Metadata: Pointer to Tier (HBM / Host DRAM / Distributed NVMe)     |
+-------------------------------------------------------------------------+

1. Deterministic Prefix Hashing

Conversations are tokenized and organized into a hierarchical prefix tree where each node represents a fixed-size chunk of tokens (e.g., 64 tokens) identified by a cryptographic hash of its token IDs and system configuration. When a new turn arrives, AttentionStore walks the tree from root to leaf, identifying the longest matching historical prefix in $O(K)$ time (sub-millisecond).

2. Multi-Tier Eviction and Cost-Aware Demotion

AttentionStore implements an intelligent tiered eviction policy based on access frequency and turn recency:

Active Hot Caches: Retained in Host DRAM while a chat session is actively exchanging messages.
Warm Demotion: If no new turn arrives within 3 minutes, the KV blocks are demoted from Host DRAM to the local NVMe storage pool via direct asynchronous I/O.
Cold Eviction: If a session remains inactive past a configurable TTL (e.g., 2 hours), the NVMe blocks are pruned or archived to object storage.

Architectural Comparison: AttentionStore vs Existing Serving Engines

The table below contrasts AttentionStore against contemporary caching and serving solutions:

Feature / Metric	vLLM (Standard)	SGLang (RadixAttention)	CacheGen (Compression)	AttentionStore (EuroSys)
Cache Storage Medium	GPU VRAM Only	GPU VRAM Only	Compressed Stream over Net	Tiered: HBM + DRAM + NVMe
Multi-Node Sharing	None (Node Silos)	None (Local Tree)	Peer-to-Peer Transfer	Disaggregated RDMA Pool
Cache Retention Policy	Local LRU in VRAM	Local LRU in VRAM	Network Streaming	Look-Ahead + Multi-Tier Demotion
I/O Latency Hiding	Synchronous	In-VRAM (No Transfer)	Sequential Streaming	Layer-Wise Asynchronous Overlap
GPU Memory Waste (Idle)	Extreme (High pinning)	Extreme (High pinning)	Low	Zero (Immediate Evacuation)
Multi-Turn TTFT Speedup	Baseline (1.0x)	3.5x (VRAM hit only)	2.8x	7.2x (86% Reduction)
Infrastructure Cost	100% (High VRAM floor)	~85% (VRAM bound)	~70%	~40% (60% Cost Reduction)

While RadixAttention provides excellent prefix caching within the limits of a single GPU’s HBM, AttentionStore scales attention reuse across the entire datacenter infrastructure.

Step-by-Step Implementation: Building a Multi-Tier KV Cache Manager

The following Python listing demonstrates the core mechanisms of AttentionStore, implementing a hierarchical cache directory, look-ahead prefetch triggers, and layer-wise streaming queues.

import asyncio
import hashlib
import time
from typing import Dict, List, Optional, Tuple

class CacheTier:
    GPU_HBM = "GPU_HBM"
    HOST_DRAM = "HOST_DRAM"
    NVME_SSD = "NVME_SSD"

class KVBlockDescriptor:
    def __init__(self, block_hash: str, token_ids: List[int], layer_id: int):
        self.block_hash = block_hash
        self.token_ids = token_ids
        self.layer_id = layer_id
        self.current_tier = CacheTier.HOST_DRAM
        self.gpu_address: Optional[int] = None
        self.host_address: Optional[int] = 0x1000
        self.nvme_offset: Optional[int] = None
        self.last_accessed = time.time()

class AttentionStoreManager:
    '''Hierarchical Disaggregated KV Cache Store with Layer-Wise Prefetching.'''
    def __init__(self, num_layers: int = 32, pcie_bandwidth_gbps: float = 63.0):
        self.num_layers = num_layers
        self.pcie_bw = pcie_bandwidth_gbps * 1e9 # Bytes/sec
        self.prefix_index: Dict[str, KVBlockDescriptor] = {}
        self.hbm_allocated_blocks = 0
        self.max_hbm_blocks = 1000

    def compute_prefix_hash(self, token_ids: List[int]) -> str:
        data = ",".join(map(str, token_ids)).encode('utf-8')
        return hashlib.sha256(data).hexdigest()[:16]

    def register_block(self, token_ids: List[int], layer_id: int) -> str:
        b_hash = self.compute_prefix_hash(token_ids)
        descriptor = KVBlockDescriptor(b_hash, token_ids, layer_id)
        self.prefix_index[f"{b_hash}_L{layer_id}"] = descriptor
        return b_hash

    async def prefetch_layer_kv(self, block_hash: str, layer_id: int, stream_id: int):
        '''Asynchronously streams a specific layer's KV block from Host DRAM to GPU HBM.'''
        key = f"{block_hash}_L{layer_id}"
        descriptor = self.prefix_index.get(key)
        if not descriptor:
            return None # Cache miss, must compute

        if descriptor.current_tier == CacheTier.HOST_DRAM:
            # Emulate non-blocking DMA transfer over PCIe
            block_bytes = len(descriptor.token_ids) * 2 * 128 * 2 # 16-bit float KV
            transfer_duration = block_bytes / self.pcie_bw
            await asyncio.sleep(transfer_duration)
            descriptor.current_tier = CacheTier.GPU_HBM
            descriptor.gpu_address = 0x8000 + layer_id * 0x100
            descriptor.last_accessed = time.time()
            return descriptor.gpu_address

    async def execute_layer_wise_pipeline(self, prompt_tokens: List[int]) -> float:
        '''Simulates running prefill while streaming KV caches layer-by-layer.'''
        start_time = time.perf_counter()
        b_hash = self.compute_prefix_hash(prompt_tokens)

        # Launch Layer 1 prefetch immediately
        prefetch_task = asyncio.create_task(self.prefetch_layer_kv(b_hash, layer_id=0, stream_id=1))

        for layer in range(self.num_layers):
            # Await the current layer's KV transfer
            await prefetch_task

            # Start prefetching the NEXT layer concurrently with current layer computation
            if layer + 1 < self.num_layers:
                prefetch_task = asyncio.create_task(
                    self.prefetch_layer_kv(b_hash, layer_id=layer+1, stream_id=1)
                )

            # Emulate layer forward computation
            await asyncio.sleep(0.0005) # 500 microseconds compute

        return time.perf_counter() - start_time

Production Benchmarks: Empirical Evaluation

The AttentionStore paper provides rigorous empirical benchmarks evaluated across synthetic multi-turn traces, real-world ShareGPT conversations, and LMSYS Chatbot Arena datasets using LLaMA-2/3 models.

1. Multi-Turn TTFT Latency Reduction Across Turns

Evaluating a 20-turn conversational session on LLaMA-3-70B where context length expands from 1,024 tokens to 32,768 tokens:

Conversational Turn	Context Length	vLLM (Recomputation)	SGLang (In-VRAM LRU)	AttentionStore (Disaggregated)	TTFT Latency Reduction
Turn 1 (Intro)	1,024 tokens	120 ms	120 ms	120 ms	Baseline
Turn 5 (Discussion)	8,192 tokens	1,240 ms	210 ms (Hit) / 1,240 (Miss)	215 ms	-82.6%
Turn 10 (Analysis)	16,384 tokens	2,480 ms	230 ms (Hit) / 2,480 (Miss)	220 ms	-91.1%
Turn 20 (Deep Q&A)	32,768 tokens	4,850 ms	260 ms (Hit) / 4,850 (Miss)	225 ms	-95.3%

Time-to-First-Token (TTFT) Across Multi-Turn Dialogue

5,000ms |------------------------------------------------------------- (vLLM Recompute: 4,850ms)
        |                                                           //
3,000ms |----------------------------------------------------------//-
        |                                                        //
1,000ms |-------------------------------------------------------//----
        |                                                     //
  250ms |====================================================//======= (AttentionStore: ~225ms)
        |_____________________________________________________________
        Turn 1           Turn 5          Turn 10         Turn 20

Under vLLM without persistent caching, TTFT grows linearly with conversation length, reaching nearly 5 seconds by Turn 20. Under AttentionStore, TTFT remains flat at ~225 milliseconds throughout the entire 20-turn session.

2. GPU Memory Footprint and Cluster Throughput

By immediately offloading idle conversational caches:

GPU HBM Consumption: Slashed by 80%, allowing nodes to dedicate over 85% of physical memory to active token generation batches.
Cluster Capacity: Accommodates up to 5.2x more concurrent multi-turn chat sessions on identical hardware infrastructure.
Operational Serving Costs: Slashed by 62.8% by replacing high-end GPU nodes with high-density CPU DRAM and NVMe storage pools.

Operational Checklist: Deploying AttentionStore in Enterprise Clusters

Engineering teams integrating AttentionStore into large-scale conversational inference clusters should verify system readiness against this operational checklist:

RoCE v2 / InfiniBand Fabric Verification: Verify that all GPU worker nodes and storage nodes communicate over non-blocking RDMA fabrics with GID indices and RoCE PFC/ECN enabled:

ibv_devinfo -v rdma link show
Look-Ahead Queue Window Tuning: Configure the look-ahead scheduling window size to 3–5 batches. Setting the window too small starves prefetching, while setting it too large risks prefetching queries that are canceled.
Host DRAM Allocation Limits: Reserve between 256 GB and 512 GB of host system RAM per 8-GPU node as the Tier 2 staging buffer, pinning buffers with mlock to prevent OS paging stalls.
Prefix Tree Hashing Granularity: Set token chunk hashing granularity between 32 and 64 tokens. Finer granularity increases tree indexing overhead; coarser granularity reduces prefix match rates.
Prometheus Telemetry Dashboard: Instrument metrics tracking attentionstore_tier_hit_ratio (HBM vs DRAM vs NVMe), attentionstore_layer_prefetch_stall_ms, and attentionstore_eviction_throughput_bytes.
Cost Governance: Cross-reference storage scaling targets with your internal docs and align cluster capacity with enterprise infrastructure pricing.

Strategic Outlook: The Era of Disaggregated AI Memory

AttentionStore marks a fundamental paradigm shift in the architecture of generative artificial intelligence. In early AI infrastructure, developers treated GPUs as closed, self-contained silos responsible for compute, temporary activations, and long-term conversational memory simultaneously.

By demonstrating that the key-value cache can be disaggregated into an external, multi-tier storage fabric, AttentionStore bridges the gap between high-performance computing and enterprise database systems.

As enterprise AI transitions from simple one-shot prompts to autonomous agent loops connecting live api-integration connectors, automated web-scraping pipelines, and continuous conversational state, disaggregated memory architectures like AttentionStore will become the industry standard for production serving.

Discover more deep dives, technical tutorials, and AI systems analysis on our homepage as we continue tracking the cutting edge of frontier machine learning research.