vAttention: Serving LLMs Without PagedAttention Guide
The Problem: The Hidden Overhead of Software PagedAttention
In high-concurrency generative AI pipelines, automated research tools, and high-frequency web scrapers like AiFetchly, deploying large language models requires managing massive dynamic memory buffers for Key-Value (KV) caches.
When PagedAttention (Kwon et al., SOSP 2023) debuted in vLLM, it addressed a critical operational challenge: static memory allocation pre-allocated full context buffers (e.g., 8,192 tokens) for every incoming request, resulting in 60% to 80% of GPU memory being wasted on unused padding. PagedAttention solved this by borrowing the concept of virtual memory from operating systems, splitting the KV cache into fixed-size physical memory pages (typically 16 or 32 tokens per block) allocated dynamically on demand.
However, PagedAttention implemented paging entirely in software-level user space, introducing three severe architectural drawbacks in production:
1. Software Page Table Dereferencing in GPU Inner Loops
In PagedAttention, because the physical memory blocks of a sequence are non-contiguous and scattered across GPU VRAM, attention kernels cannot address memory using simple continuous pointer arithmetic.
Instead, every GPU thread block must continuously consult a software-managed block table during inner attention loops:
Standard Memory Traversal vs. Software Paged Memory Traversal:
Contiguous Pointer: KV_ptr = Base_Address + (token_idx * Stride) ──► Direct 1-Cycle Math!
Paged Pointer: Logical_Page = token_idx / Block_Size
Physical_Block = Block_Table[batch_id][Logical_Page] ──► Global VRAM Fetch!
KV_ptr = Physical_Block_Base + (Offset * Stride) ──► Extra Indirection! This software address translation creates high instruction overhead, wastes register budgets on tracking block indices, and triggers memory latency stalls while fetching block table entries from VRAM.
2. Warp Divergence and Loss of Memory Coalescing
Modern GPU architectures (NVIDIA Hopper, Ada Lovelace, and Blackwell) rely on memory coalescing: when 32 threads in a warp issue loads to contiguous, 128-byte aligned memory addresses, the memory controller services the entire warp with a single high-bandwidth memory transaction.
In PagedAttention, because logical sequences cross discontinuous physical page boundaries, threads at the boundary of a block diverge:
- Threads processing tokens within the block load from one physical address space.
- Threads processing tokens in the next block must load from a completely different physical pointer.
- Memory requests become fragmented into multiple uncoalesced transactions, cutting effective memory bus bandwidth by up to 30%.
3. Incompatibility with Native High-Performance Kernels
State-of-the-art attention implementations like FlashAttention-2, FlashAttention-3, and FlashDecoding are heavily optimized for dense, contiguous tensors using hardware Tensor Memory Accelerators (TMA) and warp specialization.
Because PagedAttention alters the memory layout from a contiguous tensor into an array of pointers:
- Standard PyTorch operations and native FlashAttention kernels cannot execute over paged KV caches.
- Engine developers must write, maintain, and tune custom CUDA kernels for every single attention variant (MHA, GQA, MLA) across every GPU generation.
- Empirical profiling shows that custom PagedAttention decode kernels run 15% to 26% slower than native contiguous FlashAttention kernels on the same hardware.
The Architecture: vAttention and Operating-System-Level GPU Memory Management
To eliminate the latency penalties of software page tables while preserving zero-waste dynamic allocation, researchers from Microsoft Research and UIUC introduced vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention (Qiu et al., ASPLOS / MLSys).
The core thesis of vAttention is straightforward: Instead of emulating page tables in software inside CUDA kernel loops, leverage the underlying GPU Hardware Memory Management Unit (MMU) and CUDA Virtual Memory Management (VMM) APIs.
The Three Operational Pillars of vAttention
Modern operating systems execute virtual memory translation using hardware page tables backed by CPU MMUs. Since CUDA 10.2, NVIDIA GPUs have supported low-level Virtual Memory Management (VMM) driver primitives (cuMem* family of APIs).
vAttention utilizes these native hardware primitives to restructure KV cache management across three decoupled stages:
1. Virtual Address Space Reservation (Zero Physical VRAM Cost)
On 64-bit architectures, virtual address space is virtually inexhaustible ($2^{64}$ bytes). Reserving a virtual address range costs zero bytes of physical GPU memory:
- When a new request arrives, vAttention calls
cuMemAddressReserve()to reserve a large, contiguous virtual memory buffer corresponding to the maximum possible sequence length (e.g., 32k or 128k tokens). - The returned pointer is a standard 64-bit device memory address.
- At this stage, no physical GPU VRAM is committed.
2. On-Demand Dynamic Physical Allocation and Mapping
As autoregressive generation runs and new tokens are generated:
- When the sequence exceeds the currently mapped physical memory, vAttention invokes
cuMemCreate()to allocate a small physical memory chunk (e.g., 2 MB or 64 KB). - vAttention then calls
cuMemMap()to map this physical chunk directly to the next contiguous virtual address offset inside the reserved address space. - Finally,
cuMemSetAccess()marks the mapped region as read-write accessible by the GPU.
Because the underlying physical pages are mapped to consecutive virtual addresses, the GPU’s hardware Memory Management Unit (MMU) and Translation Lookaside Buffer (TLB) perform address translation at wire speed inside the memory controller.
3. Execution of Native, Off-the-Shelf FlashAttention Kernels
From the perspective of PyTorch and CUDA kernels, the KV cache is an ordinary, 100% contiguous tensor:
# To the attention kernel, the tensor is perfectly contiguous:
# Shape: (batch_size, num_heads, sequence_length, head_dim)
output = flash_attn_func(q, k_vattention, v_vattention) - Zero Software Lookups: Kernels use standard pointer arithmetic (
ptr + offset), eliminating block table dereferencing loops. - Full Memory Coalescing: Memory accesses remain 128-byte aligned across all thread warps, maximizing High Bandwidth Memory (HBM) utilization.
- Zero Kernel Rewriting: Any off-the-shelf attention kernel (FlashAttention-2, FlashAttention-3, FlashDecoding, Triton kernels) runs directly without modification.
Architectural Comparison: LLM KV Cache Management Strategies
| Dimension / Metric | Static Contiguous Allocation | PagedAttention (vLLM) | vAttention (CUDA VMM) |
|---|---|---|---|
| Virtual Address Space | Contiguous | Fragmented (Array of Blocks) | Contiguous (Reserved) |
| Physical VRAM Allocation | Eager (Max Context upfront) | Lazy (Dynamic Blocks) | Lazy (Dynamic Pages) |
| Memory Fragmentation | Catastrophic (60-80% waste) | Near Zero (<4% waste) | Near Zero (<4% waste) |
| Address Translation | Direct hardware math | Software Block Tables in Kernel | Hardware GPU MMU / TLB |
| Attention Kernels | Native FlashAttention | Custom Paged Kernels Only | Native FlashAttention-2/3 |
| Kernel Latency Overhead | 0% (Baseline) | +15% to +26% overhead | 0% (Matches Native Kernel) |
| Serving Throughput | Low (Limited by batch OOM) | High | 1.3x to 1.9x vs PagedAttention |
Implementation: Simulating CUDA Virtual Memory Management in Python
Below is an educational, production-grade simulation of vAttention’s memory allocation engine in Python. This implementation illustrates virtual address space reservation, lazy physical page mapping, and seamless contiguous tensor execution.
Step 1: The Virtual Memory Cache Manager
import numpy as np
from typing import Dict, List, Optional, Tuple
class VirtualKVCacheManager:
"""
Simulates vAttention's low-level CUDA Virtual Memory Management (VMM).
Reserves contiguous virtual buffers and maps physical pages on demand.
"""
def __init__(self, page_size_tokens: int = 64, d_model: int = 128, max_seq_len: int = 2048):
self.page_size_tokens = page_size_tokens
self.d_model = d_model
self.max_seq_len = max_seq_len
self.page_size_bytes = page_size_tokens * d_model * 2 # 16-bit float (2 bytes)
# Total virtual capacity reserved per request
self.virtual_capacity = max_seq_len
# Physical memory pool simulation
self.physical_allocations: Dict[int, List[int]] = {} # req_id -> list of physical page IDs
self.virtual_buffers: Dict[int, np.ndarray] = {} # req_id -> contiguous virtual array
self.mapped_tokens: Dict[int, int] = {} # req_id -> currently mapped capacity
self.current_tokens: Dict[int, int] = {} # req_id -> active sequence length
def register_request(self, req_id: int):
"""
Simulates cuMemAddressReserve(): reserves full virtual address space with zero VRAM cost.
"""
# Pre-allocate contiguous virtual memory buffer (simulated)
self.virtual_buffers[req_id] = np.zeros((self.max_seq_len, self.d_model), dtype=np.float16)
self.physical_allocations[req_id] = []
self.mapped_tokens[req_id] = 0
self.current_tokens[req_id] = 0
def append_token(self, req_id: int, key_vector: np.ndarray):
"""
Appends a generated token vector. Allocates and maps physical pages as needed.
"""
seq_idx = self.current_tokens[req_id]
# Check if new physical page is required
if seq_idx >= self.mapped_tokens[req_id]:
self._map_next_physical_page(req_id)
# Write token directly into contiguous virtual address space
self.virtual_buffers[req_id][seq_idx] = key_vector
self.current_tokens[req_id] += 1
def _map_next_physical_page(self, req_id: int):
"""
Simulates cuMemCreate() and cuMemMap(): commits a new physical memory page.
"""
new_page_id = len(self.physical_allocations[req_id])
self.physical_allocations[req_id].append(new_page_id)
self.mapped_tokens[req_id] += self.page_size_tokens
def get_contiguous_kv_slice(self, req_id: int) -> np.ndarray:
"""
Returns a contiguous slice for native FlashAttention. Zero software table lookups!
"""
active_len = self.current_tokens[req_id]
# Direct contiguous memory view
return self.virtual_buffers[req_id][:active_len]
def get_memory_stats(self, req_id: int) -> Dict[str, float]:
"""Calculates physical memory allocated vs reserved virtual memory."""
physical_kb = (len(self.physical_allocations[req_id]) * self.page_size_bytes) / 1024
virtual_kb = (self.max_seq_len * self.d_model * 2) / 1024
active_kb = (self.current_tokens[req_id] * self.d_model * 2) / 1024
return {
"active_tokens": self.current_tokens[req_id],
"physical_allocated_kb": physical_kb,
"virtual_reserved_kb": virtual_kb,
"utilization_pct": (active_kb / physical_kb) * 100 if physical_kb > 0 else 0.0
} Step 2: Demonstrating Dynamic Allocation and Native Kernel Execution
# Initialize vAttention cache manager (page size = 64 tokens, d_model = 128)
manager = VirtualKVCacheManager(page_size_tokens=64, d_model=128, max_seq_len=1024)
# Register new client request
req_id = 101
manager.register_request(req_id)
print("--- 1. Initial State (Virtual Reservation Only) ---")
initial_stats = manager.get_memory_stats(req_id)
print(f"Virtual Address Reserved: {initial_stats['virtual_reserved_kb']:.1f} KB")
print(f"Physical Memory Allocated: {initial_stats['physical_allocated_kb']:.1f} KB (Zero physical VRAM!)\n")
# Simulate generating 150 tokens step by step
np.random.seed(42)
for i in range(150):
k_vec = np.random.randn(128).astype(np.float16)
manager.append_token(req_id, k_vec)
print("--- 2. State After Generating 150 Tokens ---")
gen_stats = manager.get_memory_stats(req_id)
print(f"Active Tokens Generated: {gen_stats['active_tokens']}")
print(f"Physical Pages Allocated: {len(manager.physical_allocations[req_id])} (64 tokens per page)")
print(f"Physical Memory In Use: {gen_stats['physical_allocated_kb']:.1f} KB")
print(f"Physical Memory Efficiency: {gen_stats['utilization_pct']:.1f}%\n")
print("--- 3. Kernel Execution over Contiguous Memory ---")
# Retrieve contiguous slice directly for FlashAttention
kv_tensor = manager.get_contiguous_kv_slice(req_id)
print(f"Contiguous Tensor Shape: {kv_tensor.shape}")
print(f"Is Tensor Contiguous: {kv_tensor.flags['C_CONTIGUOUS']}")
print("Native FlashAttention can execute directly with zero block-table lookups!") When executed, this demonstrates how vAttention dynamically scales physical memory while keeping tensors contiguous:
--- 1. Initial State (Virtual Reservation Only) ---
Virtual Address Reserved: 256.0 KB
Physical Memory Allocated: 0.0 KB (Zero physical VRAM!)
--- 2. State After Generating 150 Tokens ---
Active Tokens Generated: 150
Physical Pages Allocated: 3 (64 tokens per page)
Physical Memory In Use: 48.0 KB
Physical Memory Efficiency: 78.1%
--- 3. Kernel Execution over Contiguous Memory ---
Contiguous Tensor Shape: (150, 128)
Is Tensor Contiguous: True
Native FlashAttention can execute directly with zero block-table lookups! Production Advantages for Autonomous Agents and Enterprise Scraping
In real-world data collection, multi-turn reasoning agents, and high-concurrency scraping engines—such as the infrastructure powering AiFetchly—deploying a vAttention memory subsystem delivers four foundational advantages:
- Up to 26% Reduction in Decode Latency: By removing software page-table dereferencing inside GPU attention inner loops and eliminating warp divergence at block boundaries, inference latency drops significantly across interactive token streaming.
- 100% Contiguous Kernel Portability: Infrastructure teams can immediately deploy upstream optimizations—such as FlashAttention-3, FlashDecoding, or custom Triton attention kernels—without waiting for developers to re-engineer custom paged versions.
- Zero VRAM Fragmentation with Dynamic Batching: Like PagedAttention, physical memory pages are allocated only when needed and freed immediately upon request completion, eliminating memory waste from pre-allocated context windows.
- Seamless Support for Prefix Caching: Shared system prompts and document prefixes can be mapped to multiple virtual address spaces via copy-on-write page table mapping, accelerating multi-step agent trajectories with minimal memory duplication.
Conclusion
The introduction of PagedAttention was a pivotal milestone in tackling LLM memory fragmentation, but its software-level page table emulation came with a persistent tax on kernel speed and software complexity.
By moving address translation from software kernel loops down into the GPU’s native Hardware Memory Management Unit via CUDA Virtual Memory Management (VMM), vAttention unites the best of both worlds: zero-waste dynamic memory elasticity and bare-metal FlashAttention execution speeds. As foundation model serving scales to 128k+ context windows in 2026, vAttention stands as the defining systems architecture for high-performance LLM deployment.