FlashInfer Guide: Split-KV & Paged Attention Acceleration
The Problem: Why FlashAttention Stalls in Production LLM Serving
In high-concurrency generative AI pipelines, automated research tools, and high-frequency web scrapers like AiFetchly, foundation model deployment is divided into two fundamentally distinct phases: Prefill and Decode.
While FlashAttention (Dao et al.) revolutionized offline training and long-prompt prefilling by computing attention inside GPU SRAM via tiling, it was engineered under an assumption of dense, contiguous tensor allocations.
When deployed into real-world, interactive LLM serving environments, FlashAttention encounters severe architectural limitations:
1. The GPU Under-Utilization Crisis in Single-Token Decoding
During the autoregressive decode phase, an inference server receives a single newly generated query token ($q_t \in \mathbb{R}^{1 \times d}$) per sequence and computes attention against a massive past Key-Value (KV) cache ($T$ tokens, e.g., $32,768$ tokens).
In standard FlashAttention, parallelization is mapped strictly across batch size and attention heads:
$$\text{Active Thread Blocks} = \text{Batch Size} \times \text{Number of Heads}$$
On high-end datacenter GPUs like the NVIDIA H100 (which contains 132 Streaming Multiprocessors, or SMs) or the RTX 4090 (128 SMs):
- If an agent issues a request with batch size $b = 1$ using Grouped-Query Attention (GQA) with $8$ KV heads, only 8 thread blocks are scheduled.
- As a consequence, over 120 Streaming Multiprocessors sit completely idle, starved of computational work while the active SMs crawl sequentially through 32k tokens over the memory bus.
Hardware SM Saturation in Standard Decode (Batch Size = 1, 8 Heads):
[ SM 1 ][ SM 2 ][ SM 3 ][ SM 4 ][ SM 5 ][ SM 6 ][ SM 7 ][ SM 8 ] ──► 8 SMs Active (Saturated)
[ Idle ][ Idle ][ Idle ][ Idle ][ Idle ][ Idle ][ Idle ][ Idle ] ──► 124 SMs Idle (94% GPU Waste!) 2. The Paged Memory Address Translation Tax
To prevent Out-Of-Memory (OOM) memory fragmentation, modern serving systems (like vLLM and SGLang) allocate KV caches into non-contiguous virtual pages (PagedAttention / RadixAttention).
However, running standard FlashAttention kernels over paged memory requires either:
- Copying and Flattening: Re-assembling fragmented pages into contiguous buffers before kernel launch, incurring massive GPU memory copy latency.
- Pointer-Chasing Overheads: Custom CUDA wrappers that resolve block table pointers inside inner loops, causing memory serialization and high instruction stalls.
3. Kernel Launch Overhead in Multi-Turn Pipelines
In standard serving backends, processing a single decode token requires launching multiple disjoint GPU kernels in sequence:
apply_rotary_pos_emb(RoPE kernel)kv_cache_append(Memory write kernel)paged_attention(Compute kernel)
Launching three separate kernels per transformer layer across an 80-layer foundation model triggers 240 sequential CUDA kernel launches per token, causing CPU-GPU driver synchronization bottlenecks that inflate Time-to-First-Token (TTFT) and degrade Inter-Token Latency (ITL).
The Architecture: FlashInfer’s Serving-Centric Attention Engine
To eliminate GPU idle time and unify paged memory traversal, researchers from the University of Washington, OctoAI, and the SGLang team introduced FlashInfer: Efficient and Customizable Attention Engine for LLM Serving (Ye et al., MLSys 2025).
FlashInfer abandons the rigid dense assumptions of classic FlashAttention, introducing a modular, serving-native kernel library optimized for non-uniform, fragmented, and small-batch workloads.
The Three Core Pillars of FlashInfer
1. Split-KV Parallel Decoding
FlashInfer resolves the SM under-utilization crisis by introducing Split-KV Parallelism.
Instead of assigning an entire sequence length $T$ to a single thread block, FlashInfer slices the sequence dimension into $S$ independent chunks:
$$\text{Total Thread Blocks} = \text{Batch Size} \times \text{Heads} \times S$$
If $S = 16$, a batch size 1 request with 8 heads schedules $1 \times 8 \times 16 = 128$ parallel thread blocks, saturating 100% of the GPU’s Streaming Multiprocessors simultaneously.
Each SM computes attention over its designated slice using online softmax, emitting three compact partial statistics:
- $m_s$: The maximum attention score logit observed within chunk $s$.
- $l_s$: The local denominator sum of exponentials within chunk $s$.
- $O_s$: The unnormalized weighted sum of Value vectors within chunk $s$.
Finally, a lightweight Logarithmic Reduction Kernel merges the $S$ partial outputs into the exact global attention output using numerical softmax rescaling:
$$m_{\text{global}} = \max_{s} m_s, \quad l_{\text{global}} = \sum_{s} l_s \cdot e^{m_s - m_{\text{global}}}$$
$$O_{\text{global}} = \frac{1}{l_{\text{global}}} \sum_{s} e^{m_s - m_{\text{global}}} \cdot O_s$$
This parallel split delivers up to $3.1\times$ lower decode latency on long sequences with exact mathematical equivalence to dense attention.
2. Native Page-Sparse Layout Traversal
FlashInfer implements custom warp-level address translation primitives that read directly from paged KV cache tables:
- Inlined Block Table Traversal: Resolves page physical offsets inside GPU register memory rather than issuing global memory pointer lookups.
- Zero Padding in Ragged Batches: Variables-length prompt prefill requests are packed into a single contiguous 1D array (
ragged_indices), completely eliminating dummy zero tokens.
3. Composite Kernel Fusion
FlashInfer eliminates kernel launch overhead by fusing multiple pre-attention operations into a single Composite Attention Kernel:
- Rotary Position Embedding (RoPE) is applied directly inside registers as the Query and Key tokens are loaded.
- The newly calculated Key and Value tokens are appended into the paged memory pool concurrently as attention is being computed.
- This reduces kernel launch counts from 3 to 1 per layer, slashing CPU driver dispatch latency by $66%$.
Architectural Comparison: Serving Attention Backends
| Capability / Metric | FlashAttention-2 | vLLM PagedAttention v1 | FlashInfer Composite Engine |
|---|---|---|---|
| Decode Parallelism | Batch $\times$ Heads only | Batch $\times$ Heads only | Split-KV (Batch $\times$ Heads $\times$ Slices) |
| SM Utilization (BS=1, 32k KV) | Poor ($< 15%$ occupancy) | Poor ($< 20%$ occupancy) | Optimal ($100%$ full GPU saturation) |
| Paged Memory Traversal | Requires dense flattening | Custom CUDA lookup kernel | Native Page-Sparse inlined address traversal |
| Kernel Fusion | Separate RoPE & Append | Separate RoPE & Append | Full Composite Fusion (RoPE + Append + Attn) |
| Quantized KV Cache | FP16 / BF16 only | FP8 / INT4 (Separate) | Native FP8 / FP4 Tensor Core mma acceleration |
| Serving System Adoption | General baseline | vLLM Default | SGLang Core, vLLM V1, MLC-LLM |
Implementation: Building a Reference Split-KV Parallel Attention Engine in PyTorch
Below is an educational, production-grade PyTorch implementation demonstrating the mathematical mechanics of Split-KV Parallel Decoding, showing how independent chunks compute local online softmax statistics and combine via a global reduction kernel.
Step 1: Simulating Split-KV Chunk Execution
import torch
import torch.nn.functional as F
import math
from typing import Tuple, List
def compute_chunk_attention(
q: torch.Tensor,
k_chunk: torch.Tensor,
v_chunk: torch.Tensor
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""
Simulates a single GPU Streaming Multiprocessor (SM) computing attention
over an isolated slice of the KV cache using online softmax statistics.
q: [1, Head_Dim]
k_chunk: [Chunk_Len, Head_Dim]
v_chunk: [Chunk_Len, Head_Dim]
Returns: (local_max_m, local_denom_l, local_out_O)
"""
scale = 1.0 / math.sqrt(q.size(-1))
# Dot product attention scores: [1, Chunk_Len]
scores = torch.matmul(q, k_chunk.transpose(-1, -2)) * scale
# 1. Local maximum logit for numerical stability
m_local = torch.max(scores, dim=-1, keepdim=True)[0]
# 2. Exponentiated scores and local denominator sum
exp_scores = torch.exp(scores - m_local)
l_local = torch.sum(exp_scores, dim=-1, keepdim=True)
# 3. Unnormalized local output projection
o_local = torch.matmul(exp_scores, v_chunk)
return m_local, l_local, o_local Step 2: Assembling the Split-KV Reduction Kernel
We now implement the logarithmic tree reduction that merges the partial statistics from all $S$ chunks:
def split_kv_reduce(
partials: List[Tuple[torch.Tensor, torch.Tensor, torch.Tensor]]
) -> torch.Tensor:
"""
Merges S partial attention accumulators into the exact global attention output.
partials: List of (m_s, l_s, o_s) tuples from all parallel SM slices.
"""
# Find global maximum logit across all chunks
all_m = torch.stack([p[0] for p in partials], dim=0) # [S, 1, 1]
m_global = torch.max(all_m, dim=0)[0] # [1, 1]
global_l = torch.zeros_like(partials[0][1])
global_o = torch.zeros_like(partials[0][2])
# Rescale and accumulate partial denominators and outputs
for m_s, l_s, o_s in partials:
# Rescaling factor: e^(m_s - m_global)
alpha = torch.exp(m_s - m_global)
global_l += l_s * alpha
global_o += o_s * alpha
# Final normalization
final_output = global_o / global_l
return final_output Step 3: Benchmarking Split-KV vs. Dense Attention
We verify that Split-KV parallel decoding matches full dense attention to machine precision while profiling how chunking scales across GPU SMs:
def benchmark_split_kv_verification():
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"--- Running FlashInfer Split-KV Verification on {device.upper()} ---")
head_dim = 128
seq_len = 32768 # 32k token long-context decode
num_splits = 16 # Slices mapped to 16 distinct SM thread blocks
chunk_size = seq_len // num_splits
# 1. Instantiate single query token and long KV cache
q = torch.randn(1, head_dim, device=device)
k = torch.randn(seq_len, head_dim, device=device)
v = torch.randn(seq_len, head_dim, device=device)
# 2. Standard Dense Attention Baseline (Single SM loop)
scale = 1.0 / math.sqrt(head_dim)
dense_scores = torch.matmul(q, k.transpose(-1, -2)) * scale
dense_attn = F.softmax(dense_scores, dim=-1)
expected_output = torch.matmul(dense_attn, v)
# 3. FlashInfer Split-KV Parallel Execution
partials = []
for s in range(num_splits):
start_idx = s * chunk_size
end_idx = start_idx + chunk_size
k_chunk = k[start_idx:end_idx]
v_chunk = v[start_idx:end_idx]
# Computed in parallel across GPU SMs
p = compute_chunk_attention(q, k_chunk, v_chunk)
partials.append(p)
split_output = split_kv_reduce(partials)
# Measure Numerical Precision & Equivalence
max_diff = torch.max(torch.abs(expected_output - split_output)).item()
print(f"Total KV Sequence Length: {seq_len:,} tokens")
print(f"Parallel Split Slices (S): {num_splits} independent thread blocks")
print(f"Tokens Processed per SM: {chunk_size:,} tokens/chunk")
print(f"Maximum Numerical Error: {max_diff:.8e} (Exact Mathematical Match)")
print(f"SM Occupancy Multiplier: {num_splits}x more active SMs utilized")
benchmark_split_kv_verification() When executed, this outputs:
--- Running FlashInfer Split-KV Verification on CPU ---
Total KV Sequence Length: 32,768 tokens
Parallel Split Slices (S): 16 independent thread blocks
Tokens Processed per SM: 2,048 tokens/chunk
Maximum Numerical Error: 5.96046448e-08 (Exact Mathematical Match)
SM Occupancy Multiplier: 16x more active SMs utilized Production Advantages for Autonomous Agents and Enterprise Scraping
In real-time web scraping, persistent reasoning agents, and enterprise document extraction—such as the infrastructure powering AiFetchly—deploying FlashInfer delivers four decisive production advantages:
- Sub-15ms Interactive Decoding on 32k+ Prompts: Multi-turn scraping agents executing deep extraction loops across massive HTML documents experience up to $3.1\times$ lower token generation latency, preventing agent timeouts.
- Zero Memory Copy Overhead in RadixAttention: In high-volume systems that reuse shared system prompts and DOM schemas, FlashInfer traverses tree-structured KV caches directly in paged memory without allocating intermediate contiguous buffers.
- Flawless Grouped-Query Attention (GQA) Scaling: FlashInfer features specialized tensor tiling for GQA models (such as LLaMA-3 and Qwen-2.5), avoiding redundant Key/Value duplication across query head groups.
- Hardware-Native Quantized KV Support: Supports FP8 and INT4 quantized KV caches directly inside Tensor Core pipelines, allowing inference clusters to host twice as many concurrent agent sessions per node.
Conclusion
While FlashAttention solved the memory footprint of dense prefilling, modern LLM serving is an entirely different beast characterized by non-contiguous paged memory, ragged batches, and single-token decode bottlenecks.
By parallelizing the sequence dimension via Split-KV decoding, traversing paged caches natively, and fusing RoPE and memory updates into composite kernels, FlashInfer bridges the gap between hardware architecture and serving dynamics. For AI systems engineers optimizing high-throughput, low-latency serving stacks in 2026, FlashInfer represents the state-of-the-art attention engine.