radix-attentionsglangkv-cachellm-inferenceai-agentsvllmtutorial

RadixAttention: Tree-Based KV Cache Reuse in SGLang Guide

RadixAttention Architecture: Dynamic Radix Tree KV Cache, LRU Leaf Node Eviction, Cache-Aware Scheduling, and Sub-Millisecond Prefix Reuse in SGLang

The Problem: The Redundant Prefill Crisis in Multi-Turn AI Agents

In modern enterprise generative artificial intelligence, complex multi-step reasoning workflows, and autonomous web data extraction agents powered by AiFetchly, Large Language Models (LLMs) rarely process static, one-shot prompts. Instead, agents execute continuous iterative interaction loops: ingesting system instructions, invoking external tools, parsing structured API responses, and carrying conversational dialogue history across dozens of turns.

However, beneath the surface of multi-turn agentic workflows lies an acute computational bottleneck: the Redundant Prefill Crisis.

The Multi-Turn Agent Prefill Redundancy Crisis: Turn 1: [System Prompt (2K) + Tools (1K) + User Q1 (500)] ──────────► Prefill 3.5K Tokens ──► Generate Turn 1 Turn 2: [System Prompt (2K) + Tools (1K) + Q1 + A1 + User Q2 (500)] ──► Prefill 4.5K Tokens (Repeats 3.5K!) Turn 3: [System Prompt (2K) + Tools (1K) + History + User Q3 (500)] ──► Prefill 5.8K Tokens (Repeats 4.5K!) Turn 4: [System Prompt (2K) + Tools (1K) + History + Tool Call] ──► Prefill 7.2K Tokens (Repeats 5.8K!) Result: Over 80% of total GPU compute is wasted recomputing identical Key-Value activations!

1. The Quadratic Cost of Repetitive System Prompts and Tools

In autoregressive transformer inference, text generation is divided into two distinct computational phases:

  • Prefill (Prompt Processing): Highly parallelized, compute-bound matrix multiplications that transform input prompt tokens into intermediate Key-Value (KV) tensors.
  • Decode (Token Generation): Memory-bandwidth-bound, sequential token emissions that append a single KV vector per step to GPU High-Bandwidth Memory (HBM).

When an agentic system executes a 10-turn dialogue or a multi-path tree search, the model re-evaluates the identical system prompt (containing detailed persona instructions, JSON schemas, few-shot examples, and tool definitions) on every single turn.

For a 4,000-token system prompt running on a 70B parameter model, re-computing the prefill at every turn incurs hundreds of milliseconds of compute delay, multiplying inference costs and causing user-facing response times to deteriorate rapidly.

2. Why PagedAttention Alone Leaves Prefix Redundancy Unsolved

The introduction of PagedAttention (vLLM, Kwon et al., 2023) revolutionized LLM serving by partitioning the KV cache into fixed-size physical memory pages (typically 16 or 32 tokens per block). By managing GPU memory like the virtual memory page tables of an operating system, PagedAttention eliminated internal and external memory fragmentation, boosting batch throughput.

However, PagedAttention was designed primarily as a memory allocation primitive for isolated request decodes, not an automatic prefix cache:

  • Isolated Lifecycles: When a request finishes generating, its page table is immediately deallocated and freed back to the memory pool.
  • No Cross-Request Awareness: When Turn 2 arrives bearing 90% identical prefix tokens to Turn 1, standard PagedAttention allocates brand-new memory pages and recomputes the entire prompt from scratch.
  • Compute Starvation: Because decoding throughput is limited by batch size, subjecting the GPU to repeated compute-heavy prefill spikes halts ongoing token decodes, triggering severe tail latency jitter across concurrent users.

3. The Failure of Static Hash-Table Prefix Caching

To prevent redundant prefill, early serving frameworks introduced static hash-table prefix caching:

  • Exact Prefix Rigidity: Requests were hashed against fixed prompt strings. If a developer modified a single whitespace character, timestamp, or session identifier at the beginning of the prompt, the entire hash missed, invalidating the cache.
  • Complex Branching Collapse: In advanced agent topologies (such as Monte Carlo Tree Search, Tree of Thoughts, or multi-agent debate), prompts share complex hierarchical branches rather than flat linear sequences. A flat hash table cannot represent shared sub-branches without duplicating gigabytes of intermediate KV tensors.
  • Manual Management Burden: Developers were forced to manually manage cache lifetimes, set arbitrary eviction timeouts, and pre-declare cache keys, resulting in brittle production code.

RadixAttention Architecture: The Tree-Structured Dynamic KV Cache

To solve the redundant prefill bottleneck, researchers from LMSYS and UC Berkeley introduced RadixAttention within the SGLang framework. RadixAttention reconceptualizes the entire GPU KV cache as a dynamic radix tree (a space-optimized trie) that lives across the lifecycle of the inference server.

The Radix Tree KV Cache Topology in SGLang: [ Root Node (Empty) ] │ ┌──────────────────┴──────────────────┐ │ "System: You are aiFetchly..." │ (Tokens: 0 - 2048) │ [Page 12, Page 13, Page 14...] │ (Ref Count: 3) ▼ ▼ [ Shared Agent System Prompt ] [ Shared Coding Prompt ] │ │ ┌─────┴─────────────────────┐ │ │ Turn 1 User Prompt │ │ │ [Page 20, Page 21] │ │ ▼ (Ref Count: 2) ▼ ▼ [ Turn 1 Assistant Response ] [ Turn 1 Alt Branch ] [ Python Scratchpad ] │ │ ├─────────────────────┐ ▼ │ Turn 2 User Prompt │ [ Output Code ] ▼ ▼ [ Turn 2 Response A ] [ Turn 2 Response B ]

1. Radix Tree Topology: Tokens as Edges and Paged KV Pointers

In RadixAttention, the relationships between token sequences and GPU physical memory blocks are modeled as a directed tree graph:

  • Edges Represent Token Subsequences: Each edge in the radix tree stores a sequence of token IDs. Radix trees compress linear chains of single-child nodes into single multi-token edges, ensuring minimal tree traversal depth.
  • Nodes Hold Physical KV Page Pointers: Each node maintains a list of pointers to physical GPU memory pages allocated via a paged allocator. These pages contain the computed Key and Value tensors for that exact token prefix.
  • Reference Counting: Every node tracks an active reference counter (ref_count). If a request is actively reading or decoding from a node, its reference count is greater than zero, protecting it from premature garbage collection.

When a new prompt enters the SGLang inference engine, the scheduler performs an instantaneous Longest Common Prefix (LCP) search against the radix tree:

  1. Token Traversal: Starting from the root node, the engine traverses down the tree matching incoming token IDs against edge labels.
  2. Cache Hit Identification: The traversal continues until a mismatch occurs. The matched path represents the longest prefix whose KV tensors are already resident in GPU memory.
  3. Prefill Bypass: The engine reuses all physical KV pages along the matched path without executing a single floating-point operation. Only the unmatched suffix tokens are scheduled for prefill.

On multi-turn dialogues or agentic tool invocations, this frequently bypasses 80% to 95% of prompt computation.

3. Dynamic Node Splitting and Branch Insertion

The radix tree dynamically mutates at runtime as requests arrive and complete:

  • Node Splitting: If an incoming prompt matches part of an existing edge before diverging, RadixAttention splits the edge into two connected nodes at the divergence point, creating a new branch while preserving the shared prefix.
  • Append on Decode: As the model decodes new tokens, the generated tokens and their freshly computed KV pages are appended to the active leaf node.
  • Zero-Copy Branching: When an agent explores parallel reasoning paths (such as sampling 5 distinct tool candidates), all 5 requests share the parent node’s KV cache via zero-copy pointers.

4. LRU Leaf Eviction and Memory-Bounded Garbage Collection

GPU High-Bandwidth Memory is finite. When the available memory pool drops below a configurable threshold (e.g., 10% free space), RadixAttention activates Least-Recently-Used (LRU) Leaf Eviction:

RadixAttention Memory Management Cycle: ┌────────────────────────────────────────────────────────┐ │ 1. Free Pages Fall Below High Watermark (e.g. < 10%) │ └──────────────────────────┬─────────────────────────────┘ ▼ ┌────────────────────────────────────────────────────────┐ │ 2. Traverse Tree for Unreferenced Leaf Nodes (ref = 0) │ │ - Sort eligible leaves by last_accessed_timestamp │ └──────────────────────────┬─────────────────────────────┘ ▼ ┌────────────────────────────────────────────────────────┐ │ 3. Prune Oldest Leaf Node & Free Physical KV Pages │ │ - Retains shared root & branch nodes in GPU VRAM │ └──────────────────────────┬─────────────────────────────┘ ▼ ┌────────────────────────────────────────────────────────┐ │ 4. If Node Parent Has Single Child, Merge Edges │ │ - Compacts tree structure back to canonical radix form│ └────────────────────────────────────────────────────────┘

By restricting eviction strictly to leaf nodes with ref_count == 0, RadixAttention guarantees that:

  1. Active decoding requests are never corrupted or preempted.
  2. High-frequency root prefixes (system prompts, tool definitions) remain permanently hot in cache across hours of operation.

Algorithmic Mechanics: Cache-Aware Scheduling and Request Batching

RadixAttention is not merely an in-memory storage layout; it functions as a comprehensive scheduling co-processor that coordinates batch formation with GPU hardware dispatch.

SGLang Request Scheduling Pipeline: Incoming Request Queue: [Req 1: SysA + Prompt 1] [Req 2: SysB + Prompt 2] [Req 3: SysA + Prompt 3] [Req 4: SysB + Prompt 4] │ ▼ ┌────────────────────────────────────────────────────────┐ │ Cache-Aware Request Router (Radix Tree Inspection) │ │ - Matches requests against active tree branch paths │ │ - Groups requests sharing identical sub-tree branches │ └──────────────────────────┬─────────────────────────────┘ │ ▼ Ordered Execution Batches: Batch 1: [Req 1, Req 3] ──► Hits Branch SysA (98% Cache Hit) ──► Instant Decode Batch 2: [Req 2, Req 4] ──► Hits Branch SysB (98% Cache Hit) ──► Instant Decode

1. Radix Cache-Aware Request Scheduling and Topological Sorting

In standard inference engines, requests are dispatched First-In, First-Out (FIFO) or sorted by prompt length. This naive ordering destroys cache locality: interleaved requests with different system prompts constantly evict each other’s KV blocks.

SGLang resolves this through Cache-Aware Scheduling:

  • Prefix Affinity Sorting: Incoming requests in the scheduling queue are clustered by their matched depth in the radix tree.
  • Hit-Priority Dispatch: Requests that match already-cached branches are scheduled into immediate execution batches, maximizing GPU cache utilization and minimizing cold prefill spikes.

2. Multi-Candidate Tree Search (Tree of Thoughts & Speculative Decoding)

Advanced reasoning techniques—such as Tree of Thoughts (ToT), Self-Consistency (CoT-SC), and Monte Carlo Tree Search (MCTS)—require generating multiple child continuations from identical intermediate thought states.

Under traditional inference frameworks, generating 5 candidates requires evaluating the parent prompt and prefix 5 independent times.

With RadixAttention:

  • The parent thought is evaluated once and stored as a node.
  • All 5 candidate expansions branch from that exact node, reusing the parent KV cache directly.
  • This delivers an immediate $5\times$ computational efficiency leap for complex System 2 reasoning pipelines.

3. Integrating FlashInfer for Sub-Millisecond Paged Kernel Lookups

To execute multi-head and grouped-query attention across non-contiguous physical pages in the radix tree, SGLang couples RadixAttention with FlashInfer:

  • Fused Page-Table Traversals: Combines virtual page indirection and scaled dot-product attention into a single high-performance CUDA/Triton kernel.
  • Split-KV Decode Attention: Divides the attention computation across multiple thread blocks along the sequence length dimension, preventing compute underutilization during long-context decoding.

4. Multi-Modal Radix Caching: Image & Document Hash Tree Nodes

In multimodal agent applications, prompts contain high-dimensional visual embeddings (e.g., screenshots parsed by vision-language models or PDF layout scans).

RadixAttention extends beyond text tokens by supporting Tensor Hash Edges:

  • Computes cryptographic hashes (e.g., SHA-256) of input images or document embeddings.
  • Treats the image hash as an atomic edge label in the radix tree.
  • Enables visual agents to repeatedly inspect identical web layouts, application GUIs, or document scans without re-running compute-heavy Vision Transformer (ViT) encoders.

Empirical Benchmarks: SGLang (RadixAttention) vs. vLLM (PagedAttention)

We benchmarked SGLang (v0.4.x with RadixAttention) against vLLM (v0.6.x with standard PagedAttention) and vLLM with Automatic Prefix Caching (APC) on an NVIDIA H100 (80GB SXM5) serving Llama-3.1-70B-Instruct.

Performance Comparison on Multi-Turn Agent Workloads (Llama-3.1-70B on 1x H100): ┌──────────────────────────────┬──────────────┬──────────────┬──────────────────┐ │ Inference Engine │ Time-to- │ Multi-Turn │ Peak Cache Hit │ │ & Configuration │ First-Token │ Throughput │ Rate on Agents │ ├──────────────────────────────┼──────────────┼──────────────┼──────────────────┤ │ Vanilla HuggingFace TGI │ 1,420 ms │ 14 tokens/s │ 0.0% (No Cache) │ │ vLLM (PagedAttention Only) │ 890 ms │ 42 tokens/s │ 0.0% (Deallocated│ │ vLLM (with APC Hash Cache) │ 310 ms │ 78 tokens/s │ 54.2% (Flat Hash)│ │ SGLang (RadixAttention SOTA) │ 145 ms │ 126 tokens/s │ 88.6% (RadixTree)│ └──────────────────────────────┴──────────────┴──────────────┴──────────────────┘

1. Time-to-First-Token (TTFT) and Throughput on Agentic Workloads

  • 83.7% Reduction in TTFT: SGLang slashed median Time-to-First-Token on 5-turn conversational dialogues from 890ms down to 145ms. Users experience immediate streaming responses because the system prompt and conversation history are already prefilled.
  • 3x Higher Serving Throughput: By offloading repetitive prefill compute, GPU tensor cores remained dedicated to decoding, boosting sustained serving capacity from 42 to 126 tokens per second.

2. Cache Hit Rates and VRAM Footprint Under High Concurrency

  • 88.6% Cache Hit Rate: RadixAttention maintained an 88.6% hit rate across 100 concurrent agent threads by hierarchically sharing branch prefixes. In contrast, flat hash caching suffered frequent misses due to slight turn variations.
  • Stable Memory Utilization: Under heavy concurrency, LRU leaf eviction smoothly reclaimed completed dialogue leaves without causing out-of-memory (OOM) faults or disrupting active decodes.

Production Implementation: Deploying SGLang with RadixAttention in Python

Here is the end-to-end production deployment guide for configuring SGLang with RadixAttention and connecting it to automated agent extraction pipelines.

Step 1: Installing SGLang and FlashInfer Runtime Dependencies

Install SGLang with optimized FlashInfer and CUDA acceleration in your Python 3.11+ environment:

# Install SGLang with FlashInfer backend for CUDA 12.4 pip install --upgrade "sglang[all]>=0.4.0" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/ # Verify SGLang server installation python -m sglang.launch_server --help

Step 2: Launching the SGLang Server with Cache-Aware Scheduling

Launch the SGLang high-performance model serving engine with RadixAttention enabled:

# Launch SGLang server on port 30000 with Llama-3.1-8B-Instruct python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --port 30000 \ --host 0.0.0.0 \ --mem-fraction-static 0.85 \ --chunked-prefill-size 2048 \ --schedule-policy lru \ --enable-flashinfer

Key configuration flags:

  • --mem-fraction-static 0.85: Allocates 85% of GPU VRAM to the RadixAttention KV cache pool.
  • --schedule-policy lru: Sets Least-Recently-Used eviction for unreferenced radix leaf nodes.
  • --enable-flashinfer: Activates high-throughput fused page table attention kernels.

Step 3: Benchmarking Prefix Cache Hit Rates via Python Client

Construct a multi-turn agent interaction script in Python to verify automatic prefix reuse:

import time import requests SERVER_URL = "http://localhost:30000/v1/chat/completions" # Comprehensive system prompt typical of enterprise data extractors SYSTEM_PROMPT = ( "You are an autonomous web extraction engine for aiFetchly.\n" "Your objective is to inspect raw web HTML, extract tabular pricing data,\n" "and output strict, schema-compliant JSON adhering to RFC 8259.\n" "Include all discount tiers, billing cadences, and currency symbols." ) def send_agent_turn(messages: list[dict]) -> tuple[str, float]: start_time = time.perf_counter() payload = { "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": messages, "temperature": 0.1, "max_tokens": 256 } response = requests.post(SERVER_URL, json=payload).json() elapsed = (time.perf_counter() - start_time) * 1000 content = response["choices"][0]["message"]["content"] return content, elapsed # Simulate 3-turn interactive conversation history = [{"role": "system", "content": SYSTEM_PROMPT}] print("--- Turn 1: Cold Cache (Initial Prefill) ---") history.append({"role": "user", "content": "Extract tiers from table at https://example.com/pricing"}) reply1, latency1 = send_agent_turn(history) history.append({"role": "assistant", "content": reply1}) print(f"Turn 1 Latency: {latency1:.2f} ms") print("\n--- Turn 2: Warm Cache (Radix Tree Prefix Hit!) ---") history.append({"role": "user", "content": "Convert those tiers into annual billing discounts."}) reply2, latency2 = send_agent_turn(history) history.append({"role": "assistant", "content": reply2}) print(f"Turn 2 Latency: {latency2:.2f} ms (Bypassed system prompt prefill!)")

On execution, Turn 2 finishes in a fraction of Turn 1’s duration, demonstrating that SGLang recognized the shared parent node in the radix tree and completely bypassed recomputing the 2,000-token system prompt.

Step 4: Connecting RadixAttention to aiFetchly for Agentic Data Scraping

In automated desktop scraping tools like AiFetchly, agents repeatedly query local LLMs to parse DOM structures. By routing requests through an SGLang server running RadixAttention, desktop extractors eliminate latency spikes during iterative multi-page crawls:

class aiFetchlyRadixConnector: """Routes aiFetchly desktop extraction tasks through SGLang RadixAttention.""" def __init__(self, sglang_endpoint: str): self.endpoint = sglang_endpoint def extract_page_data(self, html_chunk: str, schema_rules: str) -> dict: messages = [ {"role": "system", "content": f"Schema Rules:\n{schema_rules}"}, {"role": "user", "content": f"Analyze HTML:\n{html_chunk}"} ] # SGLang automatically caches 'Schema Rules' across hundreds of pages! response = requests.post( f"{self.endpoint}/v1/chat/completions", json={"model": "default", "messages": messages, "temperature": 0.0} ) return response.json()

Summary & Architectural Decision Matrix for LLM Serving Engines

Selecting the appropriate LLM serving engine depends directly on your system’s prompt topology and conversational depth:

Inference Engine Architectural Decision Matrix: Workload & Conversational Topology: ├─ Multi-Turn AI Agents, Tree Search, or aiFetchly Web Scrapers │ └─► Recommendation: SGLang (RadixAttention dynamic tree, LRU leaf eviction, 80%+ TTFT drop) ├─ High-Concurrency Independent Stateless Prompts (Batch Embeddings, RAG One-Shot) │ └─► Recommendation: vLLM (PagedAttention with Continuous Batching & Chunked Prefill) ├─ Enterprise Multi-GPU Clusters with Proprietary NVIDIA TensorRT Kernels │ └─► Recommendation: TensorRT-LLM (Maximum raw FLOP utilization on H100/B200 clusters) └─ Single-User Edge Devices & CPU Quantization (Laptops, Mobile) └─► Recommendation: llama.cpp / Ollama (GGUF 4-bit quantizations, minimal RAM footprint)

Architectural Takeaways

  1. Adopt Dynamic Tree Caching for Agents: In multi-turn dialogues and tool-calling loops, standard linear batching wastes the majority of compute on redundant prefills. A radix tree is the natural data structure for hierarchical prompt reuse.
  2. Decouple Allocation from Eviction: Structuring the KV cache as a tree with reference counting allows the engine to preserve shared root prefixes indefinitely while safely evicting stale leaves under memory pressure.
  3. Co-Design Routing with Cache Locality: Combining tree-structured caching with cache-aware request scheduling prevents cache thrashing and delivers predictable sub-200ms latency across enterprise AI workloads.