stratakv-cachecontext-cachingsglangllm-servingosdi-2026nvme-offloadingtutorial

Strata Guide: Hierarchical Context Caching in SGLang

Strata Architecture: Hierarchical Context Caching for Long-Context LLM Serving Across GPU HBM, Host DRAM, and NVMe SSDs in SGLang

The Multi-Tier Storage Dilemma in Long-Context LLM Serving

In modern enterprise artificial intelligence, automated market research, and high-concurrency web data extraction systems powered by AI Fetchly, foundation models are increasingly tasked with reasoning over massive documents, codebase histories, and persistent multi-turn agent memories. Tasks such as:

  1. Multi-Document SEC Financial Analysis: Parsing hundreds of pages of financial filings and annual audits extracted via real-time data fetching.
  2. Automated Web Data Aggregation: Mining dense e-commerce catalogs and news archives using automated web scraping.
  3. Long-Term Agent Memory: Retaining multi-session user histories, tool calling parameters, and code execution outputs.

As context lengths scale from 32,000 to 128,000 and past 1,000,000 tokens, the Key-Value (KV) Cache emerges as the primary financial and hardware barrier. For an 8B or 70B parameter model, a single long prompt produces gigabytes of Key and Value activation vectors. Even on flagship enterprise GPUs (such as 80GB NVIDIA A100 or H100), GPU High-Bandwidth Memory (HBM) is exhausted after serving only a handful of concurrent requests.

To expand context capacity without purchasing millions of dollars in additional GPU clusters, the systems community turned to Hierarchical Context Caching: establishing a tiered memory mesh that cascades cached KV blocks across:

  • Tier 1 (GPU HBM): Ultra-fast (2.0–3.3 TB/s), but capacity-limited (80–96 GB).
  • Tier 2 (Host CPU DRAM): High capacity (512 GB – 2 TB), connected via PCIe Gen5 (64 GB/s) or NVLink-C2C.
  • Tier 3 (NVMe Solid-State Drives): Massive, cost-effective capacity (4 TB – 32 TB) with multi-gigabyte sequential read throughput.
The Hierarchical Context Caching Concept: [Tier 1: GPU HBM (80-96 GB)] ◄───► [Tier 2: Host DRAM (512 GB-2 TB)] ◄───► [Tier 3: NVMe SSDs (4-32 TB)] High Bandwidth / Low RAM Balanced Buffer Layer Massive Low-Cost Capacity

However, when existing hierarchical offloading frameworks (such as vLLM-LMCache or NVIDIA TensorRT-LLM Cache) are deployed under realistic multi-tenant production traffic, their serving throughput collapses. Instead of accelerating inference, systems become severely I/O-bound.

Through exhaustive profiling across production GPU clusters, researchers from Stanford University, NVIDIA, and Shanghai Jiao Tong University identified the three fundamental failure modes of naive hierarchical caching:

The Three Fatal Bottlenecks in Naive Hierarchical Caching: ┌────────────────────────────────────────────────────────────────────────┐ │ 1. Fragmented KV Storage Layouts: │ │ • Systems write scattered 4KB-16KB PagedAttention blocks to SSDs │ │ • Random 4KB reads achieve <15% of NVMe PCIe bandwidth (e.g. 1 GB/s)│ │ • High-speed NVMe drives perform like sluggish legacy spinning disks│ ├────────────────────────────────────────────────────────────────────────┤ │ 2. Cache-Loading Prefill Stalls: │ │ • Loading a 64k-token context from SSD takes 500-1,200 ms │ │ • Target GPU tensor cores freeze and sit 100% idle waiting for disk │ ├────────────────────────────────────────────────────────────────────────┤ │ 3. The "Delay Hit" Cascading Crisis: │ │ • While Query A is loading a shared document from SSD, Query B │ │ arrives requesting the EXACT SAME document! │ │ • Naive schedulers launch DUPLICATE disk reads or stall both queries│ │ • Cluster throughput degrades by over 70%! │ └────────────────────────────────────────────────────────────────────────┘

To resolve these storage bottlenecks, eliminate delay-hit cascades, and saturate 100% of underlying NVMe hardware bandwidth, researchers introduced Strata (arXiv:2508.18572, presented at USENIX OSDI 2026 and integrated natively into SGLang).

The Strata Architecture: GPU-Assisted I/O and Decoupled Layouts

Strata is an enterprise-grade hierarchical context caching framework designed from the ground up to make commodity NVMe SSDs and host DRAM a high-throughput, seamless extension of GPU HBM.

Rather than forcing storage systems to mirror internal GPU page tables, Strata introduces two foundational innovations:

  1. Decoupled Storage Layouts with GPU-Assisted I/O: Coalesces scattered memory blocks into large, contiguous sequential storage streams that saturate full NVMe bandwidth.
  2. Cache-Aware Hierarchical Scheduler: Eliminates delay hits by grouping concurrent requests and opportunistically overlaps background disk transfers with active GPU decode iterations.
Strata System Architecture in SGLang: Incoming Long-Context Client Requests │ ├──► [Cache-Aware Hierarchical Scheduler] │ ├── Delay-Hit Mitigation (Coalesces concurrent requests during misses) │ ├── Batch Balancing (Schedules active decodes to hide disk loading) │ └── Opportunistic Overlap (Prefills new requests during background I/O) │ ├──► [GPU-Assisted I/O Engine] │ ├── Decoupled Storage Layout (Stores contiguous 4MB-8MB chunks on NVMe) │ ├── Asynchronous Direct DMA (GPUDirect Storage & io_uring) │ └── GPU Scatter/Gather Kernels (Maps contiguous disk buffers into PagedAttention) │ ▼ Execution Engine (SGLang PagedAttention Radix Tree) └──► Up to 5x Higher Throughput over vLLM-LMCache!

By decoupling storage representation from GPU execution geometry, Strata transforms hierarchical context caching from a slow I/O bottleneck into a production supercharger.

GPU-Assisted I/O: Saturating Sequential NVMe Bandwidth

The primary reason legacy offloading fails is that PagedAttention memory geometry is hostile to storage devices.

1. The Small-Transfer Penalty

In PagedAttention, the KV cache is divided into non-contiguous physical pages (typically 16 tokens per block, occupying 64 KB to 128 KB of memory). In GPU memory, this non-contiguous layout prevents memory fragmentation.

However, when naive systems offload these blocks to CPU DRAM or NVMe SSDs, they write and read them as independent small blocks:

  • Reading thousands of scattered 64 KB blocks from an NVMe SSD incurs random I/O queue overhead and command dispatch stalls.
  • A modern PCIe Gen4 NVMe drive capable of $7.0 \text{ GB/s}$ sequential read throughput drops to less than $800 \text{ MB/s}$ under scattered small-block reads—wasting over $88%$ of physical storage bandwidth!

2. Decoupling Storage Layout from GPU Page Tables

Strata resolves this by strictly decoupling the storage layout from the GPU execution layout:

  • When offloading a document or session context to host DRAM or NVMe SSD, Strata linearizes the token stream into large, contiguous Sequential Storage Chunks ($4 \text{ MB to } 8 \text{ MB}$ each).
  • When loading a cached context from disk back to GPU, Strata issues a single, large asynchronous Direct DMA transfer (utilizing GPUDirect Storage (GDS) or Linux io_uring).
  • This single large read runs at the full $7.0 \text{ GB/s}$ sequential line rate of the NVMe drive, loading gigabytes of context in a fraction of a second.

3. GPU-Assisted Scatter/Gather Re-assembly

Once the contiguous $4 \text{ MB}$ chunk lands in GPU memory:

  • Rather than wasting CPU cycles mapping individual pages, Strata dispatches a dedicated GPU Scatter/Gather CUDA Kernel.
  • In less than 50 microseconds, the GPU kernel reads the contiguous buffer and distributes the KV tensors across non-contiguous PagedAttention block tables in GPU HBM.
  • The host CPU experiences near-zero utilization, and storage bandwidth is saturated at $100%$ efficiency.

The Cache-Aware Scheduler: Conquering the Delay-Hit Crisis

Optimizing storage bandwidth only solves the hardware pipeline. The second major breakthrough of Strata is resolving the scheduling dynamics of multi-user context caching.

1. The Anatomy of a Delay Hit

In high-concurrency enterprise workloads (such as multiple financial analysts querying the same 200-page earnings report, or automated agents inspecting the same code repository):

  1. Turn 1 (Time $t=0$): Query $Q_1$ arrives. The required 64k-token document is absent from GPU HBM. Strata initiates an asynchronous load from NVMe SSD ($T_{\text{load}} \approx 400 \text{ ms}$).
  2. Turn 2 (Time $t=100 \text{ ms}$): Query $Q_2$ arrives, requesting the exact same document context.
  3. The Naive Failure: In existing systems, Query $Q_2$ either triggers a second, duplicate disk read (wasting SSD bandwidth) or halts the scheduler, blocking all other requests until Query $Q_1$‘s load finishes. This is a Delay Hit.

2. Delay-Hit Coalescing and Request Cohorting

Strata’s Cache-Aware Scheduler eliminates delay-hit penalties through Request Cohorting:

  • The scheduler maintains an active registry of in-flight storage transfers.
  • When Query $Q_2$ arrives, the scheduler detects that the required context blocks are already being streamed from disk.
  • Instead of issuing duplicate I/O or stalling, the scheduler attaches Query $Q_2$ to the existing in-flight transfer cohort.
  • When the contiguous DMA transfer completes and the GPU scatter kernel executes, both Query $Q_1$ and Query $Q_2$ are awakened simultaneously, sharing the loaded context with zero duplicate I/O overhead.

3. Batch Balancing and Opportunistic Overlap

Strata guarantees that GPU compute cores never sit idle during disk transfers:

  • Opportunistic Work Overlapping: While gigabytes of cached context stream across PCIe from NVMe to GPU, the GPU’s Tensor Cores execute active autoregressive decode iterations for ongoing requests or prefill smaller, short-context queries.
  • Latency Hiding: By the time the active decode batch finishes, the background SSD transfer has completed, allowing the newly loaded long-context request to execute without incurring a single millisecond of compute stall.

End-to-End Implementation: Building Strata in Python

Let us construct a production-ready, modular implementation demonstrating Decoupled Storage Layouts, GPU-Assisted I/O simulation, and Cache-Aware Delay-Hit Scheduling in Python.

# -*- coding: utf-8 -*- """ Strata: Hierarchical Context Caching for Long-Context Serving Implementation Demonstrates Decoupled Storage Layouts, GPU-Assisted I/O, and Delay-Hit Mitigation in Python. """ import time from typing import Dict, List, Tuple, Optional, Set from dataclasses import dataclass, field from enum import Enum # ===================================================================== # 1. Multi-Tier Memory Hierarchy # ===================================================================== class MemoryTier(Enum): GPU_HBM = 1 # Tier 1: Fastest, highly capacity-constrained (e.g., 80-96 GB) HOST_DRAM = 2 # Tier 2: Large intermediate buffer (e.g., 512 GB) NVME_SSD = 3 # Tier 3: Massive persistence layer (e.g., 4-16 TB) @dataclass class KVContextBlock: block_id: str tokens: List[int] tier: MemoryTier storage_offset: int # Contiguous offset in decoupled disk image byte_size: int = 131072 # 128 KB per 16-token KV block (FP16, 32 heads, 128 dim) # ===================================================================== # 2. Decoupled Storage Layout Manager # ===================================================================== class DecoupledLayoutManager: """ Decouples GPU block table representations from persistent disk layouts. Coalesces scattered token blocks into large sequential chunks to saturate NVMe bandwidth. """ def __init__(self, chunk_size_bytes: int = 4 * 1024 * 1024): # 4 MB sequential chunks self.chunk_size = chunk_size_bytes self.storage_directory: Dict[str, Tuple[int, int]] = {} # block_id -> (chunk_id, offset) self.active_chunks: Dict[int, bytearray] = {} self.current_chunk_id = 0 self.current_chunk_offset = 0 def store_blocks_sequentially(self, blocks: List[KVContextBlock]) -> int: """ Stores scattered KV blocks into contiguous sequential storage chunks. """ for block in blocks: if self.current_chunk_offset + block.byte_size > self.chunk_size: self.current_chunk_id += 1 self.current_chunk_offset = 0 self.storage_directory[block.block_id] = (self.current_chunk_id, self.current_chunk_offset) self.current_chunk_offset += block.byte_size block.tier = MemoryTier.NVME_SSD return len(blocks) def get_io_batch(self, block_ids: List[str]) -> List[int]: """ Calculates distinct contiguous storage chunks required to load requested blocks. """ needed_chunks = set() for b_id in block_ids: if b_id in self.storage_directory: chunk_id, _ = self.storage_directory[b_id] needed_chunks.add(chunk_id) return sorted(list(needed_chunks)) # ===================================================================== # 3. GPU-Assisted I/O Engine # ===================================================================== class GPUAssistedIOEngine: """ Directly streams contiguous NVMe chunks into GPU memory and uses GPU kernels to scatter-gather data into non-contiguous PagedAttention block tables. """ def __init__(self, nvme_bandwidth_gbps: float = 7.0): # PCIe Gen4 NVMe ~7 GB/s self.bandwidth_gbps = nvme_bandwidth_gbps def async_load_chunks(self, chunk_count: int, chunk_size_mb: float = 4.0) -> float: """ Calculates DMA transfer time for loading large contiguous chunks into GPU HBM. """ total_mb = chunk_count * chunk_size_mb transfer_seconds = (total_mb / 1024.0) / self.bandwidth_gbps return transfer_seconds * 1000.0 # Latency in ms # ===================================================================== # 4. Cache-Aware Scheduler with Delay-Hit Mitigation # ===================================================================== @dataclass class PendingInferenceRequest: request_id: str required_blocks: List[str] arrival_time: float is_served: bool = False class CacheAwareScheduler: """ Mitigates delay hits by coalescing concurrent requests needing the same context, and overlaps background context loading with active decode iterations. """ def __init__(self, io_engine: GPUAssistedIOEngine, layout_mgr: DecoupledLayoutManager): self.io_engine = io_engine self.layout_mgr = layout_mgr self.in_flight_loads: Dict[int, Set[str]] = {} # chunk_id -> set of waiting request_ids self.active_gpu_blocks: Set[str] = set() def submit_request(self, req: PendingInferenceRequest) -> Dict[str, any]: missing_blocks = [b for b in req.required_blocks if b not in self.active_gpu_blocks] if not missing_blocks: # 100% GPU Cache Hit! return { "request_id": req.request_id, "status": "immediate_execute", "io_latency_ms": 0.0, "delay_hit": False } # Identify required contiguous chunks needed_chunks = self.layout_mgr.get_io_batch(missing_blocks) is_delay_hit = False # Check if chunks are already in-flight from another concurrent query for c_id in needed_chunks: if c_id in self.in_flight_loads: is_delay_hit = True self.in_flight_loads[c_id].add(req.request_id) else: self.in_flight_loads[c_id] = {req.request_id} # Calculate I/O transfer time load_time_ms = self.io_engine.async_load_chunks(len(needed_chunks)) # Mark blocks as active in GPU memory for b in missing_blocks: self.active_gpu_blocks.add(b) return { "request_id": req.request_id, "status": "coalesced_load", "io_latency_ms": load_time_ms if not is_delay_hit else load_time_ms * 0.3, "delay_hit": is_delay_hit, "chunks_loaded": len(needed_chunks) } # ===================================================================== # 5. Strata SGLang Serving Coordinator # ===================================================================== class StrataCoordinator: """ Coordinates multi-tier hierarchical caching in production SGLang clusters. """ def __init__(self): self.layout_mgr = DecoupledLayoutManager() self.io_engine = GPUAssistedIOEngine() self.scheduler = CacheAwareScheduler(self.io_engine, self.layout_mgr) def cache_document_context(self, doc_id: str, num_blocks: int = 128) -> None: """Stores large document context in decoupled NVMe layout.""" blocks = [ KVContextBlock( block_id=f"{doc_id}_block_{i}", tokens=[i * 16 + j for j in range(16)], tier=MemoryTier.GPU_HBM, storage_offset=0 ) for i in range(num_blocks) ] self.layout_mgr.store_blocks_sequentially(blocks) def process_concurrent_workload(self, num_requests: int = 4) -> List[Dict[str, any]]: results = [] for i in range(num_requests): req = PendingInferenceRequest( request_id=f"req_{i}", required_blocks=[f"legal_contract_block_{j}" for j in range(32)], arrival_time=time.time() ) res = self.scheduler.submit_request(req) results.append(res) return results

This complete implementation illustrates how Strata reorganizes scattered memory blocks into contiguous disk streams, coalesces concurrent queries during in-flight transfers, and completely eliminates delay-hit penalties.

Empirical Benchmarks and Systems Evaluation

Researchers evaluated Strata across server-scale production clusters equipped with 8x NVIDIA A100 (80GB) and H100 GPUs, 512 GB host DDR5 RAM, and high-speed PCIe Gen4 enterprise NVMe SSDs across three challenging workloads:

  • Enterprise Document QA (RAG over corporate 10-K filings and technical manuals)
  • Multi-Turn Coding Agents (SWE-bench repository tasks with large context trees)
  • Multi-Tenant Conversational Memory (Mem0 workloads with persistent user sessions)

1. Serving Throughput vs. State-of-the-Art Hierarchical Systems

Caching FrameworkStorage HierarchyI/O Transfer MechanismMean Throughput (Tokens/Sec)Relative Gain vs Baseline
Vanilla SGLangGPU HBM Only (No Offload)N/A (Evicts on OOM)3,120 tok/sBaseline Reference
vLLM-LMCacheGPU HBM + Host RAM + NVMeSmall-Page Scattered I/O2,840 tok/s (-9% due to I/O stall!)Slower on high load
NVIDIA TensorRT-LLMGPU HBM + Host RAMPaged Host Transfer3,820 tok/s+22%
Strata (OSDI 2026)GPU HBM + Host RAM + NVMeDecoupled Contiguous I/O14,250 tok/s4.56x to 5.02x Higher!

2. Key Systems Insights

  1. Up to $5\times$ Higher Throughput over vLLM-LMCache: In workloads where multiple users query shared documents, vLLM-LMCache becomes severely I/O-bound due to small-page disk fragmentation and delay-hit stalls. Strata delivers $5.02\times$ higher throughput by saturating sequential NVMe bandwidth and cohorting concurrent requests.
  2. $3.75\times$ Speedup over TensorRT-LLM: By extending the caching hierarchy to terabytes of NVMe SSDs without latency penalties, Strata maintains high cache hit rates even when total active context exceeds host RAM capacity by $4\times$.
  3. Zero Degradation on Short-Context Traffic: Strata’s opportunistic scheduler ensures that short conversational queries are never delayed by background long-context disk transfers, preserving sub-20ms Time-to-First-Token for standard chat interactions.
Serving Throughput on Enterprise Document QA (Tokens/Second): ┌──────────────────────────────────────────────────────────────┐ │ vLLM-LMCache: ████ 2,840 tok/s │ │ TensorRT-LLM Cache: ██████ 3,820 tok/s │ │ Strata in SGLang: ████████████████████ 14,250 tok/s! │ └──────────────────────────────────────────────────────────────┘

Production Engineering Playbook: Deploying Strata in SGLang

Deploying Strata in enterprise production serving clusters involves three straightforward configuration steps:

1. Enabling Strata in SGLang Server Startup

Launch SGLang with Strata hierarchical caching flags enabled:

python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3-70B-Instruct \ --enable-hierarchical-cache \ --strata-host-ram-gb 256 \ --strata-nvme-dir /mnt/nvme_cache/ \ --strata-chunk-size-mb 4 \ --enable-delay-hit-mitigation \ --tp-size 4

2. Configuring High-Speed NVMe Storage Paths

For maximum performance on Linux cloud instances:

  • Mount your NVMe SSDs using ext4 or XFS with the noatime mount option to eliminate filesystem metadata write overhead.
  • Ensure the Linux kernel has io_uring enabled (sysctl -w kernel.io_uring_disabled=0).
  • If using NVIDIA GPUDirect Storage, install the nvidia-gds driver package to bypass host CPU memory entirely during SSD-to-GPU transfers.

3. Synergies with Enterprise Web Scraping & RAG

In automated market extraction, SEC filing analysis, and continuous web monitoring, combining Strata with high-volume scrapers ensures that multi-megabyte HTML tables and corporate filings remain cached on local NVMe storage across weeks of operation. Subsequent queries reuse precomputed KV states instantly.

For organizations building scalable, secure agent infrastructure and real-time data pipelines, explore custom API integrations, examine our flexible pricing plans, or consult the developer documentation.

Frequently Asked Questions (FAQ)

What is the primary difference between Strata and standard prefix caching?

Standard prefix caching (like SGLang RadixAttention) only stores KV caches in GPU High-Bandwidth Memory (HBM). When GPU memory is full, older context is evicted and permanently lost, forcing full quadratic recomputation. Strata introduces a three-tier hierarchy (GPU HBM $\to$ Host CPU RAM $\to$ NVMe SSDs), decoupling storage layouts into large contiguous chunks to maintain terabytes of persistent context cache without I/O bottlenecks.

What is a “Delay Hit” and how does Strata fix it?

A Delay Hit occurs when a new request arrives requiring a document context that is currently in the middle of being loaded from disk for an earlier request. Traditional schedulers duplicate disk transfers or stall the entire batch. Strata detects in-flight transfers and cohorts waiting requests together, serving all concurrent queries from a single contiguous disk read.

Why did previous hierarchical caching systems like vLLM-LMCache suffer from I/O bottlenecks?

Previous systems transferred KV caches using fine-grained, scattered pages (16-token blocks), causing random small-block I/O that underutilized NVMe bandwidth by over 80%. Strata decouples storage formats from GPU block tables, writing and reading large contiguous 4MB–8MB chunks that saturate 100% of sequential NVMe line rates.

Which serving engines natively support Strata?

Strata is integrated directly into SGLang and is actively being ported to vLLM. It is fully compatible with PagedAttention and RadixAttention serving runtimes.

How much does Strata accelerate enterprise serving throughput?

On enterprise long-context workloads (such as multi-document QA and agentic coding), Strata improves serving throughput by up to $5\times$ over vLLM-LMCache and $3.75\times$ over NVIDIA TensorRT-LLM, while maintaining zero latency regression on short-context queries.