StrataContext CachingKV CacheLong-Context LLMsOSDI 2026GPU-Assisted IOSGLangSystems Engineering

Strata Guide: Hierarchical Context Caching for Long LLMs

October 22, 2026

Strata Architecture: Hierarchical Context Caching for Long-Context LLMs

The expansion of context windows in frontier foundation models has fundamentally reshaped machine learning capabilities. Architectures such as Google Gemini, Qwen-2.5, DeepSeek-V3, Claude 3.5, and LLaMA-3.1 now routinely process prompts spanning from 128,000 to over one million tokens. These expansive context horizons unlock transformative reasoning over entire enterprise codebases, legal repositories, complex financial filings, and dense multi-turn agent interactions.

However, executing inference over ultra-long sequences exposes a crippling computational barrier: the quadratic cost of redundant prefill computation. When an application executes multiple queries against the same reference document or sustains a multi-turn conversation, recomputing the intermediate Key-Value (KV) activations from raw tokens for every interaction wastes thousands of GPU TeraFLOPs and inflates Time-To-First-Token (TTFT) into tens of seconds.

To circumvent this bottleneck, foundation model providers have converged on context caching (also designated as prefix caching). By preserving previously computed KV activations in memory, inference engines bypass the prompt prefill phase entirely, directly executing subsequent token generation.

Yet, as context lengths scale into hundreds of thousands of tokens, the physical footprint of uncompressed KV tensors quickly exceeds the High-Bandwidth Memory (HBM) capacity of physical GPUs. On an 80GB NVIDIA H100 accelerator, reserving KV cache for a single 128K context on a 70B parameter model demands over 32GB of HBM, exhausting memory after serving merely two concurrent sessions. Consequently, production serving clusters are forced to adopt hierarchical caching—offloading inactive KV states to host CPU DRAM, local NVMe solid-state drives, or remote memory pools.

Transferring these massive cached tensors back to GPU HBM introduces a catastrophic systems bottleneck. In production benchmarks on long-context datasets, state-of-the-art serving frameworks spend up to 74% of total prefill execution time stalled on I/O transfers, degrading end-to-end serving throughput by up to 4x.

Presented at the 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2026) by a joint research collaboration between Stanford University and NVIDIA, Strata introduces a clean-sheet hierarchical context caching architecture. By replacing conventional DMA transfers with GPU-assisted I/O, decoupling memory layouts between compute and storage tiers, and deploying a cache-aware scheduler equipped with HiRadixTree transient tracking and bubble filling, Strata achieves up to 5x lower TTFT compared to vLLM with LMCache and a 3.75x speedup over NVIDIA TensorRT-LLM on long-context workloads.

In this comprehensive architectural guide, we dissect the physics of the long-context I/O bottleneck, formalize Strata’s GPU-assisted transfer engine, implement its core layout transformation kernels and cache-aware scheduler, and benchmark performance across enterprise GPU clusters.

Problem: The Long-Context I/O Wall and Paged Fragmentation

To understand why conventional hierarchical caching collapses under long-context traffic, systems architects must examine the underlying mechanics of memory paging and PCIe transfer dynamics.

The Physics of I/O Bandwidth and Little’s Law

The achievable throughput of any memory I/O subsystem is strictly governed by the foundational queueing formulation known as Little’s Law. Let $\lambda$ denote the arrival rate of I/O requests, $C$ represent the average number of concurrent in-flight I/O operations, and $L$ denote the average latency per operation. By Little’s Law: $$C = \lambda \cdot L$$

If $X$ represents the sustained data throughput (in gigabytes per second) and $S$ denotes the average transfer size per I/O operation, the aggregate throughput is: $$X = \lambda \cdot S = \frac{C \cdot S}{L}$$

+-----------------------------------------------------------------------------+
|                LITTLE'S LAW APPLIED TO GPU MEMORY TRANSFERS                 |
+-----------------------------------------------------------------------------+
|                                                                             |
|  Attainable Bandwidth:  X = (C * S) / L                                     |
|                                                                             |
|  - C (Concurrency): Bounded by CPU thread pools & CUDA driver queues.       |
|  - L (Latency):     Fixed overhead of PCIe DMA submission & driver sync.    |
|  - S (Chunk Size):  Governed by KV cache page size (typically 16-32 tokens).|
|                                                                             |
|  Outcome: When S is small (e.g., 8KB - 64KB per page), achieving high X     |
|  requires immense concurrency C that overwhelms the host CPU driver,        |
|  causing PCIe 5.0 bandwidth utilization to collapse to 5% - 22%!           |
|                                                                             |
+-----------------------------------------------------------------------------+

To saturate modern high-speed interconnects—such as PCIe 5.0 (delivering ~64 GB/s unidirectional bandwidth) or NVIDIA Grace Hopper NVLink-C2C (delivering 900 GB/s)—individual transfer payloads $S$ must reach the megabyte range (1MB to 4MB). When transfer sizes fall into the tens of kilobytes, the fixed latency $L$ of initiating DMA transfers through host CPU driver queues dominates, causing effective link utilization to collapse.

The Conflict Between PagedAttention and Transfer Efficiency

Modern LLM serving engines (such as vLLM and SGLang) manage GPU memory using PagedAttention. PagedAttention partitions KV caches into small, fixed-size pages—typically containing 16 or 32 tokens—to eliminate internal memory fragmentation and maximize cache hit rates during prefix matching.

However, this fine-grained paging creates severe data fragmentation across the memory hierarchy:

+-----------------------------------------------------------------------------+
|              PAGED ATTENTION MEMORY FRAGMENTATION ACROSS TIERS              |
+-----------------------------------------------------------------------------+
|                                                                             |
|  Logical Context: [ Token 0 ---------------------------------- Token 8192 ] |
|                                                                             |
|  GPU HBM Physical Pages (Page Size = 32 tokens, ~64KB per layer):          |
|  [ Page 12 ] ──► [ Page 84 ] ──► [ Page 03 ] ──► [ Page 201 ] ...           |
|                                                                             |
|  Transfer Mechanism: Standard cudaMemcpyAsync() Loop                        |
|  - Driver issues 256 independent DMA transfers per layer!                   |
|  - For a 32-layer model: 8,192 tiny DMA calls per request!                  |
|  - Hardware Link Utilization: ONLY 22% of PCIe 5.0 (14 GB/s vs 64 GB/s)     |
|                                                                             |
+-----------------------------------------------------------------------------+

When an inference engine loads an 8,192-token cached context from host CPU DRAM into GPU HBM using standard asynchronous memory copies (cudaMemcpyAsync), it must issue thousands of tiny transfer commands. As benchmarked by the authors of Strata, transferring an 8K-token context with 32-token pages achieves less than 22% of theoretical PCIe 5.0 bandwidth. On platforms like the NVIDIA GH200 Grace Hopper Superchip—which replaces PCIe with 900 GB/s NVLink—bandwidth utilization plummets to an astonishing 5%, squandering the accelerator’s physical capabilities.

Why not simply enlarge the page size to 512 or 1,024 tokens to improve transfer efficiency? As demonstrated in extensive microbenchmarks on the ShareGPT and LooGLE datasets, increasing page size destroys cache hit rates. Because prefix matching operates at page boundaries, large page granularities prevent partial prefix reuse, driving average and p90 TTFT latencies up by 2x to 2.9x.

The Loading-Bound Scheduling Dilemma and Delay Hits

Existing hierarchical caching frameworks operate under the optimistic assumption that prefill computation is sufficient to hide cache loading latency. In a multi-layer transformer, engines attempt to overlap the transfer of layer $N+1$‘s KV cache over PCIe while the GPU computes attention on layer $N$.

In long-context serving, this assumption completely breaks down. Consider an enterprise query analyzing an 80,000-token financial filing: the user prompt contains 80,000 context tokens (cached in host DRAM) and only 100 new query tokens. Computing the forward pass for 100 tokens requires negligible GPU compute FLOPs (sub-50 milliseconds). However, transferring the 80,000 cached tokens over PCIe takes hundreds of milliseconds. The execution pipeline degrades from being compute-bound into a purely I/O-bound stall.

+-----------------------------------------------------------------------------+
|                        THE DELAY HIT CONCURRENCY CRISIS                     |
+-----------------------------------------------------------------------------+
|                                                                             |
|  Request 1 (T=0ms): Arrives with Context Alpha (80K Tokens)                 |
|                     - Cache Miss in GPU HBM!                                |
|                     - Triggers Full Prefill Compute on GPU                  |
|                                                                             |
|  Request 2 (T=10ms): Arrives with identical Context Alpha                   |
|                      - Asynchronous Scheduler forms next batch immediately! |
|                      - Context Alpha is NOT yet hot in HBM (In-Flight)       |
|                      - Request 2 experiences FALSE CACHE MISS!              |
|                      - Triggers Redundant 80K Token Recomputation!          |
|                                                                             |
+-----------------------------------------------------------------------------+

Furthermore, multi-tenant enterprise applications—such as distributed web scraping agents or concurrent document summarizers—frequently generate Delay Hits. A delay hit occurs when Request 2 arrives querying the exact same context while Request 1’s cache miss is still actively resolving on the GPU. Modern asynchronous schedulers, which prepare batches ahead of time, treat Request 2 as an independent cache miss, launching redundant prefill computations that waste millions of GPU cycles.

Solution: Strata Hierarchical Caching and GPU-Assisted I/O

To overcome the long-context I/O wall, Strata introduces a dual-pronged architectural solution: a high-efficiency GPU-Assisted I/O Engine that achieves near-line-rate bandwidth on small fragmented pages, coupled with an intelligent Cache-Aware Scheduling Control Plane.

+-----------------------------------------------------------------------------+
|                      STRATA ARCHITECTURAL FOUNDATIONS                       |
+-----------------------------------------------------------------------------+
|                                                                             |
|  1. GPU-ASSISTED I/O:                                                       |
|     - Replaces repetitive CPU-driven cudaMemcpyAsync() calls with           |
|       specialized, persistent CUDA transfer kernels.                         |
|     - Spawns thousands of GPU threads that stream 128-byte words directly   |
|       between pinned host DRAM and GPU global memory at line rate.          |
|     - Achieves 50 GB/s on PCIe 5.0 (~78% saturation) on 32-token pages!     |
|                                                                             |
|  2. ON-THE-FLY MEMORY LAYOUT DECOUPLING:                                    |
|     - GPU Memory: Formatted in compute-friendly Layer-First layout.         |
|     - Host/Disk Memory: Formatted in transfer-friendly Page-First layout.   |
|     - Layout transpositions performed in GPU registers at ZERO compute cost.|
|                                                                             |
|  3. CACHE-AWARE SCHEDULER:                                                  |
|     - HiRadixTree with Transient Nodes: Tracks and defers delay hits.       |
|     - Balanced Batch Formation: Pairs heavy I/O loads with compute tokens.  |
|     - Bubble Filling: Injects ongoing decode steps into unavoidable stalls. |
|                                                                             |
+-----------------------------------------------------------------------------+

The Mechanics of GPU-Assisted I/O

In conventional systems, the host CPU acts as the active master of DMA operations. To transfer hundreds of non-contiguous KV pages, the CPU issues individual driver calls that submit descriptors into hardware command queues. The overhead of crossing the kernel boundary and serializing submissions throttles concurrency.

Strata reverses this control model by leveraging GPU-Assisted I/O. Instead of calling cudaMemcpyAsync, Strata launches a specialized CUDA kernel on the accelerator:

Massive Parallelism ($C$): The transfer kernel spawns thousands of active GPU threads. Each thread is assigned a 128-byte chunk of KV cache data.
Fine-Grained Transfer Efficiency ($S$): Modern NVIDIA GPU memory controllers natively support 128-byte burst transactions. Because the kernel operates at 128-byte granularity, it streams small page chunks into registers and across PCIe without incurring DMA descriptor overhead.
Microsecond Execution: Thousands of threads load data from host pinned memory into local registers via standard memory instructions, immediately storing them into target HBM addresses.

Non-Intrusive Resource Sizing

A critical systems challenge of running I/O kernels on the GPU is SM interference: if the I/O kernel consumes too many Streaming Multiprocessors (SMs) or registers, it starves concurrent matrix multiplication (GEMM) prefill kernels, degrading performance.

The architects of Strata made a profound empirical discovery: saturating PCIe bandwidth does not require the whole GPU. On an NVIDIA H200 GPU featuring 132 SMs:

Launching just two CUDA blocks of 1,024 threads each (confining the I/O kernel to exactly 2 SMs) sustains nearly 50 GB/s transfer throughput over PCIe 5.0.
Restricting the kernel to 2 SMs and utilizing cache-bypass store instructions (st.global.cs) limits performance degradation on concurrent prefill kernels to under 5%, and on decoding kernels to under 10%.
For non-critical background evictions (GPU to CPU memory), Strata throttles the kernel to a single SM (1 block), completely preserving GPU compute capacity for user-facing inference.

For high-throughput systems—whether processing real-time telemetry from continuous data fetching backends or generating long-context analytical summaries—GPU-assisted I/O guarantees consistent multi-gigabyte streaming without stalling production compute passes.

Architecture: HiRadixTree and Two-Dimensional Layout Decoupling

The Strata system architecture separates data plane execution from control plane scheduling. The data plane is orchestrated by the Strata Cache Controller, while the control plane is driven by the Strata Scheduler operating over a hierarchical prefix structure designated as HiRadixTree.

+-----------------------------------------------------------------------------+
|                         STRATA SYSTEM ARCHITECTURE                          |
+-----------------------------------------------------------------------------+
|                                                                             |
|   Incoming User Requests (Long Context Prompts & Queries)                   |
|         │                                                                   |
|   ┌─────▼───────────────────────────────────────────────────────────────┐   |
|   │ STRATA CACHE-AWARE SCHEDULER                                        │   |
|   │ 1. HiRadixTree Traversal & Delay-Hit Deferral Engine                │   |
|   │ 2. Balanced Batch Formulator (Load-to-Compute Ratio Matching)       │   |
|   │ 3. Bubble-Filling Orchestrator (Interleaving Decode Batches)        │   |
|   └─────┬───────────────────────────────────────────────────────────────┘   |
|         │                                                                   |
|         ├──────────────────────────────────┐                                |
|         ▼                                  ▼                                |
|   ┌───────────────────────────────┐  ┌───────────────────────────────────┐  |
|   │ GPU EXECUTION ENGINE          │  │ STRATA CACHE CONTROLLER           │  |
|   │ - Prefill GEMM Kernels        │  │ - Pinned Host Memory Manager      │  |
|   │ - Autoregressive Decode Step  │  │ - Storage Prefetch Pipeline       │  |
|   │ - FlashAttention / PagedAttn  │  │ - GPU-Assisted I/O Transfer Kern  │  |
|   └─────┬─────────────────────────┘  └─────┬─────────────────────────────┘  |
|         │                                  │                                |
|         ▼                                  ▼                                |
|   ┌─────────────────────────────────────────────────────────────────────┐   |
|   │ GPU HBM: Layer-First Layout  ◄──►  HOST DRAM: Page-First Layout     │   |
|   │ [Layer 0..L][Page 0..P]            [Page 0..P][Layer 0..L]          │   |
|   │ (Zero-Cost Register Transposition during Streaming Transfer)        │   |
|   └─────────────────────────────────────────────────────────────────────┘   |
|                                                                             |
+-----------------------------------------------------------------------------+

Two-Dimensional Memory Layout Decoupling

In transformer inference engines, memory layout requirements diverge radically between computation and storage:

Compute-Friendly Layout (Layer-First): Attention kernels process tokens layer by layer. To prevent strided memory access, GPU HBM organizes KV tensors in a Layer-First layout: $$\text{Tensor}_{\text{GPU}} = [L, P, H, D]$$ where $L$ is the layer index, $P$ is the page index, $H$ is the number of attention heads, and $D$ is head dimension.
Transfer-Friendly Layout (Page-First): Storing data layer-first in host DRAM or SSD storage means that loading a single token sequence requires issuing $L$ fragmented I/O operations across non-contiguous disk or memory offsets. Storage devices demand a contiguous Page-First layout: $$\text{Tensor}_{\text{Host}} = [P, L, H, D]$$

+-----------------------------------------------------------------------------+
|             ON-THE-FLY REGISTER-LEVEL LAYOUT TRANSPOSITION                  |
+-----------------------------------------------------------------------------+
|                                                                             |
|  HOST MEMORY (Page-First Layout - Contiguous per Page):                     |
|  [ Page 0: L0, L1, L2.. ][ Page 1: L0, L1, L2.. ][ Page 2: L0, L1, L2.. ]   |
|         │                                                                   |
|         ▼ (GPU Threads Read Contiguous 128-Byte Blocks into Registers)      |
|  GPU REGISTER FILE: Offset Arithmetic:                                      |
|  dst_addr = (layer_id * total_pages + page_id) * token_stride;              |
|         │                                                                   |
|         ▼ (GPU Threads Write Non-Contiguous Chunks into HBM)                |
|  GPU HBM (Layer-First Layout - Contiguous per Layer):                       |
|  [ Layer 0: P0, P1, P2.. ][ Layer 1: P0, P1, P2.. ][ Layer 2: P0, P1.. ]    |
|                                                                             |
+-----------------------------------------------------------------------------+

In conventional systems, converting between these layouts requires launching an explicit transposition kernel, incurring extra memory read/write passes that consume memory bandwidth.

Because Strata transfers data using GPU threads, layout transformation is completely free. As a GPU thread reads a 128-byte block from host memory, it applies a single arithmetic instruction to calculate its target address in HBM before executing the store instruction. This on-the-fly transposition decouples the memory hierarchy: host RAM and NVMe SSDs maintain contiguous, transfer-friendly page blocks, while GPU HBM receives contiguous, compute-friendly layer blocks without overhead.

HiRadixTree with Transient Nodes

To manage hierarchical cached prefixes across GPU HBM, host DRAM, and NVMe disks, Strata extends SGLang’s RadixTree into HiRadixTree.

A standard RadixTree indexes cached tokens using prefix strings, mapping token sequences to memory indices. Strata introduces Transient Nodes to eliminate delay hits and coordinate asynchronous batch preparation:

In-Queue Nodes: When a new request arrives, the scheduler inserts transient nodes marked in-queue representing its requested prefix. If a subsequent request arrives matching these transient nodes, the scheduler recognizes an impending delay hit.
In-Flight Nodes: As soon as Request 1 begins prefill execution on the GPU, its transient nodes transition to in-flight.
Delay-Hit Deferral: When Request 2 matches an in-flight node, the scheduler defers Request 2 to the immediately following scheduling round rather than launching redundant prefill. Request 2 is placed at the front of the queue, executing as an instant cache hit the moment Request 1’s prefill completes.
Permanent Node Conversion: Once prefill finishes, transient nodes are converted into standard nodes, recording physical memory pointers across HBM, CPU DRAM, and SSD storage.

Implementation: GPU-Assisted I/O Kernels and Bubble-Filling Scheduler

Deploying Strata requires implementing the low-level CUDA transfer kernels and integrating the cache-aware balanced batching algorithm into the serving engine’s scheduling loop.

+-----------------------------------------------------------------------------+
|                     STRATA PIPELINE EXECUTION LIFECYCLE                     |
+-----------------------------------------------------------------------------+
|                                                                             |
|  1. Request Queue Inspection: HiRadixTree Match & Delay-Hit Check           |
|  2. Balanced Batch Formation: Pair I/O Transfers with Compute Tokens        |
|  3. Dispatch Asynchronous Transfer: Launch 2-Block GPU-Assisted I/O Kernel   |
|  4. Bubble Filling: Interleave Active Decoding Passes During I/O Wait       |
|  5. Layer-Overlapped Prefill: Execute Attention with Loaded KV States        |
|  6. Asynchronous Eviction: Back Up Cold Pages to Host Memory (1-Block Kernel)|
|                                                                             |
+-----------------------------------------------------------------------------+

Step 1: High-Performance GPU-Assisted I/O CUDA Kernel

The following CUDA kernel demonstrates Strata’s GPU-assisted streaming transfer. It reads contiguous page-first memory from pinned host DRAM and performs on-the-fly transposition into GPU HBM’s layer-first layout using 128-bit vector memory instructions (uint4):

#include <cuda_runtime.h>
#include <cstdint>

// 128-bit vector type for maximum memory transaction saturation
struct __align__(16) Chunk128 {
    uint64_t low;
    uint64_t high;
};

__global__ void strata_gpu_assisted_load_kernel(
    const Chunk128* __restrict__ host_page_first_src,
    Chunk128* __restrict__ gpu_layer_first_dst,
    const int32_t* __restrict__ page_indices,
    int num_pages,
    int num_layers,
    int chunks_per_page_layer,
    int total_gpu_pages
) {
    // Confined to 2 CUDA Blocks of 1024 threads (2 SMs total)
    int global_tid = blockIdx.x * blockDim.x + threadIdx.x;
    int total_threads = gridDim.x * blockDim.x;

    int total_chunks = num_pages * num_layers * chunks_per_page_layer;

    for (int chunk_idx = global_tid; chunk_idx < total_chunks; chunk_idx += total_threads) {
        // Decompose linear chunk index into logical hierarchy
        int p_offset = chunk_idx / (num_layers * chunks_per_page_layer);
        int rem = chunk_idx % (num_layers * chunks_per_page_layer);
        int layer_idx = rem / chunks_per_page_layer;
        int chunk_in_layer = rem % chunks_per_page_layer;

        // Resolve target physical page ID in GPU HBM
        int target_gpu_page = page_indices[p_offset];

        // Compute contiguous source index in Page-First host memory
        int64_t src_idx = (int64_t)chunk_idx;

        // On-the-fly layout transposition: compute Layer-First destination index
        // GPU Layout: [Layer, Total_GPU_Pages, Chunks_Per_Page_Layer]
        int64_t dst_idx = ((int64_t)layer_idx * total_gpu_pages * chunks_per_page_layer) +
                          ((int64_t)target_gpu_page * chunks_per_page_layer) +
                          chunk_in_layer;

        // Vectorized 128-bit load from pinned CPU DRAM into GPU registers
        Chunk128 val = host_page_first_src[src_idx];

        // Streaming non-temporal store to bypass L1/L2 cache and avoid pollution
        #if __CUDA_ARCH__ >= 700
        asm volatile(
            "st.global.cs.v2.u64 [%0], {%1, %2};"
            :
            : "l"(&gpu_layer_first_dst[dst_idx]), "l"(val.low), "l"(val.high)
            : "memory"
        );
        #else
        gpu_layer_first_dst[dst_idx] = val;
        #endif
    }
}

Step 2: Cache-Aware Scheduler and Bubble-Filling Engine

The Python implementation below illustrates the Strata scheduling loop integrated with SGLang. It implements the Balanced Batch Formation algorithm and fills I/O loading stalls with complementary decoding steps:

#!/usr/bin/env python3
# Strata Cache-Aware Scheduler and Bubble-Filling Engine
# Optimizes batch formation across compute and I/O latency regimes

from typing import List, Dict, Optional
import time

class InferenceRequest:
    def __init__(self, req_id: str, prompt_tokens: int, cached_tokens: int, target_tokens: int):
        self.req_id = req_id
        self.prompt_tokens = prompt_tokens       # New tokens needing prefill compute
        self.cached_tokens = cached_tokens       # Cached context needing I/O load
        self.target_tokens = target_tokens
        self.arrival_time = time.time()
        self.is_decoding = False

class StrataScheduler:
    def __init__(self, pcie_bandwidth_gbps: float = 50.0, compute_tput_tps: float = 12000.0):
        self.pcie_bw = pcie_bandwidth_gbps * 1e9 / 16  # Approx bytes to tokens per sec
        self.compute_tput = compute_tput_tps
        self.active_decode_batch: List[InferenceRequest] = []
        self.waiting_queue: List[InferenceRequest] = []
        self.transient_nodes: Dict[str, str] = {}  # In-queue or in-flight tracking

    def estimate_io_time(self, cached_tokens: int) -> float:
        # 16-bit FP16 KV cache bytes per token: 2 * num_layers * hidden_dim
        bytes_per_token = 2 * 32 * 4096 * 2  # Example: 70B model with GQA
        total_bytes = cached_tokens * bytes_per_token
        return total_bytes / (self.pcie_bw * 16)

    def estimate_compute_time(self, new_tokens: int) -> float:
        return new_tokens / self.compute_tput

    def form_balanced_batch(self, max_tokens: int = 8192) -> Dict:
        """
        Pairs cache-loading requests with sufficient prefill compute to mask I/O latency.
        """
        prefill_candidates = []
        current_compute_tokens = 0
        current_loading_tokens = 0

        # Step 1: Defer requests susceptible to delay hits
        valid_queue = []
        for req in self.waiting_queue:
            if req.req_id in self.transient_nodes and self.transient_nodes[req.req_id] == "in-flight":
                continue  # Defer request to prevent redundant prefill
            valid_queue.append(req)

        # Step 2: Form balanced batch
        for req in valid_queue:
            if current_compute_tokens + req.prompt_tokens > max_tokens:
                break

            io_dur = self.estimate_io_time(req.cached_tokens)
            comp_dur = self.estimate_compute_time(req.prompt_tokens)

            # Check if adding this request maintains balanced execution
            prefill_candidates.append(req)
            current_compute_tokens += req.prompt_tokens
            current_loading_tokens += req.cached_tokens

        # Step 3: Bubble filling - check if batch remains loading-bound
        total_io_time = self.estimate_io_time(current_loading_tokens)
        total_comp_time = self.estimate_compute_time(current_compute_tokens)
        bubble_time = max(0.0, total_io_time - total_comp_time)

        interleaved_decodes = []
        if bubble_time > 0.005 and self.active_decode_batch:
            # Fill the I/O bubble with concurrent decoding passes!
            print(f"[*] I/O Bubble detected ({bubble_time*1000:.1f}ms). Filling bubble with decode steps.")
            interleaved_decodes = self.active_decode_batch[:16]

        return {
            "prefill_batch": prefill_candidates,
            "bubble_decodes": interleaved_decodes,
            "io_stall_ms": bubble_time * 1000
        }

if __name__ == "__main__":
    scheduler = StrataScheduler(pcie_bandwidth_gbps=50.0, compute_tput_tps=12000.0)
    scheduler.waiting_queue = [
        InferenceRequest("doc_analysis_01", prompt_tokens=128, cached_tokens=32768, target_tokens=256),
        InferenceRequest("rag_query_02", prompt_tokens=512, cached_tokens=16384, target_tokens=128),
        InferenceRequest("code_edit_03", prompt_tokens=2048, cached_tokens=4096, target_tokens=512)
    ]
    batch_plan = scheduler.form_balanced_batch(max_tokens=8192)
    print(f"[+] Formed batch with {len(batch_plan['prefill_batch'])} prefill requests.")
    print(f"[+] Scheduled {len(batch_plan['bubble_decodes'])} bubble-filling decode passes.")

Comparison: Strata vs vLLM + LMCache vs TensorRT-LLM

To evaluate Strata’s performance across diverse deployment environments, we benchmark it against state-of-the-art long-context serving systems: vLLM coupled with LMCache, NVIDIA TensorRT-LLM, and vanilla SGLang with CPU offloading.

+----------------------------------------------------------------------------------------------------+
|                         LONG-CONTEXT SERVING FRAMEWORK COMPARISON                                  |
+--------------------------+-----------------------+------------------------+------------------------+
| Feature / Metric         | Strata (OSDI 2026)    | vLLM + LMCache (2025)  | TensorRT-LLM (2025)    |
+--------------------------+-----------------------+------------------------+------------------------+
| I/O Transfer Mechanism   | GPU-Assisted (2 SMs)  | CPU cudaMemcpyAsync()  | Host DMA Pipeline      |
| PCIe 5.0 Bandwidth Sat.  | ~ 78% (50 GB/s)       | ~ 22% (14 GB/s)        | ~ 35% (22 GB/s)        |
| Memory Layout Strategy   | Decoupled (Reg Trans) | Coupled Paged Layout   | Contiguous Chunk       |
| Page Size Flexibility    | Fine-grained (16-32)  | Fixed (32 tokens)      | Large Chunks (64+)     |
| Delay-Hit Mitigation     | HiRadixTree Transient | None (Duplicate Comp)  | None                   |
| Bubble-Filling Scheduling| Yes (Interleave Dec)  | None (FIFO Stall)      | None                   |
| TTFT Latency (LooGLE)    | Up to 5.0x Lower      | High (I/O Bound)       | 3.75x Slower           |
| Short-Context Penalty    | Zero (<1% Overhead)   | Zero                   | Baseline               |
| Open-Source Codebase     | SGLang Integration   | vLLM Plugin            | Closed Engine Driver   |
+--------------------------+-----------------------+------------------------+------------------------+

Comprehensive Evaluation Results

End-to-End Time-To-First-Token (TTFT): On the multi-document LooGLE and NarrativeQA benchmarks using Qwen-2.5-14B and LLaMA-3.1-70B, Strata achieves up to 5x lower TTFT compared to vLLM with LMCache. Because Strata saturates 78% of physical PCIe bandwidth while vLLM achieves only 22%, context loading delays are compressed by over 3.5x.
Speedup Over TensorRT-LLM: Compared to NVIDIA TensorRT-LLM, Strata delivers a 3.75x speedup on long-context queries. While TensorRT-LLM features heavily optimized compute kernels, its scheduler fails to handle delay hits and does not decouple storage and compute layouts.
No Penalty on Short Contexts: On the short-context ShareGPT benchmark (average prompt length ~680 tokens), Strata achieves identical token generation throughput and latency compared to native SGLang, verifying that its 2-SM transfer quota and transient node indexing impose zero runtime penalty when context caching is not required.

Modern enterprise systems—from agentic code generators to automated enterprise AI document extraction pipelines—can deploy Strata to scale context windows without incurring linear infrastructure cost penalties.

Production: Hardware Sizing, GH200 NVLink Analysis, and Best Practices

Deploying Strata in enterprise production environments requires sizing memory tiers correctly, configuring host DRAM buffers, and tailoring transfer parameters to target hardware architectures.

+-----------------------------------------------------------------------------+
|               STRATA THREE-TIER PRODUCTION MEMORY HIERARCHY                 |
+-----------------------------------------------------------------------------+
|                                                                             |
|  TIER 1: GPU HBM (Fastest / Lowest Capacity)                                |
|  - NVIDIA H100 (80GB) or H200 (141GB HBM3e)                                 |
|  - Holds active working set and current batch KV pages in Layer-First layout|
|                                                                             |
|  TIER 2: HOST CPU PINNED DRAM (High Capacity / Line-Rate PCIe)              |
|  - 1.5 TB to 2.0 TB DDR5 Registered ECC Memory per Server Node              |
|  - Stores warm prefixes and multi-turn session caches in Page-First layout   |
|                                                                             |
|  TIER 3: LOCAL NVMe SSD ARRAYS (Massive Capacity / Asynchronous Prefetch)   |
|  - 4x to 8x PCIe Gen5 NVMe SSDs in RAID-0 (Up to 60 GB/s aggregate read)    |
|  - Stores cold historical enterprise documents and reference manuals        |
|                                                                             |
+-----------------------------------------------------------------------------+

Grace Hopper GH200 Platform Optimization

The advantages of Strata become even more pronounced on unified memory superchips like the NVIDIA GH200 Grace Hopper Superchip:

The NVLink-C2C Bandwidth Divergence: GH200 interconnects the Grace CPU and Hopper GPU via NVLink-C2C, delivering 900 GB/s bidirectional interconnect bandwidth—roughly 14x faster than PCIe 5.0.
The Classical Transfer Failure: When conventional systems run on GH200 with 32-token pages, DMA bandwidth drops to less than 5% of NVLink’s theoretical capacity (achieving ~45 GB/s out of 900 GB/s) due to kernel launch latencies.
Strata’s NVLink Scaling: By allocating 4 CUDA blocks for GPU-assisted transfer on GH200, Strata scales transfer bandwidth past 420 GB/s, slashing the context loading time of a 128,000-token prompt from 1.2 seconds down to under 80 milliseconds.

Production Cluster Sizing and Configuration Guidelines

When sizing hardware for a production Strata deployment, infrastructure engineers should adhere to the following best practices:

Host-to-GPU Memory Ratio: Provision at least 16GB to 24GB of Host CPU DRAM for every 1GB of GPU HBM. In an 8x H100-80G node (640GB total HBM), configure 1.5TB to 2.0TB of host DDR5 memory to ensure warm cached prefixes remain resident in host memory.
Pinned Memory Allocation: Pre-allocate and pin at least 70% of host DRAM using posix_memalign() and cudaHostRegister() during server boot. Avoid dynamic on-the-fly memory pinning, which introduces blocking kernel syscalls.
Transient Node Threshold Tuning: Configure the transient node deferral threshold between 64 and 128 tokens. Thresholds below 64 tokens cause excessive deferrals on coincidental prompt overlaps, while thresholds above 256 tokens permit redundant prefills to escape deferral.
SM Quota Assignment: Keep the GPU-assisted I/O quota locked to 2 blocks (2 SMs) for inbound prefill loads. Increasing the quota to 4 SMs yields marginal bandwidth improvements (<8%) while noticeably degrading matrix multiplication throughput on the remaining compute SMs.

By resolving the fundamental physical trade-off between memory granularity and transfer bandwidth, Strata establishes the definitive systems foundation for enterprise long-context artificial intelligence infrastructure.