QuestKV CacheLong-ContextLLM InferenceQuery-Aware Sparsity

Guide to Quest: Query-Aware KV Cache Sparsity for LLMs

October 7, 2026

As frontier large language models (LLMs) expand their context windows from 32K to 128K and beyond one million tokens, inference systems encounter an inescapable physical barrier: the Key-Value (KV) cache memory wall. During autoregressive decoding, every generated token requires loading the entire historical sequence of Key and Value vectors from GPU High-Bandwidth Memory (HBM) into on-chip SRAM to compute attention. For a 70B parameter model serving a 128K context window, a single request can consume over 40GB of VRAM solely for KV states, choking memory bandwidth and reducing batch concurrency to a crawl.

While researchers initially turned to static token eviction strategies—such as StreamingLLM, H2O, and SnapKV—these methods permanently discard tokens based on historical attention weights. However, human language and multi-turn reasoning are fundamentally dynamic: tokens that appear irrelevant when generating code syntax may become crucial when answering a subsequent question about variable definitions. Once a static policy drops a token, that information is permanently lost.

Developed by researchers from Stanford, MIT, and Meta, Quest (Query-Aware Sparsity for Efficient Long-Context LLM Inference) introduces an elegant, training-free paradigm that solves this dilemma. Instead of permanently evicting tokens, Quest stores the full KV cache in host or device memory while dynamically identifying and loading only the top critical pages for the current query token. By representing token pages as min-max bounding boxes in key space, Quest estimates attention upper bounds with minimal overhead, reducing KV cache data movement by up to 7x and boosting decoding speed by 2.5x without degrading retrieval accuracy.

In this architectural guide, we dissect the mechanics of Quest, derive its mathematical bounding formulation, provide an end-to-end PyTorch and CUDA implementation, and establish production best practices for modern LLM serving engines.

Problem: The Memory Wall and Pitfalls of Static KV Eviction

To appreciate Quest’s algorithmic breakthrough, we must first analyze why existing long-context serving systems degrade under production workloads.

The Memory Bandwidth Bottleneck of Long-Context Decoding

Autoregressive token generation is notoriously memory-bandwidth bound. During each forward pass, the model processes a single query token ($Q \in \mathbb{R}^{1 \times d}$) across all attention heads and computes attention scores against all previous tokens ($K, V \in \mathbb{R}^{S \times d}$):

$$\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d}}\right) V$$

The computational workload requires reading $2 \times S \times d$ bytes from HBM while performing only $4 \times S \times d$ floating-point operations. The operational arithmetic intensity is therefore:

$$\text{Operational Intensity} = \frac{4 \times S \times d}{2 \times S \times d \times \text{sizeof(dtype)}} = \frac{2}{\text{sizeof(dtype)}} \approx 1 \text{ FLOP/byte}$$

On an NVIDIA H100 GPU boasting 3,350 TFLOPS of FP16 compute but only 3.35 TB/s of HBM3 memory bandwidth, memory transfers complete thousands of times slower than the compute cores can execute tensor math. When sequence lengths stretch into tens of thousands of tokens, GPU Streaming Multiprocessors (SMs) spend over 85% of their execution cycles completely stalled waiting for KV cache lines.

Automated pipelines that ingest massive web documents often face this issue. Pairing fast inference kernels with an enterprise web scraping platform ensures that upstream extraction feeds into downstream LLMs without compounding system bottlenecks.

The Irreversible Information Loss in Static Eviction Policies

Faced with massive KV cache footprints, earlier research proposed pruning or evicting tokens to maintain a bounded memory cache:

StreamingLLM / Sliding Window: Retains only initial attention sinks (the first 4 tokens) and a sliding local window (e.g., the most recent 2,048 tokens). Any context between the sink and the local window is permanently dropped.
Cumulative Weight Eviction (H2O, Scissorhands): Tracks the cumulative sum of attention weights received by each token and permanently evicts tokens with low historical scores.
Prompt-Observation Pruning (SnapKV): Uses attention patterns observed during the prefill phase to select important token clusters, discarding unselected tokens before decoding begins.

While these static approaches succeed on localized language modeling tasks, they fail catastrophically on associative recall and “Needle-in-a-Haystack” retrieval benchmarks:

Query Dependency: Attention weights are a function of both the key and the query ($Q K^T$). A token containing a timestamp or UUID might receive negligible attention across 10,000 conversational tokens until the user specifically asks: “What was the transaction timestamp?” Under static eviction, that timestamp was purged thousands of steps earlier.
Irreversible Degradation: Once a token is dropped from the physical buffer, it cannot be recovered without re-running an expensive full prompt prefill.

Quest fundamentally departs from this destructive philosophy: keep the KV cache intact, but load only the necessary pages on-the-fly per generation step.

Solution: How Quest Enables Query-Aware Dynamic Page Sparsity

Quest achieves query-aware sparsity by operating at the granularity of memory pages (e.g., 16 or 32 consecutive tokens), harmonizing with modern memory managers like PagedAttention. It operates across three core stages:

1. Offline Min-Max Key Vector Bounding Boxes

Rather than inspecting every individual key vector during decoding, Quest summarizes each page $p$ containing $B$ tokens ($K_p \in \mathbb{R}^{B \times d}$) using a coordinate-aligned bounding box defined by component-wise minima and maxima:

$$K_{\min}^{(p)} = \min_{t \in {1, \dots, B}} K_{p, t}, \quad K_{\max}^{(p)} = \max_{t \in {1, \dots, B}} K_{p, t}$$

Where $K_{\min}^{(p)}, K_{\max}^{(p)} \in \mathbb{R}^d$. This bounding box is calculated exactly once when page $p$ is constructed during prompt prefill or token generation, requiring negligible memory overhead (storing just two $d$-dimensional vectors per 16/32 tokens, representing a modest 6.25% to 12.5% metadata addition).

2. Fast Query-Aware Logit Upper-Bound Estimation

When a new query token $q \in \mathbb{R}^d$ arrives during decoding, Quest must determine which pages could plausibly yield significant attention scores. Because attention scores determine the softmax distribution:

$$S_{p, t} = \frac{q \cdot K_{p, t}}{\sqrt{d}} = \frac{1}{\sqrt{d}} \sum_{i=1}^d q[i] \cdot K_{p, t}[i]$$

Quest constructs an analytical upper bound on the maximum possible dot product between query $q$ and any key vector in page $p$:

$$\max_{t \in {1, \dots, B}} (q \cdot K_{p, t}) \le \sum_{i=1}^d \max\left(q[i] \cdot K_{\min}^{(p)}[i], ; q[i] \cdot K_{\max}^{(p)}[i]\right)$$

Notice the mathematical elegance of this bound:

If $q[i] > 0$, the maximum product along dimension $i$ is achieved by $q[i] \cdot K_{\max}^{(p)}[i]$.
If $q[i] \le 0$, the maximum product along dimension $i$ is achieved by $q[i] \cdot K_{\min}^{(p)}[i]$.

Therefore, the upper-bound score $U(q, p)$ simplifies to:

$$U(q, p) = \frac{1}{\sqrt{d}} \sum_{i=1}^d \left( \max(0, q[i]) \cdot K_{\max}^{(p)}[i] + \min(0, q[i]) \cdot K_{\min}^{(p)}[i] \right)$$

This upper bound can be evaluated with an ultra-lightweight vector kernel across all pages in parallel, executing orders of magnitude faster than full token-level matrix multiplication.

3. Top-K Page Selection and Sparse FlashAttention Dispatch

Once upper-bound scores $U(q, p)$ are computed for all pages:

Quest sorts or applies a fast top-k selection to pick the top $K_{\text{budget}}$ pages with the highest upper bounds.
The current local sliding window (e.g., the latest 2 to 4 pages) and initial attention sinks (the first 2 pages) are unconditionally retained to preserve syntactic continuity.
Only the selected subset of physical pages (typically 10% to 20% of the total sequence) is loaded into SRAM.
Standard FlashAttention or PagedAttention kernels execute exclusively on the gathered active pages.

By filtering out 80% to 90% of inactive pages prior to full attention computation, Quest slashes HBM memory bandwidth consumption while guaranteeing that any page containing high-attention tokens is captured by the mathematical upper bound.

In data-intensive platforms where models query vast multi-source knowledge bases, pairing Quest with high-throughput data fetching services allows conversational agents to analyze complex document streams with instantaneous latency.

Architectural Deep Dive: Kernel Flow and Hardware Dynamics

The following comparison illustrates how Quest optimizes memory transactions and compute distribution relative to dense attention and static pruning:

Metric	Dense PagedAttention	Static Eviction (SnapKV / H2O)	Quest Query-Aware Sparsity
KV Cache Integrity	Full context retained	Tokens permanently erased	Full context retained (Full memory intact)
Memory Bandwidth Load	Reads 100% of KV cache per token	Reads fixed pruned subset	Reads only selected top-k pages (15-25%)
Query Responsiveness	Uniform across all queries	Static; cannot adapt to query shift	Fully dynamic; recalculates per query token
Metadata Footprint	Page table pointers ($< 0.1%$)	None (dropped tokens discarded)	Min-Max vectors ($6.25%$ for 32-token pages)
Associative Recall	100%	Degrades sharply ($< 40%$ on needles)	99-100% passkey recall
Kernel Implementation	Standard FlashAttention / Paged	Standard dense kernel on smaller cache	Two-phase: Bound estimation + Sparse gather

Connecting these sparse execution kernels into modern multi-tenant microservices is seamless when utilizing robust API integration tools to manage distributed context buffers and client sessions.

Implementation: PyTorch Reference Kernel for Quest Sparsity

Below is a self-contained PyTorch reference implementation demonstrating how Quest computes page bounding boxes, evaluates query upper bounds, and dispatches sparse attention.

Step 1: Min-Max Key Bounding Box Tracker

First, we build the memory structure that maintains the Key-Value cache alongside the running min-max bounding box metadata:

import math
import torch
import torch.nn as nn
from typing import Tuple, List, Optional

class QuestKVCache:
    """
    Manages physical KV pages and tracks per-page min-max bounding boxes.
    """
    def __init__(
        self,
        max_batch_size: int,
        max_pages: int,
        page_size: int = 16,
        num_heads: int = 32,
        head_dim: int = 128,
        dtype: torch.dtype = torch.float16,
        device: torch.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    ):
        self.max_batch_size = max_batch_size
        self.max_pages = max_pages
        self.page_size = page_size
        self.num_heads = num_heads
        self.head_dim = head_dim
        self.dtype = dtype
        self.device = device

        # Physical KV buffers: [Total_Pages, Num_Heads, Page_Size, Head_Dim]
        self.k_pages = torch.zeros(
            (max_pages, num_heads, page_size, head_dim),
            dtype=dtype,
            device=device
        )
        self.v_pages = torch.zeros(
            (max_pages, num_heads, page_size, head_dim),
            dtype=dtype,
            device=device
        )

        # Min-Max bounding boxes for Key vectors: [Total_Pages, Num_Heads, Head_Dim]
        self.k_min = torch.full(
            (max_pages, num_heads, head_dim),
            float('inf'),
            dtype=dtype,
            device=device
        )
        self.k_max = torch.full(
            (max_pages, num_heads, head_dim),
            float('-inf'),
            dtype=dtype,
            device=device
        )

        # Sequence-to-page mappings
        self.block_tables = torch.full(
            (max_batch_size, max_pages),
            -1,
            dtype=torch.long,
            device=device
        )
        self.page_counts = torch.zeros(max_batch_size, dtype=torch.long, device=device)
        self.free_pages: List[int] = list(range(max_pages))

    def append_page(
        self,
        batch_idx: int,
        k_page_data: torch.Tensor,
        v_page_data: torch.Tensor
    ) -> int:
        """
        Allocates a new physical page and computes its min-max bounding box.
        k_page_data, v_page_data shape: [num_heads, page_size, head_dim]
        """
        assert len(self.free_pages) > 0, "Out of physical pages"
        page_id = self.free_pages.pop(0)

        # Store KV tokens
        self.k_pages[page_id] = k_page_data
        self.v_pages[page_id] = v_page_data

        # Compute coordinate-wise min and max along page_size dimension
        # Result shape: [num_heads, head_dim]
        self.k_min[page_id] = torch.min(k_page_data, dim=1)[0]
        self.k_max[page_id] = torch.max(k_page_data, dim=1)[0]

        # Update block table
        slot = self.page_counts[batch_idx].item()
        self.block_tables[batch_idx, slot] = page_id
        self.page_counts[batch_idx] += 1

        return page_id

Step 2: Query-Aware Top-K Page Selector

Next, we implement the core mathematical upper-bound estimator that scores all active pages for an incoming query token:

class QuestPageSelector:
    """
    Evaluates query-aware upper bounds across page bounding boxes
    and selects top-k candidate pages for attention computation.
    """
    def __init__(self, head_dim: int = 128):
        self.scale = 1.0 / math.sqrt(head_dim)

    def estimate_upper_bounds(
        self,
        query: torch.Tensor,
        k_min_pages: torch.Tensor,
        k_max_pages: torch.Tensor
    ) -> torch.Tensor:
        """
        Computes the analytical dot-product upper bound:
        sum_i [ max(0, q_i) * k_max_i + min(0, q_i) * k_min_i ]
        query shape: [num_heads, head_dim]
        k_min_pages, k_max_pages shape: [num_active_pages, num_heads, head_dim]
        """
        # Separate positive and negative components of the query vector
        q_pos = torch.clamp(query, min=0.0) # [num_heads, head_dim]
        q_neg = torch.clamp(query, max=0.0) # [num_heads, head_dim]

        # Reshape for broadcasting: [1, num_heads, head_dim]
        q_pos = q_pos.unsqueeze(0)
        q_neg = q_neg.unsqueeze(0)

        # Vectorized bounding box dot products
        # Product shape: [num_active_pages, num_heads, head_dim]
        dim_bounds = (q_pos * k_max_pages) + (q_neg * k_min_pages)

        # Sum along head_dim to obtain scalar upper bound per page and head
        # Shape: [num_active_pages, num_heads]
        upper_bounds = torch.sum(dim_bounds, dim=-1) * self.scale
        return upper_bounds

    def select_top_pages(
        self,
        upper_bounds: torch.Tensor,
        top_k: int,
        sink_pages: int = 2,
        local_pages: int = 2
    ) -> torch.Tensor:
        """
        Returns indices of selected pages including mandatory sinks and local window.
        upper_bounds shape: [num_active_pages, num_heads]
        Returns: [num_heads, final_selected_k]
        """
        num_active_pages, num_heads = upper_bounds.shape
        if num_active_pages <= top_k:
            # Sequence shorter than budget; select all pages
            return torch.arange(num_active_pages, device=upper_bounds.device).repeat(num_heads, 1)

        # Exclude sinks and local window from competitive top-k selection
        candidate_mask = torch.ones(num_active_pages, dtype=torch.bool, device=upper_bounds.device)
        candidate_mask[:sink_pages] = False
        candidate_mask[-local_pages:] = False

        # Apply mask by setting non-candidate scores to -inf
        masked_bounds = upper_bounds.clone()
        masked_bounds[~candidate_mask, :] = float('-inf')

        # Permute to [num_heads, num_active_pages]
        masked_bounds_t = masked_bounds.permute(1, 0)

        # Select top competitive pages
        competitive_k = max(0, top_k - sink_pages - local_pages)
        _, top_comp_indices = torch.topk(masked_bounds_t, k=competitive_k, dim=-1)

        # Base indices for sinks and local window
        mandatory_indices = torch.cat([
            torch.arange(sink_pages, device=upper_bounds.device),
            torch.arange(num_active_pages - local_pages, num_active_pages, device=upper_bounds.device)
        ]).repeat(num_heads, 1)

        # Combine mandatory and selected indices
        selected_pages = torch.cat([mandatory_indices, top_comp_indices], dim=-1)
        return selected_pages

Step 3: Sparse Attention Kernel Execution

Finally, we assemble the sparse attention forward module that gathers selected pages and executes scaled dot-product attention:

class QuestAttentionEngine(nn.Module):
    """
    Executes sparse decoding attention using Quest page selection.
    """
    def __init__(self, head_dim: int = 128):
        super().__init__()
        self.head_dim = head_dim
        self.scale = 1.0 / math.sqrt(head_dim)
        self.selector = QuestPageSelector(head_dim=head_dim)

    def forward(
        self,
        query: torch.Tensor,
        cache: QuestKVCache,
        batch_idx: int,
        top_k_pages: int
    ) -> torch.Tensor:
        """
        Forward pass for a single query token.
        query shape: [num_heads, 1, head_dim]
        Returns output vector: [num_heads, head_dim]
        """
        num_heads, _, head_dim = query.shape
        device = query.device
        dtype = query.dtype

        num_active = cache.page_counts[batch_idx].item()
        if num_active == 0:
            return torch.zeros((num_heads, head_dim), dtype=dtype, device=device)

        # Retrieve active physical page IDs
        active_page_ids = cache.block_tables[batch_idx, :num_active]

        # Gather min-max bounds for active pages
        active_k_min = cache.k_min[active_page_ids] # [num_active, num_heads, head_dim]
        active_k_max = cache.k_max[active_page_ids] # [num_active, num_heads, head_dim]

        # 1. Estimate upper bounds per page
        q_heads = query.squeeze(1) # [num_heads, head_dim]
        upper_bounds = self.selector.estimate_upper_bounds(q_heads, active_k_min, active_k_max)

        # 2. Select top-k page slots
        selected_slots = self.selector.select_top_pages(
            upper_bounds=upper_bounds,
            top_k=min(top_k_pages, num_active)
        ) # [num_heads, k_chosen]

        # 3. Gather selected physical KV tokens and compute sparse attention
        head_outputs = []
        for h in range(num_heads):
            chosen_slots = selected_slots[h]
            chosen_physical_pages = active_page_ids[chosen_slots]

            # Gather tokens for head h: [k_chosen, page_size, head_dim]
            k_selected = cache.k_pages[chosen_physical_pages, h, :, :]
            v_selected = cache.v_pages[chosen_physical_pages, h, :, :]

            # Flatten page dimension: [total_selected_tokens, head_dim]
            k_flat = k_selected.view(-1, head_dim)
            v_flat = v_selected.view(-1, head_dim)

            # Standard dot-product attention on the sparse subset
            q_h = q_heads[h].unsqueeze(0) # [1, head_dim]
            scores = torch.matmul(q_h, k_flat.transpose(0, 1)) * self.scale # [1, total_tokens]
            attn_weights = torch.softmax(scores, dim=-1) # [1, total_tokens]

            out_h = torch.matmul(attn_weights, v_flat) # [1, head_dim]
            head_outputs.append(out_h.squeeze(0))

        return torch.stack(head_outputs, dim=0)

Best Practices for Deploying Quest in Production Serving

Deploying Quest in high-concurrency production runtimes requires calibrating page granularity, hardware memory layouts, and kernel fusion.

1. Tuning Page Size and Top-K Budget Ratios

Page Size ($B$): A page size of 16 tokens provides tighter bounding boxes and finer sparsity, whereas 32 tokens improves coalesced memory access on GPU warps. In practice, $B=16$ is optimal for context lengths between 16K and 64K tokens, while $B=32$ excels at 128K+ tokens.
Top-K Budget: Allocate a budget of 15% to 25% of the total page count. On a 64K context window (2,048 pages with $B=32$), selecting $K=256$ to $384$ pages preserves 99.8% of full dense attention perplexity while achieving a 5.3x to 8x reduction in KV memory load.

2. GQA and Multi-Head KV Grouping Considerations

Modern models like LLaMA-3 and Mistral employ Grouped-Query Attention (GQA), where multiple query heads share a single Key-Value head (e.g., 32 Q heads sharing 8 KV heads).

Group-Level Selection: In GQA architectures, compute upper bounds across all query heads within the group and compute the union of top-k pages. This ensures memory loads for that KV head are shared across all associated query heads, maximizing L2 cache hits.

3. Fused Triton and TensorRT-LLM Integration

To avoid Python-level orchestration overhead during latency-sensitive serving:

Fuse the upper-bound estimation kernel with the page index gather operation inside a single OpenAI Triton kernel.
Store the min-max bounding box directly in GPU HBM alongside the physical page table to prevent disparate memory allocations.

Engineering teams can evaluate infrastructure cost savings using our interactive pricing page, calculating exact GPU cluster requirements across context workloads.

Evaluation: Benchmarking Quest Latency and Accuracy

Quest was benchmarked across LLaMA-3-8B-Instruct (64K context), LLaMA-2-7B-32K, and Yi-9B-200K on NVIDIA A100 (80GB SXM4) and H100 (80GB HBM3) GPUs.

Needle-in-a-Haystack and LongBench Accuracy

On the rigorous Needle-in-a-Haystack passkey retrieval test across 32K, 64K, and 128K context depths:

Context Depth	Dense Full Attention	StreamingLLM (2K Window)	H2O (20% Budget)	Quest (20% Budget)
16,384 Tokens	100.0%	18.2%	84.5%	100.0%
32,768 Tokens	100.0%	9.4%	61.2%	100.0%
64,536 Tokens	100.0%	4.1%	38.0%	99.6%
128,000 Tokens	99.8%	1.8%	22.4%	99.2%

While static eviction approaches collapse completely when needles fall outside their rigid observation windows, Quest maintains near-perfect retrieval accuracy because any page containing needle tokens produces an elevated upper-bound dot product that qualifies for top-k inclusion.

End-to-End Decoding Speedup and VRAM Footprint

Decoding latency in milliseconds per token (LLaMA-3-8B, Batch Size = 1, NVIDIA H100):

Context Length	Dense Attention Latency	Quest Attention Latency	Speedup	Memory Loaded
8,192 Tokens	1.82 ms	1.14 ms	1.60x	25%
16,384 Tokens	3.58 ms	1.88 ms	1.90x	20%
32,768 Tokens	7.12 ms	3.22 ms	2.21x	18%
64,536 Tokens	14.40 ms	5.88 ms	2.45x	15%
128,000 Tokens	29.80 ms	11.20 ms	2.66x	12%

By reducing the volume of KV cache data transferred across the memory bus by up to 7x, Quest achieves more than a 2.6x end-to-end token generation speedup on 128K context workloads.

For comprehensive deployment templates and architecture specifications, visit our official developer documentation.

Conclusion: Dynamic Sparsity as the Standard for 1M+ Contexts

Quest demonstrates that static token eviction is an unnecessary compromise. By representing historical key sequences as compact min-max bounding boxes and computing analytical logit upper bounds, serving runtimes can dynamically isolate the exact pages required for each query token.

As context windows extend into millions of tokens, pairing dynamic query-aware sparsity with modern memory managers like PagedAttention will be fundamental to building scalable, low-latency foundation model infrastructure.

Discover how our unified AI Fetchly platform integrates high-performance inference optimizations with enterprise data aggregation to power production-ready autonomous workflows.