qcfuseragcache-fusionllm-servingkv-cachesglangtutorial

QCFuse Guide: Query-Aware Cache Fusion for RAG Serving

October 11, 2026

QCFuse Architecture: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving

The Cache-Fusion Dilemma: Quality Loss vs. Pipeline Stalls in RAG Serving

In enterprise search engines, automated technical support copilots, and real-time knowledge discovery pipelines powered by aiFetchly, Retrieval-Augmented Generation (RAG) is the foundational architecture for grounding Large Language Models (LLMs) in external facts. By retrieving external document passages from domain corpora and injecting them into the context window, RAG delivers factual precision across complex enterprise workflows.

However, the computational efficiency of RAG systems is severely limited by a dominant operational bottleneck: the prefill stage runtime.

In data-intensive RAG services, user queries are prepended with multiple retrieved evidence chunks, causing prompt lengths to scale to 4,000–32,000+ tokens. Computing the Key-Value (KV) cache for these injected passages consumes over 70% of total request response time. To alleviate this overhead, the machine learning systems and database communities have coalesced around KV Cache Fusion: precomputing position-independent Key-Value caches (PIC) for reusable corpus documents and selectively fusing them into the active prompt at serving time.

The Selector Trade-off in Layer-Wise Cache Fusion:
┌────────────────────────────────────────────────────────────────────────┐
│ Incoming RAG Query: Multi-hop reasoning across 8 retrieved chunks       │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
               ┌────────────────────┴────────────────────┐
               ▼                                         ▼
   Lightweight Selectors (CacheBlend/FusionRAG)    Full-View Selectors (ProphetKV)
 ❌ Query-agnostic or final-layer-only signal      ❌ Evaluates user query over full context
 ❌ Misses subtle multi-hop evidence tokens        ❌ Scans all L transformer layers
 ❌ Degrades output quality on complex QA          ❌ Stalls layer-wise cache-loading pipeline!
 ❌ Fast, but inaccurate                           ❌ Accurate, but pipeline-blocking

Deconstructing the Quality vs. Efficiency Bottleneck in Token Selection

When fusing precomputed chunk caches into an active request, the model cannot simply reuse stale KV tensors as-is: because causal attention and Rotary Position Embeddings (RoPE) depend on token order and prompt context, a subset of tokens $\mathcal{P}$ must undergo selective recomputation to restore exact cross-attention alignment.

The fundamental challenge in cache fusion lies in how the system selects $\mathcal{P}$:

1. The Inadequacy of Query-Agnostic and Final-Layer Selectors
Early cache-fusion systems (such as CacheBlend and EPIC) rely on query-agnostic selection: they choose which tokens to recompute based purely on static token positions (e.g., chunk boundaries) or attention sinks. Later systems like FusionRAG introduce query signals, but restrict them to final-layer-only attention.
While fast, these methods suffer from severe semantic blind spots: on multi-hop question answering and complex entity queries, critical evidence tokens situated in early or middle transformer layers receive low final-layer attention mass. As a result, essential evidence is omitted from recomputation, causing the model to hallucinate or fail multi-hop reasoning.

2. The Pipeline-Blocking Wall of Full-View Selectors
To restore 100% full-prefill quality, advanced selectors like ProphetKV examine full-view query-aware signals: they execute an initial forward pass of the query conditioned on the entire stitched context across all $L$ transformer layers.
While ProphetKV identifies answer-bearing evidence with high precision, it creates a fatal systems deadlock: it destroys the layer-wise cache-fusion pipeline.
In modern inference engines (such as SGLang), high throughput depends on overlapping the asynchronous loading of layer $l+1$‘s KV cache from host memory over PCIe while the GPU computes layer $l$. Because a full-view selector must inspect all layers across all chunks before determining the recomputation mask $\mathcal{P}$, sparse recomputation cannot begin until the full-view scan completes, stalling the GPU and negating prefill acceleration.

To dissolve this trade-off, researchers from Tongji University, Ant Group, and the Hong Kong University of Science and Technology introduced QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving (arXiv:2606.05875). QCFuse constructs two compact evidence views—chunk-anchor query probing and critical-layer profiling—forming the recomputation mask early enough to enable fully unblocked, pipelined cache fusion.

Architectural Deep Dive: How QCFuse Operates

QCFuse functions as an intelligent compressed-view selector and pipelined execution runtime integrated directly into the SGLang serving engine.

┌─────────────────────────────────────────────────────────────────────────────┐
│                         QCFuse Serving Architecture                         │
│                                                                             │
│   [ Phase I: Offline Cache Preparation ]                                    │
│   • Precomputes per-layer PIC KV caches for reusable corpus chunks          │
│   • Identifies 10% representative Chunk Anchors (via KVzip probing)         │
│   • Profiles model-specific Critical Layers (Top-3 middle layers)           │
│               │                                                             │
│               ▼                                                             │
│   [ Phase II: Online Query-Aware Cache Fusion ]                             │
│   User Query U + Retrieved Chunks [c_1, c_2, ... c_m]                       │
│               │                                                             │
│               ▼                                                             │
│   ┌─────────────────────────────────────────────────────────────────────┐   │
│   │ Step 1: Chunk-Anchor Query Probing (Token-View Compression)         │   │
│   │ Probes query U strictly over 10% anchor tokens (C_A) in <1ms        │   │
│   └───────────────────┬─────────────────────────────────────────────────┘   │
│                       │                                                     │
│                       ▼                                                     │
│   ┌─────────────────────────────────────────────────────────────────────┐   │
│   │ Step 2: Critical-Layer Token Localization (Layer-View Compression)  │   │
│   │ Evaluates query-to-context attention across profiled layers L*      │   │
│   │ Forms recomputation set P without inspecting all L layers           │   │
│   └───────────────────┬─────────────────────────────────────────────────┘   │
│                       │                                                     │
│                       ▼ (Mask P Formed Early)                               │
│   ┌─────────────────────────────────────────────────────────────────────┐   │
│   │ Step 3: Layer-Wise Pipelined Cache Fusion (Triton Sparse Kernel)    │   │
│   │ Layer l Recomputes P  ◄── OVERLAPPED WITH ──► Layer l+1 KV Loading  │   │
│   └─────────────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────┘

The system operates across three core architectural components:

1. Chunk-Anchor Query Probing (Token-View Compression)

Rather than forcing the query to attend over tens of thousands of raw context tokens during the online selection phase, QCFuse compresses the token dimension offline:

Per-Chunk Anchor Selection: For each reusable text chunk in the corpus, QCFuse identifies a compact subset of anchor tokens ($r_a = 0.1$, or 10% of the chunk) that best summarize its semantic contents. Empirical evaluation demonstrates that using self-supervised reconstruction probes (KVzip@10%) achieves 98% of the evidence recall of a full-context scan while reducing probing cache size by 90%.
Anchor-Conditioned Query Probing: When a request arrives, QCFuse concatenates only the compact anchors of the retrieved chunks ($C_{\mathcal{A}}$) with the user query $U$. Forwarding this tiny compressed view produces query hidden states that are deeply conditioned on the retrieved knowledge, capturing multi-hop relationships without paying the latency cost of full-context attention.

2. Critical-Layer Token Localization (Layer-View Compression)

To prevent the selector from stalling the execution pipeline by scanning all 32 or 80 transformer layers, QCFuse compresses the layer dimension:

The Middle-Layer Localization Phenomenon: In transformer architectures, attention weights in the initial layers are dominated by syntactic parsing, while final layers suffer from attention-sink distortions. Extensive profiling reveals that evidence-to-query localization peaks decisively in middle layers.
Model-Specific Layer Profiling: During an offline calibration phase, QCFuse profiles the model to isolate a candidate subset of top-3 critical layers ($\mathcal{L}^*$) whose cross-attention signals best identify answer-bearing tokens.
Immediate Mask Formation: During online serving, QCFuse evaluates attention scores only within $\mathcal{L}^*$. This allows the system to determine the exact sparse recomputation set $\mathcal{P}$ immediately, before layer-wise execution begins.

3. Pipelined Layer-Wise Fusion with Custom Triton Kernels

With $\mathcal{P}$ generated upfront in sub-millisecond time, QCFuse executes an unblocked layer-wise pipelined cache fusion:

For each layer $l$, the GPU executes a custom Triton selective recomputation kernel, computing fresh Query, Key, and Value states only for token positions in $\mathcal{P}$.
Simultaneously, on an asynchronous CUDA DMA stream, the system loads the precomputed KV cache blocks for layer $l+1$ across PCIe into GPU High-Bandwidth Memory (HBM).
Memory transfer latency is 100% hidden behind sparse kernel computation, delivering a 1.7x prefill speedup over full prefill.

Architectural Comparison: QCFuse vs SOTA Cache-Fusion Systems

Architectural Dimension	Full Prefill (Vanilla SGLang)	CacheBlend (EuroSys ‘25)	FusionRAG (SIGMOD ‘26)	ProphetKV (VLDB ‘26)	QCFuse (arXiv:2606.05875)
Token View	Full Context ($N$ tokens)	Query-Agnostic (Static)	Full Context	Full Context ($N$ tokens)	Chunk Anchors (0.1N tokens)
Layer View	All $L$ Layers	Query-Agnostic	Final Layer Only ($L=1$)	All $L$ Layers	*Top-3 Critical Layers ($\mathcal{L}^$)**
Pipeline Readiness	None	Immediate	Immediate	Blocked (Waits for all $L$)	Immediate (Pipelined Overlap)
Multi-Hop Evidence Recall	100% Reference	Low (~55%)	Moderate (~68%)	High (~94%)	Near-Perfect (96.5%)
Prefill Speedup	Baseline (1.0x)	2.6x (Quality drops)	1.9x (Quality drops)	1.1x (Pipeline stalls)	1.7x over Full, 1.5x over ProphetKV
Output Response Quality	100% Baseline	82.4% ROUGE	87.1% ROUGE	98.2% ROUGE	100% Full-Prefill Level Quality
Kernel Implementation	Dense FlashAttention	Sparse Scatter	Standard Kernels	Fragmented Scatter	Custom Triton Pipelined Path

Mathematical Formulation of Compressed-View Token Selection

Let us formalize QCFuse’s compressed-view optimization objective.

The Full-View Evidence Selection Objective

Let $C = [c_1; c_2; \dots; c_m]$ be the retrieved context comprising $m$ chunks with total length $N = \sum_{i=1}^m |c_i|$ tokens, and let $U$ be the user query.

In an ideal full-view selector, the attention score assigned by query token $q \in U$ to context token $t \in C$ at layer $l$ is:

$$\alpha_{q, t}^{(l)} = \frac{\exp\left( \frac{\mathbf{q}q^{(l)} \left(\mathbf{k}t^{(l)}\right)^T}{\sqrt{d{\text{head}}}} \right)}{\sum{j \in C \cup U} \exp\left( \frac{\mathbf{q}_q^{(l)} \left(\mathbf{k}j^{(l)}\right)^T}{\sqrt{d{\text{head}}}} \right)}$$

The full-view importance score of token $t$ across all layers $\mathcal{L} = {1, \dots, L}$ is:

$$S_{\text{full}}(t) = \frac{1}{|\mathcal{L}| \cdot |U|} \sum_{l \in \mathcal{L}} \sum_{q \in U} \alpha_{q, t}^{(l)}$$

Given a recomputation budget ratio $\rho \in (0, 1)$, the ideal recomputation set is the top-$\lfloor \rho N \rfloor$ tokens under $S_{\text{full}}$. However, computing $S_{\text{full}}$ requires loading $\mathbf{k}_t^{(l)}$ for all $t \in C$ across all $l \in \mathcal{L}$.

The QCFuse Compressed-View Approximation

QCFuse replaces the full context $C$ with the compact anchor view $C_{\mathcal{A}} = \bigcup_{i=1}^m \mathcal{A}(c_i)$, where $|\mathcal{A}(c_i)| = \lceil r_a \cdot |c_i| \rceil$ (default $r_a = 0.1$).

Forwarding query $U$ conditioned on $C_{\mathcal{A}}$ yields anchor-conditioned query states $\tilde{\mathbf{q}}_q^{(l)}$. The compressed importance score is computed strictly over the small critical-layer subset $\mathcal{L}^* \subset \mathcal{L}$ ($|\mathcal{L}^*| = 3$):

$$S_{\text{QCFuse}}(t) = \frac{1}{|\mathcal{L}^| \cdot |U|} \sum_{l \in \mathcal{L}^} \sum_{q \in U} \frac{\exp\left( \frac{\tilde{\mathbf{q}}q^{(l)} \left(\mathbf{k}t^{(l)}\right)^T}{\sqrt{d{\text{head}}}} \right)}{\sum{j \in C_{\mathcal{A}} \cup U} \exp\left( \frac{\tilde{\mathbf{q}}_q^{(l)} \left(\mathbf{k}j^{(l)}\right)^T}{\sqrt{d{\text{head}}}} \right)}$$

The recomputation token set $\mathcal{P}$ is selected as:

$$\mathcal{P} = \text{TopK}\left( S_{\text{QCFuse}}, ; \lfloor \rho N \rfloor \right)$$

Because $|\mathcal{L}^*| \ll L$ and $|C_{\mathcal{A}}| \ll N$, forming $\mathcal{P}$ requires 90% fewer memory reads and <5% of the FLOPs, completing in sub-millisecond time.

Step-by-Step Implementation: Building a QCFuse Serving Engine

Below is a complete, self-contained Python reference implementation demonstrating QCFuse’s core mechanics: the Chunk Anchor Selector, the Critical-Layer Profiler, and the Pipelined Cache Fusion Simulator.

1. Data Models and System Definitions

"""
qcfuse_serving_engine.py
Reference Implementation of QCFuse: Compressed-View Query-Aware Cache Fusion.
arXiv:2606.05875 Reference Architecture for SGLang.
"""

from dataclasses import dataclass, field
from typing import List, Dict, Optional, Tuple, Set
import numpy as np
import time
import math


@dataclass
class CorpusChunk:
    chunk_id: str
    text: str
    tokens: List[int]
    anchor_indices: List[int] = field(default_factory=list)

    @property
    def num_tokens(self) -> int:
        return len(self.tokens)


@dataclass
class RAGQueryRequest:
    query_id: str
    query_text: str
    query_tokens: List[int]
    retrieved_chunks: List[CorpusChunk]
    recompute_budget_ratio: float = 0.20  # Recompute top 20% tokens

2. Offline Profiling: Chunk Anchors and Critical Layers

class OfflineQCFuseProfiler:
    """
    Profiles chunk-local anchor tokens and model-specific critical layers.
    """
    def __init__(self, total_layers: int = 32, anchor_ratio: float = 0.10):
        self.total_layers = total_layers
        self.anchor_ratio = anchor_ratio

    def extract_chunk_anchors(self, chunk: CorpusChunk) -> List[int]:
        """
        Selects representative anchor tokens (KVzip style) capturing 10% of chunk.
        """
        num_anchors = max(1, int(chunk.num_tokens * self.anchor_ratio))
        # Simulated anchor selection: boundary tokens + high-norm internal tokens
        anchor_indices = set([0, chunk.num_tokens - 1]) # Boundary anchors
        step = max(1, chunk.num_tokens // num_anchors)
        for idx in range(0, chunk.num_tokens, step):
            if len(anchor_indices) < num_anchors:
                anchor_indices.add(idx)
        chunk.anchor_indices = sorted(list(anchor_indices))
        return chunk.anchor_indices

    def profile_critical_layers(self) -> List[int]:
        """
        Identifies top-3 middle layers where query-to-context attention peaks.
        For a 32-layer model (e.g. LLaMA-3-8B), middle layers [14, 16, 18] localize evidence best.
        """
        mid = self.total_layers // 2
        critical_layers = [mid - 2, mid, mid + 2]
        return critical_layers

3. Online Compressed-View Token Selector

class CompressedViewSelector:
    """
    Online selector executing chunk-anchor probing and critical-layer localization.
    """
    def __init__(self, critical_layers: List[int], d_model: int = 4096):
        self.critical_layers = critical_layers
        self.d_model = d_model

    def select_recompute_tokens(self, request: RAGQueryRequest) -> Set[Tuple[str, int]]:
        """
        Selects global token positions for sparse recomputation in <1ms.
        Returns set of (chunk_id, token_index) tuples.
        """
        total_context_tokens = sum(c.num_tokens for c in request.retrieved_chunks)
        budget = max(1, int(total_context_tokens * request.recompute_budget_ratio))

        # Step 1: Chunk-Anchor Query Probing
        # We probe the query strictly over anchor tokens
        anchor_tokens_count = sum(len(c.anchor_indices) for c in request.retrieved_chunks)

        # Step 2: Critical-Layer Token Scoring
        # Score tokens across the 3 critical layers
        token_scores = []
        for chunk in request.retrieved_chunks:
            for idx in range(chunk.num_tokens):
                # Simulated score: high for anchors and surrounding context tokens
                is_anchor = idx in chunk.anchor_indices
                base_score = 0.8 if is_anchor else 0.2
                # Distance to nearest anchor
                min_dist = min(abs(idx - a) for a in chunk.anchor_indices) if chunk.anchor_indices else 10
                locality_boost = 1.0 / (1.0 + min_dist * 0.2)
                score = base_score * locality_boost + np.random.uniform(0.0, 0.1)
                token_scores.append((score, chunk.chunk_id, idx))

        # Select top-K tokens
        token_scores.sort(key=lambda x: x[0], reverse=True)
        selected_positions = set((cid, idx) for _, cid, idx in token_scores[:budget])
        return selected_positions

4. Pipelined Layer-Wise Fusion Execution Simulation

class PipelinedCacheFusionEngine:
    """
    Simulates SGLang execution overlapping KV cache PCIe loading with sparse recomputation.
    """
    def __init__(self, total_layers: int = 32):
        self.total_layers = total_layers

    def execute_fused_prefill(
        self,
        request: RAGQueryRequest,
        recompute_mask: Set[Tuple[str, int]]
    ) -> Dict[str, float]:
        start_time = time.perf_counter()
        total_tokens = sum(c.num_tokens for c in request.retrieved_chunks)
        recompute_count = len(recompute_mask)

        # Pipeline simulation across 32 layers
        # Compute time is proportional to recomputed tokens
        compute_time_per_layer_ms = (recompute_count / total_tokens) * 1.2 + 0.1
        # PCIe DMA loading time for layer l+1
        dma_time_per_layer_ms = 0.35

        total_pipeline_time_ms = 0.0
        for l in range(self.total_layers):
            if l == 0:
                # First layer pays initial loading
                total_pipeline_time_ms += dma_time_per_layer_ms + compute_time_per_layer_ms
            else:
                # Subsequent layers overlap DMA with sparse computation!
                total_pipeline_time_ms += max(dma_time_per_layer_ms, compute_time_per_layer_ms)

        planning_time_ms = (time.perf_counter() - start_time) * 1000.0

        # Full prefill baseline comparison
        full_prefill_time_ms = self.total_layers * 1.5

        return {
            "total_context_tokens": total_tokens,
            "recomputed_tokens": recompute_count,
            "recompute_percentage": (recompute_count / total_tokens) * 100.0,
            "pipelined_prefill_latency_ms": total_pipeline_time_ms,
            "full_prefill_latency_ms": full_prefill_time_ms,
            "speedup_factor": full_prefill_time_ms / total_pipeline_time_ms,
            "selector_planning_ms": planning_time_ms
        }


if __name__ == "__main__":
    print("=== QCFuse Compressed-View Cache Fusion Benchmark ===")

    profiler = OfflineQCFuseProfiler(total_layers=32, anchor_ratio=0.10)
    critical_layers = profiler.profile_critical_layers()
    print(f"Profiled Model-Specific Critical Layers: {critical_layers}")

    # Prepare 4 corpus chunks (512 tokens each = 2048 context tokens)
    chunks = [
        CorpusChunk(f"chunk_{i:02d}", f"Enterprise documentation chunk {i}", [100 + i] * 512)
        for i in range(4)
    ]
    for c in chunks:
        profiler.extract_chunk_anchors(c)
        print(f"[{c.chunk_id}] Extracted {len(c.anchor_indices)} Anchor Tokens (10% View)")

    # Simulated incoming multi-hop RAG query
    query_req = RAGQueryRequest(
        query_id="query_audit_01",
        query_text="What are the compliance penalties for data residency breaches?",
        query_tokens=[10, 20, 30, 40],
        retrieved_chunks=chunks,
        recompute_budget_ratio=0.20 # Recompute top 20%
    )

    selector = CompressedViewSelector(critical_layers)
    recompute_mask = selector.select_recompute_tokens(query_req)
    print(f"\nFormed Recomputation Mask P: Selected {len(recompute_mask)} tokens.")

    fusion_engine = PipelinedCacheFusionEngine(total_layers=32)
    benchmark_res = fusion_engine.execute_fused_prefill(query_req, recompute_mask)

    print("\n--- Pipeline Execution Performance ---")
    print(f"Total Context Tokens:         {benchmark_res['total_context_tokens']}")
    print(f"Tokens Selectively Recomputed: {benchmark_res['recomputed_tokens']} ({benchmark_res['recompute_percentage']:.1f}%)")
    print(f"Full Prefill Latency:         {benchmark_res['full_prefill_latency_ms']:.2f} ms")
    print(f"QCFuse Pipelined Latency:     {benchmark_res['pipelined_prefill_latency_ms']:.2f} ms")
    print(f"Net Prefill Acceleration:     {benchmark_res['speedup_factor']:.2f}x Speedup!")

Production Deployment: Configuring QCFuse with SGLang and Triton

In production Kubernetes clusters, QCFuse deploys natively as an acceleration extension inside SGLang 0.5.x worker pods.

┌──────────────────────────────────────────────────────────────────────────────────┐
│                         SGLang Production Serving Pod                            │
│                                                                                  │
│   RAG Ingress Gateway (FastAPI / gRPC)                                           │
│          │                                                                       │
│          ▼                                                                       │
│   ┌──────────────────────────────────────────────────────────────────────────┐   │
│   │ QCFuse Compressed-View Selector (Sub-1ms Anchor Probing)                 │   │
│   └──────────────────────────────────┬───────────────────────────────────────┘   │
│                                      │                                           │
│                                      ▼ (Mask P Dispatched)                       │
│   ┌──────────────────────────────────────────────────────────────────────────┐   │
│   │ SGLang Layer-Wise Cache Fusion Engine                                    │   │
│   │ • Asynchronous PCIe Host-to-Device Memory Stream                         │   │
│   │ • Triton Fused Selective KV Recomputation Kernel                         │   │
│   │ • 1.7x Faster Prefill • 100% Full-Prefill Output Quality                 │   │
│   └──────────────────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────────────────┘

1. SGLang Server Configuration Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-qcfuse-worker
  namespace: rag-inference
spec:
  replicas: 4
  template:
    metadata:
      labels:
        app: sglang-qcfuse
    spec:
      containers:
        - name: sglang
          image: lmsysorg/sglang:latest
          env:
            - name: SGLANG_ENABLE_QCFUSE
              value: 'true'
            - name: QCFUSE_ANCHOR_RATIO
              value: '0.10'
            - name: QCFUSE_CRITICAL_LAYERS
              value: '14,16,18'
            - name: QCFUSE_RECOMPUTE_RATIO
              value: '0.20'
          args:
            - '--model-path=meta-llama/Meta-Llama-3.1-70B-Instruct'
            - '--tensor-parallel-size=4'
            - '--enable-paged-attention'
          ports:
            - containerPort: 30000

Enterprise teams managing automated content harvesting can pipe ingested web knowledge directly into QCFuse-enabled SGLang instances using aiFetchly API Integration and aiFetchly Web Scraping Workflows.

Empirical Benchmark Evaluation on Frontier LLMs

QCFuse was rigorously evaluated across four open-weights foundation models (LLaMA-3-8B, LLaMA-3-70B, Qwen-2.5-7B, and Qwen-2.5-32B) across six multi-hop reasoning and RAG benchmarks (HotpotQA, 2WikiMQA, MuSiQue, PopQA, TriviaQA, and QASPER):

Prefill-Phase Speedup vs Quality Retention:
1.8x ┤                                        ╭──── QCFuse (1.7x Speedup, 100% Quality!)
1.6x ┤                                   ╭────╯
1.4x ┤                      ............. ProphetKV (1.1x Speedup, Pipeline Stalls)
1.2x ┤        ..............
1.0x ┤  ────── Full Prefill Baseline (100% Quality Reference)
0.8x ┤  ─────────────────────────────────────────────────────────────
       60% Quality           80% Quality          100% Full-Prefill Quality

End-to-End Prefill Latency (4k Context on LLaMA-3-70B):
Full Prefill:        ████████████████████ 2,450 ms
ProphetKV:           ██████████████████ 2,180 ms  (Pipeline Stalled)
CacheBlend:          █████████ 1,020 ms  (Severe 18% Quality Loss)
QCFuse:              ████████████ 1,440 ms  (1.7x Speedup, 100% Quality Match!)

Key Performance Findings:

1.7x End-to-End Prefill Speedup: At matched generation quality, QCFuse accelerates the prefill phase by an average of 1.7x over full prefill and 1.5x over ProphetKV.
100% Full-Prefill Quality Preservation: Unlike query-agnostic selectors that miss critical evidence on multi-hop questions, QCFuse’s anchor-conditioned query probing matches full-prefill accuracy across all tested benchmarks.
Unblocked Layer-Wise Pipelining: Forming the recomputation mask in $<1\text{ ms}$ allows SGLang to fully overlap asynchronous PCIe KV cache loading with Triton sparse kernel execution, eliminating GPU idle bubbles.

For detailed hardware configuration tables and enterprise sizing calculators, visit the aiFetchly Documentation and explore cluster cost savings on the aiFetchly Sizing & Pricing Portal.

Frequently Asked Questions (FAQ)

How is QCFuse different from CacheBlend and FusionRAG?

CacheBlend is query-agnostic (selecting tokens based purely on position), while FusionRAG uses only a single final-layer query signal. Both methods miss subtle multi-hop evidence tokens, causing quality degradation. QCFuse uses query-aware anchor probing and middle-layer attention signals, restoring full-prefill response accuracy while remaining fast.

Why does QCFuse focus on middle layers rather than final layers?

In transformer architectures, early layers focus on local syntax, while final layers exhibit attention sinks and output distribution smoothing. Extensive empirical profiling proves that evidence localization peaks decisively in middle layers (e.g., layers 14–18 in a 32-layer model), providing the most reliable signal for selective token recomputation.

Can QCFuse be applied to multimodal RAG or vision-language models?

Yes. QCFuse operates on the general transformer attention mechanism. Because image patches and text chunks can both be represented with anchor tokens, QCFuse readily extends to multimodal document and vision-language cache fusion pipelines.