LLM InferenceSuperchipsNVIDIA GH200KV CacheSLO SchedulingSuperInfer

SuperInfer Guide: Rotary Scheduling on GPU Superchips

October 14, 2026

The Superchip Hardware Shift in Modern LLM Serving

Serving modern large language models (LLMs) with 32k to 128k context windows has pushed conventional GPU clusters toward severe memory capacity walls. In traditional disaggregated architectures, accelerators such as the NVIDIA H100 or A100 connect to host CPUs through peripheral component interconnect express (PCIe Gen 5) buses. Under these configurations, moving key-value (KV) cache tensors between host system memory (DDR5) and high-bandwidth memory (HBM3) incurs punishing latency penalties because PCIe bandwidth tops out around 64 GB/s to 128 GB/s bidirectional.

The emergence of heterogeneous CPU-GPU superchips—most prominently the NVIDIA Grace Hopper GH200 and the Grace Blackwell GB200 architectures—fundamentally changes this physical baseline. A single GH200 superchip fuses an ARM-based 72-core Neoverse V2 Grace CPU with an Hopper GPU via NVIDIA NVLink Chip-to-Chip (NVLink-C2C), delivering an interconnect bandwidth of 900 GB/s. This interconnect provides up to 7x the bandwidth of standard PCIe Gen 5 while maintaining coherent memory addressing between 480 GB of LPDDR5X CPU DRAM and 96 GB or 144 GB of GPU HBM3e.

Despite this hardware leap, running state-of-the-art serving frameworks like vLLM, TensorRT-LLM, or Megatron-LM directly on superchips creates an unexpected bottleneck. Because these systems were architected around discrete accelerators, they rely on passive offloading or synchronous kernel execution. On superchips, naive paging yields less than 5% sustained utilization of the 900 GB/s NVLink-C2C channel. Requests stall on memory allocations, tail latencies balloon, and service-level objectives (SLOs) collapse under bursty enterprise traffic.

To bridge this architectural chasm, researchers introduced SuperInfer (presented as an Oral paper at MLSys 2026 by Jiahuan Yu, Mingtao Hu, Zichao Lin, and collaborators). SuperInfer treats the superchip not as an accelerator tethered to a host, but as a unified memory-hierarchy computing node. By implementing an operating-system-inspired proactive rotary scheduler (RotaSched) and a full-duplex asynchronous memory rotation engine (DuplexKV), SuperInfer raises time-to-first-token (TTFT) SLO attainment by up to 74.7% while boosting overall generation throughput across frontier open-weight models.

The Superchip Software Bottleneck: Why Conventional Offloading Fails

Understanding why standard LLM inference runtimes fail on superchips requires dissecting the interaction between memory bandwidth, batch scheduling policies, and attention cache dynamics. When serving production queries—such as document summarization, code generation, or agentic data-fetching—input context lengths vary widely.

Traditional LLM inference engines employ PagedAttention to allocate KV cache memory in fixed-size blocks (typically 16 or 32 tokens). While this eliminates external memory fragmentation, HBM capacity remains finite. When aggregate KV cache demand exceeds available HBM during high-concurrency spikes, standard serving engines resort to one of two mechanisms:

Recomputation: The scheduler terminates or pauses decoding requests, drops their accumulated KV cache blocks, and re-executes the prefill phase when memory becomes available. This wastes massive amounts of compute and severely damages time-between-tokens (TBT) metrics.
Synchronous Swap / Passive Offloading: The scheduler swaps victim blocks out to CPU host memory over PCIe. However, because offloading is treated as an emergency fallback rather than a primary scheduling dimension, the transfer blocks execution threads.

On a GH200 superchip, passive swapping introduces severe synchronization overhead. Although the physical NVLink-C2C link can sustain 900 GB/s, standard CUDA memory copies (cudaMemcpyAsync) invoked in response to sudden out-of-memory (OOM) triggers suffer from thread synchronization barriers and uncoalesced memory access patterns.

+-------------------------------------------------------------------------+
|                  Conventional Serving Engine on Superchip               |
|                                                                         |
|   +-------------------+    OOM Event Triggered    +-----------------+   |
|   | GPU HBM3e (96GB)  | ------------------------> | CPU Host DRAM   |   |
|   | PagedAttention    |   Synchronous Transfer    | 480GB LPDDR5X   |   |
|   +-------------------+   (<5% C2C Utilization)   +-----------------+   |
|             |                                               |           |
|             v                                               v           |
|   Decoding Blocks Blocked                        Preempted Requests     |
|   Latency Violations (SLO Drops)                 Deadlines Missed       |
+-------------------------------------------------------------------------+
                                     vs.
+-------------------------------------------------------------------------+
|                    SuperInfer Engine on GH200 Superchip                 |
|                                                                         |
|   +-------------------+   Bidirectional DMA Pipes   +---------------+   |
|   | GPU HBM3e (96GB)  | <=========================> | CPU Host DRAM |   |
|   | Active KV Pool    |    Full-Duplex NVLink-C2C   | Standby Pool  |   |
|   +-------------------+    (900 GB/s Theoretical)   +---------------+   |
|             ^                                               ^           |
|             |          RotaSched Rotary Engine              |           |
|             +-----------------------------------------------+           |
|                    Proactive Time-Sliced Migration                      |
|                    74.7% Higher TTFT SLO Attainment                     |
+-------------------------------------------------------------------------+

Furthermore, standard frameworks manage memory reactively: they wait until a memory allocation failure occurs before selecting victim sequences to swap out. By the time a request is preempted, the scheduler has already missed or compromised the client’s TTFT and TBT latency deadlines.

Core Innovation 1: RotaSched (SLO-Aware Proactive Rotary Scheduling)

To eliminate reactive preemption, SuperInfer borrows a foundational design principle from operating system process scheduling: active round-robin and time-slicing state transitions. Instead of categorizing requests strictly as either running in GPU memory or preempted to disk/host memory, RotaSched introduces a three-state execution lifecycle:

Active State (Running in HBM): The request resides in GPU HBM3e, actively participating in batch tensor-parallel or continuous-decoding GEMM/GEMV iterations.
Rotary Standby State (Buffered in CPU DRAM): The request’s KV cache is staged in the 480 GB LPDDR5X CPU memory pool. Its latency budget is actively monitored, and its blocks are kept hot for immediate promotion.
Transit State (In-Flight over NVLink-C2C): Memory chunks are actively streaming across the bidirectional C2C link in background CUDA streams, fully overlapped with ongoing attention computation.

The Latency Slack Metric

RotaSched makes migration decisions based on SLO Slack, defined as the remaining real-world time before a request violates its contracted latency deadline. For a request $i$ with arrival time $A_i$, target time-to-first-token deadline $SLO_{TTFT}$, and expected prefill compute latency $T_{prefill}(L_i)$ for context length $L_i$:

$$Slack_{TTFT}(i) = A_i + SLO_{TTFT} - (t_{now} + T_{prefill}(L_i) + T_{swap}(L_i))$$

Similarly, for an actively decoding request $j$ with target time-between-tokens deadline $SLO_{TBT}$:

$$Slack_{TBT}(j) = t_{last_token}(j) + SLO_{TBT} - (t_{now} + T_{step} + T_{swap_in})$$

When HBM allocation reaches a configurable dynamic high-water mark (e.g., 85% capacity), RotaSched does not wait for an allocation fault. Instead, it computes the slack across all active requests. Requests with ample slack ($Slack > \tau$) are identified as rotation candidates.

Rather than being aborted, these candidate requests have their KV cache blocks proactively streamed to CPU memory over the 900 GB/s link while a request with critical slack ($Slack \approx 0$) is rotated into HBM to meet its deadline. This rotation operates with minimal penalty because NVLink-C2C transfer latencies for individual token chunks are orders of magnitude lower than recomputation costs.

# Conceptual representation of RotaSched Slack Evaluation
import time
from typing import List, Dict, Optional

class RequestMetrics:
    def __init__(self, req_id: str, arrival_time: float, context_len: int, slo_ttft: float, slo_tbt: float):
        self.req_id = req_id
        self.arrival_time = arrival_time
        self.context_len = context_len
        self.slo_ttft = slo_ttft
        self.slo_tbt = slo_tbt
        self.last_token_time = arrival_time
        self.state = "ACTIVE" # ACTIVE, ROTARY_STANDBY, TRANSIT

class RotaScheduler:
    def __init__(self, c2c_bandwidth_gbps: float = 900.0, hbm_limit_gb: float = 96.0):
        self.c2c_bw = c2c_bandwidth_gbps * 1e9 # bytes per sec
        self.hbm_limit = hbm_limit_gb * 1e9
        self.current_hbm_usage = 0.0

    def compute_slack(self, req: RequestMetrics, current_time: float) -> float:
        if req.state == "ROTARY_STANDBY":
            # Target is TTFT completion
            deadline = req.arrival_time + req.slo_ttft
            est_swap_time = (req.context_len * 2 * 32 * 128 * 2) / self.c2c_bw # 16-bit KV cache bytes
            est_compute = 0.005 # Estimated kernel prefill time (sec)
            return deadline - (current_time + est_swap_time + est_compute)
        else:
            # Target is TBT deadline for decoding step
            deadline = req.last_token_time + req.slo_tbt
            return deadline - (current_time + 0.015)

    def select_rotation_victims(self, active_requests: List[RequestMetrics], current_time: float) -> List[RequestMetrics]:
        # Sort active requests by largest slack (furthest from deadline)
        sorted_requests = sorted(
            active_requests,
            key=lambda r: self.compute_slack(r, current_time),
            reverse=True
        )
        return sorted_requests

Core Innovation 2: DuplexKV (Full-Duplex NVLink-C2C Streaming Engine)

While RotaSched provides the algorithmic policy for deciding when and what to rotate, the physical movement of multi-gigabyte KV tensors is governed by DuplexKV.

Conventional host-to-device memory copies use unidirectional DMA channels or synchronous CUDA kernels that cause pipeline bubbles. On superchip hardware, the NVLink-C2C interface provides independent, symmetrical transmit (TX) and receive (RX) lanes, each running at 450 GB/s to provide 900 GB/s aggregate bandwidth.

DuplexKV exploits this physical symmetry by implementing an asynchronous, full-duplex double-buffered pipeline:

1. Bidirectional Pipelining

While the KV cache blocks of incoming requests are being swapped in from CPU DRAM to GPU HBM (RX lane), DuplexKV simultaneously evicts the blocks of low-urgency requests from GPU HBM to CPU DRAM (TX lane). Both streams run concurrently without bus contention.

2. Page-Aligned Circular Buffering

DuplexKV allocates non-blocking pinned memory rings in LPDDR5X CPU RAM that align exactly with the physical memory layout of the GPU memory management unit (MMU). By avoiding intermediate user-space buffer copies and maintaining contiguous physical page descriptors, DuplexKV achieves 92.4% of the theoretical 900 GB/s bandwidth during peak rotation bursts.

3. Compute-Transfer Overlap

LLM generation consists of memory-bound autoregressive decoding kernels (GEMV) interspersed with compute-heavy attention steps. DuplexKV schedules chunked memory transfers to run in dedicated CUDA streams that execute concurrently with tensor core math, effectively hiding KV transfer latency behind ongoing decoding steps.

+--------------------------------------------------------------------------+
|                     DuplexKV Full-Duplex Overlapping                     |
|                                                                          |
| Time (t) --->                                                            |
| CUDA Stream 0 (Compute):   [ Decoding Step N ]  [ Decoding Step N+1 ]    |
| CUDA Stream 1 (C2C In):    ====== Swap In Blocks (Req B) =======>        |
| CUDA Stream 2 (C2C Out):   <===== Swap Out Blocks (Req A) ======         |
|                                                                          |
| Result: Memory movement completely hidden behind decoding compute phase  |
+--------------------------------------------------------------------------+

Mathematical Modeling: KV Cache Footprint and Bandwidth Bounds

To appreciate why SuperInfer delivers such pronounced gains on superchips, consider the exact memory footprint of modern autoregressive models. For a model with $N_{layers}$ layers, $N_{heads}$ key-value attention heads (under Grouped-Query Attention or Multi-Head Attention), head dimension $d_{head}$, and precision bytes per parameter $P$ (where $P = 2$ for FP16/BF16 and $P = 1$ for FP8):

$$S_{token} = 2 \times N_{layers} \times N_{heads} \times d_{head} \times P \quad \text{(bytes per token)}$$

For LLaMA-3-70B ($N_{layers} = 80$, $N_{kv_heads} = 8$, $d_{head} = 128$, precision BF16 with $P=2$):

$$S_{token} = 2 \times 80 \times 8 \times 128 \times 2 = 327,680 \text{ bytes} \approx 320 \text{ KB per token}$$

A single enterprise conversation with a context length of 32,768 tokens consumes:

$$S_{req} = 32,768 \times 320 \text{ KB} \approx 10.485 \text{ GB}$$

On a standard 96 GB HBM3 GPU, serving merely 6 concurrent 32k-context requests exhausts the entire memory allocation remaining after loading the 70B model weights (which occupy ~140 GB in FP16, requiring quantization to FP8 or multi-GPU sharding).

Now examine the data transfer time across different bus interconnects:

Interconnect Standard	Theoretical Peak Bandwidth	Time to Transfer 10.48 GB KV Cache	Max Swaps/Second
PCIe Gen 4 x16	31.5 GB/s	332.8 ms	3.0 swaps/sec
PCIe Gen 5 x16	63.0 GB/s	166.4 ms	6.0 swaps/sec
NVIDIA NVLink-C2C (GH200)	900.0 GB/s	11.6 ms	85.8 swaps/sec

Transferring a 10.5 GB cache over PCIe Gen 5 requires 166.4 ms—far longer than the standard 20–50 ms inter-token decoding budget. Over PCIe, offloading causes massive stalls. Over NVLink-C2C in SuperInfer, transferring that same cache takes a mere 11.6 milliseconds, falling well inside the duration of a standard decoding iteration or forward pass.

This mathematical reality explains why SuperInfer’s rotary paradigm works exclusively and exquisitely on superchips: the hardware interconnect is finally fast enough to allow memory rotation without breaking the execution flow.

Architectural Comparison: SuperInfer vs Baseline Serving Stacks

To benchmark the architectural differences between SuperInfer and contemporary LLM inference engines, we evaluate key subsystem behaviors across unified memory architectures:

Architectural Metric	vLLM (v0.6+)	DeepSpeed-FastGen	TensorRT-LLM	SuperInfer (MLSys 2026)
Interconnect Target	PCIe Gen 4/5	PCIe Gen 4/5	Proprietary NVLink / PCIe	NVLink-C2C (Superchips)
Preemption Mechanism	Abort or Sync Swapping	Head-of-Line Blocking	Dynamic Chunking	Proactive RotaSched Slicing
C2C Bus Utilization	3.8% – 5.2%	4.1% – 6.0%	12.4% – 18.0%	88.6% – 92.4%
KV Offload Strategy	Reactive OOM Swapping	Host Memory Pinned	Host Cache Allocator	Full-Duplex DuplexKV Ring
SLO Awareness	First-Come-First-Served	Static Priority Queues	Static Deadline Profiling	Dynamic Slack Cost Function
Memory Target	Discrete GPU VRAM	Host RAM Fallback	Uniform Paged Blocks	Tiered HBM3e + LPDDR5X
TTFT SLO Attainment	Baseline (1.0x)	1.12x	1.28x	1.74x (+74.7%)

By moving beyond the PCIe accelerator paradigm, SuperInfer unlocks the full potential of high-density heterogenous superchips.

Step-by-Step Implementation and Deployment Guide

Deploying SuperInfer on NVIDIA GH200 systems requires configuring unified memory allocations, initializing the DuplexKV kernel drivers, and configuring the RotaSched SLO thresholds. The following guide walks through setting up an end-to-end SuperInfer instance.

Step 1: System Prerequisite and Kernel Parameter Setup

Ensure your NVIDIA Grace Hopper host runs Linux Kernel 6.5 or later with unified memory management (CONFIG_DEVICE_PRIVATE=y) enabled. Verify the NVLink-C2C driver registers the coherent memory window:

# Verify NVLink-C2C status on NVIDIA GH200
nvidia-smi nvlink -s
nvidia-smi -q -d MEMORY

# Check coherent CPU-GPU numa nodes
numactl --hardware

Expected output indicates Node 0 (Grace CPU, 480 GB LPDDR5X) and Node 1 (Hopper GPU, 96 GB or 144 GB HBM3e) linked with direct high-speed interconnect channels.

Step 2: SuperInfer Serving Configuration

Create an environment configuration file superinfer_config.yaml specifying memory allocation bounds, model weights, and SLO target thresholds:

model:
  name: 'meta-llama/Llama-3-70b-Instruct'
  tensor_parallel_size: 1
  pipeline_parallel_size: 1
  kv_cache_dtype: 'fp8_e5m2'
  max_model_len: 32768

hardware:
  platform: 'gh200'
  c2c_bandwidth_gb: 900
  hbm_memory_fraction: 0.88
  cpu_memory_buffer_gb: 384

rotasched:
  enabled: true
  slack_evaluation_interval_ms: 5.0
  ttft_slo_target_ms: 120.0
  tbt_slo_target_ms: 25.0
  proactive_eviction_threshold: 0.85
  recovery_headroom_tokens: 4096

duplex_kv:
  stream_concurrency: 4
  pinned_ring_buffer_pages: 65536
  enable_bidirectional_overlap: true
  direct_dma_alignment: 4096

Step 3: Launching the SuperInfer Engine

Initialize the SuperInfer server using the Python runtime bindings. Developers integrating automated pipelines can connect SuperInfer directly with custom workflows, similar to connecting an api-integration or automated web-scraping ingestion queue:

import asyncio
from superinfer.engine import SuperInferServer
from superinfer.config import ServingConfig

async def main():
    # Load configuration
    config = ServingConfig.from_yaml("superinfer_config.yaml")

    # Initialize engine with DuplexKV and RotaSched
    print("[INFO] Initializing SuperInfer on NVIDIA GH200 Superchip...")
    server = SuperInferServer(config)
    await server.start()

    print(f"[INFO] Server ready. Listening for requests with TTFT SLO={config.rotasched.ttft_slo_target_ms}ms")
    await server.serve_forever()

if __name__ == "__main__":
    asyncio.run(main())

Step 4: Client Request Submission with Latency SLO Headers

Clients pass latency expectations in their inference headers, allowing RotaSched to calculate real-time slack:

import httpx
import json

client = httpx.Client(base_url="http://localhost:8000")

payload = {
    "model": "meta-llama/Llama-3-70b-Instruct",
    "prompt": "Analyze the following high-throughput database telemetry report...",
    "max_tokens": 1024,
    "temperature": 0.2,
    "slo_constraints": {
        "ttft_ms": 150.0,
        "tbt_ms": 30.0,
        "priority_class": "critical"
    }
}

response = client.post("/v1/completions", json=payload, timeout=60.0)
print(response.json())

Production Benchmarks: Empirical Performance Evaluation

The MLSys 2026 paper presents exhaustive empirical evaluations of SuperInfer across synthetic trace workloads and real-world production traces from ShareGPT and LMSYS Chatbot Arena.

1. TTFT and TBT SLO Attainment Under Bursty Loads

Under Poisson arrival distributions representing sudden traffic bursts, standard vLLM instances suffer severe head-of-line blocking when HBM fills. Requests arriving during peak memory saturation experience massive delays while waiting for existing sequences to complete or swap over PCIe.

SuperInfer’s RotaSched maintains consistent SLO compliance by continuously rotating requests with high slack into the 480 GB LPDDR5X memory pool:

SLO Attainment vs Request Rate (LLaMA-3-70B on GH200)

100% |-------------------------------------============================= (SuperInfer: 94.2%)
     |                                   //
 80% |----------------------------------//-------------------------------
     |                                //
 60% |                               //       --------------------------- (TensorRT-LLM: 61.5%)
     |                             //
 40% |                            //          ........................... (vLLM Baseline: 54.0%)
     |                          //
 20% |                         //
     |________________________//_________________________________________
     0.5 req/s            1.0 req/s           2.0 req/s          3.5 req/s

At an arrival rate of 3.0 requests/second with context lengths averaging 16k tokens:

vLLM Baseline: Achieves 54.0% TTFT SLO attainment (drops nearly half of all request deadlines).
TensorRT-LLM: Achieves 61.5% TTFT SLO attainment.
SuperInfer: Achieves 94.2% TTFT SLO attainment, representing a net 74.7% improvement over the baseline.

2. End-to-End Throughput and Bandwidth Utilization

By overlapping memory transfers with attention execution via DuplexKV, SuperInfer achieves sustained bandwidth utilization of 812 GB/s across the NVLink-C2C bus during high-churn generation phases, compared to only 42 GB/s in unoptimized implementations:

Model Architecture	Concurrency (Batch Size)	Average Context Length	Baseline Throughput (tokens/sec)	SuperInfer Throughput (tokens/sec)	Throughput Gain
Qwen2.5-32B	64	16,384	482 tok/s	894 tok/s	+85.4%
LLaMA-3-8B	128	32,768	1,240 tok/s	2,190 tok/s	+76.6%
LLaMA-3-70B (FP8)	32	16,384	215 tok/s	368 tok/s	+71.1%

These speedups demonstrate that memory capacity limits on superchips can be effectively overcome through algorithmic co-design.

Operational Checklist: Deploying SuperInfer in Production

Before rolling out SuperInfer across an enterprise superchip cluster, engineering teams should verify system readiness against this operational checklist:

Hardware Coherency: Run numactl --hardware to verify that CPU DRAM and GPU HBM reside on contiguous, non-interleaved NUMA topologies.
HugePages Configuration: Allocate 2MB or 1GB HugePages in Linux host memory to prevent TLB thrashing during DuplexKV DMA transfers:

echo 32768 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
CUDA IPC and Unified Driver Support: Ensure NVIDIA CUDA Driver version is 550.54 or higher with Fabric Manager enabled.
SLO Profile Calibration: Profile model-specific prefill and decoding latency curves under varying context lengths ($L \in [1024, 65536]$) to populate the RotaSched slack estimator.
Monitoring and Observability: Integrate Prometheus exporters tracking nvlink_c2c_throughput_bytes, rotasched_active_rotations_total, and kv_cache_hbm_usage_ratio.
Documentation and Governance: Check system reference materials in your internal docs and align capacity planning with standard infrastructure pricing.

Strategic Implications and Future Superchip Trajectories

The design principles established by SuperInfer point directly to the future of AI computing infrastructure. As model context windows expand toward 1 million tokens and agentic loops require persistent conversational histories, traditional discrete GPU nodes will increasingly hit capacity walls.

With the rollout of next-generation superchips such as the NVIDIA Grace Blackwell GB200 NVL72—which connects 36 Grace CPUs and 72 Blackwell GPUs over a massive 1.8 TB/s bidirectional NVLink fabric—the line between host memory and accelerator memory dissolves entirely. RotaSched and DuplexKV provide the architectural blueprint for managing these petabyte-scale unified memory fabrics.

For teams building autonomous AI platforms, high-throughput agents, or production knowledge graphs, adopting superchip-native inference runtimes like SuperInfer is no longer merely an optimization—it is an architectural necessity. You can discover more guides and architectural deep dives on our homepage as we continue tracking breakthrough research across the AI infrastructure ecosystem.