Nitsum Guide: Adaptive Tensor Parallelism for Tiered SLOs

The Multi-Tier SLO Serving Bottleneck in Modern LLM Clusters

In modern hyperscale AI deployments, production inference clusters rarely process a single homogenous stream of queries. Instead, enterprise AI infrastructure—such as the real-time document extraction and agentic pipeline engines built by aiFetchly—must concurrently serve diverse, multi-tenant workloads governed by starkly contrasting Service Level Objectives (SLOs).

Modern LLM workloads naturally partition into distinct tiers:

1. Interactive Tier (Strict SLOs): Web search, autonomous agent tool calling, chatbot user queries, and live code completion demand ultra-low Time-To-First-Token (TTFT < 200 ms) and smooth, conversational Time-Per-Output-Token (TPOT < 25 ms). Violating these thresholds degrades human experience and breaks conversational timeouts.
2. Offline / Background Tier (Relaxed SLOs): Asynchronous document ingestion, web scraping batch synthesis, automated summarization, and offline evaluations prioritize aggregate throughput (tokens/second per dollar) over per-token latency. TTFT deadlines for this tier often span tens of seconds or minutes.

Despite this natural divergence in workload characteristics, state-of-the-art inference engines—including vLLM, TensorRT-LLM, TGI, and DistServe—treat parallelization configurations as rigid, static deployment constants. When deploying an 8-GPU node (such as an NVIDIA HGX H100), system operators must commit at boot time to a fixed Tensor Parallelism (TP) degree, typically choosing between static high parallelism (TP=8) or static low parallelism (e.g., four independent TP=2 replicas or eight TP=1 instances).

This fundamental trade-off forces infrastructure architects to either drastically over-provision GPU clusters—burning millions of dollars on idle accelerator reserves—or accept routine SLO degradation during peak traffic.

To dismantle this architectural barrier, systems researchers from the WukLab at UC San Diego introduced Nitsum (presented in Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism, arXiv:2605.05467). Nitsum reformulates Tensor Parallelism from an immutable startup configuration into an elastic, millisecond-level runtime control surface.

In this technical architectural guide, we dissect the design of Nitsum, examine its core mechanisms—including reload-free weight reuse and parallel KV cache migration—and walk through a complete implementation and production deployment strategy.

Why Static Cluster Parallelism Violates Tiered SLOs

To understand why conventional inference systems fail under mixed workloads, we must analyze the physics of distributed tensor execution and batch queuing.

1. The Prefill Latency vs. Decode Throughput Conflict

Autoregressive transformer inference consists of two computationally distinct phases:

1. Prefill Phase (Compute-Bound): Ingests the prompt tokens simultaneously. The computational complexity scales quadratically with sequence length $\mathcal{O}(S^2)$, while matrix multiplication arithmetic intensity is high ($\text{FLOPs} \gg \text{Memory Bytes}$). Tensor parallelism divides matrix dimension $K$ or $N$ across GPUs, accelerating prompt ingestion almost linearly when communication overhead is low.
2. Decode Phase (Memory-Bound): Generates one token at a time per sequence. At batch size $b=1$, each forward pass reads all model weights from HBM to compute a single token, yielding arithmetic intensity near zero ($\approx 1\text{ FLOP}/\text{byte}$). Tensor parallelism provides diminishing returns here because the compute duration is dominated by memory read latency and inter-GPU communication latency.

When an engine is pinned to static $TP=8$, every token generation step in the decode phase triggers cross-GPU All-Reduce collectives. At small batch sizes, GPUs spend more time waiting for NVLink synchronization than executing tensor core operations.

Conversely, when pinned to static $TP=2$, background requests achieve high throughput because GEMM matrices are wide and communication is minimized. However, when an interactive request with a 4,000-token prompt arrives, the two GPUs take over 1,200 ms to compute the prefill phase, instantly violating strict interactive SLOs.

2. Headroom Waste in Disaggregated Architectures

Recent disaggregated architectures—such as Splitwise and DistServe—decouple prefill nodes from decode nodes. However, even disaggregated systems fix their internal parallelization degrees:

• Prefill nodes are statically provisioned at high parallelism (e.g., $TP=8$).
• Decode nodes are statically provisioned at low parallelism (e.g., $TP=2$).

When interactive prompt traffic subsides during off-peak hours, the expensive prefill GPU pool sits mostly idle. Because its TP degree is fixed, the cluster cannot dynamically repurpose those GPUs to process high-throughput background decoding without triggering a full process restart and reloading model weights.

Nitsum resolves this inefficiency by enabling GPUs to shift dynamically between prefill and decode roles, and across TP granularities, in less than 50 milliseconds.

Nitsum Architectural Principles: Elastic Parallelism as a Control Surface

Nitsum structures the GPU cluster as a reconfigurable resource pool managed by a centralized, lightweight coordinator.

The system operates across three synchronized control dimensions:

1. Topology Adaptation: Dynamically scaling worker group sizes between $TP=1$, $TP=2$, $TP=4$, and $TP=8$.
2. Phase Assignment: Flexibly designating worker groups as dedicated prefill workers, dedicated decode workers, or hybrid workers based on current queue backlogs.
3. Queue Scheduling: Routing incoming prompts to the worker group that guarantees completion within the query’s residual slack budget while maximizing overall cluster goodput.

Deep Dive: Reload-Free Tensor Parallel Weight Reuse

The central innovation that allows Nitsum to reconfigure parallel topologies in milliseconds—rather than seconds—is Reload-Free Weight Reuse.

1. The Mechanics of Megatron-LM Weight Sharding

In Megatron-style tensor parallelism, linear layers inside transformer blocks are partitioned across attention and MLP projections:

• Self-Attention Query-Key-Value (QKV): Partitioned column-wise across $N$ GPUs.
• Self-Attention Output Projection: Partitioned row-wise across $N$ GPUs, followed by an All-Reduce sum.
• MLP Gate and Up Projections: Partitioned column-wise across $N$ GPUs.
• MLP Down Projection: Partitioned row-wise across $N$ GPUs, followed by an All-Reduce sum.

Mathematically, let $W \in \mathbb{R}^{d_{in} \times d_{out}}$ denote a column-parallel weight matrix. At maximum parallelism $TP_{max} = 8$, the matrix is segmented into 8 disjoint contiguous column blocks:

$$W = \left[ W^{(0)} ,|, W^{(1)} ,|, W^{(2)} ,|, W^{(3)} ,|, W^{(4)} ,|, W^{(5)} ,|, W^{(6)} ,|, W^{(7)} \right]$$

where each $W^{(i)} \in \mathbb{R}^{d_{in} \times \frac{d_{out}}{8}}$.

2. Zero-Copy Logical Slicing via NVLink Memory Mapping

In a static system, switching a GPU from $TP=8$ to $TP=2$ requires that GPU to host a tensor of width $d_{out}/2$, which corresponds to four fine-grained shards (e.g., $W^{(0..3)}$). Traditionally, an engine would allocate a new PyTorch tensor and stream weights from system RAM over PCIe at 32–64 GB/s, stalling inference for hundreds of milliseconds.

Nitsum bypasses physical copying completely. During node startup, Nitsum leverages CUDA Virtual Memory Management (VMM) and peer-to-peer NVLink mappings (cudaMemAddressRange_t and cudaMemMap):

1. Base Fine Partitioning: Each GPU allocates its primary physical memory for its native $TP_{max}$ slice.
2. Peer Direct Mapping: Through NVLink (which provides 900 GB/s bidirectional bandwidth on H100), each GPU maps the virtual address spaces of adjacent peer GPUs directly into its local Unified Virtual Addressing (UVA) map.
3. Pointer Swapping: When merging two GPUs into a lower TP group (e.g., GPU 0 and GPU 1 forming a $TP=2$ pair), GPU 0 logically updates its tensor pointer offsets to span both its local slice $W^{(0)}$ and GPU 1’s slice $W^{(1)}$.

Because this operation only modifies page tables and kernel launch arguments, reconfiguration completes in under 15 milliseconds, allowing the cluster to adapt between token generation steps.

Deep Dive: Parallel Inter-GPU KV Cache Migration Mechanics

While weights represent static read-only memory, the Key-Value (KV) cache represents dynamic, stateful memory that grows with every generated token. When Nitsum alters worker group boundaries, active requests must have their KV cache states resharded without disrupting ongoing generation.

1. PagedAttention Block Alignment

Modern LLM inference engines store KV cache in non-contiguous memory blocks (typically 16 tokens per block) managed by a page table. Nitsum extends PagedAttention by introducing Distributed Block Virtualization:

• Block metadata contains a logical_tp_stride field indicating how heads are partitioned across physical GPUs.
• When resharding between $TP=4$ and $TP=2$, the coordinator calculates the target block layout and launches parallel cudaMemcpyPeerAsync streams across the NVLink fabric.

2. Double-Buffered Zero-Pause Migration

To prevent request stalls during resharding:

1. Pre-Copy Phase: While the current worker group finishes decoding token $t$, the background DMA engine transfers historical blocks $0 \dots (t-1)$ to the target GPU group.
2. Delta Transfer: Once token $t$ is computed, only the final block delta (1 token) is synchronized.
3. Atomic Cutover: The coordinator updates the request’s routing target. Token $t+1$ executes immediately on the reconfigured worker group.

This double-buffered migration guarantees that inter-GPU KV transfers introduce zero visible generation bubbles into user-facing token streams.

Queue-Aware Joint Scheduling & Goodput Maximization Formulation

Nitsum coordinates cluster reconfiguration through a continuous mathematical optimization problem evaluated at 1-second intervals.

1. Mathematical Objective

Let $\mathcal{R}$ represent the set of active requests, partitioned into interactive queries $\mathcal{R}{int}$ and background batch queries $\mathcal{R}{bg}$.

For each request $r$, define:

$$\text{Slack}(r) = \text{SLO}(r) - \text{ElapsedTime}(r) - \text{EstimatedRemainingTime}(r, TP)$$

A request is considered SLO-compliant if and only if $\text{Slack}(r) \ge 0$.

The cluster objective is to maximize total valid goodput $G$:

$$\max_{\mathcal{C}, \mathcal{M}} \quad \sum_{r \in \mathcal{R}} w(r) \cdot \mathbb{I}(\text{Slack}(r) \ge 0) - \lambda \cdot \text{Cost}_{reconfig}(\mathcal{C} \to \mathcal{C}’)$$

where:

• $\mathcal{C}$ is the current cluster configuration (e.g., one $TP=8$ worker vs. four $TP=2$ workers).
• $\mathcal{C}’$ is the candidate cluster configuration.
• $w(r)$ is the tier weight ($w_{int} = 10.0$, $w_{bg} = 1.0$).
• $\text{Cost}{reconfig}$ is the total reconfiguration overhead ($\approx 15\text{ ms} + T{kv_transfer}$).
• $\lambda$ is a damping regularization parameter preventing excessive flapping between configurations.

2. Decision Tree Policy

To ensure scheduling decisions execute in under 1 millisecond, Nitsum evaluates a fast heuristic decision tree:

End-to-End Implementation: Building a Tiered-SLO Serving Controller

Let us now review the complete working implementation of the Nitsum serving controller. This production-grade simulation accurately models the behavior of an 8-GPU cluster serving mixed workloads under high concurrency.

To run this simulation, execute the script with Python 3:

Benchmark Evaluation: Nitsum vs. vLLM, DistServe, and Splitwise

In empirical evaluations conducted across production Azure LLM traces and bursty synthetic workloads on 8x NVIDIA H100 SXM5 nodes, Nitsum consistently outperforms static baseline architectures.

1. Multi-Tier SLO Goodput

Under mixed workloads where 30% of requests are interactive (TTFT < 200 ms) and 70% are offline batch ingestion, Nitsum achieves:

• Up to 5.3x higher goodput than vanilla vLLM configured with static $TP=8$.
• Up to 2.4x higher goodput than DistServe’s fixed prefill/decode disaggregated cluster.
• 74% reduction in P99 interactive TTFT during traffic bursts.

2. Reconfiguration Overhead Analysis

The benchmark demonstrates that Nitsum’s reload-free weight reuse incurs negligible latency overhead:

• Logical Weight Slice Switch: 11.2 ms average across 8 GPUs.
• KV Cache Peer DMA Transfer: 4.8 ms to 16.5 ms depending on active batch sequence length.
• Total Topology Transition Time: Consistently under 45 ms, easily fitting inside the typical inter-token scheduling window of distributed inference.

Production Deployment Playbook: Kubernetes, Ray, and Prometheus Monitoring

Deploying Nitsum in enterprise production environments requires orchestrating GPU affinity, configuring NVLink peer-to-peer permissions, and establishing real-time SLO monitoring.

1. Kubernetes Resource Manifest

Ensure your Kubernetes pod manifest grants IPC host sharing and NVIDIA capabilities required for zero-copy peer memory mapping:

2. Client Ingestion with Tier Headers

When sending queries from downstream applications, include explicit tier metadata headers so the Nitsum gateway can enforce appropriate scheduling rules:

For teams building automated ingestion pipelines, integrating with aiFetchly API Integration or exploring aiFetchly Data Fetching allows decoupling high-volume document scraping from interactive user query paths.

Frequently Asked Questions (FAQ)

What is Nitsum and how does it differ from vanilla vLLM?

Nitsum is an open-source distributed serving framework developed by the WukLab at UC San Diego that introduces runtime-adaptive Tensor Parallelism (TP). While vanilla vLLM requires selecting a fixed TP degree at cluster boot time (e.g., permanently locking an 8-GPU node into TP=8 or four TP=2 instances), Nitsum treats TP as an elastic, millisecond-level runtime control surface. It dynamically merges or partitions GPU worker groups based on real-time interactive queue backlogs and background batch volume.

How does Nitsum change tensor parallelism without reloading weights?

Nitsum uses a technique called Reload-Free Weight Reuse. During cluster initialization, model weights are pre-partitioned at the maximum parallelism granularity ($TP_{max}=8$). Adjacent GPUs map each other’s virtual address spaces over high-bandwidth NVLink interconnects using CUDA virtual memory management. When GPUs merge into lower TP groups, the system simply updates logical tensor pointers and kernel launch dimensions—reconfiguring weights in under 15 milliseconds without reading from host RAM or storage.

Does migrating the KV cache cause generation pauses or dropped tokens?

No. Nitsum employs a double-buffered asynchronous migration mechanism. In-flight requests continue generating tokens on their existing worker group while historical KV cache blocks are copied across NVLink via asynchronous peer-to-peer DMA streams. Only the final token state is synchronized before atomic cutover, ensuring zero token generation bubbles.

Can Nitsum be deployed alongside pipeline or data parallelism?

Yes. While Nitsum focuses on tensor parallelism inside high-bandwidth NVLink domains (typically single 8-GPU nodes), it composes cleanly with Data Parallelism (DP) across separate server nodes and Pipeline Parallelism (PP) for models exceeding single-node memory capacity.

What hardware is required to run Nitsum?

Nitsum requires multi-GPU nodes with high-speed peer-to-peer interconnects supporting direct DMA and Unified Virtual Addressing (UVA). It is optimized for NVIDIA HGX architectures (H100, H200, B200, and A100) connected via NVLink and NVSwitch. Standard PCIe-only topologies can run Nitsum, but KV cache resharding and weight pointer updates will experience slightly higher transition latencies.

Where can I access the Nitsum research paper and source code?

The Nitsum research paper is available on the arXiv preprint repository (Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism, arXiv:2605.05467), authored by researchers at the UC San Diego WukLab. You can inspect implementation details and cluster configuration templates in the open-source repository.

For more details on deploying scalable AI pipelines, consult the aiFetchly Documentation and check our enterprise aiFetchly Pricing options.