QuaRot Guide: Outlier-Free 4-Bit W4A4 LLM Quantization

The Problem: The Activation Outlier Wall in Sub-8-Bit Quantization

In modern high-concurrency generative AI pipelines, automated research engines, and high-frequency data scrapers like AiFetchly, deploying Large Language Models (LLMs) with tens or hundreds of billions of parameters requires overcoming the GPU Memory-Bandwidth Wall.

During autoregressive generation, foundation models like LLaMA-3 70B or Qwen-2.5 72B spend the overwhelming majority of their GPU cycle time moving parameter weights and Key-Value (KV) cache tensors across the High-Bandwidth Memory (HBM) bus rather than executing arithmetic calculations.

To slash memory footprints and compress models, the machine learning industry broadly adopted post-training quantization. However, sub-8-bit quantization historically encountered a severe architectural split:

1. Why Weight-Only Quantization (W4A16) Leaves Hardware Idle

Popular 4-bit quantization methods—such as AWQ (Activation-aware Weight Quantization) and GPTQ—are strictly weight-only (W4A16):

• Model weights are compressed into 4-bit integers (INT4), reducing static VRAM from 140 GB to ~35 GB for a 70B model.
• However, intermediate activations remain unquantized in 16-bit precision (FP16 or BF16).
• Before each matrix multiplication ($Y = X W$), the GPU must dynamically dequantize the 4-bit weights back into 16-bit registers.
• Consequently, matrix multiplications are executed using standard FP16 Tensor Cores. The GPU cannot use high-throughput native INT4 Tensor Cores (which deliver up to $2\times$ higher compute throughput than FP16 on NVIDIA Ampere, Ada Lovelace, and Hopper architectures).
• Furthermore, because activations remain in 16-bit, the Key-Value (KV) cache is uncompressed, frequently consuming over 40 GB of VRAM per user on 128k context windows.

2. The Emergence of Channel-Aligned Activation Outliers

Why hasn’t the industry simply quantized both weights and activations to 4-bit (W4A4)?

The fundamental roadblock is the Activation Outlier Phenomenon (Dettmers et al., 2022). In transformers with more than 6.7 billion parameters, self-attention and MLP layers develop extreme activation outliers:

• Across all sequence positions, approximately $0.1%$ of hidden channels exhibit activation magnitudes up to $100\times$ larger than typical dimensions (e.g., values reaching $+120.0$ while normal dimensions fluctuate between $-1.0$ and $+1.0$).
• These outliers are strictly coordinate-aligned: they consistently appear in the exact same channel indices across tokens, functioning as internal attention coordinators.

3. The Collapse of Naive W4A4 and SmoothQuant’s Limits

In standard uniform quantization, a continuous value $x$ is mapped to an integer grid:

$$x_{\text{quant}} = \text{clip}\left(\left\lfloor \frac{x}{S} \right\rceil, -2^{b-1}, 2^{b-1}-1\right), \quad S = \frac{\max(|X|)}{2^{b-1}-1}$$

When quantizing into 4-bit integer format ($b = 4$), the grid has only 16 discrete levels (from $-8$ to $+7$):

• If the scale factor $S$ is set to accommodate the massive outlier ($+120.0$), the quantization bin step size becomes $S \approx 15.0$.
• All normal activation values (which lie between $-1.0$ and $+1.0$) are rounded directly to 0, obliterating more than $99%$ of the model’s semantic representations.
• Conversely, if the outlier is clipped to preserve normal feature resolution, truncating the coordinator features causes catastrophic attention collapse. Naive W4A4 causes evaluation perplexity on WikiText-2 to explode from $5.4$ to $>1,000$!

Prior solutions like SmoothQuant (Xiao et al., 2023) mathematically migrated the quantization difficulty from activations to weights using a per-channel scaling factor $s \in \mathbb{R}^d$:

$$Y = X W = (X \cdot \text{diag}(s)^{-1}) \cdot (\text{diag}(s) \cdot W)$$

While SmoothQuant successfully enables 8-bit weights and activations (W8A8), it breaks down completely at 4-bit (W4A4). The 16 discrete levels of INT4 are too coarse to absorb migrated outliers without destroying weight accuracy.

The Architecture: Computational Invariance via Orthogonal Hadamard Rotations

To break through the activation outlier wall without adding learned parameters or degrading perplexity, researchers from EPFL and Microsoft Research introduced QuaRot (Quantization with Rotation) (Ashkboos et al., NeurIPS 2024 / 2025).

QuaRot introduces a foundational mathematical insight: The presence of activation outliers is an artifact of the coordinate basis, not an inherent property of the underlying vector space.

By rotating the coordinate system using orthogonal transformations, QuaRot disperses outlier energy uniformly across all channels before quantization occurs.

1. Orthogonal Matrix Invariance and Coordinate Rotation

Consider a standard linear layer mapping activations $X \in \mathbb{R}^{n \times d_{\text{in}}}$ to output $Y \in \mathbb{R}^{n \times d_{\text{out}}}$ via weight matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$:

$$Y = X W^T$$

Let $H \in \mathbb{R}^{d_{\text{in}} \times d_{\text{in}}}$ be an orthogonal matrix, satisfying:

$$H^T H = H H^T = I$$

By inserting the identity matrix $I = H^T H$ between $X$ and $W^T$, we rewrite the matrix product:

$$Y = X (H^T H) W^T = (X H^T) (W H^T)^T = \tilde{X} \tilde{W}^T$$

Where:

• $\tilde{X} = X H^T$ is the rotated activation matrix.
• $\tilde{W} = W H^T$ is the rotated weight matrix.

Because orthogonal transformations preserve inner products and Euclidean norms ($|\tilde{X}|_2 = |X|_2$), the mathematical output $Y$ is bit-exact and completely invariant!

2. Why Randomized Walsh-Hadamard Transforms (RHT) Disperse Outliers

QuaRot chooses $H$ as a Randomized Walsh-Hadamard Matrix:

$$H = \frac{1}{\sqrt{d}} H_d \cdot D$$

Where:

• $D = \text{diag}(\pm 1)$ is a diagonal matrix of independent Rademacher random variables ($\pm 1$ with probability $0.5$).
• $H_d$ is the recursive Sylvester construction of the Walsh-Hadamard matrix:

$$H_{2k} = \begin{bmatrix} H_k & H_k \ H_k & -H_k \end{bmatrix}, \quad H_1 = [1]$$

When an activation vector $x$ containing a localized outlier ($x_i \gg 0$) is multiplied by $H$, the outlier’s energy is multiplied by the entries of $H$ ($\pm 1 / \sqrt{d}$):

• Each coordinate of the rotated vector $\tilde{x}_j$ becomes a linear combination of all original coordinates:

$$\tilde{x}j = \frac{1}{\sqrt{d}} \sum{k=1}^d D_{kk} H_{jk} x_k$$

• By the Central Limit Theorem, rotating high-magnitude spikes over an orthogonal basis spreads their variance evenly across all $d$ dimensions.
• The maximum channel magnitude $\max_j |\tilde{x}_j|$ drops by $5\times$ to $10\times$, completely eliminating the outlier spike!
• With outliers extinguished, the dynamic range shrinks dramatically, allowing standard uniform 4-bit quantization to capture features with minimal round-off error.

3. The Fast Walsh-Hadamard Transform (FWHT): O(d log d) Without Multiplications

If rotating activations required a dense matrix multiplication ($X \times H^T$), the computational overhead would negate the speedups of 4-bit inference.

Crucially, multiplying by a Walsh-Hadamard matrix does not require a dense GEMM. It can be executed via the Fast Walsh-Hadamard Transform (FWHT) in:

$$\mathcal{O}(d \log_2 d) \text{ operations}$$

• The FWHT requires zero floating-point multiplications—it consists exclusively of additions and subtractions.
• In GPU execution, the FWHT is fused directly into the preceding RMSNorm or LayerNorm kernel in on-chip SRAM.
• The activation rotation adds less than $2%$ latency to the normalization layer while enabling $200%$ faster downstream INT4 GEMMs!

Deep Dive: The End-to-End QuaRot Layer Lifecycle

To achieve complete 4-bit inference across an entire transformer, QuaRot decomposes the model into four distinct stages:

1. Offline Weight Transformation and GPTQ Rounding

Because the orthogonal matrix $H$ is deterministic and known in advance, weights are rotated and quantized entirely offline:

$$\tilde{W} = W H^T$$

1. $\tilde{W}$ is computed once during model compilation.
2. Rather than using naive round-to-nearest (RTN), QuaRot applies GPTQ (Generalized Post-Training Quantization) to $\tilde{W}$.
3. Because the Hadamard rotation disperses ill-conditioned Hessian curvature across all dimensions, GPTQ converges with significantly lower reconstruction error than on raw weights.
4. The resulting 4-bit weights $\tilde{W}_{\text{INT4}}$ are packed and saved to disk. During runtime, they require zero rotation overhead.

2. Online Activation Rotation Fused into RMSNorm

During inference, incoming activations from the previous transformer layer pass through RMSNorm:

$$\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d} \sum x_i^2 + \epsilon}} \odot \gamma$$

QuaRot fuses the online rotation directly into this kernel:

1. RMSNorm normalizes and scales the activation vector in GPU registers.
2. The thread block executes a register-level Fast Walsh-Hadamard Transform (FWHT).
3. The rotated activations are quantized to 4-bit integers on the fly using a per-token symmetric scale factor:

$$S_x = \frac{\max(|\tilde{x}|)}{7.0}, \quad \tilde{x}_{\text{INT4}} = \text{clip}\left(\left\lfloor \frac{\tilde{x}}{S_x} \right\rceil, -8, 7\right)$$

4. The INT4 tensor is written to GPU shared memory ready for matrix multiplication.

3. Pure INT4 Tensor Core GEMM Execution

With both weights $\tilde{W}{\text{INT4}}$ and activations $\tilde{x}{\text{INT4}}$ in 4-bit integer representation, QuaRot launches hardware-native INT4 GEMM kernels:

$$Y_{\text{INT32}} = \text{MatMul}{\text{INT4}}\left(\tilde{x}{\text{INT4}}, \tilde{W}_{\text{INT4}}\right)$$

$$Y = S_x \cdot S_w \cdot Y_{\text{INT32}}$$

• Arithmetic executes on NVIDIA INT4 Tensor Cores at double the instruction rate of FP16.
• The GPU never dequantizes weights to 16-bit in global memory or registers, slashing memory bandwidth pressure by $75%$.

4. 4-Bit Rotated Key-Value (KV) Cache Compression

In self-attention, Keys and Values also suffer from activation outliers, preventing naive 4-bit KV caching.

QuaRot applies a head-level Hadamard rotation $H_{\text{head}} \in \mathbb{R}^{d_{\text{head}} \times d_{\text{head}}}$ to Key and Value projection outputs:

$$\tilde{K} = K H_{\text{head}}^T, \quad \tilde{V} = V H_{\text{head}}^T$$

Because attention scores are invariant under orthogonal rotation:

$$\text{Score} = Q K^T = Q (H_{\text{head}}^T H_{\text{head}}) K^T = (Q H_{\text{head}}^T) (K H_{\text{head}}^T)^T = \tilde{Q} \tilde{K}^T$$

• Both Keys and Values are stored in High-Bandwidth Memory in 4-bit INT4 precision.
• In an 80-layer 70B model with a 128k context window, this drops KV cache memory from $42\text{ GB}$ down to just $10.5\text{ GB}$, allowing a single server to handle $4\times$ more concurrent requests.

End-to-End Implementation: Building a QuaRot W4A4 Layer in PyTorch

Below is a complete, executable PyTorch implementation demonstrating the Walsh-Hadamard transform, offline weight rotation, dynamic INT4 activation quantization, and output invariance verification.

Production Benchmarks: Perplexity, Memory, and Hardware Speedups

Empirical benchmarks published across open foundation models (LLaMA-2 7B/13B/70B and LLaMA-3 8B) validate that QuaRot breaks the sub-8-bit barrier, making 4-bit weights, activations, and KV caches viable in production.

1. Downstream Perplexity on WikiText-2 (Lower is Better)

Under identical test settings, QuaRot matches or approaches full FP16 precision where prior W4A4 techniques completely collapse:

Quantization Method	Precision Format	LLaMA-2 7B	LLaMA-2 13B	LLaMA-2 70B	LLaMA-3 8B
Unquantized Baseline	FP16 (16-bit)	5.47	4.88	3.32	6.14
AWQ / GPTQ	W4A16 (Weight only)	5.62	4.98	3.44	6.38
SmoothQuant	W8A8 (8-bit)	5.54	4.92	3.37	6.22
Naive Round-to-Nearest	W4A4 (No rotation)	$>1000$ (OOM/NaN)	$>1000$	$>1000$	$>1000$
QuaRot (Full INT4)	W4A4 + KV4	5.90	5.16	3.62	6.65

• On LLaMA-2 70B, QuaRot achieves a WikiText-2 perplexity of $3.62$, closely trailing the unquantized FP16 baseline ($3.32$) while compressing the entire model, activations, and KV cache into 4-bit integers.
• On MMLU benchmarks, QuaRot retains over $96.5%$ of 16-bit reasoning accuracy, proving that Hadamard rotation preserves structural attention dynamics.

2. Decode Latency and KV Cache VRAM Reductions

By unlocking native INT4 Tensor Cores and compressing KV caches, QuaRot transforms real-world serving throughput:

Strategic Recommendations: Deploying W4A4 QuaRot in 2026

For AI infrastructure leads, inference engineers, and developers optimizing local foundation models for high-speed extraction workflows in tools like AiFetchly, deploying QuaRot delivers immediate operational efficiencies:

1. Leverage CUTLASS / FlashInfer INT4 GEMM Kernels: To realize real-world speedups from W4A4, do not simulate INT4 in PyTorch. Deploy QuaRot weights using native INT4 Tensor Core kernels (such as CUTLASS grouped INT4 GEMMs or FlashInfer INT4 paged attention).
2. Fuse FWHT into Normalization Kernels: Always fuse the Fast Walsh-Hadamard Transform directly into the preceding RMSNorm kernel. Executing FWHT in GPU shared memory prevents roundtrips to global HBM, ensuring online activation rotation adds less than $2%$ latency overhead.
3. Prioritize 4-Bit Key-Value Caching for Long-Context Agents: If your hardware does not support INT4 activation Tensor Cores (e.g., older GPUs), you can still deploy W4A16 with KV4 (4-bit KV Cache). Applying head-level Hadamard rotations to Keys and Values slashes KV memory by $75%$ without modifying attention weights.
4. Combine with GPTQ Second-Order Weight Hessian Updates: When preparing weights offline, apply GPTQ rounding to the rotated weight matrices $\tilde{W} = W H^T$. The rotation disperses ill-conditioned Hessian curvature, enabling GPTQ to achieve optimal quantization bounds.