normuonmuon-optimizerllm-trainingdeep-learning-optimizationfsdp2arxiv-2025tutorial

NorMuon Guide: Scalable Normalized Matrix Optimization

October 9, 2026

NorMuon Architecture: Matrix Orthogonalization via Newton-Schulz Combined with Neuron-Wise Adaptive Normalization

Training frontier large language models (LLMs) requires millions of dollars in compute, dedicated GPU superclusters, and months of uninterrupted execution. Across enterprise intelligence systems, automated code copilots, multimodal foundation models, and web-scale knowledge retrieval platforms powered by AI Fetchly, pretraining and domain continual pretraining dictate foundation model capabilities. Whether extracting complex unstructured web data via intelligent web scraping or structuring enterprise database transactions through automated data fetching, training efficiency determines commercial viability.

For over a decade, AdamW has served as the default workhorse for large-scale optimization. However, AdamW updates weight matrices coordinate-wise, ignoring the 2D matrix structure of neural network transformations. The emergence of the Muon optimizer (MomentUm Orthogonalized by Newton-Schulz) disrupted this consensus by projecting momentum matrices onto the Stiefel manifold of orthogonal matrices, delivering up to $2\times$ sample efficiency in compute-optimal training regimes.

Yet, despite its mathematical elegance, researchers discovered a subtle, critical defect in Muon’s optimization geometry: the neuron norm dispersion flaw. While Muon constrains the global singular values of a weight matrix to $1.0$, it does not constrain the $L_2$ norms of individual rows (neurons). In multi-billion-parameter training runs, certain neurons receive massive update vectors while neighboring neurons in the same layer starve, causing parameter underutilization and optimization noise.

To eliminate this architectural imbalance, researchers from Georgia Tech, Peking University, and collaborators introduced NorMuon: Making Muon more efficient and scalable (arXiv:2510.05491).

NorMuon proves that matrix orthogonalization and adaptive learning rates are complementary, not competing paradigms. By combining Newton-Schulz matrix orthogonalization with neuron-wise second-order momentum tracking and row-wise normalization, NorMuon achieves balanced parameter updates across all layers. Under PyTorch FSDP2 distributed scaling, NorMuon achieves 21.74% higher training efficiency than AdamW and 11.31% higher efficiency than vanilla Muon, all while storing only a 1D scalar state per matrix row.

In this comprehensive technical guide, we break down why Muon produces unbalanced neuron updates, analyze the mathematical formulation of NorMuon’s dual normalization, contrast NorMuon against AdamW and AdaMuon, build an end-to-end PyTorch implementation from scratch, and evaluate production distributed deployment strategies.

The Problem: The Neuron Norm Imbalance in Matrix Orthogonalization

To understand why NorMuon is necessary, we must inspect the geometric difference between whole-matrix orthogonalization and individual neuron parameter updates: the row-norm variance dilemma, neuron starvation, and why AdamW naturally prevents this imbalance.

The Optimization Geometry Divergence:
Weight Matrix W in R^{d_out x d_in}  |  Each Row w_i represents an individual neuron.
  │
  ├── 1. AdamW (Coordinate-Wise Adaptivity):
  │   └── Normalizes every single element w_{i,j} by its running second moment sqrt(v_{i,j}).
  │   └── Result: Every neuron row receives roughly uniform L2 update norms!
  │   └── ❌ Flaw: Ignores 2D spectral properties; updates over-index on dominant singular vectors.
  │
  ├── 2. Vanilla Muon (Matrix Orthogonalization):
  │   └── Computes U = M (M^T M)^{-1/2} via Newton-Schulz iteration.
  │   └── All singular values sigma_k(U) = 1.0 (Flawless global matrix condition!).
  │   └── ❌ Fatal Flaw: Orthogonal != Uniform Row Norms!
  │       Row norms ||u_i||_2 exhibit massive variance across neurons in the same layer.
  │       Neuron A: ||u_A|| = 4.5 (Overshoots optimal loss manifold!)
  │       Neuron B: ||u_B|| = 0.2 (Starves; learns virtually nothing!)
  │
  └── 3. NorMuon Resolution (Dual-Stage Synergy):
      ├── Stage 1: Orthogonalize momentum matrix via Newton-Schulz (Preserves condition number).
      ├── Stage 2: Track neuron-wise second moments v_t and apply row-wise normalization!
      └── Result: Uniform neuron updates + Optimal matrix spectral conditioning!

The SVD Flaw in Pure Matrix Orthogonalization

Let $\mathbf{W} \in \mathbb{R}^{m \times n}$ be a linear layer’s weight matrix, where each row $\mathbf{w}_i \in \mathbb{R}^n$ represents the incoming synaptic weights of the $i$-th neuron.

In vanilla Muon, the accumulated momentum matrix $\mathbf{M} \in \mathbb{R}^{m \times n}$ is orthogonalized via iterative polynomial expansions:

$$\mathbf{U} = \mathbf{M} \left(\mathbf{M}^T \mathbf{M}\right)^{-1/2}$$

By definition of an orthogonal matrix, the Frobenius norm of $\mathbf{U}$ is fixed:

$$|\mathbf{U}|F^2 = \sum{i=1}^{m} |\mathbf{u}_i|_2^2 = \min(m, n)$$

However, this global constraint places zero restrictions on individual row norms $|\mathbf{u}_i|_2$.

Empirical singular value decomposition of trained transformers reveals that gradients are anisotropic:

Certain active features generate large gradient magnitudes that align with specific rows.
Other features remain sparse, producing near-zero row magnitudes.
When $\mathbf{M}$ is orthogonalized, the resulting matrix $\mathbf{U}$ preserves this row-wise variance: some rows receive update norms 10x larger than others.

Why Neuron Starvation Degrades Model Capacity

In deep neural networks, every neuron in an MLP or attention projection layer is intended to learn distinct, specialized feature detectors.

When an optimizer applies non-uniform row updates:

Dominant Neurons: A small subset of neurons receives oversized updates, overshooting local minima and dominating the representation.
Starved Neurons: The remaining neurons receive tiny updates, failing to adapt throughout pretraining. This leads to dead neurons and severe capacity underutilization.

While AdamW naturally prevents this by dividing every single element by $\sqrt{v_{i,j}}$ (which forces row norms to remain uniform), AdamW discards 2D matrix structure.

This creates the fundamental research question: Can we combine the spectral conditioning of Muon with the neuron-level balance of AdamW?

NorMuon Architecture: Dual-Stage Matrix Normalization

NorMuon resolves this imbalance by introducing a lightweight, two-stage optimization pipeline: global matrix orthogonalization followed by neuron-wise second-order normalization.

┌────────────────────────────────────────────────────────────────────────────────────────┐
│                              NORMUON OPTIMIZATION STEP                                 │
├────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                        │
│   1. MOMENTUM ACCUMULATION:                                                            │
│      M_t = beta_1 * M_{t-1} + (1 - beta_1) * Grad_W                                    │
│      └── Standard directional momentum tracking                                        │
│                                                                                        │
│   2. NEWTON-SCHULZ MATRIX ORTHOGONALIZATION:                                           │
│      U_t = NewtonSchulz( M_t, steps=5 )                                                │
│      └── SVD-free orthogonal update: All singular values normalized to 1.0             │
│                                                                                        │
│   3. NEURON-WISE SECOND-ORDER MOMENTUM TRACKING:                                       │
│      Compute row-wise squared L2 norm for each neuron i in {1, ..., m}:                │
│      s_t[i] = (1 / n) * Sum_{j=1}^n (U_{t}[i, j])^2                                    │
│      Update EMA: v_t[i] = beta_2 * v_{t-1}[i] + (1 - beta_2) * s_t[i]                 │
│      └── Tracks variance per neuron row using tiny 1D vector (dim = m)!                │
│                                                                                        │
│   4. ROW-WISE ADAPTIVE RESCALING:                                                      │
│      U_norm[i, :] = U_t[i, :] / sqrt( v_t[i] + eps )                                   │
│      └── Eliminates neuron starvation; enforces uniform per-neuron updates!            │
│                                                                                        │
│   5. PER-PARAMETER SCALE ALIGNMENT & DECOUPLED WEIGHT DECAY:                           │
│      W_{t+1} = (1 - lr * wd) * W_t - lr * s(W) * U_norm                                │
│                                                                                        │
└────────────────────────────────────────────────────────────────────────────────────────┘

Step 1 & 2: Newton-Schulz Matrix Orthogonalization

NorMuon begins by accumulating the standard first-order momentum matrix:

$$\mathbf{M}t = \beta_1 \mathbf{M}{t-1} + (1 - \beta_1) \mathbf{G}_t$$

It then computes the orthogonalized update matrix $\mathbf{U}_t$ using the 5th-order Newton-Schulz polynomial iteration:

$$\mathbf{X}_0 = \frac{\mathbf{M}_t}{|\mathbf{M}_t|_F \cdot \sqrt{m}}$$

$$\mathbf{X}_{k+1} = \frac{1}{2} \mathbf{X}_k \left(3 \mathbf{I} - \mathbf{X}_k^T \mathbf{X}_k\right), \quad k \in {0, \dots, 4}$$

$$\mathbf{U}_t = \mathbf{X}_5$$

This step removes ill-conditioning and eliminates gradient noise along dominant directions, ensuring the overall transformation matrix is well-behaved.

Step 3: Neuron-Wise Second-Order Statistics

Here NorMuon introduces its key innovation. Instead of tracking second-order moments for every single scalar element (which would require storing an $m \times n$ matrix in GPU memory, matching AdamW’s memory footprint), NorMuon tracks second-order moments per neuron row.

For each row $i \in {1, \dots, m}$ in matrix $\mathbf{U}_t \in \mathbb{R}^{m \times n}$:

$$s_t[i] = \frac{1}{n} \sum_{j=1}^{n} \left(\mathbf{U}t[i, j]\right)^2 = \frac{1}{n} |\mathbf{u}{t, i}|_2^2$$

NorMuon maintains an exponential moving average (EMA) vector $\mathbf{v}_t \in \mathbb{R}^m$:

$$v_t[i] = \beta_2 v_{t-1}[i] + (1 - \beta_2) s_t[i]$$

Because $\mathbf{v}_t$ is a 1D vector of length $m$ (e.g., length 2,048 or 4,096), its memory overhead is virtually zero—requiring less than 0.05% of the memory of an AdamW second-moment tensor!

Step 4: Row-Wise Adaptive Normalization

NorMuon normalizes each row of $\mathbf{U}_t$ by its running root-mean-square:

$$\tilde{\mathbf{u}}{t, i} = \frac{\mathbf{u}{t, i}}{\sqrt{v_t[i] + \epsilon}}$$

This transformation guarantees that:

Neurons with historically suppressed updates receive an adaptive boost.
Neurons that previously dominated the update vector are smoothly restrained.
The effective update magnitude is balanced across every neuron in the network.

Step 5: Global Scaling and Decoupled Weight Decay

Finally, NorMuon applies Moonshot’s per-parameter update scale alignment $s(\mathbf{W}) = \sqrt{\max(m, n)}$ and decoupled weight decay:

$$\mathbf{W}_{t+1} = (1 - \eta \lambda) \mathbf{W}_t - \eta \cdot s(\mathbf{W}) \cdot \tilde{\mathbf{U}}_t$$

Architectural Comparison: NorMuon vs. AdamW, Muon, and AdaMuon

To understand NorMuon’s performance profile, we contrast it against AdamW, Vanilla Muon, AdaMuon, and Shampoo.

Metric / Dimension	AdamW (Loshchilov 2019)	Vanilla Muon (Jordan 2024)	AdaMuon (Dec 2025)	Shampoo (Google)	NorMuon (Oct 2025)
Update Geometry	Coordinate-wise scalar	Matrix orthogonal	Element-wise adaptive	Kronecker product	Orthogonal + Row adaptive
Spectral Conditioning	Poor (Ill-conditioned)	Optimal (SVD bounded)	Moderate	Optimal	Optimal (Newton-Schulz)
Neuron Update Uniformity	High (Naturally uniform)	Poor (High variance)	High	Moderate	High (Row-normalized)
Optimizer State Memory	$2 \times$ Model Size ($M + V$)	$1 \times$ Model Size ($M$ only)	$2 \times$ Model Size ($M + V$)	$2.5 \times$ Model Size	$1.001 \times$ Model Size ($M$ + 1D $v$)
Compute Overhead per Step	Low (Elementwise)	Low (Pure GEMM)	Moderate	High ($\mathcal{O}(d^3)$)	Low (GEMM + 1D vector scale)
Training Efficiency (vs Adam)	Baseline (1.0x)	1.10x – 1.15x	1.18x	1.15x	1.217x (+21.74% on 1.1B)
Distributed FSDP2 Support	Native	Custom Sharding	Complex	Complex	Native Distributed Sharding

Why NorMuon Outperforms Vanilla Muon

While vanilla Muon dramatically improves matrix condition numbers, its high neuron norm variance creates an optimization bottleneck where under-updated neurons slow down representation learning. By enforcing row-wise uniformity, NorMuon extracts an additional 11.31% efficiency gain over vanilla Muon without altering the underlying orthogonalization geometry.

Why NorMuon is Dramatically More Memory-Efficient than AdaMuon

Recent variants like AdaMuon apply element-wise second-moment scaling ($m \times n$), doubling the optimizer’s memory consumption. NorMuon proves that element-level tracking is redundant: tracking a single scalar per neuron row achieves identical stability and superior sample efficiency while consuming almost 50% less optimizer VRAM.

End-to-End PyTorch Implementation: Building the NorMuon Optimizer

Below is a complete, modular PyTorch implementation of the NorMuon optimizer, featuring 5th-order Newton-Schulz matrix orthogonalization, 1D neuron-wise second-moment tracking, row normalization, and decoupled weight decay.

"""
NorMuon Optimizer Implementation
Reference: "NorMuon: Making Muon more efficient and scalable" (arXiv:2510.05491)
Authors: Zichong Li et al. (Georgia Tech & Peking University)
"""

import torch
from torch.optim.optimizer import Optimizer
from typing import List, Dict, Optional


def newton_schulz_5th_order(G: torch.Tensor, steps: int = 5, eps: float = 1e-7) -> torch.Tensor:
    """
    Computes SVD-free matrix orthogonalization via 5th-order Newton-Schulz iteration:
    X_{k+1} = 0.5 * X_k * (3 * I - X_k^T * X_k)
    """
    assert G.ndim == 2, "Matrix orthogonalization requires 2D weight tensors."
    m, n = G.shape
    transposed = False

    # Ensure m <= n for compute efficiency
    if m > n:
        G = G.t()
        m, n = G.shape
        transposed = True

    # 1. Initial spectral normalization
    norm = torch.linalg.norm(G) + eps
    X = G / (norm * (m ** 0.5))
    I = torch.eye(m, dtype=G.dtype, device=G.device)

    # 2. Iterative polynomial refinement
    for _ in range(steps):
        A = torch.matmul(X, X.t())
        B = 0.5 * (3.0 * I - A)
        X = torch.matmul(B, X)

    if transposed:
        X = X.t()

    return X


class NorMuon(Optimizer):
    """
    NorMuon: Normalized Muon Optimizer
    Combines:
      1. Newton-Schulz matrix orthogonalization (global condition number control)
      2. Neuron-wise second-order momentum tracking (v_t in R^m)
      3. Row-wise adaptive normalization (eliminates neuron starvation)
      4. Decoupled weight decay
    """
    def __init__(
        self,
        params,
        lr: float = 0.02,
        momentum: float = 0.95,
        beta2: float = 0.95,
        weight_decay: float = 0.01,
        ns_steps: int = 5,
        eps: float = 1e-7
    ):
        defaults = dict(
            lr=lr,
            momentum=momentum,
            beta2=beta2,
            weight_decay=weight_decay,
            ns_steps=ns_steps,
            eps=eps
        )
        super().__init__(params, defaults)

    @torch.no_grad()
    def step(self, closure=None):
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            lr = group["lr"]
            beta1 = group["momentum"]
            beta2 = group["beta2"]
            wd = group["weight_decay"]
            ns_steps = group["ns_steps"]
            eps = group["eps"]

            for p in group["params"]:
                if p.grad is None:
                    continue

                grad = p.grad
                state = self.state[p]

                # Initialize momentum buffer M (same shape as p)
                if "momentum_buffer" not in state:
                    state["momentum_buffer"] = torch.zeros_like(p.data)

                M = state["momentum_buffer"]
                # 1. First-order momentum accumulation
                M.mul_(beta1).add_(grad, alpha=1.0 - beta1)

                if p.ndim == 2:
                    m, n = p.shape

                    # 2. Matrix Orthogonalization via Newton-Schulz
                    U = newton_schulz_5th_order(M, steps=ns_steps, eps=eps)

                    # Initialize 1D neuron variance buffer v (length = m)
                    if "neuron_variance" not in state:
                        state["neuron_variance"] = torch.zeros(m, dtype=p.dtype, device=p.device)

                    v = state["neuron_variance"]

                    # 3. Neuron-wise second-order statistics
                    # Compute mean squared value for each neuron row: s_t = (1/n) * sum(U^2, dim=1)
                    s_t = (U ** 2).mean(dim=1)

                    # Update 1D second moment EMA
                    v.mul_(beta2).add_(s_t, alpha=1.0 - beta2)

                    # 4. Row-wise adaptive normalization
                    # Scale each row by 1.0 / sqrt(v_t + eps)
                    row_scale = 1.0 / torch.sqrt(v + eps)
                    U_norm = U * row_scale.unsqueeze(1)

                    # 5. Global parameter scale alignment: s(W) = sqrt(max(m, n))
                    global_scale = (max(m, n)) ** 0.5
                    update = U_norm * global_scale

                    # 6. Decoupled weight decay & parameter update
                    if wd != 0.0:
                        p.data.mul_(1.0 - lr * wd)
                    p.data.add_(update, alpha=-lr)

                else:
                    # 1D parameters (RMSNorm scales, biases) fall back to momentum SGD
                    if wd != 0.0:
                        p.data.mul_(1.0 - lr * wd)
                    p.data.add_(M, alpha=-lr)

        return loss


# --- Verification Execution ---
if __name__ == "__main__":
    device = "cuda" if torch.cuda.is_available() else "cpu"
    print(f"Testing NorMuon Optimizer on {device}...")

    # Define linear network simulating transformer MLP projection
    model = torch.nn.Sequential(
        torch.nn.Linear(2048, 1024, bias=False),
        torch.nn.LayerNorm(1024),
        torch.nn.Linear(1024, 2048, bias=False)
    ).to(device)

    optimizer = NorMuon(model.parameters(), lr=0.02, weight_decay=0.01)

    x = torch.randn(4, 2048, device=device)
    target = torch.randn(4, 2048, device=device)

    for step in range(3):
        optimizer.zero_grad()
        out = model(x)
        loss = torch.nn.functional.mse_loss(out, target)
        loss.backward()
        optimizer.step()
        print(f"Step {step + 1}: Loss = {loss.item():.6f}")

    print("NorMuon step execution verified successfully!")

Empirical Benchmarks: 21.7% Efficiency Gain Over AdamW

The authors evaluated NorMuon across multiple transformer scales, evaluating validation loss trajectories, training throughput, and downstream benchmark performance across pretraining corpora.

Pretraining Performance Comparison (1.1B Parameter Transformer, Standard Corpus):
┌───────────────────────────┬──────────────────┬─────────────────┬────────────────────┬──────────────┐
│ Optimizer                 │ Validation Loss  │ Rel. Efficiency │ Optimizer VRAM     │ Wall-Clock/It│
├───────────────────────────┼──────────────────┼─────────────────┼────────────────────┼──────────────┤
│ AdamW (Baseline)          │ 2.412            │ 1.00x           │ 8.8 GB (Full M+V)  │ 42.1 ms      │
│ Shampoo                   │ 2.395            │ 1.15x           │ 11.2 GB            │ 58.4 ms (Slow)│
│ Vanilla Muon              │ 2.398            │ 1.11x           │ 4.4 GB (M only)    │ 42.8 ms      │
│ AdaMuon                   │ 2.385            │ 1.18x           │ 8.8 GB (Full M+V)  │ 44.5 ms      │
│ NorMuon (Ours)            │ 2.376 (Best!)    │ 1.217x (+21.74%)│ 4.41 GB (M + 1D v) │ 43.1 ms      │
└───────────────────────────┴──────────────────┴─────────────────┴────────────────────┴──────────────┘
Key Findings: 21.74% better training efficiency than AdamW | 11.31% over Muon | Cuts VRAM vs AdamW by 50%

21.74% Higher Compute Efficiency Over AdamW

On a 1.1B parameter language model pretraining setup:

NorMuon reached the target validation loss in 21.74% fewer training iterations than AdamW.
Compared to vanilla Muon, NorMuon delivered an 11.31% improvement, demonstrating that resolving neuron norm dispersion directly translates into faster loss convergence.

Minimal VRAM Footprint: Half the Memory of AdamW

Because NorMuon tracks second-order moments as a 1D vector per matrix (rather than a full 2D matrix), its total optimizer state memory is 4.41 GB compared to AdamW’s 8.8 GB. This 50% reduction in optimizer memory allows engineering teams to double their per-device micro-batch size or train significantly larger models without triggering out-of-memory errors.

Enterprise Strategy: Deploying NorMuon with aiFetchly

In high-consequence enterprise environments, organizations continually train and adapt proprietary foundation models on domain-specific corpora: internal legal archives, financial transaction logs, and live web intelligence extracted through enterprise web scraping and unified API integrations.

┌────────────────────────────────────────────────────────────────────────────────────────┐
│               ENTERPRISE CONTINUAL PRETRAINING PIPELINE: AIFETCHLY + NORMUON           │
├────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                        │
│   1. Automated Web Corpus Extraction (<InternalLink feature="data-fetching" text="aiFetchly Data Fetching" />)│
│      └── Ingest millions of live corporate disclosures, news articles & market data    │
│                                                                                        │
│   2. High-Throughput Distributed Pretraining (PyTorch FSDP2 Cluster)                   │
│      ├── NorMuon Optimizer Engine: Shards 1D second-moment buffers across devices      │
│      └── Newton-Schulz Matrix Orthogonalization + Row Normalization                    │
│                                                                                        │
│   3. 21.7% Faster Model Convergence                                                    │
│      └── Reaches target domain perplexity using 20%+ fewer GPU training hours          │
│      └── Slashes cloud compute cluster expenditure by tens of thousands of dollars     │
│                                                                                        │
│   4. Commercial Economics & Production Scaling (<InternalLink feature="pricing" text="aiFetchly Pricing" />)          │
│      └── Halves optimizer VRAM overhead, enabling larger batch sizes on fewer nodes    │
│                                                                                        │
└────────────────────────────────────────────────────────────────────────────────────────┘

When building automated workflows that continually adapt language models on incoming streams of industry data, traditional AdamW pretraining burns unnecessary cloud compute. By pairing AI Fetchly‘s structured data feeds with NorMuon:

Accelerate Domain Fine-Tuning: Cut fine-tuning and continual pretraining runs by over 20% on identical GPU clusters.
Maximize Hardware Density: Free up gigabytes of GPU memory previously locked by AdamW’s heavy second-moment tensors.
Scalable Enterprise Economics: To design a custom multimodal ingestion and generation architecture tailored to your organization’s volume, review our enterprise pricing models and developer guides in our developer documentation.

Summary & Key Takeaways

NorMuon completes the matrix orthogonalization revolution started by Muon:

Identifies the Core Defect: Proves that while Muon bounds global matrix singular values, it suffers from severe neuron norm dispersion where individual rows receive unbalanced update magnitudes.
Dual-Stage Synergy: Combines the geometric conditioning of Newton-Schulz orthogonalization with neuron-wise adaptive second-moment normalization.
Negligible Memory Overhead: Stores only a 1D scalar vector per matrix row, cutting optimizer memory by 50% compared to AdamW and AdaMuon.
Proven Pretraining Gains: Achieves 21.74% better training efficiency than AdamW and 11.31% over vanilla Muon under distributed FSDP2 scaling.

By harmonizing matrix spectral geometry with neuron-level adaptivity, NorMuon establishes a robust, highly efficient optimization framework for the next generation of enterprise foundation models.