muon-optimizernormuondga-muonllm-pretrainingnewton-schulzmatrix-optimizationdeep-learning

DGA-Muon Guide: Geometry-Aligned Optimizer for LLMs

October 9, 2026

DGA-Muon Architecture: Decoupled Geometry-Aligned Adaptive Scaling for Muon Resolving NorMuon Degeneration

The optimization of multi-billion-parameter foundation models has undergone a quiet revolution over the past year. In frontier AI laboratories, training clusters, and distributed infrastructure pipelines powered by AI Fetchly, the traditional supremacy of coordinate-wise AdamW has been challenged by matrix-aware orthogonalized optimizers—most notably Muon (MomentUm Orthogonalized by Newton–Schulz).

Originally introduced in the Modded-NanoGPT speedrun and rapidly adopted across frontier foundation model families (including Kimi, GLM, Moonlight, and DeepSeek), Muon treats two-dimensional weight matrices as cohesive geometric operators rather than flattened one-dimensional vectors. By computing the matrix polar factor via iterative Newton–Schulz polynomials, Muon equalizes singular values, delivering dramatic wall-clock training acceleration over AdamW.

To introduce adaptive learning rates into Muon, researchers developed NorMuon (Neuron-wise Normalized Muon), which equips Muon with row-wise second-moment scaling. NorMuon demonstrated strong empirical speedrun records on Modded-NanoGPT and NanoChat. However, despite its widespread adoption, NorMuon suffered from an unexplained theoretical puzzle: its adaptive scaling factors were computed directly from the orthogonalized momentum rather than historical gradients.

In a groundbreaking theoretical and empirical analysis—DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon (arXiv:2610.06578)—researcher Wenpeng Zhang exposed the fatal flaw of this design: NorMuon’s adaptivity degenerates into a trivial global scalar on square projection matrices (such as standard Q, K, V, and O attention weights), while distorting the orthogonal column directions of tall matrices. To cure this failure mode, Zhang introduces DGA-Muon, an optimizer grounded in two principled design axioms: decoupling adaptivity from orthogonalization and aligning scaling dimensions with matrix shape.

The result is the first theoretically grounded adaptive Muon optimizer with formal non-convex convergence guarantees, outperforming NorMuon and vanilla Muon across large-scale pretraining benchmarks. In this comprehensive technical guide, we break down the Orthogonalization-Adaptivity Paradox, formalize DGA-Muon’s geometry-aligned update rule, construct an end-to-end PyTorch implementation, and evaluate empirical training dynamics.

The NorMuon Degeneration Trap: The Orthogonalization-Adaptivity Paradox

To understand why DGA-Muon is necessary, we must analyze the mathematical mechanics of vanilla Muon and NorMuon.

Review of Muon and the Polar Factor

For a 2D weight matrix $W \in \mathbb{R}^{m \times n}$, Muon maintains an Exponential Moving Average (EMA) momentum buffer $M_t = \beta_1 M_{t-1} + (1 - \beta_1) G_t$. Instead of updating weights along $M_t$, Muon extracts its polar factor $O_t = \text{Orth}(M_t) = U V^\top$ (where $M_t = U \Sigma V^\top$ is the singular value decomposition):

$$W_t = W_{t-1} - \eta \cdot \text{Orth}(M_t)$$

Because computing SVD on every training step is computationally prohibitive, Muon approximates $O_t$ using high-throughput Newton–Schulz iterations:

$$X_{k+1} = X_k \left(a I + b X_k^\top X_k + c (X_k^\top X_k)^2\right)$$

How NorMuon Attempted Adaptivity

To introduce Adam-style neuron-wise adaptivity, NorMuon tracks the second moment of the orthogonalized update $O_t$ rather than the raw gradient:

$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) \cdot \text{mean}_{\text{cols}}\left(O_t \odot O_t\right)$$

$$W_t = W_{t-1} - \eta_t \cdot \frac{O_t}{\sqrt{v_t} + \delta}$$

where the division is broadcast row-wise across the $m$ rows of $O_t$.

Theorem: The Collapse of NorMuon on Square & Wide Matrices

Zhang (arXiv:2610.06578) proves a striking theoretical result: for any matrix where $m \le n$ (such as square attention projections $Q, K, V, O$ where $m=n$), the exact polar factor satisfies $O_t O_t^\top = I_m$. Consequently, for every single row $i \in {1, \dots, m}$:

$$|O_{t, i, :}|2^2 = 1 \implies \text{mean}{\text{cols}}\left(O_t \odot O_t\right)_i = \frac{1}{n}$$

Substituting this into the second-moment recursion reveals:

$$v_t = \beta_2^t v_0 + (1 - \beta_2^t) \frac{1}{n} \mathbf{1}_m = (1 - \beta_2^t) \frac{1}{n} \mathbf{1}_m$$

The second-moment vector $v_t$ is strictly uniform across all rows! The resulting scaling vector $\frac{1}{\sqrt{v_t} + \delta}$ assigns an identical scalar multiplier to every single neuron.

The intended neuron-wise adaptivity completely collapses to a trivial global scalar. In practice, NorMuon’s apparent adaptivity on square matrices arose solely as an artifact of numerical truncation in low-order Newton–Schulz iterations! As orthogonalization improves, NorMuon’s adaptivity vanishes.

The Tall Matrix Distortion ($m > n$)

In tall matrices (such as the second layer of MLP down-projections where $m > n$), the polar factor has orthonormal columns ($O_t^\top O_t = I_n$), not rows. Applying rigid row-wise scaling $D_t = \text{diag}(1 / \sqrt{v_t})$ distorts the extracted orthogonal directions:

$$(D_t O_t)^\top (D_t O_t) = O_t^\top D_t^2 O_t \ne I_n$$

Row scaling reintroduces cross-coupling between columns, destroying the exact orthogonality that Muon was invented to preserve.

The Structural Breakdown of NorMuon:
┌────────────────────────────────────────────────────────────────────────┐
│ Square Matrices (m = n, e.g., Q, K, V, O Attention Projections):        │
│ Exact Orthogonalization: O_t O_t^T = I_m                               │
│ -> Row norms are CONSTANT: ||O_t[i, :]||^2 = 1.0                       │
│ -> Second moment v_t collapses to identical scalar across all rows!    │
│ ❌ Zero neuron-wise adaptivity; degrades to scalar Muon!                │
├────────────────────────────────────────────────────────────────────────┤
│ Tall Matrices (m > n, e.g., MLP Down-Projections):                     │
│ Exact Orthogonalization: O_t^T O_t = I_n (Column Orthonormal)          │
│ -> Rigid row-wise scaling distorts column orthogonality!               │
│ ❌ Destroys the geometric structure of the polar factor!               │
└────────────────────────────────────────────────────────────────────────┘

When training enterprise foundation models that orchestrate complex tool invocations via data fetching APIs, optimizer instability and degenerate scaling degrade pretraining efficiency. A mathematically consistent matrix optimizer is essential.

DGA-Muon: Two Principled Design Axioms

To overcome the limitations of NorMuon, DGA-Muon establishes two foundational design principles:

DGA-Muon Design Axioms:
┌────────────────────────────────────────────────────────────────────────┐
│ AXIOM 1: DECOUPLE ADAPTIVITY FROM ORTHOGONALIZATION                    │
│ Compute second moments directly from raw gradients G_t, NOT from O_t!  │
│ -> Restores genuine optimization-driven adaptivity.                    │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ AXIOM 2: GEOMETRY-ALIGNED DIMENSIONAL SCALING                          │
│ • Wide Matrices (m < n): Apply ROW-WISE scaling (matches O_t O_t^T)   │
│ • Tall Matrices (m > n): Apply COLUMN-WISE scaling (matches O_t^T O_t) │
│ • Square Matrices (m = n): Either orientation is valid                │
│ -> Strictly preserves polar factor orthogonality across all regimes!   │
└────────────────────────────────────────────────────────────────────────┘

Axiom 1: Decoupled Gradient Adaptivity

DGA-Muon decouples the extraction of the orthogonal update direction from the extraction of adaptive scale factors. While the update direction $O_t = \text{Orth}(M_t)$ is extracted via Newton–Schulz iterations over momentum, the adaptive second moment $v_t$ is accumulated directly from raw gradients $G_t$:

$$\text{Wide Matrices } (m \le n): \quad v_t = \beta_2 v_{t-1} + (1 - \beta_2) \cdot \text{sum}_{\text{cols}}\left(G_t \odot G_t\right) \in \mathbb{R}^m$$

$$\text{Tall Matrices } (m > n): \quad v_t = \beta_2 v_{t-1} + (1 - \beta_2) \cdot \text{sum}_{\text{rows}}\left(G_t \odot G_t\right) \in \mathbb{R}^n$$

By accumulating raw gradient energies, $v_t$ reflects true optimization variance and curvature rather than being masked by the unitary geometry of $O_t$.

Axiom 2: Shape-Dependent Geometry-Aligned Scaling

To prevent the dimensional mismatch that destroys tall-matrix orthogonality, DGA-Muon applies scaling factors along the non-orthogonal dimension of the polar factor:

When $m \le n$ (Wide or Square): The polar factor $O_t$ has orthonormal rows ($O_t O_t^\top = I_m$). DGA-Muon applies row-wise scaling:

$$U_t = \text{diag}\left(\frac{1}{\sqrt{\hat{v}_t} + \delta}\right) O_t$$

When $m > n$ (Tall): The polar factor $O_t$ has orthonormal columns ($O_t^\top O_t = I_n$). DGA-Muon applies column-wise scaling:

$$U_t = O_t \cdot \text{diag}\left(\frac{1}{\sqrt{\hat{v}_t} + \delta}\right)$$

Zhang provides a rigorous mathematical proof: under column-wise scaling on tall matrices, the column directions remain orthogonal! The scaling merely rescales the magnitude of each orthogonal basis vector without introducing off-diagonal coupling.

Bias Correction and Adaptive Clipping

DGA-Muon incorporates standard Adam-style bias correction $\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$, ensuring unbiased second-moment estimates during early training iterations. Additionally, an adaptive clipping threshold bounds the maximum learning rate amplification, preventing training instability in deep layers.

Architectural Comparison: DGA-Muon vs. Muon and NorMuon

The table below contrasts DGA-Muon against vanilla Muon and NorMuon across mathematical and systems properties:

Optimizer Property	Vanilla Muon (Jordan, 2024)	NorMuon (Li et al., 2026)	DGA-Muon (Zhang, 2026)
Direction Extraction	Polar Factor $\text{Orth}(M_t)$	Polar Factor $\text{Orth}(M_t)$	Polar Factor $\text{Orth}(M_t)$
Adaptivity Source	None (Global Scalar)	Coupled to Orthogonalized $O_t$	Decoupled Raw Gradients $G_t$
Square Matrix Adaptivity	No (Uniform Scalar)	Collapses to Uniform Scalar!	Active Neuron-Wise Scaling
Tall Matrix Geometry	Preserves Orthonormal Cols	Distorts Orthonormal Columns	Geometry-Aligned Column Scaling
Theoretical Convergence	Heuristic Non-Convex	Unproven / Open Problem	Formal Non-Convex Proof Provided
Pretraining Performance	Fast Baseline	High on MLP, Flat on Attention	Superior Across All Layers

By repairing the theoretical foundation of matrix adaptivity, DGA-Muon ensures that attention matrices receive the same fine-grained optimization benefits as feed-forward layers.

End-to-End Implementation: Building the DGA-Muon Optimizer

Let us construct a clean, production-ready PyTorch implementation of the complete DGA-Muon optimizer, featuring high-order Newton–Schulz orthogonalization and geometry-aligned adaptive scaling.

import torch
from torch.optim.optimizer import Optimizer
from typing import List, Optional

def zeropower_via_newtonschulz5(G: torch.Tensor, steps: int = 5, eps: float = 1e-7) -> torch.Tensor:
    """
    Computes the polar factor Orth(G) = U V^T using quintic Newton-Schulz iterations.
    Equalizes singular values to 1 while preserving matrix directional geometry.
    """
    assert G.ndim >= 2
    a, b, c = (3.4445, -4.7750, 2.0315)
    X = G.bfloat16() if G.dtype == torch.bfloat16 else G.float()

    # Normalize spectral norm to ensure convergence inside unit disk
    norm = X.norm().clamp(min=eps)
    X = X / norm

    # Ensure wide orientation for iteration efficiency
    transposed = False
    if X.size(0) > X.size(1):
        X = X.t()
        transposed = True

    for _ in range(steps):
        A = X @ X.t()
        B = b * A + c * (A @ A)
        X = a * X + B @ X

    if transposed:
        X = X.t()

    return X.type_as(G)

class DGAMuon(Optimizer):
    """
    Decoupled Geometry-Aligned Muon (DGA-Muon) Optimizer.
    Combines Newton-Schulz momentum orthogonalization with decoupled raw-gradient
    second moments and shape-aligned scaling (row-wise for wide, column-wise for tall).
    """
    def __init__(
        self,
        params,
        lr: float = 0.02,
        momentum: float = 0.95,
        beta2: float = 0.99,
        delta: float = 1e-6,
        weight_decay: float = 0.01,
        ns_steps: int = 5
    ):
        defaults = dict(lr=lr, momentum=momentum, beta2=beta2, delta=delta,
                        weight_decay=weight_decay, ns_steps=ns_steps)
        super().__init__(params, defaults)

    @torch.no_grad()
    def step(self, closure=None):
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            lr = group['lr']
            beta1 = group['momentum']
            beta2 = group['beta2']
            delta = group['delta']
            wd = group['weight_decay']
            ns_steps = group['ns_steps']

            for p in group['params']:
                if p.grad is None:
                    continue

                grad = p.grad
                state = self.state[p]

                # Initialize state buffers
                if len(state) == 0:
                    state['step'] = 0
                    state['momentum'] = torch.zeros_like(p)
                    m, n = p.shape
                    # Geometry-aligned second moment allocation
                    if m <= n:
                        state['v'] = torch.zeros(m, 1, device=p.device, dtype=torch.float32)
                    else:
                        state['v'] = torch.zeros(1, n, device=p.device, dtype=torch.float32)

                state['step'] += 1
                step = state['step']
                M = state['momentum']
                v = state['v']
                m, n = p.shape

                # 1. Update Momentum: M_t = beta1 * M_{t-1} + (1 - beta1) * G_t
                M.mul_(beta1).add_(grad, alpha=1.0 - beta1)

                # 2. Extract Polar Factor via Newton-Schulz
                O = zeropower_via_newtonschulz5(M, steps=ns_steps)

                # 3. Decoupled Adaptivity: Accumulate second moments directly from raw grad G_t
                grad_f = grad.float()
                if m <= n:
                    # Wide matrix: sum across columns -> [m, 1]
                    g_sq = (grad_f * grad_f).sum(dim=1, keepdim=True)
                else:
                    # Tall matrix: sum across rows -> [1, n]
                    g_sq = (grad_f * grad_f).sum(dim=0, keepdim=True)

                v.mul_(beta2).add_(g_sq, alpha=1.0 - beta2)

                # Bias correction
                bias_correction2 = 1.0 - beta2 ** step
                v_hat = v / bias_correction2
                scale = 1.0 / (torch.sqrt(v_hat) + delta)

                # 4. Geometry-Aligned Scaling Execution
                if m <= n:
                    # Row-wise scaling
                    update = O * scale.type_as(O)
                else:
                    # Column-wise scaling
                    update = O * scale.type_as(O)

                # 5. Apply decoupled weight decay and parameter update
                if wd > 0:
                    p.mul_(1.0 - lr * wd)
                p.add_(update, alpha=-lr)

        return loss

Functional Validation Harness

Let us test our DGA-Muon optimizer on wide, square, and tall weight tensors to confirm numerical stability:

def verify_dga_muon_execution():
    torch.manual_seed(42)
    # Test parameters: wide (64x128), square (128x128), and tall (128x64)
    w_wide = torch.randn(64, 128, requires_grad=True)
    w_square = torch.randn(128, 128, requires_grad=True)
    w_tall = torch.randn(128, 64, requires_grad=True)

    optimizer = DGAMuon([w_wide, w_square, w_tall], lr=0.01, momentum=0.9, beta2=0.99)

    print('Simulating optimization steps across diverse matrix geometries...')
    for step in range(10):
        optimizer.zero_grad()
        loss = (w_wide.sum() ** 2) + (w_square.sum() ** 2) + (w_tall.sum() ** 2)
        loss.backward()
        optimizer.step()

    # Verify states
    v_wide = optimizer.state[w_wide]['v']
    v_tall = optimizer.state[w_tall]['v']
    v_sq = optimizer.state[w_square]['v']

    print(f"Wide matrix v shape: {v_wide.shape} (Row-aligned [64, 1])")
    print(f"Tall matrix v shape: {v_tall.shape} (Column-aligned [1, 64])")
    print(f"Square matrix v shape: {v_sq.shape} (Row-aligned [128, 1])")
    assert v_wide.shape == (64, 1)
    assert v_tall.shape == (1, 64)
    assert v_sq.shape == (128, 1)
    print('Verification Succeeded: DGA-Muon correctly allocates shape-aligned adaptivity!')

if __name__ == '__main__':
    verify_dga_muon_execution()

When managing distributed model training clusters that integrate with custom backend endpoints, deploying an optimizer with provable non-convex convergence prevents training blowups and ensures predictable compute schedules.

Production Benchmarks & Pretraining Telemetry

In empirical evaluations conducted by Zhang (arXiv:2610.06578), DGA-Muon was benchmarked across NanoChat and Modded-NanoGPT speedruns.

NanoChat Pretraining Loss & Validation

The table below reports pretraining validation loss and training tokens required to reach target perplexity:

Optimizer	Update Direction	Adaptivity Mechanism	Final Validation Loss	Training Steps to Threshold	Relative Convergence Speedup
AdamW	Flattened Vector	Element-Wise Gradient $G_t$	3.284	4,800 steps	1.00x (Baseline)
Muon (Vanilla)	Polar Factor $\text{Orth}(M_t)$	None (Fixed Scalar)	3.210	3,450 steps	1.39x
NorMuon	Polar Factor $\text{Orth}(M_t)$	Coupled Orthogonal $O_t$	3.178	3,120 steps	1.54x
DGA-Muon (2026)	Polar Factor $\text{Orth}(M_t)$	Decoupled Geometry-Aligned	3.142 (Best!)	2,780 steps	1.73x (vs. AdamW)

Pretraining Steps to Target Perplexity (Lower is Better):
AdamW:          [████████████████████] 4,800 steps (Baseline)
Vanilla Muon:   [██████████████      ] 3,450 steps (1.39x faster)
NorMuon:        [█████████████       ] 3,120 steps (1.54x faster)
DGA-Muon:       [███████████         ] 2,780 steps (1.73x faster SOTA!)

By restoring active neuron-wise adaptivity to attention projection layers and preserving tall-matrix column orthogonality, DGA-Muon achieves 11% faster convergence than NorMuon and 23% faster convergence than vanilla Muon.

Organizations evaluating infrastructure costs and GPU cluster reservations can explore flexible pricing structures to calculate ROI when speeding up foundation model pretraining.

Production Deployment Guide: Integrating DGA-Muon with FSDP and Megatron

DGA-Muon can be integrated directly into distributed pretraining pipelines like PyTorch Fully Sharded Data Parallel (FSDP2) or Megatron-LM:

# train_llm_dga_muon.py
import torch
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
# Import DGA-Muon optimizer
from dga_muon import DGAMuon

# Separate 2D matrix weights (for DGA-Muon) from 1D biases/norms (for AdamW)
muon_params = [p for p in model.parameters() if p.ndim == 2 and p.requires_grad]
adam_params = [p for p in model.parameters() if p.ndim < 2 and p.requires_grad]

optimizer_muon = DGAMuon(muon_params, lr=0.02, momentum=0.95, beta2=0.99, weight_decay=0.01)
optimizer_adam = torch.optim.AdamW(adam_params, lr=0.001, betas=(0.9, 0.95), weight_decay=0.01)

Engineering teams can consult the full developer documentation for detailed distributed checkpointing manifests, multi-node communication recipes, and learning rate scheduling recommendations.

Key Takeaways and Architectural Summary

The NorMuon Paradox: NorMuon’s coupling of adaptive second moments with orthogonalized momentum causes adaptivity to collapse into a trivial global scalar on square matrices ($m=n$).
Decoupled Gradient Adaptivity: DGA-Muon computes second moments directly from raw gradients $G_t$, restoring true optimization-driven adaptivity.
Geometry-Aligned Scaling: Uses row-wise scaling for wide matrices ($m < n$) and column-wise scaling for tall matrices ($m > n$), preserving polar factor orthogonality.
Provable Convergence: Establishes the first formal non-convex convergence guarantees for adaptive Muon optimizers.
1.73x Pretraining Speedup: Delivers faster wall-clock convergence over AdamW and beats NorMuon across frontier LLM pretraining runs.

DGA-Muon establishes a new gold standard in matrix-aware deep learning optimization, providing a rigorous and scalable foundation for the next generation of frontier AI models.