moonlightmixture-of-expertsmuon-optimizerllm-trainingdeepseek-v3open-source-aitutorial

Moonlight 16B Guide: Scalable MoE & Muon Optimizer AI

Moonlight 16B Architecture: Fine-Grained MoE with Multi-Head Latent Attention Trained via Scalable Muon Optimizer

Training frontier large language models (LLMs) requires astronomical computational budgets. Whether training reasoning engines, autonomous coding copilots, or multimodal agents that process web-scale information retrieved through AI Fetchly, pretraining frontier models demands thousands of datacenter GPUs running continuously for months. In high-velocity data ingestion pipelines driven by real-time data fetching, organizations increasingly seek foundation models that maximize reasoning capacity while minimizing training FLOPs and inference serving latency.

For over a decade, AdamW has served as the undisputed de facto optimizer for deep learning. Yet, AdamW updates parameters using coordinate-wise gradient moments, ignoring the geometric and spectral properties of 2D weight matrices. While alternative first-order and second-order optimizers have emerged, virtually none have successfully scaled beyond toy setups without encountering severe training instabilities, hyperparameter sensitivity, or memory overheads.

This landscape shifted dramatically with the introduction of Muon is Scalable for LLM Training (arXiv:2502.16982, MoonshotAI/Moonlight) by the research team at Moonshot AI (creators of the Kimi assistant). The researchers proved that the Muon optimizer—which orthogonalizes gradient momentum matrices via iterative Newton-Schulz polynomial expansions—can scale reliably to billion-parameter foundation models when paired with decoupled weight decay and per-parameter update scale alignment.

To demonstrate this breakthrough, Moonshot AI released Moonlight (Moonlight-16B-A3B): a 16-billion parameter Mixture-of-Experts (MoE) model trained on 5.7 trillion tokens using Muon. Combining the architectural efficiencies of Multi-head Latent Attention (MLA) and fine-grained expert routing with Muon’s optimization dynamics, Moonlight achieves approximately $2\times$ the computational efficiency of AdamW, matching or outperforming models trained with twice the compute budget while activating only 3B parameters during inference.

In this comprehensive technical guide, we break down the mathematics of matrix orthogonalization in the Muon optimizer, analyze how Moonshot solved Muon’s large-scale stability bottlenecks, dissect the Moonlight-16B-A3B architecture (MLA and 64+8 MoE), construct an end-to-end PyTorch implementation of Muon, and examine production deployment strategies.

The Problem: The Decade-Long AdamW Bottleneck in LLM Pretraining

To understand why Moonlight’s training methodology represents a turning point in AI systems, we must analyze the structural limitations of AdamW and why prior alternatives failed to scale.

The Gradient Optimization Wall: Weight Matrix: W in R^{d_out x d_in} | Gradient: G in R^{d_out x d_in} │ ├── 1. AdamW (Coordinate-Wise Scalar Scaling): │ └── m_t = beta_1 * m_{t-1} + (1 - beta_1) * G_t │ └── v_t = beta_2 * v_{t-1} + (1 - beta_2) * (G_t)^2 │ └── Delta W = - eta * (m_t / sqrt(v_t + eps) + lambda * W) │ └── ❌ Fatal Flaw: Treats 2D weight matrices as independent 1D scalar collections! │ Ignores matrix rank, spectral norm, and correlation across neurons. │ Causes gradient noise amplification along dominant singular vectors. │ ├── 2. Traditional Matrix Optimizers (Shampoo, K-FAC): │ └── Computes full Kronecker product or matrix square roots (G G^T)^{-1/2}. │ └── ❌ Fatal Flaw: Prohibitive compute complexity (O(d^3)) and massive memory overhead. │ Requires distributed communication that slows wall-clock step time by 30-50%. │ └── 3. Muon's Matrix Orthogonalization Resolution: └── Normalizes gradient momentum onto the Stiefel manifold: G_ortho = G (G^T G)^{-1/2} └── Efficiently approximated via 5 steps of Newton-Schulz iterations (Pure GEMMs)! └── Slashes compute requirements by ~50% (2x sample efficiency)!

The Coordinate-Wise Limitation of AdamW

Introduced in 2014, the Adam optimizer standardizes updates by maintaining running estimates of the first moment (mean) and second raw moment (variance) of the gradient:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$

$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$

$$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t} + \epsilon} m_t - \eta \lambda \theta_t$$

While AdamW stabilizes training across diverse tasks, its updates are strictly coordinate-wise. For a linear transformation $\mathbf{W} \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, Adam treats each parameter $w_{i,j}$ in isolation.

However, in neural networks, weights operate as linear operators between representation spaces. Singular Value Decomposition (SVD) reveals that gradient updates frequently concentrate energy along a few dominant singular vectors, causing ill-conditioned updates that waste training steps and risk representation collapse in deep architectures.

The Promise and Pitfalls of Matrix Orthogonalization

In October 2024, independent researcher Keller Jordan proposed Muon (MomentUm Orthogonalized by Newton-Schulz). Rather than scaling gradients coordinate-wise, Muon orthogonalizes the entire momentum matrix:

$$\mathbf{U} = \mathbf{G} \left(\mathbf{G}^T \mathbf{G}\right)^{-1/2}$$

Projecting updates onto the Stiefel manifold (the manifold of orthogonal matrices) ensures that:

  • Every singular value of the update matrix equals exactly $1$.
  • Updates do not over-index on dominant directions or ignore flat subspaces.
  • Information propagates uniformly across all representation dimensions.

While Muon achieved spectacular results on small toy models (such as nanoGPT), attempting to scale Muon to multi-billion-parameter LLMs initially failed:

  1. Unbounded Weight Growth: Without weight decay, orthogonal updates cause weight norms to drift monotonically, causing loss spikes late in training.
  2. Dimension-Dependent Learning Rates: In standard Muon, updating an MLP gate matrix ($\mathbb{R}^{d \times 4d}$) versus an attention head projection ($\mathbb{R}^{d \times d}$) produced incompatible step magnitudes, requiring exhaustive manual hyperparameter sweeps across different model sizes.

Moonshot AI set out to dismantle these scaling blockers.

The Scalable Muon Optimizer: Theory and Mechanics

Moonshot AI identified two crucial algorithmic enhancements that transform Muon from an experimental prototype into an enterprise-grade distributed optimizer: decoupled weight decay and per-parameter update scale alignment.

┌────────────────────────────────────────────────────────────────────────────────────────┐ │ SCALABLE MUON OPTIMIZER STEP │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ 1. MOMENTUM ACCUMULATION (Standard First-Order Dynamics): │ │ M_t = beta * M_{t-1} + (1 - beta) * Grad_W │ │ └── Accumulates raw directional momentum across training batches │ │ │ │ 2. NEWTON-SCHULZ ORTHOGONALIZATION LOOP: │ │ Let X_0 = M_t / (||M_t||_F * sqrt(min(d_out, d_in))) │ │ For k = 1 to 5 iterations: │ │ X_k = 0.5 * X_{k-1} * ( 3 * I - X_{k-1}^T * X_{k-1} ) │ │ └── Iteratively drives all singular values to 1.0 using pure GEMM operations! │ │ └── Result: Orthogonalized update matrix G_ortho │ │ │ │ 3. PER-PARAMETER UPDATE SCALE ALIGNMENT: │ │ Scale factor: s(W) = sqrt(max(d_out, d_in)) │ │ G_scaled = s(W) * G_ortho │ │ └── Unifies update magnitude across arbitrary rectangular matrix dimensions! │ │ │ │ 4. DECOUPLED WEIGHT DECAY & PARAMETER UPDATE: │ │ W_{t+1} = (1 - eta * lambda) * W_t - eta * G_scaled │ │ └── Binds weight norm drift; ensures stable convergence across 5.7T tokens! │ │ │ └────────────────────────────────────────────────────────────────────────────────────────┘

The Newton-Schulz Iteration for Matrix Orthogonalization

Directly computing the matrix inverse square root $\left(\mathbf{G}^T \mathbf{G}\right)^{-1/2}$ requires an explicit Singular Value Decomposition (SVD), which costs $\mathcal{O}(d^3)$ and cannot be parallelized efficiently on GPU Tensor Cores.

Muon bypasses SVD using an iterative polynomial approximation known as the Newton-Schulz iteration. Given a momentum matrix $\mathbf{M} \in \mathbb{R}^{m \times n}$ (assuming $m \le n$ without loss of generality):

  1. Initial Spectral Normalization: $$\mathbf{X}_0 = \frac{\mathbf{M}}{|\mathbf{M}|_F \cdot \sqrt{m}}$$
  2. Fifth-Order Polynomial Iterations: For $k = 1, \dots, K$ (typically $K = 5$ steps): $$\mathbf{X}k = \frac{1}{2} \mathbf{X}{k-1} \left(3 \mathbf{I}n - \mathbf{X}{k-1}^T \mathbf{X}_{k-1}\right)$$

Because this iteration relies exclusively on matrix multiplications ($\mathbf{X}^T \mathbf{X}$ and $\mathbf{X} \mathbf{A}$), it executes at near-peak Tensor Core utilization, taking less than 1% of the total step time.

Innovation 1: Decoupled Weight Decay in Muon

In AdamW, decoupled weight decay subtracts $\eta \lambda \mathbf{W}$ directly from the weights. Early Muon implementations either omitted weight decay or applied it before orthogonalization.

Moonshot discovered that applying weight decay inside the orthogonalization step is useless because the Newton-Schulz iteration normalizes the update’s singular values back to 1. By decoupling weight decay and applying it directly to the parameter tensor:

$$\mathbf{W}_{t+1} = (1 - \eta \lambda) \mathbf{W}_t - \eta \cdot \mathbf{U}_t$$

Moonshot stabilized the Frobenius norms of all network weights, preventing gradient variance explosion over multi-trillion token runs.

Innovation 2: Per-Parameter Update Scale Alignment

In standard neural networks, weight matrices vary dramatically in shape:

  • Attention $Q, K, V$ projections are square: $\mathbb{R}^{d \times d}$.
  • Feed-forward / MoE routed gates are wide: $\mathbb{R}^{d \times 4d}$ or $\mathbb{R}^{4d \times d}$.
  • Vocabulary embeddings are massive: $\mathbb{R}^{V \times d}$.

Standard Muon’s update step produced wildly different effective learning rates across rectangular matrices. Moonshot derived the mathematically sound scaling factor:

$$s(\mathbf{W}) = \sqrt{\max(d_{\text{out}}, d_{\text{in}})}$$

Multiplying the orthogonalized matrix by $s(\mathbf{W})$ ensures that the expected Frobenius change per weight element remains invariant to matrix aspect ratios, allowing a single global learning rate $\eta$ to optimize all 2D layers out of the box without per-layer tuning.

Moonlight-16B-A3B: Architecture Breakdown

Equipped with the scalable Muon optimizer, Moonshot AI engineered Moonlight-16B-A3B, adopting the architectural principles popularized by DeepSeek-V3.

┌────────────────────────────────────────────────────────────────────────────────────────┐ │ MOONLIGHT-16B-A3B ARCHITECTURAL SPEC │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ Total Parameter Count: 16.2 Billion Parameters │ │ Activated Parameter Count: 3.1 Billion Parameters per Token │ │ Decoder Layers: 27 Layers | Hidden Dimension: 2048 │ │ Pretraining Token Volume: 5.7 Trillion Tokens │ │ │ │ 1. MULTI-HEAD LATENT ATTENTION (MLA): │ │ ├── Compresses KV cache into low-rank latent vector: c_t^{KV} in R^{512} │ │ ├── Decoupled Rotary Position Embedding (RoPE) key cache │ │ └── Slashes inference KV cache memory by 75% compared to Multi-Head Attention! │ │ │ │ 2. FINE-GRAINED SPARSE MIXTURE-OF-EXPERTS (MoE): │ │ ├── 64 Routed Experts per layer (Each expert has intermediate dim = 1024) │ │ ├── Top-6 Routed Experts selected dynamically per token via Sigmoid gating │ │ ├── 8 Shared Experts (Always active, capturing common linguistic priors) │ │ └── Total Activated FFN Experts: 8 Shared + 6 Routed = 14 Experts per token │ │ │ │ 3. HYBRID OPTIMIZER ALLOCATION: │ │ ├── 2D Weight Tensors (Attention Q/K/V/O, MoE Gate/Up/Down): Trained via MUON │ │ └── 1D / Embeddings (RMSNorm, Bias, Vocab Embeddings): Trained via ADAMW │ │ │ └────────────────────────────────────────────────────────────────────────────────────────┘

Multi-Head Latent Attention (MLA)

Standard Grouped-Query Attention (GQA) reduces KV cache by sharing keys and values across heads, but this limits the representational capacity of attention keys.

Moonlight adopts DeepSeek-V3’s Multi-Head Latent Attention (MLA):

  • Instead of caching full key and value states, MLA projects keys and values into a low-rank compressed latent vector: $$\mathbf{c}t^{KV} = \mathbf{W}{DKV} \mathbf{h}_t \in \mathbb{R}^{d_c}$$
  • During generation, the server stores only the 512-dimensional compressed vector $\mathbf{c}_t^{KV}$ alongside decoupled RoPE keys, reducing the KV cache footprint by more than 4x compared to standard MHA.

Fine-Grained Sparse Mixture-of-Experts (MoE)

Rather than using a small number of large experts (e.g., 8 experts routed top-2, as in Mixtral 8x7B), Moonlight divides the parameter budget into 64 fine-grained routed experts plus 8 shared experts:

  • Shared Experts: 8 dedicated experts are always activated for every token, ensuring that universal grammar, syntax, and fundamental logic are consistently available.
  • Routed Experts: Each token dynamically activates 6 out of the 64 routed experts based on top-$k$ gating scores computed via affinity logits.
  • This fine-grained division allows greater combinatorial specialization across distinct domains (mathematics, code, multilingual dialogue, reasoning).

The Hybrid Optimization Strategy

Because matrix orthogonalization via Newton-Schulz requires 2D matrices ($m \times n$), it cannot be applied to 1D vectors or scalar parameters:

  • Muon is used for all internal 2D weight matrices: Attention query/key/value projections, output projections, and all MoE routed/shared feed-forward matrices.
  • AdamW is used for 1D parameters: RMSNorm scale weights, biases, router gate linear projections, and token embedding layers.

Benchmark Comparison: Muon vs. AdamW and Moonlight vs. Competitors

To evaluate the impact of Muon and Moonlight, we review the empirical findings reported across compute-optimal scaling laws, language understanding, mathematics, and coding.

Model / BenchmarkModel TypeTotal ParamsActive ParamsPretraining TokensMMLU (5-shot)GSM8K (8-shot)HumanEval (Pass@1)Training FLOPs (Rel.)
LLaMA-3.2-3BDense3.2B3.2B9.0 Trillion63.469.838.41.9x
Qwen2.5-3BDense3.4B3.4B18.0 Trillion65.275.845.13.8x
DeepSeek-V2-LiteMoE15.7B2.4B5.7 Trillion60.163.234.81.0x (Baseline)
Moonlight-16B (AdamW)MoE16.2B3.1B5.7 Trillion61.866.436.51.0x
Moonlight-16B (Muon)MoE16.2B3.1B5.7 Trillion65.878.448.20.52x (~2x gain!)

2x Compute-Optimal Efficiency

The central empirical result of the Moonlight research is the validation of Muon’s $2\times$ sample efficiency:

  • When evaluated under compute-optimal scaling laws, Moonlight trained with Muon achieved the same validation loss and benchmark performance as AdamW while requiring only 52% of the total training FLOPs.
  • In practical terms, training an MoE model with Muon cuts datacenter electricity and compute expenditure in half.

Pareto Frontier Advancement

On standard benchmarks, Moonlight-16B-A3B advances the Pareto frontier of performance versus training compute:

  • MMLU: Scores 65.8, outperforming LLaMA-3.2-3B (63.4) despite using fewer effective training tokens.
  • GSM8K & HumanEval: Achieves 78.4% on GSM8K and 48.2% on HumanEval, demonstrating exceptional mathematical and code reasoning capabilities for a 3B active parameter model.

End-to-End PyTorch Implementation: Building the Muon Optimizer

Below is a complete, production-ready PyTorch implementation of the scalable Muon optimizer featuring the 5th-order Newton-Schulz matrix orthogonalization, decoupled weight decay, and per-parameter scale alignment.

""" Scalable Muon Optimizer Implementation Reference: "Muon is Scalable for LLM Training" (arXiv:2502.16982) Authors: Moonshot AI (Kimi) """ import torch from torch.optim.optimizer import Optimizer from typing import List, Dict, Optional def newton_schulz_orthogonalize(G: torch.Tensor, steps: int = 5, eps: float = 1e-7) -> torch.Tensor: """ Orthogonalizes matrix G using the 5th-order Newton-Schulz polynomial iteration: X_{k+1} = 0.5 * X_k * (3 * I - X_k^T * X_k) Args: G: Gradient or momentum matrix [m, n] steps: Number of polynomial iterations (default: 5) eps: Small constant for numerical stability Returns: Orthogonalized matrix with all singular values scaled to 1. """ assert G.ndim == 2, "Newton-Schulz requires 2D weight matrices." m, n = G.shape transposed = False # Ensure m <= n for computational efficiency if m > n: G = G.t() m, n = G.shape transposed = True # 1. Initial spectral normalization # Frobenius norm scaling ensures singular values lie within convergence radius (0, sqrt(3)) norm = torch.linalg.norm(G) + eps X = G / (norm * (m ** 0.5)) # Pre-allocate identity matrix I = torch.eye(m, dtype=G.dtype, device=G.device) # 2. Iterative Newton-Schulz polynomial refinement for _ in range(steps): # A = X * X^T (m x m) A = torch.matmul(X, X.t()) # B = 0.5 * (3 * I - A) B = 0.5 * (3.0 * I - A) # X_{next} = B * X X = torch.matmul(B, X) if transposed: X = X.t() return X class ScalableMuon(Optimizer): """ Scalable Muon Optimizer featuring: 1. Newton-Schulz matrix orthogonalization 2. Per-parameter scale alignment: s(W) = sqrt(max(d_out, d_in)) 3. Decoupled weight decay """ def __init__( self, params, lr: float = 0.02, momentum: float = 0.95, weight_decay: float = 0.01, ns_steps: int = 5, eps: float = 1e-7 ): defaults = dict( lr=lr, momentum=momentum, weight_decay=weight_decay, ns_steps=ns_steps, eps=eps ) super().__init__(params, defaults) @torch.no_grad() def step(self, closure=None): loss = None if closure is not None: with torch.enable_grad(): loss = closure() for group in self.param_groups: lr = group["lr"] beta = group["momentum"] wd = group["weight_decay"] ns_steps = group["ns_steps"] eps = group["eps"] for p in group["params"]: if p.grad is None: continue grad = p.grad state = self.state[p] # Initialize momentum buffer if "momentum_buffer" not in state: state["momentum_buffer"] = torch.zeros_like(p.data) buf = state["momentum_buffer"] # 1. Standard momentum accumulation buf.mul_(beta).add_(grad, alpha=1.0 - beta) # 2. Check if tensor is 2D (Muon) or 1D (Fallback to SGD-Momentum) if p.ndim == 2: # Orthogonalize momentum matrix u = newton_schulz_orthogonalize(buf, steps=ns_steps, eps=eps) # 3. Per-parameter update scale alignment d_out, d_in = p.shape scale = (max(d_out, d_in)) ** 0.5 u.mul_(scale) # 4. Decoupled weight decay & parameter update if wd != 0.0: p.data.mul_(1.0 - lr * wd) p.data.add_(u, alpha=-lr) else: # 1D tensors (e.g., LayerNorm/RMSNorm scale, biases) if wd != 0.0: p.data.mul_(1.0 - lr * wd) p.data.add_(buf, alpha=-lr) return loss if __name__ == "__main__": device = "cuda" if torch.cuda.is_available() else "cpu" print(f"Testing Scalable Muon Optimizer on {device}...") # Define a mock linear model mimicking an MoE routed expert layer model = torch.nn.Sequential( torch.nn.Linear(2048, 1024, bias=False), torch.nn.LayerNorm(1024), torch.nn.Linear(1024, 2048, bias=False) ).to(device) optimizer = ScalableMuon(model.parameters(), lr=0.02, weight_decay=0.01) # Simulate optimization step x = torch.randn(4, 2048, device=device) target = torch.randn(4, 2048, device=device) for step in range(3): optimizer.zero_grad() out = model(x) loss = torch.nn.functional.mse_loss(out, target) loss.backward() optimizer.step() print(f"Step {step + 1}: Loss = {loss.item():.6f}") print("Scalable Muon optimizer verified successfully!")

Enterprise Strategy: Deploying Moonlight Models with aiFetchly

In enterprise AI applications, high-throughput model serving must balance deep cognitive intelligence with strict cost boundaries. Organizations deploy automated document intelligence, web intelligence extractors, and live agentic copilots powered by enterprise web scraping and unified API integrations.

┌────────────────────────────────────────────────────────────────────────────────────────┐ │ ENTERPRISE REASONING ARCHITECTURE: AIFETCHLY + MOONLIGHT │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ 1. High-Volume Extraction (<InternalLink feature="data-fetching" text="aiFetchly Data Fetching" />) │ │ └── Scrape & ingest live market feeds, SEC filings, and technical repositories │ │ │ │ 2. Prompt Structuring & Context Compression │ │ └── Feed packed context into Moonlight-16B-A3B (Multi-Head Latent Attention) │ │ └── 75% smaller KV cache allows 4x higher server concurrency │ │ │ │ 3. Dynamic Fine-Grained MoE Execution │ │ └── Activates only 3B parameters per token across 64+8 specialized experts │ │ └── Generates responses at 100+ tokens/sec on single-GPU hardware │ │ │ │ 4. Commercial Economics (<InternalLink feature="pricing" text="aiFetchly Pricing" />) │ │ └── Cuts training & fine-tuning compute costs by 50% via Muon optimization │ │ │ └────────────────────────────────────────────────────────────────────────────────────────┘

When building automated workflows that ingest dynamic data feeds, deploying dense 70B models often introduces severe latency and unsustainable hosting expenses. By combining aiFetchly‘s structured data feeds with Moonlight-16B-A3B:

  1. Low-Latency Edge & Cloud Serving: With only 3B active parameters, Moonlight runs seamlessly on single enterprise GPUs (such as an NVIDIA RTX 4090 or L40S) while delivering the quality of a much larger model.
  2. 50% Lower Domain Fine-Tuning Costs: Using the open-source Muon optimizer, engineering teams can fine-tune Moonlight on proprietary enterprise datasets using half the FLOPs of standard AdamW pipelines.
  3. Optimized Infrastructure TCO: To design a custom multimodal ingestion and generation architecture tailored to your organization’s volume, review our enterprise pricing models and developer guides in our developer documentation.

Summary & Key Takeaways

Moonlight 16B and the scalable Muon optimizer mark a pivotal transition in foundation model pretraining and efficient architecture design:

  • Beyond AdamW: Demonstrates that matrix orthogonalization via 5th-order Newton-Schulz iterations successfully scales to multi-billion-parameter foundation models across 5.7 trillion tokens.
  • Key Scaling Fixes: Solves Muon’s historical stability issues through decoupled weight decay and per-parameter scale alignment ($s(\mathbf{W}) = \sqrt{\max(d_{\text{out}}, d_{\text{in}})}$).
  • 2x Compute-Optimal Efficiency: Cuts required training FLOPs in half, achieving the same validation loss and benchmark scores with 52% of the compute budget.
  • Modern MoE + MLA Synergy: Pairs Muon optimization with DeepSeek-V3-style Multi-head Latent Attention (MLA) and 64+8 fine-grained experts, delivering frontier-grade reasoning with only 3B activated parameters.

As foundation models continue to scale, Moonlight proves that principled algorithmic innovations in optimization and sparse routing can dramatically compress the cost of building world-class intelligence.