apollollm-trainingoptimizersmemory-efficientadamwsgdtutorial

APOLLO Guide: SGD-Level Memory with AdamW LLM Scaling

APOLLO Architecture: Approximated Gradient Scaling, Random Projection, and SGD-Level Memory LLM Optimization

The Memory Crisis in LLM Training and the AdamW Optimization Tax

As modern enterprise systems increasingly deploy massive foundation models for multi-document synthesis, complex autonomous agent execution, and continuous data ingestion through real-time data fetching pipelines and intelligent web scraping, training and fine-tuning these models has become a dominant operational cost. In high-throughput AI platforms engineered by AiFetchly, keeping training scalable, energy-efficient, and budget-accessible is vital.

However, scaling Large Language Model (LLM) pretraining and full-parameter fine-tuning collides directly with a crippling physical bottleneck: the optimizer memory tax of AdamW.

The VRAM Memory Footprint Breakdown during LLM Training (Mixed Precision FP16/BF16): Total GPU Memory = Model Weights + Gradients + Optimizer States + Activation Memory For a 70-Billion Parameter Model (e.g., LLaMA-3.1-70B): 1. Static Model Parameters: - 70B parameters in FP16 / BF16 (2 bytes per param) ──────────► 140 GB 2. Backward Gradients: - 70B gradients in FP16 / BF16 (2 bytes per param) ───────────► 140 GB 3. Standard AdamW Optimizer States: - First Momentum (m_t) in FP32 (4 bytes per param) ──────────► 280 GB - Second Momentum (v_t) in FP32 (4 bytes per param) ─────────► 280 GB Total Optimizer Memory Tax ───────────────────────────────────► 560 GB! ════════════════════════════════════════════════════════════════════════════════ Result: AdamW optimizer states consume FOUR TIMES the memory of the model weights! Total static memory reaches 840 GB before even allocating a single batch activation!

1. The Prohibitive Cost of AdamW Optimizer States

For over a decade, AdamW (Adaptive Moment Estimation with Decoupled Weight Decay) has stood as the unchallenged gold standard for training deep neural networks. Its strength lies in its coordinate-wise adaptive learning rate:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$

$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$

$$\theta_{t+1} = \theta_t - \eta_t \left( \frac{\widehat{m}_t}{\sqrt{\widehat{v}_t} + \epsilon} + \lambda \theta_t \right)$$

where $g_t$ is the stochastic gradient, $m_t$ tracks first-order momentum, and $v_t$ maintains the running uncentered second moment (variance) for every single scalar parameter in the network.

While this element-wise preconditioning dramatically accelerates convergence and stabilizes ill-conditioned loss landscapes, storing two 32-bit floating-point tensors ($m_t$ and $v_t$) for every scalar parameter requires 8 bytes of extra memory per parameter.

To survive this memory explosion, engineering teams are forced to adopt expensive distributed system strategies:

  • ZeRO-Stage 3 & Fully Sharded Data Parallel (FSDP): Sharding optimizer states, gradients, and model weights across dozens of GPUs. While effective, cross-node communication overhead (AllGather and ReduceScatter) chokes network interconnects and caps GPU compute utilization.
  • Micro-Batch Starvation: Shrinking per-GPU batch sizes down to $B=1$ or $B=2$, leading to poor matrix-multiplication efficiency on NVIDIA Tensor Cores and dragging down overall training throughput.
  • Hardware Exclusivity: Requiring multi-node clusters of high-end enterprise GPUs (such as 8x H100 80GB nodes), pricing out research labs, startups, and edge deployments from performing custom full-parameter domain training.

2. The Failures of Prior Memory-Efficient Optimizers

Recognizing this critical bottleneck, researchers introduced several alternative memory-efficient optimizers, yet each introduced severe practical compromises:

Comparison of Prior Memory-Reduction Paradigms: ┌─────────────────────────────────────────────────────────────────────────────┐ │ 1. Stochastic Gradient Descent (SGD with Momentum): │ │ - Memory: Only stores 1 state (4 bytes/param). │ │ - Failure: Lacks adaptive second moments; training loss collapses or │ │ requires excruciating hyperparameter tuning; completely fails to │ │ match AdamW pretraining perplexity on transformers. │ ├─────────────────────────────────────────────────────────────────────────────┤ │ 2. 8-Bit Quantized Optimizers (bitsandbytes AdamW-8bit): │ │ - Memory: Compresses states from 8 bytes to 2 bytes per param. │ │ - Failure: Still retains full parameter-sized tensors; 70B model still │ │ demands 140 GB of VRAM solely for optimizer states. │ ├─────────────────────────────────────────────────────────────────────────────┤ │ 3. Adafactor (Factored Second Moments): │ │ - Memory: Decomposes 2D weight matrices into row and column sums. │ │ - Failure: Prone to numerical instability; fails on 1D biases/norms; │ │ requires distinct tuning recipes; often lags AdamW downstream quality. │ ├─────────────────────────────────────────────────────────────────────────────┤ │ 4. GaLore (Gradient Low-Rank Projection): │ │ - Memory: Projects gradients into low-rank subspace (rank r << d). │ │ - Failure: Requires periodic Singular Value Decomposition (SVD) on every │ │ weight matrix; high computational overhead; sensitive to rank choice. │ └─────────────────────────────────────────────────────────────────────────────┘

The fundamental dilemma remained unresolved: Can we achieve the ultra-low memory footprint of SGD while preserving the flawless convergence and training stability of AdamW?

Enter APOLLO: Approximated Gradient Scaling for LLM Optimization

Published at MLSys 2025 by researchers from UT Austin, Meta, and Tsinghua University (APOLLO: SGD-like Memory, AdamW-level Performance, arXiv:2412.05270), APOLLO answers this question with a definitive breakthrough.

The Paradigm Shift: From Fine-Grained Scaling to Structured Random Projection ═══════════════════════════════════════════════════════════════════════════════ AdamW Approach (High Overhead): Gradient Matrix G [M x N] ──► Second Moment V [M x N] (Full Size: M x N) Memory: 8 bytes per parameter | Element-Wise Scaling: g_{i,j} / sqrt(v_{i,j}) APOLLO Approach (SGD-Level Overhead): Gradient Matrix G [M x N] ──► Pure Random Projection Matrix P [N x r] (r = 1) │ ▼ Auxiliary Low-Rank State V_low = G * P [M x r] (Tiny Size: M x 1) Memory: < 0.1% Optimizer Overhead | Structured Coordinate Preconditioning ═══════════════════════════════════════════════════════════════════════════════

The core theoretical discovery behind APOLLO is that AdamW’s learning rate adaptation rule does not require fine-grained element-wise precision. Instead, the preconditioning effect of the second moment can be effectively coarsened as a structured learning rate update.

By approximating learning rate scaling using an auxiliary low-rank optimizer state generated via pure random projection (completely eliminating costly SVD operations), APOLLO slashes optimizer state memory by over 95% while matching or even outperforming full-precision AdamW across pretraining and fine-tuning benchmarks.

Key Breakthroughs of the APOLLO Optimizer

  1. SGD-Level Memory Footprint: With its rank-1 variant (APOLLO-Mini), the auxiliary state occupies less than 0.1% of the original AdamW memory budget.
  2. Zero SVD Computational Overhead: Unlike GaLore, APOLLO relies on fixed, deterministic random projections that add zero matrix decomposition latency.
  3. 3x Higher Training Throughput: Slashing optimizer VRAM enables up to 4x larger per-GPU micro-batch sizes, maximizing NVIDIA Tensor Core operational efficiency.
  4. Accessible Pretraining: Enables pretraining a 7-billion parameter LLaMA model on a single 16GB consumer GPU, and pretraining LLaMA-13B with standard PyTorch DistributedDataParallel (DDP) without FSDP or ZeRO-3 offloading.

Deep Dive into the APOLLO Architecture

The mathematical foundation of APOLLO centers on reformulating how adaptive second moments are approximated across high-dimensional parameter spaces.

The APOLLO Architectural Pipeline: ┌─────────────────────────────────────────────────────────────────────────────┐ │ Backward Pass: Compute Stochastic Gradient Matrix G_t in R^{M x N} │ └──────────────────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ 1. Structured Random Projection (Zero SVD overhead) │ │ - Generate orthogonal or Gaussian random projection matrix P in R^{N x r}│ │ - Project gradient into rank-r subspace: G_proj = G_t * P │ └──────────────────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ 2. Auxiliary Low-Rank Second Moment Accumulation │ │ - Track running variance of projected gradients: │ │ v_t = beta_2 * v_{t-1} + (1 - beta_2) * (G_proj)^2 in R^{M x r} │ └──────────────────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ 3. Structured Learning Rate Reconstruction & Momentum Update │ │ - Reconstruct preconditioning scale factor: S_t = sqrt(v_t * P^T) │ │ - Apply structured scaling directly to first momentum or raw gradient │ │ - Update model parameters with weight decay: theta_{t+1} = theta_t - eta │ └─────────────────────────────────────────────────────────────────────────────┘

1. Mathematical Formulation: Coarsening AdamW

Consider a 2D weight matrix $W \in \mathbb{R}^{M \times N}$ in a transformer feed-forward or projection layer, with gradient $G_t \in \mathbb{R}^{M \times N}$.

In AdamW, the second moment tensor $V_t \in \mathbb{R}^{M \times N}$ tracks each entry individually:

$$V_{t, i, j} = \beta_2 V_{t-1, i, j} + (1 - \beta_2) G_{t, i, j}^2$$

APOLLO observes that parameter gradient variances across a neural layer exhibit high directional coherence: individual channels (rows or columns) share similar orders of magnitude, while within-channel variations have minimal impact on preconditioning quality.

Therefore, APOLLO approximates the second moment as a low-rank factorization:

$$V_t \approx U_t \Sigma_t Q_t^T$$

Instead of computing dynamic singular vectors via expensive SVD updates at runtime, APOLLO fixes a random projection matrix $P \in \mathbb{R}^{N \times r}$, where $r \ll \min(M, N)$ is the rank (typically $r = 1$ in APOLLO-Mini or $r = 4$ in full APOLLO).

2. The Low-Rank Random Projection Operator

The gradient matrix $G_t$ is projected into the lower-dimensional subspace:

$$H_t = G_t P \in \mathbb{R}^{M \times r}$$

The auxiliary second moment $v_t$ is updated exclusively in this compressed subspace:

$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) (H_t \odot H_t)$$

where $\odot$ denotes Hadamard (element-wise) multiplication. Notice the dramatic dimensionality difference:

  • Standard AdamW stores $M \times N$ values (e.g., $4096 \times 4096 = 16,777,216$ floats = 67.1 MB per layer).
  • APOLLO (with $r=1$) stores $M \times 1$ values (e.g., $4096 \times 1 = 4,096$ floats = 16.3 KB per layer).
  • Optimizer state reduction: 4,096x smaller!

3. Structured Coordinate Preconditioning

To update the parameters, the learning rate scaling matrix $\Lambda_t \in \mathbb{R}^{M \times N}$ is reconstructed from the low-rank auxiliary state:

$$\Lambda_{t, i, j} = \frac{1}{\sqrt{\sum_{k=1}^r v_{t, i, k} \cdot |P_{j, k}|} + \epsilon}$$

Because $P$ is fixed or periodically refreshed using a deterministic pseudo-random seed, computing $\Lambda_t$ requires no global synchronization or cross-GPU communication.

The model update step then fuses decoupled weight decay and structured gradient scaling:

$$\theta_{t+1} = \theta_t - \eta_t \left( \Lambda_t \odot m_t + \lambda \theta_t \right)$$

For maximum memory savings, APOLLO-Mini can even replace the full first momentum $m_t$ with Nesterov accelerated gradient scaling or quantized momentum, achieving complete pretraining with virtually zero optimizer overhead.

Hands-On Implementation: Building the APOLLO Optimizer

To understand how APOLLO operates under the hood, let’s implement a complete, production-ready PyTorch optimizer implementing the APOLLO-Mini algorithm.

1. PyTorch Implementation of APOLLO-Mini

import torch from torch.optim.optimizer import Optimizer import math class ApolloMini(Optimizer): """ APOLLO-Mini: Approximated Gradient Scaling for Memory-Efficient LLM Optimization. MLSys 2025 (arXiv:2412.05270). Implements rank-1 random-projection second moment scaling with SGD-level memory costs. """ def __init__( self, params, lr: float = 1e-3, betas: tuple = (0.9, 0.999), eps: float = 1e-8, weight_decay: float = 0.01, rank: int = 1, proj_type: str = "random", ): if not 0.0 <= lr: raise ValueError(f"Invalid learning rate: {lr}") if not 0.0 <= eps: raise ValueError(f"Invalid epsilon value: {eps}") if not 0.0 <= betas[0] < 1.0: raise ValueError(f"Invalid beta parameter at index 0: {betas[0]}") if not 0.0 <= betas[1] < 1.0: raise ValueError(f"Invalid beta parameter at index 1: {betas[1]}") defaults = dict( lr=lr, betas=betas, eps=eps, weight_decay=weight_decay, rank=rank, proj_type=proj_type, ) super(ApolloMini, self).__init__(params, defaults) @torch.no_grad() def step(self, closure=None): loss = None if closure is not None: with torch.enable_grad(): loss = closure() for group in self.param_groups: lr = group["lr"] beta1, beta2 = group["betas"] eps = group["eps"] weight_decay = group["weight_decay"] rank = group["rank"] for p in group["params"]: if p.grad is None: continue grad = p.grad if grad.is_sparse: raise RuntimeError("ApolloMini does not support sparse gradients") state = self.state[p] # State initialization if len(state) == 0: state["step"] = 0 # For 2D matrices (Linear layers, attention projections): # We create an auxiliary low-rank state of shape [M, rank] if p.dim() >= 2: M = p.shape[0] N = p.numel() // M state["is_matrix"] = True state["M"] = M state["N"] = N # Generate fixed deterministic random projection vector P in R^{N x rank} # Normalized by 1 / sqrt(N) to preserve gradient magnitude generator = torch.Generator(device=p.device).manual_seed(42) P = torch.randn(N, rank, generator=generator, device=p.device, dtype=p.dtype) P = P / math.sqrt(N) state["proj"] = P # Auxiliary low-rank second moment: [M, rank] in FP32 state["exp_avg_sq_low"] = torch.zeros(M, rank, dtype=torch.float32, device=p.device) else: # For 1D tensors (biases, layer norms), fallback to standard scalar scaling state["is_matrix"] = False state["exp_avg_sq_low"] = torch.zeros_like(p, dtype=torch.float32) # First momentum state (stored in FP16 or BF16 for extreme memory efficiency) state["exp_avg"] = torch.zeros_like(p, memory_format=torch.preserve_format) state["step"] += 1 step = state["step"] exp_avg = state["exp_avg"] exp_avg_sq_low = state["exp_avg_sq_low"] # Perform decoupled weight decay if weight_decay != 0: p.mul_(1.0 - lr * weight_decay) # Update first momentum (m_t) exp_avg.mul_(beta1).add_(grad, alpha=1.0 - beta1) # Bias correction terms bias_correction1 = 1.0 - beta1 ** step bias_correction2 = 1.0 - beta2 ** step if state["is_matrix"]: M = state["M"] N = state["N"] P = state["proj"] # Reshape gradient to 2D matrix: [M, N] grad_2d = grad.view(M, N) # Step 1: Project gradient into rank-r subspace: [M, N] @ [N, rank] -> [M, rank] grad_proj = torch.matmul(grad_2d, P) # Step 2: Update low-rank second moment in FP32 exp_avg_sq_low.mul_(beta2).addcmul_(grad_proj, grad_proj, value=1.0 - beta2) # Step 3: Compute structured scale factor: sqrt(v_corrected) in [M, 1] v_hat = exp_avg_sq_low / bias_correction2 scale_row = torch.sqrt(v_hat).add_(eps) # Shape: [M, rank] # Step 4: Apply row-wise structured scaling to momentum # Broadcasting [M, 1] across [M, N] m_2d = exp_avg.view(M, N) step_size = lr / bias_correction1 # In-place parameter update update = m_2d / scale_row p.add_(update.view_as(p), alpha=-step_size) else: # 1D Parameter update (Standard adaptive scaling) exp_avg_sq_low.mul_(beta2).addcmul_(grad, grad, value=1.0 - beta2) v_hat = exp_avg_sq_low / bias_correction2 denom = torch.sqrt(v_hat).add_(eps) step_size = lr / bias_correction1 p.addcdiv_(exp_avg, denom, value=-step_size) return loss

2. Integration with Hugging Face Trainer and PyTorch Training Loop

Here is how you configure APOLLO within modern training pipelines:

import torch import torch.nn as nn from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer # Load target model model_name = "meta-llama/Llama-3.1-8B" model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=torch.bfloat16, device_map="auto" ) # Custom optimizer factory creating APOLLO-Mini def create_apollo_optimizer(model, lr=5e-4, weight_decay=0.01): decay_params = [] no_decay_params = [] for name, param in model.named_parameters(): if not param.requires_grad: continue if "bias" in name or "layernorm" in name or "norm" in name: no_decay_params.append(param) else: decay_params.append(param) optimizer_grouped_parameters = [ {"params": decay_params, "weight_decay": weight_decay}, {"params": no_decay_params, "weight_decay": 0.0}, ] return ApolloMini( optimizer_grouped_parameters, lr=lr, betas=(0.9, 0.999), eps=1e-8, rank=1 # APOLLO-Mini: Rank-1 projection ) optimizer = create_apollo_optimizer(model) print("APOLLO-Mini optimizer initialized with <0.1% memory overhead!")

Benchmark Comparison: APOLLO vs. AdamW, GaLore, and Adafactor

To evaluate the effectiveness of APOLLO, researchers conducted extensive pretraining and fine-tuning experiments across LLaMA, RoBERTa, and Mistral model architectures.

1. Pretraining Perplexity on LLaMA Models (1B to 7B Parameters)

Below is the pretraining performance comparison on the C4 and SlimPajama datasets across 100,000 steps:

OptimizerMemory per ParamTotal Optimizer Memory (7B Model)Pretraining Perplexity (LLaMA-1B)Pretraining Perplexity (LLaMA-7B)Relative Step Throughput
Standard AdamW8.0 Bytes56.0 GB14.829.451.00x (Baseline)
AdamW-8bit (bitsandbytes)2.0 Bytes14.0 GB14.859.480.94x (Slow dequant)
Adafactor~0.1 Bytes~0.7 GB15.629.891.05x
GaLore (Rank-128)1.2 Bytes8.4 GB14.909.510.82x (SVD tax)
APOLLO (Rank-4)0.3 Bytes2.1 GB14.789.421.28x
APOLLO-Mini (Rank-1)0.05 Bytes0.35 GB14.819.441.35x
Pretraining Loss Convergence Trajectory: Perplexity ▲ 16│ Adafactor (Plateaus early) 15│───────┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈ 14│ AdamW (Full 56 GB footprint) 13│ GaLore (High SVD CPU latency) 12│═══════ APOLLO-Mini (SGD memory, 0.35 GB, matches AdamW perfectly!) └────────────────────────────────────────► Training Tokens / Steps

2. GPU Hardware Throughput and Batch Scaling

Because APOLLO nearly eliminates the 56 GB optimizer footprint on a 7B model, the freed VRAM can be allocated directly to activation memory and larger batch sizes:

Throughput on 8x NVIDIA A100-80GB (LLaMA-7B Pretraining): AdamW (Max Batch Size = 8 per GPU): 12,400 tokens/sec █████ AdamW-8bit (Max Batch Size = 16 per GPU): 18,200 tokens/sec ███████ GaLore (Max Batch Size = 24 per GPU): 16,100 tokens/sec ██████ APOLLO-Mini (Max Batch Size = 32 per GPU): 37,800 tokens/sec ███████████████ (3.05x Speedup!) ═══════════════════════════════════════════════════════════════════════════════ By supporting 4x larger batch sizes without OOM crashes, APOLLO-Mini achieves over 3x higher aggregate token training throughput!

3. Comprehensive Technical Feature Matrix

Feature / DimensionStandard AdamWAdafactorGaLoreAPOLLO-Mini
Optimizer State Memory8 Bytes/param (100%)~0.1 Bytes/param (~1%)~1.2 Bytes/param (15%)< 0.05 Bytes/param (<1%)
Computational OverheadLow (Element-wise)LowHigh (Periodic SVD)Zero (Deterministic Proj)
Hyperparameter SensitivityStandard ($\alpha, \beta_1, \beta_2$)High (Requires manual tuning)High (Rank choice $r$)Low (Default AdamW recipe)
1D Layer SupportNativeFails / Needs FallbackFails / Needs FallbackNative Full-Model Support
Pretraining ConvergenceOptimalDegradedNear-OptimalOptimal (Matches/Beats AdamW)
Max Batch Size Potential1.0x (Baseline)3.5x2.5x4.0x+

Production Deployment: Integrating APOLLO into Distributed Clusters

Deploying APOLLO into high-scale enterprise environments requires optimizing data parallelism, mixed-precision scaling, and distributed gradient clipping. When planning enterprise AI infrastructure, consult AiFetchly enterprise pricing and our developer API documentation.

1. Step-by-Step Production Configuration

Follow these production best practices when rolling out APOLLO across distributed training clusters:

Step 1: Install the APOLLO Package Install the official APOLLO library from PyPI:

pip install apollo-torch

Step 2: Configure Distributed Data Parallel (DDP) Pretraining Because APOLLO eliminates optimizer state sharding dependencies, you can pretrain 7B and 13B models using standard DDP without complex ZeRO-3 configurations:

import os import torch import torch.distributed as dist from torch.nn.parallel import DistributedDataParallel as DDP from apollo import ApolloAdamW def setup_distributed_training(): dist.init_process_group(backend="nccl") local_rank = int(os.environ["LOCAL_RANK"]) torch.cuda.set_device(local_rank) return local_rank def train_production_step(): local_rank = setup_distributed_training() # Load 7B Model in BF16 model = create_foundation_model().to(local_rank).to(torch.bfloat16) model = DDP(model, device_ids=[local_rank]) # Instantiate APOLLO with Rank-1 approximation and decoupled decay optimizer = ApolloAdamW( model.parameters(), lr=3e-4, betas=(0.9, 0.999), eps=1e-8, weight_decay=0.01, rank=1, scale_type="channel", # Channel-wise structured preconditioning warmup_steps=2000 ) # Enable automatic mixed precision scaler = torch.cuda.amp.GradScaler(enabled=False) # Native BF16 needs no scaler for step, batch in enumerate(dataloader): optimizer.zero_grad(set_to_none=True) with torch.autocast(device_type="cuda", dtype=torch.bfloat16): outputs = model(**batch) loss = outputs.loss loss.backward() # Gradient clipping remains essential for stability torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) optimizer.step() if step % 100 == 0 and local_rank == 0: print(f"Step {step}: Loss = {loss.item():.4f}")

2. Production Tuning and Operational Best Practices

When operating APOLLO at scale, apply these operational guidelines:

  1. Learning Rate Alignment: Because APOLLO uses structured channel scaling, its optimal learning rate closely mirrors standard AdamW. You can directly reuse existing AdamW learning rate schedules ($3\text{e-}4$ to $5\text{e-}4$) without re-tuning.

  2. Rank Selection Strategy:

    • For Full Pretraining from Scratch: Use rank = 1 (APOLLO-Mini). Empirical testing demonstrates that rank-1 captures over 98% of the Fisher Information preconditioning curvature needed for stable convergence.
    • For Fine-Tuning Domain LLMs: Use rank = 1 or rank = 4 if downstream tasks contain severe activation spikes.

  3. Memory Savings Allocation: Reinvest the VRAM saved by APOLLO into:

    • Doubling or quadrupling batch sizes to maximize Tensor Core utilization.
    • Extending context lengths from 4K to 32K during pretraining without gradient checkpointing.
    • Eliminating offloading to CPU RAM, preventing PCIe bandwidth bottlenecks.

By replacing brute-force parameter tracking with structured random projection, APOLLO dissolves the optimizer memory wall—delivering the lightweight footprint of SGD with the gold-standard convergence of AdamW.