simpollm-alignmentdporeinforcement-learningfine-tuningtutorial

SimPO Guide: Reference-Free LLM Preference Optimization

SimPO Architecture: Reference-Free Alignment, Length-Normalized Implicit Rewards & Target Margins

The Memory and Alignment Paradox in Modern LLM Post-Training

In production AI systems, generative workflows, and enterprise intelligence pipelines powered by AI Fetchly, post-training alignment represents the critical bridge between raw next-token prediction and reliable, instruction-following behavior. Whether training specialized models for code synthesis, financial report analysis, or automated web extraction through real-time data fetching, aligning foundation models with human preferences is mandatory to prevent hallucinations, enforce formatting constraints, and ensure conversational safety.

For years, the standard alignment pipeline relied on Reinforcement Learning from Human Feedback (RLHF) via Proximal Policy Optimization (PPO). However, PPO is notoriously unstable and resource-intensive, requiring the simultaneous maintenance of four separate deep neural networks in GPU High-Bandwidth Memory (HBM): the Actor (policy $\pi_\theta$), the Critic (value model $V_\phi$), the Reward Model ($r_\psi$), and the frozen Reference Model ($\pi_{\text{ref}}$).

The Rise and Hidden Pitfalls of Direct Preference Optimization (DPO)

In 2023, Direct Preference Optimization (DPO) transformed the field by mathematically reparameterizing the RLHF objective. DPO showed that the optimal policy under the Bradley-Terry preference model can be solved analytically, allowing developers to optimize the policy network $\pi_\theta$ directly on paired preference data $(x, y_w, y_l)$ without training a separate reward model or executing online reinforcement learning:

$$\mathcal{L}{\text{DPO}}(\theta) = -\mathbb{E}{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right]$$

The DPO Architectural Overhead: ┌─────────────────────────────────────────────────────────────────┐ │ GPU 1 & 2: Active Policy Model π_θ (Trainable, FP16/BF16) │ │ • Forward Pass (y_w, y_l) │ │ • Backward Pass & Optimizer States (AdamW: 16 bytes/param) │ └─────────────────────────────────────────────────────────────────┘ ▲ │ KL Divergence Penalty ▼ ┌─────────────────────────────────────────────────────────────────┐ │ GPU 3 & 4: Frozen Reference Model π_ref (Non-Trainable, BF16) │ │ • Forward Pass (y_w, y_l) to compute log π_ref │ │ • Consumes 50% of available GPU VRAM solely for log-prob ratios │ └─────────────────────────────────────────────────────────────────┘

While DPO removed the PPO critic and reward models, enterprise engineering teams rapidly encountered two severe operational bottlenecks:

  1. The Reference Model Memory Tax: DPO requires keeping a complete, frozen copy of the supervised fine-tuned (SFT) base model ($\pi_{\text{ref}}$) in GPU memory throughout training. For a 70B parameter model in BF16 precision, loading $\pi_{\text{ref}}$ consumes 140 GB of VRAM purely to evaluate reference log-probabilities. This memory penalty restricts per-device batch sizes, forces aggressive activation checkpointing, and frequently requires double the GPU nodes.
  2. The Discrepancy Between Reward and Generation Metrics: DPO defines its implicit reward as $r(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$. However, during inference, generative LLMs do not decode using this ratio; they decode directly using the token log-probabilities $\log \pi_\theta(y_t|x, y_{<t})$. This mismatch causes gradient degradation and instability.
  3. Severe Length Bias (Verbosity Exploitation): Because DPO’s implicit reward accumulates unnormalized sums of token log-probabilities, longer sequences naturally accumulate higher variance. The optimization process frequently rewards verbose, repetitive responses over concise, factually dense completions—a phenomenon known as “length hacking.”

To resolve these systemic challenges, researchers from Princeton University introduced SimPO (Simple Preference Optimization) at NeurIPS 2024. SimPO presents an elegant, reference-free objective that aligns models using length-normalized average log-likelihoods and target reward margins, slashing GPU memory consumption by 50% while achieving top-tier performance on competitive alignment benchmarks.

Mathematical Foundations: How SimPO Redefines Implicit Rewards

SimPO eliminates the reference model entirely by redefining how the implicit reward function relates to model probabilities.

SimPO Core Architectural Innovations: ┌────────────────────────────────────────────────────────┐ │ 1. Length-Normalized Implicit Reward │ │ r_SimPO(x, y) = (β / |y|) · log π_θ(y|x) │ │ • Averages log-probabilities over sequence length │ │ • Directly matches inference decoding metric │ │ • Prevents verbosity hacking and length bias │ └───────────────────────────┬────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────┐ │ 2. Target Reward Margin (γ > 0) │ │ Enforces: r_SimPO(x, y_w) - r_SimPO(x, y_l) > γ │ │ • Replaces zero-centered sigmoid with margin offset │ │ • Prevents gradient saturation on easy pairs │ │ • Guarantees positive reward gap between answers │ └───────────────────────────┬────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────┐ │ 3. Reference-Free Loss Objective │ │ L_SimPO = -log σ( r_SimPO(y_w) - r_SimPO(y_l) - γ ) │ │ • No frozen π_ref model loaded in GPU VRAM │ │ • 50% memory reduction, 30%+ training speedup │ └────────────────────────────────────────────────────────┘

1. Length-Normalized Implicit Reward Formulation

In autoregressive language generation, the probability of a completion sequence $y = (y_1, y_2, \dots, y_{|y|})$ given prompt $x$ is:

$$\pi_\theta(y|x) = \prod_{t=1}^{|y|} \pi_\theta(y_t | x, y_{<t})$$

Taking the logarithm yields the cumulative log-probability:

$$\log \pi_\theta(y|x) = \sum_{t=1}^{|y|} \log \pi_\theta(y_t | x, y_{<t})$$

In standard DPO, the implicit reward is proportional to the unnormalized sum $\sum_{t=1}^{|y|} \log \pi_\theta(y_t | x, y_{<t})$. As sequence length $|y|$ increases, this sum scales linearly with token count. Consequently, DPO policies easily learn that generating longer texts acts as an effective shortcut to maximize reward.

SimPO solves this by introducing a sequence length-normalized reward:

$$r_{\text{SimPO}}(x, y) = \frac{\beta}{|y|} \log \pi_\theta(y|x) = \frac{\beta}{|y|} \sum_{t=1}^{|y|} \log \pi_\theta(y_t | x, y_{<t})$$

where:

  • $|y|$ is the exact token length of the generated response.
  • $\beta$ is a temperature hyperparameter that scales the reward magnitude (typically $\beta \in [2.0, 2.5]$ in SimPO, compared to $\beta \in [0.05, 0.1]$ in DPO).

By dividing by $|y|$, $r_{\text{SimPO}}(x, y)$ calculates the average log-likelihood per token. This directly aligns the reward with standard evaluation and decoding metrics (such as perplexity and beam search scoring), ensuring that shorter, dense answers compete fairly against longer completions.

2. The Target Reward Margin ($\gamma$)

In the Bradley-Terry preference framework, the probability that completion $y_w$ is preferred over $y_l$ is modeled as:

$$P(y_w \succ y_l | x) = \sigma \left( r(x, y_w) - r(x, y_l) \right)$$

where $\sigma(z) = \frac{1}{1 + e^{-z}}$ is the sigmoid function. Notice that when $r(x, y_w) = r(x, y_l)$, the preference probability is exactly $\sigma(0) = 0.5$.

In DPO, the optimization pushes the implicit reward difference to be greater than zero. However, because SimPO does not use a reference model to anchor the policy’s log-probabilities, naively optimizing $r_{\text{SimPO}}(y_w) - r_{\text{SimPO}}(y_l) > 0$ can cause optimization collapse: the policy can satisfy the inequality by minimally nudging probabilities, leading to vanishing gradients on easy examples and poor margin separation.

To enforce strict, robust discrimination, SimPO incorporates a Target Margin $\gamma > 0$:

$$P(y_w \succ y_l | x) = \sigma \left( r_{\text{SimPO}}(x, y_w) - r_{\text{SimPO}}(x, y_l) - \gamma \right)$$

This formulation enforces that the length-normalized reward of the winning completion $y_w$ must exceed the reward of the losing completion $y_l$ by at least margin $\gamma$:

$$r_{\text{SimPO}}(x, y_w) - r_{\text{SimPO}}(x, y_l) > \gamma$$

3. The Full SimPO Loss Function

Combining the length-normalized reward with the target margin yields the complete SimPO loss:

$$\mathcal{L}{\text{SimPO}}(\theta) = -\mathbb{E}{(x, y_w, y_l)} \left[ \log \sigma \left( \frac{\beta}{|y_w|} \log \pi_\theta(y_w|x) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l|x) - \gamma \right) \right]$$

This objective possesses remarkable mathematical properties:

  1. Fully Reference-Free: No term in the loss references $\pi_{\text{ref}}$.
  2. Analytic Gradient Flow: Gradients are scaled inversely by sequence lengths $|y_w|$ and $|y_l|$, preventing long outlier samples from dominating the backward pass.
  3. Hyperparameter Stability: The margin $\gamma$ acts as a regularization constraint, preventing over-optimization and drift away from the base language modeling capabilities.

SimPO vs. DPO vs. ORPO vs. KTO: Architectural Comparison

To understand where SimPO fits in the alignment ecosystem, it is essential to contrast it against alternative preference optimization methodologies.

Alignment Paradigm Decision Matrix: ┌────────────────────────────────────────────────────────────────────────┐ │ 1. RLHF (PPO): Online policy generation with reward model & critic. │ │ • Pros: Highest theoretical peak for iterative self-play. │ │ • Cons: Extreme memory footprint (4 models), fragile hyperparams. │ ├────────────────────────────────────────────────────────────────────────┤ │ 2. DPO: Offline Bradley-Terry optimization using reference model ratio. │ │ • Pros: Stable, no reinforcement learning actor-critic loops. │ │ • Cons: 2x model VRAM overhead, severe length bias exploitation. │ ├────────────────────────────────────────────────────────────────────────┤ │ 3. ORPO: Monolithic SFT + Odds Ratio penalty without reference model. │ │ • Pros: Combines instruction tuning and alignment in one pass. │ │ • Cons: Odds ratio penalty can destabilize SFT log-likelihoods. │ ├────────────────────────────────────────────────────────────────────────┤ │ 4. SimPO: Reference-free, length-normalized margin optimization. │ │ • Pros: 50% VRAM savings, zero length bias, top benchmark scores. │ │ • Cons: Requires high-quality, pre-converged SFT checkpoint. │ └────────────────────────────────────────────────────────────────────────┘

Detailed Structural Comparison

Feature / DimensionStandard DPOKTO (Kahneman-Tversky)ORPOSimPO (Princeton)
Reference Model $\pi_{\text{ref}}$ Required?Yes (Frozen in VRAM)Yes (Frozen in VRAM)NoNo
Data FormatPaired $(x, y_w, y_l)$Unpaired $(x, y, \text{label})$Paired $(x, y_w, y_l)$Paired $(x, y_w, y_l)$
Implicit Reward Formulation$\beta \log \frac{\pi_\theta}{\pi_{\text{ref}}}$Prospect utility $u(\pi_\theta, \pi_{\text{ref}})$Odds ratio $\frac{\text{odds}\theta(w)}{\text{odds}\theta(l)}$$\frac{\beta}{|y|} \log \pi_\theta(y|x)$
Length Normalization?None (Raw sum)NonePartialExact per-token average
Target Margin ($\gamma$)?None ($\gamma = 0$)NoneNoneConfigurable ($\gamma \in [0.5, 1.4]$)
GPU Memory Footprint$2\times$ Base Weights$2\times$ Base Weights$1\times$ Base Weights$1\times$ Base Weights (Lowest)
Training Step LatencyBaseline ($1.0\times$)Baseline ($1.0\times$)$0.85\times$$0.70\times$ (30% Faster)
Length Exploitation RiskHighHighLowNegligible / Zero

Why ORPO and SimPO Differ

Developers often ask how SimPO differs from ORPO (Odds Ratio Preference Optimization), as both eliminate the reference model.

  • ORPO combines supervised fine-tuning (SFT cross-entropy loss) with an odds-ratio penalty in a single training stage. Because it alters the base language modeling loss, tuning the odds-ratio coefficient is delicate; too high a weight damages perplexity, while too low a weight yields weak alignment.
  • SimPO is a dedicated second-stage alignment algorithm applied to an already converged SFT model. It isolates preference learning into a pure margin-ranking objective, preserving the foundational SFT distribution while establishing a strict margin between preferred and rejected completions.

Hands-On Implementation: PyTorch Loss Module & Hugging Face TRL

Implementing SimPO is straightforward and integrates cleanly into existing post-training pipelines.

1. PyTorch Reference Implementation of the SimPO Loss

The following self-contained PyTorch module implements the exact SimPO loss with sequence length normalization, target margin offset, and numerical stability safeguards:

import torch import torch.nn as nn import torch.nn.functional as F class SimPOLoss(nn.Module): """ Production-grade PyTorch implementation of SimPO (Simple Preference Optimization). Reference: Meng et al., 'SimPO: Simple Preference Optimization with a Reference-Free Implicit Reward' (NeurIPS 2024 Oral). """ def __init__(self, beta: float = 2.0, gamma: float = 0.5): """ Args: beta: Scaling temperature for length-normalized rewards (default: 2.0). gamma: Target reward margin enforcing positive separation (default: 0.5). """ super().__init__() self.beta = beta self.gamma = gamma def forward( self, chosen_logprobs: torch.Tensor, rejected_logprobs: torch.Tensor, chosen_lengths: torch.Tensor, rejected_lengths: torch.Tensor ) -> torch.Tensor: """ Args: chosen_logprobs: Tensor of shape (batch_size,) with cumulative log-probs of y_w. rejected_logprobs: Tensor of shape (batch_size,) with cumulative log-probs of y_l. chosen_lengths: Tensor of shape (batch_size,) with token lengths of y_w. rejected_lengths: Tensor of shape (batch_size,) with token lengths of y_l. Returns: loss: Scalar tensor representing mean SimPO objective across the batch. """ # Ensure lengths are float tensors and avoid division by zero chosen_lengths = chosen_lengths.clamp(min=1.0).float() rejected_lengths = rejected_lengths.clamp(min=1.0).float() # Compute sequence length-normalized implicit rewards # r(x, y) = (beta / |y|) * sum(log pi(y_t | x)) reward_chosen = (self.beta / chosen_lengths) * chosen_logprobs reward_rejected = (self.beta / rejected_lengths) * rejected_logprobs # Calculate reward difference with margin offset # delta = r(y_w) - r(y_l) - gamma reward_delta = reward_chosen - reward_rejected - self.gamma # SimPO loss: -log(sigmoid(reward_delta)) = F.softplus(-reward_delta) # Using softplus for superior numerical stability over log(sigmoid(x)) losses = F.softplus(-reward_delta) return losses.mean() # Verification Example if __name__ == "__main__": batch_size = 4 simpo_loss_fn = SimPOLoss(beta=2.0, gamma=0.8) # Simulated cumulative log-probabilities (negative numbers) # y_w typically has higher log-prob than y_l chosen_lp = torch.tensor([-45.2, -88.1, -24.5, -62.0]) rejected_lp = torch.tensor([-65.8, -112.4, -38.2, -71.5]) # Sequence lengths chosen_len = torch.tensor([30.0, 65.0, 18.0, 42.0]) rejected_len = torch.tensor([45.0, 70.0, 25.0, 40.0]) loss = simpo_loss_fn(chosen_lp, rejected_lp, chosen_len, rejected_len) print(f"Calculated SimPO Batch Loss: {loss.item():.4f}") assert loss.item() > 0.0

2. Training with Hugging Face TRL and CPO/SimPO Trainer

Modern versions of Hugging Face’s TRL (Transformer Reinforcement Learning) and Llama-Factory support SimPO natively via the CPOTrainer / DPOTrainer with loss_type="simpo":

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments from trl import CPOTrainer, CPOConfig from datasets import load_dataset import torch # 1. Load Pre-Trained SFT Model (No reference model needed!) model_id = "meta-llama/Llama-3.1-8B-Instruct" model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", use_cache=False ) tokenizer = AutoTokenizer.from_pretrained(model_id) tokenizer.pad_token = tokenizer.eos_token # 2. Configure SimPO Hyperparameters training_args = CPOConfig( output_dir="./llama-3.1-8b-simpo-aligned", loss_type="simpo", # Select native SimPO objective cpo_alpha=0.0, # 0.0 disables auxiliary BC/NLL loss, pure SimPO beta=2.5, # Reward scaling parameter simpo_gamma=1.4, # Target reward margin (typically 0.5 - 1.4) learning_rate=5e-7, # Low learning rate for stable alignment per_device_train_batch_size=4, gradient_accumulation_steps=4, num_train_epochs=1, bf16=True, gradient_checkpointing=True, logging_steps=10, save_strategy="epoch", warmup_ratio=0.1, lr_scheduler_type="cosine" ) # 3. Load Preference Dataset (e.g., UltraFeedback Binarized) dataset = load_dataset("HuggingFaceH4/ultrafeedback_binarized", split="train_prefs[:5000]") # 4. Instantiate Trainer and Execute Alignment trainer = CPOTrainer( model=model, args=training_args, train_dataset=dataset, tokenizer=tokenizer, max_length=2048, max_prompt_length=1024 ) # Run reference-free alignment trainer.train() trainer.save_model("./llama-3.1-8b-simpo-aligned")

When integrating automated feedback pipelines from our automated web scraping systems and custom API integration, feeding structured preference pairs directly into the SimPO training loop enables automated model fine-tuning without manual labeling friction.

Empirical Benchmarks & Performance Analysis

In the seminal paper by Princeton researchers, SimPO was rigorously evaluated against DPO, KTO, PPO, and ORPO on standard alignment benchmarks: AlpacaEval 2, Arena-Hard-v0.1, and MT-Bench.

Benchmark Results on LLaMA-3-8B-Instruct and Mistral-7B

The following table summarizes empirical performance across leading benchmarks:

Alignment AlgorithmReference Model?AlpacaEval 2 (Length-Controlled Win-Rate)Arena-Hard-v0.1 (Win-Rate vs GPT-4)MT-Bench (Score / 10)Average Response Length (Tokens)
Base SFT CheckpointNone22.6%20.6%7.82412
PPO (Online RLHF)Yes ($\pi_{\text{ref}} + V_\phi$)31.4%25.1%8.04560 (Severe inflation)
DPO ($\beta = 0.1$)Yes ($\pi_{\text{ref}}$)33.5%27.4%8.12510 (Length bias)
KTOYes ($\pi_{\text{ref}}$)32.1%26.2%8.08495
ORPONo31.9%25.8%8.01425
SimPO ($\beta=2.0, \gamma=0.5$)No (Reference-Free)36.8%31.2%8.35418 (Zero inflation)
SimPO ($\beta=2.5, \gamma=1.4$)No (Reference-Free)38.4%33.1%8.41422 (Clean conciseness)

Critical Empirical Insights

  1. Substantial Win-Rate Gains: SimPO achieves an AlpacaEval 2 Length-Controlled Win Rate of 38.4%, outperforming standard DPO (33.5%) by nearly 5 percentage points and PPO (31.4%) by 7 percentage points.
  2. Resistance to Length Hacking: Notice the average response length: while DPO expands average response length from 412 tokens to 510 tokens, and PPO inflates it to 560 tokens, SimPO maintains a disciplined 422 tokens. This demonstrates that SimPO’s length-normalized reward effectively prevents models from exploiting length as a proxy for answer quality.
  3. Hard-Bench Generalization: On Arena-Hard-v0.1 (widely considered the most demanding benchmark for LLM reasoning and professional competence), SimPO scores 33.1%, demonstrating superior adherence to complex technical specifications, math problem-solving, and structured outputs.

Production Engineering: Hyperparameters & Best Practices

Deploying SimPO successfully in enterprise post-training pipelines requires understanding its core hyperparameters and avoiding common training pitfalls.

Production Training Workflow: ┌─────────────────────────┐ ┌───────────────────────────────┐ │ Clean SFT Base Model │ ──► │ High-Quality Preference Pairs │ └─────────────────────────┘ └──────────────┬────────────────┘ │ ▼ ┌───────────────────────────────┐ │ Hyperparameter Calibration │ │ • Set β ∈ [2.0, 2.5] │ │ • Set γ ∈ [0.5, 1.4] │ │ • Learning Rate: 3e-7 to 8e-7 │ └──────────────┬────────────────┘ │ ▼ ┌───────────────────────────────┐ │ Reference-Free SimPO Training │ │ • Single Model in HBM │ │ • 50% Higher Batch Size │ └──────────────┬────────────────┘ │ ▼ Aligned Production Model

1. Calibrating Temperature $\beta$ and Target Margin $\gamma$

Unlike DPO where $\beta$ is typically chosen from $[0.01, 0.1]$, SimPO uses a length-normalized log-probability, meaning the raw numerical magnitude of the reward is smaller.

  • Recommended $\beta$: Set $\beta \in [2.0, 2.5]$. Setting $\beta < 1.0$ leads to under-confident gradients, while $\beta > 3.0$ can cause gradient explosion.
  • Recommended Margin $\gamma$: Set $\gamma \in [0.5, 1.4]$.
    • For general chat models where response styles vary widely, $\gamma = 0.5$ provides gentle, stable separation.
    • For coding, mathematical reasoning, and deterministic structured extraction, $\gamma = 1.4$ forces strict discrimination between bug-free and flawed completions.

2. Learning Rate Discipline

Because SimPO does not have a reference model acting as an explicit KL-divergence anchor, running with excessive learning rates will cause the policy to drift away from natural language fluencies.

  • Safe Learning Rate: Keep learning rates between $3 \times 10^{-7}$ and $8 \times 10^{-7}$. Never exceed $1 \times 10^{-6}$.
  • Use a cosine decay schedule with a 10% warmup phase.
  • Train for 1 to 2 epochs maximum. Over-training beyond 2 epochs on preference data leads to reward collapse and repetitive generation loops.

3. Data Filtering and Margin Hygiene

Ensure preference pairs exhibit genuine quality differentials. If $y_w$ and $y_l$ are virtually identical, enforcing a strict margin $\gamma = 1.4$ forces the model to invent arbitrary syntactic distinctions. Filter out preference pairs where the token edit distance between winning and losing answers is under 5%.

For enterprise organizations evaluating infrastructure requirements to scale alignment workloads, review our flexible pricing tiers for cost-optimized GPU compute management.

To explore real-world integration architectures and deployment blueprints, consult our comprehensive developer documentation.

Frequently Asked Questions (FAQ)

Why doesn’t SimPO need a reference model like DPO?

In DPO, the reference model $\pi_{\text{ref}}$ is mathematically required to anchor the policy and prevent it from drifting arbitrarily far from the initial SFT distribution via an implicit KL penalty. SimPO achieves distribution stability through two alternative mechanisms: first, it normalizes log-probabilities by sequence length to stabilize per-token scale; second, it introduces an explicit positive margin $\gamma$ that bounds the target separation without needing a second model. Empirical testing shows that SimPO maintains perplexity and benchmark capability without the reference anchor.

Can SimPO be used for reasoning models like DeepSeek-R1 or OpenAI o1 style chains?

Yes. In fact, SimPO is especially advantageous for reasoning and chain-of-thought (CoT) alignment because reasoning traces vary drastically in length (from 500 tokens to 8,000+ tokens). Under standard DPO, models exploit long reasoning chains simply to rack up unnormalized reward, producing bloated, repetitive internal monologues. SimPO’s length-normalized reward ensures that concise, logically sound reasoning steps receive higher reward density than rambling, redundant chains.

Does SimPO replace Supervised Fine-Tuning (SFT)?

No. SimPO is strictly an alignment algorithm, not a replacement for SFT. You must first train your base model on high-quality instruction-response pairs via SFT until convergence. Once the model knows how to follow instructions and format answers, SimPO is applied in the second stage on paired preference data to align its choices with human values, conciseness, and precision.

What are the hardware savings when switching from DPO to SimPO?

By dropping the frozen reference model, SimPO reduces memory consumption by approximately 45% to 50%. For example, aligning a 70B parameter model in 16-bit precision that previously required eight 80GB NVIDIA A100/H100 GPUs can now comfortably fit on four GPUs with the same per-device micro-batch size. In addition, eliminating the forward pass for $\pi_{\text{ref}}$ yields a 30%+ speedup in training wall-clock time.