reinforcement-learningreasoning-modelsacpopolicy-optimizationcredit-assignmentdeepseek-r1llm-training

Asymmetric Credit Policy Optimization (ACPO) Guide

Asymmetric Credit Policy Optimization (ACPO) Architecture: Mode-Local Entropy Surrogate, Mismatch Routing, and Asymmetric Token-Level Credit Assignment

Reinforcement Learning with Verifiable Rewards (RLVR)—popularized by models such as DeepSeek-R1 and OpenAI-o1—has emerged as the premier post-training methodology for frontier foundation models. In autonomous software engineering pipelines, enterprise mathematical problem solving, and complex structured extraction tasks powered by AI Fetchly, RLVR incentivizes language models to generate extensive Chain-of-Thought (CoT) trajectories, self-correcting mistakes and verifying intermediate deductions.

However, standard policy gradient algorithms (such as PPO and GRPO) operate primarily at the trajectory level. When an entire CoT sequence of 8,000 tokens receives a binary outcome reward (e.g., $+1$ for correct or $0$ for wrong), the algorithm applies an identical scalar advantage across every single token in the sequence. This creates a severe credit assignment problem: on successful trajectories, routine connective syntax is rewarded equally with pivotal breakthrough insights; on failed trajectories, trivial punctuation tokens are penalized identically with the fatal conceptual hallucination that initially derailed the proof.

To achieve token-level granularity, recent research explored entropy-aware exploration methods. Yet, as revealed by researchers from Peking University and Baidu in ACPO: Asymmetric Credit Policy Optimization via Mode-Local Entropy Surrogate (arXiv:2607.03126), standard entropy introduces fatal flaws. Full-vocabulary Shannon entropy is corrupted by long-tail distribution noise over large vocabularies ($|\mathcal{V}| > 128\text{k}$), and uncertainty carries fundamentally asymmetric meanings across success and failure trajectories: on successful rollouts, uncertainty marks creative exploration among plausible paths, whereas on failed rollouts, high entropy is merely the confused aftermath following an earlier overconfident error.

To resolve these challenges, ACPO introduces two foundational breakthroughs: replacing global entropy with a mode-local surrogate ($\delta = 1 - \pi_\theta(v^*)$), and enforcing asymmetric credit assignment that rewards uncertain exploration on positive trajectories while heavily penalizing confident mistakes on failed ones. Coupled with mismatch routing to prevent off-target gradient pollution, ACPO delivers consistent state-of-the-art gains across AIME 2024/2025 and HumanEval Pro.

In this technical guide, we break down why symmetric entropy-aware RL fails, formalize the mathematics of the mode-local entropy surrogate and asymmetric multipliers, walk through a complete PyTorch implementation, and evaluate training benchmarks.

The Credit Assignment Trap in Trajectory-Level RL

To understand why conventional RLVR algorithms plateau on complex reasoning tasks, we must analyze the temporal anatomy of a reasoning failure.

The Anatomy of Reasoning Divergence

Consider a 5,000-token Chain-of-Thought solving a competitive Olympiad geometry problem. The trajectory unfolds in three distinct phases:

  1. Initial Setup (Tokens 1 to 1,200): The model correctly defines coordinates, asserts verified theorems, and maintains low entropy (high confidence).
  2. The Divergence Point (Token 1,201): The model makes an ungrounded assumption or miscalculates an algebraic sign. Crucially, the model is highly confident in this erroneous step (assigning $>95%$ probability to the false deduction).
  3. Downstream Aftermath (Tokens 1,202 to 5,000): Having introduced a mathematical contradiction, the model encounters algebraic inconsistencies. Its generation becomes increasingly uncertain, meandering, and hesitant, leading to a sustained explosion in token-level Shannon entropy.
Entropy Profile Across a Failed Reasoning Trajectory: ┌────────────────────────────────────────────────────────────────────────┐ │ Phase 1: Correct Setup (Low Entropy, ~0.08 nats) │ ├────────────────────────────────────────────────────────────────────────┤ │ Token 1,201: FATAL OVERCONFIDENT ERROR! (Low Entropy, ~0.05 nats) │ ├────────────────────────────────────────────────────────────────────────┤ │ Phase 3: Confused Downstream Aftermath (High Entropy, ~0.89 nats) │ └───────────────────────────────────┬────────────────────────────────────┘ │ ┌───────────────────────────────┴───────────────────────────────┐ ▼ ▼ ┌─────────────────────────────────┐ ┌─────────────────────────────────┐ │ Symmetric Entropy-Aware RL │ │ ACPO Asymmetric Policy Updates │ ├─────────────────────────────────┤ ├─────────────────────────────────┤ │ • Rewards/penalizes high- │ │ • Recognizes asymmetry: │ │ entropy tokens symmetrically. │ │ - On Failure: High entropy is │ │ • Penalizes Phase 3 heavily │ │ downweighted! │ │ (the symptom, not the cause). │ │ - Penalizes Token 1,201 │ │ ❌ Fails to suppress the root │ │ (the confident error)! │ │ cause error at Token 1,201! │ │ ✔ Precise root-cause credit! │ └─────────────────────────────────┘ └─────────────────────────────────┘

Under standard symmetric entropy rules (such as the 80/20 rule or GTPO), high entropy is uniformly interpreted as a marker of critical decision-making. Consequently, the algorithm concentrates negative updates on the confused aftermath tokens (Phase 3) while barely penalizing the confident root-cause mistake at Token 1,201.

When training multi-step autonomous agents that coordinate with data fetching APIs, misdirected credit assignment causes models to become indecisive rather than correcting underlying reasoning flaws. A principled, asymmetric formulation is required.

Mathematical Foundations of ACPO

Asymmetric Credit Policy Optimization (ACPO) addresses this credit assignment failure through two core mathematical innovations: replacing full-vocabulary entropy with a bounded mode-local surrogate, and deploying asymmetric advantage multipliers.

The Mode-Local Entropy Surrogate

Given policy $\pi_\theta$, the true Shannon entropy at sequence position $t$ is defined across the entire vocabulary $\mathcal{V}$:

$$H_{t}(\theta) = -\sum_{v \in \mathcal{V}} \pi_\theta(v) \log \pi_\theta(v)$$

In frontier models where $|\mathcal{V}| > 128,000$, the long tail of low-probability tokens introduces high variance into both the entropy value and its gradients. Furthermore, differentiating full entropy produces non-local gradient terms that pull updates through thousands of unvisited tokens.

ACPO replaces Shannon entropy with the mode-local surrogate $\delta_{t} \in [0, 1]$, defined as the total probability mass residing outside the dominant greedy mode $v_t^* = \arg\max_v \pi_\theta(v)$:

$$\delta_t = 1 - \pi_\theta(v_t^*)$$

Xie et al. prove that $\delta_t$ forms a strict monotonic envelope bounding Shannon entropy:

$$-\log(1 - \delta_t) \le H_t \le -(1 - \delta_t) \log(1 - \delta_t) - \delta_t \log\left(\frac{\delta_t}{|\mathcal{V}| - 1}\right)$$

Because $\delta_t$ depends solely on the dominant mode probability $\pi_\theta(v_t^*)$, it is completely immune to long-tail vocabulary noise, bounded strictly within $[0, 1]$, and eliminates non-local gradient dispersion.

Asymmetric Credit Assignment Multipliers

With the mode-local proxy $\delta_t$ established, ACPO parameterizes the token-level policy gradient update via asymmetric credit multipliers $c_t$ based on the sign of the sequence advantage $\hat{A}$:

$$\nabla_\theta \mathcal{L}{\text{ACPO}} = \mathbb{E}\left[ \sum{t=1}^{|y|} c_t \cdot \hat{A} \cdot \nabla_\theta \log \pi_\theta(y_t \mid x, y_{<t}) \right]$$

  1. Positive Advantage Branch ($\hat{A} > 0$): On successful trajectories, high uncertainty indicates that the model explored a non-trivial, creative decision path that ultimately succeeded. To encourage fruitful exploration, ACPO scales the update by:

$$c_t^+ = 2 + \delta_t$$

The additive offset of $2$ is derived from leading-order product-rule gradient analysis to compensate for the opposing proxy derivative, ensuring that updates in confident regions remain strictly positive and never flip sign.

  1. Non-Positive Advantage Branch ($\hat{A} \le 0$): On failed trajectories, high uncertainty indicates downstream confusion, whereas high confidence indicates the fatal root-cause error. Therefore, ACPO concentrates negative penalties on confident tokens:

$$c_t^- = 1 - \delta_t = \pi_\theta(v_t^*)$$

Under this formula, tokens generated with high confidence ($\pi_\theta(v^) \to 1.0$) receive maximum negative suppression ($c_t^- \to 1.0$), while downstream confused tokens ($\pi_\theta(v^) \to 0.1$) receive negligible penalties ($c_t^- \to 0.1$).

Mismatch Routing for Mode-Misaligned Tokens

When the sampled token $y_t$ differs from the greedy mode ($y_t \ne v_t^$), differentiating the mode-local surrogate $\delta_t$ would backpropagate gradients through the un-sampled mode $v_t^$, creating off-target gradient conflict. ACPO introduces mismatch routing, applying a stop-gradient operator (sg) on misaligned tokens:

$$c_t = \begin{cases} 2 + \delta_t & \text{if } y_t = v_t^* \text{ and } \hat{A} > 0 \ 1 - \delta_t & \text{if } y_t = v_t^* \text{ and } \hat{A} \le 0 \ 3 \cdot \text{sg}[\delta_t] & \text{if } y_t \ne v_t^* \text{ and } \hat{A} > 0 \ 3 \cdot (1 - \text{sg}[\delta_t]) & \text{if } y_t \ne v_t^* \text{ and } \hat{A} \le 0 \end{cases}$$

This safeguard guarantees that the composite policy gradient remains an exact non-negative scalar multiple of the local policy gradient in both logit and parameter space, eliminating gradient reversal.

The ACPO Token-Level Credit Allocation Logic: ┌────────────────────────────────────────────────────────────────────────┐ │ Trajectory Advantage Sign: Positive (Success) vs. Negative (Failure) │ └───────────────────────────────────┬────────────────────────────────────┘ │ ┌─────────────────────────┴─────────────────────────┐ ▼ ▼ ┌───────────────────────────────────┐ ┌───────────────────────────────────┐ │ Positive Advantage (A > 0): │ │ Non-Positive Advantage (A <= 0): │ │ Objective: Reward Exploratory CoT │ │ Objective: Suppress Root Error │ ├───────────────────────────────────┤ ├───────────────────────────────────┤ │ Multiplier: c+ = 2 + δ │ │ Multiplier: c- = 1 - δ = π(v*) │ │ • High Uncertainty (δ high): │ │ • High Confidence (π high): │ │ Maximum positive reinforcement! │ │ Maximum negative suppression! │ │ • Low Uncertainty (δ low): │ │ • High Uncertainty (π low): │ │ Baseline reinforcement (offset) │ │ Downweighted aftermath penalty! │ └───────────────────────────────────┘ └───────────────────────────────────┘

Architectural Comparison: ACPO vs. Leading RL Frameworks

The table below compares ACPO against classical and contemporary reinforcement learning algorithms used for LLM reasoning post-training:

AlgorithmCredit Assignment GranularityUncertainty SignalHandles Mismatch ConflictsTraining StabilityAIME Benchmark Gain
PPO (Schulman et al.)Trajectory-Level AdvantageGlobal Entropy BonusNo (Global Additive)ModerateBaseline (1.0x)
GRPO (DeepSeek-R1)Group Normalized OutcomeNone (Outcome Supervised)N/AHighStrong Baseline
80/20 Rule (Entropy-Aware)Top-20% High Entropy TokensFull-Vocabulary Shannon HNoLow (Gradient Jitter)+1.8%
SAPO (Soft Adaptive)Gated Advantage ScalingSigmoid Confidence GatePartialModerate+2.9%
ACPO (arXiv:2607.03126)Asymmetric Token-LevelMode-Local Surrogate (δ)Yes (Mismatch Routing)Very High+5.4% - +6.8%

Unlike GRPO which treats all tokens uniformly, ACPO concentrates policy updates exactly where learning signal is densest, eliminating the noisy gradient interference that causes reasoning policy degradation.

End-to-End Implementation: Building an ACPO Loss Module

Let us construct a clean, production-ready PyTorch implementation of the ACPO loss function, featuring mode-local surrogate computation, asymmetric credit weighting, and mismatch routing.

import torch import torch.nn as nn import torch.nn.functional as F from typing import Tuple class ACPOLoss(nn.Module): """ Asymmetric Credit Policy Optimization (ACPO) Loss Module. Implements mode-local entropy surrogate weighting and mismatch routing for fine-grained token-level credit assignment in reasoning LLMs. """ def __init__(self, clip_ratio: float = 0.2): super().__init__() self.clip_ratio = clip_ratio def forward( self, policy_logits: torch.Tensor, ref_logits: torch.Tensor, action_tokens: torch.Tensor, advantages: torch.Tensor, mask: torch.Tensor ) -> Tuple[torch.Tensor, torch.Tensor]: """ Args: policy_logits: [Batch, SeqLen, VocabSize] current actor logits. ref_logits: [Batch, SeqLen, VocabSize] reference model logits. action_tokens: [Batch, SeqLen] sampled response token IDs. advantages: [Batch] sequence-level advantage scalar. mask: [Batch, SeqLen] boolean mask for response tokens. Returns: Tuple of (loss scalar, mean_delta_uncertainty). """ # 1. Compute probabilities and dominant modes policy_probs = F.softmax(policy_logits, dim=-1) mode_probs, mode_tokens = torch.max(policy_probs, dim=-1) # [Batch, SeqLen] # Mode-local entropy surrogate: delta = 1 - pi(v*) delta = (1.0 - mode_probs).clamp(min=1e-6, max=1.0 - 1e-6) # 2. Check token alignment with greedy mode: y_t == v_t* is_aligned = (action_tokens == mode_tokens) # [Batch, SeqLen] # 3. Expand sequence advantages to token level: [Batch, SeqLen] token_adv = advantages.unsqueeze(-1).expand_as(action_tokens) is_pos = (token_adv > 0.0) # 4. Asymmetric Credit Multipliers with Mismatch Routing # Aligned branch c_aligned_pos = 2.0 + delta c_aligned_neg = 1.0 - delta c_aligned = torch.where(is_pos, c_aligned_pos, c_aligned_neg) # Misaligned branch: use detached stop-gradient scale delta_detached = delta.detach() c_misaligned_pos = 3.0 * delta_detached c_misaligned_neg = 3.0 * (1.0 - delta_detached) c_misaligned = torch.where(is_pos, c_misaligned_pos, c_misaligned_neg) # Composite credit multiplier credit_weights = torch.where(is_aligned, c_aligned, c_misaligned) # 5. Policy log-ratio and PPO-style clipping log_policy = F.log_softmax(policy_logits, dim=-1) log_ref = F.log_softmax(ref_logits, dim=-1) token_log_policy = log_policy.gather(-1, action_tokens.unsqueeze(-1)).squeeze(-1) token_log_ref = log_ref.gather(-1, action_tokens.unsqueeze(-1)).squeeze(-1) ratio = torch.exp(token_log_policy - token_log_ref.detach()) # Modulate advantage with ACPO credit weights modulated_adv = credit_weights * token_adv # Clipped surrogate objective surr1 = ratio * modulated_adv surr2 = torch.clamp(ratio, 1.0 - self.clip_ratio, 1.0 + self.clip_ratio) * modulated_adv policy_loss = -torch.min(surr1, surr2) # Apply mask and compute final loss masked_loss = (policy_loss * mask).sum() / mask.sum().clamp(min=1.0) mean_delta = (delta * mask).sum() / mask.sum().clamp(min=1.0) return masked_loss, mean_delta

Functional Validation Harness

Let us verify the loss module by simulating a training step across positive and negative advantage sequences:

def verify_acpo_module(): batch_size = 2 seq_len = 8 vocab_size = 100 # Mock policy and reference logits policy_logits = torch.randn(batch_size, seq_len, vocab_size, requires_grad=True) ref_logits = policy_logits.clone().detach() action_tokens = torch.randint(0, vocab_size, (batch_size, seq_len)) advantages = torch.tensor([1.5, -1.2]) # Sequence 0: Positive, Sequence 1: Negative mask = torch.ones(batch_size, seq_len) print('Initializing ACPO Loss Module...') acpo = ACPOLoss(clip_ratio=0.2) loss, avg_delta = acpo(policy_logits, ref_logits, action_tokens, advantages, mask) print(f"ACPO Loss: {loss.item():.4f}, Mean Uncertainty Proxy (delta): {avg_delta.item():.4f}") loss.backward() assert policy_logits.grad is not None print(f"Gradient Norm on Actor Logits: {policy_logits.grad.norm().item():.4f}") print('Validation Passed: ACPO forward and backward pass successful!') if __name__ == '__main__': verify_acpo_module()

When building multi-step reasoning copilots that coordinate with custom backend endpoints, training models with advantage-aligned credit assignment ensures robust generalization on complex tool-calling and API-orchestration benchmarks.

Production Benchmarks & Empirical Evaluation

In the empirical evaluation presented by Xie et al. (arXiv:2607.03126), ACPO was benchmarked across Qwen2.5-7B-Base and Qwen3-8B-Base on competitive mathematical reasoning and coding benchmarks: AIME 2024, AIME 2025, MATH 500, and HumanEval Pro.

Reasoning Accuracy Across Model Families

The table below reports Mean@8 on mathematical benchmarks and Pass@1 on HumanEval Pro after RL post-training on 8x NVIDIA H200 GPUs:

Base Model & RL FrameworkAIME 2024 (Mean@8)AIME 2025 (Mean@8)MATH 500HumanEval Pro (Pass@1)
Qwen2.5-7B (Base)16.7%12.5%68.2%51.2%
Qwen2.5-7B + GRPO24.2%19.8%76.4%61.5%
Qwen2.5-7B + 80/20 Rule25.0%20.4%77.1%62.0%
Qwen2.5-7B + SAPO26.8%22.1%78.5%63.8%
Qwen2.5-7B + ACPO (2026)31.2% (+4.4%)26.9% (+4.8%)81.4%68.2% (+4.4%)
:---:---:---:---:---
Qwen3-8B (Base)22.4%18.2%74.5%58.0%
Qwen3-8B + GRPO32.5%27.4%82.0%67.4%
Qwen3-8B + SAPO35.1%29.8%83.6%69.1%
Qwen3-8B + ACPO (2026)39.8% (+4.7%)34.2% (+4.4%)86.2%74.5% (+5.4%)
AIME 2024 Performance Gain (Qwen3-8B Post-Training): Base Checkpoint: [█████ ] 22.4% GRPO (DeepSeek-R1):[████████ ] 32.5% SAPO: [█████████ ] 35.1% ACPO (This Work): [██████████ ] 39.8% (SOTA +4.7% over SAPO!)

ACPO consistently outperforms both entropy-aware methods and outcome-supervised RL baselines, demonstrating that asymmetric credit assignment directly enhances high-difficulty mathematical deduction.

Training Dynamics and Entropy Stability

A critical advantage of ACPO is its training stability: whereas standard Shannon entropy bonuses cause exploration entropy to collapse or explode uncontrollably, the mode-local surrogate maintains smooth, bounded exploration entropy throughout 2,000+ training iterations, enabling models to explore deeper reasoning chains without degenerative repetition.

Organizations evaluating infrastructure costs for post-training cluster deployments can explore flexible pricing structures to model compute budgets across distributed GPU clusters.

Deployment Guide: Integrating ACPO with OpenRLHF and Ray

ACPO can be integrated into high-throughput distributed RL engines like OpenRLHF or Hugging Face TRL:

# Launch distributed ACPO training on an 8x H200 GPU node ray start --head python3 -m openrlhf.cli.train_ppo \ --actor_model_path Qwen/Qwen3-8B-Base \ --reward_model_path math_verifier \ --custom_loss_fn acpo \ --tau_pos 1.0 \ --tau_neg 1.05 \ --train_batch_size 128 \ --micro_train_batch_size 16 \ --max_samples 100000 \ --output_dir ./qwen3_acpo_checkpoints

Engineering teams can consult the full developer documentation for detailed container recipes, hyperparameter tuning guides, and multi-node Ray cluster configuration manifests.

Key Takeaways and Architectural Summary

  1. The Credit Assignment Solution: ACPO solves the trajectory-level credit assignment dilemma by allocating fine-grained credit at the individual token level.
  2. Mode-Local Uncertainty Proxy: Replaces noisy full-vocabulary Shannon entropy with the probability mass outside the greedy mode ($\delta = 1 - \pi_\theta(v^*)$), providing a bounded, monotone surrogate.
  3. Asymmetric Learning Principle: Treats uncertainty asymmetrically: reinforcing creative exploration ($c^+ = 2 + \delta$) on successful paths while heavily suppressing overconfident root errors ($c^- = 1 - \delta$) on failed ones.
  4. Mismatch Routing Safeguard: Detaches proxy gradients when sampled tokens diverge from the policy mode, eliminating off-target gradient conflicts.
  5. Empirical Reasoning Superiority: Achieves state-of-the-art results on AIME 2024/2025 and HumanEval Pro, outperforming PPO, GRPO, and SAPO.

Asymmetric Credit Policy Optimization represents a major evolutionary step in reinforcement learning for reasoning LLMs, establishing a robust foundation for next-generation frontier intelligence.