rStar-Math: Small LLMs Surpass OpenAI o1 in Math via MCTS

The High-Compute Monopoly of Reasoning Models

The rise of test-time compute scaling—epitomized by proprietary frontier systems like OpenAI o1 and commercial reasoning APIs—has demonstrated that models can solve intricate mathematical, coding, and multi-step scientific problems when allocated extended thinking time. However, this breakthrough has historically come with an exorbitant operational footprint:

1. Parameter Bloat and Hosting Costs: Frontier reasoning systems rely on frontier scale (often exceeding hundreds of billions of parameters or dense MoE clusters). Deploying these architectures in enterprise production requires massive GPU clusters, resulting in prohibitive serving costs, high carbon overhead, and multi-second latency bottlenecks.
2. Knowledge Distillation Dependency: Open-source attempts to reproduce long-chain reasoning typically rely on distilling outputs directly from closed proprietary frontier models. While distillation yields short-term gains, it caps performance below the teacher, introduces legal vulnerabilities, and fails to instill genuine cognitive exploration within the student model.
3. The Brittle Hallucination of Pure Natural Language CoT: Pure natural language Chain-of-Thought (CoT) frequently succumbs to cumulative arithmetic errors, algebraic slip-ups, and logical drift. When an agent derives an equation across 30 sequential steps, a single computational error on step 4 irreparably contaminates the entire remainder of the trajectory.

For enterprise automation platforms—whether executing financial analytics, orchestrating distributed extractions via automated web scraping, or executing data pipelines through real-time data fetching—enterprises require reasoning models that are lightweight, verifiable, cost-efficient, and capable of running locally or on edge infrastructure.

Can a small, 3-billion to 7-billion parameter language model match or exceed the mathematical reasoning capability of frontier giants like OpenAI o1-preview, without relying on proprietary distillation?

Enter rStar-Math: Self-Evolved Deep Thinking via MCTS

In January 2025, researchers at Microsoft Research published rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (arXiv:2501.04519). The research provides a decisive affirmative answer to the small-model reasoning challenge:

• State-of-the-Art Benchmark Scores: rStar-Math boosts Qwen2.5-Math-7B from 58.8% to 90.0% on the competitive MATH benchmark, surpassing OpenAI o1-preview (85.5%) by +4.5%.
• Edge-Scale Superiority: On Phi3-mini-3.8B, rStar-Math elevates accuracy from 41.4% to 86.4%, outperforming o1-preview by +0.9% on a compact 3.8B footprint.
• Olympiad-Grade Performance: On the prestigious American Invitational Mathematics Examination (AIME), rStar-Math solves 53.3% (8 out of 15 problems), matching the performance of the top 20% of elite high school competition students.

Crucially, rStar-Math achieves these milestones from scratch, without distilling reasoning traces from frontier commercial models.

The Three Structural Pillars of rStar-Math

The architecture of rStar-Math rests upon three core innovations:

1. Code-Augmented Chain-of-Thought (Code-CoT): Rather than forcing the small language model to perform arithmetic, matrix operations, or large-integer factorization purely in token space, the policy SLM is trained to interleave natural language formulation with executable Python snippets. A secure sandbox executes the code, and the output is fed back into the reasoning context as a ground-truth observation.
2. Process Preference Model (PPM) Instead of Naive PRMs: Standard Process Reward Models (PRMs) attempt to assign scalar scores to intermediate reasoning steps using human annotations or LLM-as-a-judge prompts. These scores suffer from severe variance and calibration error. rStar-Math introduces the Process Preference Model, which frames step verification as a pairwise ranking task derived from empirical MCTS trajectory completion rates.
3. Four-Round Self-Evolution Recipe: The policy SLM and the PPM co-evolve iteratively. In each round, the policy model and PPM guide MCTS to solve increasingly difficult mathematical problems across a corpus of 747,000 tasks, synthesizing verified training data that fine-tunes both models for the subsequent round.

Mathematical Foundations and Architecture

To understand how rStar-Math achieves such extraordinary performance on compact models, we analyze the mathematical formulation of its Monte Carlo Tree Search, the pairwise objective of the Process Preference Model, and the self-evolution dynamics.

1. Test-Time Monte Carlo Tree Search (MCTS)

At inference time, rStar-Math treats problem-solving as search over a directed tree T = (V, E), where:

• Each node s in V represents an intermediate reasoning state (a prefix of natural language steps and code execution results).
• Each edge a = (s, s’) in E represents an action: generating the next reasoning step and executing any enclosed Python code block.

The search policy navigates the tree via the Upper Confidence Bound for Trees (UCT): a* = argmax_a [ Q(s, a) + c_puct * P(s, a) * ( sqrt(N(s)) / (1 + N(s, a)) ) ]

where:

• Q(s, a) is the action-value estimate derived from terminal solution verification and PPM step evaluations.
• P(s, a) is the prior probability assigned by the policy SLM.
• N(s) is the total visit count of parent state s.
• N(s, a) is the visit count of child state s’.
• c_puct is the exploration constant.

2. The Process Preference Model (PPM) Formulation

Conventional step-level PRMs attempt to minimize mean squared error against binary step labels: L*{PRM}(theta) = E*{(s_t, y_t)} [ (R_theta(s_t) - y_t)^2 ]

where y_t in {0, 1} indicates whether step t eventually led to a correct solution. However, because a correct terminal answer can sometimes emerge despite an erroneous intermediate step (false positives), or a flawless intermediate step can be abandoned due to subsequent mistakes (false negatives), scalar step regression exhibits extreme variance.

rStar-Math re-conceptualizes intermediate evaluation as a pairwise ranking objective. Let st^+ and s_t^- denote two competing reasoning step extensions originating from the identical prefix state s{<t}. From MCTS rollouts, we calculate the empirical success rate (win rate) of completing the problem from each branch: w(s_t) = N_correct(s_t) / N_total(s_t)

When w(st^+) - w(s_t^-) > delta (where delta is a significance margin, typically 0.2), (s_t^+, s_t^-) forms a preference pair. The Process Preference Model r_phi is trained via the Bradley-Terry preference objective: L{PPM}(phi) = - E*{(s*{<t}, st^+, s_t^-)} [ log sigma( r_phi(s{<t}, st^+) - r_phi(s{<t}, s_t^-) ) ]

where sigma(z) = 1 / (1 + exp(-z)) is the standard logistic sigmoid.

This pairwise formulation eliminates arbitrary scalar calibrations, focusing the reward model exclusively on distinguishing between promising and unpromising exploratory directions.

3. The 4-Round Iterative Self-Evolution Loop

The self-evolution engine scales model capability across four progressive rounds:

1. Round 0 (Bootstrap Initialization):
   • Fine-tune base SLMs on existing open math datasets augmented with Python tool-use examples.
   • Train an initial PPM on small-scale MCTS trajectories.
2. Round 1 (Guided Rollout on 747k Problems):
   • Execute MCTS on 747,000 math problems using Policy-0 and PPM-0.
   • Extract verified successful paths to create SFT Dataset-1.
   • Extract preference pairs from branch visit statistics to create Preference Dataset-1.
3. Round 2 (Dual Model Evolution):
   • Fine-tune Policy-1 on SFT Dataset-1.
   • Train PPM-1 on Preference Dataset-1.
   • Noticeable leap on Olympiad-grade questions as the models self-correct early false paths.
4. Round 3 & 4 (High-Order Convergence):
   • Repeat MCTS generation on previously unsolved tasks with deeper exploration limits (N_simulations = 64 to 128).
   • Terminal convergence: Qwen2.5-Math-7B reaches 90.0% on MATH, solving 8 out of 15 AIME problems.

Production Implementation: MCTS Node, PPM Loss, and REPL Execution

To illustrate the technical mechanics of rStar-Math, we present an end-to-end PyTorch and Python implementation. The code below contains:

1. The PythonSandbox executing code snippets generated by the SLM.
2. The ProcessPreferenceLoss module training the PPM using pairwise logistic ranking.
3. The MCTSNode and RStarMCTSSearch classes coordinating tree expansion, UCT calculation, and intermediate verification.

Architectural Walkthrough

1. Deterministic Execution Interleaving: Whenever the policy model emits a code block enclosed in Python delimiters, the PythonSandbox intercepts execution in an isolated sub-process. This ensures that calculations (such as modular exponentiation, SymPy factorization, or matrix inversion) are ground truth, completely eliminating arithmetic hallucinations.
2. UCT with PPM Hybrid Scoring: The uct_score method integrates both dynamic visit rewards ($Q(s, a)$) and the static ppm_score from the Process Preference Model. This prevents the tree search from getting trapped in dead-end branches.
3. PPM Bradley-Terry Optimization: By supervising the difference between preferred and dispreferred branches (r_pos - r_neg), the preference model learns step calibration without arbitrary human annotations.

Empirical Evaluation and Benchmark Analysis

In Microsoft Research’s comprehensive benchmarks, rStar-Math was evaluated against industry-leading models on challenging competition mathematics datasets.

Primary Benchmark Results

Model Architecture	Parameter Scale	Distilled from o1?	MATH Benchmark (%)	AIME 2024 (Solved / 15)	GSM8K (%)
GPT-4o (Zero-Shot)	Frontier (~200B+)	No	76.6%	13.3% (2/15)	95.8%
OpenAI o1-preview	Frontier MoE	No	85.5%	40.0% (6/15)	94.4%
Phi-3-mini-3.8B (Base)	3.8 Billion	No	41.4%	0.0% (0/15)	82.5%
Phi-3-mini + rStar-Math	3.8 Billion	No	86.4% (+45.0%)	46.7% (7/15)	96.2%
Qwen2.5-Math-7B (Base)	7.0 Billion	No	58.8%	13.3% (2/15)	88.2%
Qwen2.5-Math-7B + rStar	7.0 Billion	No	90.0% (+31.2%)	53.3% (8/15)	97.8%

Key Analytical Takeaways

1. Surpassing Frontier Models at 1/20th the Scale: rStar-Math on Qwen2.5-Math-7B outperforms OpenAI o1-preview by +4.5% on MATH (90.0% vs. 85.5%) and outperforms it on AIME (53.3% vs. 40.0%). This decisively refutes the premise that competitive math reasoning is exclusive to proprietary frontier models.
2. Self-Evolution Over Distillation: Prior open-source attempts to mirror o1 relied on training on OpenAI o1 outputs. rStar-Math proves that autonomous MCTS exploration over 747k problems enables small models to discover superior, verified reasoning paths organically.
3. Compute Efficiency: Because the policy model is only 3.8B or 7B parameters, executing 32 to 64 MCTS simulations requires significantly less GPU VRAM and overall compute than a single forward pass of a 671B frontier MoE model.

Enterprise Integration and Data Flywheels with AI Fetchly

For organizations building automated financial reasoning, technical data verification, and agentic workflows, rStar-Math provides an ideal paradigm: small, ultra-fast models executing verified deep thinking on private infrastructure.

1. Grounding MCTS Rollouts with Enterprise Live Data

In real-world enterprise applications, mathematical problem-solving is rarely purely theoretical; it involves real-time asset pricing, regulatory filings, or unstructured market data:

• Integrate unified API integration to supply external market numbers directly into the MCTS search state.
• Use real-time data fetching to retrieve live database schemas and statistical indicators.
• Automate document verification across financial PDFs and technical specs with automated web scraping.

2. High-Throughput Deployment on vLLM and TensorRT-LLM

Because rStar-Math relies on branching tree search during inference, optimizing KV cache sharing is critical:

• Tree-Branch KV Cache Sharing: Deploy using serving engines like SGLang or vLLM that support Radix Attention. Sibling nodes in the MCTS tree share identical prefix KV caches, reducing memory consumption by over 60%.
• Speculative Rollout Acceleration: Use small 3.8B models for fast tree exploration, reserving deeper simulations for high-ambiguity prompts.

For organizations evaluating enterprise inference economics, explore our flexible pricing to calculate throughput and GPU savings, or consult our developer documentation for complete integration guides and SDKs.

Summary and Key Takeaways

rStar-Math marks a watershed moment in open-source artificial intelligence:

• SLMs Can Rival Frontier Giants: 3.8B and 7B small language models achieve 90.0% on MATH and 53.3% on AIME 2024, proving that parameter scale is not the sole determinant of reasoning ability.
• Code-Augmentation Eliminates Computational Drift: Pairing natural language reasoning with sandbox Python execution guarantees mathematical and logical precision.
• Process Preference Modeling Outperforms PRMs: Pairwise preference learning from MCTS win rates overcomes the noise and bias of conventional step-level scalar scoring.
• Self-Evolution is the Future of Post-Training: Autonomous self-improvement across iterative rounds provides a scalable path toward super-human reasoning without human annotation bottlenecks.
• Enterprise Ready: Coupled with high-speed data pipelines from AI Fetchly, rStar-Math enables organizations to deploy verified, private, low-latency reasoning engines.